Searching for High-Value Molecules Using Reinforcement Learning and Transformers
Authors: Raj Ghugare
Santiago Miret
Adriana Hugessen
Mariano Phielipp
Glen Berseth
Venue: ICLR 2024 poster (2024)
Keywords: chemistry
reinforcement learning
language models
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: A new RL algorithm for better molecular discovery.

Abstract: Reinforcement learning (RL) over text representations can be effective for finding high-value policies that can search over graphs. However, RL requires careful structuring of the search space and algorithm design to be effective in this challenge. Through extensive experiments, we explore how different design choices for text grammar and algorithmic choices for training can affect an RL policy's ability to generate molecules with desired properties. We arrive at a new RL-based molecular design algorithm (ChemRLformer) and perform a thorough analysis using 25 molecule design tasks, including computationally complex protein docking simulations. From this analysis, we discover unique insights in this problem space and show that ChemRLformer achieves state-of-the-art performance while being more straightforward than prior work by demystifying which design choices are actually helpful for text-based molecule design.

SEARCHING FOR HIGH- VALUE MOLECULES USING REINFORCEMENT LEARNING AND TRANSFORMERS
Raj Ghugare! Santiago Miret 3
Adriana Hugessen 1.2
Mariano Phielipp'
Glen Berseth 1.2
Universitéé de Montréal
Mila Quebec Al Institute
3 Intel Labs
raj. -aj-ghan geuggggggremmmi 1a quebec santiago mi ret@intel.................a..e.e com, adriana www. htwphsuuulluuuuseeeeeemmmii hugessen@mila. queebee
ABSTRACT
Reinforcement learning (RL) over text representations can be effective for finding high-value policies that can search over graphs. However, RI. requires careful structuring of the search space and algorithm design to be effective in this challenge. Through extensive experiments, explore how different design choices for text grammar and algorithmic choices for training can affect an RL policy's ability generate molecules with desired properties. We arrive at new RL-based molecular design algorithm (ChemRLformer) and perform a thorough analysis using 25 molecule design tasks, including computationally complex protein docking simulations. From this analysis, we discover unique insights this problem space and show that ChemRL.frmer achieves satte-of-the-ar performance while being more straightforward than prior work by demystifying which design choices are actually helpful for text-based molecule design.
INTRODUCTION
Molecular discovery can have significant impact on our society, however, the vast search space makes it challenging to find high-value molecules. The potential of reinforcement learning (RL) methods to discover new, high-value molecules has resulted in a series of research work performed
by RL researchers focusing on learning policies as graph neural networks (GNNs) Youu al. 2018
Zhou et al. 2019 Jin et al. 2020 Fu et al., [2022: Yang et al. 2021 Bengio et al. (021)). In this formulation, the RL policy trained add atoms and bonds molecular graph representation. In this formulation there is a one-to-one mapping between molecules and their graph representation, making it easier to construct state and action spaces with Markovian dynamics. However, the action space in the graph formulation is vast as it consists of the product of candidate attachment positions and candidate attachment sequences. Graph-based data structures (such as adjacency matrices, powerful representation used describe number of design problems. including social networks (Tan et al. al.] (011)) transportation networks (Wang and Tang (202]). recommendation systems (Chen al. 2022bb), and combinatorial optimization problems (Khadka al.) 2020 Miret jet al., [2022) have been popular in this design space. However, GNNs are often difficult to train (Chen al. 2022) and cannot readily take advantage of large-scale text data that effectively describe molecular structures and properties.
trees, etc.)
In order take advantage of the richness of text-based representations for molecules, one can formulate the molecular search problem as the construction of tokens in a sequence that become molecular text. The molecular texts formulated by common text-based representations, such as SMILES [Wininger] 1988) and SELFIES (Krenn et al. (222)). can then be converted into molecular graphs with cheminformaties libraries (Landrum et al., (003)) using their respective encoding and decoding rules. However, the text-based representation can be more difficult to formulate as an MDP since there not always an exact one-to-one mapping between texts and molecules. In fact, the text-to-molecule conversion can be many-to-one, where the complexity of the dynamics in the MDP given by many-to-one mappings is non-trivial. On the other hand, the action space in molecular text design be significantly reduced given the rules of text construction imposed by a given representation. Moreover, formulating molecule discovery as sequence-generation generation has the potential to capitalize recent successes in natural language modeling (Brown et al. 2020a).

---

In this paper, we perform detailed empirical study of molecular discovery using text-based RL across more than 25 molecular properties relevant for stug-discoveryy including docking simulation for molecular ligands (Garcia-Otteóón
own based on state-of ofe-r---t literature as shown in Table | In our experiments, we evaluate two molecular text representations (SMILES, SELFIES) and the use of three neural network architectures (Multi-Layer Perceptron (Bengio et al. 2003). Recurrent Neural Network (Schmidi, 2019). Transformer /Vaswani al. 2017)) pretrained on 5 datasets of varying quality and sizes. We create ChemRLformer that achieves the highest performance across these tasks while being much simpler than previous text-based RL algorithms (Blaschke et al.] 2020a] Gao et al. 2022b). Via our detailed ablation study, we construct ChemRL.fomee and find that pretraining on aligned datasets can significantly improve performance across all molecular design tasks, even exceeding the performance of agents pretrained on 100 times larger datasets. We also show that targeted algorithmic design, such as hill-climbing in the replay buffer and regularization, further increases the performance of ChrmRLformer. To the best of our knowledge, ChemRL.ormer is the largest analysis of text-based RL methods for molecule discovery.
al., 2022] Huang et al.] 2022) and develop
algorithm
Table 1: Table showing conceptual comparisons of various text based molecular optimization methods. ChemRL combines the most successful elements of prior work.
Method
Text Representation
RI.
Architecture
Pretraining
Algorithmic Components
SMILES VAR
Goomez-Romburel et al.
2018
SMILES
X
VAE LSTM
Maximum Likelihood
SMILES
LSTM Biowa di al
2019
SMILES
X
Maximum Likelihood
BOSS
Mois
SMILES
VAE
Bayesian Optimization
REINVENT
Blasclkk d a
20200
SMILES
GRU
Replay buffer, KL.
REINVENT 2.
Hiasehie
SMILES
GRU
MC-Replay buffer. Log p. KI.
Taiga 2.0
Mazaz & al. 21/3
SMILES
TRANSFORMER
policy gradients
MoGGPT
Bagal
SMILES
V
TRANSFORMER
V
policy gradients
STONED
Nigani ei al
SELFIES
X
FC
X
Genetic algorithm
Pasithea
Shzn ml
SELFEES
PC
Deep dreaming
ChemRL.ormer (Ours)
SMILES, SELFIES /
Transformer. FC
Replay buffer, KL
2 RELATED WORK
RL for Design and Discovery: Many methods in diverse fields leverage RL to help augment a prior design method to improve performance (Yu al.] 2018 Schaff al. 2019). Other methods have explicitly included the design process in the RI. loop by training design problems together (Chen) jet al. 2 aa Ha 2019 Luck al., 2020 Kumar et al, 2022) with most prior work focusing on
robot and agent design, not molecular design. Our molecular design work creates an autoregressive structure that grows the size of the state as the agent acts in the environment.
Molecular Discovery Using Sequence-Ban Based Methods: Sequence- methods treat molecular design as a sequence of tokens that get concatenated in order. Generative models for sequence- based methods span diverse range, including variational autoencoders (VAEs) (Gómez Bom borellllli al. 2018 Alperstein et al.) 2019). recurrent neural networks (RNNs) (Gupta et al.) 2018 Bjerrum and
Threlfall 2017: Grisoni et al. 2020 Flam-Shephed
2022 and transformer models( Wang et 2019 Fabian et al. 2020 Edwards et al.) [022a]] Zeng et al. 2022 Taylor et al. 2022). The
al.]
general procedure for all the above methods to perform self-supervised generative learning to
sample molecules similar to the original dataset. MoLRL can also make use of pretrained generative models, which we then fine-tune using reinforcement learning produce enhanced molecules.
Molecular Discovery Using Search- Based Methods: Although sequence-based molecule
generation methods often provide a more structured way of learning molecular distributions, search- based methods generally have the advantage of being able to directly find molecules based on a desired property. Although a wide range of graph-based RL methods (You et al., 2018 Zhou et al. 2019; Jin et al., 2020: Fu et al. 2022, Yang et al. [ et 2021) for optimizing molecules exist, graph-based state representations introduce significant complexity to the RL problem formulation, both in the transition dynamics and action space. By contrast, text-based methods are simpler and also relatively under-explored, motivating our focus on these methods in this work. Moreover, recent work (Cieplinski et al. 2021 Gao et (022b) has shown that older text-based method REINVENT (Olivecrona et al. 2017) outperforms more complex graph-basee RL methods. Some limited extensions to Olivecrona al. (2017) have been explored, including
2

---

experimenting with newer molecular grammar designed for robust molecule generation (Gao et al. 2022b). However, there has been limited work proposing the use of language models and text-based RL for molecular discovery. Additionally, there have been limited efforts to incorporate recent advancements from the language modeling domain into these methods. For example, the a character- level LSTM network architecture used in [Olivecrona al. (2017), has not been revisited despite significant recent advances in sequence modeling (Vaswani ct al. 2017 Brown et al.) 2020b).
3
BACKGROUND
The algorithms detailed in this paper are built on top of a foundation of reinforcement learning, text-based molecule representations, and language modeling.
Reinforcement Learning: Reinforcement learning can be used to learn policies for sequential decision-making problems. Policies are optimized based on an environment that is described as a Markov Decision Process (MDP). discrete MDP defined by the tuple AS, 7) where S is the state space, A is the action space, T S x A x S' + [0, 1] is the transition function, r: r:Sx x A + R is the reward function and y is the discount rate.
For actions E A and states 8t E S, the goal of reinforcement learning is to learn policy To (st) which maps states to actions, such that:
T Σ
To(a, (x)) = arg max E (
7'r(st, at)
(1)
0
where p(r(0) is the distribution over trajectories induced by hhe and the transition function T.
Text representations for molecules: Molecules are most naturally described using a graph
structure of atoms and bonds. However, graph-based deep learning models can be difficult train,
especially at large scale (Dwivedi et al. 2022 Geisler et al. 2023). Recent works have proposed a
variety of text representations for molecules [Weininger]] 1988 Krenn al. 2020 |020 Heller et al. [013] Krenn et al.) 2022; Cheng et al.) (023), each having their distinct advantages and shortcomings.
In this study, we focus on the two most commonly used representations: SMILES (Weininger) 1988) and SELFIES (Krenn et al. 2020). Any text representation for molecules consists of a set of valid tokens, which may represent individual atoms or special characters that imply the presence of certain structures, as well as the encoding and decoding rules needed to convert between the text representation and the graph representation of a molecule. Valid texts under a grammar are those which respect both the vocabulary and the encoding/decoding rules for that grammar and, hence, can be converted into a graph representation of a molecule. SELFIES, which was developed in response to the tendency for SMIL.E-bbased deep learning models (Gó mez-Bommarelli al. 2018 Jin al.) 2018) to generate invalid molecular texts, has the useful property of providing a conversion for any text into a graph corresponding to a molecule, provided the tokens the text respect the SELFIES vocabulary. For example, the text representation of Benzene in SMILES is CI=CC=CC=CI while in SELFIES one possible representation is [CI==CCCCI- ][=Branch]]]
Language modeling: Language modeling often relies on the self-supervised task of next-token prediction for model pretraining. The general framework for next-token prediction is to train a model predict the next token a sequence autoregressively, i.e. given the previous tokens the sequence (left context). Many architectures to handle sequential data have been proposed: Recurrent Neural Networks (RNNs) (Hochreiter and Schmidhuber) 1997, Rumelharr and McClelland, 1987) are a class of models used in sequence modeling which use recursive connections in hidden layers to accumulate the left context for next-token prediction. Transformers are a more recent architecture that instead use a self-attention mechanism Vaswani et al. 2017) to capture dependencies between all tokens in a sequence. For next-token prediction tasks, attention masking is used to enforce left context, meaning that representations for tokens later in the sequence are only allowed to attend to previous tokens in the sequence. In Section[4ww we outline how we pretrain an autoregressive sequence model to predict sequences of known molecules.
3

---

Actions
01
02
03
04
0,5
46
07
E05
Palicy
805
Benzene (Cg Hg)
80
81
#2
#3
84
#5
86
87
States
Autoregressively generating benzene molecule.: An autoregressive model for sequence generation can be viewed as an RL policy where the actions a, are the next tokens to append to the sequence and the state is the concatenation of all actions taken up to time 1. A special ond-of-sequence token can terminate the episode early at time T. The text at time sT is then converted into molecule based the text-representation grammar and then scored according to scoring function that measures the alignment of the molecule with the desired properties informed by the application. Hydrogen atoms added at the end to complete the structure.
Figure 1:
4 CHEMRLFORMER GENERATING MOLECULAR STRINGS VIA REINFORCEMENT LEARNING
The space of drug-like molecules vast, and reinforcement learning methods hold great promise in improving the speed and reducing the cost of drug discovery. In this section, we describe ChemRLformer and how combining language models and tools from RL produces a sota algorithm.
MDP for molecule generation: The vocabulary and grammar for text representations of molecules can be interpreted as an MDP as described in Section where the states 8; correspond to variable length text of accumulating tokens, and the actions a₃ correspond to vocabulary defined by the text-representation. The transition function a deterministic function where the action at taken by the agent is appended to the end of the state 8 resulting in 8t+1 using the dynamics S1+1 [st, 0z] + T(sr, at). However, the corresponding transition function induced in the graph representation of molecules is more complex as it is determined by the encoding/decoding rules of the chosen text representation. For example, in the SMILES grammar, a random concatenation of valid molecule, while the SELFIES grammar constructed such that any ordering of its tokens is encoded as valid molecule.
tokens may correspond
Finally, the reward function R scores molecules according their alignment with desired chemical properties, which can involve complex material simulations. The underlying property computation of the reward function further informs the dynamics of the MDP imposed by text representation. For example, docking scores are used to estimate the binding affinity between ligands and protein targets. We discuss reward functions for molecules in more detail section Section
Pretraining policies for molecule discovery. To advance effectively within this vast search space, we make use of datasets containing large number of drug-like molecules in text format (Irwin et al.) 2012 Sterling and Irwin 2015b; Mendez al., 2019). This data is used to train autoregressive model predict tokens that conform to the grammar for drug-like molecules. instead of the random texts that are generated from a randomly initialized policy, thereby significantly simplifying the exploration problem. In particular, we pretrain network Ps on the superviee objective of next-token prediction. Although large language models can be trained with other objectives, such as corrupted text reconstruction (Edwards et al. (2022),, these models are a good fit for our purposes since they cannot generate diverse and valıd molecules without access to carefully designed prompts.
min EAND
log pola = As A Ao)
In practice a minibatch of sequences {A) .... An' ".....Am are sampled from the prior dataset D to evaluate the loss function in Equation and the parameters are trained using gradient descent.
4

---

4.1 RL FOR MOLECULE GENERATION
To generate a molecule, a ChemRLformer policy ao(a; | 8t)) is allowed autoregressively sample tokens for a fixed number of timesteps H. The start state so is always a [BOS], and the agent can terminate early by taking the end-of-sequence action [OS]. Figure I shows how RL policy can construct a Benzene molecule. Since properties the final molecule, there are no intermediate rewards and the goal of the RL policy is to maximize the expected scalar reward corresponding to the final constructed molecule, r(ST). Thus, assuming a discount rate 7
token
only interested in the
Equation |
can be rewritten more simply as:
max Bar [r(sr]]
where ST = [BOS]
[BOS], is sampled autoregressively from the policy.
Our experiments use the policy gradient algorithm (Sutton et al., 1999b) to train the RL policy because it is known to achieve state-of-the-ard performance amongst RL for molecular optimization (Olivecrona al. 2017). Deep RL policies are able to learn the non-linear global structures of molecular texts which, as we show in section [ enables them to generalize to novel and diverse molecules. However, training RL policies from scratch time-consuming and can make the exploration problem infeasibly difficult. Next, we explain how adapt recent language modelling techniques to pretrain the RL policy.
RL fine-tuning The pretrained model can directly be used sample novel drug-like molecules. These molecules, however, are not optimized for any particular property. Note that given our definition of the state s; as the concatenated history of all previous actions, this pretrained network is exactly analogous to the policy network in Equation [ Hence, by initializing To Ps. and Φ. we can fine-tune this pretrained network by optimizing Equation via the policy gradient algorithm REINFORCE (Sutton et al. 1999a). We need only to define reward function r(sT) which scores molecules according to their alignment with the desired properties. In the following experiments, show that this fine-tuning is vital for ChemRLformer to sample better molecules. We also highlight the importance of pretraining and study how the size and quality of the prior data affect the downstream ability of RL to search for high-value molecules.
EXPERIMENTAL RESULTS
Our proposed algorithm CormRL uses the best combinations of choices resulting from assessing the performance across three dimensions: (1) what pretraining factors are important to improve RL for molecular discovery (Section 5.2). (2) how the use of recent text-based molecule grammars facilitates downstream RL exploration (Section 5.3); and, lastly, (3) which specific algorithmic changes are necessary improve RL performance (Section 5.4p
5.1
EXPERIMENTAL SETUP
Tasks. We evaluate ChemRLformer against five different docking targets (Alhossary al.
2015)
(fa7, 1a 5ht1b, jak2, and braf) previously explored in the literature (Yang et al. 202 Lee
al. 2023). The docking scores used to estimate the binding affinity between ligands and protein targets are a complex function of the global molecular structure and have been proposed as chemically relevant benchmarks for molecule design algorithms (Cieplinski et al. 2021 Tripp al., 2022). In addition to the docking targets, we also evaluate on 22 pharmaceutically--reee..
oracle
functions (Huang et al. 202I; Gao et al. 2022b Brown al. 2019) (pytde tasks), which include tasks such as optimizing proxies of bioactivity, similarity to target molecules, and combinations of multiple physiochemical drug properties.
Evaluation metrics. We design our evaluation procedure with the final goal of identifying the best candidates to test in a wet lab. To discover such high-value candidate molecules, we use sota simulators that assign rewards to molecules by performing complex docking simulations (Alhossary et al. 2015) or using proxy models and chemical rules (Huang et al. 2021). Previous works limit the
Because we operate in finite MDP, a discount factor of 1 gives us an unbiased estimate of the true objective.
5

---

number of molecules sampled during evaluation around 3000 for docking tasks (Garcia-Ortegón Ortegón
iet al. 2022 Yang et al. [0021 I Lee et al. 2023) and 10000 for pytde tasks (Gao et al. |002bb Brown et al.
2019 due to the computational cost associated with these reward simulators. We allow up to 25000 unique oracle calls and up to 40000 total oracle calls (allowing repeats). We argue this better reflects the lower cost and availability of computing resources relative to wet-lab resources. From all the sampled molecules, the average score of the top-k (A: = 1, 10, 100) molecules is used as a performance metric. These top groups are an estimate of the algorithm's 's ability to discover group of top-quality candidates that could be given to a wet lab for thorough testing. We report pytde scores on a normalized basis between zero and one by default. Next, we normalize all docking scores by dividing them by -20 in our experiments. Additionally, we report diversity (Div.), defined as the averaged internal distance of the top 100 molecules, and redundancy (Red.), defined as the total number of oracle calls that an agent makes for an already evaluated molecule.
Pretraining. We study how the quality and size of prior data affect the downstream RL performance of ChemRLformer by pretraining GPT (Radford al. 2018) style transformer model on five datasets of varying sizes and quality and using the pretrained model as an initialization for the RL agent's policy network. See Table[2for the name, size, and description of all datasets used in our work. We also rank all datasets based on their quality on docking and pytdc tasks. We determine the quality of a dataset by the performance of molecules sampled from the model pretrained on that dataset. The quality of ChemRL.ormer ' pretrained model is evaluated using the top-100 molecules sampled by the pretrained model under the same evaluation setup in Appendix A.2 By default, these open-sourced datasets contain a large number of drug- like molecules in SMILES format. For our experiments, we also convert all datasets to the SELFIES format. Lastly, three different architectures are compared: fully-connected (FC), recurrent (RNN) GRU and transformer GPT style autoregressive model, and compare them on downstream RL tasks.
Table 2: Description of molecular datasets used for pretraining: Datasets are ranked according to procedure described in Section Two datasets have the same rank if their average performance lies inside one standard error of the other. The datasets are drawn from a subset of the Zine (Sterling and Irwin 2015b Irwin et al. 2022) and ChemBI. (Gaulton et al. 2012) databases.
Dataset
Size
Docking Rank
Pytde Rank
Description
Manually eurated database of bioactive molecules with drug-like properties (Gaulton al. 2012
CHEMBL
1.2M
ZINC datubase molecules curated for their pharmaccutical relevance and popularity (Gao et al. 2022h
ZINC 250K
250 K
2
2
ZINC 1M
M
Random molecules from ss 1.5 hillion molecules from the ZINC database (Sterling and Irwin 2015a ZINC IM C ZINC 10M ZINCTOUM.
ZINC 10M
10M
ZINC 100M
100
All of our experiments on pytdc tasks are run across 5 random seeds. Since docking simulations are expensive and time consuming, we run all docking experiments across random seeds. Experiments with different seeds use the same pretrained model which is only pretrained once for every dataset. Additional details about the task rewards, evaluation metrics, and the pretraining datasets and models discussed in Appendix A. A.2 and A.3 respectively.
5.2 HOW DOES PRIOR DATA AFFECT THE FINAL PERFORMANCE OF CHEMRLFORMERT
In this section, we pretrain the REINFORCE policy on datasets of varying size and quality from Table 2 Our datasets vary from small (250K) to very large (100M) sizes. Due to the parallelizability of training on larger datasets, we use the transformer policy architecture for all experiments in this section. In natural language processing (NLP), pretraining transformer models on large and diverse unlabelled datasets have been found to perform well on downstream tasks using few-shot labeled data (Brown et al. 2020b). Yet, our results in Figure 2b indicate that the quality of the prior dataset matters more than its size. We compares the scores of the molecules generated by the policy after pretraining and after RL training. Figure shows that the molecules sampled the ChEMBL dataset achieves higher scores, and hence is more aligned with both the pytde and the docking tasks. As a result, the RL agent pretrained on the ChEMBL dataset outperforms all other agents, including the ones trained on 100 times more data.
after pretraining

---

CME
ZINC 250K
ZIC IM
ZINC 10M
ZINC 100M
Pretraining performance on PYTDC
pretraining
500r
0.50
Score
0.90
100
Dlv.
Red.
(a) Performance SMILES based molecular design with pretraining (left) and with pretraining and RL (right).
Docking atter pretraining
performance on Augmented Docking after pretraining
0.75
0.75
0.25
0.25
T0p
Top 10
Top 100
Drv.
Red.
10p
Top 10
Top 100
Dou
Red.
(b) Performance SMILES- based molecular docking with pretraining (left) and with pretrainig and RL (right). Section describbs augmented docking setting with additional experiments shown in Appendix A.3 Div. and Red, the diversity and redundancy
described section (5T respectively
On the left pretrained performance datasets, such ChemBL. lead to higher-performance for both pytde and augmented docking. On the right is the performance after RL training. RL has substantial benefit for pytde tasks, while for docking tasks an augmented docking score is used to avoid reward hacking, see Figure for details.
Figure 2:
ESS-based ChemRL.frrmer Higher-quality
Results may seem surprising from an NL.P perspective, but they make sense when viewed from an RL perspective. Pretraining using next token prediction Equation ( is analogous to behavior cloning in this context, where the performance depends largely on the quality of the offline dataset (Ross et al. 2011] Ho and Ermon 2016). These results suggest that ChemRLformer might benefit from better pretraining objectives, that go beyond simple imitation learning. when trained on
large and diverse offline datasets (Kumar et al. 2023 Farebrother et al. 2023
5.3 TEXT REPRESENTATIONS AND ARCHITECTURES FOR CHEMRLFORMER
Starting with REINFORCE agent, we isolate the effect of various representations for molecules and policy network architectures on performance. All experiments in this section use ZINC-250k dataset for pretraining. Similar results obtained for other datasets are shared in the following sections. Whenever we show normalized results across different experiments, we add the individual plots Appendix C.I
Text representations. In Figure architectures and tasks across environments that base their dynamics on SELFIES and SMILES. The results show normalized scores across all architectures. Consistent with prior work (Gao al.) 2022b) we find that SMILEE-based polices generally outperforms SELFIES-baan all pytde tasks and architectures, ChemRL.former agents based on SMILES consistently achieve better rewards when compared to SELFIES-base agents across all reward metrics. Although more subtle, we observe a similar theme the docking tasks where SMILES achieves higher rewards than SELFIES on all top-K metrics. Another consistent theme in the results is that even though the diversity of top-100 molecules obtained by SELFIES is higher, the redundancy of SELFIES agents higher as well. This means that SELFIES based ChemRLformer agents explore much smaller region of the molecular space. These results suggest that the rules which allow SELFIES strings to always be converted into valid molecule can actually be detrimental the agent's exploration and search for high-value molecules, more details in Appendix C.I
compare ChemRLformer agents using different
policies. On
Architectures.
The results in Figure 4 show that the transformer and RNN have similar performance on all tasks. On the pytdc tasks, FC achieves worse performance than other

---

SELFIES SMILES
scoe
0.30
6.23
0.55
Top
Top
Top
Figure 3: Comparison between SELFIES and SMILES: The SELFIES representation makes relatively difficult for ChemRLformer agents to explore effectively leading to generally lower performance pytde and docking while scoring higher on diversity. Scores are reported for the transformer model and are averaged across all reward functions.
architectures specially made to handle strings, expected. However, docking tasks, FC obtains unusually high rewards. We find that this method performs type of reward function hacking (Amodei al. 2016) [Skalse al. 2022, Everiu (019) by exploiting corner case of the docking-based reward function which provides high rewards for long strings Carbon and Nitrogen atoms together. To evade the reward hacking of docking scores, we constructed an augmented docking score function with commonly used oracles (QED and SA scores) based on previous work (Lee et al., (023) (See Appendix AA.3 for more details). This finding shows that the REINFORCE agent can search the space well and, in this case, can be used to expose issues with current design of reward functions.
RMN
Transfommes
FYTDC
DOCKING
0.23
Top
Top 10
Top 100
Dra
Red.
kop
T5p 10
Top 100
Div.
Bed.
Comparison of different policy architectures: No single architecture clearly outperforms for molecular ChemRLformer. Although FC does better on the docking tasks, our analysis shows that it learns exploit the docking function opposed designing high-value molecules. More details about ways to tackle this issue are given in Appendix C.3 experiments for comparing transformers and RNNs are shown Appendix C.5 These experiments use the smiles text representation.
Figure 4:
Additional
5.4 REVISITING RI. ALGORITHM DESIGN CHOICES FOR CHRMRLFORMER.
Replay buffers and hill climbing. In off-policy deep RL, replay buffer is generally used to store and reuse previous trajectories for training. Although text-based RL algorithms are trained on-policy, prior work has proposed using replay buffer improve performance Blaschkee et al. (20205). Standard replay buffers Mnih al. (2013) throw away the oldest trajectories as newer ones arrive. But many text-based RI. algorithms propose to use hill-climb replay buffers, that randomly sample batch of molecules from the highest scoring molecules seen so far and add them the current mini batch Thomas et al. (2022). In Figure 5 we that using the hill-climb buffer results in significant performance boost for ChemRLformer, whereas using a standard buffer does not contribute much. Notably, the use of hill-climb replay buffer reduces diversity and increases redundancy quite substantially. The following two experiments involve combining regularisation terms with the RL objective in Equation 3 The coefficients these extra terms largely affect the final performance. To make fair comparison,
HIl Climb Replay Buffer
Rap Suffer
No Replay Buffer
PTDC
Top
"Bp 10
Top 100
Dlv.
Red.
Figure 5: Hill climbing buffer lead 13%
improvement in Top-100 rewards.
8

---

perform hyper-paraman tuning over six different values for every new regularisation term with details provided in Appendix
Should the policy to stay close to the pretrained model? Pretrained models carry information on how to build valid drug-like molecules. To ensure that ChemRL agents do not stray far away from the space of valid drug-like molecules during exploration, Olivecrona et al. (2017); Gao et al.] (2022b)
KL Prior
No KL Prior
PrTDC
2.30
scorm
constrain the KL divergence between the policy and the pretrained model by adding a KL penalty the policy gradient loss function in Equation (3). Prior works show that adding this penalty helps the agent achieve better sample efficiency (Gao et al., 20226). Yet, our results Figure 6 suggest that, when you increase the number of oracle calls in simulation, adding this penalty does not yield any additional benefit while substantially increasing the GPU memory requirement, especially when using larger models. Since invalid molecules correspond to zero rewards, the ChemRLformer agent is able to learn to avoid invalid structures on its own merit.
Top 1
Top Top 100
Div.
Red.
Figure Our experiments show little difference in
performance for multiple KL regularization terms.
LogP
Entrapy
No Regulartzation
Regularizing the policy's likelihood for exploration. RL agents classically face an exploration-exploina. dilemma, which can lead to agents getting stuck sub-optimal local maxima when not well balanced. ChemRLformer agents are not immune to this dilemma.
PYTDC
Upon encountering good, but sub-optimal molecules, agent may adjust its policy to increase the likelihood of sampling these sub-optimal molecules and, without sufficient exploration, fail to discover higher-value regions of policy space. This can be particularly detrimental during the initial learning stages.
0.82
Top
Figure Different likelihood penalization for exploration. Log P regularization is a better choice for efficient exploration for ChemRI.former.
To combat this issue, entropy regularisation, which adds a log (s) term to the RL loss, has been proposed (Haarnoja et al. 2018). This encourages the RL policy explore states with lower likelihood values. Similarly (Olvecrona al.) 2017) adds Log regularizer, which penalizes higher likelihood values by adding 1/ log (T()) term to the RI. loss. In Figure our results show that although an entropy regularizer leads to lesser redundancy, the Log regularizer boosts performance significantly by exploring more efficiently. The Log p regularizer
only penalizes the agent for being extremely certain (likelihood
tends to 1) about its actions, and mostly agnostic for lower likelihood values. This penalty is much better choice for ChemRLformer as it only activates when stuck in a local optimum of molecular space.
CONCLUSION AND FUTURE WORK
We present ChemRLformer that resulted from our empirical study of multiple algorithmic components of text-based molecular design. For future practitioners, our method suggests the following philosophy: (I) Using SMILES is better choice than SELFIES. (2) When collecting data for pretraining, the quality of molecules matter much more than the number of molecules. (3) Both transformer and RNN architectures achieve similar performance across all tasks using current datasets. (4) Incorporating components such as hill-climb buffer and Log P regularization yields substantial performance improvements. Conversely, introducing KL regularization or opting for more intricate actor-critic algorithms may result in diminished performance, at the cost of more hyperparameters and memory resources. While our analysis focused on model-free RL algorithms, learning reward model in sample efficient manner is an exciting area of future work. Our analysis also shows that RL agents were able to hack the reward functions suggesting that there is space to improve the metrics used for molecule quality.

---

REFERENCES
Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh. Fast, accurate, and reliable molecular docking with QuickVina 2. Bioinformatics, 31(13)2214-2216, 02 2015. ISSN 1367-4803, doi: 10. 10.100//oomfommatcc//b0006
bioinformatica//ttoooo
Zaccary Alperstein, Artem Cherkasov, and Jason Tyler Rolfe. All smiles variational autoencoder. arXiv preprini arXiv: 13343, 2019.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety, arXiv preprini
2016.
Viraj Bagal, Rishal Aggarwal, PK K Vinod, and U. Priyakumar. Molgpt: Molecular generation using transformer-decoder model. Journal of Chemical Information and Modeling, 62, 10 2021. doi: 10. 10.1021/acs.jcim. http://cs..cii...mo060000000 lc06600.
Emmanuel Bengio, Moksh Jain, Maksym Korablyov, Doina Precup, and Yoshua Bengio. Flow network based generative models for non-iterative diverse candidate generation, 2021. URL
https:///rxiis org/abs/////oooooooo/oo/// attaa//sss...../ 2216600399999
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. A neural probabilistic language model. J. Mach. Learn. Res., 3(null): 137-1115, mar 2003. ISSN 1532-4435.
Mostapha Benhenda. Chemgan challenge for drug discovery: can reproduce natural chemical diversity?, 2017.
Esben Jannik Bjerrum and Richard Threlfall. Molecular generation with recurrent neural networks (rnns), arXiv preprint arXiv: (el::1705.0440 0000,04462,00000 2017.
Thomas Blaschke, Josep Arús-Pous, Hongming Chen, Christian Margreitter, Christian Tyrchan, Ola Engkvist, Kostas Papadopoulos, and Atanas Patronov. Reinvent 2.0: an ai tool for de novo drug design. Journal of chemical information and modeling, (0(12):5918-5922 2020a.
Thomas Blaschke, Josep Arús-Pous, Hongming Chen, Christian Margreitter, Christian Tyrchan, Ola Engkvist, Kostas Papadopoulos, and Atanas Patronov. Reinvent 2.0: An ai tool for de novo drug design. Journal of Chemical Information and Modeling, 60, 10 2020b. 0c00915.
doi: 10.1021/acs.jeim.
Nathan Brown, Marco Fiscato, Marwin H.S. Segler, and Alain C. Vaucher. GuacaMol: Benchmarking models for de novo molecular design. Journal of Chemical Information and Modeling, 1996)-1996-1008 mar 2019. doi: 10. (0.1021/acs.jcin b0000 org/10.10211220
URL https://doi.
jeim. 8b00839
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In H Larochelle, M Ranzato, Hadsell, M F Balcan, and H Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020a. URL http::ss//pooceeennn....eee neurips wttppccccpppppeeeeeeeeeeeee files//aper/2020// #1e/2397Ccbbo0005/11188862222222
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vons Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020b.
10

---

Tianjian Chen, Zhanpeng He, and Matei Ciocarlie. Hardware policy: Mechanical and computational Co-Optimization using deep reinforcement learning. In Jens Kober, Fabio Ramos, and Claire Tomlin, editors, Proceedings of the 2020 Conference Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 1158-1173. PMLR, 2021a.
Tianlong Chen, Kaixiong Zhou, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, and Zhangyang Wang. Bag of tricks for training deeper graph neural networks: A comprehensive benchmark study. IEEE Transactions on Pattem Analysis and Machine Intelligence, 2022.
Xiaocong Chen, Lina Yao, Julian McAuley, Guanglin Zhou, and Xianzhi Wang. A survey of deep reinforcement learning in recommender systems: A systematic review and future directions. arXiv preprint ar arXiv:2109.03542 2021b.
Austin Cheng, Andy Cai, Santiago Miret, Gustavo Malkomes, Mariano Phielipp, and Alán Aspuru-Guzik. Group selfies: robust fragment-based molecular string representation. Digital Discovery, 2023.
Petros Christodoulou. Soft actor-critic for discrete action settings, 2019.
Tobiasz Cieplinski, Tomasz Danel, Sabina Podlewska, and Stanislaw Jastrzebski. We should least be able to design molecules that dock well, 2021.
Vijay Prakash Dwivedi, Chaitanya K. Joshi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. 2022.
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 375-413, Abu Dhabi, United Arab Emirates, December 2022a. Association for Computational Linguistics. URL https:// aclanthology. wwt.oog2222222 emnlp maaamll-mai.266
Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, and Heng Ji. Translation between molecules and natural language. 2022b.
Tom Everitt. Towards safe artificial general intelligence. PhD thesis, The Australian National
University (Australia), 2019.
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain- relevant auxiliary tasks. arXiv preprint arXiv:2011.13230. 2020.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G. Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks, 2023.
Daniel Flam-Shepherd, Kevin Zhu, and Alán Aspuru-Guzik. Language models can learn complex molecular distributions. Nature Communications, 13(1):3293, 2022.
Tianfan Fu, Cao Xiao, Lucas M. Glass, and Jimeng Sun. Moler: Incorporate molecule-level reward to enhance deep generative model for molecule optimization. IEEE Transactions on Knowledge and Data Engineering. 34(1 1):5459-5444 5471,,4711111 2022. doi: 10.1 10.1009/KDDE.2221.0000
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization, 2022a.
Wenhao Gao, Tianfan Fu, Jimeng Sun, and Connor W. Coley. Sample efficiency matters: A benchmark for practical molecular optimization, 2022b.
Miguel GGarcia-Ortegee Gregor NC Simm, Austin J Tripp, José Miguel Hernáneez-Lobtto Andreas Bender, and Sergio Bacallado. Dockstring: easy molecular docking yields better benchmarks for ligand design. Journal of chemical information and modeling, (2115)3386-223 2022.
Anna Gaulton, Louisa J. Bellis, A. Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, and John P. Overington. Chembl: A large-scale bioactivity database for drug discovery. Nucleic Acids Research, 40(D1):D1000DDI January 2012. ISSN 0008- doi: 10. 30.3///////

---

Simon Geisler, Tobias Schmidt, Hakan Şirin, Daniel Zügner, Aleksandar Bojchevski, and Stephan Günnemann. Robustness of graph neural networks at scale, 2023.
Rafael G6 mezz-Bombreelli Jennifer N. Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sanchez-Lengeling, Dennis Sheberla, Jorge Aguilera-Iparrguirre Timothy D. Hirzel, Ryan P. Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS Central Science, 4(2):268-276 jan 2018. doi: 00.1022/accceensee7700000033 URL httth::////// .or.ii.rg///ooo 102/110.1111111 F0022222aaccctttsssss 7bC0572
Rafael Gómez Bonbb--lo,aarlli,, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato, Benjamín Sánchez-Lengeling, Dennis Sheberla, Jorge aguilera-Iparragun Timothy Hirzel, Ryan P Adams, and Alán Aspuru-Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268 276, 2018.
Francesca Grisoni, Michael Moret, Robin Lingwood, and Gisbert Schneider. Bidirectional molecule generation with recurrent neural networks. Journal of chemical information and modeling, 60(3): 1175-1183, 2020.
Anvita Gupta, Alex T Müller, Berend JH Huisman, Jens A Fuchs, Petra Schneider, and Gisbert Schneider. Generative recurrent networks for de novo drug design. Molecular informatics, 37 (1-2):1700111, 2018.
David Ha. Reinforcement learning for improving agent design. Artif. Life, 25(4)3352-365, November 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Stephen Heller, Alan McNaught, Stephen Stein, Dmitrii Tchekhovskoi, and Igor Pletnev. Inchi-the worldwide chemical structure identifier standard. Journal of cheminformatics, 5(1):1-9, 2013.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 9(8): 1735-1780, nov 1997. ISSN 7669.7667. doi: 10.1162/neco. 1997.9.8.... 9..77.98.1735. URL https:
//doi.org/10 www2//eco..... nww.. 1997.9999 9. 8.175
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovee, Connor W. Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapcutics data commons: Machine learning datasets and tasks for drug discovery and development, 2021.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovee, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Artificial intelligence foundation for therapeutic science. Nature Chemical Biology, 10110)1003-100 2022.
John J Irwin, Teague Sterling, Michael M Mysinger, Erin S Bolstad, and Ryan G Coleman. Zine: free tool to discover chemistry for biology. Journal of chemical information and modeling, 52 (7):1757-1768. 2012.
Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. Chemformer: pre- trained transformer for computational chemistry. Machine Learning: Science and Technology, (1):015022, 2022.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation, 2018.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
12

---

Shauharda Khadka, Estelle Aflalo, Mattias Marder, Avrech Ben-David, Santiago Miret, Shie Mannor, Tamir Hazan, Hanlin Tang, and Somdeb Majumdar. Optimizing memory placement using evolutionary graph reinforcement learning. arXiv preprint arXiv:2007.07296 2020.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik. Self-referencing embedded strings (SELFIES): A 100% robust molecular string representation. Machine Learning: Science and Technology, 14))044022, oct 2020. doi: 10. (0.1088/2632-2155) aba947. URL httphttts/://oi.orre
1ww.:118882222 108882F2662222
Mario Krenn, Qianxiang Ai, Senja Barthel, Nessa Carson, Angelo Frei, Nathan C Frey, Pascal Friederich, Théophile Gaudin, Alberto Alexander Gayle, Kevin Maik Jablonka, et al. Selfies and the future of molecular string representations. Patterns, 3(10):100588, 2022.
Aviral Kumar, Amir Yazdanbakhsh, Milad Hashemi, Kevin Swersky, and Sergey Levine. Data- driven offline optimization for architecting hardware accelerators, 2022.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes, 2023.
Greg Landrum et al. Rdkit: A software suite for cheminformatics, computational chemistry, and predictive modeling. Greg Landrum, 8, 2013.
Seul Lee. Jachyeong Jo, and Sung Ju Hwang. Exploring chemical space with score-based out-of- distribution generation, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017.
Kevin Sebastian Luck, Heni Amor, and Roberto Calandra. Data-efficient Co-Adaptation of morphology and behaviour with deep reinforcement learning. In Leslie Pack Kaelbling. Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 854 869. PMLR, 2020.
Eyal Mazuz, Guy Shtar, Bracha Shapira, and Lior Rokach. Molecule generation using transformers and policy gradient reinforcement learning. Scientific Reports, 13, 05 2023. doi: 10.1038/ 0011598003-35660
David Mendez, Anna Gaulton, A Patrícia Bento, Jon Chambers, Marleen De Veij, Eloy Félix, María Paula Magariños, Juan F Mosquera, Prudence Mutowo, Michat Nowotka, al. Chembl: towards direct deposition of bioassay data. Nucleic acids research, 47(D1):D930-DD D940, 2019.
Santiago Miret, Vui Seng Chua, Mattias Marder, Mariano Phiellip. Nilesh Jain, and Somdeb Majumdar. Nburoevolution-enhanceee multi-objective optimization for mixed precissoe quantization. In Proceedings of the Genetic and Evolutionary Computation Conference, pages 1057-1065, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013.
Henry B. Moss, Daniel Beck, Javier Gonzalez, David S. Leslie, and Paul Rayson. Boss: Bayesian
optimization over string spaces, 2020.
AkshatKumar Nigam, Robert Pollice, and Alan Aspuru-Guzike Janus: Parallel tempered genetic
algorithm guided by deep neural networks for inverse molecular design, 2021.
Mareus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de novo design through deep reinforcement learning. 2017.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022.
13

---

Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant M. Jayakumar, Max Jaderberg, Raphael Lopez Kaufman,, Aidan Clark, Seb Noury, Matthew M. Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zach Vito,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ily Sutskever. understanding by generative pre-training. 2018.
Improving language
Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011.
David E. Rumelhart and James L. McClelland. Learning Internal Representations by Error Propagation, pages 318-362. MIT Press, 1987.
Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew R Walter. Jointly learning to construct and control agents using deep reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pages 9798-9805. ieeexplore. iere.orr.. May 2019.
Robin M Schmidt. Recurrent neural networks (rnns): gentle introduction and overview. arXiv preprint arXiv:1912.0591,, 2019.
Cynthia Shen, Mario Krenn, Sagi Eppel, and Alá n Aspuru-Guzik. Deep molecular dreaming: inverse machine learning for de-novo molecular design and interpretability with surjective representations. Machine Learning: Science and Technology, 2(3):03LT02, jul 2021. doi: 10. 10.1008/2632221133/0096666 URL https://doi. org 108882F2632
0r0.b....rrr.0..882222222222 21538
2Fac09d6
Joar Skalse, Nikolaus HR Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. arXiv preprint arXiv:2209.1300 2022.
Teague Sterling and John Irwin. Zine 15 ligand discovery for everyone. Journal of chemical information and modeling, 55, 10 2015a. doi: 10.1022/acssecim55005555
Teague Sterling and John J Irwin. Zine 15-ligand discovery for everyone. Journal of chemical information and modeling. 55(11))2324-2337 2015b.
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation.
Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999a. URL http::::pprooeeeess..eeeessspspp
files/papers
Eile/464d82e0 httl://5..222550000eeeeeeaeeeee3
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Proceedings the 12th International Conference on Neural Information Processing Systems, NIPS'99, 99, page 1057-1063, Cambridge, MA, USA, 1999b. MIT Press.
Qiaoyu Tan, Ninghao Liu, and Xia Hu. Deep representation learning for social network analysis. Frontiers in big Data 2:2, 2019.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: large language model for science. arXiv preprint arXiv:2211 1.09085, 2022.
Morgan Thomas, Noel O Boyle, Andreas Bender, and Chris Graaf. Augmented hill-climb increases reinforcement learning efficiency for language-basse based novo molecule generation. Journal of Cheminformatics, 14, 10 2022. doi: 10.1 11118//1133 1666113322-022-00646-..
14

---

Austin Tripp. Wenlin Chen, and José Miguel Hernández-Lobato. An evaluation framework for the objective functions of de novo drug design benchmarks. In 1CCL20022 Machine Learning for Drug Discovery, 2022. URL https: openreview httpa//wwweeeotttoooooooddoddde
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Qi Wang and Chunlei Tang. Deep reinforcement learning for transportation network combinatorial optimization: A survey. Knowledge-Basse Systems, 233:107526, 2021.
Sheng Wang, Yuzhi Guo, Yuhong Wang. Hongmao Sun, and Junzhou Huang. Smiles-bert: large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference bioinformatics, computational biology and health informatics, pages 429-436, 2019.
David Weininger. Smiles, chemical language and information system. 1. introduction to methodology and encoding rules. Chem. Inf. Comput. Sci., 28(1):31-36) feb 1988. ISSN 0095- 2338. doi: 10. 10..0//00005520055.
http://oo.oo/////////oo 10. 1021/c1005570005
Soojung Yang. Doycong Hwang, Seul Lee, Seongok Ryu, and Sung Ju Hwang. Hit and lead discovery with explorative and fragment-based molecule generation, 2021.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning, 2021.
Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation, 2018. URL https://arxiv.orz
abs/1800 02473
Wenhao Yu, C. Karen Liu, and Greg Turk. Policy transfer with strategy optimization, 2018.
Zhenni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(862), 2022.
Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N. Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific Reports, 9(1), jul 2019. doi: 10.1038/
15

---

Outline of Appendices. In Appendix A we provide details about the experimental setup. Appendiix Bi we describe our hyperparameter tuning strategy. In Appendiix Fwee include additional results from our experiments.
A EXPERIMENTAL SETUP
In this section, provide additional details about the tasks, evaluation metrics and pretraining models and data used in our work.
A.1 TASKS
Pydte tasks. These tasks are a set of pharmaccuttically-eeeeaaan oracle functions, which have been commonly used in prior work (Brown et al.) 2019 Gao et al. [002bb Huang et al. 2021 for evaluating performance across molecular discovery algorithms:
QED: quantitative estimate of drug-likeness calculated using a set of rules.
DRD2, GSK33, and JNK3: Classical machine learning models (SVMs and random forests) that provide an estimate properties like target affinity or susceptibility towards a disorder.
Celecoxib, Troglitazone, and Thiothixene rediscovery: An estimate of smiles text similarity, based on tanimoto metric, towards target molecule.
Albuterol and Mestranol similarity: Generate molecules similar to target molecule.
Isomers c7882202 and isomers c9h10n2o2pf2c: Generate molecules corresponding target molecular formula.
Medianl and Median2: molecules.
Generate molecules that are maximally similar several target
_mpp Osimertinib -mpne, mpo, fexofenadine mmo, ranolazine _mpo, perindopril.mpe amlodipine .moo, sitagliptin mpo, zaleplon mpo: Generate molecules that maximize multiple properties of targeted drug.
valsartan smarts: Generate molecules that contain a certain SMARTS pattern and certain physicochemical properties.
Most of these tasks are from the GuacaMol benchmark (Brown 2019). All oracles are calculated using the Python API provided by Therapeutics Data Commons (Huang al. 2021) and more details for these tasks can found on [teii website
Docking tasks. We used Quick Vina 2 (Alhossary et al. 2015) for calculating docking scores using the same default configuration parameters as prior works ( (Yang et al. [202]] Lee al., (023). For example, we used exhaustiveness = 1, and modes 10. We choose different protein targets to calculate docking scores: fa7 (FA7), parpl (PARP-1) 5htlb (5-HTIB), jak2 (JAK-2), and braf (BRAF). These targets were chosen by (Yang et al.) 2021 Lee et al. 2023) because the docking simulators for these targets work fairly well when compared to the ground truth. our experiments in Section 5.3 we found that text-based RL algorithms were easily able to produce chemically trivial molecules that have very high docking scores. To understand the complexity of computing docking scores, report the time taken to dock 1000 molecules in parallel using 12 CPUs table 3 We also provide the time taken to run the RL algorithm on these 1000 molecules after their docking scores are available.
Table 3: Time complexity of docking score evaluation More than half the running time is spent evaluating the docking scores.
Number of molecules
Docking time
RL update time
1000
130 seconds
74 seconds
16

---

Augmented docking tasks. In results for the standard docking tasks (Figure 3 and Figure 4), we found that using the simulated docking scores as rewards did not lead to chemically relevant molecules. Text-based RL algorithms were able to exploit their state and action spaces to design chemically trivial molecules that have very high docking scores. To tackle this issue of undesirable reward hacking, we tried reward function based on prior works (Garcfa-Ortegóó) iet al., 2023) that combine objectíves for drug-like, and synthesizable molecules with docking scores. We call tasks corresponding to this new reward function as augmented docking tasks. Concretely, we chose the same reward function from (Lee et al. 2023
al. 2022 Lee
r(s) = DDS(s)220 x QED(s) x (10- SA(s))9/
(4)
Where DS is the docking score, QED and SA are quantitative estimates of drug likeness and synthesizablity respectively.
A.2 EVALUATION METRICS
Most of the metrics we use are described in detail in Section 5.1 Here, we provide additional details about the diversity metric. We calculate the diversity of the top 100 molecules sampled by the algorithm, where higher diversity is considered better given that it increases the chances for success further wet lab experiments. In our experiments, we use the diversity evaluator from TDC (Huang et al., (202T) which defines the diversity of a set of molecules as the average pairwise Tanimoto similarity between Morgan fingerprints the molecules. See Section 2 of (Benhenda) 2017) for exact details of how Tanimoto similarity is calculated.
A.3 PRETRAINING
In this section, we provide more details about the pretraining datasets and models used our experiments.
Pretraining datasets. The ZINC 250k dataset contains approximately 250k molecules from the
ZINC database (Irwin et al.] 2012), chosen for their pharmaceutical relevance, moderate size, and popularity (Gao et al., 2022b). The CHEMBL dataset (Mendez et al., (2019) consists of approximately 2M manually curated drug-like molecules. The other 3 datasets consist of randomly selected subsets of the ZINC-15 dataset (Sterling and Irwin 2015b) that obey some chemically imposed mild constraints (Irwin et al. 2022). We test three subsets of different sizes: (1) ZINC 1M (2) ZINC 10M, and (3) ZINC 100M, to test the impact of scaling the size of pre-training data. These datasets and data-subsets, including their vocabularies, will be shared in an easily accessible format upon acceptance.
Removing outliers and unusual non drug-like compounds helps to keep the vocabulary small and improves the quality of the generative model (Blaschke al., 2020a). To achieve this, filter all datasets by removing molecules which contain 1) less than 10 or more than 50 heavy atoms and 2) molecules other than Carbon, Nitrogen, Oxygen, Fluorine, Silicon, Chlorine and Bromine. We also canonicalize and sanitize all molecules using RDKIT (Landrum et al.) 2013). For experiments that apply SELFIES, we convert all datasets to SELFIES using the Python API provided by (Krenn) et al. 2020 (Version: 2.1.1).
Apart from the experiments shown in the main paper, Appendix C.2 contains additional experiments comparing text-based RL. agents across different pretraining datasets.
Pretraining models. In Table 4 we provide details about the pretraining modes which we use in experiments. Upon acceptance, will open-source code and release the pretrained weights to support reproducible research.
We select network sizes that have been commonly used in RL (Blaschke et al. 2020a Yarats al. 2021]. Although conducting a study of scaling the model size (Kaplan et al. 2020 is out of the scope of our work, we believe that it is promising direction for future.
Since the fully connected model can only take fixed length inputs, we always input molecular text padded to certain maximum length (we used length 100 in our experiments). This padding is done
17

---

Table 4: Description model architectures used pretraining
Model
Number of Parameters
Description
FC is a fully connected neural network
FC
1.07 x 10?
with 3 hidden layers of 1024 size each.
RNN is recurrent network which consists of 3 GRU layers of hidden sizes 512 each.
RNN
4.17 x 10
GPT (Brown et al. 2020b) style transformer with 6 layers, heads and 256 embedding dimensions.
TRANSFORMER
4.78 x 10
using a special token [PAD] to convey that corresponding tokens should not considered while deciding the value of the text.
Pretraining experimental details. We pretrain FC, RNN and transformer architectures on the ZINC 250K dataset and pretrain transformer on all other datasets. All models are pretrained using the PyTorch (Paszke et al.) al.] (019) framework. All models used an initial learning rate of 1e - 3, with a cosine learning rate schedule (Loshchilov and Hutter 2017). FC and RNNs used a batch size of 128 and were trained for 10 epochs. All transformers were trained for 5 epochs, with the largest batch size that we could in the memory of single NVIDIA RTX A6000 GPU, for example, a ZINC 100M dataset. We made
batch size of 2048 for pretraining the transformer models were trained until convergence, On the ZINC 250K SMILES dataset, the FC, the RNN and the transformer model achieved validation loss of 29.417, 22.507, and 22.923 respectively.
that all
A.4 RL FINETUNING
The pretrained model is further trained using the policy gradient algorithm, REINFORCE (Sutton eet al., 1999b). Given the reward function r(sH) corresponding to the text sy, this algorithm optimizes the loss function
H Σ
po = Ae A-- An)r(sn [Ao: An))
min J(0)
(5)
[=1
where A, is the token sampled by the agent at time-step I.
A.5 REPLAY BUFFER
Although REINFORCE is an on policy method, we investigate the use of replay buffer to improve its sample efficieny Blaschke al. (2020b). To do this, we store recent molecules sampled by the agent in a replay buffer. In our experiments we store the 100 most recent molecules. At every training update, we append a small batch of molecules sampled from the replay buffer to the on- policy batch of molecules. The entire new batch is used to update the policy using the policy gradient appendix A.4
B HYPERPARAMETER TUNING
We conduct a common hyperparameter tuning strategy for all experiments. hyperparameter tuning for
Specifically, we conduct
Learning rate for different architectures Figure 4 and text grammars Figure 3
Coefficients for different likelihood regularizations Figure
Coefficients for KL regularization loss term Figure 6
We select three tasks from the pytde tasks, i.c., troglitazone rediscovery, sitagliptin mpo, and median2 for hyperparameter tuning. For each hyperparameter, select set of 5 evenly spaced
18

---

realistic values and run 5 random seeds of RL experiments per hyperparameter value. We select the hyperparameter value that achieves the best average score of the top- molecules as the final value for running all the experiments. We report the hyperparamters used for the policy gradient training table
Table 5: Hyperparamters
Name
Value
Maximum number unique molecules
25000
5.00 10 RNN and FC
Learning rate
1.00 X 10 Transformer
Batch size
64
Log p coefficient
5
KL coefficient
1.00 10
C RESULTS
C.I TEXT REPRESENTATIONS AND ARCHITECTURES FOR RL
Here, present additional results from subsection 5.3 Figure 8 shows that SMILES are better molecular grammar when compared to SELFIES across all architectures, for the text based RL algorithms that we consider. Figure compares various architectures, while keeping the molecular grammar fixed to SELFIES. The results in Figure 9 reflect our findings in Figure 4 that no single architecture clearly outperforms for molecular text-based RL. It also shows the reward hacking behavior of the docking tasks by the FC based RI. agent.
SELFIES
SMILES
Transformer
RAN
FC
0.75
5core
0.25
0.25
3.00
0.00
0.00
Top
Top
100
Div.
Top
Top Top
(a) PYTDC.
150
0.75
Score
0.25
050
025
Top
Top
10 Top
100
Div.
Red.
Top 1
Top 10 Top
100
Div.
Red.
Top
Top 10 Top
100
Div.
Red.
(b) DOCKING.
Figure 8: Comparison between SELFIES and SMILES across different architectures. These figures the individual plots corresponding the normalised plot show in Figure 3
The reason for lower value molecules for SELFIES environments can be explained by the SELFIES grammar that induces a flat optimization landscape. Many SELFIES strings can correspond to the same molecule, and in fact, once invalid action is taken, any subsequent sequence of tokens will be ignored on the resulting molecule. This makes exploration of new molccules difficult (Krenn et al. 2020 Gao et al. 20225). On the other hand, the benefit of SELFIES over SMILES in
19

---

Transformers
pyloc
docking
2.75
5core
2.50
1.00
0.53
6.00
0.03
Top
Top 10
Top
Div.
Top
Top Top 100
Red.
Comparison of different policy architectures (SELFIES): No single architecture clearly outperforms for molecular text- base RL. Although FC does better on the docking tasks, our analysis shows that it learns exploit the docking function as opposed designing high-value molecules.
Figure 9:
eliminating invalid molecule generation is mitigated by our pretraining process, which initializes SMILES-bas based policies with strong bias toward generating valid molecules. Overall, we find that SMILES-based policies, when combined with pretraining, are more effective exploring and finding high-value molecules.
C.2 PRETRAINING FOR RL
Figure (right) shows the top docking scores obtained by RL agents pre-trained different datasets when trained with on the augmented docking tasks. In Figure 10 we show the actual augmented rewards obtained by the RL agent. These results suggest that the augmented docking score is complex reward function as the RL agent is achieved minimal improvement over the prior agent. To verify this hypothesis, see that RL agents corresponding to all prior-datasets exhibit considerable improvement. Text-based RL algorithms learn search more efficiently when provided with more compute.
increased the molecule budget of the RL agent by 10 times. We indeed
CHEMBL
ZINC 250K
ZINC 1M
ZINC 10M
ZINC 100M
AUGMENTED DOCKING
0.75
score
0.50
0.25
Top 1
Top 10
Top 100
Div.
Red.
Score
0.50
0.25
Top 1
Top 10
Top 100
Div.
Red.
Figure 10: This figure shows the augmented rewards obtained by the RI. agents (Top) and data quality (Bottom) of different datasets. See (susecction A.l for how the augmented reward is calculated.
20

---

CHEMBL
ZINC 250K
ZINC
ZINC
ZINC 100M
AUGMENTED DOCKING
1.25
1.00
score
0.75
0.50
Top 1
Top 10
Top 100
Div.
Red.
Figure 11: This figure shows the augmented rewards obtained by the RI agents trained for 10 times more molecules.
C.3 REWARD HACKING
Figure 4 and Figure shoow that text-based RL agents that are trained using fully connected neural networks able to obtain unusually high rewards. This is probably because it is easier for FC agents to find actions that exploit the local structure of the reward function as RNNs and Transformers inductively biased to find global solutions. This highlights undesirable type of reward function hacking by the FC agent which provides high rewards for molecules with long strings of Carbon and Nitrogen atoms together. Similar to prior work (Lee et 2023), we augment the docking scores with objectives for drug-like and synthesizable molecules. See Appendix A for details of this task and Figure 2 and Figure 10 for results corresponding results on this task (Figure 2 and Figure IO) suggested that the augmented reward function was more aligned towards chemically relevant molecules. We also noticed that the RL agents were not able to improve lot over the prior baselines for this task. To verify whether the low performance of RL agents was because of less training data or the augmented reward function was indeed a more realistic and robust reward function, higher training budget. Given more data, all RL agents showed considerable improvements over the priors. This experiment also revealed that the agents pre-trained ZINC 1M, ZINC 10M and ZINC 100M, were able to exploit the reward function to generate unrealistic yet highly rewarding molecules. These molecules have unusually low docking scores (less than -20). Our results highlight the need for an aligned and more robust reward function to generate molecules for docking protein targets.
this task. Our initial
repeated the experiments in Figure 10) with ten times
C.4 ADDITIONAL RESULTS FOR THE IMPORTANCE OF ALGORITHMIC CHOICES FOR TEXT-BASED RL.
Section 5.4 compares various algorithmic components like replay buffers, hill climbing, KL regularisation towards the pretrained policy, and likelihood penalties and show results for PYTDC tasks. In this section, we repeat all the experiments from (ection[5.440m augmented docking tasks as well and reach the same conclusions. In Figure 15 15|we we see that using the hill-climb buffer results in significant performance boost, whereas using standard buffer does not contribute much, Figure 16 shows that Log regularization a better choice for efficient exploration when it comes to text- based RL algorithms. In Figure 17 show that penalising the policy to move away from the pretrained policy does not improve performance.
C.5 INSTABILITY OF TRANSFORMERS FOR ONLINE RL.
Many works (Parisotto et al. online reinforcement learning. To understand this in the context of text based RL, we compare transformer and an RNN based agent on the augmented docking task. To probe whether pronounced effects of this instability seen, we train both agents for 10 times more molecules (250K
2019) have pointed out the instability of training transformers using
21

---

Table 6: Performance of ChemRLformer across all PyTDC tasks.
Target
Avg Top1
Avg Top10
Avg Top
Diversity_ooy 1trrittyttpo000
Redundant,count
drd2
1.000 (0.000)
1.000 (0.000)
1.000 (0.000)
0.465 (0.052)
29907.0 (1689.767)
qed
0.948 (0.000)
0.948 (0.000)
0.948 (0.000)
0.639 (0.064)
26802.6 (460.301)
jnk3
0.896 (0.096)
0.892 (0.095)
0.883 (0.095)
0.348 (0.094)
29844.8 (3249.365)
gsk3b
1,000 (0.000)
0.997 (0.004)
0.992 (0.006)
0.407 (0.030)
26923.4 (2663.734)
celecoxib.rediscover rediscovery
1.000 (0.000)
0.889 (0.007)
0.847 (0.006)
0.285 (0.031)
28856.6 (1903.692)
troglitazone redisssovry
0.654 (0.098)
0.643 (0.089)
0.627 (0.082)
0.364
27425.4 (891.261)
thiothixene rediscovery
0.669 (0.005)
0.669 (0.005)
0.649 (0.006)
0.403 (0.012)
29158.4 (1492.667)
albuterol similarity
1,000 (0.000)
1.000 (0.000)
1.000 (0.000)
0.529 (0.034)
31551.2 (1310.567)
mestranol_similarig
0.883 (0.124)
0.873 (0.122)
0.825 (0.109)
0.371 (0.029)
30395.2 (1516.911)
coomers_c7h82200
1.000 (0.000)
1.000 (0.000)
0.950 (0.031)
0.781 (0.056)
33245.4 (1393.527)
isomers_c9h 102o22212e
0.931 (0.007)
0.925 (0.009)
0.904 (0.014)
0.598 (0.073)
31457.4 (1893.718)
medianl
0.415 (0.030)
0.395 (0.027)
0.379 (0.019)
0.425 (0.050)
27805.4 (1172.183)
median2
0.353 (0.033)
0.347 (0.031)
0.337 (0.027)
0.323 (0.045)
25513.0 (923.048)
osimertinib mpo
0.911 (0.005)
0.907 (0.005)
0.903 (0.005)
0.389 (0.033)
26083.6 (1287.568)
fexofenadine mpo
0.940 (0.020)
0.938 (0.021)
0.927 (0.019)
0,479 (0.034)
26143.0 (1100.390)
ranolazine_mpo
0.879 (0.011)
0.876 (0.010)
0.872 (0.009)
0.351 (0.050)
23383.8 (1070.
perindopril_mpo
0.637 (0.020)
0.635 (0.020)
0.632 (0.018)
0,410 (0.067)
26441.8 (938.579)
amlodipine mpo sitagliptin mpo zaleplon mpo
0.827 (0.079)
0.827 (0.080)
0.817 (0.078)
0,340 (0.066)
28469.4 (1439.003)
0.573 (0.056)
0.566 (0.056)
0.547 (0.063)
0.611 (0.025)
30156.6 (1903.446
0.631 (0.019)
0.623 (0.021)
0.605 (0.021)
0.542 (0.065)
29838.2 (531.040)
valsartan _smarts
0.000 (0.000)
0.000 (0.000)
0.000 (0.000)
0.880 (0.002)
0.0 (0.000)
normalised score
0.769 (0.010)
0.760 (0.010)
0.745 (0.009)
0.473 (0.008)
27114.3 (495.515)
Table 7: Performance of just the pretrained model across all PyTDC tasks
Target
Avg Top1
Avg Top10
Avg Top100
Diversity Hiversity.top100
Redundant.count
drd2
0.962 (0.022)
0.882 (0.018)
0.484 (0.011)
0.853 (0.002)
9.600 (4.224)
ged
0.948 (0.000)
0.947 (0.000)
0.944 (0.000)
0.853 (0.005)
9.600 (4.224)
jnk3
0.446 (0.153)
0.327 (0.034)
0.197 (0.004)
0.868 (0.004)
9.600 (4.224)
gsk3b
0.748 (0.061)
0.580 (0.027)
0.363 (0.011)
0.868 (0.002)
9.600 (4.224)
0.444 (0.019)
0.399 (0.007)
0.342 (0.002)
0.829 (0.003)
9.600 (4.224)
Doglitazzonerediscoove
0.304 (0.016) 0.281 (0.004)
0.250 (0.001)
0.829 (0.002)
9.600 (4.224)
thiothixene rediscovery
0.382 (0.013)
0.353 (0.005)
0.310 (0.001)
0.806 (0.003)
9.600 (4.224))
albuterol similarity
0.605 (0.022)
0.561 (0.012)
0.487 (0.005)
0.847 (0.003)
9.600 (4.224))
mestranol mimilatol_smmilarity
0.486 (0.015)
0.448 (0.007)
0.388 (0.003)
0.833 (0.004)
9.600 (4.2244)
coomers.c7h8222.
0.948 (0.045)
0.839 (0.032)
0.556 (0.032)
0.900 (0.003)
0..0024.224
1omersscch110222 On2o2p12cl
0.824 (0.029)
0.740 (0.019)
0.597 (0.011)
0,880 (0.003)
9.600 (4.2244)
medianl
0.285 (0.021)
0.240 (0.005)
O. 198 (0.002)
0.852 (0.003)
9.600
median2
0,243 (0.013)
0.215 (0.004)
O. 189 (0.001)
0,822 (0.002)
9,600 (4.224)
osimertinib mpo
0.812 (0.004)
0.799 (0.002)
0.771 (0.001)
0.843 (0.001)
9.600 (4.224)
fexofenadine mpo
0.723 (0.013)
0.702 (0.004)
0.670 (0.002)
0.838 (0.003)
9.600 (4.2244)
ranolazine_mpo
0.636 (0.020)
0.551 (0.012)
0.441 (0.010)
0.856 (0.003)
9.600
perindopril
0.507 (0.021)
0.472 (0.009)
0.437 (0.003)
0.811 (0.002)
9.600
amlodipine mpo
0.629 (0.020)
0.580 (0.011)
0.521 (0.003)
0.814 (0.002)
9.600
sitagliptin.mpp -mpo
0.438 (0.027)
0.378 (0.013) 0.284 (0.005)
0.858 (0.004)
9.600
zaleplon_mpo
0.513 (0.013)
0.484 (0.005)
0.443 (0.003)
0.830 (0.002)
9.600 (4.224)
valsartan smarts
0.003 (0.007)
0.000 (0.001) 0.000 (0.000)
0.876 (0.004)
9.600 (4.224)
normalised_scorf
0.566 (0.006) 0.513 (0.002)
0.422 (0.002)
0.846 (0.001)
9,600 (1.282)
22

---

Table 8: Performance of ChrmRLformer across all docking tasks.
Target
Avg Top1
Avg Top10
Avg Top100
Diversity_topl0
Redundant_count
Shtlb
17.467 (4.742)
14.903 (2.295)
13.131 (1.374)
0.870 (0.019)
16221.333 (11267.121)
jak2
13.200 (1.158)
12.370 (0.790)
11.294 (0.700)
0.853 (0.031)
19294.333 (10886.657)
fa7
18.933 (5.949)
13.860 (1.549)
11.349 (0.699)
0.884 (0.017)
5334.000 (6029.172)
braf
21.700 (11.950)
16.817 (5.709)
12.366 (0.746)
0.888 (0.013)
13023.333 (6672.632)
parp1
21.867 (8.745)
16.680 (2.110)
14.310 (0.466)
0.869 (0.024)
15175.667 (10328.254)
normalised score
18.633 (2.823)
14.926 (1.121) 12.490 (0.397)
0.873 (0.010)
13809.733 (4172.842)
Table 9: Performance of the just the the pretrained model across all docking tasks.
Target
Avg Topl
Avg Topl0
Avg Top100
Diversity top100
Redundant count
Shtlb
12.700 (0.283)
12.250 (0.136)
11.557 (0.083)
0.866 (0.003)
11.333 (1.886)
jak2
11.867 (0.287)
11.193 (0.115)
10.510 (0.077)
0.874 (0.004)
11.000 (1.414)
fa7
10.167 (0.125)
9.837 (0.019)
9.241 (0.018)
0.873 (0.003)
11.667 (3.300)
braf
12.333 (0.125)
11.853 (0.047)
11.160 (0.091)
0.869 (0.004)
11.667 (1.247)
parpl
12.833 (0.047)
12.323 (0.042)
11.588 (0.039)
0.869 (0.001)
11.333 (4.028)
normalised_score
11.980 (0.082)
11.491 (0.013) 10.811 (0.019)
0.870 (0.001)
11.400 (2.142)
molecules). In figure Figure 12 targets.
we see that both agents perform comparably across all docking
23

---

RNN
Transformers
AUGMENTED DOCKING
0.75
score
Top
Top 10
Top 100
Div.
Red.
Figure 12:
A transformer and RNN based RL agent trained for 10 times molecules augmented docking scores.
PPO
REINFORCE (ours)
PRIOR
Algorithms
0.75
score
0.50
0.25
0.00
Top
Top 10
Top 100
Div.
Red.
Comparison with another RL algorithm PPO. To keep the evaluation fair, compared both the algorithms without the replay buffer and without the KL penalty. Both algorithms used logp regularization. We made sure that both algorithms molecular optimization tasks of PyTDC, our results indicate that vanilla policy gradient algorithms stable than actor critic algorithms like PPO and achieve higher performance. We believe this occurs because PPO learns a value function which a difficult task when the reward function sparse. This is indeed the case for molecular optimization. The agent only gets non zero rewards at the last step of the episode, when the molecule is scored. This makes the value function learning highly biased. This results resonates with the findings of previous work in molecular
Figure 13:
the same pretrained policy. On the
optimization (Cieplinski et al. 2021 Gao et al. 2022b).
24

---

Figure 14: Some ChemRLformer agents able obtain unusually high docking scores by stacking together long chains rings of sulphur, phosphorus or carbon atoms.
Hill Climb Replay Buffer
Replay Buffer
No Replay Buffer
AUGMENTED DOCKING
0.75
score
0.50
0.25
Top 1
Top 10
Top 100
Div.
Red.
Figure 15: Do replay buffers help?
D DISCUSSION WITH DOMAIN EXPERTS.
Domain experts find the versatility of ChemRLformer in solving multiple tasks, especially expensive tasks like docking, useful for early-stage drug discovery. The reward-hacking insights are particularly useful as they show the shortcomings of docking score evaluations and the need for better evaluation, both simulation and real-world experiments. will add a deeper discussion on domain expert and application perspectives in the camera-ready version.
E REINFORCEMENT LEARNING FROM HUMAN FEEDBACK.
In recent years, Reinforcement Learning from Human Feedback (RLHF) has become popular framework train large language models optimise reward model. This reward model meant to represent human preferences over outcomes. The reward model is trained using datasets of sequence label pairs. These labels are provided by humans, depending on how good the sequence is. This recipe has been successful to finetune large language models Ouyang et al. (2022): Gao et al. 2022a).
While RLHF is promising direction of future work, where chemists could interact and give feedback to the RL agent to generate desirable molecules, one reason why this will be more difficult in the space of molecular optimisation than natural language is that the expertise in chemistry is much more rare. And given that training a reward model requires large amount labels, the current RLHF framework may not scale well for this case. Right now, these experts indirectly performing RLHF by creating simulations (e.g. Autodock) which we use in our experiments. Even with these simulations, which could be used to label unlimited molecules, should focus on sample
25

---

LogP Regularization
Entropy Regularization
Regularization
AUGMENTED DOCKING
0.75
Score
0.50
0.00
Top 1
Top 10
Top 100
Div.
Red.
Figure 16: Comparison of different likelihood penalization for efficient exploration
KL Prior
No KL Prior
AUGMENTED DOCKING
Score
0.50
0.25
0.00
Top 1
Top 10
Top 100
Div.
Red.
Figure 17: Is KL. regularisation with a prior necessary?
efficient RL algorithms. The reason being that the simulators aren't perfect appendix chemists will only be able to give feedback on limited number of molecules.
CK and real
F RESULTS
In this section, we add images of the top 5 molecules found by CormRl.former each of the PyTDC tasks:
tapa
Figure 18: drd2.
26

---

Figure 19: qed.
XOsd tapa
Figure 20: jnk3.
xosto
Figure 21: gsk3b.
poos
Figure 22: celecoxib rediscovery.
thoep puod x
ana,
Figure 23: troglitazone rediscovery.
Figure 24: thiothixene rediscovery.
27

---

fox
Figure 25: albuterol similarity.
topa
Figure 26: mestramol similarity.
Figure 27: isomers c7h8n202.
3ot
Figure 28: isomers w9h10n220000 On2o2pf2el.
Figure 29: median1.
Figure 30: median2.
28

---

awe
Figure 31: osimertinib mpo.
audo
Figure 32: fexofenadine mpo.
Figure 33: ranolazine mpo.
tranp
eoxo
Figure 34: perindopril mpo.
thoup
Figure 35: amlodipine mpo.
com
Figure 36: sitagliptin mpo.
29

---

tame
Figure 37: zaleplon mpo.
gimo auto thoup
Figure 38; valsartan smarts.
Figure 39: deco hop.
Figure 40: scaffold hop.
LogP Regularization
Tuned Entropy Regularization
No Regularization
PYTDC
0.75
0.50
0.25
0.00
Top 1
Top 10
Top 100
Div.
Red.
Figure 41: We replicate the comparison in fig. 7 with an automatically tuned coefficient for the entropy HHarnoja al. (2018). We follow Christodoulou (2019) to set the target entropy as -0.98 (og(1/[A]) where A is equal to the number of tokens in the vocabulary. Adding this feature does not improve the performance of entropy regularisation and explained in section 5.4 Log regularisation performs much better.
30


Reviews:
[{"review_id": "6ERUYhkHAV", "reviewer": "Reviewer_QKJh", "summary": "This paper explores the potential of reinforcement learning (RL) methods to discover new, high-value molecules and presents a new RL-based molecular design algorithm called ChemRLformer. The authors conduct extensive experiments and analysis to show that ChemRLformer achieves state-of-the-art performance while being more straightforward than prior work. The paper provides unique insights into the application of RL to molecular design and highlights the importance of careful search space structuring and algorithm design.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important problem in the field of molecular design, which has significant implications for society. The potential of RL methods to discover new, high-value molecules could have a major impact on drug discovery, materials science, and other fields.\n\n2. The paper presents a new RL-based molecular design algorithm called ChemRLformer, which achieves state-of-the-art performance while being more straightforward than prior work. The authors provide unique insights into the application of RL to molecular design and highlight the importance of careful search space structuring and algorithm design.\n\n3. The paper is well-written and easy to understand, even for readers who may not be familiar with RL or molecular design. The authors provide clear explanations of their methods and results, and use visual aids to help illustrate their points.\n\n4. The authors conduct extensive experiments and analysis using 25 molecule design tasks, including computationally complex protein docking simulations. They explore how different design choices for text grammar and algorithmic choices for training can affect an RL policy’s ability to generate molecules with desired properties.", "weaknesses": "1. It is confusing why the authors assume a discount rate of 1 in Section 4.1. Even with a finite trajectory, a discount rate smaller than 1 is not obligatory, it can also help. A discount rate of 1 can prevent the agent from learning and executing long-term tasks, as it won't appropriately discount long-term returns, leading it to prioritize immediate rewards and neglect long-term benefits. It is suggested to justify this setting.\n\n2. REINFORCE is a very classic yet old algorithm. It is highly recommend to try more recent algorithms, e.g., TROP or PPO. I am not an exert in RL for molecular optimization, but I doubt whether it is the state-of-the-art RL algorithm in this field. For example, (i) due to its reliance on stochastic policies and simple optimization methods, REINFORCE can be more susceptible to getting stuck in local optima, making it less effective in complex and high-dimensional environments; (ii) due to its on-policy mechanism, REINFORCE requires a large number of samples to estimate gradients accurately. This can make it computationally expensive and slow for complex tasks and environments. \n\n3. It would be more convincing if domain experts can help to judge the effectiveness of ChemRLformer from molecular's perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Kyx94iXss", "reviewer": "Reviewer_ANoo", "summary": "The authors present a method for molecule design using language models and RL. First, a GPT-style model is pre-trained in an autoregressive fashion. The model is then finetuned using REINFORCE algorithm to generate high-value molecules. The authors discuss several design choices and perform ablation studies to confirm their hypotheses.", "questions": "1. A most popular approach nowadays to finetuning transformers is reinforcement learning with human feedback. In RLHF, the reward preference model is trained to choose between several samples of the transformer and then the fine-tuning is performed using PPO algorithm. Have the authors considered this approach? \n1. It is not clear why the authors use Reinforce instead of more popular PPO for example. I appreciate that the authors did an ablation study on the KL constraint, but PPO has other benefits in comparison with Reinforce \n1. Details of the backbone model are not given. Is it a standard GPT model? Was the tokenizer standard or also trained? \n1. It is not clear why the authors chose GPT-style model as opposed to BERT style model, where the whole sequences can be predicted directly. This approach was used, e.g  by Cowen-Rivers et al albeit for a different problem and with a different RL algorithm. A discussion on the subject would be interesting.\n1. Haarnoja et al 2018 proposed an automatic tuning procedure of entropy that can explicitly state the target level of entropy. It would be interesting to have this as ablation as well. \n1. I am confused about referencing Mnih et al 2013 regarding the use of replay buffer for on-policy algorithms. Don’t Mnih et al use an off-policy algorithm? \n1. Furthermore, I am not quite sure what a replay buffer for on-policy algorithm means. Does it mean that we use samples for several previous iterations not just the most recent one?\n1. The results could be displayed better, with this scale the improvement of the model does not seem too large, but could be deceptive. \n1. In Figure 2 CheMBL offers high diversity and high redundancy results, which seems confusing at first. Please comment in the figure caption\n\nUPD:\nReferences :\n* Cowen-Rivers, Alexander I., et al. \"Structured Q-learning For Antibody Design.\" arXiv preprint arXiv:2209.04698 (2022).   \n* Haarnoja, Tuomas, et al. \"Soft actor-critic algorithms and applications.\" arXiv preprint arXiv:1812.05905 (2018).", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem is of high importance\n1. The paper is well-written\n1. The numerical results are well documented and contain ablation studies", "weaknesses": "1. Some of the design choices seem to be dated. For example, \n    * it is not clear why REINFORCE is used as the policy learning algorithm, as opposed for instance to PPO. \n    * there is no automatic entropy tuning for exploration.\n    * Reinforcement Learning with Human Feedback can be also an interesting approach to try\n1. The details for the backbone language model as scarce. It would be also interesting to discuss the design choices for language modeling, e.g., pretraining architecture (BERT vs GPT),  tokenizer etc\n1. It would be good to see if increasing the transformer size would lead to performance improvement. If I am not mistaken increasing the RNN size could lead to issues while training and therefore this could potentially become the strength of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f1EDHV3cdr", "reviewer": "Reviewer_AgdE", "summary": "Authors propose Transformer-based approach with Reinforcement Learning tuning for molecular generation. Approach is tested using different datasets and wide ablation study is conducted.", "questions": "Please include relevant work which is mentioned in the **Weaknesses**\n\n* What are the principal differences between you approach and Taiga?\n\n* Can you compare against MolGPT, Taiga and REINVENT (or similar approach)? \n\n* What are \"Div.\" and \"Red.\" columns?\n\n* What are the scores for different reward components (e.g. QED)? \n\n* Can you please provide examples of generated molecules? The picture of molecular graphs and real drugs for given targets is fine.\n\n* How did you make run docking process so fast? From my experience even GPU-accelerated programs for docking are quite slow. Is acceleration achieved because the docking is performed to the fixed target protein?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Promising approach for the molecular generation. Experiments are conducted using three different datasets and impact of each of them is shown. \n\nGood ablation study includes the design choices, different models architectures and even other RL algorithms beside REINFORCE are tested.", "weaknesses": "**Relevant work**\n\nSome relevant works seem to be missing: \n\n* MolGPT (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10232454/) also uses Transformer for molecular generation but without RL.\n\n* Taiga (https://pubs.acs.org/doi/10.1021/acs.jcim.1c00600) uses Transformer and REINFORCE algorithm for the molecular generation with different properties optimization.\n\n* Please also consider citing the relevant parallel work which applies offline RL for the similar problem: https://www.biorxiv.org/content/10.1101/2023.11.29.569328v1 (once it is public)\n\nThe second work seems to be very similar in terms of approach and I'm not sure about the novelty of the approach itself.\n\n** Results presentation **\n\n* I haven't noticed the comparison against any other methods.\n\n* From the plots I couldn't understand what \"Div.\" and \"Red.\" columns stand for.\n\n* I couldn't find reported scores for different tasks (e.g., similarity, QED or SA). It would be nice to see them.\n\n* There are no samples of generated molecules. RL could hack reward functions and generate molecules which make no sense. Reward hacking is mentioned by the authors but including resulting molecules is essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bS40PzdxX2", "reviewer": "Reviewer_feem", "summary": "The paper presents a study of different design choices (problem representation, neural architecture, pretraining, etc) in the space of string-based RL for molecular discovery. The paper presents extensive experiments, including more than 25 molecule design tasks, to compare different design choices. The best combination of design choices is then used to propose a new RL-based molecular design algorithm (ChemRLformer). The paper also provides a thorough discussion of the results and provides valuable insights and recommendations for practitioners in the field of molecular discovery.\n\nThe paper is well written and organized. It is easy to follow and understand. The paper is also well motivated and provides a good introduction to the field of string-based RL for molecular discovery. I recommend the paper for acceptance.", "questions": "- (Introduction) \"...our own algorithm (MoLRL)\" - what is MoLRL?\n- (Table 1) What is the difference between \"MoLRL\" and \"ChemRLformer (Ours)\"? Is MolRL the general framework and ChemRLformer the best combination of design choices based on the ablation study? Please clarify.\n- (Section 4) \"The molecular design space is complex but the benefit from finding improved options is great.\" This sentence reads a bit too colloquially. Please rephrase.\n- (Section 4) \"However, the corresponding transition function induced in the graph representation of molecules is more complex as it is determined by the encoding/decoding rules of the chosen text representation.\" Are these constraints hard-coded in situ on the transitions dynamics or are they learned? Please clarify.\n- (Figure 2) \"(a) Performance on SMILES-based molecular design with pertaining (left) and with pretrianing and RL (right).\" Please correct the typos \"pertaining\" and \"pretrianing\".\n- (Figure 2) \"b) performance on SMILES-based molecular docking with pertaining (right) and with pretrainig and RL (right).\" First (right) should be (left). Please correct.\n- (Figure 2) How does the \"pretraining only\" results (left column) are computed? Is this a zero-shot evaluation? I can't find the answer in the text. Please clarify.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Technical Soundness: The paper demonstrates a high level of technical rigor. It conducts extensive experiments that compare various design choices (such as problem representation, neural architecture, pretraining, etc.) within the realm of string-based reinforcement learning for molecular discovery. Moreover, the paper offers a comprehensive discussion of the results and imparts valuable insights and recommendations for future research.\n\n- Clarity and Organization: The paper is well-written and organized, making it easy to understand and follow. It effectively motivates the reader and provides a solid introduction to the field of string-based reinforcement learning for molecular discovery.\n\n- Significance: The paper makes a substantial contribution to the field of molecular discovery. It offers a comprehensive comparative study of diverse design options (problem representation, neural architecture, pretraining, etc.) within the domain of string-based reinforcement learning for molecular discovery.\n\n- Table 1 provides a nice summary of the different algorithms for text-based molecule design.", "weaknesses": "- Originality: While the paper offers a thorough numerical comparison of different design choices (problem representation, neural architecture, pretraining, etc.) in the context of string-based reinforcement learning for molecular discovery, its originality is somewhat constrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)