Maximum Reward Formulation In Reinforcement Learning
Authors: SaiKrishna Gottipati
Yashaswi Pathak
Rohan Nuttall
. Sahir
Raviteja Chunduru
Ahmed Touati
Sriram Ganapathi Subramanian
Matthew E. Taylor
Sarath Chandar
Venue:  (2021)
Keywords: Reinforcement Learning
Theoretical Reinforcement Learning
Drug Discovery
Molecule Generation
de novo drug design

Abstract: Reinforcement learning (RL) algorithms typically deal with maximizing the expected cumulative return (discounted or undiscounted, finite or infinite horizon). However, several crucial applications in the real world, such as drug discovery, do not fit within this framework because an RL agent only needs to identify states (molecules) that achieve the highest reward within a trajectory and does not need to optimize for the expected cumulative return. In this work, we formulate an objective function to maximize the expected maximum reward along a trajectory, derive a novel functional form of the Bellman equation, introduce the corresponding Bellman operators, and provide a proof of convergence. Using this formulation, we achieve state-of-the-art results on the task of molecule generation that mimics a real-world drug discovery pipeline.

Under review as a conference paper at ICLR 2021
MAXIMUM REWARD FORMULATION IN REINFORCE- MENT LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Reinforcement learning (RL) algorithms typically deal with maximizing the ex- pected cumulative return (discounted or undiscounted, finite or infinite horizon). However, several crucial applications the real world, such as drug discovery, do not fit within this framework because an RL agent only needs to identify states (molecules) that achieve the highest reward within a trajectory and does not need optimize for the expected cumulative return. In this work, we formulate objective function to maximize the expected maximum reward along a trajectory, propose novel functional form of the Bellman equation, introduce the correspond- ing Bellman operators, and provide a proof of convergence. Using this formulation, we achieve state-of-the-art results on the task of synthesizable molecule generation that mimics real-world drug discovery pipeline.
INTRODUCTION
Reinforcement learning (RL) algorithms typically try to maximize the cumulative finite horizon undiscounted return, R(T) Σ Tt, the infinite horizon discounted return, R(T) t== y're- is the reward obtained at time step t, 7 is the discount factor in the range [0, 1). and 7 is the agent's trajectory. consists of actions (a) sampled from the policy (r(- s)) and states (s') sampled from the probability transition function ((s" 18, a) of the underlying Markov Decision Process (MDP).
The action-valle value function (" a) for policy n is given by
Q" (s. тース
[R(T)] ag) (s. a)]
The corresponding Bellman equation for (* a) with the expected return defined as R(()) Ei=o is
(s) E [r(st, + +0² ($1+1,
(
This Bellman equation has formed the foundation of RL. However, argue that optimizing for only the maximum reward achieved in an episode is also an important goal. Reformulating the RL problem achieve the largest reward in an episode is the focus of this paper, along with empirical demonstrations in one toy and one real-world domain.
In the de novo drug design pipeline, molecule generation tries to maximize given reward function. Existing methods either optimize for the expected cumulative return, or for the reward at the end of the episode, and thus fail optimize for the very high reward molecules that may be encountered in the middle of an episode. This limits the potential of several of these reinforcement learning based drug design algorithms. We overcome this limitation by proposing a novel functional formulation of the Bellman equation:
@ (st, a.) = p [max (r(st; 7Q (8 (x41))) a
Other cases of this formulation (i.e., situations where the single best reward found, rather than the total rewards, are important) are symbolic regression (Petersen 2020). Udrescu & Tegmark (2020) which is interested in finding the single best model, active localization (Chaplot al. (2018) 8)) must find the robot's 's one most likely pose, green chemistry (Koch et al. (2019) wants to identify the one best product formulation, and other domains that use RT. for generative purposes.

---

Under review as a conference paper at ICLR 2021
This paper's contributions are to:
Derive novel functional form of the Bellman equation, called max-Bellman, to optimize for the maximum reward in an episode.
Introduce the corresponding evaluation and optimality operators, and prove the convergence of Q-learning with the max-Bellman formulation.
Test a toy environment and draw further insights with a comparison between Q-learning and Q-learning with our max-Bellman formulation.
Use this max- Bellman formulation to generate synthesizable molecules in an environment that mimics the real drug discovery pipeline, and demonstrate significant improvements over the existing state-of- sute-of-the-ar -ar methods.
2 RELATED WORK
This section briefly introduces fundamental RL concepts and the paper's main application domain.
2.1 REINFORCEMENT LEARNING
Bellman's 's dynamic programming paper (Bellman 1954) introduced the notions of optimality and convergence of functional equations. This has been applied in many domains, from control theory to economics. The concept of an MDP was proposed the book Dynamic Programming and Markov Processes (Howard) 1960) (although some variants of this formulation already existed in the 1950s). These two concepts of Bellman equation and MDP are the foundations of modern RL. Q-learning was formally introduced in (Watkins & Dayan 1992) and different convergence guarantees were further developed in (Jaakkola et al.) 1993) and (Szepesvári 1997). Q-learning convergence to the optimal Q-value (Q") has been proved under several important assumptions. One fundamental assumption is that the environment has finite (and discrete) state and action spaces and each of the states and actions be visited infinitely often. The learning rate assumption is the second important assumption, where the sum of learning rates over infinite episodes is assumed to go to infinity in the limit, whereas the sum of squares of the learning rates are assumed be a finite value (Tsitsiklis) 1994 Kamihigashi & Le Van 2013). Under similar sets of assumptions, the on-policy version of Q-learning known as Sarsa, has also been proven to converge the optimal Q-value in the limit (Singh al. 2000).
Recently, RL algorithms have seen large empirical successes as neural networks started being used as function approximators (Mnih al., 2016). Tabular methods cannot be applied large state and action spaces as these methods are linear in the state space and polynomial in the action spaces in both time and memory. Deep reinforcement learning (DRL.) methods on the other hand, can approximate the Q-function or the policy using neural networks, parameterized by the weights of the corresponding neural networks. In this case, RL algorithms easily generalize across states, which improves the learning speed (time complexity) and sample efficiency of the algorithm. Some popular Deep RL algorithms include DQN (Mnih et al. 2013). PPO (Schulman et al. 2017). A2C (Mnih et al. 2016), SAC (Haarnoja et al. 2018), TD3
Fujimoto al. (018b), etc.
2.2
DE NOVO DRUG DESIGN
De novo drug design is well-studied problem and has been tackled by several methods, including evolutionary algorithms (Brown et al. 2004):Jensenn) (2019); Ahn et al. (020)), generative models (Simonovsky & Komodakis (2018):
al. (2018) Winter et al. (2019): Din p Olivecrona et al. (2017)) and reinforcement learning based approaches (You et al. (2018a) Zhou et al. (2018)). Wh While the effectiveness of the generated molecules using these approaches has been demonstrated on standard benchmarks such as Guacamol (Brown et al. (2019)), the issue of synthesizability remains a problem. While all the above approaches generate molecules that optimize a given reward function, they do not account for whether the molecules can actually be effectively synthesized, an important practical consideration. Gao & Coley (2020) further highlighted this issue of synthesizability by using synthesis planning program to quantify how often the molecules generated using these existing approaches can be readily synthesized. To attempt to solve this issue, Bradshaw et al. (019) used
et al. (2018;; [) et al. (2018) &); Grifiitss & Hermández-
2

---

Under review as a conference paper at ICLR 2021
variational auto-encoders based approach optimize the reward function with single-step reactions. Korovina et al. (2019) employed a random selection reactants and reaction conditions at every time step of multi-step process. PGFS (policy gradient for forward synthesis) from (Gottiptii et al. (2020) generates molecules via multi-step chemical synthesis and simultaneously optimized for the given reward function. PGFS leveraged TD3 algorithm (Fujimoto et al. (20T8a)), and like existing approaches, optimizes for the usual objective of total expected discounted return.
3
METHOD
This section briefly describes the previous attempts optimizing for the maximum reward in an episode, defines the Q-function and new functional form of the Bellman equation, defines the corresponding max-Bellman operators, and proves its convergence properties.
Quah & Quek (2006) also try to formulate maximum reward objective function. They define the Q-function and derive the max- -ex-Bellman equation in the following way (rephrased according to this paper's notation):
Q (st, a4)
E max (7" ae)) (5t, az), (st, 122
π V(st, E E AxxA
(2)
Then, the corresponding functional form of Bellman equation (i.e., the max-Bellman formulation) was derived as follows:
max (7" (rSss, av)) ($t,, as), P2t
Quaxx(s ae) = E
P saa))) max ((())) a.). E (--1r))(115p)
(1 (x"'""((s,,a))) (8141, ((2)), 122+1
max
=Beei en.e)) max (r(8,, at), QQ (81+1, (111111111)1)
However, the second equality is incorrect, which can be shown with a counter example. Consider an MDP with four states and zero-reward absorbing state. Assume a fixed deterministic policy and 7 close to 1. Let be the start state, with deterministic reward r(s1) 1. With probability 81 goes to $2. Let r(s2) 0, and with equal probability 0.5 transitions to either 83 or sa- The immediate reward r(s3) 2 and r(s4) O, after that it always goes to the absorbing state with 0 rewards thereafter. Since there can be only trajectories, one with a maximum reward of 2 and the other with maximum reward of have Q (s1) 1.5. but Qmex(s) (s2) and max(r(s1). (m))((2)) 1 き 1.5. This is because the expectation and max operators are not interchangeable.
While the counter example above is not applicable in the current setting of PGFS (as it has a deterministic environment and uses deterministic policy gradient algorithm (TD3)), the definition of the Q-function given in equation ( does not generalize to stochastic environments. Moreover, in order to be able accommodate multiple possible products of chemical reaction and truly leverage a stochastic environment (and stochastic policy gradient algorithms), the Q-function should be able recursively optimize for the expectation of maximum of reward the current time step and future expectations. Therefore, we define the Q-function as:
Quae a.) E, s ((13)0))) max (r(S). (1)ss.a)))))) 7B,, 12 P( Pllaa (ar))))) [max (r(s1+1, (1())41:444))....)
wos (-1141)))))
(44200((
Then, the corresponding functional form of Bellman equation (i.e., max- max-Bellman formulation) can be obtained follows:
Q Tn (st, a+) = E Es (s,,a)) max (r(st, at), Q (#1+1, a++1))
(5)
de41owat-1ee (-1atr)
While the work Quah & Quek (2006) is focused on deriving the various forms the max- max-Bellman equations, the major contributions of our paper 1) propose the max-Bellman equation with
3

---

Under review as a conference paper at ICLR 2021
particular motivation towards drug discovery, 2) define the corresponding operators, 3) provide their convergence proofs, and 4) validate the performance on a toy domain and a real world domain (de-novo drug design). Other related ideas and formulations are described in Appendix section [
Based on Equation 5 we can now define the max- max-Bellman evaluation operator and the max- Bex-Bellman optimality operator. For any function Q S A - IK and for any state-action pair (8,a). a),
(),*))(s, _ max
r(s, a), rB, P((ea()) d'mx((16'')
1Q ,d'))
max Q(s', .21)
(M & max
r(s,,),)BBssa P(
a'EAA
M and M (8 x A A (8 x - IR) are operators that takes in a Q function and returns another modified Q function by assigning the Q value of the state s, action a, to be the maximum of the reward obtained at the same state and action (s, a) and the discounted future expected Q value. Proposition 1. The operators have the following properties.
Monotonicity: let Q1, Q1:Q2: S A R. such that Q 2 Q2 element-wise, Then, M Q1 2 Q2 and M Q1 M Q2
Contraction: both operators are 7-contruction in supremum i.e., for any Q1, Q2
S x A-R
|M Q1 - 22 s>1Q Q211ao
(6)
|| M222o S s Q211a
Proof. We will provide proof only for M The proof of M is similar and is provided in Section of the Appendix.
Monotonicity:
Let Q1 and Q2 be two functions such that Qı a) 2 Q2(s, a) for any state-action
pair (s,a) a) E S A. We then have:
max (1 (2(s. ((s,,)))))a))
a'EA
By the definition of max, we obtain:
max (1(),d)) max Q2(s, d'),vs Vs d'EAA
a'EA
and by linearity of expectation, we have:
max Q1 (1', .0) 2 2TB.P((190)) a'EA
yE P((a,a
max a'EA
max (2(s'aa') v(s,a)
Since,
M²Qi(s,a) 27
max (1''
a'EA
we get:
(*,i(s,a) 2 3B.
max Q2(s''a))
a'Ed
Moreover, because M+ 2 r(s,a), we obtain:
M) 2 max ((s.,a), P max Qz (((6s,01)) ())
which is the desired result.
4

---

Under review as a conference paper at ICLR 2021
Denote fs p(.l.) a the expected action-value function of the next state. By using the fact that max(x, y) = 0.3(2 y vl). V(z, y) E R²a we obtain:
Contraction:
max(r, fi) max(r, f2) = 0.5 (r fi - 0.5 + f2 + Ir f2l)
0.5 (fi + 1r - -arr----aa f2l)
S 0.5 (fi - 1h+r--f -irr (r--fo -f)l))
0.5 (fi -f2 Ii f2l)
$ [ - fzl
Therefore
|M - M0 max(r, fi)
- |/ /21loc
$ >ll max (..))) max
a'
s 11Q Q211
(max is a non-expansion)
The left hand side of this equation represents the largest difference between the two Q-functions. Recall that y lies in the range [0, 1) and hence all differences are guaranteed go zero in the limit. The Banach fixed point theorem then lets us conclude that the operator M admits a fixed point.
We denote the fixed point of M by Q* max- Based on equation 5
we see that Qmax is the fixed point of M the next proposition, we will prove that Qinax corresponds to optimal action-value function in the sense that Q = max, Q
Proposition (Optimal policy). The deterministic policy is defined as T²(ss arg max (s, a). n², the greedy policy with respect to Qiaac is the optimal policy and for any stationary policy Q Qinax 2 Q
Proof. By the definition of greediness and the fact that Q is the fixed point of MC, we have: n (s) = arg max.5.A (s,ax(s, a) + M Q* M "==Qu = Q This proves that Q is the fixed point of the evaluation operator M which implies that Q is the action-value function of n².
For any function Q x A + and any policy ग, we have M 2 MTQ. Using monotonicity, have
(Q MM 2 M( 2 MT (M
We then use induction to show that for any 2 1, (M) > (M. As both operators are contractions, by the fixed-point theorem, (M*)*QQ and (Mr)"d converge Qiax and @ respectively, when n goes to infinity. We conclude then that Q Q max
Thus, we have shown that and optimality operator defined based on our novel formulation of Bellman equation converges a fixed point (Proposition) [ and that fixed point the optimal policy (Proposition
4 EXPERIMENTS
This section shows the benefits of using max-Ban Bellman in a simple grid world and in the real-world domain of drug discovery.
5

---

Under review as a conference paper at ICLR 2021
4.1 ToY EXAMPLE GOLD MINING ENVIRONMENT
Our first demonstration is on a toy domain with multiple goldmines in a 3 x 12 grid The agent starts in the bottom left corner and each time step, can choose from among the four cardinal actions: up. down, left and right. The environment is deterministic with respect to the action transitions and the agent cannot leave the grid. All states in the grid that are labeled with values other than -1 represent goldmines and each value represents the reward that the agent will collect upon reaching that particular state. Transitions into a non-goldmine state results in a reward of -1, goldmine's reward can be accessed only once and after it has been mined, its goldmine status is revoked, and the reward received upon further visitation is The episode terminates after time steps and the discount factor is 0.99. The observation is a single integer denoting the state number of the agent. Thus, this a non- non-Markovian setting since the environment observation does not communicate which goldmines have already been visited, and also because the time step information is not included in the state observation.
The environment is shown in Figure 1 If the agent goes up to the top and continues to move right, it will only get a cumulative return 26.8. it instead traverses the bottom row to the right, it receive cumulative return of 27.5, which is the highest cumulative return possible in this environment.
1.0
1.
12
13
144
15
2.0
2.1
7.2
9.0
-1.0
-1.0
-1.0
-8.0
-8.0
-8.0
-8.0
-8.0
-8.0
-8.0
-8.0
880
880
880
-1.0
2.1
2.2
2.3
244
2.5
2.6
2.7
2.8
2.9
6.0
Figure 1: A visualization of the gold-mining toy example. The bottom left state, shown in green, denotes the starting state. States with values other than -1 denote the goldmines and the values denote their respective rewards.
We test both Q-learning and Max-Q Q (i.e. Q-learning based on proposed max- -ex-Bellman formulation), on this environment. As usual, the one step TD update rule for Q-learning is:
Q(s;, at) Q(s,, a,) + a(re + max ((5++1 a) Q(s.. at))
The one step TD update rule for Max flxx-Qia is:
Q(s;,a;) Q(,,, a,) a(max(re: max (()+11;a)) Q(+,. 0z))
For both algorithms, we use learning rate a = 0.001, and decay epsilon from 0.2 to 0.0 linearly over 50000 episodes, after which remains fixed at O. Figure shows the learned Q-values and Figure shows a difference in the final behavior learned by the two algorithms. Q-learning seems to prefer the path with highest cumulative return (along the bottom row) while Max-Q prefers the path with highest maximum reward (reward of +-9 in the top row). The learned policies consistently reflect this behavior. Over 10 independent runs of each algorithm. Q-learning's 's policy always converges moving along the bottom row and achieves expected cumulative return of 27.5. On the other hand, Max-Q always prefers the top row and achieves expected cumulative return of 26.8, but accomplishes its goal of maximizing the expected maximum reward in an episode (i.e., reaching the highest rewarding state). Also, Max-Q has worse initial learning performance in terms the cumulative return, which can be explained by the agent wanting to move from the bottom row to the top row, despite the -8 penalty. This desire to move upwards at any cost is because the agent is pulled towards the +9, and does not care about any intermediate negative rewards that it may encounter.
Figure shows a quantitative comparison between Q-learning and Max-Q. -. Figure 4a shows a comparison in terms of the average episodic return. Q-lcarning achieved optimal performance in terms of cumulative return and therefore has no incentive to direct the agent towards the maximum
'The environment and algorithm code is submitted as supplementary material and will be open sourced after acceptance for replication purposes.

---

Under review as a conference paper at ICLR 2021
(b)
Figure 2: The learned q-values for (a) Q-learning (b) Max-Q
(a)
(4) Average return
(b) Maximum reward per episode
Figure 3: Learned policies for (a) Q-learning (b) Max-
Figure 4: Comparison between Q-learning and Max-Q
reward of +9. Max-Q on the other hand, converges to the path of following the top row to reach the highest reward +9. This can be seen more clearly Figure 4b) which shows a comparison the maximum reward obtained in each episode. Each curve is averaged over 10 runs, and the shaded region represents standard deviation. We also perform experiments fully Markovian setting for similar environment. These results are shown in the Appendix section-C
4.2
DRUG DISCOVERY
We give a brief summary of PGFS (Gottipati et al. (2020)) for de novo drug design here and then incorporate the max-Ban max-Bellman formulation derived above. PGFS operates in the realm off off-colicy continuous action space algorithms. The actor module II that consists of f and π networks predicts continuous action (that in the space defined by the feature representations all second reactants). Specifically, the network takes in the current state (reactant-1 R(()) input and outputs the best reaction template T. The network takes in both (!)) and T inputs and outputs the continuous action The environment then takes in and computes k closest valid second reactants (R(2)), For each of these R(2)s, we compute the corresponding product of the chemical reaction between R()) and 12(2), compute the reward for the obtained produce and choose the product (next state, 824-1 that corresponds the highest reward. All these quantities are stored in the replay buffer. The authors leveraged TD3 (Fujimoto et al.) 8a) algorithm for updating actor (f, π) and critic (Q) networks. More specifically, the following steps are followed after sampling random minibatch from the buffer (replay memory):
First, the actions for the next time step (T++1 and a;+1) are computed by passing the state input (R(")) through the target actor network (i.e., the parameters of the actor networks are not updated in this process). Then, the one step TD target is computed:
+ min Critic-target (R Ti+1,, a;411
j=1,2
In the proposed approach "MB (max-Bellman) PGFS", we compute the one-step TD target as
y/ max r min Critic-target (RR a.+1)
j=1.2
The critic loss and the policy loss are defined as:
Leittic = Σ 1y Q((R(") " a))
7

---

Under review as a conference paper at ICLR 2021
Cpolicy Σ Critic(Re)) (c)))(RR(
Only few reaction templates (about 8 percent) are valid for a given state (reactant-1) Thus, when the actor networks were randomly initialized, they choose invalid template most of the time during initial phases of training. Thus, minimizing cross entropy loss between the template predicted by the f-network and the actual valid template chosen enables faster training i.e., like the PGFS algorithm. we also minimize an auxiliary loss to enable stronger gradient updates during the initial phases of training.
Lauxll - -OTT (o)(F(((()))
and, the total actor loss Cactor is summation of the policy loss Epoliey and auxiliary loss Cauxit.
Lactor
Lpplicy + Lamiit
The parameters 82 of the Q-network are updated by minimizing the critic loss Leritics and the parameters 8 and f of the actor networks f and и are updated my minimizing the actor loss Luctor- more detailed description of the algorithm, pseudo code and hyper parameters used in given in Section of the Appendix.
We compared the performance of the proposed formulation "MB (max-Bellman) PGFS" with PGFS, and random search (RS) where reaction templates and reactants are chosen randomly at every time step. starting from initial reactant randomly sampled from ENAMINE dataset (which contains set of roughly 150,000 reactants). We evaluated the approach on five rewards: QED (that measures the drug like-ness: [Bickerton al, 22022), clogP (that measures lipophilicity: You et al. (018b)) and activity predictions against three targets (Gottipati et al. (2020): HIV-RT, HIV-INT, HIV-CCRS. performance on all these rewards across different metrics
While PGFS demonstrated state-of-ther (maximum reward achieved, mean of top-100 rewards, performance
validation set, etc.) when
compared to the existing de novo drug design approaches (including the ones that do not implicitly explicitly account for synthesizability and just optimize directly for the given reward function), we show that the proposed approach (PGFS+MB) performed better than PGFS (i.e., better than the existing state-of-the-ard methods) on all the five rewards across all the metrics. For a fairness in comparison, like PGFS, we only performed hyper parameter tuning over policy noise and noise clip and trained only for 400,000 time steps. However, the proposed formulation, we noticed that the performance is sensitive to the discount factor and the optimal 7 is different for each reward.
Table compares the maximum reward achieved during the entire course of training of 400,000 time steps. We notice that the proposed approach PGFS+MB achieved highest reward compared to existing state-of-the-ar approaches. Table 4 compares the mean (and standard deviation) of top 100 rewards (i.e., molecules) achieved by each of the methods over the entire course of training with and without applicability domain (AD) (2010), Gottipati et al. (2020)) We again note that the proposed formulation performed better than existing methods without AD. In F we compare the performance based on the rewards achieved starting from fixed validation set of 2000 initial reactants. For all the three HIV reward functions, RL approaches (i.e., PGFS and RS) in
all three rewards both with and
notice
that PGFS+MB performed better than existing reaction-basee terms of reward achieved at every time step of the episode (Figure (Sa), and in terms of maximum reward achieved in each episode (Figure 5)). Further experimental details and results are provided in Section E of the Appendix.
Table 1: Performance comparison of reaction based de novo drug design algorithms in terms of maximum reward achieved
Method QED
clogP
RT
INT
CCR5
ENAMINEBB
0.948
5,51
7.49
6.71
8.63
RS
0.948
8.86
7.65
7.25
8.79 (8.86)
POFS
0.948
27.22
7.89
7.55
9.05
PGFS
0.948
27.60
7.97
7.67
9.20 (9.26)
8

---

Under review as a conference paper at ICLR 2021
Table Statisties of the top-100 100 produced molecules with highest predicted HIV scores for every reaction-baie based method used and Enamine's building blocks
NO AD
AD
Scoring
INT
CCR5
RT
INT
CCR5
ENAMINEBB
6.87 + 0.11
6.32 + 0.12
7.10 + 0.27
6.87 + 0.11
6.32
+ 0.12
6.89 + 0.32
RS
7.39 1: 0.10
6.87 1: 0.13
8.65 1: 0.06
7.31
0.11
6.87
1 0.13
8.56 1 0.08
PGFS
7,81 + 0.03
8.96
- 0.04
7.63
+ 0.09
7.15
+ 0.08
8.93 + 0.05
PGFS+MB
7.81 + 0.01
7,51 + 0.02
9.06 - 0.01
7.75
1 0.04
7.51
+ 0.04
9.01 + 0.05
7.0
6.5
7.0
6.0
6.5
(a)
5.5
4.5
5.0
time stop
step
time stop
RS PGPS
400
500
NB-PPGSS
counts
300
counts
200
(b)
300
130
200
100
100
hiv_cor5
hiv_
NV
Figure 5: Comparison of the three methods: RS (blue), PGFS (orange) and PGFS+MB (green) based on the rewards achieved starting with the 2000 initial reactants from fixed validation set.
5 CONCLUSION
In this paper, we introduced novel functional form of the Bellman equation to optimize for the maximum reward achieved at any time step in an episode. We introduced the corresponding evaluation and optimality operators and proved the convergence of Q-learning algorithm. We further showed that the proposed max-Bellman formulation can be applied to deep reinforcement learning algorithms by demonstrating tate-of-the-results the task of de novo drug design across several reward functions and metrics.
REFERENCES
Sungsoo Ahn, Junsu Kim, Hankook Lee, and Jinwoo Shin. Guiding deep molecular optimization with genetic exploration, 2020.
Sebastian Becker, Patrick Cheridito, and Arnulf Jentzen. Deep optimal stopping, 2020.
Richard Bellman. Some applications of the theory of dynamic programming review. Oper. Res.. ((3):275-288, 1954.
G Richard Bickerton, Gaia V Paolini, Jérémy Besnard, Sorel Muresan, and Andrew L Hopkins. Quantifying the chemical beauty of drugs. Nature chemistry, 4(2):90, 2012.
John Bradshaw, Brooks Paige, Matt Kusner, Marwin H. S. Segler, and José Miguel Hernández- Lobato. A model to search for synthesizable molecules. CoRR, abs/1906.05221, 2019.
Nathan Brown, Ben McKay, François Gilardoni, and Johamn Gasteiger. graph-based genetic algorithm and its application to the multiobjective evolution of median molecules. Journal of chemical information and computer sciences, 44(3): 1079-107777 2004.

---

Under review as a conference paper at ICLR 2021
Nathan Brown, Marco Fiscato, Marwin HS Segler, and Alain C Vaucher. Guacamol: benchmarking models for de novo molecular design. Journal of chemical information and modeling, 59(3): 1096-1108, 2019.
Alexander Button, Daniel Merk, Jan A Hiss, and Gisbert Schneider. Automated de molecular design by bybrid machine intelligence and rule-driven chemical synthesis. Nature machine intelligence, 1(7):307-31,, 2019.
Kai-Wei Chang, Akshay Krishnamurthy. Alekh Agarwal, Hal Daumé III, and John Langford. Learning to search better than your teacher. CoRR, abs/1502.002066, 2015. URL
abs/15005 02206
Devendra Singh Chaplot, Emilio Parisotto, and Ruslan Salakhutdinov. Active neural localization. In International Conference on Learning Representations, 2018. URL /ttttpss///opeeeeeeeee
66b
Xinyun Chen and Yuandong Tian. Learning perform local rewriting for combinatorial optimization.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Buc, E. Fox, and R. Garnetr (eds.), Advances Neural Information Processing Systems, volume 32. pp. 6281-6292. Curran As- sociates, Inc., 2019, URL httLLttttp//::oceeeeeeee neurips.........p..p...... cc
Scott Fujimoto. Herke van Hoof, and David Meger. Addressing function upproximation error in actor-critic methods. CoRR, abs/1802.09947,, 2018a.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint arXiv: 1802.09477 2018b.
Wenhao Gao and Connor W. Coley. The synthesizability of molecules proposed by generative models. Journal of Chemical Information and Modeling, Apr 2020. ISSN 1549-960X. doi: 10.1021/aassjeimm.00001 0c00174.
Rafael Gömez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato. Sánchez-Lengeling,
Benjamín Sánchez- Lengeling. Dennis Sheberla, Jorge Aguilera-parraguirge Iparraguirre, Timothy D) Hirzel, Ryan P Adams, and Alán Aspuru-Guzik, Automatic chemical design using data-driven continuous representation of molecules. ACS central science, 4(2):268-22 276, 2018.
Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Karam M. Thomas, Simon Blackburn, Connor W. Coley, Jian Tang, Sarath Chandar, and Yoshua Bengio. Learning to navigate the synthetically accessible chemical space using reinforcement learning. In Proceedings of the 37th International Conference on International Conference on Machine Learning, ICML'20, 2020.
Ryan-Rhy Griffiths and José Miguel Hermández-Lobate Constrained bayesian optimization for automatic chemical design using variational autoencoders. Chem. Sci., 11:577-586, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. arXiv preprint arXiv:1801.01290 2018.
R. A. Howard. Dynamic Programming and Markov Processes. MIT Press, Cambridge, MA, 1960.
Tommi Jaakkola, Michael I. Jordan, and Satinder P. Singh. Convergence of stochastic iterative dynamic programming algorithms. In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS'93, pp. 703-710, San Francisco, CA, USA, 1993. Morgan Kaufmann Publishers Inc.
Jan H Jensen. A graph- genetic algorithm and generative model/monte carlo tree search for the exploration of chemical space. Chemical science, 10(12):3567-3555 3572, 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2323-2332. Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
10

---

Under review as a conference paper at ICLR 2021
Takashi Kamihigashi and Cuong Le Van. Necessary and sufficient conditions for solution of the bellman equation to be the value function: A general principle. 2015.
Mathilde Koch, Thomas Duigou, and Jean-Loup Faulon. retrosynthesis. bioRxiv, 2019.
Reinforcement learning for bio-
Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric P. Xing. ChemBO: Bayesian Optimization of Small Organic Molecules with Synthesizable Recommendations. arXiv:1908.0144 [physics, stat], August 2019. arXiv: 1908.01425.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 518(7540):599-8 533, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Mareus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of cheminformatics, 9(1):48, 2017.
Giscung Park, Whiyoung Jung, Sungho Choi, and Youngchul Sung. Model ensemble- intrinsic reward for sparse reward reinforcement learning, 2020. UtLaatpps://ppeneeveeew
forum? ddsmmm/dsSSxxx34xt
Brenden Petersen. Deep symbolic regression: Recovering mathematical expressions from data via
risk-seeking policy gradients, 2020.
Mariya Popova, Olexandr Isayev, and Alexander Tropsha. Deep reinforcement learning for de novo drug design. Science advances, 4(7)xeaa 2018.
K.H. Quah and Chai Quek. Maximum reward reinforcement learning: A non-cumulative re-
ward criterion. Expert Systems with Applications, 31(2):351 359, 2006. ISSN 0957-4174. doi: http:///o..og///...11///eee....o 2005000000 09.044. URLpttp:: www. www.ssseeenedrer.....mm com/
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom van de Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. volume 80 of Proceedings of Machine Learning Research, pp. 4344 4353, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR. http: html
//proceedings.no wtt.gss.mm..p..ers.... wwwpppewwsseeeeeeeee.eeeppepeeee wwwp//////eeleeeeeeeeeeeeeaaaeee
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Chence Shi*, Minkai Xu Zhaocheng Zhu, Weinan Zhang, Ming Zhang, and Jian Tang. Graph (af): flow-based autoregressive model for molecular graph generation. In International Conference on Learning Representations, 2020.
Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, pp. 412-422. Springer, 2018.
Satinder Singh, Tommi Jaakkola, Michael Littman, and Csaba Szepesvári. Convergence results for single-step on-policy sinforcement-leamiing algorithms. Machine learning, 38(3):287-308, 2000.
Csaba Szepesvári. The asymptotic convergence-rate of q-learning. In NIPS, pp. 1064-1070, 1997.
Alexander Tropsha. Best practices for qsar model development, validation, and exploitation. Molecu- lar informatics, 29(6-7):476-488) 2010.
11

---

Under review as a conference paper at ICLR 2021
Alexander Trott, Stephan Zheng, Caiming Xiong, and Richard Socher. Keeping your dis- tance: Solving sparse reward tasks using self-balancing shaped rewards. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue. E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, pp. 10376-10386. Curran Asso- ciates, Inc., 2019. URL httptps///proceseddnngg neur reeuripp... pps wttppppppp/../../0/. 220///2119//F// [4222222000006649990000223333-32
pdf
John N Tsitsiklis. Asynchronous stochastic approximation and q-learning, Machine learning, 16(3):
185-202, 1994.
Silviu-Marian Udrescu and Max Tegmark. Ai feynman: physics-inspired method for symbolic regression. Science Advances, 6(16), 2020.
Christopher J. C. H. Watkins and Peter Dayan. Technical note: q -learning. Mach. Learn., 8(3-4):
279-292, May 1992.
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 28(1):31-36, 1988.
Robin Winter, Floriane Montanari, Andreas Steffen, Hans Briem, Frank Noé, and Djork-Amé Clevert. Efficient multi-objective molecular optimization a continuous latent space. Chemical science, 10(34)88016-800 8024, 2019.
Jiaxuan You, Bowen Liu, Rex Ying, Vijay S. Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. CoRR, abs/1806.02477 2018a.
Jiaxuan You. Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph convolutional
policy network for goal-directed molecular graph generation. In Advances in neural information processing systems, pp. 6410-6421, 201 8b.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. (eds.), Advances in Neural Information Processing Systems, volume 31, pp. 0164444654 Curran As- sociates, Inc., 2018. URL wttp:ttttts:///oeeednngg www.rdiiinnnnnnnnnnn.ne.om neurips /ttprr/spppppppppp/000/////////
and R. Garnett
pdf
Zhenpeng Zhou, Steven M. Kearne,, Li Li, Richard N. Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. CoRR, abs/1 100.0088,,,, 2018.
12

---

Under review as a conference paper at ICLR 2021
A PROOF -MONOTONICITY AND CONTRACTION PROPERTIES OF M
Proof. Monotonicity: Let Q1 and Q2 be two functions such that Qi a) 2 Q2(s, a) for any state- action pair (s,a) E 8 x A. By linearity of expectation, we have (Bp (Q(s'' Z ((Bs))() [Q2(s', d')l (s, As (s, a) z (BBxsP()s.,) (Qi(s',a') we get M Q( 2 /
(0z(s',a')) Moreover, because M (1(,,),) 2 r(s,a), we obtain M²=Q2(s.a)
M M²Q1(s,a) 2 max (B/ (Q2(s',a'))
which is the desired result.
Contraction: Let us denote the expected action-value function of the next state by fi obtaining the following equation:
fi(s, (P [Qi(s', .')))
a'~x(-ls')
By using the fact that max(r, v) = 0.5(z +
E R2, we obtain
max(r, fi) max(r, f2) = 0.5 (r + -- - 0.5 + f2 + Ir f2l)
0.5 (ʃi -2 frlt-r-----r---
I 0.5 (fi - -----ff f-r---r--
0.5 (fi -f2 Ii f2l)
< 1/1 - f2]
Therefore
| - M 02 = maax(r, f2)
S 11 - /21100
S
- E Q2(: a²) llol
= MI (Qi(;,d') - Q2(, d'))
S 7ll max (Qi a') - Q2(: ())
a'
< 7 max [(Q)(, Q2(: a())
?1|Q1 Q211a0
B OTHER RELATED IDEAS
While the discussion far has been focused on formulating novel functional form of the Bellman equation for optimizing for the expected maximum reward in an episode (and proving its convergence properties and applying it on toy domain and more complex de-novo drug design environment), there are potentially other possible ways to approach this problem, some of which were tried by us.
possible alternate approach is to just try optimizing for reward the final time step (You al. (2018a), Shit et al. (2020)). This leads to a sparse reward problem and the training becomes harder. still active research area and several approaches were introduced to address this issue of sparse rewards. For example, shaping the rewards (Trott et al. (2019),, learning auxiliary tasks (Riedmiller al. (2018), learning intrinsic rewards (Zheng et al. 2018). ) using ensemble of models to learn intrinsic rewards (Park et al. (2020)). However, these approaches optimize only for the reward at the end the episode, whereas, the maximum reward can occur at any time step in the episode. Optimal stopping based approaches (Becker et al. (2020) learns to stop the stream of rewards they receive but to the best of our knowledge, there isn't any approach with a thorough theoretical proof that simultaneously learns to stop the episode and to optimize for the reward the end of episode in a reinforcement learning framework. Nonetheless, we tried to have a policy network that can choose an additional 'stop' action. The rewards did not converge the policy network got biased
13

---

Under review as a conference paper at ICLR 2021
towards choosing 'stop' action most of the time. We tried other tricks like shaping the reward in the range [0.1] (instead of [-1, ID. using unshaped rewards (range [0, oc) ), using difference of rewards (Chen & Tian (2019)) and adding penalty for choosing the stop action. Even though these ideas interesting, they do not have a thorough theoretical foundation relating to the objective of maximum reward optimization and they performed worse than PGFS. An interesting future work would be to combine the optimal stopping framework with the actor-critic framework to solve the maximum
objective problem. This outside the scope of our current work. Chang et al. (2015) is potentially another way to search in the policy space on which our max- Bellman formulation could be tried.
C MARKOVIAN GOLDMINING TOY EXAMPLE
We also run a comparison between Q-learning and Max-Q on a fully Markovian setting on a similar goldmining environment. The Markovian property is enforced by not removing the rewards once the goldmines have been visited, and by also including the timestep information in the state observation. From the learning perspective, the latter is enforced simply by making the Q-value table dimensional with the first dimension referencing the timestep.
The markovian setting is similar toy domain with multiple goldmines in 6 grid. The agent starts in the bottom left corner and at each time step, can choose from among the four cardinal actions: up, down, left and right. The environment deterministic with respect the action transitions and the agent cannot leave the grid. All states in the grid that are labeled with values other than -1 represent goldmines and each value represents the reward that the agent will collect upon reaching that particular state. Transitions into non- non- non-goldmine state results in a reward of -1. The episode terminates after time steps and the discount factor is 7 = 0.99. The observation a tuple consisting of an integer denoting the state number of the agent and an integer denoting the current time step.
The environment is shown in Figure 6 If the agent goes up to the top row and continues to move right, it will only get cumulative return of 6. If instead traverses the bottom row to the right, it can receive a cumulative return of 9, which is the highest cumulative return possible in this environment.
-1.0
-10
-1.0
10.0
-1.0
-1.0
-1.0
-8.0
-8.0
-8.0
-8.0
-8.0
-1.0
-1.0
-1.0
6.0
6.0
Figure 6: A visualization of the gold-mining toy example. The bottom left state, shown in green, denotes the starting state. States with values other than -1 denote the goldmines and the values denote their respective rewards.
A comparison of cumulative return and maximum reward is shown in Figure g Q-learning learns the optimal policy while Max-Q learning the policy reach the highest rewarding state. Thus, consistent results are obtained for the Markovian setting as well. The exploration schedule and hyperparameters are the same as those used for the non-Markovian setting.
For the Markovian setting, the grid was made smaller and the number of goldmines were reduced. These changes were made only to make the task easier to solve. Since the rewards do not vanish in this setting, not applying these minor changes meant that it was much harder exploration problem and was observed that the agent simply oscillated between states and took much longer to converge to the optimal policy. The increased variance in the average return for Max-Q in Figure 8a is as expected, since there is no incentive for the Max-Q agent to avoid the -8 penalties. This high variance is not observed in the non-Mart Mankovaan toy example only because the rewards guide the agent around the negative reward filled middle row.
It should be noted that in the Markovian setting. since the Q-value table has dimensions, there is no straightforward way to visualize the learned policy. Figure / has been obtained by plotting the greedy action for each state after applying the mean across the time axis of the Q-value table.
14

---

Under review as a conference paper at ICLR 2021
(a)
(a) Average return
Maximum reward per episode
Figure 7: Learned policies for (a) Q-learning (b) Max-Q
Figure 8: Comparison between Q-learning and Max-Q
D PSEUDO CODE PGFS+MB
In this section, we explain the PGFS+MB algorithm in detail and provide the pseudo code.
The actor module consists of networks and क. The role of the actor module is to compute the action for given state In this case, the state is the reactant R(1), and the action outputs are reaction template T and tensor in the space defined by feature representation of all second reactants. We use ECFP feature representation (Morgan circular molecular fingerprint bit vector of size 1024 and radius 2) for state inputs (reactant-1) 1) and RLV2 feature representation (a custom feature representation) for action outputs (reactant-2). We used the following features RLV2: MaxEStatelndexx MinEStateIndex, MinAbsESttateldeex QED, MolWt, FpDensityMoogan! Bal- abanJ, PEOE-VS SMR-VSA7, SlogP-VSA3, SlogP-VSAS. EState- EState- EState- VSA4, EState-VSAS, EState- FractionCSP3, MolLogP, Kappa2. PEOE- VSA2, WVR-VSAS, SMR-VSA6, EState- VSA7, Chi4y, SMR-VSAIO SlogP-VSA4, VSA4, SlogP- VS96, EState-VSAS EState- VSA-EState9.
TEOE-VSAL 1, PEOE-VSA6, PEOE- VSA7, PEOE-VSA8, PEOE-VSA9
First, the reactant R is passed through the fettwwrr to compute the template tensor T that contains probability each of the reaction templates.
T= f=
For a given reactant R(), only few reaction templates are eligible to participate a reaction involving R(1), Thus, all the invalid reaction templates are masked off by multiplying element- wise with template mask mask, which is binary tensor with value 1 if its valid template, 0 otherwise.
7 = @ Taakk
Finally, the reaction template in met tensor format is obtained by applying Gumbel softmax operation to the masked off template T. It is parameterized by temperature parameter that is slowly decayed from 1.0 0.1
T = GumbelSoftmax(T.n
The one-hot template along with the reactant R()) is passed through the x network to obtain the action a in the space defined by feature representation of all second reactants.
( (TT)
The critic module consists of the Q-network and computes the Q(x, a) values. this case, it takes in the reactant R()), reaction template T, action a and compute its Q value: Q(R(),T
For the fairness in comparison, we used the exact same network sizes as described in the PGFS paper i.e., The fettwwrk has four fully connected layers with 256, 128, 128 neurons in the hidden layers. The network has four fully connected layers with 256, 256, 167 neurons in the hidden layers. All the hidden layers use ReLU activation whereas the final layer uses tanh activation. The Q-network also has four fully connected layers with 256, 64, 16 neurons in the hidden layers, with ReLU activation for all the hidden layers and linear activation for the final layer.
The environment takes in the current state and action and computes the next state and reward. First, it computes the set of second reactants that are eligible to participate in a reaction involving chosen reaction template T'
(2) GetValidRRectaans((7
15

---

Under review as a conference paper at ICLR 2021
The l valid reactants closest the action then obtained by passing the action and set of valid second reactants ((2) through the k nearest neighbours module.
A = KNN(a,, R!2))
For each of these i second reactants, we compute the corresponding products R(4) obtained involving the reactant ((1) and reaction template T by passing them through a forward reaction predictor
module, and then compute the corresponding rewards by passing the obtained products through scoring function prediction module.
R1 ForwardReaction/ (RaiiiiooonRR)) ,
Rewards ScoringFunction
Then, the product and the reward corresponding to the maximum reward are chosen and returned by the environment. all our experiments, we use
During the optimization ("backward")) phase, we compute the actions for next time step T.+1 a++1 using target actor network randomly sampled mini-batch.
T1+1, a++1 Actor-target( R(!)
We then compute one-step TD (temporal difference) target Vi (using the proposed max-Bellman formulation) as the maximum of reward at the current time step and discounted Q value (computed by critic-target) for the state, next action pair. To incorporate the clipped double Q-learning formulation used in TD3 (Fujimoto et al. (2018a)) to prevent over-estimation bias, we use two critics and only take the minimum of the critics.
y/ max(ra min ( ( 1=12
Note that this is different from PGFS Gottipati et al. (020)) where the authors compute the TD target using the standard Bellman formulation: 1/s r; 7 minj=112 Critic-target(( ( Te+1) a.+1). We then compute the critic loss Leritic as the mean squared error between the one-step TD target y: and the Q-value (computed by critic) the current state, action pair.
Centi = Elu - CRITIICR ( a2)
Σ
The policy loss Epolicy is negative of the critic value of the state, action pair where the actions computed by the current version of the actor network
ppllicy Σ AATORRE
Like PGFS, to enable faster learning during initial phases of the training, we also minimize an auxiliary loss which is the cross entropy loss between the predicted template and the actual valid template
Launi E(T!" (R
Thus, the total actor loss is the sum of policy loss and the auxiliary loss
Lactor = Epoticy + Lauxil
The parameters of all the actor and critie networks (f, n, Q) are updated by minimizing the actor and critic losses respectively.
min Lactors Lecitic
16

---

Under review as a conference paper at ICLR 2021
Algorithm PGFS+MB
1: procedure ())
2:
T < f(R())
3: T + T O mask
4: T GumbelSoftmax (T,7)
5: a x(R()TT)
6: return T,a
7: procedure CRITIC(R T. a)
8: return Q(RL),, a)
9: procedure (N a)
R(2) t GelValidReactans((2)
A KNN(a,
12:
(1))
1+1 < ForwardReaction (T)))
13: Rewards + ScoringFunction (F (
TG, R(D done arg max Rewards
15:
return R241 Ft. done
16:
procedure BACKWAR BACKWARD(buffer minibatch)
17:
Ti+1, 0++1 Actor-target (R
18:
Y: f max 7 min Critic-target( ( (11)),41)))
(1)
19:
Ceritic + E, 1/: CRITIC(R aT;
20:
Cpolicy + - E, CRITIC ACTOR(R()))
Lowvit t- - Ex(T()) If (R (R(")))
21:
22:
Lactor +- Lpolicy Lauxtt
23:
min Lenctory Leritic
procedure MAIN(J, n. Q) for episode = 1, M do sample RQ()) for t 0, N do
25:
27:
28:
Tr, +- Actor(
29:
[e(1)
Ri41, Tt, done + env.step( (R. Tr, at)
store (R("), Tr, at, R!l)) 1. Tr. done) in buffer
31:
sample random minibatch from buffer (anibard(minibatch)
E EXPERIMENTAL SETUP AND RESULTS
Our experimental setup same as that of PGFS but include here for the sake of completeness. The set of 150,560 reactants used in this study were taken from the Enamine Building Block catalogue Global stock and are represented as Simplified molecular-input line-entry gystem(SMILES) Weininger (1988). The 97 reaction templates were taken from Button et al. (2019) and are represented as SMILES arbitrary target specification (SMARTS). For given class of reaction the template contains the pattern of reactive centres and the product across unimolecular and bimolecular reactions.
We test the performance our model on five different rewards which are estimated from the molecular structure of the compound. One of the most used measure for frug-likeelinss, is quantitative estimate of drug-likeliness (QED) Bickerton et al. 2012). Octanol-water partition coefficient (clogP) You al. 20181 8b) measures the lipophilicity the aliphatic nature of the molecule which determines the solubility of the molecule in aqueous phase and increases with the size of the molecule. The other three rewards related specifically to the Human Immunodeficiency Virus (HIV), which extends the applicability of our method to biological targets, C-C chemokine receptor type 5 (CCR5), HIV Integrase (INT), HIV Reverse Transcriptase (RT).
17

---

Under review as a conference paper at ICLR 2021
Even though the main motivation of this work is to build existing reaction- methods (PGFS) to ensure that the molecules generated are synthesizable, we also include the comparison with other non-reaction based methods that were believed to achieve state-of-the-art results in the last few years. These methods include junction-tree variational auto encoder (JTVAE) (Jin et al. (2018)) Graph Convolutional Policy Network (GCPN) (You et al. (20T8b)), Molecule Swarm Optimization (MSO) (Winter et al. (2019))
Table 3: Performance comparison of de novo drug design algorithms in terms of maximum reward achieved
Method
QED
clogP
RT
INT
CCRS
ENAMINEBB
0.948
5.51
7.49
6.71
8.63
RS
0.948
8.86
7.65
7.25
8.79 (8.86)
GCPN
0,948
7,98
7.42(7.45)
8.20(8.62)
JT-VAR
0.925
5.30
7.58
7.25
8.15 (8.23)
MSO
0.948
26.10
7.76
7.28
8.68 (8.77)
PGES
0.948
27.22
7.89
9.05
PGFS+MB
0.948
27.60
7.97
9.20
Table 4: Statistics of the top-100 produced molecules with highest predicted HIV scores of de novo drug design algorithms and Enamine's 's building blocks
NO AD
AD
Scoring
RT
INT
CCRS
RT
INT
CCR5
ENAMINEBB
6.87
0.11
6.32
1 0.12
7.10 1 0.27
6.87 1 0.11
6.32 1 0.12
6.89
0.332
RS
7.39 0.10
6.87
+ 0.13
8.65 + 0.06
7.31
+ 0.11
6.87 0.13
8.56 4.008
GCPN
7.07 0.10
6.18
0.09
7
6.90
0.13
6.16 0.09
6,95 0.05
JTVAE
7.20 0.122
0.14
7.60
0.16
7.20 1 0.12
6.75
6.7510.14
7.44
MSO
7.46 0.12
8.23 1 0.24
7.36 1 0.15
6.84
7.92
1 0.61
PGFS
7,81 0.03
7.16
0.09
8.96
+ 0.04
7.63 + 0.09
7.15 + 0.08
8.93 + 0.05
PGFS+MB
7.81
0.01
7.51 - 0.02
9.06 = 0.01
7.75 + 0.04
7.51 = 0.04
9.01 0.05
Maommm E3
Eprociion
Figure 7: Comparison of the PGFS (blue) and PGFS+MB (orange) based on the maximum reward achieved in each episode during the course of training
SuBsaaTió Frreotion
Reduckin ariootts
Rowedd 7:38
Reward 7.51
Rewerd
Figure 8: visualization of synthesis path for generating highest reward molecule (obtained using PGFS+MB) against HIV-RT target
18

---

Under review as a conference paper at ICLR 2021
Grea Clmaee 12
Are fommaton
Wileramn Eber
Rewad 544
Reward 1..00
Figure 9: visualization of synthesis path for generating highest reward molecule (obtained using PGFS+MB) against HIV-INT target
Eirratio
Raward:8.3
Rawart: 922
Figure 10: A visualization of synthesis path for generating highest reward molecule (obtained using PGFS+MB) against HIV-CCRS target
19

---

Under review as a conference paper at ICLR 2021
8.52
12.55
18:30
22.28
27.80
Figure 11: A visualization of synthesis path for generating highest reward clogp molecule (obtained using PGFS+MB)
20


Reviews:
[{"review_id": "kWf6Pobr5mx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThis paper proposes a new reinforcement algorithm based on max-Bellman operator which trains the policy to optimize the maximum reward achieved in a trajectory, i.e., $R(\\tau) = \\max_{t\\geq 0} \\gamma^{t}r_{t}$. The authors analyze how the newly proposed max-Bellman operator leads to an optimal policy. Experiments on a toy task and de novo drug design tasks show better performance compared to the considered baselines. I think the proposed idea is promising, timely, and impactful. However, I think (a) the problem setting should be clarified and (b) the empirical evaluations are below the standard of ICLR conference. Especially, regarding (b), the experiments only compare with the Bellman operator under MDP with cumulative rewards that do not align with the true objective. However, researchers already know how to design MDPs with cumulative rewards aligning with the true objective and this paper should consider this as their most important baseline.\n\nPros:\n- This paper tackles an important class of problems, where the agent aims to identify states (or solutions) that achieve the highest score. Examples include drug discovery, (mathematical) combinatorial optimization problems, and program synthesis. \n- The proposed algorithm is fundamental and can be extended to many problems. \n\nCons:\n- This paper confusingly use the terminology \"reward function\" to indicate two meaning. First, it is used as a reward function associated with a Markov decision process (MDP) (taking $s_{t}$ and $a_{t}$ as inputs), where the agent is trained to optimize the cumulative summation or maximization of rewards over the trajectory. Second, it is used as a score function (taking molecule as input) that is associated with the given problem and independent of MDP (used for solving the problem). The distinction between two functions is very important and existing methods often shape rewards different from the scoring criteria. \n- In the experiments, the authors only compare the Bellman and max-Bellman operators defined on an identical MDP. Especially, the MDP is designed so that only the max-Bellman operator aligns with optimization of the true objective of the problem. This dismisses how one can also design MDP so that the Bellman operator can optimize the true objective of the problem. \\\nIn drug design, [You et al. 2018, Shi et al., 2020] consider assigning the desired score of the drug as the reward only at the terminal state of the MDP. This allows the Bellman operator to properly solve the drug design problem without any modification. I note how this approach is briefly mentioned in the introduction. Authors claim that they fail to optimize for the very high reward molecules that may be encountered in the middle of the episode. While I partly agree with such a claim, this should be empirically verified to prove the empirical superiority of the max-Bellman operator over the Bellman operator. \\\nIn combinatorial optimization and program synthesis, e.g., [Chen et al. 2019], it is common to assign the difference of scores between intermediate states, i.e., $r_{t} = c_{t}- c_{t-1}$ where $c_{t}$ is the true objective of the problem evaluated at state $s_{t}$. This also alleviates the mismatch between the training of RL and the true objective of the problem.\n- For the drug design experiment, I also suggest the authors provide a comparison on the maximum reward per episode during training, e.g., Figure 4 (b), to ablate the effect of the best max-Bellman operator. \n- In molecule generation tasks, it is common to provide an illustration of the generated molecules to check whether if they are indeed helpful and synthesizable in real-life. This aspect is especially important since PGFS was initially proposed to constrain the drug design over molecules with a synthesizable structure. \n- Given the fundamental nature of the proposed algorithms, I encourage the authors to provide more demonstrations on the superiority of the max-Bellman operator. Especially, I think it is important to compare with settings where the agents receive rewards at end of the episode, e.g., [You et al. 2018]. This paper only compares with the case where the cumulative summation of score function is maximized.\n- In (No Ad, RT) column of Table 2, PGFS+MB is marked bold even though it achieves the same score as PGFS.\n\n[You et al. 2018] Graph convolutional policy network for goal-directed molecular graph generation\n\n[Chen et al. 2019] Learning to Perform Local Rewriting for Combinatorial Optimization\n\n[Shi et al. 2020] GraphAF: a flow-based autoregressive model for molecular graph generation\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XT_aWlOMtE8", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary:\n\nThis paper proposes a modified bellman equation for reinforcement learning that optimizes the maximum expected single step reward along a trajectory, instead of the maximum cumulative reward. This formulation is applied to the generation of molecules with optimized properties of interest. A recently published molecule generation algorithm, that constructs molecules step wise via the (predicted) chemical reactions of building blocks, is modified with this new bellman formulation, and shows modest improvements in optimizing for some HIV activity targets.\n\n##########################################################################\n\nReasons for score: \n\nOverall, I vote for acceptance. The proposed max-bellman formulation seems to make sense in the context of molecule generation. However, the performance of this formulation in the molecule optimization task is not particularly impressive.  \n\n##########################################################################\n\nStrengths:\n\n*Paper is generally written and structured clearly\n\n*(From a non-RL expert), the proposed max-bellman formulation seems to make sense for molecule generation\n\nWeaknesses:\n\n*Performance improvement of the max-bellman formulation in the molecule optimization task is pretty modest. Eg from table 2, the PGFS model + max bellman only shows significant improvements over the vanilla PGFS baseline in 1 out of the 3 HIV tasks (with applicability domain restrictions on the property predictor)\n\n##########################################################################\n\nOther comments:\n\n*“For all the three HIV reward functions, we notice that PGFS+MB performed better than existing reaction-based RL approaches (i.e, PGFS and RS) in terms of reward achieved at every time step of the episode” – why is it important that the hiv_x scores are compared at each step. Shouldn’t we only care about the top hiv_x scores independent of the step?\n\n*“However, in the proposed formulation, we noticed that the performance is sensitive to the discount factor and the optimal is different for each reward.” – how is the discount factor tuned for the different models in the molecule generation task\n\n*What are the summary statistics of the top 100 produced molecules for QED and clogP?\n\n*In table 1 and figure 3, do the reported HIV-RT, HIV-INT, HIV-CCR5 results have the applicability domain (AD) restrictions?\n\n*What is the termination criteria for the PGFS model? \n\n*One of the public molecule optimization benchmarks, Guacamol, was mentioned in the paper. What is the reasoning for not using that standard evaluation suite in the evaluation of this work?\n\n##########################################################################\n\nThanks for the response to my feedback. Unfortunately, after reading the reviews/responses from the other reviewers, I have decreased my rating to 5, due to some concerns related the technical aspects of the paper\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LYQrVfNjffI", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\n\nThis paper proposes a max reward instead of cumulative reward objective for reinforcement learning. This objective is primarily motivated by applications like chemical synthesis where the goal is for the RL agent to generate the most desirable state possible. The paper then defines the corresponding varaint of the Bellman operator (the max-Bellman operator) and proves tabular convergence guarantees by a contraction argument. Some experiments in a gridworld and simulated chemical synthesis indicate that this objective modification can improve prior algorithms. \n\n--------------------------------------------------------------------\n\n**Strengths:**\n\n1. Formulation of the problem. It is a useful insight that many application may want to use reinforcement learning as a generative model over the state space. In these cases, cumulative reward may not be the best objective.\n2. Theory showing algorithm is sound. The derivation of the modified Bellman operator and subsequent algorithm is done cleanly. The proof of contraction and convergence \n3. Proof of concept that the new objective is useful. The application to chemical synthesis shows that the new problem formulation is able to improve performance on a relevant task.\n\n--------------------------------------------------------------------\n\n**Weaknesses:**\n\n1. Novelty and comparison with prior work. This is the primary weakness for this paper and there are a few ways it manifest itself. The main issue is that a quick search for related work turned up the 2006 paper \"Maximum reward reinforcement learning: A non-cumulative reward criterion\" by Quah and Quek [1]. From what I can tell, they propose the exact same problem formulation and algorithm based on a modified Bellman equation as this paper. Granted, the motivation and subsequent theory are different and I assume the authors had no knowledge of the Quah and Quek paper (I did not either before looking for related work). This does not invalidate the usefulness of presenting this max reward formulation again to a new community for new reasons, but this paper should be cited and claims of novelty should be reduced. A broader discussion of prior work that looks at non-cumulative objectives like reward at the final state would also be helpful to contextualize the paper.\n2. Clarity of the motivation for new formulation. Intuitively it is true that a generative model may only care about the maximum reward on a trajectory, but there is not a clear and formal decription of the separation between problems that can be represented as max reward versus cumulative reward problems. For example, it is not clear that doing something like using a time dependent reward that only has nonzero reward at the last state in a trajectory cannot capture most of the relevant generative problems. A more formal description of the sorts of problems that are not representable as cumulative reward problems would be useful.\n3. The toy example problem leaves some unanswered questions. Specifically, the use of a non-markovian problem makes the example somewhat suspect. By using a non-markovian problem, the example is now outside of the setting being discussed in the rest of the paper. It is not clear that the non-markovian nature of the problem is necessary to make the max-Bellman algorithm look better, so I would suggest coming up with a modified example that respects the MDP structure while still showing the utility of max-Bellman.\n4. Description of experimental setting is lacking. It is entirely unclear from reading the paper what exactly the experiments are doing, just that they use the ENAMINE dataset of reactants and several reward functions are cited. The state and action spaces are not defined. The transition dynamics are not defined. And the reward functions are not described or defined in a self-contained way. All of this information should be provided (in the appendix, no need to put it in the main text). Without this information it is difficult to judge how useful the experiments are, especially for someone not intimately familiar with related work from the RL for drug design community (as would be the case for most readers at ICLR).\n5. Presentation of experimental results is unclear. There is not much analysis of each of the figures which leaves some things unclear. For example, in figure 3(a) it appears that MB outperforms cumulative reward at every timestep. This would seem to mean that the MB algorithm in fact gets higher cumulative reward than an algorithm trained to maximize cumulative reward. This would immediately call into question the story about why MB is a useful modification if it actually leads to better performance on the cumulativ reward objective in this specific task. I may be misunderstaning this plot, but this lack of clarity is a problem. As another example, the gaps in table 1 seem relatively small between PGFS and PGFS+MB. A discussion about the scale of the improvement we can expect from MB would be useful to contextualize these results.\n\n--------------------------------------------------------------------\n\n**Recommendation:**\n\nI gave this paper a score of 5 (weak reject). This reflects the fact that I think the paper introduces an interesting problem and clean solution, but does not do a good job connecting to prior work and has a few issues with clarity especially in the experiments. I gave a confidence of 3 primarily because I am not very familiar with the literature on ML/RL for drug design so I cannot precisely guage the potential impact of the paper on that subfield. \n\nI am willing to increase my score if the authors provide a more comprehensive connection to prior work and improve the clarity and experiments section based on the weaknesses listed above.\n\n--------------------------------------------------------------------\n\n**Questions for the authors:**\n\n1. Is there potentially a connection between the proposed maximum reward formulation (especially for chemical synthesis) and the learning to search approach to structured prediction problems (see e.g. [2])?\n2. Is there a connection between the proposed maximum reward formulation and optimal stopping problems?\n\n--------------------------------------------------------------------\n\n**Additional feedback:**\n\nTypos:\n\n- The last paragraph on page 1 is not grammatically correct. Each phrase should be formulated like \"symbolic regression which is interested in\" instead of \"symbolic regession is interested in\".\n- In section 2.1 it should be \"RL algorithms easily generalize across states\" instead of \"the RL algorithms are easily generalized across states\"\n- In the leftmost column on table 2, the PGFS score of 7.81 ought to be bolded as well since it is equal to the score of PGFS+MB.\n\n\n\n[1] Quah, Kian Hong, and Chai Quek. \"Maximum reward reinforcement learning: A non-cumulative reward criterion.\" *Expert Systems with Applications* 31, no. 2 (2006): 351-359.\n\n[2] Chang, Kai-Wei, Akshay Krishnamurthy, Alekh Agarwal, Hal Daume, and John Langford. \"Learning to search better than your teacher.\" In *International Conference on Machine Learning*, pp. 2058-2066. PMLR, 2015.", "extra_scores": {}, "extra_text": {}}, {"review_id": "52yofTegTAH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Motivated by the de novo drug design, this submission proposed a new objective in reinforcement learning, i.e., to maximize the expected maximum rather than the accumulated reward along trajectories. The authors defined the corresponding Bellman operator, and then proved its theoretical properties, including monotonicity and contraction. In the experiments, the authors first showed on a simulated grid that when compared with Q-learning, the proposed Max-Q algorithm can achieve higher maximum rewards along trajectories. Finally, the authors tested on de novo drug design task, by modifying the TD target in the previous PGFS algorithm. The new variant achieved better performance across different metrics.\n\nThis work has a good motivation, from a practical perspective. The authors also investigated theoretical properties of the proposed operator, and the results are well presented, which I also appreciate. Furthermore, the simulation results are consistent with theoretical properties to show different characteristics of the Max-Q algorithm. The results on de novo drug discovery are encouraging, as the proposed method consistently outperforms its competitors, thus highlighting its practical significance. On the other hand, some descriptions are a bit confusing and more clarification is needed.\n\nMy detailed comments and questions are as follow:\n1. In Section 2.2., could you provide more explanation on how the maximum reward along a trajectory is related to the the issue of synthesizability there?\n2. When proving contraction in Page 4, f_i looks not to be a function of (s, a). Also, can you add the input, i.e. (s, a), for $r$ in the proof to make it more explicit?\n3. What exactly is the reward $r$ in drug discovery in Section 4.2? How is $r$ related to the metrics in Tables 1 and 2, e.g., RT & INT?\n4. In Page 7, the definition for L_{auxil} is unclear. How does the two terms interact each other?", "extra_scores": {}, "extra_text": {}}, {"review_id": "2HSaoSBznl2", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I like the idea of finding the max-reward policy, but the entire framework seems to stand on a shaky ground.\n\nThe main problem is with Equation (3), where the second equality is wrong. It is easy to come up with a counter-example. Consider an MDP with 4 states plus a zero-reward absorbing state. Assume a fixed deterministic policy and $\\gamma$ close to 1. Let $s_1$ be the start state, with deterministic reward $r(s_1)=1$. With probability 1 $s_1$ goes to $s_2$. Let $r(s_2)=0$, and with equal probability it transitions to either $s_3$ or $s_4$. The immediate reward $r(s_3)=2$ and $r(s_4)=0$, after that it always goes to the absorbing state with 0 rewards thereafter. Since there can be only 2 trajectories, one with max-reward 2 and the other with max-reward 1, we have that $Q(s_1)=1.5$, but it is easy to see that $Q(s_2)=1$, and $\\max(r_1,Q(s_2))=1\\neq 1.5$.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers an alternative to the standard MDP formulation, motived by the novo drug design problem.  The formulation is meant to optimize a notion of expected maximum reward along the trajectory rather than the expected sum of rewards.  The formulation is presented through a variation of the Bellman equation.  Thus mode of presentation does not make it entirely clear what the fundamental problem is and whether it is the right formulation for the application.  The reviewers point out some problems with the analysis.   Experiments compare the proposed max-Q algorithm to Q-learning and demonstrate that it achieves higher maximum reward.  Experiments involving novo drug design show promise.

This looks like an interesting idea and direction, but the consensus view is that the project deserves further work and polish.
Decision: Reject