Biases in Evaluation of Molecular Optimization Methods and Bias Reduction Strategies
Authors: Hiroshi Kajino
Kohei Miyaguchi
Takayuki Osogami
Venue: Submitted to ICLR 2023 (2023)

Abstract: We are interested in in silico evaluation methodology for molecular optimization methods. Given a sample of molecules and their properties of our interest, we wish not only to train a generator of molecules that can find those optimized with respect to a target property but also to evaluate its performance accurately. A common practice is to train a predictor of the target property on the sample and use it for both training and evaluating the generator. We theoretically investigate this evaluation methodology and show that it potentially suffers from two biases; one is due to misspecification of the predictor and the other to reusing the same sample for training and evaluation. We discuss bias reduction methods for each of the biases, and empirically investigate their effectiveness.


Under review as a conference paper at ICLR 2023
BIASES IN EVALUATION OF MOLECULAR OPTIMIZA- TION METHODS AND BIAS REDUCTION STRATEGIES
Anonymous authors Paper under double-blind review
ABSTRACT
We are interested in in silico evaluation methodology for molecular optimization methods. Given a sample of molecules and their properties of our interest, we wish not only train a generator of molecules that can find those optimized with respect to a target property but also to evaluate its performance accurately. A common practice to train a predictor of the target property on the sample and use it for both training and evaluating the generator. We theoretically investigate this evaluation methodology and show that potentially suffers from two biases; one is due misspecificationn of the predictor and the other reusing the same sample for training and evaluation. We discuss bias reduction methods for each the biases, and empirically investigate their effectiveness.
INTRODUCTION
Molecular optimization aims to discover novel molecules with improved properties, which is often formulated as reinforcement learning by modeling the construction of a molecule using a Markov decision process. The performance of such agents measured by the quality of generated molecules. In the community of machine learning, most of the molecular optimization methods have been verified by computer simulation. Since most of the generated molecules are novel, their properties are unknown and we have to resort to predictor to estimate the properties. However, little attention has been paid how reliable such estimates are, except for few empirical studies (Renz et al., 2019) Langevin et al., 2022), making the existing performance estimates less reliable. In this paper, we study the statistical properties of such performance estimators to enhance our understanding of the evaluation protocol and we discuss several directions to improve it.
Let us first introduce a common practice to estimate the performance. Let S be set of molecules, ; S* + R be property function evaluating the target property of the input molecule, and D {(m,. f" (m))) E S* x ] be a sample. We typically train a predictor fmm; D) using D, regard it the true property function, and follow the standard evaluation protocol of online reinforcement learning. That is, an agent is trained so as to optimize the properties of discovered molecules computed by f (m; D), and its performance is estimated by letting it generate novel molecules and estimating their properties by (m; D). We call this plug-in in performance estimator (section 2.
Our research question is how accurate the plug-in performance estimator is compared the true performance computed by f+. We first point out that the plug-in performance estimator is biased two ways, indicating that misspecificatiin bias comes from the deviation between the predictor and the true property function evaluated over the molecules discovered by the learned agent. This bias is closely related to the one encountered in covariate shift (Shimodaira) 2000). It grows if molecules discovered by the agent become dissimilar to those used to train the predictor. The second bias called a reusing bias is caused by reusing the same dataset for training and testing the agent. Due to these biases, the plug-in performance estimator is not necessarily a good estimator of the true performance.
reliable in general (section 2.2 The first bias called model
We then discuss strategies to reduce these two biases. Section 3.1 introduces three approaches to reducing the misspecification bias. Since it is caused by covariate shift, it can be reduced by training the predictor taking it into account (section 3.1.1) and/or by constraining the agent that the generated molecules become similar to those the sample (...2). Yet another approach use more sophisticated estimator called doubly-robust performance estimator (section 3.1.3).

---

Under review as a conference paper at ICLR 2023
Our idea correct the reusing bias comes from the analogy to model selection (Konishi & Kitagawa 2007), whose objective is to estimate the test performance by correcting the bias of the training performance, i.e., the performance computed by reusing the same dataset for training and testing. Given the analogy, one may consider train-test split could be the first choice. We however argue that it is not effective that applied to model selection due to the key difference between our setting and model selection; the test set in model selection is used to take expectation, while that in our setting is used to train predictor, which is much more complex than expectation. This complexity introduces a non-neg legligibbe bias to the train-test split estimator, resulting in a less accurate bias estimation (section 3.2.T). We instead propose to use a bootstrap method in section which is proven to estimate the reusing bias more accurately than the train-test split method does.
We empirically validate our theory section 4 First, we quantify the two biases, and confirm that both of them are non-negligible, and the reusing bias increases as the sample size decreases, as predicted by our theory. Second, we assess the effectiveness of the bias reduction methods, and confirm that the reusing bias can be corrected, while the misspecification bias can be reduced but at the cost of performance degradation of the agent.
Notation. For any distribution G, let G ~ GN denote the empirical distribution of a sample of N items independently drawn from G. For set X, let og be Dirac's delta distribution X. For any integer M E N, let [M] = (O....,M M- - 1]. For any set A, P(A) denotes the set of probability distributions defined over A.
Problem setting. We define molecular optimization problem using a Markov decision pro- cess (MDP) of length H + 1 (H E N). See appendix A for concrete examples. Let S be a set of states, and s E S be the terminal state. Let S* C S be subset of states that correspond to valid molecules and the rest of the states correspond to possibly incomplete representations of molecules (invalid molecules). Let A be set of actions that transform valid or invalid molecule into another one. There exists the terminal action To E A that evaluates the property of the molecule at step H, after which the state transits to the terminal state sT. For each step h [H + 1]. let Th: S x A + PPS) be a state transition distribution, rh S x A + IR be a reward function, and Po PP)) be the initial state distribution. We assume that the set of states at step H is limited to St, and the reward function is defined as r a) = 0 for h E [H] and (H a_) = (s) for E S*. Let LeMM= - (,,A, A, If Po, HH be the dynamical model of the MDP. Throughout the paper, we assume we know M and omit the dependency on it in expressions.
II be a policy modeled by a probability distribution over A conditioned on S. At each step h E [H + 1]. the agent takes action as sampled from Tr (- sn). The performance of a policy is measured by the expected cumulative = E [f*(((H)) where E- is the expectation with respect to the Markov process induced by applying policy on M. Letting PK E P(S*) be the distribution of states visited by policy æ at step h E [H + 1). the expected cumulative reward is alternatively expressed as, (*) Es (S).
Let II be the set of policies and ग { (- }))
reward, (T) == E
H E
- Th (S,.
In practice, the property function is not available and instead a sample from it, D {m,,. (f (m,,)) S x RI m is available. Let us assume that each tuple is independently distributed according to G EEN P(S R). Let Gs E P(S+) be the marginalized distribution over S* induced from G. For theoretical reason clarified in appendix B we use the empirical distribution of the sample, â E P(S* x R), rather than the sample itself (assumption 10) and we call & an empirical distribution and a sample interchangeably.
Let us define a policy learner a P(S x R) + II, an algorithm to learn a policy from distribution over S* x R. It typically receives sample Ô and outputs a policy, which we denote î == a2 (G). Our objective is to evaluate its performance J (+) only given access to Orr, G, and M.
BIASES OF PLUG-IN PERFORMANCE ESTIMATOR
A widely used approach to estimating (*) is plug-in performance estimator (section 2.1 We point out that it is biased in two ways (section 2.2) and theoretically characterize these biases in sections 2. and 2.4
2

---

Under review as a conference paper at ICLR 2023
2.1 PLUG-IN PERFORMANCE ESTIMATOR
For any function f: S* - IR and policy æ, let us define plug-in performance function,
Je ( f) = E² [f())
(I)
Let ay: P(S x - (S* - RR) be an algorithm to learn a predictor, typically by minimizing the loss function averaged over the input distribution. Let न ar (G) be a policy trained using and f a= be a predictor trained using the same G. Then, the plug-in performance estimator is defined as Jpr(#, f). which is often used proxy for the true performance, J (+)..
2.2 BIAS DECOMPOSITION
The plug-in performance estimator is biased in two ways; the first bias comes from model misspecification of the predictor, and the second one due to reusing the same sample for learning policy and a predictor. Let us define Jri(G1, G2) (G1)) The quantity JFI(G1. denotes the estimated performance of a policy trained with distribution G₁ evaluated by predictor trained with G2, and (pr(G1, G2) denotes the deviation of the estimated performance from the ground truth. Then, the bias we care is denoted by GG (ri G), which is decomposed shown in theorem Theorem 1. The bias decomposed into a reusing bias and a misspecification bias as follows:
(n(aa=(Gi), (G2)) and
(pI G2) = Jr1(Gı,G.)
BG-ONAPI(C G) =BG-aN [Jpr(G, G) GP G) + Jpn(G, J =B [JPI(G, G) - Jp, G)] + B API(G,
Reasing bins
Mistpecification blas
2.3 MISSPECIRICATION BIAS
Letting f == (G),,, the squared misspecification Ap((G,G)2 is upperbounded by Jensen's inequality as,
ApI (10) f
S Esupj ( -
Assuming that f = argmin, Esway (f()) - ( holds, the bias increases f fails to predict the properties of molecules generated by policy # which occurs when the predictor is misspecified (i.e., f f f*)) and pi and Gs are largely deviated (i.e., the discovered molecules are similar those in the sample).
2.4 REUSING BIAS
The former term of equation
(4) quantifies the bias caused by reusing the same finite sample for training and testing policy, which we call a reusing bias"
(G) = E () Jp( G)],
Let theoretically analyze the reusing bias, assuming the sample size N is moderately large such that
the asymptotic expansions are valid but O(1/N) term cannot be ignored. We show in proposition that the reusing bias is (1/N). See appendix B for the assumptions and appendix D.2 for its proof.
Proposition 2. Under assumptions 70 and 72
1 M IN Ex-o (2 (1.1) (ōx G, G) ((0.2)
+(02)) (8x G, G3) O(1/N9),
holds, indicating that b = O(1NN) where j(1.1) and F(022) Fréchet derivative of Jrn(GI, G2) at (G1, G2) G).
G.G are the (1, 1)st) and (0, 2)
'The reusing bias caused by sample reuse as well as the finiteness of the sample, which is clear when the policy is independent from G the reusing bias still exists such case if f f
3

---

Under review as a conference paper at ICLR 2023
In particular, if the policy is optimal and the estimated property function is unbiased, i.e., B f fo (which true at least for a linear model), we can prove that the bias is optimistic (proposition 3). See appendix for its proof.
Proposition 3. Assume E GG f = 8 and ft = argmax rnn JPI(M, f) hold. Then, (G) 2 0.
3 BIAS REDUCTION STRATEGIES
We have witnessed that the plug-in in performance estimator is biased in two ways. In this section, discuss how to reduce these biases to obtain reliable performance estimates.
3.1 REDUCING MISSPECIFICATION BIAS
There are mainly three approaches to reducing the misspecification bias, Apr(G, G). The first one to train the predictor considering the covariate shift, mismatch between training and testing distributions (section 3.1.I. The second approach is to constrain a policy such that the molecules discovered by the policy become similar to those in the sample & (section 3.1.2). These are mainly motivated by minimizing the right-hand side of equation The third one is motivated by a standard technique in contextual bandit, the doubly-robust performunce estimator instead of the plug-in performance estimator (section 3.1.3).
Before going into details, let us introduce the notion of importance weight, which is used extensively to reduce the misspecification bias. Let F E P(S*) be any probability distribution over molecules whose support is larger than that of Pt.- Let (pT (D)/F)(s) == p (s)) (s E S*) denote the importance weight between them, and let aw 11 x P(S*) + (S+ Rzo) denote an algorithm that receives a policy and distribution over molecules and outputs the importance weight between the state distribution induced by the policy and the distribution. We typically use the algorithm by substituting sample & from G for the distribution, expecting that On (T,, G) 2s PH/G.
3.1.1 COVARIATE SHIFT
The misspecification bias can be reduced by minimizing the right-hand side of equation 3 which the mean squared error over S ~ pý.. The predictor f usually trained by minimizing (swGss[[SS - S (S))² with respect to I and does not necessarily minimize the right-hand side of equation 3 due to covariate shift (Shimodaira] 2000), i.e., the mismatch between the training and testing distributions. One approach suggested by the author to alleviating it is to train the predictor by weighted maximum-liken likelihodd estimation. Let us define the algorithm as,
(2 G) argmin Es (FS)nn) (S)) jeF
(5)
where w is any importance weight and X E [0, 1] controls the bias and variance of the estimated predictor By substituting ar (w, G) for (G)) the misspecification bias will be reduced.
3.1.2 CONSTRAIN POLICY
The first approach does not always work. If pû, and G are not close enough, the effective sample size of the weighted
makiium--ikkelhhoo estimation becomes small, leading to poor estimation. This suggests that not all policy learners can be accurately evaluated; those whose state distribution pzz is deviated from G are difficult be evaluated.
Let us assume that the policy is obtained by solving the following optimization problem: az (G) = argmin,en {(;; G). While natural approach is to add divergence between the generator and the data distribution P to the objective function as a regularization term, it is computationally expensive, especially when the length of MDP, H. is large. We instead propose to regularize the policy, inspired by behavior cloning (Fujimoto& & Gu 2021). Let us first introduce behavior cloning, and then, discuss how to apply its idea to our problem setting.
While x = 1 is optimal for N + it will increase the variance for finite sample size N, and smaller A is favored.
4

---

Under review as a conference paper at ICLR 2023
Behavior cloning regularizes the policy so that the policy imitates behavior policy that generates the data. Let us assume that there exists a behavior policy T6 that induces the data distribution, i.e., PH P (s) (s) for s S", which may not be available in our setting. Behavior cloning employs the following regularized objective function: {(T; ()=; v
#+1 Σx=o Bs,,p, "haap,;,,,n (S,)m((S)) [log (AA, Ss,]], where // 2 0 is a hyperparameter controlling behavior cloning. The larger v is, the more the learned policy resembles the behavior policy, which in turn will make pi close the data distribution, and thus, we expect to reduce the misspecification bias.
technical challenge in applying behavior cloning to our setting that Tb is not available. Our key observation to this challenge is that while тb is not available, it is often the case that a trajectory towards each molecule the dataset can reconstructed. For example, in an MDP that constructs a molecule atom-wisely You et al. 2018), such trajectory is easily obtained by removing atoms one by one from the molecule; in another MDP that constructs molecule by chemical reactions (Gottipati et al. 2020), since each molecule in the dataset is assumed to be synthesizable (because the molecules in the dataset do exist in reality and thus are synthesizable), such a trajectory is easily obtained at least for those molecules in the dataset. Letting =b (m) (So. du, $1, .i,...., 8H m) be (potentially random) function to reconstruct a trajectory from a molecule, we propose to train a policy with regularization the data distribution by the following optimization problem:
H
o (G) = argmin <(T;; G) *ell
EM~GR
So. Su (Mr) [log (AA [h))
(6)
H-1
h=
Given the discussion above, least o o*( (G) can be computed. Although this regularization is not sufficient to constrain the divergence between piz and G' (which has been discussed in the literature of imitation learning), we consider the idea of behavior cloning is simple yet effective heuristic, which will be investigated in the experiment.
3.1.3 DOUBLY- ROBUST PERFORMANCE ESTIMATOR
The third approach to reducing the misspecification bias is a doubly-robus performance estimator, which has been applied in contextual bandit (Dudik et al.] 2014) and offline reinforcement learn- ing (Tang et al. 2020) as an alternative to the plug-in performance estimator. Noticing that the performance can also be estimated via importance sampling, which we call an importance-sampllee performance estimator, the doubly-robust performance estimator combines these two estimators so to inherit their benefits.
Importance-Samplinn Performance Estimator. Given that (T) = (SH)
Es-Gs (pz//C / (S) holds, obtain the importance-sampling performance estimator by substituting an importance weight model for the true importance weight. For any importance weight w: w: St - R2 and distribution F E P(S x RR, let define an importance-sampling perfor- Then, we obtain the importance-sampling performance estimator as Jis( , G). where ú a.. (ft, G).
mance function as, (w, P) = ES-FF w(S)
Doubly-Robust Performance Estimator. The doubly-rouust performance function combines the plug-in and ayportance-sampling performance functions as follows:
JDR (TT w, f. F) = Es-Fs [w(S)(f((S) This performance function is a combination of the two performance functions in that it is related to them as, JDN((T,) S, F) = (F, f) and JD,(TT, w, 0, F)- = Jis( F). By substituting म,, 16, f. and & for the arguments, we obtain the doubly-robust performance estimator as JDR(#,, w. f. G). G2),
f(S)) + E" f(SH).
(7)
Let us define, JOR (G1, G2) = (a ) (G) G2). Then, the mis- specification bias is expressed as, ADR ADR(C, = JDR (G, G) (+) Es-Gs Es-Gs(w (S) (ph, (G))(()) - (o)))) where = = (n, G). This suggests that the misspecification bias disappears if the predictor or the importance weight is well-spe.
Discussion. Notice that the misspecification biases of JrI and JIs are given by the followings:
(pi G) =Jp((f, f) ((t = EgnGs [ ())) (
Als(G, G) (1) G) - (+t = Es~GS (S) (C//C)()))
5

---

Under review as a conference paper at ICLR 2023
(S) (5) <1)(ss) holds, the misspecification bias We can deduce that for S Gs - of JDR will be smaller than that of Jis. and (ii if (S) 1 1 [C) < (S) holds, the misspecification bias of JDR will be smaller than that of JPL. Therefore, if we can learn both of the predictor and the importance weight well, the doubly-robust performance estimator is preferred to the others. Otherwise, the doubly-robust one can be worse than the others.
3.1.4 SUMMARY
We have introduced three approaches to reducing misspecification bias. The first one trains the predictor by weighted maximum likelihood estimation (equation 5). The second one constrains the policy by behavior cloning (equation 6). The third one is the doubly-robust performance es- timator (equation 7_ Taking these into consideration, let the combined performance function be, JDR (G1, G2) = (DR(O at (G G2), G2).aa(((.)))) (a
G2), G2), G2), and the com-
bined performance estimator be JR. (G,G). We call the importance weight and the predictor an evaluator. Note that proposition 2ioioonnhhhiis. for the combined performance estimator by further assuming that w is normalized and entire. Proposition [3) holds for the importance sampling performance estimator by further assuming the unbiasedness of the importance weight, but we have not found natural assumptions for the doubly-robust one. See appendix for details.
3.2 REDUCING REUSING BIAS
Given the discussion in the previous section, let us define the reusing bias for any J JIs J as, bd BG the reusing bias. Our approach is to estimate the reusing bias and substract it from the performance estimator. Such bias reduction has been extensively discussed in the literature of information criteria (Konishi & Kitagawa| 2007). which aim to estimate the test performance of a predictor in a supervised learning setting by correcting the bias of its training performance. There are mainly two approaches: train-test split method and bootstrap method.
((G.G)-((Ca
and let us discuss how to reduce
3.2.1 BIAS ESTIMATION BY TRAIN-TEST SPLIT
The first approach estimates the bias via train-test split of the sample. The sample D is randomly split into Dtrain and Dtest such that D Dtrain n Drest = 0 and Derain U Drest = D. Let Girain and Grest denote the corresponding empirical distributions. The reusing bias estimated by Aspli(Gr = B[J(Gurain: Grrain) J (Gurann Grest)], where the expectation is with respect to the random split.
While this estimator seems to be reasonable, it is not recommended for our problem setting due to the bias of the bias estimator. As demonstrated in proposition 4 the train-test split estimator has O(1/N) bias, the same order as the bias 6~ (G') itself, and therefore, it is not reliable. Such a bias is due to the non-linearlity of [G1, G2) with respect to G2, the distribution used for testing") See appendix D.2 for its proof and appendix for the comparison with supervised learning.
Propasition 4. Suppose we randomly divide the sample such that |Duraiul |Dueal = A: (1-A) for some À E (0, 1). Under assumptions 10 and 12 EG [baplit (G)] = ba (G) + O(1/N) holds.
Note that direct estimation of test performance by J(Grraann Grest) is not recommended similarly, unless the size of the test sample is sufficiently large. See appendix G for detailed discussion.
3.2.2 BOOTSTRAP BIAS ESTIMATION
An alternative approach to estimating the reusing bias (cquation 4 is bootstrap (Efron& & Tibshirani 1994). A bootstrap estimator of the reusing bias 8dd (G) is obtained by plugging G into G: 1o5(6 (G) Ex (ī G*) J( Ct). Let ((m) (m E [M] be bootstrap sample obtained by uniform- randomly sampling data points N times from the original sample & with replacement. Then, its Monte-Carlo approximation is, 6- (G) = M ( G(m) - -(( G)). In contrast to
Thhe standard supervised learning scenario does not suffer from this hias because the performance estimator linear with respect the testing distribution.

---

Under review as a conference paper at ICLR 2023
the train-test split method, the bootstrap bias estimation estimate the bias stated in proposition See appendix D.2 for its proof.
Proposition 5. Under assumptions
10 and
12
R 8a6
- (G) O(1 holds.
3.2.3 SUMMARY
We have introduced two reusing- bias estimators, referring to the literature of information criteria. We have found that the train-test split estimator, one of the most popular estimators, cannot reliably estimate the bias in our problem setting, although it works in supervised learning. In contrast, the bootstrap bias estimator is shown to be less biased than the train-test split estimator and can estimate the reusing bias more reliably. Therefore, we conclude that the bootstrap bias estimator is preferable to the train-test split estimator.
From computational point of view, the bootstrap bias estimator requires us to train M agents and M +1 evaluators. We set M 20 in the experiments given the result of a preliminary experiment. Since the bootstrap procedure can be easily parallelized with low overhead, its wall-clock time can be reduced proportion to the computational resource.
4 EMPIRICAL STUDIES
Let us empirically quantify the two biases as well as the effectiveness of the bias reduction methods. We first describe our experimental setup. See appendix H for full details to ensure reproduciability.
Molecular representation. All of the functions defined over molecules use the 1024-bit Morgan fingerprint (Morgan 1965 Rogers& & Hahn 2010) with radius 2 as feature extractor.
Environment and Agent. We employ the environment and the agent by Gottipati et al. (2020) with minor modifications. The agent receives molecule as the current state, and outputs an action consisting of a reaction template and a reactant. The environment, receiving the action, applies the chemical reaction defined by the action to the current molecule to generate product, which is then set as the next state. This procedure is repeated for H times, and lastly the agent takes action T to be rewarded by the property of the final product. We H 1 to reduce the variance in the estimated performance and better highlight the biases and their reduction. The agent is implemented by actor-critic using fully-connected neural networks.
We use the reaction templates curated by Button et al. (2019) and prepare the reactants from the set of commercially available substances in the same way as the original environment. The number of reaction templates is 64, 15 of which require one reactant, and 49 of which require two reactants. The number of reactants is 150,560.
Evaluators. As a predictor, we use fully-connected neural network with one hidden layer of 96 units with softplus activations except for the last layer. trained by minimizing the risk defined over S ~ Gs. As the importance weight, we use the kernel unconstrained least-squares importance fitting (KuLSI) (Kanamori et al.) 2012). particular, we use the trained predictor except for the last linear transformation as feature extractor and compute the linear kernel using it.
Evaluation framework. To evaluate the biases, we need the true property function f", which however is not available in general. We thus design semi-synthetic experiment using woal-world dataset Do {m,,, f (m,,) E S* x R R),,N,,,, While any function S* + R can be used as the true property function f*, we substituted the predictor provided by Gottipati et al. (2020) for f" which was trained with the ChEMBL database (Gaulton et al. (017) predict pl plCso value associated with C-C chemokine receptor type 5 (CCR5). With this property function, we have full access to the environment, and can construct offline dataset D of an arbitrary sample size by running random policy on M, which is available our setting.
To decompose the bias into the misspecification bias and the reusing bias, need f the predictor obtained with full access the data-generating distribution G. We approximate it by az (G..stt). where Grest is the empirical distribution induced by a large sample Drest of size 10a constructed independently of D. This approximation is valid if Drest. is sufficiently large (see proposition| 23} Then, the misspecification bias be estimated by J(G, Grest) J (+t and the reusing bias by
7

---

Under review as a conference paper at ICLR 2023
Deeforounnce
Behavior claring confficient
cloring JULPI
Hisqpe Eisiie Nay. Seusing Box
Figure Lines show means and shaded areas show standard deviations. (Left) Biases vs. the ) to the misspecification bias. (Middle) Comparison between bias reduction methods. (Right) Comparison between the misspecification bias, reusing bias, and the estimated reusing bias.
sample size. Jpr(f, - [f. f fo)) corresponds to the reusing bias and JfI(f, f (ft
- Grest). The performance estimators are defined by the expectation with respect to a trajectory of policy, and we estimate them by Monte-Carke Carlo approximation with 1,000 trajectories.
Quantifying the two biases. First, we quantify the misspecification and reusing biases. In specific, we study the relationship between these biases and the sample size. We vary the training sample size N in {2 2" .7..... 213]. For each N, we generate five pairs of train and test sets, and evaluate the biases indicated above. We report the means and standard deviations.
Figure (left) illustrates the result. We have three observations. First, when N = 2", the misspecifica- J* demonstrating that both are non-negligible. Second, for 2 27, the reusing bias increased as the size of the training sample decreased, which coincides with proposition 2 The results for N 27 did not coincide with because the sample size not large enough for asymptotic expansion to be justified. rather stable across different training sample
tion bias, (p,(f J- (f), was roughly twice large as the reusing bias, Jpn(ft, Jpr
Third, the ground-truth performance of the policies sizes. We found that the policies were similar to each other, suggesting that this environment has local optimum with reasonably good performance (cf., the performance of random policy is around 5.8). This also suggests that the policy learner in our experiment was insensitive to the particular sample, and the reusing bias in this case mainly caused by the finiteness of the sample train the predictor, not by reusing the same sample.
Quantifying Bias Reduction Methods. We then study the effectiveness of the bias reduction methods presented in section 3 Since the behavior cloning coefficient will control the trade-off between the misspecification bias and the performance of the learned policy, it should be determined according to the user's requirement, i.e., whether the accuracy of performance estimation the actual performance is prioritized. Therefore, we design an experiment to evaluate the effectiveness of the bias reduction methods, varying in the range {2 ...... 24).
Let J (b, b2 (+.-)) be the plug-in performance estimator with covariate shift (b) or without it (b) = -) and with bootstrap bias reduction (b2 = +) or without it (b2 = -). Let us define Jbyby accordingly for the doubly-robust performance estimator. We compare the performance
estimates by J , JPT Jprt. and JRR. to see the effectiveness of each bias reduction strategy.
Figure (middle) illustrates the performance estimates for N = Since JTR performs signifi- cantly worse than the baseline JF1 we omit it from the figure. See appendix [ for the full result. We observe that the bootstrap bias reduction worked well, while the benefit of the covariate shift strategy marginal. This indicates that the importance weight estimation did not work well in this setting.
Figure (right) illustrates the biases in J/i and the reusing bias estimated by the bootstrap method. As expected, the misspecification bias tends decrease as increase The reusing bias is under-esting eader-esttimate,, but the estimated reusing bias contributes to bias correction.
In summary, we confirm that (i) behavior cloning can reduce the misspecification bias the expense of performance degradation, (ii) the reusing bias can estimated and corrected by bootstrap, and (iii) the methods using importance weights did not perform well in our setting.
8

---

Under review as a conference paper at ICLR 2023
5 RELATED WORK
Our primary contribution is the comprehensive study of theoretically-sound evaluation methodology for in silico molecular optimization algorithms using real-world data. Since the pioneering work by Gommez-Bobareelii et al. (2018). a number of studies on this topic have been published in the communities of machine learning and cheminformatics to advance the state-of-the-an While some of them (Gómez-Bore Bombarelli et al., 2016; Takeda et al. 2020 Das et al. 2021 have been validated in vitro, many others have been evaluated silico.
Early studies (Kusner et al. 2017) adopted the octanol-water partition coefficient, log P. penalized by the synthetic accessibility score (Ertl & Schuffenhauer (2009) and the number of long rings as the target property to be maximized. The score can be easily computed by RDKit, and is often implicitly regarded as a reliable score computed by an accurate simulator. Some recently consider that the log P optimization is not appropriate as a benchmark task because it is easy to optimize (Brown et al., 2019) or its prediction can be inaccurate (Yang et al., (022), and alternative benchmark tasks have been investigated; some of them propose suite of benchmark tasks (Brown et al. 2019 Polykovskiy
2020) and the others use other property functions trained by real-world data (Olivecrona et al. 2017: Li al. 2018a: Jin al. 2020 Gottipati al. 2020 Xie et al.. (2021). However, most of the current evaluation protocols rely on the naive plug-in performance estimator.
et al.
As far as we are aware of, there are at least two empirical studies concerning about potential biases in the plug-in performance estimator. Renz et al. (2119) pointed out that the plug-in performance estimator is biased due to data reuse and random initialization of the predictor, while a follow-up study by Langevin et al. (2022) attributed the bias to the train-test split used by |Rezz et al. (2019); the train and test sets were far from being identically distributed. While these two pioneering studies shed light on the potential flaw in the plug-in performance estimator, we have not fully understand it partially because these studies are empirical.
Our contribution to this line of studies is that we not only empirically but also theoretically demonstrate potential biases in the current evaluation methodology and present bias reduction methods. This also unveils why the log P optimization task has been hacked and suggests that the alternative benchmark tasks will be hacked as long as no bias reduction method is applied. The log P function implemented in RDKit (Wildman & Crippen 1999) is obtained by fitting a linear model to dataset of experimental log P values, and is in fact predictor. Our theory suggests that unless the bias reduction methods are applied, the learned agent generates unrealistic molecules that are far from those in the dataset (which has been often reported in log P optimization), and the resultant performance estimate is biased. This mechanism is also valid for the alternative benchmark tasks, and we conjecture they will also be hacked sooner or later. It also suggests that by incorporating bias reduction methods, we can reliably estimate the performance and therefore can safely compare different methods even when using the log P optimization task.
Our work shares similar objective with a seminal work by Ito al. (2018). which aims to reduce the reusing bias that appears when solving an optimization problem w whose parameters are estimated from data. A major contribution this literature to relax their assummption that the predictor is well-specified. This introduces the concept of misspecification bias, which was confirmed to be non-negligible in our application. Another minor contribution is to formalize their reusing-bias correction method by bootstrap and investigate the theoretical properties.
6 CONCLUSION AND FUTURE WORK
We have discussed that the plug-in performance estimator is biased in two ways; one is due to model misspecification and the other is due to reusing the same dataset for training and testing. In order to reduce these biases to obtain more accurate estimates, policy such that the state distribution stays close to the data distribution and (ii) correct the bias by bootstrapping if it is non-negligible and we can afford to do it.
recommend to (i) add constraint to the
A future research direction is to improve the importance weight estimation so that the other bias reduction methods work. Another is to constrain policy with less performance degradation. Since the methods using variational autoencoders 2019) can naturally generate molecules similar to those in the data, such methods could be reevaluated.
et al. 2018 Jin et al. 2018 Kajino

---

Under review as a conference paper at ICLR 2023
REFERENCES
Nathan Brown, Marco Fiscato, Marwin H S Segler, and Alain C. Vaucher. GuacaMol: Benchmarking Models for de Novo Molecular Design. Journal of Chemical Information and Modeling, 59(3):
1096-1108, 2019.
Alexander Button. Daniel Merk, Jan A Hiss, and Gisbert Schneider. Automated de novo molecular design by hybrid machine intelligence and rule-driven chemical synthesis. Nature Machine Intelligence, 1(7):307-315, 2019. ISSN 2522-5839, doi: 10. 0011088/42226--09900666
URL
httts:://doi.com www.blo..rgg100. 1038 s2256442226622 0067-7
Payel Das, Tom Sercu, Kahini Wadhawan, Inkit Padhi, Sebastian Gehrmann, Flaviu Cipcigan, Vijil Chenthamarakshan, Hendrik Strobelt, Cicero dos Santos, Yun-Yu Chen, Yi Yan Yang, Jeremy K Tan, James Hedrick, Jason Crain, and Aleksandra Mojsilovic. Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations. Nature Biomedical Engineering, 536))613-633 2021. ISSN 2157-846X, doi: 10. 10.10038/41551-0010006063 x. URL https://dolt www.oo.orgg/00/ 101.:011111111111110000--1111110
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011. ISSN 15324435. http:// jmlrr wttp://rrrpppppppee///ssaaa htt.2//duhhhllll html
Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly Robust Policy Evaluation and Optimization. Statistical Science, 29(4):485-511, dec 2014. ISSN 08834237, 21688745, URL http:// /www jstor. wttgg/staabee4432888
Bradley Efron and Robert J Tibshirani. An introduction to the bootstrap. CRC press, 1994.
Peter Ertl and Ansgar Schuffenhauer. Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions. Journal of Chemin- formatics, 1(1):8, 2009. ISSN 1758-2946 doi: 10.1186/1758- 71.1186/7558-244611-3 URL https
Scott Fujimoto and Shixiang Gu. A Minimalist Approach to Offline Reinforcement Learning. In Beygelzimer, Y Dauphin, P Liang, and J Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
new. forum?id=
Q32U7d2WXpcc
Anna Gaulton, Anne Hersey, Michal Nowotka, A Patrícia Bento, Jon Chambers, David Mendez, Prudence Mutowo, Francis Atkinson, Louisa J Bellis, Elena Cibrián-Uhalte, Mark Davies, Nathan Dedman, Anneli Karlsson, María Paula Magariños, John Overington, George Papadatos, Ines Smit, and Andrew R Leach. The ChEMBL database in 2017. Nucleic acids research, 45(D1): D945-D954, jan 2017. ISSN 1362-4962 (Electronic). doi: 10. 10.1093/nar/gkw 1074.
Rafael G6óez--Bobbrede Jorge Aguilera-paarrguuirre, Timothy D Hirzel, David Duvenaud, Dougal Maclaurin, Martin A Blood-Forsythe, Hyun Sik Chae, Markus Einzinger, Dong- long-Gwang Ha, Tony Wu, Georgios Markopoulos, Soonok Jeon, Hosuk Kang, Hiroshi Miyazaki, Masaki Numata, Sunghan Kim, Wenliang Huang, Seong Ik Hong, Mare Baldo, Ryan P Adams, and Alan Aspuru- Guzik. Design of efficient molecular organic light-emitting diodes by a high-throughput virtual screening and experimental approach. Nature Materials, 15(10):1 120- 127, 2016.
Rafael Gómez-Bombarelli, Jennifer N Wei, David Duvenaud, José Miguel Hernández-Lobato Benjamín Sänchez-Lengelhee Dennis Sheberla, Jorge Aguilera-Iparae Iparragurre, Timothy D Hirzel, Ryan P Adams, and Alán Aspuru-Guzit Guzik. Automatic chemical design using data-driven continuous representation of molecules. ACS Central Science, 2018.
Sai Krishna Gottipati, Boris Sattarov, Sufeng Niu, Yashaswi Pathak, Haoran Wei, Shengchao Liu, Shengchao Liu, Simon Blackburn, Karam Thomas, Connor Coley, Jian Tang, Sarath Chandar, and Yoshua Bengio. Learning to Navigate The Synthetically Accessible Chemical Space Using Reinforcement Learning. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 36799 PMLR, 2020. URL gottipat 120a..100.... html
mlr. www....ee................ press/vi19.
10

---

Under review as a conference paper at ICLR 2023
Shinji Ito, Akihiro Yabe, and Ryohei Fujimaki. Unbiased Objective Estimation in Predictive Optimiza- tion. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2176 PMLR, R, 2018. htR.http::/// httett::/proceede ngs m.....mm....................mm.. ml www...................m...
html
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction Tree Variational Autoencoder for Molecular Graph Generation. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.
PMLR, 2018. URL
mlr. prwss satess/v88 /v80/
jin18a.. html.
Wengong Jin, Dr.Regina Barzilay, and Tommi Jaakkola. Multi-Objective Molecule Generation using Interpretable Substructures. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learn- ing Research, pp. 4849-4859. PMLR, 2020. URL httlttss//ppccceedinngg/ mlr press/ v119/jin20b.rr html
Hiroshi Kajino. Molecular Hypergraph Grammar with Its Application to Molecular Optimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp.
3183-3191. PMLR, 2019. /t//ittt://prrcced.ng mwr..mmrr... /aa html
Takafumi Kanamori, Taiji Suzuki, and Masashi Sugiyama. Statistical analysis of kernel-based least-squares density-ratio estimation. Machine Learning. 86(3):335-367, 2012. ISSN 1573-0565. doi: 10:10077510994--11-56663 111994-011-5220
URL
Sadanori Konishi and Genshiro Kitagawa. Information Criteria and Statistical Modeling. Springer Publishing Company, Incorporated, 1st edition, 2007. ISBN 0387718869.
Mario Krenn, Florian Häse, AkshatKumar Nigam, Pascal Friederich, and Alan Aspuru-Guzik, Self- referencing embedded strings (SELFIES): 100% robust molecular string representation. Machine Learning: Science and Technology, 1(4):45024, oct 2020. doi: 10. 00.100882633--155/aaaaa.. URL
httpps:/doii wwr.oi..rggg/00 088 / 632 2153/aba944
Matt J Kusner, Brooks Paige, and José Miguel Hermández-Lobato Grammar Variational Autoencoder. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1954. html
PMLR, 2017. URL h
mlr www.s.....rrememmmmmmmmmmmmm.see wwwpppresss.vieee wttpe///////..seee//l/
Maxime Langevin, Rodolphe Vuilleumier, and Mare Bianciotto. Explaining and avoiding failure modes in goal-directed generation of small molecules. Journal of Cheminformatics, 14(1):20, 2022. ISSN 1758-2946. doi: 10.1 20111866s13322 13311-02--000011 URL
Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of Cheminformattics, 10(1) 2018a. ISSN 1758-2946. doi: 10. (/86/>13321-018-022 0287--6. UtLbttpss://// //t:////ii.../ org. 111111/////11111 018 1101100220 0287-6
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning Deep Generative Models of Graphs, 2018b.
L. Morgan. The Generation of Unique Machine Description for Chemical Structures-A Technique Developed at Chemical Abstracts Service. Journal of Chemical Documentation, 1(2):107-113, may 1965. ISSN 0021-9576. doi: 10.1021/c1600170 72011 URL
e160017a018)
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-novo design through deep reinforcement learning. Journal of Cheminformatiis, 9(1):48, 2017. ISSN 1758-2946. doi: 10.1 10.1186//133211-777223355 -
URL httptttst///d.og//////
11

---

Under review as a conference paper at ICLR 2023
Abhishek Panigrahi, Raghav Somani, Navin Goyal, and Praneeth Netrapalli. Non-Gaussianity of Stochastic Gradient Noise, 2019.
Daniil Polykovskiy, Alexander Zhebrak, Benjamin Sanchez-Lengeling, Sergey Golovanov, Oktai
Tatanov, Stanislav Belyaev, Rauf Kurbanov, Aleksey Artamonov, Vladimir Aladinskiy, Mark Veselov, Artur Kadurin, Simon Johansson, Hongming Chen, Sergey Nikolenko, Alán Aspuru- Guzik, and Alex Zhavoronkov. Molecular Sets (MOSES): A Benchmarking Platform for Molecular Generation Models. Frontiers in Pharmacology, 11:1931, 2020. ISSN 1663-9812. doi: 10. 2889//ppar:222.555666444 Ephar. 2020 565644
Philipp Renz, Dries Van Rompaey, Jörg Kurt Wegner, Sepp Hochreiter, and Güitter Klambauer. On failure modes in molecule generation and optimizzation. Drug Discovery Today: Technologies, 32- 33:55-63, 2019. ISSN 1740-6749. doi: http:////bbooooggoo/oooooooooddd 003. URL https
//www. , sciencedi rect. com wwt.c/ooooc/eeeeeeeeeeeeeeeeeeee wttp:////w....ee.........a/a////
David Rogers and Mathew Hahn. Extended-Coomecttiity Connectivity Fingerprints. Journal of Chemical Informa- tion and Modeling. 50(5):742-754, may 2010. ISSN 1549-9596. doi: 10. 10.100//c1100050. URL
https: //pp:://di.. org/ 10. wtw.or0....//1111000000000010000 1021/c1100050
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- log-likelihood function. Journal of Statistical Planming and Inference, 90(2):227-244 224, 2000. ISSN 0378-3758, doi: https:///dii.rrg /ttp:///00.000100000007775000000
URL https:://////.. :/tp:://wwwwww /www
sci sciencedirect com httmm//ceeeneeere.ee//...../
Seiji Takeda, Toshiyuki Hama, Hssiang-Han Hsu, Victoria A Piunova, Dmitry Zubarev, Daniel P Sanders, Jed W Pitera, Makoto Kogoh, Takumi Hongo, Yenwei Cheng, Wolf Bocanett, Hideaki Nakashika, Akihiro Fujita, Yuta Tsuchiya, Katsuhiko Hino, Kentaro Yano, Shuichi Hirose, Hiroki Toda, Yasumitsu Orii, and Daiju Nakano. Molecular Inverse-Dssige Platform for Material Association for Computing Machinery, New York, NY, USA, 2020. ISBN 9781 1784440379984. httpttss:: http://ooi..ooo URL.https:///oi..rrr 11tp://...11/344411111 3011944486644433338 6
Industries, pp.
Ziyang Tang, Yihao Feng, Lihong Li, Dengyong Zhou, and Qiang Liu. Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation. In International Conference on Learning
Representations, 2020. URL wwt.thtts//// httLhttss////pneeeee http::::///ppeeeeiiee wttp:r/wwwweeeeeeeeooooooeee/edd http://oormmmm1-1111111-g111 GANtDr
David Weininger. SMILES, a chemical language and information system. 1. Introduction to method- ology and encoding rules, Journal of Chemical Information and Computer Sciences, 28(1):31-36, 1988.
Scott A Wildman and Gordon M Crippen. Prediction of Physicochemical Parameters by Atomic Contributions. Journal of Chemical Information and Computer Sciences, 396))868-877, sep 1999. ISSN 0095-2338. doi: 10.1027//1900007.. URL https: 0000///00/00000001010011 ci9903071
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the Noisy Gradient Descent that Generalizes as SGD. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10367-10376. PMLR, 2020. | wwwpssss...prrrerseeeeeeeee htt.: htt.11///200.hhmlll
Yutong Xie, Chence Shi, Hao Zhou, Yuwei Yang, Weinan Zhang, Yong Yu, and Lei Li. (MARS): Markov Molecular Sampling for Multi-objective Drug Discovery. In International Confer- ence on Learning Representations, 2021. URL htttp:://oppneeeeeeee
kHSu4ebxFXY
Xiufeng Yang. Tanuj Aasawat, and Kazuki Yoshizoe. Practical Massively Parallel Monte-Carlo Tree Search Applied to Molecular Design. In International Conference on Learning Representations, 2021. URL http:///pppeeviveeeeee
net. forum? d= ttt 7Vdo JAIK
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay Pande, and Jure Leskovec. Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation. In Advances in Neural Information Processing Systems 31, pp. 6412-6422, 2018.
12


Reviews:
[{"review_id": "pNTbidr9ir", "reviewer": "Reviewer_EYRk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, the authors observe that “plug-in performance estimators”, such as those commonly used in molecular generation and other reinforcement learning-esque settings, suffer from two kinds of bias. Misspecification bias results from policies which generate molecules far from those in the training set, while reuse bias can result from using the same data in training the plug-in estimator as well as the generator. The authors propose several approaches to correct for both types of bias. A set of empirical results demonstrate that both types of bias can arise, at least in a limited form, and that some of the proposed correction approaches can reduce the bias, though typically with a tradeoff in performance or computational cost.", "summary_of_the_review": "Overall, such generative, RL-esque approaches have received much attention recently. This work provides a reasonable framework for analytically decoupling and evaluating two types of bias in such approaches. Thus, I believe it would be of interest to many folks at ICLR.\n\nStill, as described above, the evaluation framework relies on a large number of choices. Thus, it is unclear if the specific conclusions from this analysis would hold in other molecular generation settings. Further, it is not clear if such distributions/oracle models would be available for other domains.", "strength_and_weaknesses": "The main strength of this work lies in the explicit decomposition of the biases to allow further analysis. The empirical evaluation effectively highlights the effect of the biases and some correction methods.\n\nTwo weaknesses of the current work concern its generalizability. As discussed in more detail below, it is not clear how often these biases may empirically come up. For example, even in experiments presumably designed to demonstrate this behavior, the experiments did not show that reusing samples in the plug-in estimator and generator led to problems. Second, due to the numerous modeling choices, it is not clear to what extent the specific observations relating to molecular generation are relevant for similar settings. At the least, showing that the observations hold using multiple settings within molecular generation would improve confidence in the approach and conclusions in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SVAs0AidRRG", "reviewer": "Reviewer_zS58", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper analyses reasons for why the real-world performance of molecular optimization algorithms does not match the predicted performance during evaluation, which they call _bias_. Specifically, they consider the setup where a fixed dataset of molecules with labels is used to train a policy and a predictive model that is used to estimate the performance of that policy. They propose to decompose the bias into two terms: one resulting from a mismatch between the true data-labelling function and the predictive model (called _misspecification_), and one from the same dataset being used to train the predictive model and the policy (called _reuse_). The authors discuss some strategies to mitigate both of these biases, although they seem to find that they are either ineffective or hurt performance.", "summary_of_the_review": "The main strength of this paper is the insightful decomposition of the bias of molecular optimization evaluation. However, I think the paper is made worse by its choice of a limiting MDP formalism, excessive math (which both lowers readability and may possibly be wrong, e.g. formalizing molecule space as a Banach space), and limited experiments. Altogether I think this makes the paper borderline, so I will tentatively recommend rejection at this time.", "strength_and_weaknesses": "I think the main strengths of this paper are the following:\n\n- Focuses on relevant problem: evaluation of molecular optimization methods using predictive models is a challenging problem with no clear solution\n- Decomposition of bias into two terms is insightful and potentially actionable, if the biases can be dealt with separately. I think the insight made by this paper is generally valuable\n- Experiments section is interesting because they quantify these biases, albeit for a toy problem\n\nHowever, I think the paper also has some concerning weaknesses:\n\n- Formalizing the problem as an MDP in section 1 seems unnatural to me. While many molecular optimization methods do construct molecules sequentially (e.g. string-based RL), many others do not (e.g. genetic algorithms, deep generative models). The MDP formulation does not effectively capture these methods. At the same time, it seems that the only part of the MDP formulation which is essential to the authors' analysis is the reward for terminal states ($f^*$). I suggest that the authors change the problem formulation to simple optimization (e.g. solving $x^* = \\max_x f^*(x)$). This is more general and also easier to understand. It would also allow the formalism to include SMILES-based methods, which I understand currently fall outside the formalism.\n- Notation is often confusing, particularly when the same letters are used to represent different types of objects. For example, $\\alpha_\\pi$ maps a dataset to a policy, while $\\alpha_f$ maps a dataset to a predictor. $J_{PI}$ maps policies and functions to performance, while $\\tilde{J}$ maps datasets to final performances (with a hidden dependence on alpha). $\\Delta_{PI}$ contains an implicit dependence on $f^*$. Overall I found this very confusing and constantly had to refer back to section 1-2 in order to understand what all the variables meant. I suggest the authors remove all implicit dependencies (e.g. things which depend on $\\alpha$ should have $\\alpha$ as a subscript) and use different letters for different types of objects.\n- I think the mathematical formalism was a bit excessive, and also possibly not correct. For example, proposition 2 depends on two very technical assumptions from the appendix, which the authors intuitively describe as \"$\\alpha$ produces policies which only depend on the data distribution, and has a Taylor expansion. These assumptions are not really justified, and for the sake of clarity could probably be stated more simply. I actually think these assumptions might be inappropriate: for example, definition 11 defines the Taylor expansion as occurring between two Banach spaces (essentially a vector space with a norm). I don't think molecule space is a vector space since there is no obvious notion of what the addition or scalar multiplication operation might be. If molecule space is not a vector space then most of the analysis of the paper does not apply to the case of molecules. This is just one example. While I appreciate the authors attempt to be mathematically precise, I think the underlying point of this paper is fairly simple and it would be better if they tried to introduce only as much mathematical formalism as is necessary.\n- Experiments are only done on a toy problem with a predictor oracle. Given that one of the main points of the paper is the bias that comes from using a ML predictor as an oracle function, I think it would be a better choice to use a non-ML oracle function as the ground-truth, for example the goal-directed benchmarks from GuacaMol (even though these are toy functions)\n- The solutions proposed for reducing bias seem to only be applicable to a small class of molecular optimization algorithms. For example, bootstrap, train-test split, and policy constraint only seem applicable to RL methods trained with imitation learning (I could be wrong about this though). These limitations are not clearly stated in the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "N_rfyqh3e92", "reviewer": "Reviewer_rwC3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets molecular optimization, where we want to modify/generate the structure of molecules to optimize some target properties. For this problem, this paper provided two contributions: 1) theoretical clarification on problematic biases for a common practice of using the same data for both training and evaluating the generator. 2) practical solutions to these biases, particularly the uses of (i) importance sampling estimators, (ii) doubly robust estimators, and (iii) bootstrapping/cross-validation. Empirical studies were also made using the setup of molecular optimization as reinforcement learning by Gottipati et al. (2000).", "summary_of_the_review": "This paper studied important bias problems in RL as a context in molecular optimization. However, it has unclear implications and contexts: 1) why this paper needs to target molecular optimization is unconvincing because both theoretical and practical parts seemed to be discussed in a much more general and abstract setup, and implications of the obtained theorems in the context of molecular optimization are unclear. 2) The use of \"cumulative\" reward sounds a bit unnatural for the goal of molecular optimization (Though this might just come from the other existing work...). 3) The contents of practice (bias reduction) parts are similar to well-known discussions in an OPE or contextual bandit context. In particularly, \"DM(direct method)\" corresponds to \"the plugin bias\" in this paper, \"IS(importance sampling method)\" corresponds to \"covariate shift\" in this paper, \"DR(doubly robust)\" corresponds to DR (3.1.3) in this paper. ", "strength_and_weaknesses": "[Strength]\n\n- This paper focuses on very interesting and important bias problems when we use ML for molecular optimization or conditioned molecular generation for optimizing a given target property. A common practice would be the \"direct method or plug-in method,\" that first learns a model of the system and then use it to estimate the performance of the evaluation policy, which causes misspecification bias and reusing bias.\n\n- Theoretical considerations are grounded on a general setup of Markov decision process with some mild conditions. \n\n- Not only providing theoretical results, but the paper also provides actionable practical solutions to reduce the analyzed biases. In particular, the uses of (i) importance sampling estimators, (ii) doubly robust estimators, and (iii) bootstrapping/cross-validation.\n\n\n[Weakness]\n\n- As in the paper's title, this paper targets molecular optimization, but the theoretical/technical considerations were done in a very general and abstract setup, and the obtained implications from theoretical parts were unclear. This paper's theoretical part might be better to be reconsidered in a much broader setup. For examples of three main results (Theorem1, Proposition 2, Proposition 3), Theorem 1 seemed a general decomposition, Proposition 2 is an accurate estimate of the bias WITHOUT comparing to other unavoidable uncertainty (at least, need to relate this to some uncertainty quantification of aleatoric and epistemic uncertainties?), Proposition 3 just provides positivity not referring whether this amount is negligible or not. So from the theoretical parts, we are still not sure if these biases are problematic or negligible.\n\n- The fact that theoretical analyses are mathematically solid negatively emphasizes the gap between practical situations and formal definitions. For example, the paper formulated molecular optimization as reinforcement learning to maximize the \"expected cumulative reward,\" but the goal of molecular optimization is usually just to have better-performing molecules than originals and doesn't need to maximize \"cumulative\" scores or doesn't have nice intermediate molecules during optimization. In this sense, the target task sounds like \"pure exploration\" rather than RL having an exploitation-exploration dilemma. \"maximum reward\" (for example, below [1][2]) would be more natural than \"cumulative or averaged reward,\" but this change seems to violate some basic assumptions on continuity of this paper. (For example, the entirety of Banach space?), and shake the values of the obtained theoretical results. \n\n[1] Quah & Quek, Maximum reward reinforcement learning: A non-cumulative reward criterion. (2006)\n    https://doi.org/10.1016/j.eswa.2005.09.054\n\n[2] Gottipati et al., Maximum Reward Formulation In Reinforcement Learning. (2020)\n    https://arxiv.org/abs/2010.03744 \n\n- The entire contents share strong similarity to the similar discussion in OPE (off-policy evaluation), and would be incremental given that  OPE is one of the RL-related recent hot topics in ML. The paper cited DudÍk et al. 2014 in a contextual bandit context, but this work came from ICML 2011 by the same authors https://arxiv.org/abs/1103.4601 and relevant research along this line is now being intensively investigated in ML. One of the highly cited results in this context would be [3] below, and this paper analyzed the plugin bias under study as \"DM (direct method),\" \"IS (importance sampling method)\" such as Inverse Probability Weighting (IPW), and \"DR (doubly robust),\" which is very similar to this paper. Of course, full RL (in this paper) and contextual bandits/OPE would differ in many points, but these relevant contexts should be incorporated to make contributions much clearer.\n\n[3] Farajtabar et al, More Robust Doubly Robust Off-policy Evaluation. (ICML 2018)\n    https://arxiv.org/abs/1802.03493\n\n- Similarly, the use of some out-of-sample estimators (like cross-validation or bootstrapping) sounds like standard practice for these situations (plug-in situations?), and I'm not that clear that this proposal is somewhat novel. At least, the use of bootstrapping in OPE (for example [4] below) also needs to be appropriately positioned in a related context. \n\n[4] Hao et al., Bootstrapping Fitted Q-Evaluation for Off-Policy Inference (ICML 2021)\n    https://arxiv.org/abs/2102.03607", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8Kciqol-TrQ", "reviewer": "Reviewer_JgZe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a theoretical take on the well-established intuition that RL-based molecular optimization approaches essentially \"attack\" locally-overfit predictor functions and result in bad, unrealistic molecular designs on account of these \"biases\" in the predictive model. They split this bias into a misspecification bias due to lack of overlap between the training data for the predictor and the actual distribution of (optimal) molecules generated by RL and a reusing bias due to the use of the same data for train and test the policy. Then, the propose solutions based on covariate shift, policy constrain, and doubly-robust performance estimators; and bias estimatio by train-test split and boostrapping respectively. \nThey provide an example of the quantification and reduction of these biases in a single problem", "summary_of_the_review": "I find this work shining much needed light in the crux of RL-based molecular optimization and with higher-than-usual rigor, trying to classify the sources of bias that drive RL optimizer towards bad performance in molecular design, as well as proposing potential fixes and testing them. I miss a few more experiments (with non-ML but hard oracles [perhaps quantum chemistry?]) to further quantify the issue ~ their use of an NN-based surrogate oracle creates its own set of challenges", "strength_and_weaknesses": "Strengths\n\nThis is the first work - to my knowledge - to take a serious, theoretical take on a problem that plagues RL-based molecular optimization strategies and makes them little more than useless. The distribution mismatch between the predictor training data and the huge expressive power of RL optimizers always results in the overfit predictors being attacked by the RL optimizer. I find the partitioning of the biases and the proposed solutions above the average theoretical rigor of the works in the area. \n\nThe paper provides compelling arguments to support the intuition of why RL models struggle to invent realistic, useful molecules and how to mitigate it. The fact that mispecificaton bias seems to dominate, especially with larger datasets is also intuitive. \n\nWeaknesses \nI missed more numerical examples and real world evidence of the implications of this analysis. The proposed example is fine, but it is very minor. Just one RL, on one chemical space, for one task seems scarce. Specially because the oracle is itself a fitted function, which is likley oversmooth outside the training data (and likely somewhat overfit in its training data). The implications of this are not clear, whether what the authors report is a particularly good or a particularly bad case. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject