Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this work, the authors observe that “plug-in performance estimators”, such as those commonly used in molecular generation and other reinforcement learning-esque settings, suffer from two kinds of bias. Misspecification bias results from policies which generate molecules far from those in the training set, while reuse bias can result from using the same data in training the plug-in estimator as well as the generator. The authors propose several approaches to correct for both types of bias. A set of empirical results demonstrate that both types of bias can arise, at least in a limited form, and that some of the proposed correction approaches can reduce the bias, though typically with a tradeoff in performance or computational cost.
strength_and_weaknesses: The main strength of this work lies in the explicit decomposition of the biases to allow further analysis. The empirical evaluation effectively highlights the effect of the biases and some correction methods.

Two weaknesses of the current work concern its generalizability. As discussed in more detail below, it is not clear how often these biases may empirically come up. For example, even in experiments presumably designed to demonstrate this behavior, the experiments did not show that reusing samples in the plug-in estimator and generator led to problems. Second, due to the numerous modeling choices, it is not clear to what extent the specific observations relating to molecular generation are relevant for similar settings. At the least, showing that the observations hold using multiple settings within molecular generation would improve confidence in the approach and conclusions in the paper.
clarity,_quality,_novelty_and_reproducibility: Novelty

The main novelty of the approach lies in the explicit decomposition of the bias of the generator into the two terms. This then allows the authors to investigate each term independently, in contrast to prior similar approaches in which the biases were intertwined.

Quality

I did not verify the derivations in detail, but they mostly follow from known results, so they seemed correct. The proposed empirical evaluation framework was technically reasonable for the setting considered in this work.

The proposed approaches to address the biases seemed reasonable.

One general limitation of the current analysis is that the “reuse” bias only arises in a limited manner. In particular, in the empirical analysis, the generator did not exhibit any bias due to sample reuse, but only due to limited training data. Thus, even in experiments designed by the authors, we are unable to evaluate whether sample reuse leads to empirical problems.

Another limitation of the current analysis is that it is not clear how much the specific observations about molecular generation generalize. While the implemented set of experiments show that the biases exist with these exact choices of distributions and models (though the “reuse” bias only shows up in a limited form as mentioned above), it is not clear whether these findings generalize. Concretely, as highlighted within the experimental design description, even within molecular generation, the hyperparameter space of predictor model class, generator model class, surrogate model approximations, etc., is huge. Thus, evaluating other molecular generation techniques within this framework would improve confidence in the generalizability of both the concepts and the specific conclusions for molecular generation.

The generalizability of the work could also be improved by discussing whether similar analyses would hold for non-MDP approaches, such as variational autoencoders. (Analysis of VAEs is hinted at in the conclusions of the paper, but it is not clear why the proposed approach would depend on a sequential generative process.)

Clarity

At some points, the notation becomes rather dense and difficult to follow. The combined performance function in Section 3.1.4 particularly jumps out. The clarity of the work could be improved by possibly condensing some of the notation, or at least adding a symbols table. Still, I appreciate that the theoretical analysis necessitates a fair amount of notation.

Otherwise, the paper is generally well structured and easy enough to follow.

The authors also provide good references to put the work in context, including connections to other related research communities like model selection and the various information criteria.

The clarity and impact of the work could be improved by highlighting which distributions (or surrogate/oracle models are needed to evaluate each type of bias. The paper could also be improved by clarifying whether the types of bias could be investigated independently when the necessary distributions/oracle models are not available for one of the biases.

Reproducibility

The work seems generally reproducible. While I did not run it, the supplementary material includes code with limited documentation. It also follows standard best practices like distributing python code as installable packages.
summary_of_the_review: Overall, such generative, RL-esque approaches have received much attention recently. This work provides a reasonable framework for analytically decoupling and evaluating two types of bias in such approaches. Thus, I believe it would be of interest to many folks at ICLR.

Still, as described above, the evaluation framework relies on a large number of choices. Thus, it is unclear if the specific conclusions from this analysis would hold in other molecular generation settings. Further, it is not clear if such distributions/oracle models would be available for other domains.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a theoretical take on the well-established intuition that RL-based molecular optimization approaches essentially "attack" locally-overfit predictor functions and result in bad, unrealistic molecular designs on account of these "biases" in the predictive model. They split this bias into a misspecification bias due to lack of overlap between the training data for the predictor and the actual distribution of (optimal) molecules generated by RL and a reusing bias due to the use of the same data for train and test the policy. Then, the propose solutions based on covariate shift, policy constrain, and doubly-robust performance estimators; and bias estimatio by train-test split and boostrapping respectively. 
They provide an example of the quantification and reduction of these biases in a single problem
strength_and_weaknesses: Strengths

This is the first work - to my knowledge - to take a serious, theoretical take on a problem that plagues RL-based molecular optimization strategies and makes them little more than useless. The distribution mismatch between the predictor training data and the huge expressive power of RL optimizers always results in the overfit predictors being attacked by the RL optimizer. I find the partitioning of the biases and the proposed solutions above the average theoretical rigor of the works in the area. 

The paper provides compelling arguments to support the intuition of why RL models struggle to invent realistic, useful molecules and how to mitigate it. The fact that mispecificaton bias seems to dominate, especially with larger datasets is also intuitive. 

Weaknesses 
I missed more numerical examples and real world evidence of the implications of this analysis. The proposed example is fine, but it is very minor. Just one RL, on one chemical space, for one task seems scarce. Specially because the oracle is itself a fitted function, which is likley oversmooth outside the training data (and likely somewhat overfit in its training data). The implications of this are not clear, whether what the authors report is a particularly good or a particularly bad case. 
clarity,_quality,_novelty_and_reproducibility: The analysis is novel and the description is relatively clear. The formalism could perhaps be compressed to make way for a few more experiments. 
summary_of_the_review: I find this work shining much needed light in the crux of RL-based molecular optimization and with higher-than-usual rigor, trying to classify the sources of bias that drive RL optimizer towards bad performance in molecular design, as well as proposing potential fixes and testing them. I miss a few more experiments (with non-ML but hard oracles [perhaps quantum chemistry?]) to further quantify the issue ~ their use of an NN-based surrogate oracle creates its own set of challenges
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper analyses reasons for why the real-world performance of molecular optimization algorithms does not match the predicted performance during evaluation, which they call _bias_. Specifically, they consider the setup where a fixed dataset of molecules with labels is used to train a policy and a predictive model that is used to estimate the performance of that policy. They propose to decompose the bias into two terms: one resulting from a mismatch between the true data-labelling function and the predictive model (called _misspecification_), and one from the same dataset being used to train the predictive model and the policy (called _reuse_). The authors discuss some strategies to mitigate both of these biases, although they seem to find that they are either ineffective or hurt performance.
strength_and_weaknesses: I think the main strengths of this paper are the following:

- Focuses on relevant problem: evaluation of molecular optimization methods using predictive models is a challenging problem with no clear solution
- Decomposition of bias into two terms is insightful and potentially actionable, if the biases can be dealt with separately. I think the insight made by this paper is generally valuable
- Experiments section is interesting because they quantify these biases, albeit for a toy problem

However, I think the paper also has some concerning weaknesses:

- Formalizing the problem as an MDP in section 1 seems unnatural to me. While many molecular optimization methods do construct molecules sequentially (e.g. string-based RL), many others do not (e.g. genetic algorithms, deep generative models). The MDP formulation does not effectively capture these methods. At the same time, it seems that the only part of the MDP formulation which is essential to the authors' analysis is the reward for terminal states ($f^*$). I suggest that the authors change the problem formulation to simple optimization (e.g. solving $x^* = \max_x f^*(x)$). This is more general and also easier to understand. It would also allow the formalism to include SMILES-based methods, which I understand currently fall outside the formalism.
- Notation is often confusing, particularly when the same letters are used to represent different types of objects. For example, $\alpha_\pi$ maps a dataset to a policy, while $\alpha_f$ maps a dataset to a predictor. $J_{PI}$ maps policies and functions to performance, while $\tilde{J}$ maps datasets to final performances (with a hidden dependence on alpha). $\Delta_{PI}$ contains an implicit dependence on $f^*$. Overall I found this very confusing and constantly had to refer back to section 1-2 in order to understand what all the variables meant. I suggest the authors remove all implicit dependencies (e.g. things which depend on $\alpha$ should have $\alpha$ as a subscript) and use different letters for different types of objects.
- I think the mathematical formalism was a bit excessive, and also possibly not correct. For example, proposition 2 depends on two very technical assumptions from the appendix, which the authors intuitively describe as "$\alpha$ produces policies which only depend on the data distribution, and has a Taylor expansion. These assumptions are not really justified, and for the sake of clarity could probably be stated more simply. I actually think these assumptions might be inappropriate: for example, definition 11 defines the Taylor expansion as occurring between two Banach spaces (essentially a vector space with a norm). I don't think molecule space is a vector space since there is no obvious notion of what the addition or scalar multiplication operation might be. If molecule space is not a vector space then most of the analysis of the paper does not apply to the case of molecules. This is just one example. While I appreciate the authors attempt to be mathematically precise, I think the underlying point of this paper is fairly simple and it would be better if they tried to introduce only as much mathematical formalism as is necessary.
- Experiments are only done on a toy problem with a predictor oracle. Given that one of the main points of the paper is the bias that comes from using a ML predictor as an oracle function, I think it would be a better choice to use a non-ML oracle function as the ground-truth, for example the goal-directed benchmarks from GuacaMol (even though these are toy functions)
- The solutions proposed for reducing bias seem to only be applicable to a small class of molecular optimization algorithms. For example, bootstrap, train-test split, and policy constraint only seem applicable to RL methods trained with imitation learning (I could be wrong about this though). These limitations are not clearly stated in the paper.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is ok: the main takeaways are stated pretty clearly, but I think this paper is too heavy on notation: there are many variables with different subscripts and superscripts, only a fraction of which are used multiple times throughout the paper. The implications of many of the theorems are not discussed clearly.

Novelty/originality: while I am not specifically aware of other works which break down the bias of optimization algorithms in this way, I have seen many works which aim to decompose the bias/variance of different estimators in an insightful way, particularly from the statistics literature. It would be nice to see this discussed more. In the context of this work, and the empirical work from Renz and Langevin (both cited in the paper), I would say the novelty is moderate.

Quality: I think the quality is ok. The decomposition of the bias (the key part of the paper) is a good contribution, but I think the solutions proposed are not as promising and don't seem to work well in practice (as shown in section 4). I think formulating it as a MDP also lowers the overall quality of the work.

Reproducibility: code was included, I think this is very reproducible.
summary_of_the_review: The main strength of this paper is the insightful decomposition of the bias of molecular optimization evaluation. However, I think the paper is made worse by its choice of a limiting MDP formalism, excessive math (which both lowers readability and may possibly be wrong, e.g. formalizing molecule space as a Banach space), and limited experiments. Altogether I think this makes the paper borderline, so I will tentatively recommend rejection at this time.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper targets molecular optimization, where we want to modify/generate the structure of molecules to optimize some target properties. For this problem, this paper provided two contributions: 1) theoretical clarification on problematic biases for a common practice of using the same data for both training and evaluating the generator. 2) practical solutions to these biases, particularly the uses of (i) importance sampling estimators, (ii) doubly robust estimators, and (iii) bootstrapping/cross-validation. Empirical studies were also made using the setup of molecular optimization as reinforcement learning by Gottipati et al. (2000).
strength_and_weaknesses: [Strength]

- This paper focuses on very interesting and important bias problems when we use ML for molecular optimization or conditioned molecular generation for optimizing a given target property. A common practice would be the "direct method or plug-in method," that first learns a model of the system and then use it to estimate the performance of the evaluation policy, which causes misspecification bias and reusing bias.

- Theoretical considerations are grounded on a general setup of Markov decision process with some mild conditions. 

- Not only providing theoretical results, but the paper also provides actionable practical solutions to reduce the analyzed biases. In particular, the uses of (i) importance sampling estimators, (ii) doubly robust estimators, and (iii) bootstrapping/cross-validation.


[Weakness]

- As in the paper's title, this paper targets molecular optimization, but the theoretical/technical considerations were done in a very general and abstract setup, and the obtained implications from theoretical parts were unclear. This paper's theoretical part might be better to be reconsidered in a much broader setup. For examples of three main results (Theorem1, Proposition 2, Proposition 3), Theorem 1 seemed a general decomposition, Proposition 2 is an accurate estimate of the bias WITHOUT comparing to other unavoidable uncertainty (at least, need to relate this to some uncertainty quantification of aleatoric and epistemic uncertainties?), Proposition 3 just provides positivity not referring whether this amount is negligible or not. So from the theoretical parts, we are still not sure if these biases are problematic or negligible.

- The fact that theoretical analyses are mathematically solid negatively emphasizes the gap between practical situations and formal definitions. For example, the paper formulated molecular optimization as reinforcement learning to maximize the "expected cumulative reward," but the goal of molecular optimization is usually just to have better-performing molecules than originals and doesn't need to maximize "cumulative" scores or doesn't have nice intermediate molecules during optimization. In this sense, the target task sounds like "pure exploration" rather than RL having an exploitation-exploration dilemma. "maximum reward" (for example, below [1][2]) would be more natural than "cumulative or averaged reward," but this change seems to violate some basic assumptions on continuity of this paper. (For example, the entirety of Banach space?), and shake the values of the obtained theoretical results. 

[1] Quah & Quek, Maximum reward reinforcement learning: A non-cumulative reward criterion. (2006)
    https://doi.org/10.1016/j.eswa.2005.09.054

[2] Gottipati et al., Maximum Reward Formulation In Reinforcement Learning. (2020)
    https://arxiv.org/abs/2010.03744 

- The entire contents share strong similarity to the similar discussion in OPE (off-policy evaluation), and would be incremental given that  OPE is one of the RL-related recent hot topics in ML. The paper cited DudÍk et al. 2014 in a contextual bandit context, but this work came from ICML 2011 by the same authors https://arxiv.org/abs/1103.4601 and relevant research along this line is now being intensively investigated in ML. One of the highly cited results in this context would be [3] below, and this paper analyzed the plugin bias under study as "DM (direct method)," "IS (importance sampling method)" such as Inverse Probability Weighting (IPW), and "DR (doubly robust)," which is very similar to this paper. Of course, full RL (in this paper) and contextual bandits/OPE would differ in many points, but these relevant contexts should be incorporated to make contributions much clearer.

[3] Farajtabar et al, More Robust Doubly Robust Off-policy Evaluation. (ICML 2018)
    https://arxiv.org/abs/1802.03493

- Similarly, the use of some out-of-sample estimators (like cross-validation or bootstrapping) sounds like standard practice for these situations (plug-in situations?), and I'm not that clear that this proposal is somewhat novel. At least, the use of bootstrapping in OPE (for example [4] below) also needs to be appropriately positioned in a related context. 

[4] Hao et al., Bootstrapping Fitted Q-Evaluation for Off-Policy Inference (ICML 2021)
    https://arxiv.org/abs/2102.03607
clarity,_quality,_novelty_and_reproducibility: The paper is well written with sufficient supplementary information, and the issues and problems analyzed in the paper are definitely important and interesting, and presumably reproducible ones. In contrast, novelty wise, I am still unsure, as I described several concerns in the above [Weakness] section. 
summary_of_the_review: This paper studied important bias problems in RL as a context in molecular optimization. However, it has unclear implications and contexts: 1) why this paper needs to target molecular optimization is unconvincing because both theoretical and practical parts seemed to be discussed in a much more general and abstract setup, and implications of the obtained theorems in the context of molecular optimization are unclear. 2) The use of "cumulative" reward sounds a bit unnatural for the goal of molecular optimization (Though this might just come from the other existing work...). 3) The contents of practice (bias reduction) parts are similar to well-known discussions in an OPE or contextual bandit context. In particularly, "DM(direct method)" corresponds to "the plugin bias" in this paper, "IS(importance sampling method)" corresponds to "covariate shift" in this paper, "DR(doubly robust)" corresponds to DR (3.1.3) in this paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
Summary:

This paper analyses reasons for why the real-world performance of molecular optimization algorithms does not match the predicted performance during evaluation, which they call bias. Specifically, they consider the setup where a fixed dataset of molecules with labels is used to train a policy and a predictive model that is used to estimate the performance of that policy. They propose to decompose the bias into two terms: one resulting from a mismatch between the true data-labelling function and the predictive model (called misspecification), and one from the same dataset being used to train the predictive model and the policy (called reuse). The authors discuss some strategies to mitigate both of these biases, although they seem to find that they are either ineffective or hurt performance.

Strengths:

- Focuses on relevant problem: evaluation of molecular optimization methods using predictive models is a challenging problem with no clear solution
- Decomposition of bias into two terms is insightful and potentially actionable, if the biases can be dealt with separately. I think the insight made by this paper is generally valuable
- Experiments section is interesting because they quantify these biases, albeit for a toy problem
- explicit decomposition of the biases to allow further analysis. The empirical evaluation effectively highlights the effect of the biases and some correction methods.

Weaknesses:

- Notation is often confusing
- The mathematical formalism was a bit excessive
- Experiments are only done on a toy problem 
- The solutions proposed for reducing bias seem to only be applicable to a small class of molecular optimization algorithms
- it is not clear how often these biases may empirically come up. 
- due to the numerous modeling choices, it is not clear to what extent the specific observations relating to molecular generation are relevant for similar settings

Recommendation:

This is a borderline paper. Two reviewers vote for rejection (one strongly and one midly). One reviewer votes strongly for acceptance and one tends slightly towards rejection. In the discussion, the most positive reviewer did not participate and the other reivewers agreed that the practical impact is really not there and that the theoretical results are not particularly compelling. Because of this, the reviwers agreed on rejection. I, therefore, decide to reject the paper and encourage the authors to improve the paper and resubmit to another conference.

justification_for_why_not_higher_score: The reviewers agreed on rejection.
Decision: Reject