Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: &nbsp;

The authors provide an extensive benchmark for molecular property optimisation together with a near-exhaustive empirical evaluation of molecule generation architectures. I believe this project is timely, well-needed and that the NeurIPS Datasets and Benchmarks track is an excellent venue for this work. 

I do however have concerns about some of the claims made in the submission, particularly with respect to the performance of the BO and VAE-BO methods in light of recent work [1, 2]. As such, my evaluation at this stage is borderline as it is unclear whether it will be possible to address all of the issues in the time allocated for author response. 

If however the authors can address the issues adequately, I will be willing to substantially raise my score and argue for the acceptance of this paper. I believe this will be a very important and impactful benchmark and it would be a great shame if the paper is not published in either this round or the next round of NeurIPS Datasets and Benchmarks. 

In terms of facilitating communication, it would be great if the authors could reply (as much as is possible) with partial responses to the points raised as opposed to providing the full author response at the end of the discussion period. The latter approach leaves little time to review the response and update the score.

&nbsp;
strengths: &nbsp;

1. The empirical evaluation is near-exhaustive.

2. The benchmark is informative. Of particular interest is the finding that older models such as REINVENT are performant when benchmarked on new datasets. I believe this is an important message for the molecule generation community.

3. The benchmark focusses on key indicators of performance in real-world molecular discovery campaigns such as sample efficiency.

&nbsp;
weaknesses: &nbsp;

## **MAJOR POINTS** ##

&nbsp;

1. The early stopping criterion of 5 iterations mentioned in appendix B3 seems to be far too harsh, especially for methods such as Bayesian optimisation which can often plateau during the optimisation trace due to exploratory actions. Most importantly, the results from [1], who also use the Therapeutics Data Commons benchmark show that various VAE BO methods [1,2,8,9] achieve the strongest performance and also obtain different orderings in terms of the baseline models. It would be great to see the performance comparison in terms of the raw scores (as opposed to the AUC) between [1] and the current submission.

&nbsp;

2. In terms of the statement,

&nbsp;

> Most papers do not report how many times the oracle function is called to achieve the reported results (i.e. how many candidate molecules were evaluated), except in rare cases

&nbsp;

It would be great if the following papers were also cited as the rare cases! [1, 2]

&nbsp;

3. In terms of the claim,

&nbsp;

> Our results show that none of the existing molecular optimization algorithms are efficient enough to solve a de novo molecular optimization problem within a realistic oracle budget of hundreds of experiments, and “state-of-the-art” methods often fail to outperform their predecessors

&nbsp;

I am not convinced that the authors yet have sufficient empirical evidence to make this claim given the conditions under which the models were run e.g. the early stopping of 5 iterations without performance improvement for Bayesian optimisation methods. I would recommend moderating the claims as they could potentially discourage research into otherwise promising methods. I don't believe moderating the claims takes away from the contributions of the paper, the majority of benchmarks are submitted without extensive empirical evaluation and the authors have gone above and beyond the remit in this regard.

&nbsp;

4. In terms of the statement,

&nbsp;

> but this is not an issue for more recent methods. We believe this is because current language models are better able to learn the grammar of SMILES strings, which has flattened the advantage of SELFIES.

&nbsp;

One of the reasons why the original SMILES VAE struggled to generate valid outputs was shown empirically in [3], namely a disconnect between the VAE and the Bayesian optimisation scheme. More specifically, the exploratory component of the acquisition function encourages points to be queried which lie far from the training data manifold in latent space. The constrained Bayesian optimisation approach introduced in [3] alleviated some of the problems with SMILES validity by learning a feasible (high validity) region of VAE latent space. As such, the validity concerns with the SMILES grammar are unlikely to be solely due to the quality of the language model, at least in the case of the VAE models, and I believe this is worth mentioning!

&nbsp;


## **MINOR POINTS** ##

&nbsp;

1. It would be great if the references appeared in numbered order e.g. via the LaTeX command:

&nbsp;

> \bibliographystyle{unsrtnat}

> \setcitestyle{numbers,open={[},close={]},citesep={,}}

&nbsp;

2. For the GP BO method, in addition to Morgan fingerprints, "fragprints" [4] i.e. concatentations of RDKit fragment vectors (count vectors over functional groups in the molecule) with the Morgan fingerprint bit vectors have been found to outperform Morgan fingerprints in the prediction of transition wavelengths. Given that the choice of featurisation is open-ended, frapgrints way be worth mentioning as an alternative choice for some tasks. The combination of the "local" fingerprint features and the "global" fragment features appears to perform well over a range of MoleculeNet tasks. 

&nbsp;

3. The batch size for BO mentioned in the appendix (1,180) is very large relative to the optimisation horizon (10,000). Presumably the batch size is too high due to the scalability issues? Is this for an exact GP?

&nbsp;

4. In terms of the statement on line 41,

&nbsp;

> Further, some papers only report results on trivial oracles [33] like quantitative estimate of drug-likeness (QED) or penalized octanol-water partition coefficient (LogP)

&nbsp;

It might be an idea to expand on the meaning of trivial here. Maximising the penalized octanol-water coefficient is not meaningful from the point of view of drug discovery yet it may act as a toy task for differentiating machine learning architectures based on their performance. In contrast, while QED has some relation to drug-likeness, the metric is generally too easy to be used for model comparison.

&nbsp;

5. In Table 1, [1] could be added in the entry corresponding to VAE + BO + SELFIES. This work uses a transformer architecture which appears to be more powerful than the RNN/GRU implementations of the SMILES VAE from Gomez Bombarelli et al. and Polykoskiy et al.

&nbsp;

6. In section 2.1 it is probably worth explicitly referencing the auxiliary properties of black-box optimisation problems [5] e.g. while the analytic form of the function is unknown, the function may be queried pointwise, the gradients of the function are inaccessible and there is a high cost (financially in addition to time) incurred by querying the black-box. In addition, it might be worth describing the nature of oracles in laboratory synthesis, where querying a (high fidelity) experimental property is much more time-intensive relative to computing a (lower fidelity) property value *in silico*.

&nbsp;

7. I would recommend using a different Greek letter for the design space since it bears a resemblance to the symbol for the oracle.

&nbsp;

8. In terms of the statement

&nbsp;

> Two-dimensional (2D) graphs can intuitively define molecular identities to a first approximation (ignoring stereochemistry)

&nbsp;

Although this is a widely-cited limitation of 2D molecular graph representations, it may be worth omitting that 2D graphs ignore stereochemistry since in theory, it should be possible to incorporate stereochemical information through node/edge attributes and graph transformations [6].

&nbsp;

9. In terms of the statement

&nbsp;

> Screening (a.k.a. virtual screening) involves searching over a pre-enumerated library of molecules exhaustively. We include Screening as a baseline, which randomly samples ZINC 250k.

&nbsp;

It might be worth removing "exhaustively" since this seems to be at odds with "random sampling" in the next sentence.

&nbsp;

10. In terms of the use of MolPAL, it may be worth mentioning that many other choices of surrogates could be used.

&nbsp;

11. In relation to the statement,

&nbsp;

> We included a string-based model, BO over String Space (BOSS) [32], and a synthesis-based model, ChemBO [27], but do not obtain meaningful results even with early stopping 

&nbsp;

It is probably worth citing the poor scaling of the SSK kernel in particular here as the reason and pointing to sparse implementations as a potential solution.

&nbsp;

12. Reference 26, Auto-encoding variational Bayes was published at ICLR 2014.

&nbsp;

13. It would be great if BoTorch [7] was cited given that is used for the experiments.

&nbsp;

14. In section B.4 of the appendix,

&nbsp;

> Worth to mention that due to the nonparametric essence, Bayesian optimization methods scales poorly with the data size, the run process is notoriously and intolerably slow even dealing 1k training data

&nbsp;

This is likely because the authors are using exact GP implementations in the codebase as far as I can see for both GP BO and the SMILES VAE. [1-3, 8] as well as the original automatic chemical design paper by Gomez-Bombarelli et al. all use sparse GPs for scalability. On the other hand, BOSS, at the current point in time, is constrained to be an exact GP as there is no sparse implementation of the SSK string kernel. The exact version of this algorithm indeed scales very poorly.

&nbsp;

15. It would be great if Adam was cited given that it is used [10].

&nbsp;

16. Line 594, typo, "determinantal point process".

&nbsp;

17. Some typos in the references e.g. missing capitalisation for "Bayesian" and missing capitalisations after colons in the titles of papers.

&nbsp;

## **REFERENCES** ##

&nbsp;

[1] Maus et al. Local Latent Space Bayesian Optimization over Structured Inputs. arXiv, 2022.

[2] Grosnit et al. High-Dimensional Bayesian optimisation with Variational Autoencoders and Deep Metric Learning. arXiv, 2021.

[3] Griffiths and Hernández-Lobato, Constrained Bayesian Optimization for Automatic Chemical Design using Variational Autoencoders. Chemical Science, 2020.

[4] Thawani et al. The Photoswitch Dataset: A Molecular Machine Learning Benchmark for the Advancement of Synthetic Chemistry. arXiv 2020.

[5] Garnett, Bayesian Optimization, Cambridge University Press, 2022.

[6] Andersen et al. Chemical Graph Transformation with Stereo-Information. In International Conference on Graph Transformation, 2017.

[7] Balandat et al. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. NeurIPS, 2022.

[8] Tripp et al. 2020. Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining. NeurIPS, 2020.

[9] Eriksson et al. Scalable Global Optimization via Local Bayesian optimization. NeurIPS, 2019.

[10] Kingma and Ba, Adam: A Method for Stochastic Optimisation, ICLR, 2015.

&nbsp;
correctness: &nbsp;
cf. main response
&nbsp;
clarity: &nbsp;
The paper is excellently written and presented.
&nbsp;
relation to prior work: &nbsp;

cf. main response

&nbsp;
additional feedback: &nbsp;

Adequately addressed.

&nbsp;
review_title: Excellent Benchmark, Concerns with Claims

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work proposes a new evaluation metric balancing with sample efficiency, inspired by real application scenarios. With new evaluation metric, this work re-benchmarks all proposed methods and discovers that the traditional methods are still very competitive and SMILES-based representation of molecules do not have convincing deficiencies. 

This paper has clear motivation and proposes a significant overlooked issue in molecular optimization. Then the proposed corresponding evaluation metric is reasonable and intuitive. The survey and taxonomy construction for molecular optimization methods are comprehensive and systematic. Re-benchmarking all proposed methods is a great effort for this community. 

strengths: (1)	Reasonable evaluation metric taking sample efficiency into consideration with clear motivation; 

(2)	Sample efficiency is very important for molecular lead optimization in real applications, this reviewer values the significance of this contribution very much;

(3)	The organization of current methods is systematic and logical;

(4)	Re-benchmarking leads to some important empirical findings and new insights; 

(5)	Significant contributions for further developments of molecular optimization and lead to real impacts on pharmaceutical industries in the future. 

weaknesses: This paper does not propose new dataset. As admitted by the authors, they did not exhaustively explore every method and thoroughly tune every hyperparameter.  Some other important quantities are not measured.  
correctness: Yes. The claims made in the submission is correct. The evaluation method proposed is very reasonable. This reviewer agrees that sample efficiency is truly an important and overlooked evaluation metric. 
clarity: Yes, this paper is well written. Clear motivations for new benchmark proposal, reasonable evaluation metrics design. 
relation to prior work: Yes, this work clearly points out the drawbacks of previous works and proposes corresponding solutions. 
additional feedback: No
review_title: PMO review

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This article presents a benchmark for molecule optimization to facilitate the transparent and reproducible evaluation of algorithmic advances in molecular optimization, consisting of 25 molecular optimization algorithms and 23 tasks with a particular focus on sample efficiency.  Compared with previous work, this benchmark takes into account the limits in practical scenarios of molecular design. The final experimental results also yielded some interesting conclusions.
strengths: 1. This paper integrates existing molecular optimization algorithms using a unified framework and then aligns them for comparison. The overall workload is large, and the work has practical significance.
2. This paper considers the oracle budget problem in more practical scenarios of molecular design, which has been ignored in previous work.
3. The writing of the article is good, and the code is clear.

weaknesses: 1. This article only considers the evaluation on single objective in each time. However, practical drug development is usually a multi-objective optimization, such as considering both efficacy and synthesis route. So, the “practical” in the paper is a bit over claimed.
2. Why is the number of oracle calls important? Since oracle calls can also be calculated, why limit it to 10000?

correctness: The evaluation method of the paper is correct, but can be improved to be more in line with the actual development of drugs. For example, more objectives are considered at the same time in a single optimization.
clarity: The paper is written well.
relation to prior work: This article clearly illustrates the relationship with the previous work. That is, the efficiency problem in actual drug development is considered and oracle calls is limited. The molecular optimization algorithms are compared under the same criteria.
additional feedback: In practical sense, multiple objectives should be considered in molecule optimization.
review_title: A benchmark with significance but a little too simple

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: **My summary of the paper**: This paper proposes a benchmark for molecular optimization (i.e. optimizing a function $f$ over the space of molecules). Noting that real world functions $f$ are either real-world experiments or expensive simulations, the proposed PMO benchmark tests _sample-efficiency_ of molecular optimization algorithms (i.e. can they produce optimized molecules with few calls to $f$). Specifically, the benchmark consists of a new set of metrics for a set of previously published objective functions (from Guacamol and TDC) to optimized with at most 10k function calls. The authors evaluate a relatively "out of the box" version of 25 different algorithms (with a bit of tuning) and show that 1) performance is much lower in the limited sample regime and 2) recent "state-of-the-art" algorithms don't actually seem to do any better than established older algorithms in this regime.

The authors don't make an explicit list of their contributions in the paper, but I would state their contributions as:
1. Selection of 23 objective functions + metrics to define the PMO benchmark
2. Evaluation and comparison of 25 previously-published algorithms on the PMO benchmark.
3. Several consequential conclusions from these experiments, chiefly that GraphGA and REINVENT seem to generally outperform most other (more complex and more recently published) methods

**Summary of my review**: In general I liked the work and think it is a step in the right direction: sample-efficiency is very important and very overlooked.
Reading the introduction to the paper, my understanding is that the authors propose PMO partly to stop the flow of papers which test their algorithms on unrealistic tasks and compare to very weak baselines, concluding from this that their method is "state of the art" despite being nearly worthless in practice.
My overarching perspective when judging this work is "does PMO achieve this goal".
Taking this and the scope of NeurIPS D&B into account, and given that this work 
does not propose new algorithms, objective functions, or datasets,
I think that the relevant questions for evaluating this work are:
1. Are the proposed metrics and setup of PMO good choices?
2. Is the specific choice of objective functions in PMO good?
3. Are the experiments with baseline algorithms meaningful and correct? Will the reported results hold new algorithms using the PMO benchmark to a high standard?

My opinion on these criteria are 1) metrics/setup make sense, although the choice of 10k is somewhat arbitrary 2) objectives are ok but inherit biases from Guacamol 3) some baselines are good, but I worry that many of them are implemented sub-optimally which understates their potential performance.
Ultimately I think that PMO is a promising first step towards a better molecular optimization benchmark but only incrementally improves upon existing benchmarks.
My opinion therefore is to recommend against acceptance at this time, although I believe that a revised version of PMO could potentially be accepted to a venue like D&B in the future.

strengths: - *Correct insight about sample efficiency*: many alleged "SOTA" algorithms require so many oracle calls that they would never be used in practice. Pointing this out and trying to correct it is worthwhile.
- *Addresses need for benchmarks in molecule optimization*: I agree with the authors that there are little/no standardized benchmarks which makes comparison difficult, and think that PMO contributes to this (although the difference between it and TDC/Guacamol is not very large).
- *Good metrics*: I like the AUC metrics proposed by the authors, even though I think they have some issues (discussed below). In particular, including full plots of the optimization trajectory is very useful.
- *Extensive experiments*: the authors tested a large number of baseline methods which is good (even though I believe that some methods were not implemented optimally; see below). Quality aside, including a large number of baselines is better than including a smaller number.
- *Good writing*: paper is very clear and well-written
weaknesses: - Metrics don't include diversity which I think may not make "topk" very meaningful (see correctness)
- Potential problem with setup: budget of 10k seems somewhat arbitrary (see correctness)
- Does not address effect of datasets on performance (see correctness)
- The selection of objective functions is not very diverse, and inherits some strong underlying biases of Guacamol. I worry that PMO functions are not very representative of functions in drug design more generally. (see correctness)
- Some baselines seem to be sub-optimally implemented which I believe results in their performance being understated (see "correctness"). This may sound like a minor issue, but I believe this renders the performance numbers in Tables 2, 3, etc not very meaningful, and therefore will be artificially easy to beat if subsequent papers use PMO to evaluate their algorithms (i.e. the main purpose of baseline algorithms).
correctness: Here I will comment on the weaknesses mentioned above, which in general I feel fall under the domain of "correctness".
In general, these comments come from my experience both with many of the baseline algorithms tested here,
and with the objective functions studied; I have spent over 2 years working on molecular optimization and am very familiar with the methods in this paper.

**Metrics**

AUC is a sensible metric to me because it rewards both good final performance and good initial performance.
My concern is about the choice of top-10:
Guacamol + fingerprint-based objective functions in general tend not to be very sensitive to small changes in molecular structure;
in my experience if one high-scoring molecule is known then it is easy to produce many high-scoring variations of that molecule (although some algorithms do this better than others).
This raises 2 concerns:
1. I don't think that measuring an algorithm's ability to produce variations of the same molecule is as important as finding a good initial molecule; variations of this molecule can be produced later. This matches the existing pipeline of "hit identification" and "lead optimization" in real-world drug discovery.
2. Practically, I worry that measuring the "top 10" performance will not be much different than measuring "top 1" performance.

I think that it would be better to mainly report top1 performance: this mitigates any potential concerns about diversity (which you deem "future work" anyway), and would make PMO more closely aligned with the hit identification task in drug discovery.
However, you could instead add some sort of diversity penalty, either as a hard constraint (often hard to incorporate) or a soft penalty (forcing you to pick an arbitrary trade-off between objective value and diversity).
I think that both of these options are better than using the top 10 metric and disregarding diversity.

**Benchmark setup: 10k budget**

I would like to hear the authors' reasons for choosing 10k as the oracle budget. This choice seemed somewhat arbitrary to me, and was not justified in the paper. I personally think that it is quite an awkward number: too high for real-world experiments (where even 1000 is a very high number of sequential experiments), but too low for in-silico screening (where even moderately expensive functions could be evaluated at least $10^5$ times). This makes me worried that the PMO setup does not represent real-world applications very well.

**Benchmark setup: choice of dataset**

To my knowledge, every molecular optimization algorithm uses data in some ways: for example to train a predictive or generative model, but at the very least to pick a "starting point" for iterative algorithms.
In general, it should be clear that data available impacts the difficulty of the optimization problem; problems are easier in general if there is data close to the desired optima, and more difficult if data is far from it.
For example, algorithms like MCMC or genetic algorithms which perform a guided random walk are much more likely to stumble across an optimum the closer they are to it, and will do so more quickly.
Therefore I believe that a dataset not only _influences_ the performance of algorithms, but is actually _an essential part of the definition of an optimization problem_.
Therefore, any benchmark must at the very least standardize or control the available data, and ideally deliberately choose the dataset to set the difficulty of the optimization problem.

Regarding standardization, I think that the authors did this reasonably well: they used the same subset of ZINC for all their baselines.
I would however like to see the authors more explicitly state that ZINC is the _only_ dataset permitted
when using the benchmark, so that future researchers cannot report spuriously high scores in the future achieved via careful dataset selection.
I think this would address the issue of standardization.

Regarding the choice of dataset, the authors chose to use ZINC because of its "pharmaceutical relevance, moderate size, and popularity" (line 201).
While ZINC does have all these properties, I think that the authors could have chosen their dataset more intentionally:
for example, they authors could imitate the Guacamol paper and choose the dataset more deliberately to influence the difficulty of the problem.
In Guacamol, the authors decided to make the tasks "exploratory", and filtered their training dataset to remove molecules with high objective function values or molecules too similar to the molecules which the objective functions were based on, so that any algorithm would need to explore molecules far away from the training set to receive a good score.
I think that the authors should make a similar explicit choice (either the same as Guacamol or perhaps testing "exploitative" algorithms instead),
and choose the dataset accordingly (if exploratory behaviour is desired then using the Guacamol dataset itself might be a wise choice).
Barring this, as a minimum I think the authors should have examined ZINC more closely with respect to the objective functions chosen,
for example performing analysis of the distribution of objective function values on ZINC for all objectives, and perhaps some post-hoc analysis of the molecules found during optimization to see how closely they match ZINC.

Overall, I think the dataset and its impact are under-discussed and under-examined.

**Choice of objective functions**

The authors based their benchmark off of a suite of previously published objective functions, chiefly Guacamol, but with some random-forest/SVM models too. Although the high number of objective functions may give the appearance that a wide variety of functions are being tested, I believe this is not the case based on my experience working with these exact functions.

- Guacamol functions are almost all based on fingerprint similarity to a real-world drug molecule, and therefore are generally 1) fairly smooth w.r.t. changes in molecular structure 2) unimodal 3) amenable to fairly strong performance from random datasets of generally drug-like molecules. These functions are very easy to optimize with random-walk like algorithms, as there tends to be a smooth path from any starting molecule towards the optimum via a series of small molecular changes which monotonically increase the objective function value. While some real-world drug design objectives resemble this (e.g. optimizing an initial "hit" molecule), others do not (e.g. finding an initial hit molecule, which is almost always a multi-modal function). 
- Model-based oracles (JNK3/GSK3b/DRD2) suffer from what I would call a "fragment bias": because they are trained on extremely sparse fingerprint vectors (i.e. vectors representing the counts or presence of certain substructures), in my experience there tends to be a simple relationship between certain structures and the predicted score (i.e. some structures increase the score while others decrease them). Therefore these functions tend to give a high score to extremely large molecules which have many copies of "positive" substructures and few copies of "negative" substructures. I think this is unnaturally well-suited to "fragment-based" algorithms which can learn which fragments are helpful/harmful and then add helpful fragments.
- QED is generally very easy to optimize: table 4 shows that almost every algorithm achieves similar performance (including random screening) with minimal variation.

My suggestions to the authors would be:

1. Include some more diverse functions not based on fingerprints (perhaps something based on docking? a few docking oracles have been released in the past year)
2. Re-balance the functions to have at most 1/3 based on fingerprint similarity
3. Exclude QED
4. Try to categorize functions as "exploratory" vs "exploitative" based on the strategy needed to optimize them, and try to balance the benchmark between exploratory and exploitative functions.

**Baselines**

Although I am pleased that the authors attempted to investigate a large number of baselines, I believe that there are some issues with the way that the authors have implemented them.

The most widespread issue is the way in which each algorithm uses the starting dataset. In addition to my thoughts on the importance of the starting data, I have 2 concerns about the way the baseline algorithms use this data.

1. Extremely similar algorithms use the starting data differently. For example, consider genetic algorithms, whose behaviour depends strongly on the starting population in a similar way (see discussion of "random walk" algorithms above), and have similar failure modes (e.g. insufficient diversity in the population leading to lack of progress). Despite this, all GAs tested seem to choose their starting population differently: STONED uses 500 random smiles, Synnet uses ~200 random smiles (can't tell exactly how many), SMILES GA uses 400 random smiles, SELFIES GA uses just the single smiles "C", Graph GA uses 120 random smiles, and DST uses 2000 random smiles. Surely some of the performance differences between these GAs results from this, instead of the different mutation operations or population selection mechanisms. I think this introduces unnecessary bias towards certain algorithms, and moreover the random selection of smiles introduces an extra source of variation. I think that the starting smiles should be identical between different algorithms (perhaps chosen with the same random seed) to eliminate this source of random variation.
2. Often the chose method of using the data is sub-optimal. In my experience with GAs for molecules, I think that the best method is generally to use a large fraction of the evaluation budget to randomly evaluate molecules (at least 10%), which is used to create the initial population for the GA (e.g. selecting the best molecules, or a diverse subset). Only DST seems to do this, the rest do something which I believe is worse. This would explain why some GAs do not exceed the performance of screening on certain tasks in Table 4. This also more closely matches the original Guacamol implementations, which do a large amount of random screening before starting the GA (often they start from the top few hundred molecules after randomly choosing several tens of thousands).

A second issue is early stopping, which the authors apply to many methods. I think that this is bad practice. Stochastic algorithms often proceed via random, low-probability jumps. If the probability of making the necessary jump is sufficiently low then it will take many attempts until the jump is made. In my experience using a variety of these algorithms, there are often long periods of no progress before a sudden increase. Early stopping removes the possibility of seeing this. One particular example where I have seen this in my own experiments is GraphGA (try running it for ~10x as long as the original Guacamol paper and you will see that the performance does not really stop increasing and is much higher than the performance in the table of that paper). By doing early stopping, I think that the authors' results systematically understate the true performance.

A third issue is batch sizes: many of the algorithms use very large batch sizes, which gives fewer opportunities to iteratively improve on the molecules found. For example, DoG-Gen has a batch size of several thousand, meaning it only has ~3 opportunities to iterate on a molecule, while other algorithms have perhaps 10x as many iterations. I would like to see the authors:
1. Choose the batch size consistently, at least within algorithms of the same class. Not doing this just introduces an additional source of bias.
2. Use small batch sizes where possible, since this will generally lead to better performance given a fixed number of calls to $f$

A fourth issue: I think there may be a bug in your code for screening. I downloaded your version of the ZINC dataset and computed the objective function for a random subset of 100k molecules for Zaleplon MPO (where screening does poorly). It looks like the 99th percentile in the dataset is ~0.39, which makes the AUC of 0.072 seem surprisingly low. You should double check that this is implemented correctly. In my experience, screening can be a frustratingly strong baseline.

Overall, I have serious concerns that at least half of the baselines are are set up in a highly sub-optimal way.
I understand that implementing all baselines optimally might require the authors to be an expert in all the baselines, which is a lot of work; the authors acknowledge this in their conclusion (line 306).
It may also seem like I am just asking the authors to "do more tuning" which is a common request from conference reviewers almost regardless of how much tuning was done.
However, I feel in this case that asking for better tuning of the baselines is justified because:
1. Some key parameters were not tuned at all, and as an expert familiar with these methods I believe that their default setting is poor.
2. I think it is important to have very strong baselines when proposing a new benchmark to ensure that it is actually challenging to achieve SOTA performance.
3. Disregarding how good the tuning was, things like batch sizes and starting datasets are _inconsistent_ between different algorithms, which makes comparing them less meaningful. Comparing algorithms is another important purpose of baselines.
4. The baselines are, in my opinion, the biggest contribution of this paper, and therefore should be held to the generally high standard that the NeurIPS D&B track has set for itself.
clarity: In general this paper was very well-written so I don't have much feedback. One thing that I think would be helpful though is an equation defining AUC precisely: I think I know what it means but would like to see an equation to ensure that I am interpreting it correctly.
relation to prior work: I think that this paper cites and discusses all prior work appropriately. One suggestion I have is to analyze the commonalities between different algorithms more, rather than treating all algorithms as a fixed set of design choices. For example, all GAs have populations, and mainly differ in how mutations occur. REINVENT and hill-climbing are essentially just 2 different ways of training the same policy. I think if you do this the class of algorithms that you consider might be more general and clearer to the reader.
additional feedback: **Alternative focus of the paper**: I think that defining a good benchmark for molecular optimization is very difficult, so I understand that many critiques made in my review might not be easy to address. If this is the case, one thing you could do is re-package this work as an empirical paper for a cheminformatics journal. I think that a lot of people in chemistry tend to use methods "out of the box" and with a small sample budget. Your results in these settings (namely that GAs/REINVENT are strong choices) would be of great interest to many practitioners, leading to an impactful paper.

**Recommendation for baselines**: I think for this kind of paper it would be much better to have 5 really well-tuned baselines (perhaps a representative method from many classes of algorithms) rather than a large number of moderately-tuned baselines, just as your current set of 23 moderately-tuned baselines would be better than 100 very poorly configured baselines.
review_title: Paper with insightful experiments but proposed benchmark needs more work

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper provided a standardized and thorough benchmark for molecular optimization. It is good to have such a benchmark for future molecular optimization task to follow and compare. This work benchmarked thoroughly 25 representative generative methods over 23 oracle functions. Compared to previous benchmarks, this work provides several new oracle functions. The author also suggested the AUC Top-X as a metric instead of current Top-X to consider sample efficiency and optimization ability. Moreover, the paper also gave several useful experimental results. The main weakness for this paper is the newly added oracle functions and previous ones are not very useful for real drug discovery MPO (multi-properties optimization) or PMO tasks. I suggest to add docking score, synthetic feasibility assessment, ADMET properties and so on in future work.  
strengths: 1. Provided a standardized and thorough benchmark for molecular optimization, including 23 already used oracle functions and the introduced optimization ability and sample efficiency in this work.
2. Very thorough summary and comparison of molecular generation methods, even some older algorithms that not very often compared by current algorithms. Therefore, the current benchmark and metrics are important.
3. Suggest firstly the AUC Top-X as a metric instead of current Top-X to consider sample efficiency and optimization ability for PMO task.
4. It's good to compare SMILES with SELFIES using different molecular generation methods and give us a sound conclusion about the two common molecular representation methods. 
weaknesses: Compared with previous benchmark Guacamol, only added QED, DRD2, GSK3, JNK3 and two more functions, however, more relative to real drug discovery functions or properties are still missed. For example, docking scores as they mentioned, synthetic feasibility assessment, and some key ADMET properties. Because most of the current oracle functions are not very useful for real drug discovery, even the ones used in Table 2 of this work. 

correctness: The evaluation methods and experiment design appropriate and performed correctly as a benchmark.
clarity: This paper is well written and organized.
relation to prior work: The comparison with previous methods, such as Guacamol and TDC, are clearly discussed.
additional feedback: No
review_title: A good benchmark for molecular optimization/generation and also some good points are validated by experiments, but some new and useful oracle functions are needed.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper creates an open and unified benchmark to fairly evaluate existing molecular optimization algorithms. Different from previously used evaluation metrics, the proposed benchmark particularly focuses on sample efficiency. Comprehensive experimental studies are conducted to evaluate and compare previous molecular optimization methods.
strengths: (+) This work provides an excellent and convenient toolkit for unified evaluation of molecular optimization algorithms, which is very helpful for new comers in this research area and easy to fairly compare different optimization methods.  
(+) This work emphasizes the importance of sample efficiency, which is actually critical in practical molecular optimization applications but has been long ignored in previous molecular optimization method. This insightful point makes a significant contribution in leading researchers in this area to consider more about the real-world application scenarios when developping their methods.  
(+) The experimental results show that some old methods are in fact comparable or even more powerful than most recent methods. This discovery is really enlightening and can motivate researchers to rethink the values of old methods.
weaknesses: (-) This work focuses on the single-objective molecular optimization problem. However, in some real-world applications, we need to solve the multi-objective molecular optimization problem. I hope in the future, the authors can improve the current benchmark by including multi-objective optimization methods and evaluation.  
(-) A typo exists in line 76: "demotes" should be "denotes".
correctness: The claims of this work are sound and well supported by solid experimental results. All evaluation methods and experiment design are reasonable and performed correctly.
clarity: This paper is clear and well-organized.
relation to prior work: This paper clearly discusses the relation to prior work and clarifies the major difference in evaluation metrics between the proposed benchmark and prior work.
additional feedback: N/A
review_title: An awesome molecular optimization benchmark providing a chance to evaluate the sample efficiency

Meta-review: metareview: Five out of the six reviewers are supportive to this work. There have been very extensive discussions between the authors and reviewers during rebuttals, especially the one reviewer who provided slightly negative recommendations. Overall, I feel most of the concerns have been addressed and thus recommend an accept.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept