SDDBench: A Benchmark for Synthesizable Drug Design

Authors: Songtao Liu, Zhengkai Tu, Hanjun Dai, Peng Liu
Primary area: datasets and benchmarks
Keywords: Drug Discovery

Abstract:
A significant challenge in wet lab experiments with current drug design generative models is the trade-off between pharmacological properties and synthesizability. Molecules predicted to have highly desirable properties are often difficult to synthesize, while those that are easily synthesizable tend to exhibit less favorable properties. As a result, evaluating the synthesizability of molecules in general drug design scenarios remains a significant challenge in the field of drug discovery. The commonly used synthetic accessibility (SA) score aims to evaluate the ease of synthesizing generated molecules, but it falls short of guaranteeing that synthetic routes can actually be found. Inspired by recent advances in top-down synthetic route generation and forward reaction prediction, we propose a new, data-driven metric to evaluate molecule synthesizability. This novel metric leverages the synergistic duality between retrosynthetic planners and reaction predictors, both of which are trained on extensive reaction datasets. To demonstrate the efficacy of our metric, we conduct a comprehensive evaluation of round-trip scores across a range of representative molecule generative models.

Decision: reject
Reviewer scores: 3, 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730069499942

thread_type:
official_review

<thread_object>
creator: reviewer_nvcx

<content>
summary:
This paper proposes a new metric to assess the synthesizability of molecules. The presented approach makes use of data-driven reaction prediction and retrosythesis models to directly measure if feasible synthetic pathways are available for a given molecule. This kind of scoring metric is of high importance in light of the surge of generative drug design models whose outputs are rarely validated experimentally due to difficulties in making novel molecules in the lab.

strengths:
### Overall idea and evaluation approach

The paper addresses an important limitation of many computational drug design efforts. Chemical synthesis is often the key bottleneck preventing generated compounds from being tested experimentally in a wet lab. 
The authors come to the plausible conclusion that a stringent synthesizability metric should evaluate the feasibility of the whole synthesis route for a given molecule. Unlike previous approaches they not only perform retrosynthesis planning but also employ a reaction prediction model to determine whether the target molecule can indeed be obtained from the predicted starting building blocks.
The resulting score based on the Tanimoto similarity of the target molecule and the output molecule after this round-trip procedure is well motivated and convincing in my opinion. The specifically trained reaction and retrosynthesis models seem to achieve accuracies close to those of state-of-the-art models from what I can tell, indicating that the technical implementation of the score is sound.


### Presentation of results & clarity
The evaluation procedure has a clear structure that first aims to establish the usefulness of the proposed synthesizability score, and then benchmarks several popular structure-based drug design models with the newly defined metric.
The paper also introduces the topic nicely and covers a wide range of relevant related work.

weaknesses:
### Scope of this work
Based on the title I expected the paper to present a new benchmarking suite. Instead, it only introduces a single new metric, the "round-trip score" that was applied to the molecules generated by seven recent models for structure-based drug design. While the general structure of this evaluation procedure is reasonable as discussed above, I would argue the execution is insufficient to justify the claim of establishing a new benchmark. Consequently, I recommend to revise the title and claims to more accurately reflect the scope of this work.

The analysis and discussion in Section 5, which presents the benchmark, lacks depth. The paper does not provide any potential reasons for the differences between the benchmarked methods that could inform the future development of such algorithms.
To improve the paper, the authors could for example correlate their score with various molecular properties (e.g. molecular weight, number of rotatable bonds, hydrophobicity score, ...) and analyze the results. It would also be valuable to discuss case studies based on molecules with particularly low or high round-trip scores.

Lastly, the paper does not provide guidelines on how the synthesizability objective could be balanced with other (potentially opposing) objectives. For instance, I imagine molecule size to play a crucial role. Smaller molecules are often easier to synthesize but bind their targets less specifically.


### Empirical results

Assuming that the main focus of this paper is the new synthesizability metric rather than the benchmark in Section 5, I would have hoped for a more thorough evaluation thereof.
If I understand correctly, the same retrosynthesis model that is part of the round-trip score computation is also used in the definition of successful and failed cases in the evaluation in Section 3.2.2. Similarly, the reaction prediction model seems to be used in the "molecule-wise match" scenario. 
Given that the success criterion seems to be coupled to components of the newly proposed score, it is unclear to me how meaningful the comparison with the SA score is.
Furthermore, the empirical evaluation of the round-trip score should be extended to other baselines besides the SA score. Obvious candidates would be:
- SYBA [1],
- SCScore [2],
- RAscore [3].

These scores are more recent than the SA score but still commonly employed in this domain, which makes it easy for readers to interpret the presented results.

Finally, while the full forward-backward reaction prediction is well motivated, I suspect that its main limitation will be speed–at least compared to heuristics-based synthesizability scores. A comprehensive evaluation should therefore also include the time component, and could for example discuss the pareto front of high accuracy and low run time. To this end, the authors could add a runtime analysis in their evaluation.


### Presentation

To me, it was not always easy to follow the flow of the paper. Most concepts are correctly defined somewhere but not always in the order that would have seemed natural to me. Consequently, I had to jump back and forth several times while reading the paper.
The first sections include a few definitions that are not required for understanding the method, e.g.
- the definition of ligands and proteins in Section 2.1,
- the definition of the gap between metric scores in Equation 2.

At the same time, important concepts like the the set-wise exact match criterion and its difference to the molecule-wise (round-trip) match could have been explained more clearly.

The first sentence of the _Results_ paragraph of Section 3.2.2 introduces a new definition of successful cases. This should have been explained earlier. In general, it might be cleaner to move all the results to Section 5 and limit the discussion in Section 3 to a description of the method.

### References

[1] Voršilák, Milan, et al. "SYBA: Bayesian estimation of synthetic accessibility of organic compounds." Journal of cheminformatics 12 (2020): 1-13. 

[2] Coley, Connor W., et al. "SCScore: synthetic complexity learned from a reaction corpus." Journal of chemical information and modeling 58.2 (2018): 252-261.

[3] Thakkar, Amol, et al. "Retrosynthetic accessibility score (RAscore)–rapid machine learned synthesizability classification from AI driven retrosynthetic planning." Chemical science 12.9 (2021): 3339-3349.

questions:
- Why is the search success rate not a reliable metric (line 252)? This is an interesting point but the paper could discuss this claim more and ideally provide empirical evidence.
- The definition of the round-trip score using Tanimoto similarity between molecules appears rather _ad hoc_ and the connection with the theoretical objective stated in Equation 1 is not clear. How is the following sentence from the Conclusion Section justified? -- "This score evaluates the **likelihood** that a retrosynthetic planner, trained on current reaction data, can predict feasible synthetic routes for these molecules"
- Why do you train new reaction & retrosynthesis models? Have you also experimented with established tools (e.g. AIZynthFinder) or pretrained models?
- How do you justify the definition of the success criterion to evaluate the reliability of the round-trip score? Doesn't it depend on components of the score itself?
- What is the typical run time to compute this score? How does it compare to other synthesizability scores?



### Minor comments
- line 77: typo "t" missing in first
- Figure 5 & 6: legend should define meaning of error bars
- avoid using words like "exceptional" when describing own results (e.g. line 361)
- it is unclear to me what the "FusionRetro setup" introduced in Section 5 is referring to
</content>
</thread_object>

</thread>
<thread>
creation date:
1730202745381

thread_type:
official_review

<thread_object>
creator: reviewer_s5zs

<content>
summary:
The paper draws attention to synthesizability in generative design and proposes the "round-trip score" which assesses synthesizability by whether the predictions of a retro- and forward-synthetic model overlap, and otherwise, how similar the predictions are. The results contrast several 3D generative models and show that better property metrics such as docking score, does not necessarily equate to better synthesizability.

strengths:
* Synthesizability is an important problem in generative design
* Motivation of using Round-Trip score by comparing to SA score and that it can better distinguish between "successful" and "failed" molecules
* Comparison of several 3D models

weaknesses:
Overall, while synthesizability is an important problem, the proposition to assess generated molecules with retrosynthesis models has been reported in cheminformatics and ML literature.

* The idea of assessing generated molecules by retrosynthesis models is routinely used. In cheminformatics literature, this has been very common and many works benchmark proxy scores, such as SA, SC, RA scores with retrosynthesis models to study their correlation [1]. Other works explicitly highlight the use of retrosynthesis models to assess the synthetic feasibility of generated molecules [2, 3]. 

* At conferences, work on synthesis constrained generative models explicitly tackle synthesizability. Here I will cite a few of them [4, 5, 6].

* At conferences, work on synthesis constrained generative models assess generated molecules using retrosynthesis models. Here I will cite a few of them [7, 8, 9]. Although I acknowledge some of these are concurrent works.

* This paper, in addition to [10] cited by the authors, also explicitly highlight the problem of synthesizability of generated molecules [11]. This paper also discusses the use of surrogate models aimed to predict the output of retrosynthesis models and have routinely been used to assess synthesizability. Some specific examples are RAscore [12] and RetroGNN [13]. While this is not exactly the same as directly assessing using retrosynthesis models, these models are trained on the output of retrosynthesis models and is a similar idea.

* The authors stated in the introduction that a minor change in functional group can make a synthesizable molecule, unsynthesizable. The proposed metric to compute the Tanimoto similarity between the reconstructed and generated molecule as a proxy for synthetic accessibility could suffer from more error. Molecules with 0.9 Tanimoto similarity can have quite different atom arrangements that affect reactivity and therefore synthesizability. It would be interesting to take a series (share common structural features) of known synthesizable molecules, run retrosynthetic analysis on them, and then run the forward model on the routes. This could provide insights into the sensitivity of the proposed Round-Trip score on seemingly small structural changes. It would also be interesting to see how many molecules fail the round-trip despite being known to be synthesizable, due to limitation of training data.

* The main text should also discuss the properties of the generated molecules (docking score, QED, etc.). I know the focus of the paper is on synthesizability but synthesizable molecules with poor properties will not be further considered in practical applications.

### **References**
[1] Synthesizability correlation: https://jcheminf.biomedcentral.com/articles/10.1186/s13321-023-00678-z

[2] Retrosynthesis models to assess generated molecules 1: https://pubs.rsc.org/en/content/articlelanding/2023/sc/d3sc03781a

[3] Retrosynthesis models to assess generated molecules 2: https://pubs.rsc.org/en/content/articlelanding/2024/md/d3md00651d

[4] MOLECULE CHEF: https://arxiv.org/abs/1906.05221

[5] DoG-Gen: https://arxiv.org/abs/2012.11522

[6] SynNet: https://arxiv.org/abs/2110.06389

[7] SynFlowNet: https://arxiv.org/abs/2405.01155

[8] Reaction GFlowNet: https://arxiv.org/abs/2406.08506v1

[9] RxnFlow: https://arxiv.org/abs/2410.04542

[10] Synthesizability Problem 1: https://pubs.acs.org/doi/full/10.1021/acs.jcim.0c00174

[11] Synthesizability Problem 2: https://www.sciencedirect.com/science/article/pii/S0959440X2300132X?via%3Dihub#bib53

[12] RAscore: https://pubs.rsc.org/en/content/articlelanding/2021/sc/d0sc05401a

[13] RetroGNN: https://pubs.acs.org/doi/10.1021/acs.jcim.1c01476

questions:
1. When classifying the test set molecules as “successfully” solved, the metric is whether the solved routes have starting materials identical to the reference routes. As eluded by the authors, many starting materials can lead to the same product. The authors use the classified “successful” and “failed” sets to assess round-trip score using the forward model. How does the separation change when considering all molecules that have a solved retrosynthetic route (even if not identical to the reference route)? This can be heavily biased because the retro and forward are trained with no failed reactions. If a retro route does not lead to the reference route’s starting material, it does not necessarily mean it is incorrect.

2. Pocket2Mol generally generates smaller molecules. The metrics for it may be high because many of the generated molecules are almost directly building blocks. It would be informative for context to include molecule size/number of heavy atoms as another metric. 

3. A specific question about Pocket2Mol is in Table 1, the authors show that the `Search Success Rate` is highest. What is the route length distribution of these "successful" molecules? Are any directly in the building blocks?

4. How many molecules out of the 10,000 generated molecules do not have a solved retrosynthesis route according to NeuralSym? This is an important metric missing to contextualise the round-trip score.

5. What are the building blocks used?

Minor comment: The section on AGI breaks the flow of the text and in my opinion, not needed. More data will always lead to better models and is not exclusive to reaction data or language data (like in the GPT example given by the authors).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730415761065

thread_type:
official_review

<thread_object>
creator: reviewer_1qx2

<content>
summary:
In this work, the authors focus on synthesizability assessment for molecules proposed by structure-based drug design (SBDD) models. Molecular generators (especially structure-based) often generate molecules that maximise a target metric but are hard or impossible to synthesize. To address this limitation, the authors propose a new metric, called Round-Trip Score. The metric works as follows: given a molecule proposed by some molecular generation model, a synthesis plan is proposed by a retrosynthesis planner. A different, single-step forward synthesis model is then used to simulate wet lab experiments and sample the next intermediate products until a final molecule is reached. The Round-trip score is given by the tanimoto similarity between the final molecule obtained through simulated forward synthesis and the initial query molecule. They evaluate the reliability of the proposed metric using the USPTO reaction dataset, and use it to benchmark a variety of existing SBDD models.

strengths:
I believe that this work addresses an important limitation of many molecular generation models i.e. the lack of synthesizability guarantees of the proposed molecules. The ability of ML-driven methods to propose synthetically accessible compounds represents an important gap for the successful applications in real-world impactful problems such as drug discovery. While not the only possible approach, combining unconstrained molecular generators with retrosynthetic planners is an interesting direction for addressing this limitation. The authors propose a reasonable metric combining two different models trained on reaction datasets to assess the synthesizability of a given compound. I found the paper to be generally well written. 

In particular:
1. The Background section is clear and well presented.
2. The Related Work section is substantive and informative (although there might be some missing areas, see below).
3. The experiments benchmarking existing SBDD models in Section 5 and in appendix are informative.
4. I found the discussion on lines 355-366 comparing the performance of the forward prediction model and retrosynthesis planner to be informative.

weaknesses:
Despite the merits of this work (see above), I also have important concerns. I believe that the proposed metric (Round Trip score) is reasonable and can bring value by being used for evaluating (or even guiding) the ability of SBDD models to generate synthesizable compounds. However, my main concern is about the reliability evaluation of the proposed metric:

1. It seems to me that the "wrong" target is set for that metric: the authors mention multiple times that the proposed Round-Trip score aims to "assess the likelihood that these proposed molecules can have feasible synthetic routes predicted by the retrosynthetic planner" (lines 401-402). This is *what* the metric is doing, how it is built, but what it should *aim* to do is to actually inform the experimenter on whether a proposed molecule is likely to be synthesizable in real-life. Therefore, the evaluation of the reliability of that metric should seek to assess to what extent this Round Trip score correctly distinguishes between molecules that are known to be synthesisable, and molecules that are known not to be synthesisable. However, both of the categories described in Figure 5 ("Successful" cases and "Failed" cases) belong to the first category: molecules that are known to be synthesizable: "We employ Neuralsym+Beam Search (beam size is 2) as our retrosynthetic planner to predict synthetic  routes for target molecules in the test dataset. We then compare these predicted routes to the reference  routes provided in the dataset. If the starting materials in our predicted routes match those in the  reference routes, we categorize these molecules as successful cases. Conversely, if they don’t match,  we classify them as failed cases." (lines 320-323). Therefore, observing a gap in the Round Trip score between these two categories essentially highlights a failure mode of the Round Trip score in assessing real-life synthesizability. I believe authors should instead report the proportion of such failure cases (i.e. the size of these relative groups: Successful Cases v.s. Failed Cases) as a measurement of the performance of the retrosynthesis planner component of the metric, and complement this analysis by evaluating the Round Trip score accuracy on negative examples as well (molecules that are known not to be synthesizable). The fact that the Round Trip score is high is simply a measure of the effectiveness of the forward synthesis prediction component (which shows very positive results) but not of the reliability of the metric as a whole. To be clear, the proposed metric still has merits and could be useful, but its evaluation appears flawed and thus its development incomplete. I might be missing something, in which case I believe this matter should be clarified in the manuscript, but otherwise, for the above reasons, I would currently recommend this paper not to be accepted in its current form.

In addition, here are additional points that I think could contribute to improving the paper in its current form:

2. Some claims made in the first paragraph on the introduction should be supported by appropriate citations. For exemple, on lines 30-32: "Drug design is a fundamental problem in machine learning for drug discovery. However, when these computationally predicted molecules are put to the test in wet lab experiments, a critical issue often  arises: many of them prove to be unsynthesizable in practice". From experience I agree with this statement, but it should be supported by some references (many papers have studied and addressed the poor synthesizability of compounds proposed by ML-driven molecular generators). The same goes for this other claim: "The sensitivity of chemical reactions is such that even minor change in functional groups  can potentially prevent a reaction from happening as anticipated." (lines 41-42).

3. The first contribution listed by the authors includes this sentence: "In our view, a molecule is synthesizable if retrosynthetic planners, trained on existing reaction data, can predict a feasible  synthetic route for it" (lines 74-76). In my opinion, while the use of data-driven methods for assessing synthesizability has merits and is worth studying, such a strong formulation represents a highly objectionable claim. ML-based retrosynthesis planners have flaws and blindspots and do not represent perfect oracles to assess a molecule's synthesizability, as highlighted by authors on lines 298-300, and in their own experiments in Section 3.2.2 as detailed above. One could argue, for example, that forward synthesis search methods based on actual reactions (rather than predicted reactions) give much stronger guarantees for synthesizability. I thus believe this claim on lines 74-76 to be incorrect and that it should be rephrased. For approximation, an *approximation* of synthesizability might be given by the success rate of a retrosynthesis planner. This would represent to me a weaker but more accurate claim.

4. While the Related Work section is well structured and pretty exhaustive, I believe that the positioning of this work could be greatly improved by explaining more clearly how its focus area (SBDD) cannot be tackled by other types of synthesis-aware molecular generation methods, which aims at producing molecules and synthesis path simultaneously, e.g. [1,2,3,4,5,6]. This line of work is briefly mentioned on lines 54-55 but should be much more exhaustively covered as this relates to the positioning of this contribution in the broader community. 

5. Similarly, the Related Work should also cover alternative synthesizability score e.g. SC score [7].

6. About the claim: "Our work is the firs to bridge the gap between drug design and retrosynthetic planning." (lines 77-80). This seems like a strong claim to me. There is at least this concurrent work by Guo & Schwaller [8] which does combine both family of methods, and I would be rather surprised that no previous work combined retrosynthesis planners and molecular generation.

Minor Comments:
1. typo line 77: is the first*
2. Figure 4 could be made clearer by highlighting in different colours or boxes the different steps of the score computation (retrosynthesis planning, forward synthesis simulation, molecule comparison).
3. The first paragraphs of Section 3.2.2 (under "Settings") could be made clearer. For example, Neuralsym is cited but not defined (while the reference is useful, the manuscript should be standalone).
4. The availability of the code is mentioned in Appendix (line 927) but no link is given. It would be good to at least clearly mention that the code will be made publicly available upon publication.

References:
- [1] Vinkers, H. M., de Jonge, M. R., Daeyaert, F. F., Heeres, J., Koymans, L. M., van Lenthe, J. H., ... & Janssen, P. A. (2003). Synopsis: synthesize and optimize system in silico. Journal of medicinal chemistry, 46(13), 2765-2773.
- [2] Gottipati, S. K., Sattarov, B., Niu, S., Pathak, Y., Wei, H., Liu, S., ... & Bengio, Y. (2020, November). Learning to navigate the synthetically accessible chemical space using reinforcement learning. In International conference on machine learning (pp. 3668-3679). PMLR.
- [3] Horwood, J., & Noutahi, E. (2020). Molecular design in synthetically accessible chemical space via deep reinforcement learning. ACS omega, 5(51), 32984-32994.
- [4] Gao, W., Mercado, R., & Coley, C. W. (2021). Amortized tree generation for bottom-up synthesis planning and synthesizable molecular design. arXiv preprint arXiv:2110.06389.
- [5] Cretu, M., Harris, C., Roy, J., Bengio, E., & Liò, P. (2024). Synflownet: Towards molecule design with guaranteed synthesis pathways. arXiv preprint arXiv:2405.01155.
- [6] Luo, S., Gao, W., Wu, Z., Peng, J., Coley, C. W., & Ma, J. (2024). Projecting Molecules into Synthesizable Chemical Spaces. arXiv preprint arXiv:2406.04628.
- [7] Parrot, M., Tajmouati, H., da Silva, V. B. R., Atwood, B. R., Fourcade, R., Gaston-Mathé, Y., ... & Perron, Q. (2023). Integrating synthetic accessibility with AI-based generative drug design. Journal of Cheminformatics, 15(1), 83.
- [8] Guo, J., & Schwaller, P. (2024). Directly optimizing for synthesizability in generative molecular design using retrosynthesis models. arXiv preprint arXiv:2407.12186.

questions:
1. Why would we expect retrosynthesis planners to be less limited than template-based synthesis planners? In particular, on lines 218-222, it is mentioned that "While previous methods (Bradshaw  et al., 2019b; Gao et al., 2022a) have attempted to simultaneously generate molecules and their  synthetic routes, they face limitations. For instance, the approach in (Gao et al., 2022a) is constrained  by the use of a limited number of reaction templates, which hinders scalability.". From my view, retrosynthesis planners rely on retrosynthesis predictors, which have been trained on finite dataset of reactions, so these models are also limited to the set of reactions that have been seen during training. How well can these models generalise to new reactions? This concern is further emphasized in the conclusion, lines 538-539: "To enhance the  robustness of our evaluation method, we advocate for the release of additional reaction data.". If more reaction data was available, couldn't competing methods based on templates simply extract more templates from this additional data? Thus aren't *both* family of methods limited by existing data?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730987557840

thread_type:
official_review

<thread_object>
creator: reviewer_gqp7

<content>
summary:
This paper presents SDDBENCH, a benchmark for evaluating the synthesizability of molecules generated by drug design models. It incorporates a "round-trip score," which assesses whether generated molecules can be practically synthesized by combining retrosynthetic planning with forward reaction prediction. The benchmark attempts to address a gap in drug discovery where generated molecules often display desirable pharmacological properties but pose challenges for real-world synthesis. The authors apply the round-trip score across various 3D structure-based drug design generative models, highlighting their synthesizability, although no baseline is given.

strengths:
- Synthesis is an important issue for generative models and the authors propose to address this with a new synthesis benchmark.
- The background on retrosynthesis and reactions is good.
- Quality of the methods figures is good

weaknesses:
**Main Weaknesses:**

- The language is extremely vapid, making it difficult to pick out the genuine contributions, and the arguments/paragraphs are not well structured.
    - The 1st paragraph of the introduction reports about 10 scientific facts about synthesis from the literature and yet only cites 1 deep learning article half-way through. This, in my opinion, should be reason enough for a rejection.
    - The structure of the paper is strange, with it first proposing the method, doing experiments to validate the method, then describing related work/background before finally benchmarking the models (in about 1 page).
- There is a significant lack of novelty in the ideas presented in the proposed metrics. This a further compounded by a number of novelty and contribution claims that are unjustifiable.
    - For instance, “Our work is the firs to bridge the gap between drug design and retrosynthetic planning.” This is a big claim to make given the authors cite many such works that do exactly this.
    - Guo and Schwaller, 2024 [1] recently directly optimised generative models using retrosynthesis planners. 
    - Furthermore, the use of retrosynthesis tools as an evaluation metrics has been used in many works.  [2, 3]
    - The paper also does not clearly communicate the need for the round trip score, which itself is a multi step extension of Schwaller et al, 2020 [4]. In Figure 5 right, it seems the successful cases all have a round trip score of 1, whereas the failed cases consistently have a round trip score implying negligible similarity. Can the authors then please explain the added value of round trip score in the molecule-wise setting over just reporting retrosynthesis success rates?
- The authors do not critically discuss the limitations of their work beyond stating there is a limitation in the amount of reaction data available.
- An interesting literature has emerged recently that aims to tackle the problem of synthetic accessibility by incorporating reaction-based constraints to the generative process into generative models [2, 3, 5]. However, mainly autoregressive and diffusion-baed models specialised for structure-based drug design were accessed? If this benchmarking study is to be useful these lines of work should be considered too.
- The authors mention provided code in Section A.1 but there is no link.
- As far I can tell, a random split was used. How is data leakage controlled?
- In Table 1, the metrics of the training data and test set could be included. I would also consider ChEMBL molecules as a baseline?



**Minor points:**

- Figure 5: Figure not clear. What model(s) is this for? What do the error bars represent?
- Figure 6: Can the error bars for failed and successful cases be made staggered/non-overlapping. It is very confusing currently.
- There is no background information on any of the other benchmarking efforts in drug discovery. E.g. physical realism like PoseCheck [6]
- Citing ~40 retro-synthesis models in the background is excessive and also not useful. I would instead cite as different categories are described. There are no citations for models in the “two-stage learning paradigm”.

Spelling mistakes:
- L077 firs -> first
- L302 have -> has

1. Guo, Jeff, and Philippe Schwaller. "Directly optimizing for synthesizability in generative molecular design using retrosynthesis models." arXiv preprint arXiv:2407.12186 (2024).
2. Stanley, Megan, and Marwin Segler. "Fake it until you make it? generative de novo design and virtual screening of synthesizable molecules." Current Opinion in Structural Biology 82 (2023): 102658.
3. Cretu, Miruna, et al. "Synflownet: Towards molecule design with guaranteed synthesis pathways." arXiv preprint arXiv:2405.01155 (2024).
4. Schwaller, Philippe, et al. "Predicting retrosynthetic pathways using transformer-based models and a hyper-graph exploration strategy." Chemical science 11.12 (2020): 3316-3325.
5. Bradshaw, John, et al. "A model to search for synthesizable molecules." Advances in Neural Information Processing Systems 32 (2019).
6. Harris, Charles, et al. "Posecheck: Generative models for 3d structure-based drug design produce unrealistic poses." NeurIPS 2023 Generative AI and Biology (GenBio) Workshop. 2023.

questions:
- Can the authors clarify exactly what data split they use to train the CrossDocked models?
- How robust is the proposed metric to the exact retrosynthesis tool used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731445348287

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We believe that some reviewers may lack sufficient background in retrosynthetic planning, current evaluation methods for retrosynthetic algorithms, and the definition of synthesizability. We recommend that reviewers refer to [1] for a more precise evaluation of submissions within this field in the future. It appears that some reviewers continue to rely on metrics such as search success rate, which, as we have explained, can lead to imprecise evaluations.

Additionally, synthesizability metrics like the SA score and SCScore do not ensure that a feasible synthetic route can be identified for a molecule, as we have emphasized throughout this paper. The true goal of synthesizability is to find an actual feasible synthetic route. Regardless of how high these metric scores may be, they cannot guarantee the discovery of a feasible synthesis route. Therefore, we propose that future metrics should be aligned with the retrosynthetic planner—a direction to which this paper is dedicated.

[1] FusionRetro: Molecule Representation Fusion via In-Context Learning for Retrosynthetic Planning, ICML 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732673880601

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Figure 2 provides the comparison of evaluation metrics for retrosynthetic planning. The search success rate deems both routes successful, while the matching-based metric correctly identifies the top route as incorrect and the bottom route as correct, demonstrating its superior reliability. Please check it. Thanks!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes SDDBench, introducing a ``round-trip score'' metric to evaluate molecule synthesizability in drug design. The metric combines retrosynthetic planning with forward reaction prediction to assess whether generated molecules can be practically synthesized. 

The authors apply this metric to benchmark various 3D structure-based drug design generative models. The work demonstrates several strengths in addressing a critical challenge in drug discovery where computationally designed molecules often prove difficult to synthesize. The authors propose an empirical approach combining retrosynthetic planning and forward reaction prediction, present their methodology clearly, and include a comprehensive related work section (which is occasionally missing in many of the recent works). The application to multiple contemporary structure-based drug design models showcases practical utility.

However, significant weaknesses emerge in three key areas. First, the novelty is limited as multiple reviewers note that using retrosynthesis models for evaluating synthesizability is already common practice, particularly in synthesis-constrained generative models. Second, the empirical validation relies on a small evaluation dataset of only 100 molecules, as highlighted by Reviewers s5ZS and 1qx2, with limited comparison against established synthesizability metrics beyond SA score and insufficient analysis of failure modes. Third, methodological concerns persist around the round-trip score's superiority over search success rate, the arbitrary nature of the 0.9 Tanimoto similarity threshold, and potential bias in the evaluation methodology due to coupling between the success criterion and components of the proposed score, as noted by Reviewer 1qx2.

The insufficient validation with just 100 molecules undermines confidence in the metric's reliability, despite authors citing evaluation time constraints. The limited advancement over existing methods, identified by multiple reviewers, was not adequately addressed in the rebuttal. Furthermore, methodological limitations including dependence on reference routes restrict practical applicability, while the evaluation methodology's circular nature raises concerns about the metric's reliability. Accordingly, the manuscript cannot be recommended for acceptance in its present form for ICLR. 

For future submissions, the authors should conduct larger-scale validation studies, establish reliability of the metric, provide more thorough comparisons with existing synthesizability metrics, better demonstrate practical advantages over existing methods, and address the methodological concerns raised during review. While the paper addresses an important problem in drug discovery, the current submission falls short of ICLR's standards for methodological rigor and novel contribution.