Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a retrosynthesis model which first predicts the set of graph edits transforming the target into incomplete molecules called synthons, then the model learns to expand synthons into complete molecules by attaching relevant leaving groups. 
main_review: The paper is overall well written and easy to follow. 

Point of criticism about this paper.

(1) Synthon completion boils down retrosynthesis (a generative problem) to a much simpler classification problem (classify leaving groups).

First, it is possible that the leaving groups of the unseen test set is not included in the predefined leaving groups. i.e. not a generative model.

Second, (Line 163) 50M samples are used to extract leaving group candidates. The testing examples are included in 50M, which is not allowed. 

Third, it is not fair to compare a classification problem with generative models (other methods)

(2) The ideas of "identifying reaction center and then complete the synthons" is largely the same with previous work, such as G2G, retroXert

(3) The performance is not superior compared with Dual model etc. 

limitations_and_societal_impact: yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: 
This paper presents GraphRetro, a single-step retrosynthesis model that represents an advance from last years' NeurIPS state of the art model on semi-template-based modeling of the USPTO-50k dataset. GraphRetro uses a graph neural network that predicts edits to transform a target into synthons and then expands the synthons into full molecules by attaching leaving groups (with the synthons and leaving groups learned from the training set). The authors compare the predictions on the USPTO-50k to existing baselines, and show examples of predictions, including wrong ones. Because of the particular construction of the model, the authors can decompose the sources of errors as wrong edit predictions or wrong synthon completions.

main_review: The paper represents an improvement on the semi-template based methods for retrosynthesis. The presentation is clear and the model presents significant additions compared to RetroXpert.

Although I like the organizational power of classifying retrosynthesis models as template-based, template-free, and semi-template-based, I think that as long as no data leakage happens and the templates are only generated using the training sets of a particular benchmark, then all these methods could be compared against each other in a particular benchmark (and yes, I know that we sorely need additional public benchmarks in this area). In that light, I think that this paper shows that the energy-based views of retrosynthesis outperform GraphRetro by a respectable margin in most categories, both in the template based, but importantly, also in most of the template-free methods. I understand the fundamental hesitation with including template-based methods in the comparison because they might not generalize to new reactions, but wouldn't a template-free method outperforming a semi-template-based model suggest that the former is much better yet for generalizing to unseen domains? Despite this important limitation of the work, I think that the contribution is still notable, and I hope that the authors will manage to amend this paper and publish it soon.

limitations_and_societal_impact: 
Because of the analysis on the errors, the authors do a good job at documenting certain limitations of this model. An important limitation of the paper, however, is the lack of a discussion of the results of the dualTB and dualTF models.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: ## Summary. 
The authors propose a new method for single-step retrosynthesis prediction. Their method first predicts graph edits using a message passing network yielding synthons which are then completed by adding leaving groups. The leaving groups are chosen from a fixed set using another MPN.

main_review: ## General comments: 
The proposed method is reasonably novel. It appears technically sound but we found a range 
of statements that are not convincing as detailed below.  Overall the submission is clearly written. 
In the current form, the significance is strongly limited due to flaws in the method comparison and evaluation.

## Reproducibility
Code is provided as Supplementary Material which makes the work sufficiently reproducible.

## Originality
The proposed method is reasonably novel within the scope of ML methods for chemical 
synthesis. However, it is unclear how the broader ML community profits from the 
methodology in this work. 

## Clarity
The paper is clearly structured and written. Notation is appropriate and 
follows standards in ML. Figures and tables are well designed. 
Figure 1 could be a bit more informative by displaying architectural
components, such as the encoder.

## Quality
The quality of this work is limited due to several incorrect claims, 
missing analyses and unclear formulations (see "Detailed comments" below). 
Furthermore, the method comparison leaves out many methods that 
should actually be compared. 

## Significance
It is unclear whether the proposed method represents and advance at all.
Due to missing error bars and statistical tests between methods, the 
performance values could just arise by chance (see "Detailed comments" below). 


### Detailed comments:
- L9f: “This decomposition simplifies the architecture, making its predictions more interpretable, and also amenable to manual correction.” - It is not clear which architecture is simplified. The method is more interpretable compared to what? How is interpretability measured in the first place? Given any method that predicts precursors from a product one could run an atom-mapping algorithm and also find where the edit happened making it interpretable as well. Any method is amenable to manual correction. Has it been demonstrated that manual correction is easier for this method.
  
- L25ff: “Despite their interpretability, these methods fail to generalize to new reactions.” - If generalization means non-trivial performance on a test set this statement is factually wrong. In fact Table 1 shows template-based methods outperform other ones.  
  
- L26f: “Template-free methods bypass templates by learning a direct mapping from the SMILES [24] representations of the product to reactants.” - Template-free methods need not use the SMILES representation of the molecules.
  
- L27: “Despite their greater generalization potential,..” - It’s unclear how generalization potential is measured or any argument provided why template-free methods have a greater one. For example reaction-templates might be sufficiently flexible to model chemical reactions which would render the mentioned statement false. The argument should either be dropped, extended or a reference should be provided.
- L38: Typo. Double period
  
- L44: “This formulation also simplifies our architecture” - Same as in the abstract. Compared to what is the architecture simplified.
  
- L45: “predictions more transparent, interpretable and amenable to manual correction” - I think that these claims are unsupported or not elaborated on enough. Also consider the comment regarding L9f here. What does transparent mean in this context?
  
- L46: “The benchmark dataset for evaluating retrosynthesis models is USPTO-50k, ..” - USPTO-50k is not the definitive single benchmark dataset for this task. Maybe add “A commonly used/An established benchmark”.
  
- L47ff: “The dataset contains an unexpected shortcut towards predicting the edit, in that the product atom with atom-mapping 1 is part of the edit in 75% of the cases, allowing predictions that depend on the position of the atom to overestimate performance.” - I think this should be formulated differently. Since the dataset contains the test outputs there obviously is a shortcut to predicting the right reactants. We think a better formulation woud be to state that there is an unexpected way of test set leakage. Minor note on grammar: The predictions do not overestimate performance but lead to overestimated performance.
  
- L51f: “.., GRAPH RETRO achieves a top-1 accuracy of 53.7% when the reaction class is not known, outperforming both template-free and semi-template-based methods.” - Why are template-based methods not mentioned at this point? Tetko et al. [1] report an almost identical performance 53.5% with a template-free method and DualTF in Table 1 has almost identical performance and is better at top-n acc. with higher values of n. It makes the impression that the metrics reported here are cherry picked and it’s not clear why a whole top-performing model category is not mentioned here. 
  
- L56f: “Templates are [...] or extracted algorithmically from large databases” - Templates can also be extracted from single reactions or small databases. We think that a better wording would be “extracted from atom-mapped reactions”.
  
- L57f: “Exhaustively applying large template sets is expensive due to the involved subgraph matching procedure.” - This argument is not uncommon in retrosynthesis papers, but usually no primary source is provided that actually compares this to alternative approaches. [2] actually seems to suggest that template-based methods might be faster than Transformers or a method based on GNNs (GLN). We think that data should be presented that speed issues are actually a problem for template-based methods and not for other approaches.
  
- L62: “Despite their interpretability, these methods fail to generalize outside their rule set.” - Of course they fail to generalize out of their rule set but if the rule set sufficiently covers the test set, which is attested by superior performance (Table 1) this should not matter. The proposed method could also be said to fail outside the leaving group vocabulary, but it is not a problem because these groups have good coverage. 
  
- L65: “Linearizing molecules as strings does not utilize the inherently rich chemical structure.” - How can SMILES strings encode a chemical structure without the chemical structure?

- L108: “Retrosynthesis pairs are atom-mapped” - They are not necessarily mapped.
  
- L149ff: “Unlike a typical classification problem where the labels are independent, edits can have possible dependencies between each other. For example, bonds part of a stable system such as an aromatic ring have a greater tendency to remain unchanged (label 0). We attempt to leverage such dependencies to update initial edit scores.” - I don’t really understand this sentence. Could you maybe elaborate?
  
- L209f: “Evaluating retrosynthesis models is challenging as multiple sets of reactants can be generated from the same product.” - If one is strict I think this should be reformulated as “as a product can be synthesized from different reactant sets”. “no unique solution”, “one-to-many problem” may also be useful phrases.
  
- L243: There are more papers that propose methods on this task, see e.g. [2]. These methods should be included in the comparison or there should be a discussion of why they are excluded.
Error bars should be provided in Table 1. If significant improvement cannot be established we think it would be better to tone down “Barring DUALTB, the top-1 accuracy is also better than other template-free and template-based methods. (L252f)” - to only stating comparable performance.

- L253ff: “For larger n, one reason for lower top-n accuracies than most template-based  methods is that templates already contain combinations of leaving group patterns.” - I don’t understand this argument and I think that this is not backed up by data. Could you elaborate?
  
- L260: “GRAPH RETRO also outperforms all the template-free methods in top-n accuracy.” - This is not true as DUALTF outperforms GraphRetro for n=1,3 if the reaction class is known.
  
- L261: “for GRAPH RETRO are also better than most template-based and template-free methods.” - Most other methods are better than GraphRetro on top-10 accuracy. I also don’t think that “most” is really relevant here as one can always add more models that perform worse. 

- L294: “This limitation is partly remedied by our model design, that allows for user intervention to correct the edit.“ - It is not shown why this is easier for this model than for others. If a user intervenes she could also solve the whole task which might be just as hard.

- L302ff: “Previous methods for single-step retrosynthesis either restrict prediction to a template set, are insensitive to molecular graph structure or generate molecules from scratch. We address these shortcomings [...]” - I don’t agree that these things are necessarily shortcomings. A restricted template set is not a problem it if has good coverage of the test set. Coverage is at least 92% in this setting, so I don’t think it’s the limiting factor. I think no algorithm better than random guessing can be insensitive to graph structure. SMILES based models also are sensitive to the graph structure via the linear representation of the graph. It is also not necessarily a shortcoming to construct molecules from scratch. It surely may use up more computational ressources or suffer from worse performance as a result, but the data doesn’t back this up.

- L307f “Our model outperforms previous semi-template-methods by significant margins on the benchmark dataset.” - This statement is too general as performance is not better than that of previous methods for all values of n. 

References:
[1] Tetko, I. V., Karpov, P., Van Deursen, R., & Godin, G. (2020). State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis. Nature communications, 11(1), 1-11.
[2] Seidl, P., Renz, P., Dyubankova, N., Neves, P., Verhoeven, J., Wegner, J. K., ... & Klambauer, G. (2021). Modern Hopfield Networks for Few-and Zero-Shot Reaction Prediction. arXiv preprint arXiv:2104.03279.
limitations_and_societal_impact: Limitations are addressed. 
Societal impacts are absent.
review_title: null

Meta-review: While the reviewers appreciated the clear presentation they ultimately felt that the authors did not make a strong enough case for why their method should be preferred to template-based and template-free methods. The main argument for rejection is this: it's reasonable to compare all methods against each other, and if we do so this work underperforms. The authors offered a few responses: (i) They pointed to different evaluation metrics for these classes of models, but this seems to be an argument in favor of either template-based methods (as they are more constrained by chemical knowledge) or template-free methods (as they train a single model end-to-end); (ii) They argue that their model is more interpretable which makes it worth underperforming template-free methods. This argument is undercut by the fact that template-based methods are arguably as interpretable and outperform their work; (iii) They argue that their model is able to leverage the graphical structure of molecules. While this is true it doesn't seem like it was worth it, the work needed to sacrifice performance compared to all other work (even other semi-template based work). My recommendation is to either adapt this work to be template-free by extending duality enforcement for graph structures, adapt it to go fully template-based, or submit this to a more specialized venue as recommended by Reviewer 1dhj.
Decision: Accept (Poster)