Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A new algorithm for retrosynthesis prediction based on a new GNN architecture and semi templates is presented, which yields state of the art results on some benchmark tasks.
main_review: *strengths*
- good results
- reasonable model

*weaknesses*
ablation of representation is missing: It could be argued that the most gains are from the new GNN architecture. It could be instructive to try out the same model with the same architecture that G2G, GLN, or NeuralSym uses. Or maybe with DRGAT,  G2G, GLN, or NeuralSym would also perform better?

baselines is missing: I would suggest to add
- Sacha et al - MEGAN model https://arxiv.org/abs/2006.15426
- Sun et al https://arxiv.org/abs/2007.13437 
- Seidl et al https://arxiv.org/abs/2104.03279 to the results table, which outperforms SemiRetro in top10 accuracy.



notes:
In the introduction, the authors write:
`Fortunately, with the rapid accumulation of chemical data, machine learning is promising to solve this problem (Szymkuc et al., 2016; Coley et al., 2018; Segler et al., 2018).` - However, Szymkuc et al 2016 argue *against* the use of data-driven approaches, so I would suggest to not to cite them in this context.

summary_of_the_review: interesting model, experiments could be stronger
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: * This paper is interested in the problem of one-step retrosynthesis, i.e., predicting the reactants that formed a given product.
* For this task, the authors propose a new model (SemiRetro) that combines ideas from previous template-based (TB) and template-free (TF) methods. 
* SemiRetro consists of two parts:  
	i. First, DRGAT (a new GNN architecture based on edge attention) predicts the reaction center, and breaks down the product molecule into synthons.  
	ii. Second, "semi-templates" (chosen from a fixed set by a classification model) is used to complete the synthons and assemble the reactants.
* The authors show:
	1. DRGAT works much better than previous methods for identifying reaction centers (specifically the R-GCN used in Shi et al. (2020)'s G2G) -- see Table 2.
	2. That the use of semi-templates allows SemiRetro to cover a wider range of reactions than previous template-based methods while also using less unique templates -- see Figure 5.
	3. SemiRetro outperforms many previous TB and TF methods in terms of top-k accuracy, particularly in the reaction class unknown case -- see Table 4.

main_review: # Strengths
S1. The approach obtains superior performance (wrt accuracy) compared to many previous retrosynthesis methods. 

S2. The "semi-templates" used can cover a larger space of reactions compared with using a similar number of full templates (Figure 5). This leads to improved training efficiency.

S3. The method is faster to train and takes up less GPU space (which I assume also means it has less parameters...?) than many previous methods. (Table 6).

S4. I found the paper clear to read. In particular the figures were helpful for explaining the approach.

# Weaknesses
W1. A large part of the performance of this approach seems to stem from DRGAT's superior reaction center prediction (compared to e.g. the R-GCN used by Shi et al., 2020) -- in particular see Table 2. However, it is not clear exactly where this performance comes from: as well as introducing an edge based attention mechanism the authors also use more layers than the R-GCN. It would be helpful to have an ablation study. How does it compare generally to well-tuned, off-the-shelf GNNs...?

W2. I have concerns about the novelty of the paper. SemiRetro seems quite similar to Somnath et al. (2020)'s GraphRetro (that method also has a two step process that first predicts the reaction center to create synthons and then adds "leaving groups" to create reactants). I would like to have seen a more detailed description of how the work here differs? For instance, is predicting semi-templates distinctly different to predicting leaving groups?

W3. SemiRetro's training efficiency and scalability are highlighted as one of the advantages of the method (see e.g. abstract) and the speed improvements listed in Table 6 seem impressive. It would therefore have been nice to see experiments on larger retrosynthesis datasets, e.g. the larger USPTO one used by Dai el al. (2019), to show the practical advantages of this.

W4. In the retrosynthesis experiments there are missing comparisons to state-of-the-art (SOTA) methods, e.g.:  
> Sun, Ruoxi, et al. "Energy-based View of Retrosynthesis." arXiv preprint arXiv:2007.13437 (2020).  

> Sacha, Mikołaj, et al. "Molecule edit graph attention network: modeling chemical reactions as sequences of graph edits." Journal of Chemical Information and Modeling 61.7 (2021): 3273-3284.    Preprint: https://arxiv.org/abs/2006.15426

It would be good to have these included.

# Questions to authors
Q1. Can you compare to other synthon completion methods in Table 3. For instance, Somnath et al. (2020)?

Q2. How do you make use of the reaction class when it is known? Do you just use it to mask out inapplicable templates or do you also provide the information to the reaction center network?

Q3. In the right hand subplot of Figure 5, the dotted green light appears to end above 1. Is this correct?

Q4. I'm a little confused by equation 5, in particular the second readout term. Are the node embeddings, $\mathbf{h_k}$, taken from message passing run on (a) the template graph or (b) from the synthon graph?

Q5. On p.6 you state: "Once reaction centers, synthons, and corresponding semi-templates are known, we can deduce reactants with almost 100% accuracy." Why _"almost"_ 100%? When does it fail?

Q6. I would be interested to have more details about why you think SemiRetro has better "interpretability" than previous approaches? (Previous approaches have also used templates, and the reaction center prediction here still seems to be done in a black box manner?)

# Typos/Minor Comments
A few minor comments I had (all easily fixable) are:  
i. p.1 "(seeing Fig. 1)" to "(see Fig. 1)" .  
ii. Figure 5 -- the subcaptions are both labeled a.  
iii. p.6 "Reactants can be determinately generated from semi-templates and synthons": "determinately" should be "deterministically"?  
iv. Sum term in yellow box on right-hand part of Figure 3, I believe you mean $i \in \mathcal{N}_j$?  
v. p.7 "As mentioned in previous works, the USPTO-50k dataset contains a shortcut in that the product atom with atom-mapping ”1” is part of the reaction center in 75% of the cases.". It's great that you draw attention to the issues with information leakage and atom mapping. However, perhaps provide a reference to the previous works for non-expert readers.  
vi. p.7 "Implement details" to "Implementation details".  

summary_of_the_review: SemiRetro seems an interesting method for retrosynthesis that combines ideas from both template-based and template-free approaches. It also seems to get impressive empirical performance (with the caveat that some SOTA methods are missing from the comparison -- see W4). However, I have currently gone with a lower overall score as I worry the method is similar to previous work (see W2) and I feel the paper could do a better job at explaining where the performance gains come from (see W1).

The other scores:
- correctness: see Q6 & W3. 
- technical novelty: see W2.
- empirical novelty: the paper mainly makes use of existing benchmarks/evaluation techniques.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a novel template-based retrosynthesis prediction method. The authors reduce template redundancy and obtain 150 abstract semi-templates, which can cover 96.9% of data samples. Besides, the authors propose directed relational graph attention to find the reaction center. Experiments show the proposed method can achieve superior retrosynthesis performance.

main_review: Strengths:

The paper is well organized and written. Easy to follow.

The authors propose to break full-template into several simpler semi-templates, and conduct synthon completion based on the semi-templates. The template redundancy can be reduced since many semi-templates are reduplicative. As a result, they can obtain 150 semi-templates that cover 96.9% of reactions. 

To better recognize the reaction center, a directional relational graph attention layer (DRGAT) is introduced to extract more expressive molecular features.

The proposed method achieves the SOTA 64.4% and 55.3% top-1 accuracy for reaction class known and unknown settings, respectively. Besides, the authors also show the detailed performance of two steps of the retrosynthesis method.


Weaknesses:

The proposed semi-template method is interesting, however, it has already been published in a journal article [1]. The authors of [1] name the semi-template as the local reaction template. If I understand them right, the semi-template in this paper and local reaction template are the same stuff. Please correct me if I am wrong. If that is the case, it will make the paper less significant. 

The presented DRGAT layer is based on the previous framework, what is the major contribution or novelty of DRGAT that comes from the authors? Could the authors clarify that?

What is the detailed composition of atom and bond features mentioned in Table 1? I can not find them in the submission file.

Have the authors removed the original mapping numbers from test data? If not, could the authors run inference without the original mapping numbers, and confirm the reported results can be reproduced? As the authors mentioned, this is an information shortcut with the USPTO dataset. Sometimes, it is really difficult to recognize the shortcut. So it is best to completely remove the potential of shortcuts by replacing the original mapping numbers for test data.

The proposed method seems quite generalizable, could the authors also report results on the USPTO-full dataset like GLN to demonstrate the scalability of the proposed method?

[1] Deep Retrosynthetic Reaction Prediction using Local Reactivity and Global Attention. Shuan Chen and Yousung Jung. JACS Au 2021 1 (10), 1612-1620.


summary_of_the_review: The proposed method has achieved impressive experimental results on the small dataset USPTO-50K. The proposed semi-template is novel and effective, however, it has been published in previous work. The presented DRGAT is quite incremental. The synthon completion method is the same as the GraphRetro. The overall framework is very similar to previous work G2Gs and GraphRetro. In summary, except for the performance boost, no new ideas or methods are proposed in this paper, and the contribution and novelty of this paper are limited.

*****************************************
After reading the author's response. I am quite confident there is an information leak in both G2Gs and this work. The authors did not canonicalize the test SMILES during model evaluation which is not a correct way to evaluate their method. They can not reproduce the reported results with canonicalized test SMILES. Please the authors canonicalize the test SMILES and update experimental results.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies a retrosynthesis problem of chemical reactions. 
This paper proposes a new variant of retrosynthesis models, with mainly two new components: DRGAT for powerful embedding computation and the semi-templates for accurate synthon completions. 
Especially, the propose semi-template takes the local (neighboring) bonding structure into consideration. This significantly reduces the redundancy of the templates, allowing connecting the template-based retrosynthesis (less scalable but powerful) and the template-free retrosynthesis (scalable but less powerful). 
The experimental validations show the efficacy of the new components for each subtask, and the final retrosynthesis performances exceeds the existing retrosynthesis models. 
main_review: (Sorry I re-organize the review to meet the guideline. No changes in the contents/opinions from the initial submission.)

### points

(+): a new variant of retrosynthesis problem using a local-structure aware semi-template

(++): good numerical performances confirmed in the sub-problems and the main retrosynthesis

(-): insufficient explanations of DRGAT

(-): possible missing reference and comparison

(-); visibility of figures are generally low

### comments

I like the idea of this paper, and enjoy reading the manuscript. 

In my understanding, the main stream of the proposed model follows the typical template-free retrosynthesis. 
In the current TF approaches, the center identification sub-problem affects a lot in the final retrosynthesis problem. 
The proposed DRGAT seems performing great in this sub-problem (Sec. 5.2). 
However, the explanation of the DRGAT model only appears in the Figure 3. I expect the revised manuscript add a few sentences/equations to convey the main idea of the model. 

The semi-template model for synthon completion is quite interesting. 
It seems for me a good alternative of the GraphRetro's residual prediction approach. 
Question: Why GraphRetro and RetroXpert are not studied in the Sec. 5.3? In terms of the synthon completion, GraphRetro's residual modeling may perform as good as the proposed semi-template.  

I found that one of the important previous work is missing from the discussions and the experiments; energy-based models.
Please discuss the relationship with the proposed model, and if it is appropriate, compare in experiments. 

@misc{
sun2021energybased,
title={Energy-based View of Retrosynthesis},
author={Ruoxi Sun and Hanjun Dai and Li Li and Steven Kearnes and Bo Dai},
year={2021},
url={https://openreview.net/forum?id=0Hj3tFCSjUd}
}


Readability: In general, figures are low-visibility. Especially, atoms, bonds, molecules, equations are not displayed well on screens. 
I hope these figures are improved in the camera ready as much as possible. For example, thick/strong lines and fonts, larger symbols, and so on. 
summary_of_the_review: It is great to see the stead improvement of the retrosynthesis. 
The semi-template idea sounds reasonable and good to me. 
The explanations on the DRGAT is clearly not enough, and in general, figures are hard to see on the screen. 
My positive score is with the expectation that the insufficient explanations and low-visibility figures are properly addressed in revisions or the camera-ready. 

=============================================================

After author feedback;
Thank you again authors for the efforts to revise the manuscript and discussions. 
One of the reviewers raise the concern about the leakage problem of the USPTO dataset, and I agree the problem is important. 
Some strong competitors are also introduced by other reviewers. 
As a result, we observe a few but important updates that are not reflected in the revised manuscript. 
I still believe the proposed semi-template approach is new and valuable to the community, but this time I cannot recommend this manuscript for acceptance. I expect the authors to come up with a manuscript including all new updates in near-future conferences.
review_title: null

Meta-review: This paper proposes a new semiretro algorithm by combining the two major approaches of retrysynthesis, the template based method and the template free method - breaking a full-template into several semi-templates and embedding them into the two-step template-free framework.  They also obtained state of the art performance in this task based on the recent GNN architecture. Although all reviewers were satisfied with the idea and excellent performance of this paper, the possibility of information leakage in their experiments was raised through the discussion period, and the authors seem to agree to this to some extent.

In conclusion, it is difficult to say that accurate and rigorous experimental verification of the proposed method has been made yet. I encourage the authors to resubmit the paper after correcting errors in their experiments.
Decision: Reject