Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This submission tackles the problem of one-step retrosynthesis, a classic problem in chemistry, through the lens of energy-based models. The framing of the problem in this manner is elegant. The quantitative results may need to be clarified or resolved to ensure that the comparison to prior SOTA is fair (i.e., not using an additional model or leaking data of the ground truth reactants).
main_review: 
The framework presented here is an elegant way to think about the complementarity between retrosynthesis and forward synthesis. It is described well. 

1.	This is a revised submission from ICLR 2021 where the authors were unfortunately punished for lacking comparisons to SOTA RetroXpert and GraphRetro, both of which ended up having data leakages that inflated performance by a large margin. Please refer to https://github.com/uta-smile/RetroXpert for an explanation of the leak. Were the sequence-based models provided with atom-mapped SMILES or unmapped SMILES? Were the product and reactant SMILES strings recanonicalized after stripping atom mapping using RDKit? The current SOTA top-1 accuracy when reaction type is unknown is closer to ~53% using the revised RetroXpert and GraphRetro results. The high results reported in Table 2 may be explained by the fact that the “Generalized” model implicitly uses information about the true reactants (see comment below). 

2.	The notation in Equation 6 is confusing to me; is $p(X|y) = \exp(- E_\theta(X,y))$ what the authors intended to convey?

3.	The categorization of methods into template-based and semi-template based has an unclear role to me. Is there a need to draw these strong distinctions? Atom mapping can be approximated through maximum common substructure heuristics, so it is “free” in a sense. None of the datasets used for benchmarking these tools relied on manual labeling (cf. line 226). This limitation of methods requiring atom mapping is overstated.

4.	The setting displayed in Table 1 where the inferred reaction center is provided is an unusual choice as it does not reflect a real use case for these models in chemistry. Providing the inferred reaction center (which is inferred given knowledge of the true reactants) to the “Generalized” model makes the comparison to previously reported results invalid, but would explain the 69.9% reported in Table 2 in the absence of the data leak.

If the quantitative results are clarified or resolved to ensure that the comparison to prior SOTA is fair (i.e., not using an additional model or leaking data of the ground truth reactants) I believe this paper is deserving of a 7 (accept).

-----
I have raised my score to 7, as the authors have addressed the (potential) information leak and atypical evaluation with the generalized model
limitations_and_societal_impact: Limitations are not discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper summarizes many different models for for retrosynthesis under the EBM framework.
It introduces dual training.
By performing reaction center annotation, state of the art results are claimed.
main_review: The authors unify many recently proposed models for chemical retrosynthesis under the framework of energy based models (EBM)
Furthermore they propose dual training for re-ranking the predictions.
The authors report state of the art performance.

Several works (e.g. Segler 2018, Coley 2019) have combined retro-prediction with forward scoring for re-ranking, so the novelty of this aspect is somewhat limited.

The presentation of the paper is somewhat dense, which mostly due to the length limit.

Overall i think the paper could eventually fit into NEURIPS and is interesting, but there are several open questions which need to be addressed before the publication can be recommended. (see update after rebuttal)


Please note that retroXpert https://github.com/uta-smile/RetroXpert and graphretro https://arxiv.org/abs/2006.07038 results have been updated (they are lower than originally reported). so these results should be updated.

Questions:

The description of the reaction center assignment using Schwaller's method is very brief, and in the current form unclear how it is implemented. Reaction enters can only be assigned to full reactions (you need to have the reactants AND product for that). It is unclear how this can actually work during inference. Given the later corrected results of the 2 papers results I am a bit skeptical how this is actually done, and I would like to see a very clear and long description on what is actually done here.

I also had a look at the code, but could not find the data, or how this additional step is performed in the code.

I am looking forward to discuss this in the rebuttal phase.

______
Edit after discussion:

This reviewer has raised their score to reflect the improvements the authors will make to the manuscript after discussion with the reviewers.
This paper should be published in NeurIPS


limitations_and_societal_impact: No concerns wrt Societal Impact: basic research.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper attempts to describe sequence-based and graph-based one-step retrosynthesis models, which are the current mainstreams of retrosynthesis, in a unified manner with the Energy-Based Model (EBM).
This paper reformulates several existing models with the EBM, mainly focusing on the difference in the energy function definitions. 
In addition, a new EBM variant is proposed in this paper, and its performance is updated the State-of-the-Art.　

Aside from the mainstreams of the machine learning community, AIzynthfinder recently draws many attention as a free and powerful multi-step retrosynthesis engine for computational chemistry. Some comments on this engine may be beneficial for placing the submitted work in a broader context. 

Thakkar A, Kogej T, Reymond J-L, et al (2019) Datasets and their influence on the development of computer assisted synthesis planning tools in the pharmaceutical domain. Chem Sci. https://doi.org/10.1039/C9SC04944D
Genheden S, Thakkar A, Chadimova V, et al (2020) AiZynthFinder: a fast, robust and flexible open-source software for retrosynthetic planning. J. Cheminf. https://jcheminf.biomedcentral.com/articles/10.1186/s13321-020-00472-1

main_review: ## Originality: 
As far as I know, this paper is the first attempt to describe various Retrosynthesis models in a unified manner in EBM, and it is novel in this respect.
The bi-directional training of sequence models (Dual in this paper) has been shown to be useful in several previous studies, as cited in the text, and is not novel in itself.
However, the use of sampling to stably apply the method to retrosynthesis is a good idea. 

## Quality: 
I find no evident errors in terms of technical content. 

## Clarity: 
It is good that the paper carefully introduces the EBM.
On the other hand, since many models have to be covered in one paper, the explanation of each model is limited to a minimum.   I think that the current manuscript requires the reader to have some expertises in retrosynthesis.
In Sec. 2, various existing models are actually formulated by EBM. 
In the current manuscript, readers have to go back and forth between pages while remembering the formulas to confirm the differences between the model equations.
If it is possible, it would be easier for readers if the qualitative characteristics of each formula are summarized in one table in the appendix.

## Significance:
Numerical experiments show that a new State-of-the-Art for the one-step retrosynthesis has been achieved, which strongly supports the usefulness of the generalized Dual model proposed in the paper.
In addition, the fact that an unsupervised atom mapping transformer can achieve great performance is good news for readers who have difficulty in handling graph data (in fact, it is not an easy task for many ones).
The Appendix contains many additional experiments and information to evaluate the model from various points of view, which is useful to help the reader understand it.


(+) EBM interpretation of several retrosynthesis models are new
(+) Specific formulation for dual (bi-directional) training of generalized sequence models.
(+) new SotA
(+) appendix informative
(-) cumbersome notations and presentations (partially inevitable because this paper covers many models: understandable)

## after feedbacks
Feedbacks from authors and additional experiment report solves many concerns of reviewers. 
I believe this paper is worth publication in NeurIPS.  
I keep my positive score. 
limitations_and_societal_impact: No specific opinions
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a unified framework with energy-based models for retrosynthesis prediction. Different methods (sequence and graph methods) can be untied with different energy functions within the framework. A dual variant model is presented and it performs consistent forward and backward training. Experimental results demonstrate this consistent training improves the retrosynthesis prediction significantly.

main_review: The authors present a general EBM framework to unify existing sequence-based and graph-based methods. Through investigating different designs of the EMB framework, a dual variant model is proved to achieve the best performance on USPTO-50K dataset. The proposed dual model also trains a forward reaction prediction model, combined with the retrosynthesis prediction, the top-k predictions can be ranked according to the overall prediction scores. While previous methods rank predictions without incorporating forward-prediction scores.

One major concern for me is that the authors present their methods and variants with abstract formulations, while implementation details are largely ignored. This makes their method difficult to understand. For example, line #101, p(Xi|X1:i-1, y) is parameterized by the Transformer model h(p, q), how does h(p, q) output a logits vector for each value in vocabulary? Does the formula p(Xi|X1:i-1, y) indicate the input is X1:i-1 and y, and output is Xi? Line 142, could the authors add more details to explain how KL divergence is related to the duality constraints?

Another concern for me is the preprocessing of the USPTO-50k dataset. It is found that there is a potential information leak with the dataset reported by the authors of RetroXpert. Have the authors removed the atom mapping and then canonicalized the product SMILES during the inference? Have the authors also run rxnmapper on testing data? It is not necessary to have mapping information for testing data, the authors should remove any mapping information for testing data and then run the inference. Are the authors aware of any potential information leak with the proposed method?

The authors claim that "we propose a unified energy-based model (EBM) framework that integrates sequence- and graph-based models for retrosynthesis." and I do not think this is a significant contribution. EBM is a general framework and it is not surprising that EBM can be used to explain different models, especially the graph-based method GLN itself is energy-based. 

The Transformer has been extensively explored for retrosynthesis, the authors did not clarify the difference between their different variants and existing Transformer-based methods. The proposed generalized sequence models at line #216 are similar to the RetroXpert, except that the synthons are directly predicted here. The ordered model at line #98 is similar to the one-stage Transformer methods such as SCROP.

Line #223, previous methods infer reaction center from provided atom-mapping information, which is extracted from third-party software like Indigo. The authors use rxnmapper to find the atom-mapping information. I do not think there is an essential difference.

What is the difference between formula (4) and (13)？

When incorporating templates, what is the difference between the proposed method and GLN except for the sequence and graph-based?

Could the authors also report results on USPTO-full as GLN to demonstrate the scalability? 

Line #250, the explanation of the superscript should be moved to line #173 when it first appears.

Line#121, the RHS of the equation in Algorithm 2 should have a minus symbol.

 

limitations_and_societal_impact: Not
review_title: null

Meta-review: This paper tries to include all the existing retrosynthesis algorithms into the framework of Energy-Based Model (EBM). Based the this, it derives a dual model to measure the energy of a pair (Reactants, Product), which depends on both the backward, forward energy function and their consistency. Experiments demonstrate this will improve the top-1 accuracy by 1.3~1.5%, compared with only using the backward model. 
Decision: Accept (Poster)