Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: SUMMARY

This paper uses the statistical physics-inspired energy-based model formalism to study the by now "canonical" problem of retrosynthesis using deep learning. The authors use an interesting variant that combines forward and backward prediction. The authors use template-based and template-free models.


PROS

- This reviewer believes that energy-based models have an elegance and connection to statistical mechanics that should be explored more in the area of machine learning. This work goes in this interesting direction.
- Based on the above, and as far as the reviewer is appraised, this is a unique, non-derivative direction in the field and therefore deserving of consideration for acceptance.
- The dual model seems to be very useful given the increase in template-based and non-template-based model performance. This could be applied to other transformer-based tasks in chemistry and graph-based ML
- The authors compared their models to a variety of SOTA models and approaches, they also were thorough and explored both DeepSMILES and SELFIES.


CONS
- Some of the mathematical formalism could be moved to supplementary to allow for better discussion.


MINOR FORMATTING
- The authors may want to give the manuscript a pass for grammar. There are missing articles in a few sentences.

review_title: Review of Energy-Based view of Retrosynthesis

final_score: rating: 5: Marginally below acceptance threshold
review_content: ### Summary of the paper
This paper proposes an energy based model (EBM) for retrosynthesis. The best model (dual model) leverages the duality of retrosynthesis and reaction prediction. The EBM contains three factors: prior on reactants $p(X)$, forward reaction probability $p(y | X) and backward posterior $P(X|y)$. The duality loss penalizes the KL divergence of the two directions. The forward predictor is trained on a mixture of original data and samples drawn from the backward predictor. The dual model shows improvement over template-based and template-free baselines.

### Strength
1. The dual model is novel and interesting. EBM provides a unified view of forward and backward reaction prediction. It will be interesting to see how this improves the forward prediction performance.
2. EBM is a flexible framework, which can be applied to both template-based and template-free approaches.

### Weakness (and questions)
1. Clarity: The paper describes EBM view of many methods, ranging from sequence and graph based methods. As a result, each subsection is too sketchy and lots of details are missing. For example:
 1. For Dual-TB (ours), what is the exact model architecture? I understand that the candidates are generated from reaction templates, but what's the parametrization of $p_\alpha, p_\gamma, p_\eta$? Is it GLN?
 2. For Dual-TF (ours), what is the model architecture? Is it transformer?
 3. What is the augmented USPTO 50k? Is it randomized SMILES? For Dual-TB (ours), is it trained on augmented USPTO 50k? Why SMILES randomization matters for templated based methods?
 4. $X$ is a set of compounds, how do you generate a set of compounds in an order-invariant manner?
 5. For duality constraint, you sample from backward predictor using beam search. Why not the other way around? Why not sample from forward predictor and train your backward predictor on the additional samples?

2. At inference time, it is hard to sample from EBM. Therefore, authors propose to rank the candidates generated from other models (reaction templates or transformers). This really limits the performance (and applicability) of the approach. To my knowledge, you can do MCMC based on Langevin dynamics to sample from EBM. Is this not possible for retrosynthesis?

3. The best model is this paper actually performs much worse than the state-of-the-art models. For instance, RetroXpert (Yan et al., NeurIPS 2020) achieves 65.6% top-1 accuracy (reaction type unknown) and 70.4% top-1 accuracy (reaction type known). This is much better than the dual model (55.2%, 67.7%). Somnath et al, 2020 also achieves 64.2% top-1 accuracy (reaction type unknown), which is much higher than dual model (55.2%).

### Overall evaluation
I vote for weak reject of the paper, primarily due to the weaker result and lack of clarity. I believe the dual formulation can be applied to these above state-of-the-art models (if code is available). I suspect the weak result is primarily due to the base model (e.g., transformer). I am happy to adjust my score if there are stronger results and the clarity can be improved. One suggestion for clarity is to put perturbed / bidirectional models into appendix since they are not helpful anyway... 

### Post Rebuttal
I would like to thank authors for their response. I think the paper needs to be improved further to get accepted. I do believe that the proposed dual learning method is promising, but empirical evaluation is still lacking. So my review score stays the same.

[1] Yan et al., RetroXpert: Decompose Retrosynthesis Prediction like A Chemist, NeurIPS 2020.

[2] Somnath et al., Learning Graph Models for Template-Free Retrosynthesis, 2020
review_title: Clarity needs to be improved

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors use the framework of energy based models to describe several known approaches for ML-based retrosynthesis in a unified way. This allows to combine retrosynthetic (backward) and reaction prediction (forward) in a principled way.  
Based on this analysis, a dual model is proposed which used a duality constraint as a regulariser, leading to improved performance over several baselines models, but not over SOTA (see below!).

The theoretical analysis alone is very interesting, and well described. However, I have a few concerns with missing ablation experiments and the positioning, which could be addressed to make the paper stronger in my opinion.

Overall, I think this paper should be accepted at ICLR after the following points have been addressed (or the authors commit to provide the additional data in the camera ready version if the time of the rebuttal phase is too short to run additional experiments). However, in the current form, the paper is not ready. My evaluation is for the current form of the paper, and I am happy to change it significantly during the rebuttal.


Ablation Experiments:

Is the duality constraint actually needed? It would be important to perform the ablation experiment where $\beta$ is set to 0.
In other words, what happens if you just use $\log p(X|y) + \log  p(y|X)$ to rank the candidates? If the authors perform these additional experiments and report the numbers, I will increase my score regardless of the outcome of the experiments.


Prior work:

The state of the art claim is not correct. Yan et al achieve higher performance on the same dataset https://chemrxiv.org/articles/preprint/Interpretable_Retrosynthesis_Prediction_in_Two_Steps/11869692/2 which has already been published last February! 
Therefore, please remove the SOTA claim from the paper, and acknowledge the Yan et al work. 
For this reviewer well-motivated modelling, honest analysis and proper experimentation is more important than chasing SOTA, and not achieving SOTA will not affect the evaluation negatively.

Also, the connection of forward reaction and retrosynthesis prediction via Bayes Theorem has been studied here https://arxiv.org/abs/2003.03190 which should be acknowledged.

Other work preceding has also used a combination of backward and forward prediction. For example, Segler et al Nature 2018 and Coley et al Science 2019 use a model for p(X|y) to propose disconnections combined with a model for p(X,y) to remove low probability solution.

The usefulness of SMILES augmentation has been previously shown, please cite the previous work by Arus Pous et al https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0393-0 

The Coley et al 2017a paper that the authors cite in the introduction is a wonderful paper, however, it is not concerned with retrosynthesis. I would suggest to cite the review article by Strieth-Kalthoff et al https://doi.org/10.1039/C9CS00786E instead, which provides a good overview over ML approaches for (retro)synthesis. Furthermore, I would suggest to cite Segler & Waller 2017 in the introduction, which was the first paper to suggest deep neural networks for both retrosynthesis & reaction prediction (see also https://doi.org/10.1016/j.ddtec.2020.06.002 )
Coley 2017a should be cited later in the context of forward prediction.

Comments:

Table 3 is a bit unclear. With Type, do you mean the reaction type is given?

small things:
- in eq 13, the is a low dot, should this have been a centred dot for multiplication?

- Please don't use Google house fonts for your figures,  to maintain anonymity.


review_title: Interesting analysis of the problem of retrosynthesis using EBMs

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Summary #
This paper introduces a re-interpretation of seq2seq and graph2graph retrosynthesis models with energy based models (EBMs). 
EBM is a general log-linear framework to model joint distributions of a feature variable X and a target variable y, first introduced to connect discriminative (DNN) models with generative (DNN) models. This paper shows the reformulations of typical seq2seq retrosynthesis models and graph2graph models. Also, the paper proposes a dual training model that optimizes both the forward path model and the backward path model. 
Experimental results show that the dual training models perform better than conventional retrosynthesis models in template-based and template-free retrosynthesis frameworks. 

# Comments #

I have two major concerns. 

First, I cannot clearly understand a new insight or knowledge that is brought by the EBM-based re-formulation of retrosynthesis models. 
Any probabilistic models can be described by the full joint distribution of all variables. In my understanding, the EBM is a way of modeling the joints with the log-linear model plus the potential function. 
It seems for me that the current manuscript is successful in re-wiring the existing retrosynthesis frameworks into EBMs, but that is all. 
Please clarify what readers can learn about "connections and ... differences between models, ...understanding of model design [abstract]" from the EMB modelings. 

Second, the State-of-the-Art of the retrosynthesis in the manuscript is somewhat outdated. 
To the best of my knowledge, the current best-performing retrosynthesis models are [1] and [2], and the scores of these models are higher than the reported results of the proposed dual models. 

I think the current manuscript needs modifications to appropriately cite these papers and replace the standpoint of the submitted work. 

Personally, I am interested in how the EMB can interpret these latest models. 


[1] Somnath+, "Learning graph models for template-free retrosynthesis", arXiv:2006.07038, June 2020. 
[2] Yan+, "RetroXpert: Decompose retrosynthesis prediction like a chemist", chemrxiv:11869692.v3, June 2020. 

# Evaluation points #

(+) First to apply the EBM for retrosynthesis prediction models

(+) Proposed a forward/backward simultaneous (dual) training

(--) It is unclear what we can learn by rewriting the retrosynthesis models with EBM. 

(-) Reported results do not update the State-of-the-Art prediction accuracy of retrosynthesis models in the literature

review_title: ICLR 2021 Conference Paper2550 AnonReviewer3

Meta-review: Before the discussion phase nearly all reviewers had doubts about the comparison of the current work with state-of-the-art works (notably Yan et al., 2020, RetroXpert, and GraphRETRO). The authors then compared with these works and emphasized that these works rely on hand-crafted features. They argue that the fairest comparison is the one where each method uses the same sort of features during train/test time. This is because in certain real world settings we may not have accurate estimates of such features (e.g., atom mappings, templates, reaction centers). However, in the revised version of the paper the authors did not adhere to this concept of fair comparison in Table 4 of Appendix A.4. Here their method uses reaction centers as input while baselines do not. While the authors claimed that the comparison here was designed to show how reaction centers provided as input improved performance, this doesn't seem like a good way to show it: to isolate the improvement due to reaction center inputs you should fix everything else, i.e., the rest of the method. 

Apart from the above contradiction, I buy the arguments of reviewers that distinguishing between methods that use hand-crafted features and those that do not is not a meaningful distinction. One can apply atom-mapping or reaction center discovery algorithms as data preprocessing before applying other methods. Ablation studies where such preprocessing is added or removed are interesting, but it is completely fair for any method to use such preprocessing before applying their method, it is up to the modeller. 

I would have argued for acceptance had the authors either (a) just included results from SOTA methods (one, RetroXpert was published 1 month after the ICLR submission deadline), and/or (b) reran their approach with such preprocessing. However the authors ended up hurting the submission by emphasizing a difference between using handcrafted features and not, then contradicting their experimental setup in Table 4.

This is a good paper, but I agree it is not ready to be accepted at ICLR. I recommend the authors do the following: (a) use any preprocessing they want for their method and compare with the state-of-the-art, (b) if they want they can run their method without any preprocessing as an interesting ablation study, (c) remove Table 4 (as (b) already does this type of an ablation study), (d) describe recent work through the lense of EBM, (e) resubmit to a strong ML conference. The new submission will be much stronger.
Decision: Reject