Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: - a benchmark for retrosynthesis is proposed
- a model for retrosynthesis is presented
- the authors claim state of the art

there are only limited novel contributions in this paper
strength_and_weaknesses: ### strengths
- clear description
- interesting memory model

### weaknesses:
- the paper has several incorrect definitions
- the paper yet proposes another benchmark for retrosynthesis even though there are already several existing ones
- the presented model is not properly compared
- the baselines in the paper are inconsistent with previous work
- somewhat limited ML model novelty
- some proposed reactions in the case study are chemically incorrect
clarity,_quality,_novelty_and_reproducibility: ### quality
- the paper has several incorrect definitions. 
  - in chemistry, a synthetic route and synthetic tree are the same 
  - the depth of a tree is not particularly important, what is more important is the total number of steps in the tree. with this definition, the authors are solving a different task previous work, and comparison to previous work is therefore not very meaningful, as a different metric was used
  - it is unclear why a transformer model is needed. RETROGRAPH https://arxiv.org/abs/2206.11477 shows a transformer is not needed for adding in contextual information in retrosynthesis search
- the molecular case study contains incorrect reactions, which casts doubt on the usefulness of the method in practice
- the results in table 1 are inconsistent with previous works (e.g. GLN should be stronger than neuralsym, single step transformer stronger than neuralsym and GLN). it is unclear whether the baselines have been tuned properly.
### clarity
good

### originality
1) The benchmark proposed in this paper is very close to the PAroutes  benchmark, which was already published in Feb 2022 https://chemrxiv.org/engage/chemrxiv/article-details/621c86f3c3e9da4f737b0047 - why do we need yet another benchmark?
2) The use of graph and contextual information has already been proposed in RETROGRAPH https://arxiv.org/abs/2206.11477 earlier in 2022 
summary_of_the_review: The proposed benchmark and approach are a small variation of previously existing work. Therefore this reviewer cannot justify to recommend  this work for publication in a top ML conference.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a transformer with memory of reaction-trees for a new multi-step-retrosynthesis benchmark.
strength_and_weaknesses: ## Strengths:
 - Propose a new Benchmark for Multi-Step-Retrosynthesis
 - Propose Metro: Memory-Enhanced Transformer for RetrOsynthetic planning by extending Transformer with an additional memory module
    - reaction route as context information
    - can control the synthesis path

## Weaknesses:
 - Missing Important Related Work: 
   - Section for Memory Enhanced Architectures, completely missing [1]
   - Missing Template-Based Methods in Related Works section [2] [3]
 - More clarification on why extracted Reaction-Trees are seen as optimal
 - No code for reproducibility is provided
 - Architecture needs further explanations, e.g. Decoder, Tokenizer, "Concatenated Embedding" + residual connection, ...
 - The proposed dataset is a subset of USPTO-Full, please provide a comparison e.g. a scatter plot, of performance differences between e.g. USPTO-Full results and the results obtained on your benchmark set

## Minor Comments
 - Was SMILES augmentation [4] performed or any other type of augmentation?
 - 5.1.1 paragraph about SMILES representation 2 might fit more in the subsequent paragraph 5.1.2, but is nothing inherent about the architecture
 - Please provide variance for point estimate in Table 1 as well as the significance
 - Is any standardization performed on molecules?
 - Please further describe the meaning behind A B C D E F in Figure 2
 - Appendix B.2.: Also describe the hyperparameter search space, as well as how they were obtained
 - Further discussion on why extracted Reaction-Trees are seen as optimal/better; e.g. there might exist faster/more optimal reaction routes

### References
 - [1] Ramsauer, H., Schäfl, B., Lehner, J., Seidl, P., Widrich, M., Adler, T., ... & Hochreiter, S. (2020). Hopfield networks is all you need. 
 - [2] Chen, S., & Jung, Y. (2021). Deep retrosynthetic reaction prediction using local reactivity and global attention. JACS Au, 1(10), 1612-1620.
 - [3] Seidl, P., Renz, P., Dyubankova, N., Neves, P., Verhoeven, J., Wegner, J. K., ... & Klambauer, G. (2021). Modern Hopfield Networks for Few-and Zero-Shot Reaction Prediction.
 - [4] Tetko, I. V., Karpov, P., Van Deursen, R., & Godin, G. (2020). State-of-the-art augmented NLP transformer models for direct and single-step retrosynthesis. Nature communications, 11(1), 1-11.
clarity,_quality,_novelty_and_reproducibility: ## Clarity
The paper is written quite well, but needs further details in aforementioned topices

## Quality
OK

## Novelty
The dataset is novel, as well as the application of a Memory Module of reaction-trees

## Reproducibility
No code provided
summary_of_the_review: The paper introduces a new interesting benchmark for multi-step retrosynthesis that might be influential in the community of retrosynthesis, however no code or dataset is provided.
The architecture seems interesting, but lacks connection to related work, as well as further clarifications.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is concerned about retrosynthetic planning, where the task is, given a target molecule, to find a reaction tree to synthesize the target molecule from a set of starting compounds. The proposed method called Metro is based on single-step retrosynthesis transformer, but is equipped with a memory module so that it can make use of the retrosynthetic route as context information for the purpose of predicting the reactants. Another contribution is yet another preprocessing of the USPTO dataset, resulting in a new benchmark. The resultant dataset consists of reaction trees extracted from the dataset, and its test dataset is much larger than that in previous benchmark tasks.

The authors conducted empirical studies using the proposed benchmark. In terms of top-$k$ accuracy scores, the proposed method achieves more than 10 points improvements over the transformer model (= Metro - memory module). The authors also confirm that the accuracy degrades as the depth of the reaction tree increases.
strength_and_weaknesses: ## Strength
- The experimental results indicate that the proposed method achieves better accuracy scores than baseline methods, including a transformer-based method. The result suggests that the proposed memory module contributes to the performance improvement, validating their approach.
- The authors propose a novel benchmark task considering the limitations of the existing one. In addition, since the details are well described in the paper, it will be not difficult to reproduce the task.

## Weakness
- __Why context information:__ Although the authors discuss why the context information helps in Section 5.3, I am not very convinced of it. The authors claim that the context information can prune the reaction search space, but as far as I understand, many of the existing methods utilize search algorithms for pruning. In fact, all of the methods compared in the experiments use the same DFS algorithm, which is not very fair for the existing methods.
- __Baseline methods do not cover the existing methods introduced in the introduction:__ Since many of the existing methods combine ML and planning algorithms, it is necessary to compare the proposed method with other methods utilizing a wide variety of search algorithms.
- __Presentation needs improvement:__ I had a bit difficulty in understanding the proposed algorithm, because the paper does not provide the whole algorithm (Algorithm 1 was helpful to understand it). In particular, although Figure 2 provides the overview of the proposed method, at first, it is difficult to understand why it has three inputs, A, B, and D. I would appreciate it if the authors could provide the whole algorithm first, and dive into the details after that. Similarly, although the authors pointed out three limitations in the third paragraph of the introduction, they came out suddenly, and I was not very motivated from them.
clarity,_quality,_novelty_and_reproducibility: ## Clarity
As elaborated in the weakness section, there is a room for improvement in terms of clarity.

## Quality
As elaborated in the weakness section, the use of context information is not well motivated to me. At least, it is necessary to compare Metro with existing methods using a variety of search algorithms.

## Novelty
As far as I am aware of, the preprocessing of the USPTO dataset is novel and therefore the proposed setting is novel.

## Reproducibility
Although the hyperparameters are provided in the appendix, the overall learning algorithm is not explicitly described, and it is difficult to identify the exact algorithm. I would suggest the authors to explicitly describe the learning algorithm and/or release the source code to further improve the reproducibility.
summary_of_the_review: I am not positive about accepting this paper, mainly because I am still not motivated to utilize context information. I understand that while it can bias the retrosynthetic process and can prune the search space, its advantage over the existing methods has not been clarified because of the selection of baseline methods. If there is any misunderstanding, I would appreciate it if the authors could point them out.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a new benchmark and transformer model for retrosynthesis prediction. The model particularly makes use of the context of the retrosynthesis routes and is hence rather different from existing ones. It does so by using a memory module and an extension of transformer, using matrices in place of the values (i.e., the value matrix is a tensor here).

strength_and_weaknesses: (+) It is true that existing route data is lacking, so the initiative to propose new data is good. Similarly the idea of including context information makes totally sense to me.

(+) The results look promising.

(-) From an ML perspective, the paper/model contains few novelty. 
 
(-) The paper proposes a new dataset as benchmark and one of the main contributions, but it does neither give much detail about the data nor does it provide a detailed comparison to existing works, apart from two sentences. There are recent datasets and benchmarks, e.g. [1], which should be mentioned and compared to. The related work section does not even contain a paragraph about datasets.

[1] Samuel Genheden and Esben Bjerrum, PaRoutes: towards a framework for benchmarking retrosynthesis route predictions, https://pubs.rsc.org/en/content/articlehtml/2022/dd/d2dd00015f

(-) The evaluation is lacking in that the comparison is with single-step models only. If additional (context) information is included, it is likely that the results will be better. It is interesting to see the magnitude of the improvement, however, to learn about the practical effectiveness I think other experiments are needed. Single-step models are usually applied within a planning algorithm. Since the authors focus on the routes, instead of single steps, I would suggest to evaluate the model also in a more realistic setting. For example, it would be interesting to see how close the proposed model (i.e., simple chaining of the transformer predictions) gets to Retro* with the MLP as single-step model, or how it performs when replacing the MLP by the transformer.

================================================================================

Minor Comments

- sounds wrong to me:
"For two trees ..., we think these are different trees." 
"Moreover, what needs to be declared is that molecule synthesis..."
- There are some typos in Algorithm 1
- Why do G2Gs and Graphretro perform that bad?
- What does this mean?: 
"2) the dataset is based on reaction route instead of reaction tree."
"Note that in this paper, the default order of the reaction route is the retrosynthesis order."
- The case study in the evaluation, to me it is unclear what it shows specifically, apart from the fact that the proposed model improves upon transformer.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. I think in terms of quality and novelty it is lacking (see details above). Code is not provided, yet the model description is clear and straightforward and one could reimplement it.
summary_of_the_review: The paper proposes a new benchmark and model for retrosynthesis prediction, yet it's missing an evaluation which would justify those. 
Altogether, I do not think that the paper can be accepted at the current stage and that it would need quite some revision.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
In this paper, the authors constructed a retrosynthesis benchmark, which includes the multi-step retrosynthesis path as the prediction target. The authors also proposed a memory-enhanced transformer model, which empirically achieve strong empirical performances. 

The major concerns raised by most of the reviewers lies in the following aspects:

- The novelty of the model. The model architecture is relatively straightforward. 

- The presentation of the paper is not clear, making the paper difficult to follow. 

- The literature is not fully covered, including the methods proposed in chemistry, which makes the paper is not well-positioned. 

- The experiments results are not well presented, which cause confusion and insufficient comparison impression from reviewers. 


I do believe the paper contains contribution to the community. However, the current version does not clearly emphasize the novelty and significance. I suggested the authors can consider the comments provided by the reviewers to improve the draft. 
justification_for_why_not_higher_score: 
Although the paper indeed contains new insight for retrosynthesis with new benchmarks, the current version of the paper does not demonstrate the significance of the contributions among the existing literature, which makes the paper less interesting. 

I believe more effort is needed to improve the paper.
Decision: Reject