Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper poses an approach to retrosynthesis that addresses the challenges of (i) availability of reactants and (ii) generalization to unseen templates. To achieve this the authors reformulate retrosynthesis as the selection of reactants from a fixed set, in the case of the USPTO database this is the set of 671,578 commercially available reactants used in the database. Their reactant selection framework RetCL uses GNNs to calculate selection scores for all candidate molecules. A novel contrastive training scheme is used to learn the selection score, which is computed as the cosine similarity between embeddings of the product and the reactants computed by the graph neural networks. The authors provide a good summary of related work in this area. 

My major questions for the authors of this paper concern the limitation imposed by restricting to a specific candidate set. For example, the results reported on USPTO-50k are very impressive, particularly the ability to generalize to heldout reaction types. However, the model is never challenged by reactions for which the reactants are not present in the candidate set, and it is completely unclear how the model would perform in this scenario. This is important, because to tackle the chemically relevant retrosynthesis problem it is exactly necessary to solve reactions where the reactants may not be present in the 671,518 reactants present in USPTO-50k. 

I find it a bit surprising that the authors do not address this constraint in the main text, and I would ask that they carry out experiments where they restrict the candidate set to a subset of the 671,518 reactants yet consider reactions from the whole database, and ask how severe this restriction is in terms of the solutions obtained by the model (it would be necessary to build these splits carefully, and provide results for multiple splits). There are other arguments that the authors could also make to defend this point in addition to these experiments - but the current state in which the issue is ignored is not satisfactory. 

The description of the contrastive training approach is clear and coherent. The addition of hard negatives to the batches to improve learning is interesting, and the results of the ablation study speak to the important role that it plays. 

review_title: How does this extend to the real world task?

final_score: rating: 4: Ok but not good enough - rejection
review_content: RETCL enumerates all of the candidate molecules (all US patent dataset, all 671k) based on selection scores computed by graph neural networks. The cosine similarity between products and reactants are used to design scores, which is later used for training. The way of ‘cosine similarity to bridge reactants to products are interesting. 



Q1:  Used test data as selection set during training 

Training using all US patent dataset (671k) as candidates for RETCL to select from leaked test data (USPTO50K test data) during training. All US patent dataset is a superset of the USPTO50K data.  “Selection based algorithm” tends to achieve overly optimistic results due to this reason. 

Q2: How to generalize? 
It is great the paper shows that RETCL generalizes well to unseen templates
However, if we select upon an existing dataset (though very large). How does RETCL possibly yield totally unseen reactants (i.e. not in any existing dataset)? 


Q3 Approximation of C
Sec 2.3 
Computing p(R|P, Rgiven, C)  and q(P|R, C)  requires summing over all candidates in C, which is necessary for proper probability, but computational expensive. 
How does these probabilities be approximated by a mini-batch of reactions? 
The batch statistics are very different from the C (all US patent dataset, all 671k)




review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: ### Summary of the paper
This paper proposes a sequential reactant selection scheme for retrosynthesis. In each step, the model gives a ranking of reactants based on previously chosen reactants $R_{given}$. After all the reactants are selected, the model checks whether the chosen reactants result in desired product. The ranking module $\psi$ is trained via contrastive learning. The negative reactant candidates are constrained to be similar to the positive reactant.

### Strength and weakness
1. The method proposes a selection based approach to address one of the weaknesses of template-free approaches -- the predicted reactants may be commercially unavailable. This is indeed an important issue that needs to be addressed. However, I am afraid the proposed approach is an overkill. If we only select reactants appeared in the USPTO database, the model cannot generalize to new reactions which involves new reactants not in the USPTO database. This is problematic for two reasons:
 1. For instance, if the model is evaluated on a harder test set where ground truth reactants are not in USPTO database, I think the model will fail (with 0% top-1 accuracy). 
 2. Moreover, in multi-step retrosynthesis, you are allowed to make new intermediate compounds from commercially available compounds in order to make your final product. I don't see why we have to choose reactants only from commercially available compounds. 
2. Scalability: The total number of commercially available compounds are up to $10^9$ (e.g., Enamine REAL database). I am concerned that the neural network based ranking will run very slowly and cannot scale to larger sets of compounds. 
3. In section 3.4, authors evaluate their approach on a harder test set with novel reaction templates. In my opinion, template-free approaches are also capable of generalizing to novel reaction templates. Why there is no comparison to G2G or transformer in table 5?
4. The proposed method is quite straightforward, with limited technical novelty in my opinion. I am afraid the contrastive learning part is just a straightforward application of negative sampling.
5. The result looks very strong on the USPTO-50K test set. 

### Overall evaluation
I vote for weak reject mainly for two reasons.
1. The method seems incapable of generalizing to new reactant compounds outside of the commercial library. 
2. The approach has limited novelty to ICLR audience.

### Post rebuttal
I would like to thank the authors for their valuable response. The experimental results seem strong, but technical novelty is still limited. My review score remains the same. I believe this paper can make great impact if submitted to a chemistry journal. 

### Suggestions
Despite my negative evaluation, I do appreciate the point authors are trying to address. I think the paper can be significantly strengthened if you can loosen the "commercially available" constraint. For instance, you can loosen the constraint to "synthesizable" -- choosing the reactants that can be synthesized from a given set of building blocks (via reactions)! In multi-step retrosynthesis, you are allowed to make new compounds from your building blocks for the sake of making your final products.
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This submission describes an approach to single-step retrosynthesis based on contrastive learning that selects reactants that can be used to synthesize a target product in a single step. The stated contributions are (1) an approach to retrosynthesis that is constrained to only select “available” starting materials; and (2) a novel contrastive learning scheme with hard negative mining. The use of a contrastive loss to learn an embedding of reactants and their products that exhibit the property that the sum of reactant vectors has a high cosine similarity to the product vector is clever. This is a nice way to give structure to a continuous vector space. The strategy of hard negative mining is also clever and identifies the examples that would intuitively be the most informative.

I take some issue with the premise of constraining retrosynthetic recommendations to an enumerated list. This works for small corpora like the ones used here, but in reality, retrosynthesis is a multi-step process where most reactants are *not* commercially available and the one-step retrosynthetic expansion must be repeated recursively. This approach is fundamentally unable to operate on reaction products where multiple synthetic steps are required, which represent the challenging cases. That is why in the multi-step evaluations, the authors rely on the Transformer model to propose intermediate structures. Appendix D also suggests that in the pathway search experiments, knowledge of the routes in advance was required to construct the set of all starting materials to select from.

The empirical evaluation, as a result of the premise, is somewhat flawed. By constraining reactant proposals to an enumerated list of reactants extracted from a parent database, the authors have simplified the task in comparison to previous approaches, making a head-to-head comparison of accuracy less informative. The evaluation in 3.4 generalizing to unseen templates could simply be an indication that the model is learning atom conservation (excepting leaving groups) and to maximize substructure overlap with the products. The model has access to a restricted list of possible starting materials of which very few are likely to be plausible precursors for a given product. This advantage invalidates the comparison in Table 5; this is not evidence of generalization, since the test set answers were included in the set of candidates.

While the contrastive learning approach is clever, this work uses a contrived formulation for retrosynthesis that is not applicable to multi-step planning and the experiments do not support the conclusions drawn.
review_title: Clever contrastive learning, but the formulation and experiments are problematic

Meta-review: While the authors appreciated the proposed contrastive training scheme and the strong related work summary, all authors agreed that the approach was severeley limited by being a pure selection-based method. Without the help of another model that proposes molecules, the approach can only select reactants from an existing set. As target molecules become more complicated, the modeller must make a choice: (a) use a much larger initial candidate set which hopefully encompases all molecules necessary to make the target molecule, or (b) use another model to propose new intermediate molecules. The authors went with (b) which harmed their novelty claim: a big reason why retrosynthesis is hard is because of the need to generate unseen molecules, and if this is left to an already proposed model, the current approach is not adding much methodological novelty. While their approach does improve upon existing work in the multi-step setting, there's even more recent work that has not been compared against (e.g., https://arxiv.org/pdf/2006.07038.pdf) so the improved performance may be outperformed.

The fix is straightforward: modify the methodology to also propose intermediate molecules. This will fix the novelty complaint and strengthen the practicality argument: practitioners could directly use this approach to discover synthesis routes. The authors could slightly update the related work, add comparisons against recent methods, and take into account the other feedback given by the authors. The paper is very nicely written, the proposed changes are purely methodological, and not insurmountable in my opinion. I would urge the authors to make these changes which I believe will result in a very nice paper.
Decision: Reject