Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed a new method for retrosynthesis, which does not require the mapping numbers and extracting templates from the literature.

Basically, the model consists of a graph-based encoder and a sequence based encoder. The encoder consists of local aggregation from neighbors and global attention using a new positional method. The decoder is a Transformer model with relative positional encoding.

The method achieved promising results on several retrosynthesis datasets.

main_review: 1.	From Eqn.(1) to Eqn.(5), you choose to use a complex gating mechanism to aggregate information. Is every component necessary? What if using a simple GCN or GAT?
2.	I think the model contains more parameters than conventional retrosynthesis models like conventional Transformer, GLN. Could you please show compare the number of parameters of different methods?
3.	The authors should provide some real cases to show how the method outperforms previous baselines, and why the method can obtain good results without templates.


Missing References:

1.	Dual-view Molecule Pre-training, https://arxiv.org/abs/2106.10234, the authors also work on retrosynthesis using Transformer and GNN models. A comparison is necessary.

summary_of_the_review: The results in this paper are good, although the method itself is not quite novel.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a graph-to-sequence architecture called Graph2SMILES for the retrosynthesis and the reaction outcome prediction. Graph2SMILES uses an attention-augmented D-MPNN encoder to capture the local information and a global attention encoder with graph-aware positional embeddings to capture the global information. Experiments show that Graph2SMILES is competitive with Transformer baselines but does not outperform state-of-the-art methods on tasks of the one-step retrosynthesis and the reaction outcome prediction. 
main_review: The main strengths of this paper are as follows.
1.	This paper proposes Graph2SMILES, which is a graph-to-sequence architecture without using sequence representations of input SMILES. Therefore, Graph2SMILES is permutation invariant to the input and does not need the data augmentation.
2.	Graph2SMILES has a wide range of potential applications because it can serve as a drop-in replacement for Transformer in many tasks involving the molecule(s)-to-molecule(s) transformation.

My major concerns are as follows.
1.	This paper states that Graph2SMILES achieves state-of-the-art top-1 accuracy on common benchmarks among methods that do not use reaction templates, atom mapping, pretraining, or data augmentation strategies. The authors claim that integrating the above features or techniques with Graph2SMILES could improve the performance. However, they do not conduct experiments to demonstrate their claim. Besides, as the aforementioned techniques are commonly seen in predictive chemistry tasks, the authors may want to explain why they do not equip Graph2SMILES with these techniques.
2.	D-GAT is a variant of D-GCN with attention-based message updates. However, D-GAT does not outperform D-GCN in terms of the top-1 accuracy, which is the basis for comparison throughout the discussion in this paper. According to Table 1, D-GAT has a small advantage over D-GCN only in terms of the top-5 and top-10 accuracies in the reaction outcome prediction.
3.	Graph2SMILES involves calculating pairwise shortest path lengths between atoms, which can be computationally prohibitive. The authors may want to compare Graph2SMILES against baselines in terms of the computational complexity.


summary_of_the_review: This paper studies two important problems in the computer-aided organic chemistry and proposes a graph-to-sequence architecture called Graph2SMILES. However, the empirical results do not show a superior performance of Graph2SMILES to existing methods, and the technical contribution is incremental.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a graph-to-SMILES framework, which incorporates several recently developed engineering techniques from the community, for synthesis planning and reaction outcome prediction tasks. The proposed method leverages graph neural networks and Transformer attention model to encode the graph inputs and then utilizes a Transformer decoder to generate the SMILES string as outputs. Experiments on benchmark retrosynthesis and reaction prediction tasks show that the proposed approach outperformed the vanilla SMILES-to-SMILES transformer baseline, but obtained inferior results than some other advanced methods. The paper is interesting, but both the technical novelty and the experimental studies are weak to me.
main_review: The proposed framework integrates several recently developed engineering techniques and empirically shows its superior performance over vanilla SMILES-to-SMILES transformer baseline. This paper provides another comparison baseline for research on retrosynthesis and reaction prediction. Nevertheless, I have the following concerns regarding the paper.

1.	The proposed framework is similar to the NERF approach (Bi et al. ICML 2021) as cited by the authors. NERF formulates the reaction prediction problem as a graph-to-graph translation problem. Also, NERF first leverages graph neural networks to capture the local information in individual molecules and then utilizes a Transformer encoder to further models the intermolecular interactions between nodes from multiple molecules. Furthermore, NERF uses a Transformer decoder to decode the output as graph. These are almost the same as that in the method proposed in the paper. The only different to me is that the NERF uses a Transformer to decode the output into graph directly (in a non-autoregressive fashion), while the proposed method here uses the Transformer to decode the output into SMILES strings (in an autoregressive fashion). In this sense, the novelty of this paper is limited to me. Note: I think NERF can naturally apply to two or more molecules since the Transformer encoder is used by considering all node embeddings from multiple molecules as a node set. 
2.	Experimentally, the proposed method is not directly compared with NERF. I think such comparison is necessary since as shown in the paper, NERF outperforms the SMILES-to-SMILES transformer baseline, and even the augmented version of it, to which the proposed method here obtained inferior performance. The two methods are similar and closely related. I would expect the paper to include NERF into the main results in Table 1. Also, for the USPTO_STEREO_mixed task, I wonder what the reason was for only comparing with the vanilla Transformer, and why not comparing with the Augmented Transformer or the state-of-the-art method Chemformer?
3.	Results in Table 1 show that, the proposed method is inferior to the Transformer baseline with simple augmentation. Also, 1% less than the tested method Chemformer, which makes the paper’s contribution less significant to me.
4.	The claim in the Abstract “molecular graph encoders that mitigates the need for input data augmentation” is a strong claim to me. Nevertheless, there is no evidence to support that claim. The slightly better performance over the SMILES-to-SMILES transformer baseline is not a convincing evidence to me. Input data augmentation may play a significant role on regularizing the deep neural networks. I think better justification to support the claim is necessary. 
5.	The statement in the last sentence of the first paragraph on Page2:  “[SMILES augmentation]…be interpreted as evidence of the ineffectiveness of the SMILES representation itself.” I think this hypothesis may need better support and analysis. To me, the augmentation of SMILES strings can act as a model regularization method, which helps the trained model to generalize well to unseen data, and may not directly infer the ineffectiveness of the SMILES representation itself. 
6.	I am not fully understand the claim in the second paragraph of Page2 “… we guarantee the permutation invariance of Graph2SMILES to the input, eliminating the need for input-side augmentation altogether.” I think it would be useful to specify how and why so.
7.	The proposed method integrates several performance engineering techniques (such as attention weights and multi-headed attention in the graph encoder, integration of shortest path length in the positional embeddings etc.), so where the improvement is really coming from is not clear to me. In the ablation study in Table4, both the positional embedding and global attention are key to the Transformer’s performance, so the performance degradation is expected when remove them: Transformer expects a positional embedding to work, and without a global attention the encoder will not be able to capture information from multiple molecules (their graphs are disconnected). 

summary_of_the_review: The proposed method is similar to NERF as proposed by Bi et al., so the technical novelty is limited. Also, the experiment study missed important comparison baselines. Furthermore, a more comprehensive ablation study is needed since several engineering techniques are employed, and it is difficult to tell where the performance improvement is really coming from when compared to the Transformer vanilla model.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a GNN-based extension of transformers, which have been shown to be effective for reaction prediction etc. before. In particular, the GNN-based embedding of molecules in the reaction embeddings overcomes the artificial bias inherent in the often applied sequence embeddings. The experiments show that there are sometimes increases of performance in reaction and retrosynthesis prediction - and the approach could be applied to similar problems.



main_review: (+)

Using a sequence-independent encoding in the easy-to-use transformer makes sense and is a research question which is interesting for the - I think mostly, AI in chemistry - community.

(-)

- As far as I can see, the technical novelty is limited. The proposal is a combination of rather well-known methods.

 D-MPNN (where attention is added) and

re-parameterized, relative positional encodings 

 (using 0 to represent atoms in different molecules)

in transformers

- It is unclear if the proposed attention in the GNN is useful: Since we only see "For reaction outcome prediction, there is a small advantage of using D-GAT over D-GCN", and the ablation results are missing.

- The results in Table 1 are only convincing for USPTO_STEREO_mixed.

- The experimental comparison for retrosynthesis compares to methods which use different forms of pretraining or augmentation, which makes sense. It shows that the augmentation still provides advantages. And, as far as I understand, the graph-based molecule embedding entails that Smiles augmentation would not improve your results further? Also, it would make sense to include the ablation results for Graph2Smiles just using the transformer into Table 2 directly. 


-------------------------------
Other Comments


- It is unclear to me what "Our hyperparameters for D-GAT and D-GCN are adapted from GraphRetro" means.
- In (8), s_uv should be AttnSum(...)
- How exactly is \mathcal{B}_u,v used in the learnable \tilde{r}_u,v? 
- Table 4: 
Since the paper proposes the attention-based GNN, the ablation should be provided for that model.
- Table 4: What is "no global attention encoder"? Just the combination of GNN embeddings w/o transformer? Since transformer is the baseline, I would not consider this as an ablation setting.
- Table 4: How do the results look on the other tasks? For retrosynthesis, all open existing systems that yield full retrosynthesis trees which I know use top50 (or similar), so the fact that the base model is better at top10 (already) renders the analysis questionable.
- "We include part of our code to reproduce some speciﬁc results" - Why not all?

summary_of_the_review: Overall, I think the authors' proposed model for reaction prediction makes sense. However, as mentioned above, the paper's writing could be improved, the technical contribution is limited, and the experiments also show only limited improvements. Altogether, I therefore suggest to reject the paper at the current moment. I am happy to adjust my score in case I missed critical parts.
review_title: null

Meta-review: While the reviewers appreciated the method's ability to replace transformer models and SMILES data augmentation their main concerns were with (a) the experimental section, and (b) the technical innovation over prior work, which updated drafts of the paper did not fully resolve. Specifically for (a) this work performs very similarly to prior work: for reaction outcome prediction the proposed method improves top-1/3/5 for USPTO_STEREO_mixed but is outperformed by prior work for top-1/5/10 for USPTO_460k_mixed; for retrosynthesis the model is outperformed for USPTO_full and only outperforms prior work that does not use templates/atom-mapping/augmentation for top-1 on USPTO_50k. The authors argue that their method should be preferred because their method does not require templates, atom-mapping, and data augmentation. The reviewers agree that template-free and atom-mapping-free methods are more widely applicable. However, the benefits of being augmentation-free is not convincingly stated by the authors who only state that their approach is beneficial by "simplifying data preprocessing and potentially saving training time." The authors should have empirically verified these claim by reporting training time, because it is not obvious that their model which requires pairwise shortest path lengths is actually faster to train.
For (b) the reviewers believed that the paper lacked technical novelty given recent work (e.g., NERF). The authors should more clearly distinguish this work from past work (e.g., graphical depictions and finer past work categorization may help with this).
Given the similar performance to prior work, the lack of evidence to support training time claims, and the limited technical novelty, I believe this work should be rejected at this time. Once these things are clarified this paper will be improved.
Decision: Reject