Permutation invariant graph-to-sequence model for template-free retrosynthesis and reaction prediction
Authors: Zhengkai Tu
Connor W. Coley
Venue: ICLR 2022 Submitted (2022)
Keywords: retrosynthesis
reaction prediction
graph neural network
Transformer
positional embedding

Abstract: Synthesis planning and reaction outcome prediction are two fundamental problems in computer-aided organic chemistry for which a variety of data-driven approaches have emerged. Natural language approaches that model each problem as a SMILES-to-SMILES translation lead to a simple end-to-end formulation, reduce the need for data preprocessing, and enable the use of well-optimized machine translation model architectures. However, SMILES representations are not an efficient representation for capturing information about molecular structure, as evidenced by the success of SMILES augmentation to boost empirical performance. Here, we describe a novel Graph2SMILES model that combines the power of Transformer models for text generation with the permutation invariance of molecular graph encoders. As an end-to-end architecture, Graph2SMILES can be used as a drop-in replacement for the Transformer in any task involving molecule(s)-to-molecule(s) transformations. In our encoder, an attention-augmented directed message passing neural network (D-MPNN) captures local chemical environments, and the global attention encoder allows for long-range and intermolecular interactions, enhanced by graph-aware positional embedding.  Graph2SMILES improves the top-1 accuracy of the Transformer baselines by $1.7\%$ and $1.9\%$ for reaction outcome prediction on USPTO_480k and USPTO_STEREO datasets respectively, and by $9.8\%$ for one-step retrosynthesis on the USPTO_50k dataset.

Under review as a conference paper at ICLR 2022
PERMUTATION INVARIANT GRAPH-TO-SEQUENCE MODEL FOR TEMPLATE-FREB RETROSYNTHESIS AND REACTION PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Synthesis planning and reaction outcome prediction are two fundamental prob- lems in computer-aided organic chemistry for which a variety of data-driven ap- proaches have emerged. Natural language approaches that model each problem a SMILES-to-SMILLES translation lead to simple end-to-end formulation, reduce the need for data preprocessing, and enable the use of well-optimized machine translation model architectures. However, SMILES representations are not an ef- ficient representation for capturing information about molecular structures, idenced by the success of SMILES augmentation to boost empirical performance. Here, describe a novel Graph2SMILES model that combines the power of Transformer models for text generation with the permutation invariance of molec- ular graph encoders that mitigates the need for input data augmentation. As an end-to-end architecture, Graph2SMILES can be used as drop-in replacement for the Transformer any task involving Molecule((-to-mmoleeleee tions. In our encoder, an network (D-MPNN) captures local chemical environments, and the global atten- tion encoder allows for long-range and intermolecular interactions, enhanced by graph-aware positional embedding. Graph2SMILES improves the top-1 accuracy of the Transformer baselines by 1.7% and 1.9% for reaction outcome prediction on USPTO 480k -80k and USPTO USPTO_STEREO one-step retrosynthesis on the USPTO_50K dataset.
transforma- directed message passing neural
STEREO datasets respectively, and by 9.8% for
INTRODUCTION
Retrosynthetic analysis (Corey) 1988 Corey & Cheng. 1989) and its reverse problem, reaction out- come prediction (Corey & Wipke 1969), are two fundamental problems in computer-aided organic synthesis. The former tries to propose possible reaction precursors given the desirable product, whereas the latter aims to predict the major products given reactants. Historically, they were tackled using rule-based expert systems such as LHASA (Corey et al. 1972). Recent developments in ma- chine learning have led number of template- based, graph edit-based, and translation-based methods, for which we give detailed review in Section [.1] For both tasks, translation- proaches have grown popular, possibly because the end-to-end formulation is procedurally simple. Since most organic molecules can be represented as SMILES strings (Neglur et al.) (2005), retrosyn- thesis can be cast as translation from product SMILES to reactant SMILES (Liu et al. 2017), and so can reaction outcome prediction (Nam & Kim 2016). Modeling these tasks as machine trans- lation problems enables the use of neural architectures that are well-studied and well-optimized in the field of Natural Language Processing (NLP). Several of the best performing models across mul- tiple benchmark datasets TTetko et al.) 2020 Trwin et al. 2021 Sun al.] 2020 Seo et al.) 202T] Wang et al., 2021b) have used the Transformer architecture (Vaswani et al. 2017 as the backbone on SMILES representations, showing the effectiveness of translation based formulation.
ap-
However, SMILES representations do not provide bijective mappings to molecular structure. As result, data augmentation with chemically-equivalee SMILES (Bjerrum 2017) has become a com- mon practice to improve empirical performance. Augmenting the training data with just equivalent reaction SMILES by permuting both the inputs and the outputs can already produce noticeable im- provements of 0.8% 4.3% (Schwaller et al. 2019 Seo al. 2021). Incorporating (Wang al.

---

Under review as a conference paper at ICLR 2022
2021b), up 100 (Tetko et al. 2020)
equivalent SMILES provide additional gains. While these efforts demonstrate the effectiveness of SMILES augmentation. this introduces a non-trivial choice to how much augmentation should be done. The performance has not saturated even with 100 augmented SMILES (Terko et al. 2020). but this may significantly complicate the model pipeline, especially during test time which typically necessitates de-duplication, ensembling, and heuristic-based scoring (Wang et al., 2021b Tetko et al. (020)).
In this paper, we propose a novel graph-to-sequeence architecture called Graph2SMILES solve the tasks of retrosynthesis and reaction prediction. We first design a sequential graph encoder with an ie-atteetion-aee aughenteed directed message passing neural network (D-MPNN) based on Yang et al. 2019) and optionally augmented by attention- based message updates, followed by a Transformer- based global attention encoder with graph-aware positional embedding. We then pair the graph encoder with Transformer decoder to transform molecular graph inputs into SMILES outputs, without using sequence representations of input SMILES at all. As such, we guarantee the permu- tation invariance of Graph2SMILES to the input, eliminating the need for input-side augmentation altogether. Our main contributions can summarized as follows:
1. We propose Graph2SMILES architecture with two encoder components modeling local and global atomic interactions respectively, to address forward prediction and one-step retrosynthesis as graph-to-sequence tasks.
2. We design graph-aware positional embedding to further enhance performance. It is easily generalizable to graphs containing two or more molecules, while not requiring any pre- training or joint training with auxiliary tasks.
3. We demonstrate the adequacy of graph representations alone by showing that Graph2SMILES outperforms Transformer baselines on predictive chemistry tasks without needing any input-side SMILES augmentation.
Our Graph2SMILES architecture achieves state-of-the-arr top-1 accuracy on common benchmarks among methods that do not make use of reaction templates, atom mapping, pretraining, data augmentation strategies. We emphasize that Graph2SMILEES is a backbone architecture, and hence drop-in replacement for the Transformer model. As such, Graph2SMILES can be plugged into any method for molecular transformation tasks that uses the Transformer model, while retraining the benefits from techniques or features orthogonal to the architecture itself.
2
METHODS
2.1 GRAPH AND SEQUENCE REPRESENTATIONS OF MOLECULES
There are multiple ways representing molecular structures, such as by molecular fingerprints (Rogers& & Hahn 2010), by SMILES strings (Weininger] 1988), or molecular graphs with atoms as nodes and bonds as edges. We represent the input molecule(s) as graphs, and the output molecule(s) as SMILES strings, thus modeling both reaction prediction and retrosynthesis transformations.
sraph-to-sqqennee
Formally, let 9in denotes the molecular graph input, which can contain multiple subgraphs for dif- ferent molecules, with a total of N atoms. Following the convention in Sommath et al. (2020), we describe Gin (V, E) with atoms V and bonds E. Each atom and each directed bond (u, v) E & from atom to v has its feature vector as Lu. E RR The de- tails of the atom and bond features used can be found in Appendix A We build the input molecular graphs from their SMILES strings with RDKit (Landrum, (2016). Note that all feature vectors are invariant the order of atoms and bonds, as well as how the original SMILES strings are written. We represent the output as sequence of SMILES tokens Spat (81, 82,..... ). where the tokens { are obtained from the canonical SMILES using the regex tokenizer in Schwaller et (209))
has feature vector x., R²,
2.2 GRAPH2SMILES
The Graph2SMILES model is variant of the encoder-decoder model (Cho al. (014) commonly used for machine translation. Figure [ displays the architecture of Graph2SMILES with the per- mutation invariant graph encoding process shown within the blue dashed box. We replace the
2

---

Under review as a conference paper at ICLR 2022
Permutation inveriant
Autoregrestive decoding
Molecular graphs
graph encoding
features
BrCetcaneel [cckereccl.Oddcceeoj444/4e
Source SMILES
ROME SMILES reader
ROKR featurization
Globa
Encoder-dacoder attention
Decoder
Torget SMILES
representations
Atom representations (with both local contexts)
representations
encoder
Helper
Output
011
117
features
Global
GGN
atti..
2ox
15407
012...
100.
Compute shortest
219
Embuddéngg lookup
1
path lengths
12111
Graph-aware
Transformrr
Adjacency
Patrwise distance
postiioaa embedding
decoder
Figure 1: Model architecture for Graph2SMILES, Top: the overall flowchart. Bottom left: details of permutation invariant graph encoding. Bottom right: details of autoregressive decoding.
encoder part of the standard Transformer model (Vaswani et al. 2017) used in Molecular Trans- former (Schwaller et al. 2019) with a atventin augmented directed message passing encoder, followed by global attention encoder with carefully designed graph-aware positional embedding. Each module has its intuitive function: the D-MPNN captures the local chemical context, the global attention encoder allows for global-level information exchange, and the graph-aware positional em- bedding enables the attention encoder to make use of topological information more explicitly. The permutation invariant encoding process eliminates the need for SMILES augmentation for the input side altogether, simplifying data preprocessing and potentially saving training time.
2.2.1 ATTENTION AUGMENTED DIRECTED MESSAGE PASSING ENCODER
The first module of the graph encoder is D-MPNN (Yngg et al.
2019
with Gated Recurrent
Units (GRUs) (Cho et al. 2014) used for message updates (Jin et al. 2018 Unlike atom-oriented message updates in edge-aware MPNNs (Hu et al.
Somnath et al. 2020)
2020 Yan et al.
2020
Mao et al. 2021 Wang et al. 2021a), updates in D-MPNN are oriented towards directed bonds to prevent totters, messages being passed mack-and-fortd between neighbors (Mahé et al. 2004
Yang et al. 2019). We optionally augment the D-MPNN with attention based message updates inspired from Graph Attention Network (Veličković et al., 2018) Brody et al. (202T) We term this variant as Directed Graph Attention Network (D-GAT) and refer to the original D-MPNN variant used in Somnath al.) 2020) Directed Graph Convolutional Network (D-GCN), while keeping both D-GAT and D-GCN as possible design choices for the D-MPNN.
For D-GAT, at each message passing step f, the message m [+1 associated with each directed bond (u, v) E 8 is updated using
$ Suu AttnSum Eus Bavi (m' (m wEN(())))
(1)
Zue (W: [Iui Fuvi Suv]
bz)
(2)
Tas (W, (Eni Puni 8av]
+ by)
(3)
May tanh (W [awi Duu]
+ U ruo + b)
(4)
m up (1 Zuv) Suv + Zuv C Max
(5)
where W. W,, W. U and b,, br, b are the learnable weights and biases respectively. is the sigmoid function, "." indicates concatenation, and O is the element-wise product. AttnSum the attention-based message aggregation defined
3

---

Under review as a conference paper at ICLR 2022
Cwu = a LeakyReL.U (Weh [Dui Dwos minu + bytk)
(6)
(Ew))
ayn N(EN( exp (Ewn)
(7)
AttnSum
Tus Zuns (minnu) (minuweNNurd wEN(uN))
Σ
awn (W, (W,min + b.)
=EN(u))
where W byk. a are the learnable parameters for the attention scores, and We. by are the param- eters for the value vectors. We have omitted a self-loop in message aggregation, as did have any noticeable effect performance from our experiments. In Eqn 3 we simplify the reset gate to be shared for all incoming edges, in contrast to (Somnath al. (2020) where separate reset gate defined for each edge.
Lastly, after T iterations, we obtaín the atom representations h. with similar attention- aggre- gation (with different parameters) over the bond messages coming into each atom followed by single output layer with weight W, and GELU activation (Hendrycks & Gimpel (016).
m. AttnSum'
Lu.
m (EN(())
h GELU (Wo [mu; m..)
(10)
We use multi-headed attention in our formulation similar to Brody et al. (002))
2.2.2 GLOBAL ATTENTION ENCODER WITH GRAPH- AWARE POSITIONAL EMBEDDING
To capture global interactions, the atom representations coming of the D-MPNN are fed into global attention encoder, which is a variant of the Transformer encoder. We incorporate graph- aware positional embedding, adapted from the relative positional embedding used in Transformer-
XL (Dai et al., 2019b) as follows. Firstly, in the standard Transformer either with sinusoidal encod- ing Vaswani et al. 2017 Schwaller et al. 2019) or learnable (Devlin et al. 2019) absolute positional embedding, the attention score between atoms and can be decomposed as
abe h W Wah. h W W&P. P.. W.W.WW, + p" W.! WaP.
(11)
where W.WWe are weights for the keys and queries, and Pa. Pv are the absolute positional encoding embedding corresponding to atoms and Similar Transformer-XI, four terms. Instead of using sequence-based relative positional embedding Tu-v, we use a learnable embedding term Tu.o that dependent on the shortest path length between and
reparameterize the
W. + d²) WK.RT., [h W. + +c") Whhh + WW.WT
h T WT. + +c.) Whhe + W.. + d=) Pu..
(12)
The trainable biases renamed and d to avoid confusion and shared across all layers. Intu- itively, the two terms in Equ (12) model the interactions between inputs (h,, and h.), and between input and relative graph position (h., and Fu,,) respectively. Unlike Transformer-X1, we forgo the inductive bias built into the sinusoidal encoding, merging WK.RFu, into single learnable Fa.e- This makes the relative positional embedding easily generalizable to atoms not within the same molecule, which partieular useful for reaction outcome prediction and, more generally, tasks with more than two input molecules. We also bucket the distances similar to Raffel et al. (2020) such that
distance(uu (u, v),
if distance (1, v) 8
if 8 < distance(u, v) < 15
Buy
9,
15 < distance(u, v) and u, v are in the same molecule if u, are not in the same molecule
10,
Byye is then used to look up Fu,o in the trainable positional embedding matrix. The rest of the global attention encoder mostly follows standard Transformer with ct al. 2017), layer normalization (Ba et al,, 2016). and position-wii wise feed forward layers, except no positional information is added to the value vectors within each Transformer layer.
self-attention (Vaswani
4

---

Under review as a conference paper at ICLR 2022
2.2.3 SEQUENCE DECODER
We use Transformer-basst autoregressive decoder to decode from the atom representations after the global attention encoder. Each output token is generated by attending to all atoms with encoder- decoder attention (Bahdanau et al., 2015; Vaswani al., 2017), while also attending all tokens that have already been generated. Following [Seo et al. 2021), we set max relative positions to 4 for the decoder, thereby enabling the usage sequence- based
based relative positional embedding used in Shaw et al. 2018) and implemented by OpenNMT (Klein et al. 2017).
2.3
MODEL TRAINING
The Graph2SMILES model is then trained maximize the conditional likelihood
a
p (San 9mm) = P (81, .2,.... sn II
po 181111111 9
(13)
3
RELATED WORK
3.1 REACTION OUTCOME PREDICTION AND ONE-STEP RETROSYNTHESIS
One approach to computer-aided reaction prediction and retrosynthesis is to make use of chemical reaction rules based on subgraph pattern matching that are formalized as reaction templates, as in expert systems such LHASA (Corey al. 1972) and SYNTHIA (Szymkué et al. 2016). More recent efforts have used neural networks to model the two tasks as template classification (Segler & Waller 2017 Baylon et al. 2019 Dai al. 2019a Chen & Jung 2021), template ranking based on molecular similarity (Coley et al. 2017). These template-based approaches select the top ranked templates, which can then be applied to transform the input molecules into the outputs.
For template-based based methods, there is an inevitable tradcoff between template generality and speci- ficity. Further, these methods cannot generalize to unseen templates. As remediation for such in- trinsic limitations, number of template-fred approaches have emerged over the recent years, which can be categorized into graph edit-based and translation-basee based. The first category models reaction
prediction or retrosynthesis as graph transformations (Jin et al. [017]] Coley et al.] [019 Do et al.)
2021 Qian et al. 2020). Variants of graph edit methods 202 and semi remplate-based methods where reaction centers are first identified, followed by a graph or sequence recovery stage (Shi et al., 2020) Yan cet al. 2020: Somnath et al. 20200) Wang et al. (2021b). Translation- based formulations,
2019 Bradshaw al., 2019 Sacha et al. include electron flow prediction (Bi et al.
the other
hand, approach the problems as SMILES-to-SSi MS-SLL translation, typically with sequence models
such Recurrent Neural Networks (Nam & Kim 2016 Schwaller et al.) 2018 Liu al. (017) or
the Transformer (Schwaller et al. 2019 Lin et al.
2020 Lee et al. 2019 Duan et al.) 20200 Tetko
al. 2020). Variants of these approaches design additional stages such as pretraining and reranking
(Irwin et al. 1:21: Zhu et al., 2021]] Zheng el al.]
2020 Sun et al. 2020). or use information about
graph topology to enhance performance (Yoo et al. 2020 Seo al. 2021 Mao et al. 202T).
Our approach is similar to GET (Mao et al.) 2021) which also solves one-step retrosynthesis with graph-enhanced encoders and sequence decoders. Unlike GET which concatenates the SMILES sequence embeddings and learned atom representations, thereby not guaranteeing permutation in- variance, we do not use the sequence representation at all in our encoder. Yet Graph2SMILEES yields significant improvement over GET on USPTO_500 demonstrating the power of our graph encoder and the adequacy of graph representations alone.
3.2 ADAPTING THE TRANSFORMER ENCODER FOR MOLECULAR REPRESENTATION
The idea of modeling graphs using the Transformer architecture is not new. In the domain of molec- ular representation, the Molecular Attention Transformer (Maziarka et al.) 2020) and GeoT Kwak et al. 2021 inject atomic distance information into when computing attention scores. However, the computation of such distance information itself requires sampling of 3D conformers, thereby introducing additional source of variations and breaking order invariance. An alternative is using the lengths of shortest paths between atoms. GRAT (Yoo et al., 2020) uses these lengths to pa- rameterize the optional scale and bias for the attention score, whereas PAGTN (Chen et al. 2019)
5

---

Under review as a conference paper at ICLR 2022
treats them as path features in its additive attention. Our use of pairwise shortest path lengths be- tween atoms for our Transformer-basee global attention encoder are inspired by GRAT and PAGTN. Contrary to their usage of these lengths additional features, we explicitly separate the effect of graph topology using graph-aware relative positional embedding, considering the success of such specially designed embedding in graph representation (Ying al. (02T) and other domains (Shaw et al., 20188 Dai et al.) [2019b] Wann et al., 2019 Guo et al. (020). Our formulation of relative positional embedding builds on top of its counterpart in Transformer-XL (Dai et al. 2019b), which we found to be empirically superior than using single learnable bias as in T5 (Raffel et al. 2020)
and Graphormer (Ying et al. (02I).
3.3 COMBINATION OF GRAPH NEURAL NETWORKS AND TRANSFORMER
Combining graph encoder and Transformer encoder in a sequential manner has been explored in GET (Mao et al. (021) and NERF (Bi et al. (02T), as well as in Wang et al. (2021a) and GROVER does not use the (Rong et al.J (020) albeit for different molecular learning tasks. Graph25M
sequence representation an input as in GET, require the output molecular graph in the encoder as in NERF Also, none of these related works retain the explicit information about graph topology before passing the atom representations into the attention encoder like we do, which we show to be important in the ablation study in Section 4.5 Similarly, the graph-to-sequence formulation itself has been used in NLP for conditional text generation tasks such as SQL-to-text (Xu et al.) b)) and AMR-to-text (Cai & Lam, 2020). Our encoder is different from these prior studies; most notably, the graph-aware positional embedding is designed to easily generalize to more than two disconnected graphs, which is typical for reaction outcome prediction.
4 EXPERIMENTS
4.1 DATASETS
We evaluate model performance by top-n test accuracies on four USPTO datasets derived from reaction data originally curated by Lowe (012).. The details of these datasets are summarized in Appendix C For reaction outcome prediction, we evaluate on the USPTO 480k mixed and USPTO STEREO mixed datasets following Schwaaler et al. (2019). The suffix mixed indi- cates that the reactants and reagents have not been separated based on which species contribute heavy atoms to the product. While USPTO_480 USPTO_STEREO was filtered to lesser extent, retaining stereochemical information and reactions forming or breaking aromatic bonds. For one-step retrosynthesis, we evaluate on the USPTO _full and USPTO_ 2_50k datasets without reaction type, both of which have been used as benchmarks for retrosynthesis. We count a prediction as correct only if it matches the ground truth output SMILES exactly, including all stereochemistry but excluding atom mapping, after canonicalization by RDKit.
has been preprocessed by Jin et al.] (2017),
4.2 IMPLEMENTATION DETAILSS
For D-GAT and D-GCN, we change the hidden size to 256 from 300 used in GraphRetro (Somnath) et al. 2020 to be more consistent with the global attention encoder. The number of message updat- ing steps is set to 4. For the global attention encoder, following Molecular Transformer and GTA, fix the embedding and hidden sizes dmades to 256, the filter size for Transformer to 2048, the number of attention heads to 8, and the numbers of layers for both the attention encoder and the Transformer decoder to 6. We train our model using Adam optimizer (Kingma Bal 2015) with Noam learning rate scheduler (Vaswani et al. 2017). Similar to Schwaller et al. (2019). we group reactions with similar number of SMILES tokens together, batch by the maximal number of token count, and scale the D-MPNN outputs by /dmodel before feeding into the attention encoder. The details of the hyperparameters used for different datasets are summarized in Appendix D We save the model checkpoints every 5000 steps, select the best checkpoints based on the top-I accuracy on the validation sets, and report the performance on the held-out test sets. Beam search is used to generate the output SMILES during inference with a beam size of 30. We filter out any SMILES that cannot be parsed by RDKit and keep the remaining as our final list of proposed candidates for evaluation.

---

Under review as a conference paper at ICLR 2022
4.3 RESULTS ON REACTION OUTCOME PREDICTION
Table summarizes the results of Graph2SMILES and other existing works on reaction outcome prediction. We only include methods that perform evaluations on the USPTO 240k mixed dataset in the table, and provide brief comparison of other methods that only evaluate on the less challenging USPTO_4800 10__440k__ag Beok_ssparated data in Appendix [ All excluded methods show inferior performance to the Molecular Transformer (MT), with the exception of NERF (Bi et al. 2021) which has 0.3 point improvement top-1 accuracy.
Table 1: Results for reaction outcome
prediction on USPTO_400 _80k mixed and USPTO_STEREO_min Best results for complete columns highlighted in bold.
Methods
Top-n accuracy (%)
3
USPTO_480 480k mixed
MEGAN (Sacha et al. 2021)
86.3
92.4
94.0
95.4
Molecular Transtormer (Schwaller et
2019
88.6
93.5
94.2
94.9
Graph2SMILES (D-GCN) (ours)
90.3
94.0
94.6
95.2
Graph2SMILES (D-GAT)
(ours)
90.3
94.0
94.8
95.3
Augmented Transformer
Tetko et al.
2020
90.6
96.1
Chemformer (Irwin et al.
2021
91.3
93.7
94.0
USPTO_STEREO _m
Molecular Transformer (Schwaller et al.
2019
76.2
84.3
85.8
Graph2SMILES (D-GAT) (ours)
78.1
84.5
85.7
86.7
Graph2SMILES (D-GCN) (ours)
78.1
84.6
85.8
86.8
For the USPTO 480k mixed dataset, Graph2SMILES improves upon the MT baseline with 1.7, 0.5 and 0.4 point increases in top-1, 3 and 10 accuracies respectively. Similarly, for the USPTO_STERIS STEREO_mixed dataset, Graph2SMII.ES improves the top-1 accuracy of the MT base- line by 1.9 points, with minor improvement for top-3 accuracy. While there is still gap between Graph2SMILES and Augmented Transformer or Chemformer, our approach does not use test-time data augmentation and ensembling in Augmented Transformer (Tetko al. 2020), nor have performed pretraining as Chemformer (Irwin et al. 2021) whose models have up to 10 times as many parameters Graph2SMILES. Ensembling and pretraining are potential directions for im- proving Graph2SMILES as they are still compatible with our backbone replacement for the Trans- former. For reaction outcome prediction, there is small advantage of using D-GAT over D-GCN, with improvements of up to 0.2 points on top-n accuracy.
4.4 RESULTS ON STEP RETROSYNTHESIS
We compare the results of one-step retrosynthesis on USPTO full of Graph2SMILES with all ex- isting methods that report results on this dataset, to the best of our knowledge. As can be seen from Table 2 Graph2SMILES achieves higher top-1 accuracy than all methods except GTA (Seo) et al. al, 2021), while not using any templates, atom mapping, or output-side data augmentation. These additional features techniques, which have been demonstrated to improve the performance of Transformer variants (Appendix D are orthogonal to the graph-to- topph-to--eqennce architecture itself, and therefore potentially improve Graph2SMILES well. For example, atom mapping be used to enhance Graph2SMiLES with graph- truncated cross-attention as in GTA.
The much smaller USPTO_50 50k dataset has been benchmarked more extensively. We compare the results in Table 3 marking only the usage of the same set features and techniques used as in Table 2 for succinetness The first group of rows shows that across methods that do not reaction
'GraphZSMILES can also benefit from other techniques such as latent variable modeling (Chen et al. 2020 which we demonstrate in Appendix [ From Table 10, using latent classes N 2 can already boost the top- accuracies of the D-GCN variant from 72.9 79.5, and of D-GAT from 73.9 77.7.

---

Under review as a conference paper at ICLR 2022
reaction templates used; Map.: atom- mapping required; Aug.: output data augmentation used. Best results are highlighted in bold.
Table 2: Retrosynthesis results USPTO_ full. Templ:
Methods
Top-n accuracy (%)
Features / techniques used
10
Templ.
Map.
Aug.
RetroSim (Coley et 2017
32.8
56.1
X
MEGAN Sacha et al. 2021
33.6
63.9
X
/
x
NeuralSym (Segles er & Waller 2017)
35.8
60.8
X
GLN (Dai et al. 2019a)
39.3
63.7
J
X
Transformer baselmne (Zhu et al. 2021
42.9
66.8
X
X
RetroPrime (Wang al. 202 lb
44.1
68.5
X
ANKAK
Aug. Transformer (Teiko et al. 2020
44.4
73.3
X
X
DMP fusion (Zhu al. 2021
45.0
67.9
X
Graph2SMILES (D-GAA)
45.7
62.9
X
X
X
Graph2SMILES (D-GCN) (ours)
45.7
63.4
X
GTA Seo et al. 2021
46.6
70.4
X
templates, atom mapping, or output SMILES augmentation. Graph2SMILES achieves the best top- accuracy, improving the Transformer baseline (Lin et al. 2020) by 9.8 points from 43.1 to 52.9. From our experiments, we observe that it is possible boost top-n accuracies for n > 1 at the expense of sacrificing the top-1 accuracy (even slightly), creating some room for tradeoff. To avoid over-tuning and giving overly optimistic results, however, we only report the test results for models with the highest top-1 accuracy during validation. Unlike in reaction outcome prediction, using D- GAT does not have clear advantage over D-GCN for retrosynthesis, yielding only improvement on top-5 and 10 accuracies for USPTO_500 2_50k as in Table 3
Table 3: Retrosynthesis results on USPTO_500 without reaction type. Templ.: reaction templates
used; Map.: atom-mapping required; Aug.: output data augmentation used. Best results for each group of rows are highlighted bold.
Methods
Top-n accuracy (%)
Features / techniques used
3
5
10
Templ.
Map.
Aug.
AutoSynRoute (Lin et al.
2020
43.1
64.6
71.8
78.7
X
X
GET (Mao et al. 2021)
44.9
58.8
62.4
65.9
X
X
x
DMP Tusion (Zhu
2021
46.1
65.2
70.4
74.3
X
X
Tied Transformer Kum et al.
2021
47.1
67.2
73.5
78.5
X
X
x
Graph2SMILES (D-GAT) (ours)
51.2
66.3
70.4
73.9
X
X
X
Graph2SMILES (D-GCN) (ours)
52.9
66.5
70.0
72.9
X
X
X
MEGAN Sacha et al.
202
48.1
70.7
78.4
86.1
X
X
G2Gs Shi el al. 2020
48.9
67.6
72.5
75.5
X
X
RetroXpert Yan al.
2020
50.4
61.1
62.3
63.4
X
GTA (Soo et al. 2021
51.1
67.6
74.8
81.6
X
RetroPrime (Wag al. 2021b
51.4
70.8
74.0
76.1
GLN DDai et al. 2019a)
52.5
69.0
75.6
83.7
Aug. Transtormer (Tetko al.
2020
53.2
80.5
85.2
X
X
X
LocalRetro (Chen & Jung 202
53.4
77.5
85,9
92.4
GraphRetro (Somnath et al. 2020
53.7
68.3
72.2
75.5
Chemformer (Irwin et al. 2021
54.3
62.3
63.0
EBM (Dual-TB) (Sun al.
2020
55.2
74.6
80.5
86.9
The second group of rows in Table incluues methods that use additional features techniques. Graph2SMILES beats a number methods in this group with top-1 accuracy of 52.9 without using any of templates, atom mapping output SMILES augmentation. EBM (Dual-TB) (Sun)

---

Under review as a conference paper at ICLR 2022
et al. 2020) achieves the SOTA for top-1 accuracy, and LocalRetro (Chen & Jung 2021 for top-3, and 10 accuracies respectively. Both make use of templates and atom mapping, which seem to be empirically helpful for this small dataset. Similar to how Graph2SMIILES can benefit from using atom mapping as discussed earlier, we can make use of templates for potential gain, e.g. by retaining only candidates with reaction templates that have been seen in the training set.
4.5 ABLATION STUDY
We perform ablation study to explore the effects of various components of Graph2SMILES by removing the D-MPNN, positional embedding, or the global attention encoder. We summarize the results on USPTO_500 for which the quantitative effect is most conspicuous. From the results in Table the removal of D-MPNN decreases the top-1 accuracy to 49.9. Note that this setup is architecturally similar to the Transformer baseline, but uses atom features rather than SMILES sequence as inputs. Getting rid of the graph-aware positional embedding leads to a drop in top- accuracy of 2.1 points for D-GCN and 0.6 points for D-GAT. The effect of removing the global attention encoder is more significant, decreasing the top-1 accuracy by up to 8.3 points. We therefore conclude that all three components of our encoder in Graph2SMILES, namely the D-MPNN, the graph-aware positional embedding, and the global attention encoder, are important. They work collectively capture meaningful representation of the input molecular graphs.
Table 4: Ablation study for Graph2SMILES on USPTO_500
Architecture
Top-1
Top-3
Top-5
Top-10
Transformer baseline, AutoSynRoute (Lin al. 2020
43.1
64.6
71.8
78.7
Graph2SMILES without D-MPNN
49.9
67.1
71.7
75.3
Graph2SMILES (D-GCN) no graph-aware positional embedding
52.9
66.5
70.0
72.9
50.8
65.2
69.4
73.6
global attention encoder
44.6
60.7
65.2
69.6
Graph2SMILES (D-GAT) no graph-aware positional embedding
51.2
66.3
70.4
73.9
50.6
65.7
69.8
73.2
global attention encoder
44.9
59.3
64.1
68.0
DISCUSSION
Throughout our experiments, demonstrate the advantage over vanilla Transformer, we have focused on the baseline Graph2SMILES model, forgoing the benefits of using additional features and techniques for performance engineering. Although we have used the top-1 accuracy as basis for comparison throughout our discussion, recognize its limitations especially for retrosynthesis, which can have many equally plausible options. Similarly, the datasets we use reaction outcome prediction not perfectly detailed, with legitimate ambiguity in the identity of the major product. While Graph2SMILES shows strong performance for top-1 accuracy and can replace Transformer model with minimal modification to the pipeline, there could be cases when top-n accuracy is more relevant (e.g. in multi-step planning applications). In those cases, the aforementioned performance engineering techniques would be necessary boost the top-n performance of Graph2SMILES.
CONCLUSION
In this paper, we present a novel Graph2SMILES model for template-free reaction outcome pre- diction and retrosynthesis. The permutation invariance its D-MPNN and graph-aware positional embedding eliminates the need for any input-side SMILES augmentation, while achieving notice- able improvement over the Transformer baselines, especially for top-1 accuracy. Graph2SMILES therefore attractive drop-in replacement for any methods that the Transformer model for molecular transformation tasks. Further gain may be possible through performance engineering tricks that orthogonal to the architecture itself, which will be investigated in future work.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We hereby declare that all of our reported results are reproducible from our existing code base, subject to minor deviations due to hardware-level numerical uncertainties as we have observed for some GPU models (e.g. V100), We include part of our code to reproduce some specific results, with self-explanatory README file in the supplementary materials. We will open-source all scripts to reproduce any result in the manuscript along with pretrained Graph2SMII.ES checkpoints after the review period, regardless of whether the manuscript is accepted.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Javier L. Baylon, Nicholas A. Cilfone, Jeffrey R. Gulcher, and Thomas W. Chittenden. Enhanc- ing retrosynthetic reaction prediction with deep learning using multiscale reaction classification. Journal of Chemical Information and Modeling, (9(2):673-885 2019. doi: 10. 10.102//acs.jcmm. 8b00801. PMID: 30642173.
Hangrui Bi, Hengyi Wang, Chence Shi, Connor Coley Jian Tang, and Hongyu Guo. Non- autoregressive electron redistribution modeling for reaction prediction. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 904-913. PMLR, 18-24 Jul 2021.
Esben Jannik Bjerrum. SMILES enumeration as data augmentation for neural network modeling of molecules. CoRR, 2017. URL
John Bradshaw, Matt J. Kusner, Brooks Paige, Marwin H. S. Segler, and José Miguel Hernández- Lobato. A generative model for electron paths. International Conference on Learning Repre- sentations, 2019.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? CoRR, 2021.
URL /ttL:::::///// /ttppp::///axiiiu /aaxiviivvv. wwt.:oggg//aaaoo...oo..oo.o. httpg//b//211... 114911155.4111111 14491
Deng Cai and Wai Lam. Graph transformer for graph-to-scquence learning. Proceedings of the
AAAI Conference Artificial Intelligence, 34(05):7464-7471, Apr. 2020. doi: 10.1609/aaii.
v34i05.6243.
Benson Chen, Regina Barzilay, and Tommi S. Jaakkola. Path-augmented graph transformer network. CORR, 2019. URL
12712
Benson Chen, Tianxiao Shen, Tommi S. Jaakkola, and Regina Barzilay. Learning to make general- izable and diverse predictions for retrosynthesis, 2020. URL https:://penrevieer nett Forum Forum?id-Bygf-ZNKs
Shuan Chen and Yousung Jung. Deep retrosynthetic reaction prediction using local reactivity and global attention. JACS Au, 2021. doi: 10.1021/jacsau.Ic0022 c000244 URL /tttp:://oi..rrs
10. 10.1021/ http://aaaaauucc00433. 1c00246
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Pruceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1734,173, Doha, Qatar, October 2014. Associ- ation for Computational Linguistics. doi: 10.3 15/v 1/D14-1199
Connor W. Coley, Luke Rogers, William H. Green, and Klavs F. Jensen. Computer-assisted ret- rosynthesis based on molecular similarity. ACS Central Science, 3(12):1237-12255 2017. doi: 10. 10.1021 hac0///accceetee.00003555.. PMID: 29296663.
10

---

Under review as a conference paper at ICLR 2022
Connor W. Coley, Wengong Jin, Luke Rogers, Timothy F. Jamison, Tommi S. Jaakkola, William H. Green, Regina Barzilay, and Klavs F. Jensen. A graph-convolutionaa neural network model for the prediction of chemical reactivity. Chem. Sci., 10:370-377, 2019. doi: (0.1039/C8SC04228D.
E. J. Corey. Robert Robinson lecture. retrosynthetic thinking essentials and examples. Chem. Soc. Rev., 113131,,,, 1988. doi: 10. 10.1039/CS9888100011.
E. J. Corey and W. Todd Wipke. Computer-assisted design of complex organic syntheses. Science, 166(3902):178-199)) 1969. doi: 10. 1 1126//ciece.166..393 166.3902.778
E. J. Corey, Richard D. Cramer, and W. Jeffrey Howe. computer-assisted synthetic analysis for complex molecules. methods and procedures for machine generation of synthetic intermediates. Journal of the American Chemical Society, 94(2):440 459, 1972. doi: 10. 10.1021/ja007577022..
E.J. Corey and X.M Cheng. The Logic of Chemical Synthesis. Wiley, 1989.
Hanjun Dai, Chengtao Li, Connor Coley, Bo Dai, and Le Song. Retrosynthesis prediction with conditional graph logic network. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019a.
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer- XL: Attentive language models beyond a fixed-length context. Proceedings of the 57th Annual Meeting the Association for Computational Linguistics, pp. 2988-2998. Florence, Italy, July 2019b. Association for Computational Linguistics. doi: 10.1 20.16633v1/P19/ 1285.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/113//1//111143
Kien Do, Truyen Tran, and Svetha Venkatesh. Graph transformation policy network for chemi- cal reaction prediction. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '19, pp. 750-760, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362016 doi: 10. 00.114//392550033339988
Hongliang Duan, Ling Wang, Chengyun Zhang, Lin Guo, and Jianjun Li. Retrosynthesis with attention-based NMT model and chemical analysis of "wrong" predictions. RSC Adv, 10: 1371- 1378, 2020. doi: 10.1039/C9RA08535..
Meng-Hao Guo, Junxiong Cai, Zheng-Ning Liu, Tai-Jiang Mu. Ralph R. Martin, and Shi-Min Hu. PCT: point cloud transformer. CoRR, 2020. URL https::/arxii...rr http://wwwiv..rr////////2///2 09688
Dan Hendrycks and Kevin Gimpel. Bridging nonlincarities and stochastic regularizers with gaussian error linear units. CoRR, 2016.
1606. 08415
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference on Learning Representations, 2020.
R. Irwin, S. Dimitriadis, J. He, and E. Bjerrum. Chemformer: A pre-trained transformer for compu- tational chemistry, 2021.
Wengong Jin, Connor Coley, Regina Barzilay, and Tommi Jaakkola. Predicting organic reaction out- comes with Weisfeiler- network. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molecular graph generation. In Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pp. 2323-2332. PMLR, 10-15 Jul 2018.
11

---

Under review as a conference paper at ICLR 2022
Eunji Kim, Dongseon Lee, Youngchun Kwon, Min Sik Park, and Youn-Suk Choi. Valid, plausi- ble, and diverse retrosynthesis using tied two-way transformers with latent variables. Journal of Chemical Information and Modeling, 61(1):123-133, 2021. doi: 11.02///cc..cim.00000777. PMID: 33410697.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. OpenNMT: Open- source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstra- tions, pp. 67-72, Vancouver, Canada, July 2017. Association for Computational Linguistics.
Bumju Kwak, Jeonghee Jo, Byunghan Lee, and Sungroh Yoon. Geometry-aware transformer for molecular property prediction. CoRR, 2021. URL https: httpp://aaxxxvoorr/////////
15516
Greg Landrum. Rdkit: Open-source cheminformattics software, 2016. URL Utthtts::////tthb
Com/ http:///tw.cckkkeeeeellasse///ss
Alpha A. Lee, Qingyi Yang. Vishnu Sresht, Peter Bolgar, Xinjun Hou, Jacquelyn L. Klug-
McLeod, and Christopher R. Butler. Molecular transformer unifies reaction prediction and retrosynthesis across pharma chemical space. Chem. Commun. 55:12152-12155, 2019, doi: 10.1039/C9CC051221.
Kangjie Lin, Youjun Xu, Jianfeng Pei, and Luhua Lai. Automatic retrosynthetic route planning using template-free models. Chem. Sci., 1 11:3355-3364, 2020. doi: 10. 10.1039/C9S0036666
Bowen Liu, Bharath Ramsundar, Prasad Kawthekar, Jade Shi, Joseph Gomes, Quang Luu Nguyen, Stephen Ho, Jack Sloane, Paul Wender, and Vijay Pande. Retrosynthetic reaction prediction using neural requence-to-sequeee models. ACS Central Science, 3(10):1103-1113 2017. doi: 11102//acseentscc.00000333 PMID: 29104927.
D. M. Lowe. Extraction of chemical structures and reactions from the literature (doctoral thesis), 2012.
Pierre Mahé, Nobuhisa Ueda, Tatsuya Akutsu, Jean-Luc Perret, and Jean-Philippe Vert. Extensions of marginalized graph kernels, Proceedings of the Twenty-First International Conference on Machine Learning. ICML '04, pp. 70, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581 133855. doi: 10.1145/00 101544.....
Kelong Mao, Xi Xiao, Tingyang Xu, Yu Rong, Junzhou Huang, and Peilin Zhao. Molecular graph enhanced transformer for retrosynthesis prediction. Neurocomputing, 457:193-202, 2021. ISSN 0925-2312. doi: https://doi.org/ 10. 1016/...euccom.mm2 2et..2222.......
Lukasz Maziarka, Tomasz Danel, Slawomir Mucha, Krzysztof Rataj, Jacek Tabor, and Stanislaw Jastrzebski. Molecule attention transformer. CoRR, 2020. URL 2002. 08264
Juno Nam and Jurae Kim. Linking the neural machine translation and the prediction of organic chemistry reactions. CORR, abs/1612.09529, 2016.
Greeshma Neglur, Robert L. Grossman, and Bing Liu. Assigning unique keys to chemical com- pounds for data integration: Some interesting counter examples. In Data Integration in the Life Sciences, pp. 145-157, Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3-540- 31879-8.
Wesley Wei Qian, Nathan T. Russell, Claire L. W. Simons, Yunan Luo, Martin D. Burke, and Jian Peng. Integrating deep neural networks and symbolic inference for organic reactivity prediction. ChemRxiv, 2020. doi: 0.264344chemrxiv.l 116.s....166663.3 1659563.v1.
12

---

Under review as a conference paper at ICLR 2022
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jml www.pr/pppppp////app. w/tp:///222000020 html
David Rogers and Mathew Hahn. dateeddddcoonneeee mation and Modeling, 50(5):742-754, 2010. doi: 10.1021/c11005500 PMID: 20426451.
fingerprints. Journal of Chemical Infor-
Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying WEI, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12559-12577
Curran Associates, Inc., 2020.
Mikolaj Sacha, Mikotaj Blaž, Piotr Byrski, Paweł Dąbrowski- Rabrowski-Tumanski, Mikolaj Chrominski, Rafal Loska, Pawel Włodarczyk-
(todarrzyy-PPruzyisskt Pruszyúski, and Stanislaw Jastrzebski. Molecule edit graph attention network: Modeling chemical reactions as sequences of graph edits. Journal of Chemical Informa- fion and Modeling. 61(7):3273-3284, 2021.
doi: 1/t.///c...cim.....
PMID: 34251814.
Philippe Schwaller, Théophile Gaudin, Dávid Lányi, Costas Bekas, and Teodoro Laino. "Found in translation": predicting outcomes of complex organic chemistry reactions using neural sequence- (o-sequence models. Chem. Sci., 9:6091-6098, 2018.
doi: 10.1039/CSSC02339E..
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A. Hunter, Costas Bekas, and Alpha A. Lee. Molecular transformer: A model for uncertainty- action prediction. ACS Central Science, 5(9):1572-1583, 2019. doi: 10. http://cccceettccc..000000000000
chemical re-
Marwin H. S. Segler and Mark P. Waller. Neural-symbolic machine learning for retrosynthesis and reaction prediction. Chemistry European Journal, 23(25):5966-599 -9771,, 2017. doi: https: //doi.org/10. 100.000022ccem
Seung-Woo Seo, You Young Song, June Yong Yang, Seohui Bac, Hankook Lee, Jinwoo Shin, Sung Ju Hwang, and Eunho Yang. Gta: Graph truncated attention for retrosynthesis, Proceedings of the AAAI Conference on Artificial Intelligence, 35(1):531-534 May 2021.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp- 464-468, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10. 10.18653/v 10133//1/N1--2077
Chence Shi, Minkai Xu, Hongyu Guo, Ming Zhang, and Jian Tang. A graph to graphs framework for retrosynthesis prediction. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 8818 8827. PMLR, 13-18 Jul 2020.
Vignesh Ram Somnath, Charlotte Bunne, Connor W. Coley, Andreas Krause, and Regina Barzilay. Learning graph models for template-free retrosynthesis. International Conference on Machine Learning (ICML) Workshop on Graph Representation Learning and Beyond (GRL+), 2020. URL
httpps://grl
g thub. i htppps/pps///// /611 .ddf
Ruoxi Sun, Hanjun Dai, Li Li, Steven Keamne,, and Bo Dai. Energy-based view of retrosynthesis, 2020.
Sara Szymkuć, Ewa P. Gajewska, Tomasz Klucznik, Karol Molga, Piotr Dittwald, Michat Startek, Michal Bajczyk, and Bartosz A. Grzybowski. Computer-assisted synthetic planning: The end of the beginning. Angewandte Chemie International Edition, 55(20):504455977, 2016. doi: https://do..org/10 wtt.://oorr....0000/aeeee2......
1002/anie.201506100..
V. Tetko, P. Karpov, and R. Van Deursen. Sfate-of-the-art art augmented NLP transformer models for direct and single-step retro synthesis. Nature Communications, 11, 2020.
13

---

Under review as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser. and Illia Polosukhin. Attention is all you need. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
Jiahao Wang, Shuangjia Zheng. Jianwen Chen, and Yuedong Yang. Meta learning for low-resource molecular optimization. Journal of Chemical Information and Modeling, (1( 2021a. doi: Jcmi2///cs..ccmm0000 1416. PMID: 33729779.
Xiaorui Wang, Yuquan Li, Jiezhong Qiu, Guangyong Chen, Huanxiang Liu, Benben Liao, Chang- Yu Hsieh, and Xiaojun Yao. Retroprime: A diverse, plausible and transformer- baseddd method for single-step retrosynthesis predictions. Chemical Engineering Journal, 420: 2021b. ISSN 1385-8947. doi: https://doi.org/ 10. 10w.//..ce...21......... 1.2994..
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. Self-attention with structural position representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joinl Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1403-1109, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10. 1/D 1EDD99-114 1145.
David Weininger. SMILES, chemical language and information system. 1. Introduction method- ology and encoding rules. Journal of Chemical Information and Computer Sciences, 28(1):31-36, 1988. doi: 10.1021/c:0005::000.
Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng, and Vadim Sheinin. SQL-to-text generation with graph-to-sequence model. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, Brussels, Belgium, October-November 2018a. Association for Computational Linguistics.
Kun Xu, Lingfei Wu, Zhiguo Wang, Yansong Feng. Michael Witbrock, and Vadim Sheinin. Graph2seq: Graph to sequence learning with attention based neural networks. CoRR, abs/1 00011804.002.2 2018b.
Chaochao Yan, Qianggang Ding, Peilin Zhao, Shuangjia Zheng, Jinyu Yang, Yang Yu, and Jun- zhou Huang. Retroxpert: Decompose retrosynthesis prediction like chemist. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 11248-1 1258. Curran Associates, Inc., 2020.
Kevin Yang. Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay. Analyzing learned molecular representations for property prediction. Journal of Chemical Information and Modeling, 59(8):3370-3388, 2019. doi: 10. 10.1001/acs.jcimm.eb 9b00233.. PMTD: 18361484.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Yan Liu. Do transformers really perform bad for graph representation? CoRR, 2021. URL
https: /ttt://aaavvcc////ee www.:/ogggssaooo.
05234
Sanghyun Yoo, Young-Seok Kim, Kang Hyun Lee, Kuhwan Jeong, Junhwi Choi, Hoshik Lee, and
Young Sang Choi. Graph-aware transformer: Is attention all graphs need? CoRR, 2020. URL
htttps:/aazx
http://b//20050 05213
Shuangjia Zheng, Jiahua Rao, Zhongyue Zhang, Jun Xu, and Yuedong Yang. Predicting retrosyn- thetic reactions using self-corrected transformer neural networks. Journal of Chemical Informa- tion and Modeling. 60(1):47-55 2020. doi: 10. J.../9994
9b00949, PMID: 31825611.
Jinhua Zhu, Yingee Xia, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Dual-view molecule pre-training. CoRR, 2021. URL httpstts:///axiv
10234
14

---

Under review as a conference paper at ICLR 2022
A APPENDIX: ATOM AND BOND FEATURES USED
T summarizes the atom and bond features used in Graph2SMILES. Most features were adapted from GraphRetro (Somnath et al. 2020), with the addition of chiral features (R/S and E/Z).
Table 5: Atom and bond features.
Feature
Possible values
Size
Atom Feature
Atom symbol
C, O etc.
65
Degree of the atom Formal charge of the atom Valency of the atom Hybridization of the atom Number of associated hydrogens
{d 2;0 <
10
(d 2
{d E <
sp, sp², sp'ssppd. dp
0, 1, 3, 4,5
5
Chirality
R. S. unspecified
Part of an aromatic ring
True, false
2
Bond Feature
Bond type
Single, double, triple, aromatic, other
Cis-trans isomerism
E, 7., unspecified
3
Conjugated
True, false
Part of ring
True, false
2
B APPENDIX: OTHER METHODS FOR REACTION OUTCOME PREDICTION
Table Results for reaction outcome prediction on USPTO USPTO_4SOK __00__sspprateed for methods excluded in Section 4.3 sorted by top-1 accuracy. Best results are highlighted in bold.
Methods
Top-n accuracy (%)
5
WLN/WLDN (Jin et al.
2017
79.6
87.7
89.2
Seq2Seg Schwaller
2018
80.3
86.2
87.5
GTPN (Do et al. 2019)
83.2
86.0
86.5
WLDNS (Coley
2019
85.6
92.8
93.4
GRAT Yoo et al. 2020)
88.3
Symbolic (Qian al.
2020
90.4
94.1
95.0
NERF Bi et al. 202
90.7
93.3
93.7
Molecular Transformer (Schwaller et al.
2019
90.4
94.6
95.3
summarizss the results for methods excluded in Section 4.3 for reaction outcome prediction, most of which report the values on the less challenging USPTO 480k booooktteeatedd dataset, which the reagents have been heuristically separated from the reactants. Only NERF shows marginal im- provement of 0.3 points for top-1 accuracy over Molecular Transformer, whereas all other methods cannot perform as well. We therefore use Molecular Transformer as our baseline in Section 4.3
Note that ELECTRO (Bradshaw et al. 2019 tests on simpler subset of reactions with linear elec- tron flow (LEF), and we therefore exclude from the quantitative comparison.
15

---

Under review as a conference paper at ICLR 2022
C
APPENDIX: SUMMARY OF FOUR USPTO DATASETS USED
Table Statistics of USPTO datasets used.
Dataset
Source
Train size
Validation size
Test size
USPTO 480k mixed USPTO STEREEOEEE mixed
MT repof
409.035
30,000
40,000
MT repo
902.581
50,131
50,258
USPTO_500 50k
GLN repot
40,008
5,001
5,007
USPTO_ful!
GLN repo
810,496
101,311
101,311
hhaps///gthubbbom///ajnnnnnDiii D..GN.
D APPENDIX: HYPERPARAMETER SETTING
Table 8: Hyperparameter setting used in the experiments for different datasets. Best settings selected based validation are highlighted in bold if multiple values have been experimented.
Dataset
Parameter
Value(s)
All
Embedding size
256, 512
Hidden size (same among all modules)
256, 512
Filter size in Transformer
2048
Number of D-MPNN layers
2, 4,
Number of D-GAT attention heads
8
Attention encoder layers
4,
Attention encoder heads
8
Decoder layers
4.
Decoder heads
8
Number of accumulation steps
4
USPTO_480k
Batch type
Source token counts
Batch size
4096
Total number of steps
300,000
Noam learning rate factor Dropout
2
0.1
USPTO_STEREO
Batch type
Source token counts
Batch size
4096
Total number of steps
400,000
Noam learning rate factor
2
Dropout
0.1
USPTO_50K
Batch type
Source token counts
Batch size
4096
Total number of steps
200,000
Noam learning rate factor
2, 4
Dropout
0.1, 0.3
USPTO_ full
Batch type
Source+target token counts
Batch size
8192
Total number of steps
400,000
Noam learning rate factor
2
Dropout
0.1
16

---

Under review as a conference paper at ICLR 2022
E APPENDIX: EFFECTIVENESS OF ADDITIONAL FEATURES OR TECHNIQUES
Table 9: Published results USPTO 50k without reaction type, that demonstrate the effectiveness of additional features or techniques for the Transformer model and variants
Method
Top-
Top- 3
Top-
Top-
10
EBM reranking using Transformer (Sun et al. 2020
cemplate-free proposal
53.6
70.7
74.6
77.0
on proposal constrained by templates
55.2
74.6
80.5
86.9
GTA, non-augmented (Sco et al. 2021
without cross-attention
46.8
65.2
70.5
74.9
with cross- aotsenatt (using atom mapping)
47.3
66.7
72.3
76.5
DMP, with pretrained Transformer encoder (Zhu et al. 2021
Transformer baseline
42.3
61.9
67.5
72.9
fusion with pretrained ChemBERTa encoder
43.9
62.2
68.0
73.1
fusion with pretrained DMP encoder
46.1
65.2
70.4
74.3
Augmented Transformer Tetko et al. 2020 x20 augmentation for products only
42.5
x20 augmentation for both products and reactants
48.0
SCROP [Zheng al. 2020
Transformer baseline
43.3
59.1
64.0
67.0
reranked with syntax corrector
43.7
60.0
65.2
68.7
Latent Transformer, non-augmented (Chen al. 2020
no latent variable (N=1)
42.0
57.0
61.9
65.7
with latent variable (N=2)
42.1
60.0
64.9
70.3
For all groups of rows in Table the first row of numbers are for the Transformer variants where fea- used, and the rest of the rows otherwise. In turn, these results
tures techniques of interest illustrate the effectiveness of using templates (EBM), atom-mapping (GTA), pretraining (DMP), output-side augmentation (Augmented Transformer), syntax-based reranking (SCROP) and varia- tional inference (Latent Transformer). Inclusion of these features or techniques clearly improves the accuracies across the board over respective baselines, as highlighted in bold.
Table 10: Graph2SMILES results on USPTO_500 without reaction type with latent variables
Model
Top-1
Top-3
Top-5
Top-10
Graph2SMILES (D-GCN)
latent variable (N=l)
52.9
66.5
70.0 75.2
72.9
with latent variable (N=2)
52.0
70.2
79.5
Graph2SMILES (D-GAT)
no latent variable (N=1)
51.2
66.3
70.4
73.9
with latent variable (N=2)
50.3
68.8
73.7
77.7
As an illustration of how Graph2SMILES can benefit from additional features techniques, briefly experiment with one of them, latent variable modeling similar to Chen et al, (2020) and Kim al. 202T). using N 2 latent classes with a uniform prior. As in Table this noticeably increases the top-n accuracies for both D-GCN and D-GAT, which can be particularly relevant for multi-step planning.
17

---

Under review as a conference paper at ICLR 2022
F APPENDIX: VISUALIZATION OF CASES WHERE GRAPH2SMILES OUTPERFORMS THE TRANSFORMER BASELINE
We include some real cases for which Graph2SMILES gives the correct top-1 prediction but the Molecular Transformer baseline cannot, for the reaction outcome prediction task on USPTO 480k_ _0_0444kkieed__ mixed. For each row in Figure 2 the reactants shown the left, the ground truth product (which is also the correct prediction by Graph2SMILES) is shown in the center, and the erroneous prediction by the Molecular Transformer baseline shown on the right.
Reactants
Graph2SMILES (correct))
Molecular Transformer (wrong)
Reactants
Graph2SMILES (correct))
Molecular Transformer (wrong)
Hp
Reactants
Graph2SMILES (correct))
Molecular Transformer (wrong)
Reactants
Graph2SMILES (correct))
Molecular Transformer (wrong)
Figure 2: Real cases for which Graph2SMILES predicts the major product correctly on USPTO 480k 480k_mixed. Top 2 rows: N-capping followed by intramolecular ring closing. Third row:
deamination. Fourth row: regioselective substitution.
18

---

Under review as a conference paper at ICLR 2022
The first two represent two intramolecular ring forming reactions after an N-alkylation ami- dation. As shown by the right-most molecules, without explicit knowledge of the reactant graphs, Molecular Transformer instead predicts substitution reactions at the benzylic carbon and at the phenyl carbon respectively. Both predictions seem to be based solely on local chemical environ- ments, with the first being plausible nucleophilic substitution benzyl bromide, but the second nonsensical since such electrophilic aromatic substitution would not happen with the nucleophilic phenylamine. Graph2SMILES,
the other hand, correctly predicts the ring formation. We there- fore hypothesize that graph representation may be more powerful at encoding beyond local contexts via its global attention mechanism, which is potentially important for such reactions. Further quan- titative evaluation is left future work, since explainability is not trivial task for template-free models.
The third row represents the removal of the amine group attached to the benzene ring, for which Graph2SMILEES gives the correct prediction, but the Molecular Transformer predicts the formation of an aryl hydrazine, possibly by detecting the extra nitrogen in the input SMILES (from DMF) but failing to realize as merely spectator solvent without full recognition of its graph structure. The last row is where Graph2SMILES predicts the correct regioselectivity (ortho the nitro group) for the aromatic substitution reaction. This may be interpreted as better capability of Graph2SMILES learn longer range topological information compared to the Transformer baseline.
para to
19


Reviews:
[{"review_id": "yD3y7HieAoL", "reviewer": "Reviewer_3c52", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of this paper are as follows.\n1.\tThis paper proposes Graph2SMILES, which is a graph-to-sequence architecture without using sequence representations of input SMILES. Therefore, Graph2SMILES is permutation invariant to the input and does not need the data augmentation.\n2.\tGraph2SMILES has a wide range of potential applications because it can serve as a drop-in replacement for Transformer in many tasks involving the molecule(s)-to-molecule(s) transformation.\n\nMy major concerns are as follows.\n1.\tThis paper states that Graph2SMILES achieves state-of-the-art top-1 accuracy on common benchmarks among methods that do not use reaction templates, atom mapping, pretraining, or data augmentation strategies. The authors claim that integrating the above features or techniques with Graph2SMILES could improve the performance. However, they do not conduct experiments to demonstrate their claim. Besides, as the aforementioned techniques are commonly seen in predictive chemistry tasks, the authors may want to explain why they do not equip Graph2SMILES with these techniques.\n2.\tD-GAT is a variant of D-GCN with attention-based message updates. However, D-GAT does not outperform D-GCN in terms of the top-1 accuracy, which is the basis for comparison throughout the discussion in this paper. According to Table 1, D-GAT has a small advantage over D-GCN only in terms of the top-5 and top-10 accuracies in the reaction outcome prediction.\n3.\tGraph2SMILES involves calculating pairwise shortest path lengths between atoms, which can be computationally prohibitive. The authors may want to compare Graph2SMILES against baselines in terms of the computational complexity.\n\n", "summary_of_the_paper": "This paper proposes a graph-to-sequence architecture called Graph2SMILES for the retrosynthesis and the reaction outcome prediction. Graph2SMILES uses an attention-augmented D-MPNN encoder to capture the local information and a global attention encoder with graph-aware positional embeddings to capture the global information. Experiments show that Graph2SMILES is competitive with Transformer baselines but does not outperform state-of-the-art methods on tasks of the one-step retrosynthesis and the reaction outcome prediction. ", "summary_of_the_review": "This paper studies two important problems in the computer-aided organic chemistry and proposes a graph-to-sequence architecture called Graph2SMILES. However, the empirical results do not show a superior performance of Graph2SMILES to existing methods, and the technical contribution is incremental.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hNcuqExBE4k", "reviewer": "Reviewer_jeTQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.\tFrom Eqn.(1) to Eqn.(5), you choose to use a complex gating mechanism to aggregate information. Is every component necessary? What if using a simple GCN or GAT?\n2.\tI think the model contains more parameters than conventional retrosynthesis models like conventional Transformer, GLN. Could you please show compare the number of parameters of different methods?\n3.\tThe authors should provide some real cases to show how the method outperforms previous baselines, and why the method can obtain good results without templates.\n\n\nMissing References:\n\n1.\tDual-view Molecule Pre-training, https://arxiv.org/abs/2106.10234, the authors also work on retrosynthesis using Transformer and GNN models. A comparison is necessary.\n", "summary_of_the_paper": "The authors proposed a new method for retrosynthesis, which does not require the mapping numbers and extracting templates from the literature.\n\nBasically, the model consists of a graph-based encoder and a sequence based encoder. The encoder consists of local aggregation from neighbors and global attention using a new positional method. The decoder is a Transformer model with relative positional encoding.\n\nThe method achieved promising results on several retrosynthesis datasets.\n", "summary_of_the_review": "The results in this paper are good, although the method itself is not quite novel.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NWgRYLmj6HE", "reviewer": "Reviewer_Wxfa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+)\n\nUsing a sequence-independent encoding in the easy-to-use transformer makes sense and is a research question which is interesting for the - I think mostly, AI in chemistry - community.\n\n(-)\n\n- As far as I can see, the technical novelty is limited. The proposal is a combination of rather well-known methods.\n\n D-MPNN (where attention is added) and\n\nre-parameterized, relative positional encodings \n\n (using 0 to represent atoms in different molecules)\n\nin transformers\n\n- It is unclear if the proposed attention in the GNN is useful: Since we only see \"For reaction outcome prediction, there is a small advantage of using D-GAT over D-GCN\", and the ablation results are missing.\n\n- The results in Table 1 are only convincing for USPTO_STEREO_mixed.\n\n- The experimental comparison for retrosynthesis compares to methods which use different forms of pretraining or augmentation, which makes sense. It shows that the augmentation still provides advantages. And, as far as I understand, the graph-based molecule embedding entails that Smiles augmentation would not improve your results further? Also, it would make sense to include the ablation results for Graph2Smiles just using the transformer into Table 2 directly. \n\n\n-------------------------------\nOther Comments\n\n\n- It is unclear to me what \"Our hyperparameters for D-GAT and D-GCN are adapted from GraphRetro\" means.\n- In (8), s_uv should be AttnSum(...)\n- How exactly is \\mathcal{B}_u,v used in the learnable \\tilde{r}_u,v? \n- Table 4: \nSince the paper proposes the attention-based GNN, the ablation should be provided for that model.\n- Table 4: What is \"no global attention encoder\"? Just the combination of GNN embeddings w/o transformer? Since transformer is the baseline, I would not consider this as an ablation setting.\n- Table 4: How do the results look on the other tasks? For retrosynthesis, all open existing systems that yield full retrosynthesis trees which I know use top50 (or similar), so the fact that the base model is better at top10 (already) renders the analysis questionable.\n- \"We include part of our code to reproduce some speciﬁc results\" - Why not all?\n", "summary_of_the_paper": "The paper proposes a GNN-based extension of transformers, which have been shown to be effective for reaction prediction etc. before. In particular, the GNN-based embedding of molecules in the reaction embeddings overcomes the artificial bias inherent in the often applied sequence embeddings. The experiments show that there are sometimes increases of performance in reaction and retrosynthesis prediction - and the approach could be applied to similar problems.\n\n\n", "summary_of_the_review": "Overall, I think the authors' proposed model for reaction prediction makes sense. However, as mentioned above, the paper's writing could be improved, the technical contribution is limited, and the experiments also show only limited improvements. Altogether, I therefore suggest to reject the paper at the current moment. I am happy to adjust my score in case I missed critical parts.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "AXWkgle9wkC", "reviewer": "Reviewer_6G2y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed framework integrates several recently developed engineering techniques and empirically shows its superior performance over vanilla SMILES-to-SMILES transformer baseline. This paper provides another comparison baseline for research on retrosynthesis and reaction prediction. Nevertheless, I have the following concerns regarding the paper.\n\n1.\tThe proposed framework is similar to the NERF approach (Bi et al. ICML 2021) as cited by the authors. NERF formulates the reaction prediction problem as a graph-to-graph translation problem. Also, NERF first leverages graph neural networks to capture the local information in individual molecules and then utilizes a Transformer encoder to further models the intermolecular interactions between nodes from multiple molecules. Furthermore, NERF uses a Transformer decoder to decode the output as graph. These are almost the same as that in the method proposed in the paper. The only different to me is that the NERF uses a Transformer to decode the output into graph directly (in a non-autoregressive fashion), while the proposed method here uses the Transformer to decode the output into SMILES strings (in an autoregressive fashion). In this sense, the novelty of this paper is limited to me. Note: I think NERF can naturally apply to two or more molecules since the Transformer encoder is used by considering all node embeddings from multiple molecules as a node set. \n2.\tExperimentally, the proposed method is not directly compared with NERF. I think such comparison is necessary since as shown in the paper, NERF outperforms the SMILES-to-SMILES transformer baseline, and even the augmented version of it, to which the proposed method here obtained inferior performance. The two methods are similar and closely related. I would expect the paper to include NERF into the main results in Table 1. Also, for the USPTO_STEREO_mixed task, I wonder what the reason was for only comparing with the vanilla Transformer, and why not comparing with the Augmented Transformer or the state-of-the-art method Chemformer?\n3.\tResults in Table 1 show that, the proposed method is inferior to the Transformer baseline with simple augmentation. Also, 1% less than the tested method Chemformer, which makes the paper’s contribution less significant to me.\n4.\tThe claim in the Abstract “molecular graph encoders that mitigates the need for input data augmentation” is a strong claim to me. Nevertheless, there is no evidence to support that claim. The slightly better performance over the SMILES-to-SMILES transformer baseline is not a convincing evidence to me. Input data augmentation may play a significant role on regularizing the deep neural networks. I think better justification to support the claim is necessary. \n5.\tThe statement in the last sentence of the first paragraph on Page2:  “[SMILES augmentation]…be interpreted as evidence of the ineffectiveness of the SMILES representation itself.” I think this hypothesis may need better support and analysis. To me, the augmentation of SMILES strings can act as a model regularization method, which helps the trained model to generalize well to unseen data, and may not directly infer the ineffectiveness of the SMILES representation itself. \n6.\tI am not fully understand the claim in the second paragraph of Page2 “… we guarantee the permutation invariance of Graph2SMILES to the input, eliminating the need for input-side augmentation altogether.” I think it would be useful to specify how and why so.\n7.\tThe proposed method integrates several performance engineering techniques (such as attention weights and multi-headed attention in the graph encoder, integration of shortest path length in the positional embeddings etc.), so where the improvement is really coming from is not clear to me. In the ablation study in Table4, both the positional embedding and global attention are key to the Transformer’s performance, so the performance degradation is expected when remove them: Transformer expects a positional embedding to work, and without a global attention the encoder will not be able to capture information from multiple molecules (their graphs are disconnected). \n", "summary_of_the_paper": "This paper proposes a graph-to-SMILES framework, which incorporates several recently developed engineering techniques from the community, for synthesis planning and reaction outcome prediction tasks. The proposed method leverages graph neural networks and Transformer attention model to encode the graph inputs and then utilizes a Transformer decoder to generate the SMILES string as outputs. Experiments on benchmark retrosynthesis and reaction prediction tasks show that the proposed approach outperformed the vanilla SMILES-to-SMILES transformer baseline, but obtained inferior results than some other advanced methods. The paper is interesting, but both the technical novelty and the experimental studies are weak to me.", "summary_of_the_review": "The proposed method is similar to NERF as proposed by Bi et al., so the technical novelty is limited. Also, the experiment study missed important comparison baselines. Furthermore, a more comprehensive ablation study is needed since several engineering techniques are employed, and it is difficult to tell where the performance improvement is really coming from when compared to the Transformer vanilla model.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While the reviewers appreciated the method's ability to replace transformer models and SMILES data augmentation their main concerns were with (a) the experimental section, and (b) the technical innovation over prior work, which updated drafts of the paper did not fully resolve. Specifically for (a) this work performs very similarly to prior work: for reaction outcome prediction the proposed method improves top-1/3/5 for USPTO_STEREO_mixed but is outperformed by prior work for top-1/5/10 for USPTO_460k_mixed; for retrosynthesis the model is outperformed for USPTO_full and only outperforms prior work that does not use templates/atom-mapping/augmentation for top-1 on USPTO_50k. The authors argue that their method should be preferred because their method does not require templates, atom-mapping, and data augmentation. The reviewers agree that template-free and atom-mapping-free methods are more widely applicable. However, the benefits of being augmentation-free is not convincingly stated by the authors who only state that their approach is beneficial by "simplifying data preprocessing and potentially saving training time." The authors should have empirically verified these claim by reporting training time, because it is not obvious that their model which requires pairwise shortest path lengths is actually faster to train.
For (b) the reviewers believed that the paper lacked technical novelty given recent work (e.g., NERF). The authors should more clearly distinguish this work from past work (e.g., graphical depictions and finer past work categorization may help with this).
Given the similar performance to prior work, the lack of evidence to support training time claims, and the limited technical novelty, I believe this work should be rejected at this time. Once these things are clarified this paper will be improved.
Decision: Reject