ReaKE: Contrastive Molecular Representation Learning with Chemical Synthetic Knowledge Graph
Authors: Yi Wang
Shuangjia Zheng
Jiahua Rao
Yunan Luo
Yuedong Yang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Molecular representation learning has demonstrated great promise in bridging machine learning and chemical science and in supporting novel chemical discoveries. State-of-the-art methods mostly employ graph neural networks (GNNs) with self-supervised learning (SSL) and extra chemical reaction knowledge to empower the learned embeddings. However, prior works ignore three major issues in modeling reaction data, that is abnormal energy flow, ambiguous embeddings, and sparse embedding space problems. To address these problems, we propose ReaKE, a chemical synthetic knowledge graph-driven pre-training framework for molecular representation learning. We first construct a large-scale chemical synthetic knowledge graph comprising reactants, products and reaction rules. We then propose triplet-level and graph-level contrastive learning strategies to jointly optimize the knowledge graph and molecular embeddings. Representations learned by ReaKE can capture intermolecular relationships reflected in the semantic knowledge graph and molecular structures. By comparing with other state-of-the-art methods, we show that ReaKE can achieve competitive performance on the reaction prediction pretext task and the learned representations transfer well to various downstream tasks, including reaction classification, yield prediction, and molecule property prediction. Further visualization shows that the learned representations can capture the fine-grained differences both between reactions and between molecules.

Under review as a conference paper at ICLR 2023
REAKE: CONTRASTIVE MOLECULAR REPRESEN- TATION LEARNING WITH CHEMICAL SYNTHESIS
KNOWLEDGE GRAPH
Anonymous authors Paper under double-bling blind review
ABSTRACT
Molecular representation learning has demonstrated great promise in bridging ma- chine learning and chemical science and in supporting novel chemical discoveries. State-of-the-arr methods mostly employ graph neural networks (GNNs) with self- supervised learning (SSL.) and extra chemical reaction knowledge to empower the learned embeddings. However, prior works ignore three major issues modeling reaction data, that is abnormal energy flow, ambiguous embeddings, and sparse embedding space problems. To alleviate these problems, we propose ReaKE, a chemical synthesis knowledge graph-driven pre-training framework for molecular representation learning. We first construct a large-scale chemical synthesis knowl- edge graph comprising reactants, products and reaction rules. We then propose triplet-level and graph-level contrastive learning strategies to jointly optimize the knowledge graph and molecular embeddings. Representations learned by ReaKE can capture the changes between the before and after of a reaction (template in- formation) without prior information. Extensive experiments of downstream tasks and visualization demonstrate the effectiveness of our method compared with the state-of-ae art methods.
INTRODUCTION
Organic chemistry is rapidly developed with the growing interest in big data technology (Schwaller et al. [2021b). Among them, reaction prediction becomes necessary component of retro-synthesis analysis or virtual library generation for drug jesign(Kayala & Baldi, 2011). However, the predic- tion of chemical reaction outcomes in terms of products, yields" reaction rates with computa- tional approaches remains a formidable undertaking. In the last years, natural language processing (NLP)-based methods showed robustness and effectiveness in representing molecules and reaction prediction(Schwalee (Schwaller et al. 2020), these methods treat the precursors' Simplified molecular-input line-entry system (SMILEST as text. While effective, they face the challenges of dealing with molecules' structural information.
technology(Schwalleer
To handle this challenge, researchers leverage the ascendency of Graph neural networks (GNNs) in modeling 2D molecular structures et al.] 2019 Yang et al.) 2019 Liu et all 2022 Ma et al.
2022 Still, there exists the problem of predicting out-of-disrribution data samples since labeled data are limited and the chemical space is complex Wu et al. 2018). Thus, some recent methods employ self-supervised learning (SSL)strategies to use unlabelled data, including designing special
pretext tasks and applying the contrastive learning framework You et al. 2020 Zhang et al. 2020
Xu et al. 2021 Wang et al. 2022 Li et al. 2022). However, SSL on molecular graph structures remains challenging as the current approaches mostly lack domain knowledge in chemical synthe- sis. Recent studies have pointed out that pre-trained GNNs with random node/edge masking gives limited improvements and often lead to negative transfer downstream tasks Hu al. (2020); Stärk et al. (202T), as the perturbations actions on graph structures can hurt the structural inductive
RRaction yield measure of the quantity of moles of product formed in relation the reactant con- sumed, obtained in a chemical reaction, usually expressed as a percentage.
Pt is specification for unambiguously describing molecular structures in ASCII strings. For example, the SMIL .FS string of ethanol is 'CCO'.

---

Under review as a conference paper at ICLR 2023
bias of molecules. More recently, few studies inject extra chemical reaction knowledge into SSL training to empower the learned embeddings Wen et al. 2022; Wang et al. 2021). Among them, the state-of-art method MolR (Wang et al., 2021) preserves the equivalence of molecules with respect to chemical reactions in the embedding space. (i.e., forcing the sum of reactant embeddings and the sum of product embeddings be equal for each chemical equation. F
Albeit promising, the chemical reaction-aware method face the following three problems:
(1) Abnormal energy flow: all chemical reactions are accompanied by changes in entropy, and changes in entropy require reaction conditions such as temperature and pressure to trigger. Un- der the equivalence assumption of the previous method, the reactants and products can flow with each other as long as the embedding is equal, which violates the principle of entropy increase in the second law of thermodynamies. For example, given a reaction
B -> C and a reaction of D + F. - C, it will result in A + B - D + F., but that reaction might not occur. (2) Ambiguous embeddings: the previous method assumes that the embeddings of reactants and products are equal in embedding space, however, reactants and products are often similar but totally different in property, this assumption will lead to lack of discrimination between reactants and products in the embedding space, for example, incorrectly predicting products as reactants, more detailed examples are in [f Appendix [F] (such as No.5 and No.72 reactions).
(3) Sparse embedding space: since the amount of recorded chemical reactions is limited, the embed- ding spaces of reactants and products learned by the previous methods are sparse and lack smooth- ness, which may lead to a large offset of embeddings when making small perturbation to the reaction. i.c., if I have an A + B reaction, there will be e(A) = e(B), e(-) represents the em- bedding function. Suppose small perturbation a (removing an atom outside the reaction center) is done on both A and B, it may cause a large offset due to the sparsity of the chemical space and make c(A o) * o)..
To address these problems, we develop ReaKE, novel deep learning framework that learns chemistr--meniinfful molecular representations from graph-in-graph data architecture, i.e., a knowledge graph (KG) that connects 2D molecular graphs using reaction templates. First, to al- leviate the energy flow and the ambiguous embedding problems, we construct a chemical synthesis knowledge graph and build explicit connections between molecules through reaction template in- formation. This can introduce the changes in reaction sites as the trigger conditions of flow between molecules, but also help distinguish reactants and products in the embedding space. Then, for solv- ing the sparse embedding space problem, we further design a functional group- SSL method for reaction triplet-level representation learning, which can help build a denser chemical embedding space. Finally, we propose a reaction-aware contrastive learning strategy to improve the efficiency of the knowledge graph-level training.
Extensive experiments demonstrate that the representations Iearned by our proposed model can ben- efit wide range of downstream tasks which require chemical synthesis priors information. For example, ReaKE achieves 6.8% absolute Hit@1 1 gain in pretext reaction prediction, an average of 9.4% absolute F1 score gain in reaction classifications, and an average 4% R² improvement in yield predictions over existing state-of-the-art methods, respectively. Further visualization studies indicate that our reaction representations can not only categorize reactions clearly but also capture discriminative properties of reaction templates.
2
METHODS
An illustrative overview of proposed method of molecular pre-training with Reaction Knowledge Embedding (ReaKE) is presented in Fig. T In this section, we first introduce the definition of chemical synthesis knowledge graph (section 2.1), as schematically shown in Fig. 2(a) Then we depict the joint learning of the triplet-level encoder and the knowledge encoder at the graph-level (section 2.2), followed by the overall pre-training objects (section 2.3).
Knowledge graph embedding (KGE) aims to encode components of a KG into a low-dimensional continuous vector space to support the downstream graph operations and knowledge reuse.
'For example, given the chemical equation of Fischer esterification of acetic acid and ethanol: CH₃COOH + H2H + CH,COOC, Hs + H₂O, MolR assumes the еcи,соон + εc.H,oи ecH,cooc,H, ed,o also holds, where e(-) represents molecule embedding function.
2

---

Under review as a conference paper at ICLR 2023
Knowledge graph
Knowledge
Negative sampling
Contrastive
Knowledge embedding
loss
B
Gyyr
Product
CC
0o0oc1)
2200c022
enooder
Claseifier
Praperty
pradiction
[CC cosce1
Reaciants
NNc cococt
Dilerence
Classifier
Triplet augmentation
classilicttion
B2 A
encoder
Regressor
Yiald prediction
FC Jayer
Triplat encoder
Downstream lasks
Figure 1: Overview of ReaKE. Our model joint learns molecular embeddings the triplct-level and knowledge embeddings at the graph-level. The pre-trained model is evaluated on three representative tasks, including reaction classification, yield prediction and molecular property prediction.
2.1 DEFINITION OF CHEMICAL SYNTHESIS KNOWLEDGE GRAPH
Chemically speaking, the flow between reactants and products needs to be triggered by environ- mental conditions such temperature and pressure, and the reaction conditions finally reflected on the reaction site which can be represented as the reaction template Therefore, the introduction of template information can help avoid the abnormal energy flow problem. In this sense, we formulate the molecular SSL task as multi-scale knowledge graph embedding (KGE) task Wang al. (2017).
We first introduce a Chemical Synthesis Knowledge Graph (CSKG) that is constructed by chemical reaction data. We define the reactant set R (21.72....) as head entities set, the product set P = {P1. } as tail entities set, and the reaction template set T = {t1, tt..t as relations set. Then, we will have triplet set CSKG. For example, as shown in Fig.
B form Hg Hg + H14 N2, we can extract its template R2 CO R2 R3 NH, > R2 R2 CN R3 as relation and build triplets [CsHHOO template, C N2).. (CoHsN template, (1 More detailed information about CSKG in Appendix[B] To integrate the multi-scale representations of molecules and knowledge graphs, utilize joint contrastive learning fuse the heterogeneous information.
{b. by.b2.... b2,...)
}. where b, (rr, La pi). Triplets
((a) if we have reaction C&H
2.2 JOINT CONTRASTIVE LEARNING FOR TRIPLETS AND THE KNOWLEDGE GRAPH
In this section, introduce our joint learning of triplet-level molecular representation and graph- level knowledge representation. The triplet-level learning is on the individual triplet and the graph- level learning is on a batch of triplets. The main purpose of triplet-level learning to construct relatively dense and smooth embedding space for triplets while the graph-level learning aims at im- proving the model's ability distinguishing noise samples and making the training more effective. Next, we will explain in detail.
2.2.1 MOLECULAR REPRESENTATION LEARNING THE TRIPLET-LEVEL
Molecular Encoder with Graph Neural Network. Let G = (V.E) denote molecular graph with atoms V (U1, U2, ) and bonds m d2, }. Atom attributes are set element type, total
'The reaction template expresses the reaction mechanism and shows the structural changes before and after the reaction. Its change characteristies he the standard of reaction classification.
3

---

Under review as a conference paper at ICLR 2023
Functonal group
Molecule
Knowledge graph
ritatio!
Negative triplets
Augmentation
Augmertation C
Chericcll reaction
GNN Encoder
GNN Encoder
Change
Change
Template
Diference
Sampling entty
Sampling ently
(a) Chemical knowledge graph.
(b) Molecular augmentation.
(c) Negative sampling strategy.
Figure 2: Key points for ReaKE. (a) The construction of chemical synthesis knowledge graph, atoms with red color are reaction center. (b) The reaction-aware negative sampling strategy batch. (c) The sepersuu learning with functional group-bace based augmentation for molecules.
degree, atom is in ring not, the number of connected hydrogen atoms, atom is aromatic, valence, radical electrons and so on. These atom properties are represented as a one-hot vector and we obtain total of 69-dimensional atomic features. GNNs utilize the graph connectivity features to learn representations of atoms and further the entire molecule. Generally, GNNs follow a message passing paradigm, which each atom representation iteratively updated by aggregating the representations of its neighbor atoms. At the 1 layer, GNN updates the atom embedding h; of atom U as:
well node
Update (N! (1))))) Aggregate ((N eV,,)) U (x(H))
(1)
where (2v)) ) is the neighbor set of atom U/. Aggregate() is an aggregation function depending on the architecture of GNN, more details are in Appendix After 7. layers, the final representation of molecule G is defined as follows:
eg = Readout ((h y V)),
(2)
where Readout function collects the representation of all atoms and obtains the final molecule representation by pooling operation.
Triplet-level Contrastive Learning. As shown in the triplet encoder of Fig. For an individ- ual triplet, we generate more possible triplets to make the chemical embedding space denser and smoother. Thus, small shifts in the embeddings of reactants and products would not affect the correctness of triplets and improve the generalized ability unseen cases. Specially, we obtain an augmented version of triplet by keeping the template (relation) and generating a new reactant (head entity) and new product (tail entity). We keep the reaction center functional group and drop atoms (randomly or crop from edge to center) outside the reaction center with a ratio 33 An illustrative example of molecular augmentation is shown in Fig. 2(b)
For a triplet (r, f, p), the augmentation applied twice on reactant r and product separately, resulting in 2 new triplets (r(1): tt P()) and (, After molecular embedding and template embedding (section 2.2.2), get (e,(1), et, (p()) and (er(2): et, Cp(2)): assume e,- + e, Cp. the augmentation loss is defined as follows:
L Lang (p(p))))<(()))) s(Er()); er(2)))
where s is the distance function, s(er())) Cr(2) = ||er(1) - ex(2) I|2 denotes the 12 norm between two embeddings.
4

---

Under review as a conference paper at ICLR 2023
2.2.2 KNOWLEDGE REPRESENTATION LEARNING THE GRAPH-LEVEL.
In addition to the training at the triplet level, we further leverage contrastive knowledge embedding at the graph level to improve the model's ability of distinguishing noise samples and make the training more effective.
Template Encoder. To describe the inference relation from reactant molecule to product molecule, we utilize reaction templates to represent the changes that occur in the chemical process. An illus- trative template example has been shown in Fig, 2(a) In particular, we leverage the toolkit RDChi- ral(Coley et al. 2019) to extract templates with radius 1. Given a reaction F1 + F2 ...... + r - r is the reactant set and {PI, P2, ... P ) is the product set, defined the template 4,, + 4r2 +... lxx + lp + 1m2 + ... + Lp, and a GNN encoder g(-), the
P1 + P Pq. (r,,
.Pg)
relation embedding is defined follows:
e Σ ((t - s(pp)) j=1
(4)
where & denotes the number reactants, ni the number products. Note that the GNN encoder g(-) we used here is different from the molecule encoder.
Graph-Level Contrastive Learning. In addition, we also propose contrastive strategies for knowl- edge graph embedding. In particular, focus the design of negative sampling Zhang et al. (2019). Negative sampling is a crucial part of the KGE training process, the generation of hard negative samples can greatly improve the efficiency and quality of training. However, previous approaches employ indiscriminate negative sampling in a batch, which can easily generate uninfor- mative wrong negatives. Thus, we propose reaction-aware negative sampling avoid same class negatives, which is shown in Fig. 2(c). The negative sampling strategy is detailed as follows:
(ri, Pt) E B, we sample the head entity r tail entity p in the rest triplets to construct negative triplets set By. Head entities and tail entities in triples whose relation are 4 are excluded, that is we do not sample molecules of the same class as or p;. The set B' used to construct negative samples can be formulated as:
Denote mini-batch B (b1, ........ bn of size n. For triplet
...
B' {bx E Blby (rk, tk Pk)) tk # t.}
The entity set in B' is denoted as (R', P'} where the reactant entity set and the product entity set represented as R' and P', respectively. The negative triplets set B can then be defined as follows:
B (1' 4:)) Rt'} U ((ri, P.Dp/ e P'}
(6)
After filtering out same class negatives, utilize TransE(Bordes al. 2013) as the basic training objective because it is effective and simple in capturing asymmetry, inversion, and composition relations. Thus, the overall KG loss is defined follows:
1 Ealr deell,, (p)))
LKG
de (er: Cp.)
where m is the number of negative samples, it is up to 2(n - 1). a means the sigmoid function, which helps avoid learning easy triplets. (r', 44,p') the jth negative triplet B/. 7 the margin hyperparameter and the distance function de ep) = Il(e, + et) - cpll2-
cp) = (l(e, + et) - -ppl2
(8)
2.3 OPTIMIZATION ORJECTIVES
Overall, the model is trained jointly with a weighted sum of knowledge embedding loss and molecule augmentation loss mentioned above, where stands for the trade-off parameter. is described as follows:
L = LKG XL
(9)
5

---

Under review as a conference paper at ICLR 2023
3 EXPERIMENTS
In this section, we conduct various experiments to demonstrate the generality our model. We first the embedding- srarddeig--a chemical reaction prediction task to validate whether the framework effec- tively solves the abnormal energy flow and the ambiguous embedding issues. Then, we introduce two reaction-related downstream tasks: reaction classification and yield prediction, to investigate whether our model could capture the changes between the before and after of a reaction (template information) without prior information. Finally, we explore the generalization ability of ReaKE through molecular property prediction tasks. The details of the datasets we used in the pretext and downstream tasks are shown in Appendix
3.1 CHEMICAL REACTION PREDICTION
Baselines. Following the evaluation protocol of MoIR<Wang et al. 2021). we compare our mod- els with several state-of-the-an molecular representation methods, including Moleevec(Jaeger et al.] 2018), MolBERT (i & Jiang 2021), and MolR. In the baselines, reactants and products em- bedded by molecule encoder, and the dot product of two embeddings is used for ranking product candidates. The pre-training setup be found in Appendix
Evaluation Protocol. The reaction prediction task aims to predict the real product's ranking in all candidates of given reactants SMILES. In the test set, considering the direct use of templates or reaction types will lead to the problem of product data leakage, which is not conducive to our product ranking task, we split the template into reactant templates {4r 'r::".. ;,r)) ,.r, and product template (tp fp2. ... }- All products and its ty in the test set are treated as candidates. For reactants
in a reaction, we calculate the embedding distance between (e, + Σ g(tr)) and all product candidates' (ep +
and rank the candidates by distance. Then, the true product's ranking is used to calculate mean reciprocal rank (MRR), mean rank (MR), and top-k hit ratio (Hit@k) which are standard evaluation metrics in KG models. Higher MRR, higher Hit@k k and lower MR indicate that model achieves better performance. Note that the results from Mol2vec to
MolR are from Wang et al. 2021
Table 1: Results of product ranking prediction.
Methods
MRR
MR
Hit@1
Hit@3
Hit@5
Hit@ 10
Mol2vec
0.681
483.7
0.614
0.725
0.759
0.798
Mol2vec-FTI
0.688 0.000
417.6 = 0.1
0.620 = 0.000
0.734 = 0.000
0.767 + 0.000
0.806 + 0.000
MolBert
0.708
460.7
0.623
0.768
0.811
0.858
MolBert-FTI
0.731
0.000
457.9 = 0.0
0.649
0.000
0.790
0.000
0.831
0.000
0.873 + 0.000
MolBERT-FI2
0.776 0.000
459.6 = 0.2
0.708
0.000
0.827
0.000
0.859 +:: 0.000
0.891 +: 0.000
MolR
0.918 0.000
27.4 + 0.4
0.882 = 0.000
0.949
0.001
0.960 + 0.001
0.970 + 0.000
ReaKE-SAGE
0.953 1 0.001
4.2
0,930 = 0.001
0.973 = 0.001
0.980 + 0.000
0.987 +:2 0.000
ReaKE-GAT
0.965 0.001
6.6
0.946 + 0.001
0.982 0.001
0.986 + 0.001
0.990 + 0.000
ReaKE-GCN
0.966 0.000
0.1
0.948
= 0.000
0.983 0.000
0.987
- 0.001
0,991 + 0.000
ReaKE-TA TAG
0.967 0.000
2.9 0.0
0.950 T 0.000
0.982 0.000
0.987 0.000
0.992 1 0.000
Results. As shown in Tab. 1 ReaKE gains about 4.9% MRR and 6.8% Hit@1 performance enhancement against the baseline model MolR and outperforms all Bert-based models. In addition, ReaKE outperforms the baseline regardless of which GNN is combined. Appendix F shows the dice similarity between the predicted product and ground-truth product and cases where our method predicted correctly but other methods did not. Taken together, these results indicate that our method can add triggering conditions between reactants and products, avoid the situation of predicting prod- ucts as reactants and can learn the changes between reactants and products, which also confirms the effectiveness of introducing templates.
3.2 REACTION CLASSIFICATION
The goal of reaction classification task is to predict the cat- egory of the given reaction. We consider several molecule fingerprints and reaction fingerprints as baselines, including CCrhhar et al. 1985), DRFP(Schheider et al. 2015), RXNFP RXNFP(Schwalllr (Schwaller et al. 2021a) and RxnRep Wen et al. 2022 AP3 is atom-pairs molecular fingerprint method
Baselines and Evaluation Protocol.

---

Under review as a conference paper at ICLR 2023
with maximum path length of three. DRFP creates a hash binary fingerprint based on the symmet-
ric difference between substructures.
To evaluate the effectiveness of the Iearned representations, use our pre-trained model a feature extractor and obtain the final reaction representations by calculating the difference between reactants and products. Then,
train MLP for reaction classification. Following the few-shot setting of SxnRep(Wen et al. 2022), instead of using the entire training set, we sample 4, 8, 16, 32, 64, 128 reactions per class to simulate the situation of a small dataset. Every experiment repeated five times with the resampling of training data. In addition, we visualize the initial embedding space directly encoded by the pre-trained model. We demonstrate the distribution of chemical reaction embeddings with different classes and the distribution of reactants and products within a type of reaction. Note that the results from AP3 to RxnRep are from Wen al. 2022 For MolR, we use the provided model
Table 2: Classification Fi score on the Schneider dataset.
Methods
reactions per class
reactions per class
16 reactions
32 reactions per class
64 reactions
128 reactions per class
per class
per class
AP3
0.518
0.004
0.620 0.004
0,703 = 0.006
0.761 = 0.002
0.799 + 0.004 0.338 + 0.006
0.828 0.004
DRFP
0.100 0.005
0.129 0.004
0.199 0.008
0.266 = 0.007
0.398 0.002
RXNFP
0.322 0.012
0,394
0.013
0.471 + 0.010
0.531 + 0.006
0.575 + 0.005
0.618
0.004
RxnRep
0.441
0.010
0.634
0.003
0.767
0.003
0.831 + 0.002
0.875
0.003
0.900
0.002
MolR
0.629 0.007
0.722 0.009
0.803 0.006
0.862 = 0.006
0.901 + 0.004
0.900 0.027
ReaKE-SAGR
0.796 0.011
0.860
0.010
0.892
0.004
0.908 + 0.003
0.918 + 0.003
0.933 0.002
RcaKE-GAT
0.778 0.012
0.844
0.004
0.876
0.005
0.897 = 0.005
0.917 + 0.003
0.923 0.007
ReaKE-GCN
0.765
0.008
0,842
0.008
0.877
0.009
0.898 0.003
0.918 1 0.004
0.923 0,016
ReaKE-TAG
0.821 + 0.011
0,882 1 0.003
0.901 + 0.005
0.916 + 0.004
0.928 1 0.003
0.935 0.004
(a) RXNFP.
(b) MolR.
(c) ReaKE.
Figure 3: t-SNE visualization of RXNFP, MolR and ReaKE on schneider dataset. (a), (b)and (c)show the fingerprint distribution of the entire schneider dataset, with 46 colors representing 46 categories of reactions.
Results. The results under transfer feature extraction setting are illustrated in Tab. 2 ReaKE achieves 19.2%. 16%, 9.8%, 5.4%, 2.7%, 3.5% Fi score gain over the state-of-art model with 4, 8, 16, 32, 64, 128 reactions per class as the training set. It demonstrates our learned embeddings can transfer well downstream tasks with small training set. And our model could capture the changes between the before and after of reaction (template information) without prior information.
t-SNE visualizations of RXNFP, MolR and our method on the Schneider dataset are shown in Fig- 6 We perform two kinds of visualizations: One is the visualization on the reaction fingerprint level as shown in Fig. ([a] Fig. (bb and Fig. B[c] that obtain the embeddings of all reactions from pre-trained model, and then conduct t-SNE visualization to see the distribution of the initial embeddings in the embedding space. As shown Fig. 3(c there is a clear boundary between each category and almost no overlapping parts compared with RXNFP and MolR. Furthermore, the reactions the same category are clustered in multiple clusters, which demonstrates that our reaction representations can also capture discriminative properties about reaction templates. The other is the visualization on the molecular level which can be found Appendix D
7

---

Under review as a conference paper at ICLR 2023
3.3 YIELD PREDICTION
Baselines and Evaluation Protocol. Yield prediction aims at predicting the yield of given reac- tion. For comparison, choose the DFT-based method, DRFP (Echneieeeeerrr et al. 2013). Yield-BERT and its augmented version Yield-BER' (aug.) (Schwaller et al. (020) as basclines. Yield-BER is
an extension of the learned RXNFP fingerprint with regression layer.
Table R of yield prediction Buchwald Hartwig reactions.
Test
Test 2
Test 3
Test 4
Avg. 1-4
DFT
0.80
0.77
0.64
0.54
0.69
DREP
0.81 + 0.010
0.83 + 0.003
0.71 + 0.001
0.49 + 0.004
0.71 0.160
Yield-BERT
0.84 + 0.010
0.84 + 0.030
0.75 0.040
0.49 0.050
0.73
Yield-BERT
0.80 = 0.010
0.88 + 0.020
0.56 + 0.080
0.43 + 0.040
0.58 0.330
MolR
0.68 0.003
0.84 0.002
0.61 0.006
0.51 0.003
0.66
0.120
ReaKE
0.87 + 0.002
0.89 + 0.002
0.67 + 0.004
0.57 + 0.005
0.75 0.140
A 0.9140
0.6702 ARE -.0.323
8 6115 Awaa 15-00
5.13
NAI 8.2kd
Prodicted
Predicted
Prodiicted
Measured
Measured
Measured
Measured
(a) Test 1.
(b) Test 2.
(c) Test 3.
(d) Test
Figure 4: Fine-tune performance R² scores (%) on Testl-4 of ReaKE.
We obtain the reaction representations from our pre-trained model, and then feed it into xg- boost(Chen & Guestrin 2016). We leveraged the four tests in DRFP(Schneider
al. 2015) to show the results under splits based on isoxazole additives with 5 times run. We report the mean and standard deviation of R² score. Note that the results from DFT to Yield-BERTTaug.) are from (Schneider et al. 2015).
Result. apply feature extraction setting and fine-tune setting on the yield prediction task. The results under two settings are shown in Tab. and Fig. I separately. For the feature extraction, our method gains about 2% average R² value enhancement on T4s For the fine-tuning setting, we achieve 4% average R improvement model gains about 7% R² improvement. In conclusion, our method can capture subtle changes in the reaction and improve the performance of prediction yield prediction, which also confirms that pretext template information is beneficial for downstream tasks.
Test1-4. In addition, in Testl and Test4, fine-tune
3.4 MOLECULAR PROPERTY PREDICTION
CITA
(a)
Random split.
(b) Scaffold split.
Figure 5: Fine-tune performance ROC-AUC scores (%) on property prediction benchmarks.
8

---

Under review as a conference paper at ICLR 2023
Baselines and Evaluation Protocol.The goal of molecular property prediction is to predict labels of given molecules. To further confirm our pre-trained model's potential when addressing tasks not related to chemical synthesis. apply ReaKE to molecular property predictions. We select the rule-based methods ECFP4(Rogers & Hahn 2010), MACCCS/Heikamp & Bajorath 2011), and the self- self-supervised learning method MolR as baselines. Due to the large gap between the pretext dataset and property datasets, we verify our method by adding an MLP layer under the fine-tuning setting. We demonstrate the average ROC-AUC score of the datasets under random splits and scaffold splits, we apply 5-fold cross- validation to random splits and run experiments 5 times with different random seeds under scaffold splits.
Results. The evaluation results under fine-tune setting are illustrated in Fig. 5 achieve 1.5% and 3.0% gain on the random splits and scaffold splits. It suggests the representation can also transfer well to molecular-related tasks.
On average, we
4 RELATED WORK
Sequence-based methods. One line of works to represent molecules/reaction are BERT-
based(Devlin et al. 2018) or Transformer-based(/V Vaswani et al. 2017) models, such as the SMILES-
BERT Wang et al. 2019 MoI-BERT & Jiang 2021), Mol- Transformer Schwaller et al.
TransformerSchwwllle
2019
PXNFP(Schwalle et al. 2021a), K-BERT(WI (Wu et al. 2022) and KV-PLM et al. 2022). These
methods treated SMILES sequences of molecules/reaction as text and generated efficient molec-
BERT or Transformer models. These methods only use the sequence information of SMILES, however, molecular biological activ- ity largely depends on its structure.
ularfreaction embeddings by designing
Structure-based methods. The other line of studies focus on topological information of molecules. Some traditional methods are topology-basee hashed fingerprints. such as AP3 (Carbart et al. 1985), ECFee(Rogers & Hahn 2010)) and DRFP(Schneider et al.) 2015). they capture the extended- connectivity and leverage the linear path or radial substructure of molecules/reactions. And recent studies mostly leverage GNN-based methods to model the 2D structure of molecules. Among them, self-superveised Learning mSSL)methods are actively proposed to maximize the use of unlabeled data, such as GraphCL et al.) 2020), MICRO-Graph Zhang et al.] 2020), GraphLoG((uu et al.) 202T. MoICLR (Wang et al.] al.] 2022) and KPGT et al.] 2022). The above methods leverage con- trastive learning on molecules and focus on extracting subgraph patterns for more comprehensive
molccular representations.
Knowledge-based SSL Methods. However, recent studies have pointed out that pre-trained GNNs with node prediction, context prediction, and motif prediction pretext tasks gives limited improve- ments and often lead to negative transfer on downstream tasks (Hu et al.) 2020 Stärk et al. 2021
Sun 2022). Thus, It is necessary to provide domain knowledge and design complex pretext task since this can force model learn more useful information. Recent studies inject extra chemical reac- tion knowledge into SSL training to empower the learned embeddings. For example, RxnRep (Wen)
al. 2022) leverages the chemical reaction data and makes the two augmented representations of a reaction similar to each other but distinct from different reactions. MolR [Wang et al./ 202T preserves the equivalence of molecules with respect to chemical reactions in the embedding space.
5 CONCLUSION
In this work, we propose a simple yet effective chemical synthesis knowledge graph to tackle the challenges in modeling reaction data. We introduce the changes reaction sites as the trigger condi- tions of flow between molecules and build explicit connections between molecules through reaction template information. Besides, we joint learning the triplet-level molecular representation and the graph-level knowledge representation. Comprehensive experiments over multiple benchmark down- stream tasks consistently demonstrate the effectiveness of our method.
Further directions to explore be described the following points: First. can take into account the 3D structure of molecules and reactions. Second, it is interesting to consider external factors related to the reaction (i.e., temperature, reaction conditions, etc.).
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Derek T Ahneman, Jesús G Estrada, Shishi Lin, Spencer D Dreher, and Abigail G Doyle. Predicting reaction performance in c-n cross-coupling using machine learning. Science, 360(6385):186- 190, 2018.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data, Advances in neural information pro- cessing systems, 26, 2013.
Raymond E Carhart, Dennis H Smith, and R Venkataraghavan. Atom pairs as molecular features in structure-activityy studies: definition and applications. Journal of Chemical Information and Computer Sciences, 25(2):64-73, 1985.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Connor W Coley, William H Green, and Klavs F Jensen. Rdchiral: An rdkit wrapper for handling stereochemistry in retrosynthetic template extraction and application. Journal of chemical infor- mation and modeling, 59(6)2559-2531 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 18Xiv:1810.044 04,00, 2018.
Jian Du Shanghang Zhang, Guanhang Wu, José MF Moura, and Soummya Kar. Topology adaptive graph convolutional networks. arXiv preprint arXiv: 1710. 17001//7 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Kathrin Heikamp and Jürgen Bajorath. How do 2d fingerprints detect structurally diverse active compounds? revealing compound subset-specific fingerprint features through systematic selec- tion. Journal of chemical information and modeling, 51(9):2254-2265, 2011.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In 8th International Conference Learning Representations, ICLR. OpenReview.net, 2020.
Sabrina Jaeger, Simone Fulle, and Samo Turk. Mol2vec: unsupervised machine learning approach with chemical intuition. Journal of chemical information and modeling, 58(1):27-35, 2018.
Matthew Kayala and Pierre Baldi. A machine learning approach to predict chemical reactions. Advances in Neural Information Processing Systems, 24, 2011.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: (tt...26000207 2016.
Han Li, Dan Zhao, and Jianyang Zeng. Kpgt: Knowledge-guided pre-training of graph transformer for molecular property prediction. arXiv preprint arXiv:2206.03334, 2022.
Juncai Li and Xiaofei Jiang. Mol-bert: An effective molecular representation with bert for molecular property prediction. Wireless Communications and Mobile Computing, 2021, 2021.
Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised repre- sentation for graphs, with applications to molecules. Advances in neural information processing systems, 32, 2019.
Songtao Liu, Rex Ying, Hanze Dong, Lanqing Li, Tingyang Xu, Yu Rong, Peilin Zhao, Junzhou Huang, and Dinghao Wu. Local augmentation for graph neural networks. In International Con- ference Machine Learning, pp. 14054-14077 PMLR, 2022.
Hehuan Ma, Yatao Bian, Yu Rong, Wenbing Huang, Tingyang Xu, Weiyang Xie, Geyan Ye, and Jun- zhou Huang. Cross-dependent graph neural networks for molecular property prediction. Bioin- formatics, 38(7):2003-2000 2009, 2022.
10

---

Under review as a conference paper at ICLR 2023
Daniel Probst, Philippe Schwaller, and Jean-Louis Reymond. Reaction classification and yield pre- diction using the differential reaction fingerprint drfp. Digital discovery, 1(2)911977, 2022.
David Rogers and Mathew Hahn. iirended-connecting fingerprints. Journal of chemical informa- tion and modeling, 50(5):742-755 2010.
Nadine Schneider, Daniel Lowe, Roger A Sayle, and Gregory A Landrum. Development of a novel fingerprint for chemical reactions and its application to large-scale reaction classification and similarity. Journal of chemical information and modeling, 55(1):39-53, 2015.
Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Christopher A Hunter, Costas Bekas, and Alpha A Lee. Molecular transformer: a model for uncertainty-calibraeed chemical reaction prediction. ACS central science, 5(9):1572-5883, 2019.
Philippe Schwaller, Alain C Vaucher, Teodoro Laino, and Jean-Louis Reymond. Data augmentation strategies to improve reaction yield predictions and estimate uncertainty. 2020.
Philippe Schwaller, Daniel Probst, Alain C Vaucher, Vishnu H Nair, David Kreutter, Teodoro Laino, and Lean-Louis Reymond. Mapping the space of chemical reactions using attention- neural networks. Nature Machine Intelligence, 3(2):144-))) 152, 2021a.
Philippe Schwaller, Alain C. Vaucher, Teodoro Laino, and Jean-Louis Reymond. Prediction of chem- ical reaction yields using deep learning. Machine learning: science and technology, 2(1):015016, 2021b.
Hannes Stärk, Dominique Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Günnemann, and Pietro Liò. 3d infomax improves gnns for molecular property prediction. arXiv preprint arXiv:21 10.011000444 10.04126, 2021.
Ruoxi Sun. Does gnn pretraining help molecular representation? arXiv preprint arXiv:2207.06010, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. stat, 1050:20, 2017.
Hongwei Wang, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin Burke. Chemical-reactioonaave molecule representation learning. International Conference Learming Representations, 2021.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(1)): 2744,2743, 2017.
Sheng Wang. Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. Smiles-bert large scale unsupervised pre-training for molecular property prediction. In Proceedings of the 10th ACM international conference on bioinformatics, computational biology and health informatics, pp. 429 436, 2019.
Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279- 287, 2022.
Mingjian Wen, Samuel M Blau, Xiaowei Xie, Shyam Dwaraknath, and Kristin A Persson. Improv- ing machine learning performance on small chemical reaction data with unsupervised contrastive pretraining. Chemical science, 13(5): 144--1458,8 2022.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: a benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530, 2018.

---

Under review as a conference paper at ICLR 2023
Zhenxing Wu, Dejun Jiang, Jike Wang. Xujun Zhang, Hongyan Du, Lurong Pan, Chang- Yu Hsieh, Dongsheng Cao, and Tingjun Hou. Knowledge-baid bert: a method to extract molecular features like computational chemists. Briefings in Bioinformaties, 23 1310 2022.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure, In International Conference on Machine Learning. pp. 11548-11588 PMLR, 2021.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Hua Gao, Angel Guzman-Peree Timothy Hopper, Brian P Kelley, Andrew Palmer, Volker Settels, et al. Are learned molecular representa- tions ready for prime time? 2019.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 33:5812-5823, 2020.
Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable human professionals. Nature communications, 13(1):1-11, 2022.
Shichang Zhang. Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv preprint arXiv:2012.12533 2020.
Yongqi Zhang, Quanming Yao, Yingxia Shao, and Lei Chen. Nscaching: simple and efficient neg- ative sampling for knowledge graph embedding. In 2019 IEEE 35th International Conference Data Engineering (ICDE), pp. 614-625. IEEE, 2019.
Shwangjia Zheng, Jiahua Rao, Zhongyue Zhang, Jun Xu, and Yuedong Yang. Predicting retrosyn- thetic reactions using celf-corretted transformer neural networks. Journal of chemical information and modeling, 60(1):47-55 2019.
12

---

Under review as a conference paper at ICLR 2023
A DETAILS OF GNN ARCHITECTURES
We use SMILES string of molecules as input and then change SMILES into Mol through "rd- kit. Kit.Chem" D finally access atoms and edges in Mol and build graphs. The molecular graphs are the input of GNN encoders. Following MolR Wang et al. 2021 we apply 4 kinds of GNN architecture based on Deep Graph Library (DGL The above GNNs all follow pessage-passingg framework, i.., at each iteration, every node aggregates information from its local neighbor, thus we illustrate the detailed message passing pattern of the GNNs:
Graph Convolutional Network (ipt) & Welling 2016 In GCN, the features of neighbors are summed directly, that is, we treat the neighbors of the current node equally. The feature matrix of the t layer are as follows:
H o(D (DD+*AD + ((--))))))
(10)
where W((-1) is the weight matrix (2- 1) layer, Ā is the adjacency matrix with added self-loop, DRK = Σ, Ang. and (-) means the activation function.
et al. 2017). In GAT, it aggregates neighbor nodes through the attention mechanism, and realizes the adaptive allocation of different neighbor weights. The feature of node i at the 1th layer is:
Graph Attention Network
o
Σ Oiy ( JEV(O)
(11)
where Ony represents the attention score between node and its neighbor j. azj = softmaxy (ery) and eig = a aWh,, Wh,), a()) is a shared attentional mechanism.
Graph Sample and Aggregate (GraphSage) (Hamilton et al. 2017
GraphSage, it learns the embeddings of each node in an inductive mode and updates embedding by sampling and aggregating. can naturally generalize to unseen nodes.
W = (ggregates((h! (1-1), Yo E ((i)))
(12)
o(W Concat(h
(1 (1-1) 2))
where Aggregate(-) can be mean aggregator, LSTM aggregator and pooling aggregator.
Topology Adaptive Graph Convolutional Network (TAGCN) et al. 2017). It is variant of GCN, TAGCN uses K graph convolution kernels to extract local features of different sizes, and retains K as hyperparameter.
K 1 AD -
H
( = o >
(D-+
(13)
k==
where K stands for the number of local filters, O is a weight matrix of the i hop. Totally, TAGCN can extract local features on a of size-1 up to size-K receptive fields.
k
B DATASETS
Pre-training Dataset. The dataset leverage for constructing the chemical synthesis knowledge graph here is 2YPTO-479K(Zheng et al. 2019). It contains reactions with up to five reactants and only one product. By removing reactions from which cannot extract templates and others that contain the same reactants and products, we finally obtain 407,039 training reactions, 29,848 vali- dation reactions and 39, 39,802 testing reactions. We further convert the training reactions into 587,403 triplets with 103,339 reaction templates.
Downstream Task Datasets. For the reaction classification task, we use the Schneider dataset (Schneider et al. 2015). It is derived from the Schneider 50k dataset, which is a descendant of the
hhtt:///ttki...nndddo/s//orrrrrr Cht.tkkhii...hhhmmmmmm hhwps//www.dgl.ai/
13

---

Under review as a conference paper at ICLR 2023
USPTO dataset of patent reactions. After further cleaning, obtain 38,800 reactions with 46 reac- tion types. We split it into training set with 31,002 reactions, a validation set with 3,896 reactions and test set with 3,902 reactions. For the yield prediction task, palladium-catalyzzed Buchwald- Hartwig C-N cross-coupling reactions dataset(AAhennt performance in experiment. It includes 3,955 reactions labeled with yield. These reactions composed of 15 aryl halides, methylaniline. 4 Buchwald ligands, 1 Pd catalyst, bases and 23 additives. In addition, following drfp(Probst al. (022), used four out-of-sample splits based on isoxazole additives and created 70/10/20 train/valid/test split. For the molecular property pre- diction task, we use four Open Graph Benchmark (OGB) datasets with their standard scaffold splits and random splits. The datasets are BACE, Tox21, Clintox and HIV Bataset(Wuu et al. 2018).
et al.] 2018) is utilized to evaluate model
C PRE-TRAINING SETUP
We consider four GNN models as molecule encoders, they are GCpt & Welling. 2016 GAT al., 2017). SAGE(Hamilton et al. 2017), and TAG(Du al. 2017). In all mod- els. we use a 2-layer GNN with a sum pooling Readout function and project the representation to 1024-dimensional latent space for both molecule encoder and template encoder. Besides, we opti- mize our model using Adam optimizer with learning rate of 5e-5. As a default setting, we use margin value y of 4.0, a molecule augmentation drop ratio 3 0.7, and trade-off parameter À of 1.0. Furthermore, we train at batch-size 1024 for 30 epochs on NVIDIA GeForce GTX 1080Ti GPU. Pre-training on 587k triplets of CSKG takes 5.8 hours.
Our pre-training model consists of Graph Isomorphism Network (GIN) from ? with 5 layers and 300 hidden dimensions and a residual convolutional neural network (ResNet-34) ? with 63.5M parameters. We pre-train the model for 100 epochs using a batch size of 1024 on 8 NVIDIA 3090TI GPUs. We use the Adam optimizer with an initial learning rate of 3e-4 and weight decay of 0.02. We take image with resolution of 128x 128. The margin y is pairs for MIGA takes 8 hours, far less than 26 hours for ContextPred and 48 hours for GraphCL on 280k molecules of GEOM-Drugs.
4. Pre-training
750k graph-image
D VISUALIZATION ON SCHNEIDER DATASET
We further demonstrate the distribution of reactants and products within type of reaction in Schnei- der dataset.
(a) DRFP.
(b) RXNEP.
(c) ReaKE.
Figure 6: t-SNE visualization of ReaKE, DRFP, RXNEP on schneider dataset. (a), (b)and (c) demon- strate the distribution of reactant and product embeddings within the reactions of the same class, label O stands for reactants and label stands for products.
6(b) and Fig. 6(c) which is gaining the embeddings of reactants and products in class of Schneider reactions, and then using t-SNE visualization to explore the distribution. As shown in Fig. [(() the reactants and products form clusters separately, which suggests the Ambiguous embeddings problem is alleviated, and
The visualization on the molecular level shown in Fig. 6(a)
Fig.
14

---

Under review as a conference paper at ICLR 2023
there exist differences between the embeddings of the reactant and the product. Besides, model can aggregate molecules with the same functional group.
E ABLATION STUDY
We further analyze the contribution of different components under two kinds augmentation in our ReaKE. The drop atom mode randomly crops atoms outside the reaction center, and the subgraph mode crops atoms from edge to center. The variants are as follows: (1) w/o neg: w/o negative sampling; (2) w/o aug. w/o molecular augmentation; (3) w/o temp: w/o adding reaction templates information.
The ablation results on chemical reaction prediction reported in Appendix removal of the template component leads to the most significant performance drop, which is align with our assumption that template information can benefit molecular modeling. Note that molecular augmentation can bring obvious improvements. We show two kinds of augmentations: drop atom mode and drop subgraph mode. The exclusion of both strategies will decrease the results, showing the importance of augmentation. We also try to iterate the estimation w/o negative sampling but observe a performance drop given more iterations.
E In specific, the
Table Ablation study results on chemical reaction prediction.
Method
MRR
MR
Hit@1 1
@it@10
ReaKE (drop atom)
0.967
2.8
0.950
0.992
w/o neg
0.925
11.6
0,900
0.967
-/o aug
0.953
3.7
0,930
0.987
w/o temp
0.917
33.9
0.880
0.970
ReaKE (subgraph)
0.967
3.3
0.950
0.991
w/o neg
0.922
18.0
0.896
0.966
w/o aug
0.953
3.7
0.930
0.987
w/o temp
0.914
40.4
0.880
0.965
15

---

Under review as a conference paper at ICLR 2023
F CASE STUDY
To explore whether our method can effectively solve the Ambiguous embeddings problem in the pretext task, we statisties the distribution of dice FP similarities between predicted products and real products. We except the cases where both our method and MolR predict the true product for demonstrating the differences more explicitly.
Dice similarity
Dice similarity
0-0.2
0-0.2
0.4-0.6
0.4-0.6
0.6-0.8
0.6-0.8
25%
7%
9%
16%
8%
3%
(a) MolR.
(b) ReaKE.
Figure 7: Pie chart of dice FP similarities distribution of predicted and real products.
We also conduct case study on USPTO-479k and select the first 100 reactions the test dataset to see if our optimization is effective compared with MolR. The detailed results are demonstrated on Table remove examples where all methods predict correctly.
Table
Case study on the first 100 reactions USPTO-479k test set.
Ground-truth product
Predicted product by ReaKE
Predicted product by MolR
Index
Reactant(s)
5
same as ground-truth
same as ground-truth
16

---

Under review as a conference paper at ICLR 2023
66
b11
72
same as ground-truth
79
same as ground-truth
82
same as ground-truth
As shown in Table 5 our method models chemical reaction data in detail and avoids some cases where the predicted product value is consistent with the reactant (such as the No.5 reaction and the No.72 reaction). This phenomenon also confirms that our initial analysis of the problem and optimization measures are effective. Besides, there are reactions (such as the No. of 66 reaction) that cannot be predicted correctly by both methods, which due the fact that only the main product is retained in USPTO-479k, and the by-product is omitted. This deficiency leads to inaccurate prediction, but the correct product ranking predicted by our method still better than previous methods.
17


Reviews:
[{"review_id": "W33jQfh92q", "reviewer": "Reviewer_EYxB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a molecular representation learning method based on pre-training with chemical synthetic knowledge graph. The proposed method tackles and solves the limitations in previous works that use chemical reaction knowledge in self-supervised learning.", "summary_of_the_review": "The motivation of the work is strong, and the proposed approach is simple and practical to deal with the previous limitations. However, the experiments need more clear descriptions and convincing comparisons.", "strength_and_weaknesses": "Pros:\n1. The paper is well-organized and easy to follow.\n2. The proposed approach is simple and useful to deal with the previous limitations.\n\nCons:\n1. The experiments are not clearly described and exhibited:\n    - It is unclear what the \"TAG\" model is. In Appendix A, GCN, GAT, SAGE, and TAG all lack their references. \n    - From the results shown in Table 1 and Table 2, it is not necessary to include the results of ReaKE-SAGE/GAT/GCN, since ReaKE-TAG consistently performs better than the other three variants. \n    - It is unclear whether the results of the baseline models come from their original papers, or from the authors' own experiments. For example, the results of MolR in Table 1 are directly from its original paper [1]. However, the results of MolR in Table 2 and 3 are not found in [1]. The authors do not mention how they get the results. \n    - For the visualization in Figure 3 and 6, the baselines being shown are only DRFP and RXNFP, which perform much worse than the proposed method as shown in Table 2. It would be more convincing to see the visualization of MolR, since it is the most competitive baseline.\n    - For the molecule property prediction task, the results of ECFP4 and MolR shown in Figure 5 of this paper and the results in MolR's original paper (Table 3 in [1]) are very different. The authors should clarify this mismatch. Besides, it would be more clear to just use a table to list the values in Figure 5 for comparison. \n2. The orders of Figure 2b and Figure 2c can be exchanged to match the orders of the corresponding paragraphs.\n3. The authors should double-check if the related papers are correctly referenced. For example, MolR has been already accepted by ICLR 2022 rather than just on arxiv.\n4. Typo: In section 2.2.2, \"knoledge\" --> \"knowledge\"\n\n[1] Wang, Hongwei, Weijiang Li, Xiaomeng Jin, Kyunghyun Cho, Heng Ji, Jiawei Han, and Martin Burke. \"Chemical-Reaction-Aware Molecule Representation Learning.\" In International Conference on Learning Representations. 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CcjYEy6kxR", "reviewer": "Reviewer_oZTX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new framework, named ReaKE for learning representations for chemicals to better predict chemical reactions\nThe main contributions of this work are three folds as claimed by the authors: (1) chemical synthesis KG of reactants, products and; (2) contrastive learning strategies in KG triple level and molecular graph level; (3) joint training on reaction and molecular level which are essentially two types of graph. \nExperiments have demonstrated that ReaKE outperforms other SOTA models and the effectiveness of ReaKE modules with ablation studies. \n", "summary_of_the_review": "This paper proposed ReaKE as a comprehensive framework for chemical-KG representation learning both from the reaction knowledge graph and the molecular atom graph. The major issue of this paper is clarity on model details which makes it hard to fully understand the motivation and rationale of the learning strategies and training protocol. \n\nSome other questions and suggestions are as follows: \n1. There is no related work in this paper which brings difficulties to understand the position of the paper. It is suggested that a separate section discuss the distinction and how the baseline models are selected. Some works are missing regarding SSL in graph neural networks [1], graph-sequence hybrid models [2], and multi-view KG joint training [3]. \n2. Why are traditional KG embedding methods on reaction KG (TransE) not usable for comparison for downstream tasks?\n3. No report of scalability and computing resources used for the complex learning framework. \n4. As a minor issue, the term \"synthetic“ may cause confusion at first when it comes to \"synthetic KG\" which by default means manually generated datasets instead of real-world ones. \"Chemical Synthesis Knowledge Graph (CSKG)\" may be used to avoid misunderstanding. \n\n\nReferences: \n1. Xie, Y., Xu, Z., Zhang, J., Wang, Z., & Ji, S. (2022). Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence.\n2. Wang, Z., Liu, M., Luo, Y., Xu, Z., Xie, Y., Wang, L., ... & Ji, S. (2022). Advanced graph and sequence neural networks for molecular property prediction and drug discovery. Bioinformatics, 38(9), 2579-2586.\n3. Hao, J., Ju, C. J. T., Chen, M., Sun, Y., Zaniolo, C., & Wang, W. (2020, September). Bio-JOIE: Joint representation learning of biological knowledge bases. In Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics (pp. 1-10).\n\nFor other issues such as clarity and novelty, please check the corresponding sections.", "strength_and_weaknesses": "Strength:\n1. Well-established problem and formulation of chemical synthesis problem from the perspective of \"multi-view\" KG representation learning \n2. The proposed framework is well-designed and mostly technically sound. \n3. Extensive experiments from multiple applications and comparison with a large group of the state-of-the-art baseline approaches\n\nWeaknesses: \n1. The writing clarity needs improvement, especially on the term definitions, and specific mathematical functions. \n2. Insufficient justifications for the rationale of the proposed model and training design.\n2. Some important baselines are not included in the comparison, discussion, or related work. It is unclear to distinguish previous work on graph learning and SMILE-based models. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9bkgy1UX_q", "reviewer": "Reviewer_ToSM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper mainly proposes a chemical synthetic knowledge graph (ReaKE) to handle with different downstream tasks relevant to molecules. The proposed framework mainly addresses three problems: abnormal energy flow, ambiguous embeddings and sparse embedding space. Experimental part shows that the ReaKE improves the results in various tasks.", "summary_of_the_review": "The paper proposes chemical knowledge graph ReaKE to learn better for different downstream tasks. \nThe method consists of several points which appear in existing works (contrastive learning, augmentation and negative sampling), but the combination can be new and has not been applied for molecular applications. I would recognize this paper as successfully adapt existing tricks to molecular related applications.\n\nOverall, I think the paper is marginally below the acceptance threshold currently. ", "strength_and_weaknesses": "Pros:\n\n- The paper is easy to follow.\n\n- The three problems the paper given are important for the chemical reaction-aware self-supervised methods and hard to completely solve.\n\n- The paper uses the chemical reaction data differently compared to existing works and get good experiment results on various tasks.\n\nCons:\n\n- The paper does not have a related work part to orderly introduce the existing works and some related works mentioned in the paper are not well introduced. For example, the four GNN models used in experiment part are not introduced in the paper.\n\n- Although the paper has experiments for various datasets, in some tasks such as chemical reaction prediction, reaction classification and yield prediction, there is only one dataset used. I think more datasets should be used to verify the efficiency of the proposed framework.\n\nConcerns:\n\n- In the example in Section 2.1, I cannot understand why one reaction can be transformed into two triplets, as there should be simultaneously two input molecules to get the output molecules.\n\n- In Triplet-level Contrastive Learning in section 2.2.1, the molecules obtained after the drop step might not exist. Is it meaningful to predict these molecules that do not exist in real world?\n\n- In table 1 in section 3.2, the results except MR are stable and almost all the stds are zero. Why?\n\n- In table 2 in section 3.3, the results of DRFP and RXNFP are bad, what is the reason? Are there baselines that can perform better?\n\n- The baselines for MPP tasks are too old. Are there any newer baselines? For example, some NLP based methods are mentioned in introduction. Can these methods applied in MPP tasks?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7eD4qn-ZBO", "reviewer": "Reviewer_5dFJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a self-supervised representation method that is based on chemical reaction data. The paper points out that existing methods have disadvantages such as abnormal energy flow,  ambiguous embeddings, and sparse embedding space. To address these disadvantages of the existing reaction-based representation learning, they propose to use a chemical synthetic knowledge graph that connects molecules using reaction templates and individual graphs that represent the 2D structure of the molecules.  They claim that the intorduction of the reaction knowleadge graph can elivate the energy flow and the ambiguous embedding problems. Besides, they propose to use of a functional group-augmented SSL method for reaction triplet representation learning which they believe will solve the sparse embedding space problem.", "summary_of_the_review": "The paper overall is written clearly, but I have some minor comments or questions as following\n\n1. How does the contrastive knowledge embedding\nat the graph, level make the triplet embeddings more evenly distributed on the embedding space? also, there is a typo here in the paper knoledge graph instead of the knowledge graph.\n\n\n\n2. I am still a bit confused about the intuition behind this: \"The main purpose of triplet-level learning is to construct a\nsmooth latent embedding space for reactants and products while graph-level learning aims to make\ntriplet embeddings distributed on the embedding space more evenly\"\n\n3. In the section titled \"Triplet-level Contrastive Learning\",  the loss seems to be simply on the augmentations of the reactant and augmentations of the products, not sure why it is called triplet-level contrastive learning, this title is a bit confusing. I was wondering without any template or triplet, just doing the same augmentation on each of the molecules and applying similar loss would have the same effect. Also in the appendix, the latent of this model seems to separate the reactant and product very well, how did this happen? also, do we really want to separate the reactant and product? because a product in this reaction could be a reactant in another reaction I think.\n\n", "strength_and_weaknesses": "Strength:\nThey consider the issues such as abnormal energy flow and ambiguous embeddings which were neglected before and proposed valid ways to address them.  The constructive loss defined on the knowledge graph seems to be intuitive and interesting.\n\n\nweakness:", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject