Substructure-Atom Cross Attention for Molecular Representation Learning
Authors: Jiye Kim
Seungbeom Lee
Dongwoo Kim
Sungsoo Ahn
Jaesik Park
Venue: Submitted to ICLR 2023 (2023)
Keywords: Molecular representation learning
Molecular Substructure
Cross-attention

Abstract: Designing a neural network architecture for molecular representation is crucial for AI-driven drug discovery and molecule design. In this work, we propose a new framework for molecular representation learning. Our contribution is threefold: (a) demonstrating the usefulness of incorporating substructures to node-wise features from molecules, (b) designing two branch networks consisting of a transformer and a graph neural network so that the networks fused with asymmetric attention, and (c) not requiring heuristic features and computationally-expensive information from molecules. Using 1.8 million molecules collected from ChEMBL and PubChem database, we pretrain our network to learn a general representation of molecules with minimal supervision. The experimental results show that our pretrained network achieves competitive performance on 11 downstream tasks for molecular property prediction. 

Under review as a conference paper at ICLR 2023
SUBSTRUCTURE-ATOM CROSS ATTENTION FOR MOLECULAR REPRESENTATION LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Designing a neural network architecture for molecular representation is crucial for Al-driven drug discovery and molecule design. In this work, we propose new framework for molecular representation learning. Our contribution is threefold: (a) demonstrating the usefulness of incorporating substructures node-wise features from molecules, (b) designing two branch networks consisting of a transformer and graph neural network so that the networks fused with asymmetric attention, and (c) not requiring heuristic features and computationally-xxensii information from molecules. Using 1.8 million molecules collected from ChEMBL and PubChem database, we pretrain our network to learn general representation of molecules with minimal supervision. The experimental results show that our pretrained network achieves competitive performance 11 downstream tasks for molecular property prediction.
INTRODUCTION
Predicting properties of molecules is one of the fundamental concerns in various fields. For instance, researchers apply deep neural networks (DNNs) to replace expensive real-world experiments to measure the molecular properties of a drug candidate, e.g., the capability of permeating the blood- brain barrier, solubility, and affinity. Such an attempt significantly reduces wet-lab experimentation that often takes more than ten years and costs $1 million (Hughes et al. 2011 Mohs & Greig 2017).
Among the DNN architectures, graph neural networks (GNNs) and Transformers are widely adopted to recognize graph structure of molecules. GNN architectures for molecular representation learning include message-passeeee -aagsngggg neural network (MPNN) and directed MPNN (Gilmer et al. [017 Yang et al. 2019), where they investigate how to obtain effective node, edge, and graph representation. GNNs are powerful in capturing local information of node, but may lack the ability encode information from far-away nodes due to over-smoothing and over-squashing issues (Li al. 2018 Alon & Yahav 2020). On the other hand, Transformer- based architectures such as MAT (Maziarka et al.) (2200 and Graphormer (Ying et al., 202T) augment the self self-attention layer a vanilla Transformer using high-order graph connectivity information. Transformers can encode global information they consider attention between every pair of nodes from the first layer. To guide a structural bias in the attention mechanism, previous work relies on heuristic features such as the shortest path between two nodes since the naive Transformers cannot recognize the graph structure.
From the understanding of chemical structure, it is known that meaningful substructures can be found across different molecules, also known as motif or fragments (Murray & Rees 2009). For example, carbon rings and NO2 groups are typical substructures contributed mutagenicity (Debnath et al. 1991) showing that proper usage of substructures can help property prediction. Molecular substructures are often represented as molecular fingerprints or molecular fragmentation. Molecular fingerprints such as MACCS (Molecular ACCess System) keys (Durant et al., 2002) and Extended- Connectivity Fingerprints (ECFPs) Rogers & Hahn (2010) represent a molecule into fixed binary vector where each bit indicates the presence of certain motif in the molecule. With a predefined fragmentation dictionary, such as BRICS (Degen et al. 2008 or tree decomposition (Jin et al. 2018). molecule can be decomposed into distinct partitions. Interestingly, machine learning algorithms that utilize molecular substructures still show competitive performance on some datasets to deep learning models (Hu et al. 2020. Maziarka et al. 2020).

---

Under review as a conference paper at ICLR 2023
(a) Indistinguishable molecules by traditional GNNs
(b) Indistinguishable molecules by MACCS keys
Figure 1: Traditional GNNs, such as GCN and GAT, cannot distinguish the two molecular graphs in (a). However, it can be easily distinguished through simple 6-ring and S-ring substructure information. On the other hand, the two molecules in (b) have similar substructures, so atom features from neighborhood are necessary discriminate the molecules.
We propose fusion architecture between GNN and Transformer to incorporate molecular graph information and molecular substructures. Molecular substructures and graph are encoded through Transformer and GNN, respectively. The Transformer is designed to recognize the molecular substructures, where the substructure information is mixed through self-attention to obtain better representations. With the Transformer only architecture, however, local information of molecules, such as atoms, bonds, and connectivity, can be lost from the structures. For example, the two molecules shown in Figure Тb share the same representation with MACCS keys while having different structures. To overcome, we use a separate GNN branch for preserving local information. In our model, we inject the GNN feature into the intermediate Transformer layers through the fusing network. In this way, substructures and local node information are interactively fused, producing final representation for molecular graphs.
We name our network Substructure-Aton Cross Attention (SACA) as it uses substructure as well as atom information molecules and fuses them through cross-attention. The architectural choices allows us avoid heuristic features such the shortest-path or 3-dimensional distance in attention layers, which are required in many existing Transformer architectures for molecular representation learning (Maziarka et al., 2020 Ying al. 2021. Furthermore, model reduces the complexity for attention calculation over the node-level Transformer models from O(N to ONN), where N is the number of atoms. To demonstrate the empirical effectiveness of the proposed network and see the ability to capture the general representation of molecules, we evaluate our model on 11 downstream tasks from MoleculeNet (Wu et al. 2018). Our approach achieves the competitive performance on 1 downstream tasks.
In what follows, we summarize the key contributions and benefits.
We propose novel network that combines the information from substructures and node features in a molecule. Our model combines the advantages of both Transformer and GNN architectures represent the information given to each architecture.
We show the effectiveness of model for molecular representation learning. Our model achieves competitive performance upon strong baseline models on 11 molecular property tasks.
Our model does require computationnalyyexxpssii heuristic information of molecular graphs.
The source code and pretrained networks will be released in the public domain upon the paper acceptance.
2 RELATED WORK
2.1 ARCHITECTURES FOR MOLECULAR REPRESENTATION LEARNING
Graph neural networks (GNNs) Most common architecture for molecular representation learning is the GNNs since molecules be naturally represented graph structure; node atom, an edge as a connection. Researchers have actively investigated variations of GNN architectures (Gilmer et al. 2017 Yang et al. 2019 Xiong et al. 2019 Song al. 2020), for molecules. For
2

---

Under review as a conference paper at ICLR 2023
example, MPNN (Gilmer et al. 2017) generalizes the message passing frameworks and explores some variants that predict molecular properties. Directed MPNN (DMPNN) (Yang et al., 2019) proposed to replace the node-based message by edge-based messages to avoid unnecessary message loops. Next, communicative MPNN (CMPNN) (Song et al. 2020) improved DMPNN by additionally considering the node-edge interaction during the message passing phase. AttentiveFP (Xiong et al.. 2019) extends the graph attention mechanism to allow for nonlocal effects at the intramolecular level.
Despite the advance of GNN architectures for molecular representation learning, there are known problems in GNN, such as over-smoothing and over-squashing problems (Li et al., 2018 Alon & Yahav 2020), which means the node representations become too similar, and the information from far nodes does not propagate well as the number of neighbors increases exponentially. Furthermore, Xu et al. (2019) analyzes expressive powers of standard GNNs following neighborhood aggregation scheme and shows that they are bounded to Weisfeiler-Lehmman test (WI-test) (Weisfeiler & Leman 1968). Therefore, GNNs with standard message passing cannot learn to discern a simple substructure such as cycles. For example, the two molecules in Ia cannot be distinguished by WL-test, hence standard GNNs cannot distinguish these two molecules (Bodnar et al. 202T). A solution to this limited representation power of GNNs is to directly incorporate important substructures in the representation learning framework.
With recent advance of Transformer architectures and their promising performance in various domains, including NLP and computer vision (Devlin et al. 2019) [Dosoviskkiy et al.) 2020). Transformer-baat based architectures for molecular representation learning (Maziarka et al.) 2020 Ying
Transformers
et al. |02 Rong et al. 2020; Maziarka et al. 202T) have been developed. Transformer architecture
calculates pair- wise attention between every node from the first layer. Therefore, it can effectively
capture the global information of a graph. However, a vanilla Transformer architecture (Vaswani al. 2017) is not directly applicable for molecular graph representation because cannot incorporate structural information such as the edge and connectivity in graphs. To bridge the gap, advanced Transformer architectures alter the self-attention layer (Maziarka et al., 2020 Ying et al., [02T]] Maziarka et al. 2021)) incorporate message passing networks into Transformer architectures for input feature (Rong et al. 2020). Specifically, MAT (Maziarka et al., 2020) uses adjacency and distance matrix of atoms augment the self-attention layer. GROVER (Rong et al. 2020) runs Dynamic Message Passing Network (dyMPN) over the input node and edge features to extract queries, keys and values for self-attention layers. CoMPT (Chen et al.] (202I) uses the shortest path information between two nodes, and Graphormer (Ying et al. 2021) encodes node's degree, edges and the shortest path with edges between two nodes for molecular data. Although these graph-specific features such as the shortest path or 3D distance are found to be useful in graph representation, these features can be heuristic and impose excessive computational overhead, which limits the model's applicability to large molecules such as proteins. Our proposed model does not require any computationally expensive heuristic features such as 3D information or shortest path for preprocessing or computation in attention layer.
2.2 MOLECULAR SUBSTRUCTURE
Substructure information extracted from molecules has been widely used in molecular generation. property prediction, and virtual screening (Willett al.) 1998| Brown & Martin 1996 Bccket & Bajorath| 2007] Jin et al., 2018). ECFPs (Rogers & Hahn, 2010) encode existing substructures within circular distance from each atom in a molecule. PMTNN (Ramsundar et al. 2015) is a multi-task network that takes ECFPs as an input to predict molecular properties on various datasets.
MACCS keys (Durant et al. 2002) extract molecules substructure depending on the presence of pre-defined functional groups One example of the usage of MACCS keys is to encode known ligands of each protein, which leads to an improvement of prediction performance in protein- ligand interaction (Li et al. 2019). Molecular fragmentation method such as BRICS fragmentation (Degen et al. 2008 Zhang et 202T) or tree decomposition (Jin et al. 2018) is to decompose molecules in non-overlapping partitions with pre-defined rules. BRICS fragmentation divides molecules following chemical reaction based rules. Tree decomposition proposed in Jin et al. (2018) also extracts junction tree by contracting certain edges. One node in the junction tree represents substructure of original molecules. Our model receives molecular substructures as input to supplement GNN's expressivity and it can flexibly encode any substructure vocabulary.
Our model combines Transformer and GNN, with molecular substructures and molecular graph as inputs each network. There existing studies that have also considered combining GNN
3

---

Under review as a conference paper at ICLR 2023
and Transformer architecture (Zhu et al. 2021a bj Yang et al. 2021 to learn graph representations. Specifically, DMP (Zhu et al., 202 la) representation of molecules and train both branches using consistency loss match the two outputs of input molecules. PoseGTAC (Zhu et al., 2021 and GraphFormers (Yang et al. (021) combine enlarge the receptive field
utilizes GNN and Transformer to encode graphs and SMILES
GNN and Transformer layer alternatively Transformer. Our model is the first to encode substructures through Transformer and inject atom features through separate GNN. In this way, we preserve both substructures and local atom features of molecules.
mix the output of
3 MODEL
In this section, we explain the architecture of our model.
Selentntttionn
Overall architecture Figure 2 shows the overall architecture of our model. Our net- Trans-
XM
work consists of two branches: former branch that uses the molecular sub- structures as input and (ii) GNN branch that uses molecular graph as input. The two branches have different roles. First, the Transformer branch is intended to cap- ture global information of molecules. It receives the molecular substructures that have important role molecular prop- erties, but cannot be easily captured by GNNs, and learn the overall representation of molecules. On the other hand, the GNN branch is intended to capture local node in- formation of molecules. The different levels of information are mixed through the fusing network in the Transformer branch.
C1S
CHs
Embedding
Fusing Network
GNN
CHb
Self' Attention
x.V
CH₂
MLP
Subernacturs
SMILES
Figure Overall architecture. Our model consists of Transformer and GNN branches. Transformer encodes molecular substructures, while GNN encodes atomic information. The self-attention and fusing network are alternatively stacked N times, followed by M more self self-attention layer refine the substructure feature. The final CLS token is used to predict the molecular properties.
Transformer branch Our Transformer branch incorporate both molecular sub- structure information and local node fea- tures. The input token for Transformer is the substructure embeddings of molecules. Predefined substructures are first detected and then projected into separate embedding vectors. For example, Figuue 2] substruc- tures such N-Heterocycle, carbon-oxyger bond and methyl group are detected and embedded into learnable embedding vectors. To identify substructure from the input molecules, we use MACCS keys (Durant et al./ (2002), which indicates the presence of motifs in molecule. Note that archi- tecture is not limited to certain molecular substructures, but it can flexibly receive any substructure vocabulary. The embeddings of substructures mixed together and refined they passed through the self self-attention module.
The substructure embeddings after self-attention layer are fused with node embeddings from separate GNN branch. The fusing network computes cross-attention between substructures and nodes where substructures are used as query and nodes are used as key and value. The detailed computation of the fusing network is shown in ingiiuere 3] In the fusing network, the cross-attention between each pair of substructure embedding and node embedding is computed.
To be specific, for a given molecule having atoms and m extracted substructures, we have sub- structure embeddings F, E m and node embeddings E,, Raxd where d is the embedding dimension. Then, the cross-attention is computed as follows:
(E,,)) (F,,
Attention(Q) (Q, K, K,V) V) = Softmax
(1)
ddz
4

---

Under review as a conference paper at ICLR 2023
where Wo. WK, Wy E Raxdx are learnable parameters. The cross coss-attentiinn module outputs E', Rmxd, Through this fusing network, the substructure embeddings aggregate the local information from node embeddings. Structurally important nodes are aggregated with more weights. Instead of designing heuristic weights on the nodes, nodes related to graph-level property by cross-attention. Additionally, as the attention is computed between substructures and nodes, the space and time complexity of the self and cross attention map of our model is linear to the number of atoms, i.e., O(N), whereas other transformer architectures for molecular graphs have quadratic complexity.
model can learn to select structurally important
The self-attention hetween substructures and fus- ing network with node embeddings are repeated iteratively N times. Before making the final pre- diction, we add M self-attention layers at the end of the network for making refinement on the substructures. We add a residual connection from the input tokens to every output of the fus- ing network. This ensures the input structural information last throughout the entire network. Furthermore, we use a CLS token similar to spe- cial classification token in BERT (Devlin et al. 2019) to aggregate the global representation of molecules. The CLS token is shown Figure It is a learnable latent vector and passed to the Transformer branch attached to the substructure tokens as input. CLS token has been used many Transformer- based architectures (Maziarka et al.) [2020] Ying et al.) (021) for molecular representation learning. We attach an MI.P head to the final CLS token to perform graph-level prediction tasks.
Node Fimbeeding
nxd
nx
rxXd
mxd
Atiention Scores
Outputs
Structural Embedding
Figure 3: Illustration of structural and node em- bedding for cross attention computation. Cross attention is computed with the illustrated query, key and value.
GNN branch The GNN branch is used to extract local node features from molecular graphs. For GNN architecture, we use GIN (Xu et al.) 2019) with jumping knowledge (Xu et al. 2018). The computed node features are injected into the Transformer through the fusing network with the same hierarchy. For example, O-hop node representations are injected into the first cross-attention module and 1-hop representations for the second cross-attention module. This allows the model to encode local node features progressively from the shallow to deeper layers.
The computation of node representation through GNNs and injection the Transformer allow us to take advantage of both GNNs and Transformer architectures. Substructures that are hard to be captured by GNNs, but essential to molecular properties, are first detected and encoded through Transformer. Meanwhile, local node information that can be lost in using substructures alone is effectively captured by GNNs and fused with substructures. Additionally, our architecture does not require computationally expensive high-order graph-level information. As Transformer cannot naturally incorporate graph's 's edge connectivity information, existing work (Rong et al.) 2020) Ying al., 202 Chen et al., (021) mainly focuses how to add structural bias such the shortest path or 3D distance between two nodes into the Transformer self-attention computation. However, these structural biases require computationally expensive preprocessing of the molecular datasets. Our network that utilizes GNN as separate branch can avoid these limitations.
4 EXPERIMENT
4.1 EXPERIMENTAL SETTING
Pretraining We pretrain our network obtain molecular representation transferable various molecular datasets and tasks. For pretraining, we extracted 200 real- real-valued descriptors of physico- chemical properties from the pretraining datasets using RDKit (Landrum) 2016) and train our network to predict these properties. As the 200 molecular descriptors include diverse set of molecular properties, the model can learn a representation of molecules that can be used for various downstream tasks.
We collected 1,858,081 number of unlabeled molecules from ChEMBL and PubChem databases (Kim et al. 2016| Gaulton et al. 2012). We preprocessed the pretraining dataset to remove
Dataset
5

---

Under review as a conference paper at ICLR 2023
Table Statistics of eleven datasets from MoleculeNet (Wu et al.
2018 used for downstream tasks.
(a) Classification tasks
(b) Regression tasks
Dataset
Size
Tasks
Metric
Dataset
Size
# Tasks
Metric
ВBBP
2,039
ROC-AUC
FreeSolv
642
1
RMSE
SIDER
1,427
27
ROC-AUC
ESOL
1,128
RMSE
ClinTox
1,478
2
ROC-AUC
Lipophilicity
4,200
1
RMSE
BACE
1,513
ROC-AUC
QM7
6,830
1
MAE
Tox21
7,831
12
ROC-AUC
QM8
21,786
12
MAE
ToxCast
8,575
617
ROC-AUC
duplicated molecules in the pretraining dataset and keep the downstream dataset intact. Further details of pretraining dataset collection procedure are presented in [Appenddx D ChEMBL and PubChem are large-scale databases that include a variety of chemical and physical properties, and biological utilize MACCS key (Durant al.
activities of molecules. To obtain molecular substructures, 2002), a 166 dimensional vector that indicates a presence of certain substructure in molecules, and extract this for every molecule using RDKit (Landrumi 2016). We use OGB package (Hu et al. 2020) to convert SMILES (Weininger 1988), a text-representation for molecules, to molecular graphs
Implementation details We set M 4 and N 3, where M and N are defined in|section 3 We set 508-dimensional hidden units and 16 attention heads. When pretraining, we used the AdamW optimizer (Loshchilov Hutter 2018) with a learning rate of le-4. We divide the pretraining dataset into a 9:1 ratio and use them for training and validation sets. The model is trained for 10 epochs, and the model with the best validation loss is used for downstream tasks. Further details for pretraining setting is presented in Appendix D
4.2 DOWNSTREAM TASKS
Tasks We evaluate the performance on six classification tasks (BBBP, SIDER, ClinToxx BACE, Tox21, and ToxCast) and five regression tasks (FreeSolv, ESOL, Lipo, QM7 and QM8) from MoleculeNet (Wu et al., 2018). The dataset statistics are summarized in Table I Each task is related molecular property from lewelevel, for example, water solubility in ESOL to high-level, possibility of blood-brain barrier penetraion in BBBP. Further details about the downstream tasks are available Appendix B
Experimental setting To evaluate each model, we use 3 different scaffold splits (Wu et al. 2018) following Rong et al. (2020). Scaffold split divides structurally different molecules into different subsets and provides more challenging and realistic test environment. From the pretrained model, we replace the last MLP layer of the network with the task-specific MLP heads. For each downstream dataset, train our model for 100 epochs and report the test score corresponding validation epoch. We tune hyperparameters with Bayesian optimization search with a budget 100 for learning rate, dropout, weight decay and the number of last prediction heads. The hyperparameter search range provided in AAppnddxx D
the best
We compare the performance of model on the downstream tasks with several GNN and Trans- former based tate-off-the-arts approaches for molecule representation learning. TF-Robust (Ram- sundar al., 2015) is DNN-based model that takes molecular fingerprints. GNN-based models include GraphConv (Kipf & Welling 2017), Weave (Kearnes et al., 2016) and SchNet (Schütt et al.
2017) which are graph convolutional networks, MPNN (Gimmrr al. 2017), DMPNN ( (Yang et al. 2019). MGCN (Lu et al. 2019) and CMPNN (Song et al. 2020) which are GNN models considering
the edge features during message passing. AttentiveFP (Xiong et al. 2019) is an extension of graph attention network for molecule representation.
coansformer-based models include GROVER (Rong cet al. 2020), MAT (Maziarka et al. 2020), Graphormer (Ying et al., 202 I) and CoMPT (Chen et al. 2021). Among the baselines, N-GRAM (Liu et al. 2019)) Hu et al. (2019). GraphLoG (Xu et al.
2021). MAT, GROVER, Graphormer, GEM (Fang et al. (022) and MPG (Li et al. 2021) are models that pretraining strategies. We report GROVER base model for fair comparison in terms of the number of parameters. To reproduce the results for models using pretraining strategies, we use
the pretrained model made available by the authors. We reproduce the results of MPG due to the

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison on small-scale datasets. We report the average and standard deviation (in brackets) over three splits, We mark the best and the second-best performances in bold yellow and light yellow, respectively.
Classification Tasks
Method
BBBP
SIDER 1
ClinTox 1
BACE f
Tox21 f
ToxCast t
Rank
TF_ Robust Ramsundar al.
2015
.600.m
607.000
.765,mm
.24..m
698
.585
17.5
Weave (Kearnes
2016
837,mm
_543.o4
823
799m
.74
.678.20
18.5
GraphConv (Kipl & Welling
2017
877mm
.593,n0
845
.84..m
.772mm
.650kmm
15.2
SchNet Schüll et al. 20171
847ume
.545.000
.717
.750umm
.767
.679
19.2
MPNN
.13,m
.595.cc
.879um
8155m
808
12.7
DMPNN Yang el 2019
9919, 19
.632.uch
.897
.852uss
826
.718,am
6.0
MGCN Lu et al. 2019
850use
.552.mm
.644psz
.344
.707um
.663,,00
20.2
Attentive Xiong ot
2019
908k290
.605.u0
.933um
.63mm
.07
.579,00
11.5
CMPNN Song
.400.om
.612:000
.311m0
868,029
.85
.722,005)
5.5
CoMPT Chen et al 2021
930kpm9
605.o0
8 818mm
85 (049))
.790
.716
10.2
Molformer Wu ot
.266ua7
616:015
674gs54)
8842m2
.773mms
.688,009
13.2
N-GRAM Lio al
2019
9120m
6322.00
855 (57
876.mm
769
9.4
Hu. ct.al Hu al. 2019
9155p
614:000
762.us
(027))
811
.714.0
10.2
MAT' (Maziarka 2012
.92,,0m
617:
885
830
.8 $10,ors
.712.
9.3
GROVER Kong et N0A0
936,00
656..00
925ini3)
.78.0.9
819
.2.3
2.3
Graphormet Ying al. 2012
.38mmm
625
.913mm
.48.am
(01
.718co07)
6.8
GraphL LGG
al.
913,m
.95,.00
.749.mm
845.om
.773.mm
.677
15.5
GraphMVP Liu
.923um
.617umm
7740pmm
.850
.788
.6944.
11.5
MGSSI Zhang et
.92mmm
616,
.763mm
847.m
.795mm
.695(.m
12.5
MPG* cl 2021
.922um
.288un
864umz
.800ne
.712.00m
7.8
GEM inggen
.92 Lumo
.033,
882,am
8815mm
.720k
7.6
Ours
.34,m1
646:100
.935 (M
877umm
.29
.730
1.8
Regression Tasks
Method
Pre.
FreeSolv 4
ESOL
Lipo
QM7
QM8
Rank
TF_ Robust RRamsundar al
2015
4.122
1.722.
909
120 220.68..
.024
15.2
Weave
Keames et al. 2016
2.398.
1.158
813umm
94.7mm
.022 sm))
11.8
Graph Lov [Knn & Welling
2017
2.900
1.068 oms
.712.om
118.9 118.9
.021c
12.2
SchNel Schuit al
2017
3,215
1.045.
909
74.2mmm
.020 000
11.4
MPNN Gilmer
201
28155.m
1.167
.672.0m
113.0una
.01.5.
10.0
DMPNN
Yang of ml
2019
2. 2.777.9
.900 258)
653mmam
105 8
.0143
7.8
MGCN La al. 2019
3.349.am
1.266
1.1 11u4t
77. 77.6am
.022, 203)
13.6
AttentiveFP (xrong et
2019
2.030
.853 se00
.650mmm
126.
.0282, z01)
8.8
CMPNN Song
X0A0
2,254.159
881
606.aam
70.60m
.01 36:
4.8
CoMPT Chen ci 2021
225
.98
632
65.311.0
.0145..22)
5.6
N-GRAM iiu al. 2019
2,512,
1.100 1000
876mm
125 61.5)
.0320 063)
14.0
MAT (Mazirrka et al 20201
2.116.152)
833
668um25
93. 6.1.5
.01 0128
6.6
GROV FF [Rong
1.592
.888
.563.am
72.5m
.0172,
4.4
Graphorner (Ying
2.089
827
.64mmm
171,300
.0140
7.2
MPG Li ot al 2021
2.44.500
926,
682(mm
10.7
GFM Fang at 2022
2.21,500
.885
.617com
58.9
35: 2013
4.4
Ours
1.750
.82 073)
.575mm
63.5p.4
.0134.00 203)
1.6
Table 3: Results on ZINC.
Table 4: Results on OGBG-MoIHIV.
Method
test MAE
Method
AUC
GAT
Velickovid al. 2018
.384
GCN+VN
Kipf & Welling 2017
75.99tu.
GIN
No
.387
GIN-VN No ol 2019
77.07
PNA
Cosso al 2020
(188uea)
PNA Curso al 220
79. 19.05
Graph Transfoenker Diivedd Brewsn 2020
.2261s44
Beami
79.70mmm
Transformer RWIE Dinvadi et a
.310
Graphormet + FLAG Ying et al. 2021
80,51
SAN Kreuzer al.
.139
Krewxer
2021
77.85m03
Graphomer
221
122
GSN [Bounitss
2022
77.99kmm
K- K-Subgraph SAT
Chen
.994ue
HMGNN
PPN You Goo
80.20
80.80ku.0
different splits used in Li et al. 2021 MPG and the results can be found in the appendix.
We also evaluate our model with the same data split used for
Note that there is a mismatch between the splits used in the original paper of MPG and the one used in the author's 's repository. For both cases, our model performs better than MPG under the same splits. The additional experiments are available in Appendix G
7

---

Under review as a conference paper at ICLR 2023
Lipophilicity (U)
QM8 (L)
BBBP (1)
ToxCast (1)
0.85
0.02
00191
0.94
0.73
0.75
0.738
0.018
0.0166
0.92
0.924
0.72
0.72
0.016
0.907
0.71
0.71
0.65
0.549
0,0143
0.9
0.596
0.583
0.014
00138
0.55
0.012
0.69
Withour GNN
End-Concat
Begin-Concat
Random Embedding
Ours
Figure 4: Ablation study. Four variations of our model (Without GNN, End-Concat, Begin-Concat, Random Embedding) on four downstream datasets. The results show the necessity of each component of our model.
ToxCast (1)
0.75
20
R,
0.701
0.706
0.7
0.669
0.856
0.65
-10
0.6
GCN Oursges
OursoN
-20
-10
20
-20
GIN
Ours
Figure Comparison between standard three GNNs and our model on ToxCast dataset.
Figure 6: t-SNE embedding of molecules with different sub- structures.
Results Table shows the overall results of the baselines and our model on 11 MoleculeNet datasets. Our model achieved the best performance on five downstream tasks: ClinTox, Tox21, ToxCast, ESOL and QM8, and the second best performance on five downstream tasks: SIDER, BACE, FreeSolv, Lipo and QM7. We also computed the average rank for the classification and regression tasks, separately. Our model achieved the best average rank among all compared models. The result shows that our model generalizes well across different downstream tasks, which means local information aggregated from GNN can propagate globally as interacting with structural information in Transformer. We find that some substructures weight high attention nodes consisting of the substructures, and CLS token used for prediction also focuses on the substructures from attention scores in self-attention.
We further evaluate model two
molecular property prediction benchmarks, OGBG- MolHIV (Hu et al. 2020) and ZINC (Dwivedi et al.) (222)). For ZINC, following the setting in Chen al. (2022), trained additional slim model with total parameter of 381,737 and used RWPE (Dwivedi et al.] (02T) added to atom features. Table and TTble 4 summarize the performance of our model. On both datasets, our model consistently outperforms previous Transformer and GNN based models.
4.3 ABLATION STUDY
We report ablation study to justifyy each component and flexibility of our model architecture. For the ablation study, we reduce the hyperparameter search space to only learning rates ( (le-3, 5e-4, le-4, 5e-5, 1e-5, 5e-6) to facilitate the comparison between different models. We report the test scores based on the hyperparameters from the best validation scores.
Ablation on model components Figure 4]shows the performance comparison between four varia- tions of our model on four downstream datasets. We first verify the performance of our model without GNN branch. To do that, we replace all cross-attention layers with self-attention layers and exclude GNN branch. Begin-Concer and End-Concat examine different ways of combining substructure and local features. Begin- Concat runs vanilla- toailll--rrfffommr encoder on top of a concatenated atom and substructure embeddings. End-Concat and concatenates the substructure feature at the end to make the final prediction. Random Embedding examines the effect of substructural embeddings by replacing them with random learnable embed- dings. Figure 4 shows our model outperforms other variations on all datasets, which justifies the necessity of each component.
vanilla-transfomea encoder top of atom embeddings
8

---

Under review as a conference paper at ICLR 2023
Different GNN branch Our model can flexibly utilize other GNN architectures our GNN branch. We test the changes in performance when our model design is applied with different GNN architectures. Figure 5) shows the comparison between commonly-used GNNs, i..., Graph Convolutional Network (GCN), Graph Attention Network (GAT) (Veličković et al. 2018 and GIN, and models with the GNN branch switched to each corresponding GNN ToxCast dataset. There is a significant performance improvement when our model is adopted to each GNN model. Performance comparison when utilizing 3D-aware GNN is presented in Appendix F
4.4 ANALYSIS
Effectiveness of substructures Figure 6 shows the t-SNE embeddings of molecules with different substructures. Our model discriminates molecules with similar substructures (aromatic rings with or two different atoms) better than GIN.
shows the attention weights in a cross-attention and self-attention layer. In the cross-attention layer, each row and column correspond to substructure and node in the given molecule, we can interpret the at- tention score as the degree of focus between the substructures and nodes. We check where the CLS token gives more attention since it is used for prediction. We find out the CLS token gives strong attention to specific substructures. More interestingly, often substructure gives more attention on the nodes that consists the substructure itself. Despite of not having any structural information between input substruc- tures and nodes as input, our model can identify structurally related nodes in the cross attention layer, showcasing the ability of understanding molecular structure.
Attention map Figure 7
Alsnn bolees
Atorn Iokene
(a) Ring-Chaing Chain-Rinngggg bond
(b) Chlorine atom
Figure 7: strongly attends to Ring-Chain Ring bond and Cl substructures of the input molecule. The cross at- tention maps show the substructures capture the atoms related to the substructure.
Attention visualization. CLS token
Computation time for graph features Figure shows the time required in millisecond compute the MACCS keys, shortest path and 3D distance for each molecule with different number of atoms. As shown the figure, the time required for 3D distance and the shortest path increases dramatically the number of atoms in- crease, which make these features impossible to be applied in large molecules. However, identifying the MACCS key substructures does not depend on the input molecule's ' number of atoms.
13017
MACCS Age
Skuttor Donace
1o'
3D Distuee
Time (ms)
10"
351
10'
10
100
200
300
400
Number atoooms
Figure 8: Time required to compute each feature per molecule.
5 CONCLUSION
In this paper, we propose a novel framework that incorporates Trans- former and GNN architecture for molecular representation learning. Our model takes advantages of the two architectures to aggregate substructure and local information. With the cross attention mechanism in fusing network, our model could achieve state-of-the-arr performance on various molecular property prediction benchmarks. Overall, our work highlights the effectiveness of SACA for molecular representation learning. We plan release the source code and pretrained neural networks in the public domain.
Limitation and future work While we show the effectiveness of our model to predict molecular property, the proposed approach has not yet been validated with large-scale molecular graph such as proteins. Applying or modifying our model to proteins which contain 5,000 to 50,000 atoms would be an interesting direction for the future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Uri Alon and Eran Yahav. On the Bottleneck of Graph Neural Networks and its Practical Implications. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Dominique Beaini, Saro Passaro, Vincent Létourneau, Will Hamilton, Gabriele Corso, and Pietro Liò. Directional graph networks. In International Conference on Machine Learning, pp. 748-758 PMLR, 2021.
Cristian Bodnar, Fabrizio Frasca, Nina Otter, Yuguang Wang, Pietro Lio, Guido F Montufar, and Michael Bronstein. Weisfeiler and lehman go cellular: Cw networks. Advances in Neural Information Processing Systems, 14:2625-26400 2021.
Giorgos Bouritsas, Fabrizio Frasca, Stefanos P Zafeiriou, and Michael Bronstein. Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Transactions Pattern Analysis and Machine Intelligence, 2022.
Robert D Brown and Yvonne C Martin. Use of structure- activity data to compare structure based clustering methods and descriptors for in compound selection. Journal of Chemical Information and Computer Sciences, 36(3):572-584 1996.
Hanxuan Cai, Huimin Zhang, Duancheng Zhao, Jingxing Wu, and Ling Wang. Fp-gnn: a ver- satile deep learning architecture for enhanced molecular property prediction. arXiv preprint 2rXiv:2205.03834, 2022.
Dexiong Chen, Leslie O' O'Bray, and Karsten Borgwardt. Structure-aware transformer for graph representation learning. In International Conference on Machine Learning, pp. 3469-3489. PMLR, 2022.
Jianwen Chen, Shuangjia Zheng, Ying Song. Jiahua Rao, and Yuedong Yang. Learning attributed graph representations with communicative message passing transformer. In IJCAI, 2021.
Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Liò, and Petar Veličković. Principal neighbourhood aggregation for graph nets. Advances Neural Information Processing Systems, 33: 13:00-111337,, 2020.
Tomasz Danel, Przemyslaw Spurek, Jacek Tabor, Marek Šmieja, Lukasz Struski, Agnieszka Stowik, and Lukasz Maziarka. Spatial graph convolutional networks. In International Conference on Neural Information Processing, pp. 668-675. Springer, 2020.
Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34 797,, 1991.
Jörg Degen, Christof Sigscheid-Gerladee Andrea Zaliani, and Matthias Rarey. On the art of compiling and using' drug-like chemical fragment spaces. ChemMedChem: Chemistry Enabling Drug Discovery, 3(10): (113)):503-150000 2008.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference Learning Representations (ICLR), 2020.
Joseph L Durant, Burton A Leland, Douglas R Henry, and James G Nourse. Reoptimization of MDL keys for use in drug discovery. Journal of chemical information and computer sciences, 2002.
Vijay Prakash Dwivedi and Xavier Bresson. generalization transformer networks to graphs. arXiv preprint arXiv:2012.09669 2020.
10

---

Under review as a conference paper at ICLR 2023
Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.
Vijay Prakash Dwivedi, Anh Tuan Luu, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Graph neural networks with learnable structural and positional representations. arXiv preprint 1r000:2110.07875. 2021.
Hanna Eckert and Jürgen Bajorath. Molecular similarity analysis in virtual screening: foundations, limitations and novel approaches. Drug discovery today, 12(5-6))225-233 2007.
Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers. Marco Fiscato, and Mohamed Ahmed. Molecular representation learning with language models and domain-relevant auxiliary tasks. arXiv preprint arXiv:201 13230, 2020.
Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geometry-enhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127-134, 2022.
Yin Fang, Haihong Yang. Xiang Zhuang, Xin Shao, Xiaohui Fan, and Huajun Chen. Knowledge- aware contrastive molecular graph learning. arXiv preprint 2tt::22003.30445 2021.
Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, Bissan Al-Lazikani, et al. Chembl: a large-scale bioactivity database for drug discovery. Nucleic acids research, 2012.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovee. Strategies for Pre-training Graph Neural Networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint urXiv:2005.00663 2020.
James P Hughes, Stephen Rees, S Barrett Kalindjian, and Karen L Philpott. Principles of early drug discovery. British journal of pharmacology, 162(6):1399 1249, 2011.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction tree variational autoencoder for molec- ular graph generation. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, pp. 595-608, 2016.
Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. PubChem substance and compound databases. Nucleic acids research, 2016.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. Proceedings of the International Conference on Learning Representations (ICLR), 2017.
Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. Advances in Neural Information Processing Systems, 34:21618 00166182211 616-21629, 2021.
Greg Landrum. Rdkit: Open-source cheminformatics software. 2016. URL https: /ttps:///qithi. Som./rdkitt/rdkit//eeeeeeess
ease 201 2016_09_4
11

---

Under review as a conference paper at ICLR 2023
Li Li, Ching Chick Koh, Daniel Reker, JB Brown, Haishuai Wang, Nicholas Kcone Lee, Hien-haw Liow, Hao Dai, Huai-Meng Fan, Luonan Chen, et al. Predicting protein-ligand interactions based bow- www-pharmacoloiical space and bayesian additive regression trees. Scientific reports, 9(1):
1-12, 2019.
Pengyong Li, Jun Wang, Yixuan Qiao, Hao Chen, Yihuan Yu, Xiaojun Yao, Peng Gao, Guotong Xie, and Sen Song. An effective self- superviee framework for learning expressive molecular global representations to drug discovery. Briefings in Bioinformatics, 22(6):bbab109, 2021.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI conference artificial intelligence, 2018.
Shengchao Liu, Mehmet F Demirel, and Yingyu Liang. N-gram graph: Simple unsupervised repre- sentation for graphs, with applications to molecules. Advances in neural information processing systems, 32, 2019.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. arXiv preprint urXiv:2110.07722 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- ence on Learning Representations, 2018.
Chengqiang Lu, Qi Liu, Chao Wang. Zhenya Huang, Peize Lin, and Lixin He. Molecular property prediction: A multilevel quantum interactions modeling perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp- 1062-1060, 2019.
tukasz Maziarka, Tomasz Danel, Stawomir Mucha, Krzysztof Rataj, Jacek Tabor, and Stanistaw Jastrzebski. Molecule attention transformer. arXiv preprint arXiv:2002.08266 2020.
Lukasz Maziarka, Dawid Majchrowski, Tomasz Danel, Piotr Gainski, Jacek Tabor, Igor Podolak, Pawel Morkisz, and Stanislaw Jastrzębski. Relative Molecule Self-Attention Transformer. arXiv preprint 1rXiv:2110.05844 2021.
Richard C Mohs and Nigel Greig. Drug discovery and development: Role of basic biological research. Alzheimer's & Dementia: Translational Research & Clinical Interventions, 3(4):651-657) 2017.
Christopher W Murray and David C Rees. The rise of fragment-basee drug discovery. Nature chemistry, 2009.
Bharath Ramsundar, Steven Kearnes, Patrick Riley, Dale Webster, David Konerding, and Vijay Pande. Massively multitask networks for drug discovery. arXiv preprini arXiv: 150.002072. 2015.
David Rogers and Mathew Hahn. Extended-connecttvvg fingerprints. Journal of chemical information and modeling, 2010.
Yu Rong. Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. GROVER: Self-supervised Message Passing Transformer on Large-scale Molecular Data. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre
Tkatchenko, and Klaus-Robel Müller, Schnet: continuous-fin filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Ying Song, Shuangjia Zheng, Zhangming Niu, Zhang-Hua Fu, Yutong Lu, and Yuedong Yang. Communicative representation learning on attributed molecular graphs. In IJCAI, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. Proceedings of the International Conference on Learning Representations (ICIR), 2018.
12

---

Under review as a conference paper at ICLR 2023
David Weininger. Smiles, a chemical language and information system. 1. introduction to methodol- ogy and encoding rules. Journal of chemical information and computer sciences, 1988.
Boris Weisfeiler and Andrei Leman. The reduction of a graph to canonical form and the algebra which appears therein. N7I, Series, 2(9):12-16, 1968.
Peter Willett John M Barnard, and Geoffrey M Downs. Chemical similarity searching. Journal of chemical information and computer sciences, 38(6):983-996 1998.
Fang Wu, Qiang Zhang, Dragomir Radev, Jiyu Cui, Wen Zhang, Huabin Xing, Ningyu Zhang, and Huajum Chen. 3ddtransformer: Molecular representation with transformer 3d space. arXiv preprint arXiv:2110.0119)) 2021a.
Fang Wu, Qiang Zhang, Dragomir Radev, Jiyu Cui, Wen Zhang, Huabin Xing, Ningyu Zhang, and Huajun Chen. Molformer: Motif-based transformer on 3d heterogeneous molecular graphs. arXiv preprint arXiv:2110.01191, 2021
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 2018.
Zhaoping Xiong, Dingyan Wang, Xiaohong Liu, Feisheng Zhong, Xiaozhe Wan, Xutong Li, Zhaojun Li, Xiaomin Luo, Kaixian Chen, Hualiang Jiang, et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism. Journal of medicinal chemistry, 63(16):8749-8766 2019.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks, In International Conference on Machine Learning, pp. 5453-5462. PMLR, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pp. 11548-11555 PMLR, 2021.
Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, $4:28798-288 28,10, 2021.
Kevin Yang, Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, et al. Analyzing learned molecular representations for property prediction. Journal of chemical information and modeling, 2019.
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do Transformers Really Perform Bad for Graph Representation? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Zhaoning Yu and Hongyang Gao. Molecular representation learning via heterogeneous motif graph neural networks. In International Conference on Machine Learning, pp. 25581-25594 PMLR, 2022.
Zaixi Zhang, Qi Liu, Hao Wang, Chengqiang Lu, and Chee-Kong Lee. Motif-based graph self- supervised learning for molecular property prediction. Advances in Neural Information Processing Systems, 11115870-15822, 2021.
Jinhua Zhu, Yingce Xia, Tao Qin, Wengang Zhou, Houqiang Li, and Tie- Yan Liu. Dual-view molecule pre-training. arXiv preprint arXiv:2106. 2002244406.. 2021a.
Yiran Zhu, Xing Xu, Fumin Shen, Yanli Ji, Lianli Gao, and Heng Tao Shen. Posegtac: Graph transformer encoder-decoder with atrous convolution for 3d human pose estimation. In IJCAI, pp. 1359-1365, b02
13

---

Under review as a conference paper at ICLR 2023
A NODE AND EDGE FEATURES
In this section, we present the node and edge features of molecules used for GNN branch. We used OGB package (Hu al., 2020) convert SMILES strings (Weininger 1988) molecular graphs. The molecular graphs are encoded through GIN (Xu et al. 2019) and injected into the Transformer branch by the cross-attention. The molecular graphs have the following node and edge features.
Node features.
Each node has the following 9 dimensional features as shown in Table 5
Table 5; Node Features.
Index
Description
Range
Atomic num Chirality
[1. 118), other
unspecified, tetrahedral ew, tetrahedral cew, other
Degree Formal Charge
[0, 10], other
[-5, 5], other
Num Hydrogen
[0, 8). other
5
Num Radical Electron
[0, 4]. other
Hybridization
SP, SP2, SP3, SP3D, SP3D2, other
Is Aromatic
False, True
8
Is in Ring
False, True
Edge features.
Each edge has the following 3 dimensional features shown in
Table 6
Table 6: Edge Features.
Index
Description
Range
0
Bond Type
single, double, triple, aromatic, other stereonone, stereoz, stereoe, stereocis, stereotrans, stereoany False. True
1
Bond Stereo
2
Is Conjugated
14

---

Under review as a conference paper at ICLR 2023
B DETAILS OF DOWNSTREAM DATASETS
We used 11 binary graph classification and regression datasets: BBBP, SIDER, ClinTox, BACE, Tox21, ToxCast, FreeSolv, ESOL, Lipophilicity. QM7 and QM8 from Moleculenet (Wu et al. 201 8). The details each dataset are shown in Table Through the various datasets, we can test the generalization ability of our pretrained model.
Table Detailed description for each downstream dataset.
Pattsset
Descripion
B8BP
Binary classilication predict molecule's blood-brain barnerr penctration ability
SIDER
Marketed drugs with its adverse drug reactions
ClinTox
Qualitative data of upproved the HDA and those that have failed clinical trials for toxicity
Binary classification
predict
binding
Tox21
Qualitative taxicity measurements on 12 biological targets
ToxCast
Toxizology data for large library of compounds based on in vitro high throughput screening. including experiments on over 600 tasks Regression task predict bydration free energy small molccules water
FreeSoly
predict Experimental results of octanol/water distribution coefficient
Regression
solubility terms log sobability
per litre
Lipophilicity
QM7
A subser of GDB-13 13 composed of all molecules of to 23 atoms (inchuding 7 heavy atoms N, O, and S), toralling 7165 molscules Computer generated quanuum mechanical properties
QM8
C MOLECULAR STRUCTURAL KEYS
In this section, explain the details of molecular substructures that used for model, i.e., Molecular ACCess System (MACCS) keys (Durant et al., (2002). We extract MACCS keys for each molecule using RDKit package (Landrum 2016). MACCS keys have 166-dimensional features where each binary label indicates the presence of a particular substructure in the given molecule. For example, the 139th index of MACCS keys indicates the presence of the Hydroxy group (-OH), and the 162nd index indicates the presence of the aromatic ring in molecule. The full list of 166 MACCS keys can be found in the document
HO
0000000000
00000000000000000000000000000000
0000000000
00000000000000000000000000000000
0000000000
01100110001100110000000000000011
0000001000 000
1000
00001000000001000000000000000000
011110
111110
Figure 9: MACCS keys encoding.
Figure showw examples how MACCS keys encode substructure information into a binary bit string. The canonical SMILES representations of the left and right molecules are C1=CC=C(C=C1)0 and ccC CNC(CCC))CCCC())111 cocs1, respectively.
Table 8 shows the MACCS keys statistics for pretraining and downstream datasets. It includes the average and the median number of MACCS keys that molecules in each dataset contain. Also, the minimum and the maximum number of MACCS keys for each dataset are reported.
15

---

Under review as a conference paper at ICLR 2023
Table 8: MACCS keys statistics for pretraining and downstream dataset.
Dataset
Average
Median
Min
Max
Pretraining dataset
52.12 + 13.33
53
1
106
BBBP
46.03 14.48
46
2
SIDER
46.61 + 17.93
46
1
105
Clin'ox
46.13 16.54
2
BACE
61.04 + 12.53
60
21
93
Tox21
32.73 + 16.63
30
2
99
ToxCast
33.50 = 17.30
31
2
101
FreeSolv
15.26 + 9.78
13
1
62
ESOL
23.63 + 15.52
19
1
76
Lipophilicity
51.61 + 14.26
52
7
93
QM7
18.75 7.80
18
50
QM8
23.00 + 8.10
23
48
D EXPERIMENTAL SETTING
In this section, we explain further details of our experimental setting.
Pretraining. The default hyperparameters for pretraining are listed in Table 9 All attention layers have 16 attention heads and 768 hidden dimension. Our model has 41M parameters.
Table 9: Model configurations and hyperparameters for pretraining.
Parameter
M
N
3
Model hidden dimension
768
FFN inner-layer dimension
768
of attention heads
16
Learning rate
0.0001
Epoch
10
Dropout
0.1
Batch size
32
Hyperparameter Search Range. For each downstream task, we search for the best hyperparameter combinations. We perform the Bayesian optimization over the validation set and use the hyperparam- eters for the best validation score report the test score. The hyperparameter range that searched over is shown in Table 10
Table 10: Finetuning hyperparameter search range.
Hyperparameter
Description
Range
Learninng rate # of MLP layers
The learning rate The number of last MLP layers Dropout ratio Weight decay
0.000001 0.001
1, 2, 3
Dropout
0.0 0.5
Weight decay
0.0, 0.001, 0.0001, 0.00001
16

---

Under review as a conference paper at ICLR 2023
Pretraining Task. To pretrain our network, extract the 200 real-valued descriptors for each molecule using RDKit package (Landrum, 2016). This task is proposed by (Fabian et al. (020). Through this task, can make the model to learn the physicochemical properties of the input molecules.
Details to obtain pretraining dataset. We describe the pretraining dataset collection procedure. (I) We collected 9,37 unlabeled molecules from ChEMBL and PubChem databases. (2) We canonicalized the SMILES using RDKit obtain unique representation of molecules and remove all duplicated molecules in the pretraining dataset. (3) We retrieved all overlapping molecules between the pretraining and downstream datasets. (4) We removed the overlapping molecules from the pretraining dataset and keep the downstream dataset intact, Such procedure results in 2,271,376 413,295 = 1,858,081 molecules in the pretraining dataset. Table II shows the statistics of overlapping molecules between each downstream dataset and the originally collected pretraining dataset.
Table 11: Overlapping molecules between original pretraining dataset and each downstream dataset.
BBBP
SIDER
Clintox
BACE
Tox21
ToxCast
FreeSolv
ESOL
1ipo
QM7
QM8
# mols
2039
1427
1478
1513
7831
8576
642
1128
4200
6830
21786
Overlapping mols
1029
956
631
3
7105
8243
459
826
1260
529
869
Ratio
50.5%
67.0%
42.7%
0.2%
90.7%
71.5%
73.2%
30.0%
7.7%
4.0%
E PARAMETERS AND SPACE COMPLEXITY OF DIFFERENT MODELS
We present the number of parameters of transformer- molecular representation learning models in Table 12, Except for CoMPT, our model has a comparable or lower number of parameters than other wransformer-based models.
We also present the space complexity in Table 12
The space complexity for our attention computation linear to the number of atoms (i.e., O(z) where n is the number of atoms). This is because the number of MACCS key, the substructure that is used in our model, is bounded by 166. On the other hand, the space complexity of other transformer- based molecule representation learning models is quadratic to the number of atoms (i.e., O(n?)).
Table 12: Parameter and complexity comparison between different models.
Parameters
Space Complexity for Attention
GROVER (Rong al.
2020
48M
O(n))
Graphormer (Ying 2021
47M
O(n2)
MAT Maziarka et al 2020)
42M
(nn))
CoMPT (Chen et al.
2021
2.7M
Onn))
Ours
41M
O(n)
F ADDITIONAL ABLATION STUDY
Change GNN branch to 3D-aware GNN To observe the effect of the choice of GNN architecture for GNN branch, we conduct additional experiments with 3D-aware GNN. We changed our GNN branch from GIN to SGCN (Danel et al. 2020) that utilizes 3d coordinates of input molecules. Please note that the two models are not pre-trained and instead trained on the downstream dataset from scratch. Figure 10 shows that the model with 3D-aware GNN shows better performance than our original model on QM7 dataset whose task is closely related to 3D information. The result shows the flexibility of our framework that can be further tuned by using tppr-appe GNN architecture.
17

---

Under review as a conference paper at ICLR 2023
Other substructures input tokens for Transformer Despite of the fact that our model utilizes MACCS keys for the input of Transformer, model flexibly receive any substructure vo- cabulary. We conduct experiments using other substructures from ECFP fingerprints, Tree decomposition and Pubchem Fingerprint. 70 Table 13 summarize the results. While other substructures achieve competitive performance on most of the downstream tasks, MACCS keys shows the best performance on average.
QM7 (L)
75
71.2
65
We presumed the performance decrease when using ECFP or Tree decomposition as follows. First, they do not encode explicit pre- defined functional groups as their motifs. Particularly, ECFP defines motifs as subgraph within a certain diameter around each node. This mechanism similar to how GNN encodes node representations they both see the local subgraph information around each node. As we utilize GNN branch, as separate branch, the use of ECFP fingerprint might not bring new information. Secondly, the possible number of substructures for data-driven motifs are much larger than MACCS key. Specifically, ECFP commonly encodes molecules into 1024 bits, which might make challenging for the model to learn the relationship between each substructure and node. Also, Pubchem Fingerprint defines larger substructure vocabulary of 881 than MACCS key. The result shows that larger vocabulary does not always bring performance improvement.
61.2
60
OUrSGIn
OuTSSGON
Comparison of our model where the GNN branch is replaced with 3D GNN.
Figure 10:
Table 13: Comparison between different substructures applied to our model.
Method
BBBP
BACE f
Tox21 f
ToxCast
FreeSolv
ESOL +
Lipo J
QM8
MACCS key
0.934
0.868
0.818
0.725
2.00
0.878
0.582
0.0140
ECFP 4
0.925
0.869
0.818
0.716
2.52
0.900
0.596
0.0152
ECFP 6
0.903
0.861
0.818
0.709
2.30
0.949
0.592
0.0151
Tree Decomposition
0.925
0.848
0.796
0.715
2.37
0.885
0.614
Pubchem Fingerprint
0.922
0.853
0.802
0.709
1.83
0.977
0.719
G PERFORMANCE ON MPG SPLIT
As MPG uses different splits to GROVER on the paper, we reproduce result our model on same split that MPG used for fair comparison. As shown in Table 14 our model outperforms MPG on most of the downstream datasets.
Table 14: Comparison between MPG and our model on the split from MPG paper.
Method
BBBP
SIDER 1
Clintox
BACE
Tox21
ToxCast f
FreeSolv +
ESOL J
Lipo 4
MPG
0.922
0.661
0.963
0.920
0.837
0.748 0.748
1,269 1.278
0.741
0.556
Ours
0.943
0.665
0.958
0.931
0.847
0.719
0.546
H MODEL VARIANTS
In this section, we illustrate the difference between 4 variants of models that utilize molecular substructures, as shown in Figure 11 The first model is Extra Atom-level feature, which concatenates substructure features to atom features that make up the substructure, and forwards them through the Transformer. The second model Begin-Concat, which tokenizes substructures along with atom tokens and uses them input for the model. The third model is End-Concat, and this model forwards only atom tokens through Transformer, followed by concatenating substructure feature before the last fully-com cally-connected layer. The fourth model is proposed model, which forwards substructure tokens through Transformer, node representations through a separate GNN and mixes these through cross attention.
18

---

Under review as a conference paper at ICLR 2023
Self Attention
Self Attention
Self Artention
Cross Attention
CHI
Self Atlention
Extra Atom-level feature
(b) Begin-Concat
(c) End-Coneat
Ours
Figure 11: Model variants that incorporate molecular substructures and atoms.
These different ways of mixing substructures and atoms have been explored in existing work (Fang
al. 2021: Cai et al. 2022 Wu et al., 2021a) and the first three variants have some limitations.
First, Extra Atom-leve feature makes it difficult for the model to separate the semantic meanings of substructures from atoms only. For example, the Hydroxy group (-OH) is important by itself, but is hard to obtain this substructure information from hydrogen and oxygen atom features implicitly. Second, Begin-Concar, the direct concatenation of substructure and atom tokens has limitation that the model does not differentiate the two different types of tokens. A full self attention is computed between concatenated substructure and atom tokens at the same level. Lastly, End-Concat, there interaction between substructures and atoms except for the last linear layer. In subsecccio 4.3) we have also shown Begin-Conee and End-Concat have poor performance than our model, which fuses two types of information through cross attention. This exploration of model variants shows our model proposes a good way to mix the two types of information.
I OTHERS
URLs for pretrained model for baselines. We use the following sources for pretrained models to reproduce the results.
Graphormer:
MAT: Mt/.rppp///athhhb.coo//rr/g///1/
GraphLoG: http:///wwt.sctttcooo///eeeennnr
http:///fttiiicce//ddddd/////ald preetrainad_comppooo/hheeeee
GEM:
19


Reviews:
[{"review_id": "yQUR5svvnw", "reviewer": "Reviewer_bDKw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a transformer-based architecture for molecular property prediction. The proposed method represents a molecule at two different levels: substructure-level and atom-level. The substructures of a molecule are extracted through MACCS fingerprint. Each substructure is embedded as a token and encoded by a transformer. The atom-level embedding is learned through a standard GNN and it is fused into the substructure embedding. The model pre-trained on a large collection of unlabelled molecules and fine-tuned on MoleculeNet benchmark datasets. ", "summary_of_the_review": "Overall, there are several concerns of the paper that needs to be addressed:\n1) Novelty compared to MolFormer and other previous work\n2) The cross attention is not linear time complexity based on equation 1, which contradicts with author's claim.\nTherefore, I vote for weak rejection of this paper. ", "strength_and_weaknesses": "Strength\n* The empirical evaluation is very comprehensive and includes most of the state-of-the-art baselines. \n* The results showed that the method outperforms all the baselines (averaged over 15 datasets).\n\nWeakness\n* The novelty of the proposed architecture is a bit limited. Similar architectures have been proposed before, e.g. MolFormer [1]. It would be good to discuss the technical differences and compare with MolFormer on the same datasets.\n* The proposed architecture treats each substructure as a discrete token and there is no relational attention between two substructures (e.g., the distance between two substructures, how many shared atoms, etc). In other words, the substructure transformer is a set transformer with no relational information between the substructures. It that's the case, the proposed model is essentially a transformer over MACCS keys + GNN + cross attention between the two. I believe that relational attention should be included, which could further improve the method.\n\n[1] Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs, arxiv 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "jXQrh32n-xM", "reviewer": "Reviewer_JtFD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary of the paper\nThe paper proposed a simple molecule representation learning method that combines representation from a transformer and a GNN network to leverage the strengths of each method. While transformers are good at learning frequent substructures or motifs in a molecule database, GNN provides additional information about missing structures due to the linearization of molecule graphs.\n\nThe authors also pretrained their proposed networks with 1.8 M molecules with the objective of predicting 200 real-valued descriptors of physical-chemical properties from the pretraining datasets using RDKit.\n\nThe experiments on 11 downstream tasks demonstrated significant results  of the proposed methods compared to different transformers  and GNN networks both with pretrained/non-pretrained models \n\n\n\n\n\n", "summary_of_the_review": "Summary of my comments\nThe paper was well-written and I see that its method for combining strengths from two different representations of learning architecture is an interesting practice but a combination of these two known techniques is not technically strong enough for an ICLR paper. \nI would recommend the authors validate the fusion approach against a simple linear ensemble of the baseline methods listed in your paper. Also, it is important to remove the test molecules of the benchmark datasets from the pretraining data.\n", "strength_and_weaknesses": "Strengths\nThe paper is well written. The results look promising. The idea of the fusion of the representation from two different architectures to leverage the strengths of both methods is a good practice worth sharing with the application community.\n\nThe examples in Figure 1 are an interesting motivation example for leveraging motif substructure in molecules to improve the disadvantage of GAT GNN.\n\nWeaknesses\nThis is rather a simple approach trivially combining existing techniques. For the ICLR conference, I would like to see more depth on technical contribution. I would suggest comparing the following simple baselines:\n\n + Ensemble of GNN and Transformers Models, just consider a linear model with the same weight as all the baseline models\n\n+ Recent works have considered learning a space that combines information from 2D and 3D data GraphMVP, please look at https://arxiv.org/abs/2110.07728, this is a good baseline as GraphMVP help fix the issues that models learned from 2D data that miss information about the 3D arrangement of the atoms in the molecules.\n\n\nPotential leakage during pretraining. It is important to remove test molecules from the data used for pretraining. This removal is important to make sure that there is no leakage happens.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LpY1aCPMQjd", "reviewer": "Reviewer_cHkX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "# Summary\nSubstructure-Atom Cross Attention for Molecular Representation Learning\n\n## What is the problem?\nPrediction of molecular properties via deep learning methods is important for AI-aided drug discovery.\n\n## Why is it impactful?\nDrug discovery is expensive, so better machine learning methods to solve this task could help accelerate drug discovery at lower costs.\n\n## Why is it technically challenging/interesting (e.g., why do naive approaches not work)?\nProcessing molecular graphs is challenging, as important features exist at various scales within the input graphs, and we have a puacity of labeled data to train high-capacity methods.\n\n## What is this paper's contribution?\nThis paper proposes to use extracted molecular substructures (extracted via MACCS keys) and atom-level features simultaneously via a two-branched neural network approach, where molecular substructures are processed via a transformer architecture and atoms via a GNN architecture.\n\nIn addition, the authors use a large-scale multi-task pre-training approach to initialize their architecture, thereby helping ameliorate for the small dataset scale available here.\n\n## How do these methods compare to prior works?\nWhile there are prior works that use both GNNs and transformers together, and approaches that leverage graph transformers directly, I am not aware of any papers that jointly leverage atomic features and MACCS substructures simultanesouly.\n\n## How do they validate their contributions?\nThe authors validate their contribution against a variety of baselines from the MoleculeNet benchmark, finding that their approach achieves rank 1.8 on average and 1.6 on average in classification and regression tasks, respectively. Baselines appear to be compelling, and span both pre-trained and non-pre-trained approaches.\n", "summary_of_the_review": "I think this is a very strong, well-written paper proposing a meaningful solution to an important problem and demonstrating the utility of that solution via compelling results.", "strength_and_weaknesses": "# Strengths and Weaknesses\n\n## Key Strengths (reasons I would advocate this paper be accepted)\n  1. I think this approach is well motivated, aimed towards an impactful problem, and well presented.\n  2. The results you present are compelling, and compare against meaningful baselines using appropriate evaluation metrics.\n\n## Key Weaknesses (reasons I would advocate this paper be rejected)\n  None.\n\n## Minor Strengths (things I like, but wouldn't sway me on their own)\n  1. I like that you include ablation studies motivating the design choices of your system.\n  \n## Minor Weaknesses (things I dislike, but wouldn't sway me on their own)\n  1. There are some missing details; for example, how are positional encodings generalized for their substructure transformer in this case? In addition, while hyperparameter search is described, architecture search is not (e.g., how many layers are used).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "BMhCinj8Kp", "reviewer": "Reviewer_Jvvo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel network architecture called Substructure-Atom Cross Attention (SACA) for molecular graphs that effectively combines Transformers and GNNs. Substructural patterns are important in molecular tasks, as we see in traditional chemoinformatics fingerprints like ECFP and fragmentation like BRICS/RECAP, but it is also widely known to be essentially hard for GNN to recognize substructure isomorphism. GNN is known to be upper bounded by 1-WL, whereas higher-order WL usually requires heavy computational cost. So explicitly incorporating substructural motifs can be a nice inductive bias to improve the prediction of GNNs in molecular tasks. The paper's simple idea is 1) prepare a motif vocabulary (e.g., MACCS keys), 2) for an input molecule, motif patterns detected from SMILES are fed as tokens to Transformers, 3) in parallel, the molecular graph is fed to GNN to get the node features representing local information around each node. 4) Then, GNN node features are used as keys & values in a Cross Attention layer with Transformer features as queries, 5) build a stack of Self Attention -> Cross Attention + a stack of Self Attention along with the CLS token for global output. The empirical studies demonstrated that this substructure-Transformer + molecule-GNN performed well in 11 downstream tasks after pretraining with a large ChEMBL + PubChem dataset.\n", "summary_of_the_review": "This paper presents a novel network architecture called Substructure-Atom Cross Attention (SACA) for molecular graphs that effectively combines Transformers and GNNs. It is based on a simple idea of the use of predefined motifs (MACCS keys). The on-bit motifs are fed into Transformer as tokens (via embedding layers), while the input molecular graph is processed by a GNN and feedback to the Transformer through the Cross Attention layer. Even though I have several concerns, as described in the [Weaknesses] section, I liked the paper overall. It would always be nice to see that a simple idea worked. On the other hand, I believe that MACCS keys should be replaced by better ones to make inputs (almost) always valid.\n", "strength_and_weaknesses": "[Strength]\n\n- The paper presents a simple, easy-to-understand, and effective design pattern to fuse Transformers and GNNs for molecular tasks with good empirical supports. It can incorporate the predefined knowledge of motif vocabulary into representation learning from molecules.\n\n- It was very nice to see the presented method didn't try to explicitly match the predefined subgraph motifs to the input molecules, and this enables a computationally efficient and clean architecture. It would require large-scale pretraining but the relationship between motifs and input molecules is indirectly acquired in a data-driven way. In contrast, typical research considering substructure-aware methods often requires a bit costly computations such as graph matching, subgraph isomorphism, motif tree compositions, etc.\n\n- The details are carefully followed by ablation studies (without GNN vs End-Concat vs Begin-Concat vs Random Embedding vs proposed, the results when we use different GNNs instead of GIN), experimental validations cover a wider range of competitive methods including several Graph Transformers, the pretraining is done with a large 1.8M record dataset collected from ChEMBL and PubChem, complementary analysis for practical uses such as visualization of learned features by tSNE, visualization of attention weights, and computation time.\n\n[Weaknesses]\n\n- One of the concerns is a potential risk of data leakage in evaluations. The paper's empirical studies are grounded on the test performance on 11 tasks of MoleculeNet. But the base model is pretrained with a large (self-collected?) dataset of 1,858,081 molecules from PubChem and ChEMBL that might also cover some/large parts of the MoleculeNet dataset. Though I understand that the pretext tasks for pretraining (to predict 200 RDKit-calculated molecular descriptors) are completely different from the MoleculeNet tasks.\n\n- Another concern is about the use of MACCS keys. It is a set of 166 keys and usually considered to be a bit weak to represent molecules in the entire ChEMBL and PubChem database. This is basically why PubChem developed their own version of substructural keys called PubChem Fingerprint that is a 881-bit-long structural key defined at https://pubchemdocs.ncbi.nlm.nih.gov/data-specification   \n   \nIf the input molecule doesn't have any of 166 structural keys, the input to Transformer branch (the main branch) becomes empty, and this model doesn't make any sense. This would be one of the most well-known disadvantages to use predefined motifs (as in MACCS keys) rather than motifs occurring in the dataset (as in ECFP). So what was the average number of on-bit keys out of 166 motifs? This case would not be likely to happen?  \n  \nAlso, it seems no problem if we just use data-driven motifs, by BRICS or ECFP, if the on bits are hashed into a fixed number of bits. Have you tested any other options than MACCS keys, particularly data-driven ones such as BRICS motifs or ECFP keys?\n\n- Related to the above point, the following three related methods [1] [2] [3], which were not included in the empirical evaluation of the paper ([1] was cited though), use data-driven motifs with GNNs or Transformers or both. Of particular interest, the \"Structure-Aware Transformer (SAT)\" [3] is based on Transformer + GNN like, First, extract k-hop subgraphs, process them by GNNs, and fed outputs into Transformer as Q and K with V as input molecule. So it would be highly appreciated to discuss the relationship. Is this pattern \"End-Concat\"..? (First, motifs occurring in the input molecule are fed into GNNs, and their outputs are fed into Transformer. See the picture at https://github.com/BorgwardtLab/SAT)\n\n[1] Zhang et al, Motif-based Graph Self-Supervised Learning for Molecular Property Prediction. (NeurIPS 2021)\n    https://arxiv.org/abs/2110.00987\n\n[2] Yu and Gao, Molecular Representation Learning via Heterogeneous Motif Graph Neural Networks (ICML2022)\n    https://arxiv.org/abs/2202.00529\n\n[3] Chen et. al, Structure-Aware Transformer for Graph Representation Learning (ICML2022)\n    https://arxiv.org/abs/2202.03036\n\nDisclaimer: I have no relationships with the authors of [1] [2] [3], just in case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AhyHmmSRAr", "reviewer": "Reviewer_vSFi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework that incorporates Transformer and GNN for molecular representation learning. In particular, the Transformer branch encodes molecular substructures by using molecular fingerprints as input, and the GNN branch extracts local node features from molecular graphs to provide embeddings for the fusion network's cross-attention in the Transformer branch. ", "summary_of_the_review": "Given the limited novelty and the unconvincing comparisons to the baselines, I would not recommend an acceptance. ", "strength_and_weaknesses": "Pros:\n1. The paper is well-organized and easy to follow. \n2. The ablation study is well-designed. \n\nCons:\n1. The idea of combining the information from molecular fingerprints and node embeddings learned from GNN is not novel and has been proposed in [1, 2]. \n2. The results shown in Table 2 are not convincing enough. The authors use random scaffold split and rerun the experiments 3 times following GROVER. Since the results between GROVER and the proposed model are very close (usually smaller than the standard deviation), it would be necessary to know if the differences are statistically significant. Otherwise, a fixed scaffold split may be used for fair comparisons. Besides, when comparing the proposed model to MPG, the proposed model outperforms MPG as shown in Table 2, while their performances are on par as shown in Table 2. I'm concerned about whether the other models would also have such different performances when changing the splitting way. Lastly, there is work [3] using pretraining strategy that has better results on the datasets being compared.\n3. The authors claim that their model has O(n) complexity according to the self and cross attention map in the Transformer branch. However, for the cross attention map, the complexity is actually O(nm), where n is the number of atoms, and m is the number of substructures. Since the number of substructures should be linear to the number of atoms in a molecule, I think the resulting complexity should be O(n^2).\n4. There is a mismatch between the shape of the rectangles and the size of the matrics as shown in Figure 3. For example, a 3*4 grid and a 2*4 grid all represent an m*d matrix, which is misleading.\n\n[1] Fang, Y., Yang, H., Zhuang, X., Shao, X., Fan, X. and Chen, H., 2021. Knowledge-aware contrastive molecular graph learning. arXiv preprint arXiv:2103.13047.\n\n[2] Cai, H., Zhang, H., Zhao, D., Wu, J. and Wang, L., 2022. FP-GNN: a versatile deep learning architecture for enhanced molecular property prediction. arXiv preprint arXiv:2205.03834.\n\n[3] Xia, J., Zheng, J., Tan, C., Wang, G. and Li, S.Z., 2022. Towards effective and generalizable fine-tuning for pre-trained molecular graph models. bioRxiv.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject