Empowering Graph Representation Learning with Paired Training and Graph Co-Attention
Authors: Andreea Deac
Yu-Hsiang Huang
Petar Velickovic
Pietro Lio
Jian Tang
Venue:  (2020)
Keywords: graph neural networks
graph co-attention
paired graphs
molecular properties
drug-drug interaction

Abstract: Through many recent advances in graph representation learning, performance achieved on tasks involving graph-structured data has substantially increased in recent years---mostly on tasks involving node-level predictions. The setup of prediction tasks over entire graphs (such as property prediction for a molecule, or side-effect prediction for a drug), however, proves to be more challenging, as the algorithm must combine evidence about several structurally relevant patches of the graph into a single prediction.
Most prior work attempts to predict these graph-level properties while considering only one graph at a time---not allowing the learner to directly leverage structural similarities and motifs across graphs. Here we propose a setup in which a graph neural network receives pairs of graphs at once, and extend it with a co-attentional layer that allows node representations to easily exchange structural information across them. We first show that such a setup provides natural benefits on a pairwise graph classification task (drug-drug interaction prediction), and then expand to a more generic graph regression setup: enhancing predictions over QM9, a standard molecular prediction benchmark. Our setup is flexible, powerful and makes no assumptions about the underlying dataset properties, beyond anticipating the existence of multiple training graphs.

Under review as a conference paper at ICLR 2020
EMPOWERING GRAPH REPRESENTATION LEARNING WITH PAIRED TRAINING AND GRAPH CO-ATTENTION
Anonymous authors Paper under double-blind review
ABSTRACT
Through many recent advances in graph representation learning, performance achieved on tasks involving graph-structured data has substantially increased in recent years-mostly on tasks involving node-level predictions. The setup of prediction tasks over entire graphs (such property prediction for molecule, or side-effect prediction for drug), however, proves be more challenging, as the algorithm must combine evidence about several structurally relevant patches of the graph into a single prediction. Most prior work attempts to predict these graph-level properties while considering only one graph at time-not not allowing the learner to directly leverage structural similarities and motifs across graphs. Here we propose setup in which graph neural network receives pairs of graphs at once, and extend it with a co-attentional layer that allows node representations easily exchange structural information across them. We first show that such setup provides natural benefits on a pairwise graph classification task (drug-drug interaction prediction), and then expand more generic graph regression and graph classification setups: enhancing predictions over QM9, D&D and PROTEINS, standard molecular prediction benchmarks. Our setup flexible, powerful and makes no assumptions about the underlying dataset properties, beyond anticipating the existence of multiple training graphs.
INTRODUCTION AND RELATED WORK
We study the task of gruph-level representation learning: i.e., computing representations of entire input graphs, for the purposes of downstream tasks (such as graph classification or regression). This is typically step-up in complexity compared to node classification the learning algorithm must aggregate useful structural information across the graph into a single prediction- relyinng only this global supervision signal (as opposed to having feedback from every node/edge of the graph).
link prediction, given that
Perhaps the highest challenge this kind of architecture must face is inductivity and generalisation acmosy structures. Specifically, an inductive model must be readily applicable across several graph itnucuues--inchuding unseen during training. Additionally, the model is tasked with discovering interesting structural "motifs" across the entire dataset of graphs, whose presence or absence may help determine the overall predictions.
However, even enabling inductivity is not traditionally simple task in graph representation learning, as many prior approaches (Bruna al., 2013, Perozzi et al. 2014 Defferrard et al., 2016) not inductive by design. Furthermore, even the models that are currently used for graph-level representation learning: e.g. Gilmer al. (2017) et al. (2018); Xu et al. (2018); et al. (2019). operate over only single graph at a time making it challenging for them to reason about common substructures across graphs from graph-level supervision signal alone.
In this manuscript, we propose the approach of paired training-iss learning representations over pairs of input graphs at once. Intuitively, as long as we allow for dataflow between the representations of the two graphs within a pair, this allows the graph neural network to directly observe related (sub)structures from other inputs, to solidify its decision making. We note that in the context of sraph-structured inputs this may be particularly useful as, unlike simpler inputs such images text, there are no guarantees that different graphs within a dataset will have equal or even similar overall structure. To facilitate this dataflow, we propose the usage of graph co-attention for exchanging

---

Under review as a conference paper at ICLR 2020
representations of nodes across the two graphs. Intuitively, this operator performs attention (Bahdanau et al. 2014 Vaswani et al. (017)) over the fully-connected bipartite graph, with one part corresponding to all nodes in one graph. This allows every node of the first graph to detect and reuse useful patch representations in the second graph (in a form of hierarchical graph matching), and vice-versa.
Initially, validate model performance a pairwise graph classification task classifying drug pairs for side effects caused by drug-drug interactions (DDI) (Jin et al. 2017] ZZitikk et al. (2018) In this setting, pairwise approach is natural, as we inherently have classify pairs of graphs. We demonstrate that learning a jooint representation using graph co-attention provides substantial benefits to predictive power, setting the state-of oftte-of-the-rrr the-art result on this task.
From there, we demonstrate the applicability of our approach to arbitrary multi-graph datasets; for this, we leverage the QM9 dataset for predicting quantum chemistry properties of small molecules (Ramakrishnan et al. 22014). As such, it represents a challenging graph regression problem. We propose using paired training to perform regression on two molecules at once, demonstrating clear benefits to doing so. In similar vein, we execute variants of our model standard graph kernel classification benchmarks (Kersting et al. 2016), showing advantages to generic graph classification.
Our approach paves the way to a promising direction for graph-level prediction tasks, that is in principle applicable to any kind of multi-graph dataset, especially under availability of large quantities of labelled examples.
Our model builds up on a large existing body of work in graph convolutional networks (Bruna et al. 2013; Defferrard et al. 2016 Kipf & Welling 201 6a Gilmer al. 2017 Veličković al. 2018))
input processing (such the chemical representation (Gilmer et al. 2017: De Cao & Kipf [0088 You et al. (0188) of the drugs leveraged here). Furthermore, we build up on work proposing co-atiention (Lu et al. 2016; [Dac et al., (018) as mechanism allow for individual sel-structured datasets (such as nodes in multimodal graphs) to interact. Specifically, such mechanisms have already been used for explicit matching of graph structure motifs (Li et al. 2019). and therefore represent a natural methodology for our purposes.
that have substantially advanced the state-of ofate-of-the-eoee the-art in many tasks requiring graph
Overall, these (and related) techniques lie within the domain of graph representation learning, one of the latest major challenges of machine learning (Bronstein et al., 2017 Hamilton et al. 2017 Battaglia et al. (018). with transformative potential across wide spectrum potential applications. extending outside the biochemical domain.
2 ARCHITECTURE
In this section, we will present the main building blocks used within our architecture for paired graph representation learning. This will span a discussion of the way the input to the model is encoded, followed by an overview of the individual computational steps of the model. As the tasks considered in this manuscript operate over molecules, we will treat all inputs molecular graphs.
2.1 INPUTS
The molecules, dus are represented as graphs consisting of atoms, (dz) as nodes, and bonds between (dz)
those atoms (a:d) .a. a((z)) as edges. For each atom, the following input features are recorded: the atom number, the number of hydrogen atoms attached to this atom, and the atomic charge. For each (d.). bond, a discrete bond rype (e.g. single, double etc.) is encoded as a learnable input edge vector, e(g
2.2 MESSAGE PASSING
Within each of the two molecules separately, our model applies a series of message passing (Gilmer et al. 2017) layers. Herein, nodes are allowed to send arbitrary vector messages to each other along the edges of the graph, and each node then aggregates all the messages sent it.
Let (dx) n denote the features of atom of molecule x, at time step t. Initially these
2

---

Under review as a conference paper at ICLR 2020
set to projected input features, i.e.:
(d=) 13 - fs
(a (4))) (a(4
where fi is small multilayer perceptron (MLP) neural network. In all our experiments, this MLP (and all subsequently referenced MLPs and projections) projects its input to 50 features. Note that this effectively encodes one-hot inputs such atom type into a learnable embedding.
Considering atoms i and of molecule x, connected by a bond with edge vector (d.) eij
start by computing the message, mly sent along the edge j - i. The message takes into account both the features of node j and the features of the edge between i and j:
(dz) ml =
(1
where I and f are small MLPs, and @ is elementwise vector-vector multiplication.
Afterwards, every atom aggregates the messages sent to it via summation. This results in an internal message for atom i, mf:
(de)m
Σ
(da)
(3)
VJEN()
where the neighbourhood, (i),,, defines the of atoms linked to i by edge.
2.3 CO-ATTENTION
Message passing provides a robust mechanism for encoding within-molecule representations of atoms. For learning appropriate joint molecule-molecule representation, however, allow atoms to interact across molecule boundaries via co-aftentional mechanism (Deac et al. 2018).
Consider two atoms, i and j. of molecules and y, respectively. Let their features at time step t be (d and (d)),,, just as before. For every such pair, we compute an attentional coefficient, alj using a simplified version of the Transformer (Vaswani et al. 2017) attention mechanism:
al softmax, ((w wg ((w((")1---(((0))))))))))
where W is a learnable projection matrix, (: is the inner product, and the softmax is taken across all nodes from the second molecule. The coefficients aly may be interpreted the importance of atom j's features to atom i.
These coefficients are then used to compute outer message for atom of molecule (d expressed as a linear combination (weighted by aty) of all (projected) atom features from molecule y:
(da) (dz) nt = Σ di W (do))
(5)
VjEdy
where W is a learnable projection matrix.
was shown recently that multi-head attention can stabilise the learning process, as well as learn information at different conceptual levels (Veličković et al., 2018 Qiu et al. (018). As such, the mechanism of Equation[]ja independently replicated across attention heads (we chose K 8), and the resulting vectors are concatenated and MLP-transformed to provide the final outer message for each atom.
K Maly (4) (w (W))))))
(d=) ft
Σ
(6)
k=IvjEd,
where f is a small MLP and is featurewise vector concatenation.
3

---

Under review as a conference paper at ICLR 2020
(du (dulynlly
Vola mnda
Drug
Drug
Figure 1: The illustration single step of message passing (computing the inner message, (4e) m))), and co-attention (computing the outer message (d)) on two nodes (i and u) of molecule
The computation Equations is replicated analogously for outer messages from molecule x to atoms of molecule
We have also attempted to use other popular attention mechanisms for computing the of values-such as the original Transformer (Vaswani et al., 2017), GAT-like (Veličković et al.] (018) and tanh (Bahdanau al. 2014) attention findntion-finding them yield weaker performance than the approach outlined here.
single step of message passing and co-attention, as used by our model, is illustrated by Figure
2.4
UPDATE FUNCTION
Once the inner messages, ( (obtained through message passing), as well as the outer messages, n; (obtained through co-attention) are computed for every atom of each of the two molecules (dzldy), we use them to derive the next-level features, (d. (d+),f, for each atom.
At each step, this is done by aggregating (via summation) the previous features (represent- ing a skip connection (He et al. 2016)), the inner messages and outer messages, followed by layer normalisation (Ba et al. 2016):
(d=) h Larer ((.)) + (d.) m + (d.));)
The operations Equations 7 are then repeated for T propagation steps here, we set Refer to Figure 2 for complete visualisation of our architecture.
As will be demonstrated by results, using co-attention to enable the model to propagate the information between two molecules from the beginning-Bee thus learning joint representation of the vallubly contributes the predictive power of the model.
2.5 READOUT AND SCORING
As we're making predictions on the level of entire molecules (or molecule-molecule pairs), we need compress individual atom representations into molecule-level representations. In order to obtain the molecule-leve vectors d,,, we apply simple summation of its constituent atom feature vectors after the final layer (i.e. after propagation steps have been applied):
d₂ = Σ fr (d=)))
(8)
Vjeds
where f, is small MLP.
4

---

Under review as a conference paper at ICLR 2020
Message passing
m
Brug 14 (1)02)
Update
Readhut
dr
Co-attention
(Aa) n (d
Update
14/1 f. (1)
dg
Message passing
x'I' steps
Figure 2: A high-level overview of our paired training architecture. The next-level features of atom i of molecule (d,) are derived by combining its input features, (d=) 2-1 its inner message, m4,),, computed using message passing, and its outer message, (dx) computed using co-attention over the second molecule, dy.
Once the molecule vectors computed for both d₂ and dy, they be used make pre- dictions on either x and individually, or to make pairwise predictions-the exact setup varies depending on the considered task.
PAIRWISE GRAPH CLASSIFICATION: POLYPHARMACY SIDE EFFECTS
We now directly evaluate the benefits of our model on a pairwise graph classification task. This represents natural testhed for our framework, within the dataset, graphs already given in paired form for training and evaluation.
Despite the apparent specificity of this setup, it is central to many relevant and important problems across the sciences. We study the task of polypharmacy side effect prediction, to be described below.
3.1 MOTIVATION
Diseases are often caused by complex biological processes which cannot be treated by individual drugs and thus introduce the need for concurrent use of multiple medications. Similarly, drug combinations are needed when patients suffer from multiple medical conditions. However, the downside of such treatment (referred as polypharmacy) that increases the risk of adverse side effects, caused by the chemical- physscall incompatibility of the drugs.
Uniquely within the related work in this area, our model learns a robust representation of drugs by leveraging joint information early on in the learning process. Our method outperforms previous state-of-the-art models in terms predictive power, while using just the molecular structure of the drugs. It thus allows for the method be applied to novel drug combinations, as well as permitting the detection to be performed at the preclinical phase.
3.2 DATASET AND PREPROCESSING
The drug drug-druguin interaction data was chosen to be in agreement with the dataset used by Decagon (Zitnik al. 2018). It is obtained by filtering the TWOSIDES side-effect dataset (Tatonetti et al.
5

---

Under review as a conference paper at ICLR 2020
No
Joheveation!
Bradnanli!
Gonmenaattall bind?
DDI predictor (hiuary)
pDi prodicisr (mmili-lated)
HO
Zachoandiz
HO.
HD
HD
Drog
Figure 3: An overview of the binary (left) and multi-label (right) drug-drug interaction (DDI) task. In both cases, two drugs (represented as their molecular structures) are provided to the model in order to predict existence or absence of adverse interactions. For binary classification, the DDI predictor is also given particular side effect input, and is required specifically predict existence absence of it. For multi-label classification, the DDI predictor simultaneously predicts existence or absence of all side-effects under consideration.
2012), which consists of associations which cannot be clearly attributed either drug alone (that is, those efff-effect reported for any individual drug in the OFFSIDE dataset (Taoonetti et al. 2012)).
The side effects, se2, one-hot encoded from set of 964 polypharmacy side-effect types that occurred in at least 500 drug pairs. The model input and loss function varies depending on whether are performing binary classification for given side effect, multi-label classification for all side effects at once (Figure further details may be found in Appendix A of 4,576,785 positive examples. As we aim sample balanced number of positive and negative examples, the overall dataset size for training our model is 9,153,570 examples.
The full dataset consists
3.3 EXPERIMENTAL SETUP
We will refer to our model as MHCADDI (multi-head co-attentive drug-drug interactions), and MHCADDI-ML for multi-label classification. We perform detailed ablation studies, to stress the importance of various components of MHCADDI (especially, its co-attention mechanism). The following variations are considered:
MPNN-Concat: removing co-attention, i.e. learning drug representations independently;
Late-Outer: where co-attention messages are not aggregated until the last layer;
CADDI: only
attention head.
Additionally, we compare our method with several existing strong baselines: the Multitask Dyadic Prediction (Jin al. 2017) relying on proximal gradients and substructure fingerprints, tensor factorisation methods RESCAL (Nickel et al. (011)) and DEDICOM (Papalexakis et al. (017), the network embedding method Deep Walk (Perozzi al. 2014), and the multiresolution link prediction method over drug-protein protein interaction graphs, Decagon (Zitnik al. (2018). We omit comparisons with traditional machine learning baselines (such as support vector machines or random forests) as prior work (Jin al. 2017) has already found them be significantly underperforming on this task.
3.4 RESULTS
We perform stratified 10-fold crossvalidation on the derived datasct. For each model, we report the area under the receiver operating characteristic (ROC) curve averaged across the 10 folds. The results of our study may be found Table 1
From this study, may conclude that learning joint drug-drug representations by simultaneously combining internal message-passing layers and co-attention between drugs is a highly beneficial approach, consistently outperforming the strong baselines considered here (even when additional
data sources are used, as in the case of Decagon). Furthermore, comparing between architectural variants, we can conclude that (further qualitative studies may found in Appendix A):
The fact that all our co- co-attentive architectures outperformed MPNN-Come implies that is beneficial learn drug-drug representations jointly rather than separately.

---

Under review as a conference paper at ICLR 2020
Table I: Comparative evaluation results after stratified 10-fold crossvalidation.
AUROC
Drug-Fingerpriie (Jin et al.
2017)
0.744
RESCAL (Nickel 2011
0.693
DEDICOM Papalexakis ct al
2017
0.705
Walk Walk PPerozzi et al.
2014
0.761
Concatenated Teatures Zitni al.
2018
0.793
Decagon Zitnik al.
2018)
0.872
MPNN-Coneat (ours)
0,661
Late-Outer (ours)
0.724
CADDI (ours)
0.778
MHCADDI (ours)
0.882
MHCADDI-ML (ours)
0.819
Furthermore, the outperformance Late-Outer by (MH)CA-DDI further demonstrates that it is useful provide the cross- modal information earlier in the learning pipeline.
Lastly, the comparative evaluation of MHCADDI against CADDI further shows the benefit inferring multiple mechanisms of drug-drug interaction simultaneously, rather than anticipating only one.
We note that further relevant research in particully in unsupervised learning-coue be used to further enhance our results. Given the link prediction nature the dataset, the method of Kipp & Welling (0166) could be useful for appropriately pre-training the GNNs used here, for example.
GRAPH REGRESSION: QUANTUM CHEMISTRY
Having confirmed that our paired training paradigm with graph co-attention yields significant returns on polypharmacy side effect prediction, now direct attention to generalising to arbitrary multi-graph datasets (even without explicitly given pairings).
We consider the challenging graph regression setup of predicting quantum-chemicdd properties of small molecules, on the QM9 dataset (Ramakrishnan et al. (2014), as previously done by e.g. Gilmer et al. (2017); Lu et al. 2019). QM9 consists of ~ 130,000 molecules with 12 properties for each molecule, yielding 12 regression tasks. We preprocess the dataset similarly as in prior work (by transforming the ground-truth values of the twelve quantum-cheemica properties under study to the [0, 1] range). We use 10,000 randomly-chose molecules for testing and validation each; the remainder is used for training.
4.1 EXPERIMENTAL SETUP
We perform paired training on QM9 by pairing each input graph with K different training graphs (these pairs are fixed and chosen in advance, for consistency and reproducibility purposes). We compare the following variants:
(self): only pair molecules to themselves. This setup is equivalent an MPNN Gilmer et al. 2017) from representational standpoint, as the co-attention layers allow for unrestricted dataflow between atoms in a molecule.
= 1 (k-NN): using a pre-trained Grammar VAE (Kusner et al. (017)) model, pair each molecule with its nearest neighbour.
K 1 (random): pair each molecule to randomly chosen training molecule.
K E [2. 5]: pair each molecule to K randomly chosen training molecules, and addition- ally to themselves.
(k-NN): pair each molecule to itself and with K nearest neighbours obtained using pre-trained GrammarVAF.
7

---

Under review as a conference paper at ICLR 2020
Table 2: Mean absolute emrors (MAE) the twelve prediction tasks of QM9.
K =
K ==1
K=1 (random)
K =
K =
K
K = 5
K =
Property
(self)
(K-NN)
(k-NN)
mu
0.6267
1.1058
0.6354
0,6182
0.6242
0.6178
0.6142
0.6160
alpha
0.6893
2.2989
0.6913
0,6550
0.6647
0.6647
0.6523
0.6590
HOMO
0.0055
0.0140
0.0056
0.0053
0.0055
0.0054
0.0053
0.0053
LUMO
0.0057
0.0196
0.0060
0.0055
0.0055
0.0056
0.0055
0.0054
eap
0.0077
0.0242
0.0082
0.0075
0.0076
0.0076
0.0076
0.0074
R2
59.249
147.25
61.309
55.563
57.229
57.341
55.789
52.823
ZPVE
0.0005
0.0025
0.0005
0.0005
0.0005
0.0005
0.0004
0.0004
UO
0.2912
33,575
0.1451
0.1970
0.1902
0.1872
0.1337
0.2552
0.2980
29.247
0.1610
0.1557
0.1713
0.2523
0.1416
0.1984
H
0,3870
23.029
0.2587
0,2055
0.2124
0.2319
0.1363
0.2816
G
0.3441
18.554
0.1990
0.1972
0.1948
0.2574
0.1737
0.2444
Cv
0.3781
1.6162
0.3908
0.3665
0.3706
0.3727
0.3520
0.3534
Average
5.1902
21.3949
5.3176
4.8314
4.9731
4.9954
4.8343
4.6208
bayer
500000
baper
140000
120000
1200000
NuTbor 10250
100001
100000
80000
60000
4000
40000
20000
-20
-10
-85
-25
Ertopy
Entropy
Figure 4: Histograms of entropies of co-attentional coefficients, for the three layers of trained K co-attentive model (left) and the uniform distribution (right).
When training pair of molecules, simultaneously optimise the mean absolute (MAE) both them, for all the regression targets. At validation- or test-time, we only look at predictions for the molecule, discarding the representation of the paired training molecule. Furthermore, whenever K > 1, we average the predictions across all pairings at test time.
4.2 RESULTS
The results on the QM9 dataset study are provided in Table 2 may be noted that already allowing for one more random pairing on top of the self-pair& (K = 2) yields improved performance across all of the prediction tasks. These gains remain consistent, and usually do not change much, when increasing the number of pairings beyond 2. This confirms the benefits of paired training in the graph regression setup.
Furthermore, we provide qualitative evidence for the co-attention's mechanism of action, demonstrat- ing that learns attentional probability distributions that are more complex than uniform aggregation- i.c. that focus on specific regions of the paired graph. We visualise, for the K 3 model, the histograms of co-attention coefficient entropies across the testing dataset, and compare it against the histogram for the uniform distribution, in Figure This clearly demonstrates that not only the co-attention mechanism learns complex interactions across the paired molecules, but also that learss different mechanisms of interaction across the three co-attentive layers.
8

---

Under review as a conference paper at ICLR 2020
Table 3: Classification accuracy percentages after 10-fold cross-validation..
Datasets
Model
D&D
Proteins
MPNN Gilmer et al.
2017
70.04
68.15
Late-Attn (ours)
68.59
71.49
Co-Attn (ours)
74.11
71.14
5 GRAPH CLASSIFICATION
Finally, we turn our attention to the standard graph kernel classification datasets of Kersting et al. (2016). We focus on two molecular-base binary graph classification datasets: D&D and PROTEINS. In both cases, it is required to classify protein structures based whether they enzymes. D&D comprises 1,178 graphs of protein structures (with 284 nodes and 715 edges on average). PROTEINS comprises 1,113 graphs (with 39 nodes and 72 edges average).
For both datasets, we train a co-attentive GNN with propagation rules as described in the Architecture section. The GNN consists of three interleaved layers of message passing and co-attention, computing 64 features each. As is common-pris paantocceeaccceeeeeee for these datasets (see e.g. Ying et al. (2018)) we provide select
the accuracy results them after 10-fold crossvalidation. For paired training purposes, a random training graph to pair with for each batch, and otherwise proceed as described for QM9, Random pairing was sufficient demonstrate clear gains, and thus we leave more complex pairings for future analysis.
For evaluation purposes, we compare our co-attentive GNN with paired training against version without co co-attention (referred to as MPNN), well as version with only one co-attentive layer at the end (referred to as Late-Aitn).
The results of this study may be found in Table 3 Evidently, the co-attentive method improves over corresponding MPNN baseline on both datasets. Furthermore, on D&D, it outperforms the late-applied co-attention, providing evidence of useful hierarchical structure matching between the two paired graphs.
6 CONCLUSIONS
We have presented novel way of training graph neural networks multi-graph datasets. relying on making predictions jointly, in pairs of graphs-the paired training approach. Additionally, allowed for arbitrary representation exchange between these graphs by way of co-attentive mechanisms. The two combined allow for extraction of stronger and more robust representations as opposed single-graph learning, which is a claim verified across several established molecular prediction tasks: polypharmacy side effect prediction (where we set a state-of-the- result), quantum chemistry properties prediction and graph classification. As flexible and generic approach which doesn't it rely on dataset properties in any way, so long as it consists of multiple graphs, we believe it be useful direction to explore for graph representation learning as whole.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint 100x:::44090044 2014.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalee Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1 2018.

---

Under review as a conference paper at ICLR 2020
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18 42, 2017.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv preprint arXiv: 1312.233,, 2013.
Nicola De Cao and Thomas Kipf. Molgan: An implicit generative model for small molecular graphs. arXiv preprint ar05: / 2018.
Andreea Deac, Petar Veličković, and Pictro Sormanni. Attentive cross-modal paratope prediction.
arXiv preprint 1806. 04398, 2018.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pp. 3844-3852, 2016.
Justin Gilmer, Samuel Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. CoRR, abs/1 2017. URL http:://rxxy
azq/abs/1704: 01212
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference artificial intelligence and statistics, pp. 249-256, 2010.
William Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and 2017.
applications. arXiv preprint
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Bo Jin, Haoyu Yang, Cao Xiao, Ping Zhang, Xiaopeng Wei, and Fei Wang. Multitask dyadic prediction and its application in prediction adverse drug-drug interaction. In AAAI, 2017.
Kristian Kersting, Nils M. Kriege, Christopher Morris, Petra Mutzel, and Marion Neumann. Bench- mark data sets for graph kernels, 2016. URL httpp://graapkkermee
as ws..cstuuu.... tu-dortmund
de
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412iv:441..00 144166988. 2014.
Thomas N Kipf and Max Welling. Semi-supervice classification with graph convolutional networks. arXiv preprint 1609. 029072 2016a.
Thomas Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arX1v:1611.07308. 2016b.
Matt J Kusner, Brooks Paige, and José Miguel Hermández-Lobato. Grammar variational autoencoder. In Proceedings of the 34th International Conference on Machine Learning-Voon Voluue 70, pp. 1945- 1954. JMLR. org, 2017.
Yujia Li, Chenjie Gu, Thomas Dullien, Oriol Vinyals, and Pushmeet Kohli. Graph matching networks for learning the similarity of graph structured objects. arXiv preprint arXiv:1904.12780 2019.
Chengqiang Lu, Qi Liu, Chao Wang, Zhenya Huang, Peize Lin, and Lixin He. Molecular property pre- diction: A multilevel quantum interactions modeling perspective. arXiv preprint arXiv: 10011, 2019.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hicrarchical question-iman image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pp. 289-297, 2016.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2579 2008
10

---

Under review as a conference paper at ICLR 2020
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way way model for collective learning on multi-relational data. In ICML, volume 11, pp. 809-816, 2011.
Evangelos Papalexakis, Christos Faloutsos, and Nicholas D Sidiropoulos. Tensors for data mining and data fusion: Models, applications, and scalable algorithms. ACM Transactions on Intelligent Systems and Technology (TIST), 8(2):16, 2017.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social representa- tions. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710. ACM, 2014.
JZ Qiu, Jian Tang, Hao Ma. YX Dong, KS Wang, and J Tang. Deepinf: Modeling influence locality in large social networks. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 18). 2018.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1:140022, 2014.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine 2014.
Learning Research,
Nicholas Tatonetti, Patrick P. Ye, Roxana Daneshjou, and Russ B. Altman. Data-driven prediction of drug effects and interactions. 2012. /tt.atpp.//aa...rr...22//////ta. Imed
3003377
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017. URL
fttp:://axxiv.or//aaab
03762
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liô, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
URL httphhtss///peeeeeeeeee .wt.:/neeeeeeeeeeeeeee forum? f id=rJXMpikC2
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural 2018
networks? arXiv preprint vrXiv:1810.000
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In Advances in Neural 2018
Information Processing Systems, pp.
Hee-Giun Yoon, Hyun-Je Song, Seong-Bae Park, and Young Park. translation-based knowledge graph embedding preserving logical property of relations. In HLT-NAACI, 2016.
Jiaxuan You, Bowen Liu, Rex Ying, Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. arXiv preprinl arXiv: 010..000000433. 2018.
Marinka Zitnik, Monica Agrawal, and Jure Leskovee, Modeling polypharmacy side effects with graph convolutional networks. Bioinformatiies 34(13):4457-446) 2018, doi: 10. 10..109//iioinformaaicc bty294. URL http://///....ooooooo//////////.
A POLYPHARMACY SIDE EFFECT TASK DETAILS
A.1 DATASET GENERATION
For binary classification (Figure 3 (Left)), the input our model is triplet of two drugs and a side effect (dg, dy. se,), requiring a binary decision on whether drugs x and adversely interact to cause side effect 2.
For multi-label classification (Figure 3 (Right)), the input to model is pair of two drugs (dg, dy), requiring 964 simultaneous binary decisions on whether drugs x and adversely interact cause each of the considered side effects. Note that, in terms of learning pressure, this model requires more robust joint representations of pairs of drugs- as they need to be useful for all side-effect predictions at once.
11

---

Under review as a conference paper at ICLR 2020
In order compensate for the fact that TWOSIDES contains only positive samples, appropriate negative sampling is performed for the binary classification task:
During training, tuples (dz dy sez), where d, and are chosen from the dataset and dy is chosen random from the set of drugs different from dy in the true samples (dg dy sez).
During validation and testing, we simply randomly sample two distinct drugs which do not appear the positive dataset.
A.2 TRAINING SETUP
To further regularise the model, dropout (Srivastava al. 2014) with p 0.2 applied the output of every intermediate layer. The model is initialised using Xavier initialisation (Glorot & Bengio 2010 and trained using the Adam SGD optimiser (Kingma & Ba 2014), with early stopping. The learning rate after f iterations, is derived using an exponentially decaying schedule:
?) = 0.001 0.9
(9)
A.3 BINARY CLASSIFICATION
In the binary classification case, the side effect vector se, is provided as input. We then leverage scoring function f similar to the one used by Yoon et al. Yoon et al. (2016) to express the likelihood of this side effect occuring:
f(dz, dyy seg) M,dr + se, M,d, || se; M,d
(10)
where || lla is the La norm, and M, and M, represent the head node and tail node space mapping matrices, respectively.
The model is then trained end-to-end with gradient descent to optimise margin-based ranking loss:
C= E Σ max(0, f(das dy, sex) f(d dy. sex)))
(11)
da:dy.8es di de
where (dz, dy) is drug-drug pair exhibiting side effect se,, and (dz, dgy) is drug-drug pair exhibiting it. >> 0 is the margin hyperparameter.
A.4 MULTI-LABEL CLASSIFICATION
Here, all side effects are predicted simultaneously, and accordingly we define a prediction layer, parametrised by learnable weight matrix W, and bias by- This layer consumes the concatenation of the two drug vectors and projects it into score for each of the 964 side effects. These scores are then converted into probabilities, Vey- of each side effect, 2, occurring between drugs and y using the logistic sigmoid nonlincarity, a, applied elementwise:
Vig = o
[|dy] b,))
(12)
We can then once again train the model end-to-end with gradient descent, using binary cross-entropy against the ground truth value, V2 (which is binary label indicating whether side effect actually occurs between drugs x and y). The loss function is as follows:
LBCE
Σ
Wey log Viy + (1- Viyy) log (1-1)) (1-
(13)
d...
A.5 QUALITATIVE RESULTS
The effectiveness of the learnt joint representations may be investigated qualitatively well, and to that end, we have devised a controlled visualisation experiment. Our objective is to investigate the distribution of learnt drug-drug embeddings with respect individual side effects.
12

---

Under review as a conference paper at ICLR 2020
Figure 5: t-SNE projections of 200 learnt drug-drug representations, across side-effects. Each side effect is colour-coded with shade of red for positive drug-drug pairs and shade of green for negative pairs.
We start with a pre-trained MHCADDI model. For each side effect in a sample of 10, we have randomly sampled 100 drug-drug pairs exhibiting this side effect, and 100 drug- pairs not exhibiting it (both unseen by the model during training). For these pairs, we derived their embeddings, and projected them into dimensions using t-SNE Maaten & Hinton (2008).
The visualised embeddings may be seen Figure for individual side-effects. and Figure for a combined plot. These plots demonstrate that, across single sidc-effect, there is a discernible clustering in the projected 2D space of drug-drug interactions. Furthermore, the combined plot demonstrates that strongly side-effect inducing drug-drug pairs tend to be clustered together, and away from the pairs that do induce side-effects.
13

---

Under review as a conference paper at ICLR 2020
Figure 6: t-SNE projection of 2000 learnt drug-drug representations, sampled to cover 10 side effects. Each side effect colour-coded with a shade of red for positive drug-drug pairs and shade of green for negative drug-drug pairs.
14


Reviews:
[{"review_id": "rkgsaIwdqH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a model to classify pairs of graphs which is used to predict sided effects caused by drug-drug-interactions (DDI). \n\nThe contribution of this work is to add attention connections between two graphs such that each node operation from one graph can attend on the nodes of the other graph. The paper shows good results in DDI prediction, although the performance gap with previous works (Zitnik et al., 2018) is modest.\n\nIn the related work they mention some works from Graph Neural Networks literature. But works from the benchmark experiments are not explained. I think they could also explain which are the similarities and differences of the proposed method vs these works they are comparing to.\n\nAnother way of improving the paper could be running more experiments beyond the QM9 dataset to corroborate the good performance of the algorithm.\n\nIn equation (2), a message that goes from node “j” to node “i” does not include node “i” as input into the edge operation. I think the GNN would be more powerful if both nodes \"i\" and \"j\" are input into the edge operation.\n\nIn summary, the main contribution of the paper is to add attention connections between two graphs. I do not feel it is innovative enough.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hkg6hDIotB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work injects a multi-head co-attention mechanism in GCN that allows one drug to attends to another drug during drug side effect prediction. The motivation is good with limited technical novelty. The paper is well-written and well organized.\n\n\nFor MHCADDI, it is performing binary classification for all side effect labels. It is different from Decagon’s setting, hence not comparable. Maybe also include Decagon-Binary?\n\n\nMissing baseline: as its main innovation is using co-attention, it should compare with concatenated embedding generated from Graph Attention Network so that we know co-attention is better than independent attention on each drug (seems the authors have already attempted to do so but did not report it). Current baselines such as Decagon only use GCN with no attention mechanism. It could be also benefited by including VGAE. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1xagNeTFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a method to extend graph-based learning with a co-attentional layer. Feeding graphs pairwisely into the model allows nodes to easily exchange information with nodes in other graphs as well as within the graph. This method outperforms other previous ones on a pairwise graph classification task (drug-drug interaction prediction).\nThis model is generalized from Neural Message Passing for Quantum Chemistry (Justin Gilmer et al.) and Graph Attention Networks (Petar Velickovic et al.), but most ideas are directly from the two previous papers. Combining the two methods do provide insights into understanding the interactions between graphs and get really good results on DDI prediction, but the novelty is limited.\nQuestions:\n1 Are atoms encoded as only atom numbers, charges and connected hydrogen atoms? Because some atoms might have much larger atom numbers than others, e.g. carbon (6) and sulfur (16), will there be some scale problems? Will one-hot encoding of atom type help (like in Neural Message Passing for Quantum Chemistry)?\n2 According to the paper, bond types will be encoded as e_{ij}. But in molecules, bond type is way more complex than only single/double/triple bonds, especially for drug molecules which are enriched for aromatic systems. For example, bonds in benzene or pyridine rings are between single and double (also not necessarily 3/2). Are there other possible methods to encode graph edges?\n3 In result table 2 of Section 4 (quantum chemistry), I didn’t see a principle of choosing K value and choosing neighbors because different properties reaches the lowest MAE at different K values. This might cause some confusion in real application. Moreover, the authors should compare the performance with previous methods.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes combining paired attention with co-attention. The reviewers have remarked that the paper is will written and that the experiments provide some new insights into this combination. Initially, some additional experiments were proposed, which were addressed by the authors in the rebuttal and the new version of the paper. However, ICLR is becoming a very competitive conference where novelty is an important criteria for acceptance, and unfortunately the paper was considered to lack the novelty to be presented at ICLR.
Decision: Reject