Towards Interpretable Molecular Graph Representation Learning
Authors: Emmanuel Noutahi
Dominique Beani
Julien Horwood
Prudencio Tossou
Venue:  (2020)
Keywords: molecular graphs
graph pooling
hierarchical
GNN
Laplacian
drug discovery

Abstract: Recent work in graph neural networks (GNNs) has led to improvements in molecular activity and property prediction tasks. Unfortunately, GNNs often fail to capture the relative importance of interactions between molecular substructures, in part due to the absence of efficient intermediate pooling steps. To address these issues, we propose LaPool (Laplacian Pooling), a novel, data-driven, and interpretable hierarchical graph pooling method that takes into account both node features and graph structure to improve molecular understanding.
We benchmark LaPool and show that it not only outperforms recent GNNs on molecular graph understanding and prediction tasks but also remains highly competitive on other graph types. We then demonstrate the improved interpretability achieved with LaPool using both qualitative and quantitative assessments, highlighting its potential applications in drug discovery.

Under review as a conference paper at ICLR 2020
TOWARDS INTERPRETABLE MOLECULAR GRAPH REP- RESENTATION LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Recent work in graph neural networks (GNNs) has led to improvements in molecu- lar activity and property prediction tasks. Unfortunately, GNNs often fail to capture the relative importance of interactions between molecular substructures, in part due to the absence of efficient intermediate pooling steps. To address these issues, we propose LaPool (Laplacian Pooling), a novel, data-driven, and interpretable hierarchical graph pooling method that takes into account both node features and graph structure to improve molecular understanding. We benchmark LaPool and show that it not only outperforms recent GNNs on molecular graph understanding and prediction tasks but also remains highly competitive on other graph types. We then demonstrate the improved interpretability achieved with LaPool using both qualitative and quantitative assessments, highlighting its potential applications drug discovery.
INTRODUCTION
Following the recent rise of deep learning for image and speech processing, there has been great interest in generalizing convolutional neural networks to arbitrary graph-structured data (Gilmer et al.) et al. 2015 Xu et al. 2018). To this end, graph neural networks (GNN) falling into either spectral-based or spatial- based approaches have been proposed. Spectral methods define the graph convolution (GC) as a filtering operator of the graph signal (Defferrard et al. (016)) while spatial methods define the GC as a message passing and aggregation across nodes (Henaff et al.. 20155) Xu et al. 2018 Jin et al. 2018). In drug discovery, GNNs have been very successful across several molecular graph classification and generation tasks. In particular, they outperform predetermined molecular fingerprints and string-basee approaches for molecular property prediction and the de novo generation of drug-like compounds (Jin et al. 2018 Li et al. 2018b).
However, the node feature update performed by GNNs introduces important limitations. For instance, experimental results indicate a performance decrease for deeper GNNs due to the signal smoothing effect of each GC layer (Li et al. 2018a). This limits the network's 's depth and restricts the receptive field of the vertices in the graph to a few-hop neighborhood, which is insufficient to properly capture local structures, relationships between nodes, and subgraph importance in sparse graphs such as molecules. For example, at least three consecutive GC layers are needed for atoms at the opposite side of benzene ring to exchange information. This issue exacerbated by the single global pooling step performed at the end of most GNNs that ignores any hierarchical structure within the graph.
To cope with these limitations, graph coarsening (pooling) methods have been proposed reduce graph size and enable long- distance interaction between nodes. The earliest proposed methods relied solely on deterministic clustering the graphs, making them non- differentiable and indeeendeenn (Jin et al.) 2018 8 Dafna and Guestrin 2009 von Luxburg, 2007 Ma et al.) 2019). In contrast, more recent methods use node features but, as we will show, are unable to preserve the graph structures after pooling (Ying et al. 2018 Gao and Ji 2018), limiting their interpretability.
Borrowing from theory in graph signal processing, we propose LaPool (Laplacian Pooling), a differentiable pooling method that takes into account both the graph structure and its node features. LaPool performs a dynamic and hierarchical segmentation of graphs by selecting a set centroid nodes as cluster representatives (centroids) using the graph Laplacian, then lears sparse assignment of the remaining nodes (followers) into these clusters using an attention mechanism. The primary contributions of this paper are summarized below:

---

Under review as a conference paper at ICLR 2020
We propose novel and differentiable pooling module (LaPool) that can be incorporated into existing GNNs to yield more expressive networks for molecular data.
We propose graph structure understanding dataset for benchmarking GNNs that is based on molecular substructure prediction.
We show that LaPool outperforms recently proposed graph pooling layers on both discrimi- native and generative molecular graph benchmarks, while also remaining competitive on other graph benchmarks.
We highlight the improved interpretability achieved by LaPool using both qualitative and quantitative assessments.
We argue that the enhanced performance and interpretability achieved by LaPool can improve our understanding of molecular structure-activity relationships, and therefore has important applications in drug discovery.
RELATED WORK
In this section, we first introduce related work on graph pooling, then provide an overview of techniques used in computational drug discovery to put our work into context. As our focus herein is on graph pooling, we refer readers to (Wu et al. 2019) for an overview of recent progress GNNs.
In traditional GNN architectures, global sum/averagee/max-poollo node embeddings into a graph-level representation. Recently, more sophisticated methods have been proposed. For example, [Li et al. (2015) uses a gated mechanism, Zhang et al.) (2018) proposed SortPool which sorts node features before feeding them into a ID convolution, while in (Gilmer et al. 2017)) node feature averaging was substituted by a Set2Set architecture. Although these new global aggregation methods have been shown to outperform standard global pooling, they completely overlook the rich structural information on graphs which has been shown necessary for building
layers have been used to aggregate
effective GNN models (Ying et al. 2018 Ma et al. 2019).
Consequently, hierarchical graph pooling methods have been proposed. They act by reducing graph size and increasing node receptive fields without increasing network depth. However, in contrast to the regular structure of images, graphs are irregular and complex, making it challenging properly pool nodes together. Certain hierarchical graph pooling methods therefore rely on deterministic and non-differentiable clustering to segment the graph (Defferrard et al., 2016 Jin et al.) 2018). More recently, differentiable hierarchical graph pooling layers have been proposed. Ying et al. (2018) proposed DiffPool, pooling layer that performs ämiilartyy-bssed node clustering using soft affinity matrix learned by a GNN. Likewise, Graph U-Net was proposed in (Gao and Ji, (0188) as sampling method that retains and propagates only the top-k nodes at each pooling step based on a learned importance value.
In computer-aided drug discovery, methods such as virtual screening and de novo drug design serve as efficient complements to physical high-throughput screening of large molecular libraries. For example, virtual screening, which aims to accurately predict molecular properties directly from molecular structure, can play an important role in rapidly triaging promising compounds early in drug discovery (Subramaniam et al. 2008). Importantly, data-driven virtual screening approaches that leverage advances in deep learning, rather than me-determined features such molecular fingerprints (Rogers and Hahn| 2010) and SMILES string representations, have been shown to dramatically improve prediction accuracy (Kearnes al. 2016 Wu et al. 2018). Similarly, advances in generative models have enabled the application of deep generative techniques such as VAE (Kingma and Welling 2013) and GAN (Goodfellow et al.) (2014) to the de novo design of drug-like molecules. The first molecular generative models (c.g. Grammar- VAE (Kusner et al. 2017)) resorted to generating string representations of molecules (via SMILES), which resulted in a high proportion of invalid structures due to the complex syntax of SMILES. Graph generative models have since been developed (e.g. JT-VAE (Jin et al. (2118) GraphVAE (Simonovsky and Komodakis), MoIGAN (De Cao and Kipl 2018). MolMP (Li et al. 2018)), etc.) and have been shown to improve the validity and novelty of generated molecules. In addition, these methods allow conditional molecule generation via Bayesian optimization reinforcement learning (Jin et al., 2018 Olivecrona et al.) 2017] Assouel et al. 2018 Li et al.] 2018d: You et al., 2018a). In this work, we are mainly interested in the impact of molecular representation on generative performance as opposed to the optimization procedure itself.
2

---

Under review as a conference paper at ICLR 2020
3
GRAPH LAPLACIAN POOLING
X cx Ar1 -c
CH
Graph convolution (GC) forward propagelin GE
Centroid selection
Node-to to-c clusters to-lutte--sssss mapping Build wlfinity matric that the controlds.
Pooling
Select centrolds baved their local
Use Coto pool the adjecency matrix and features into
motecular grapts
Figure 1: Overview of the proposed Laplacian Pooling (LaPool) method
reliable pooling operator should maintain the overall structure and comnectivity of a graph. LaPool achieves this by taking into account the local structure defined by the neighborhood of each node. As shown in Figure [ the method uses a standard GC layer with a centroid selection and a follower selection step. First, the centroids of the graph are selected based on the local signal variation (see Section 3.2). Next, LaPool learns an affinity matrix C' using a distance normalized attention mechanism assign all nodes of the graph the centroids (see Section 3.3p. Finally, the affinity matrix is used to coarsen the graph. These steps are detailed below.
3.1 PRELIMINARIES
Notation Let G AX) be an undirected graph, where V = (U1....up Un is its vertex set, A E 13,xn denotes its adjacency matrix, and X = [x1,... Xn] E R is the node feature matrix with each node v, having d-dimensional feature z₁. X can also be viewed as d-dimensional signal on G (Shuman et al. 2012). Without loss of generality we may assume a fixed ordering of the A. and X. The neighborhood
nodes that respected node U E V is the set of nodes separated from v; by path of length at most h and is denoted by (h:())) For simplicity, we will use (1s) to refer to the set of nodes adjacent to ui.
radius h (or h-hop) neighborhood of
Graph Signal For any graph G, its graph Laplacian matrix T, is defined as I, = D A, where D is a diagonal matrix with Di,s being the degree of node v;- The graph Laplacian a difference operator and can be used to define the smoothness s(X) (the extent at which the signal changes between connected nodes) of a signal X on G. For I-dimensional signal [fl....,fa]
a Σ i.j
sff = (fT Lf) 2
Ar 1
f2)
Graph Neural Networks We consider GNNs that act in the graph spatial domain message passing (Gimerr al. (0017). We focus on the Graph Isomorphism Network (GIN) (Xu al. 20018), which uses SUM-aggregator on messages received by each node to achieve better understanding of the graph structure:
x = M/
11-1) x
Σ (EE)(E))
( 1-" (1-1)
where My is neural network with trainable parameters O z; is the feature vector for node u, by N(())) are the neighbors of o and / is the layer number. Notice the term Ai,j that takes into account the edge weight between nodes and by when A is not binary.
In this work, we focus mainly on molecular graphs in supervised settings where, given a molecule and its corresponding molecular graph Gm, aim predict some properties of m. Molecular graphs present two particularities: (1) they are often sparse and (2) there is no regularity in the graph signal (non-smooth variation) adjacent nodes tend not have similar features.
3

---

Under review as a conference paper at ICLR 2020
3.2 GRAPH DOWNSAMPLING
This section details how LaPool downsamples the original graph by selecting a set Ve of nodes as centroids after consecutive GC layers.
Centroid Selection For any given vertex 04 we can define a local measure 8; of intensity of signal variation around v,. As s; measures how different the signal residing at node oy from its neighbors, greater than
we are interested in the set Vc of nodes that have an intensity of signal variation their neighborhood. In this work, use a definition of local signal variation similar the local normalized neighboring signal variation described in (Chen et al. 2015b), with the only difference being the absence of degree normalization:
8, 8- 1 | (EN((o)
5= 8p
Vc = topy (IS,
(3)
where topy ST. corresponds the top l nodes with the greatest intensity of signal variation, and where || 112., corresponds to taking the vector norms over the d-dimensional rows of LX. Instead of using the direct neighbors, one can also generalize the computation of S in equation an /h hop neighborhood by taking powers of the Laplacian.
Observe that the GC layers preceding each pooling step perform a smoothing of the graph signal and thus act as a low-pass filter. Eq. (3) emphasizes instead the high variation regions, resulting overall in filtering of X that attenuates low and high feet-freeuency noise, yet retains the important signal information. The intuition of using the Laplacian maxima for selecting the centroids is that smooth signal can be very well approximated using linear interpolation between local maxima and minima. This is in contrast with most approaches in GSP that use the lower frequencies for signal conservation but requires the signal to be k-bandlimited (Ma et al. 2019 Chen et al. 201 a). For a ID signal, LaPool selects points, usually near the maxima/minima, where the derivative changes the most and is hardest to interpolate linearly. For molecular graphs, this often corresponds to sampling a subset of nodes critical for reconstructing the original molecule.
Dynamic Selection of the Centroids The method presented in Eq. 3 implies the selection of centroids. Because the optimal value of & can be graph-dependent and might result in densely located centroids, we explore alternative in which we dynamically select the nodes with signal variation s; greater than its neighbors 8j:
Vc for E V A uz Aijs; > 0}
(4)
3.3 LEARNING THE NODE-TO-CLUSTER ASSIGNMENT MATRIX
Once the set Ve of centroid nodes is determined, we compute a mapping of the remaining "follower" nodes VF V\ Ve into the new clusters formed by the nodes in Vo. This mapping gives the cluster
assignment C [c.,...c,]] E [0, 1|mxm s.t. Vi 1c] 1, where each row c, corresponds to the affinity of node v; towards each the m clusters in Ve.
Let (4) be the node embedding matrix at arbitrary layer and x the embedding of the "centroids". We compute using a soft-attention mechanism (Graves et al., 2014) measured by the cosine similarity between X and x!!)
8 sparsemax
v; E Vo
C;
B
x
otherwise
(5)
| x
where 6,,, is the Kronecker delta and sparsemax (Laha al. 2016 Martins and Astudillo, 2016), an alternative to the softmax operator defined by:
sparsemax(z)
arg min |pp z||2 which corresponds the euclidean projection of onto the K- k-1
probability simplex A k-1 = {p RK 1pp 1,p 2 0). The sparsemax operator ensures the sparsity the attention coefficients and encourages the assignment of each node to a single centroid. It further alleviates the need for entropy minimization done in DiffPool.
4

---

Under review as a conference paper at ICLR 2020
Eq. (5) also prevents the selected centroid nodes from being assigned to other clusters. Moreover, notice the term 3, that regularizes the value of the attention for each node. We can define 3 dive
where ve is the shortest path distance between each node o VP and centroids in Vc. Although this regularization incurs cost (((V)) it will strengthen the affinity to closer centroids and ensure the connectivity of the resulting pooled graph. Note that this regularization is considered as a hyperparameter of the layer and can be turned off, alternatively, the mapping can be restricted to centroids within a fixed hop neighborhood of each node.
Finally, after C' computed at layer I, the coarsened graph G(I+1)_ = (V(+1), AU+1),, x(++)) is computed using Eq. (6). as in (Ying et al. 2018). In these equations, My is neural network with trainable parameters L that is used to update the embedding of nodes in G(I+1) after the mapping.
1 = c( C() E RRIE 1
x++ My (())
(6)
This process be repeated by feeding the new graph G(++) into another GNN layer.
3.4 PROPERTIES OF THE LAPOOL METHOD
Preservation of Structural Information In addition to identifying graph nodes as centroids for pooling in a data-driven way, LaPool retains the feature content of the other nodes in a graph via the soft-assignment of followers to their centroids.
Substructure Identification By construction, the soft assignment of nodes to centroids clusters existing substructures of the graph together, thus identifying important subgraphs according to the classification task. By controlling for differences signal variations within neighborhoods, we encourage these clusters to be spread out across different areas of the graph.
Dynamic Cluster Dimension As discussed in section 3.2, LaPool offers the unique flexibility of determining the clustering dynamically, when training graphs sequentially, or statically when performing batch training.
Permutation Invariance It is trivial to show that LaPool is permutation invariant as long as the GNN used as its basis is permutation invariant, since both the graph downsampling (Eq. 3 and the node mapping (Eq. ( not affected by any permutation the vertex set.
Emphasizing the Strong Features Similar to how most CNNs implement a max max-pooling layer to emphasize the strong features, LaPool does so by selecting the nodes with high signal as centroids. For molecular graphs, the centroids are biased towards high degree nodes and atoms different than their neighbors (e.g. Nitrogen in Carbon ring).
RESULTS AND DISCUSSION
A fundamental objective of LaPool is to learn interpretable representation of sparse graphs, notably molecular substructures. We argue that this is an essential step towards shedding light upon the decision process within neural networks and ultimately increasing their utility in the design of new drug-like molecules. This implies that GNNs should be able to identify semantically important substructure components from molecular graphs, and eventually reconstruct these graphs from such components. This stems from the intuition that molecular validity and functional properties derive more from chemical fragments than individual atoms.
Our experimental results thus aim to empirically demonstrate the following properties of LaPool, as benchmarked against current state-of-the-ar ahe pooling models and the Graph Isomorphism Network:
LaPool's s consideration of semantically important information such as node distance trans- lates to improved performance on molecular understanding and molecular activity prediction tasks.
Visualization of LaPool's 's behaviour at the pooling layer demonstrates its ability identify coherent and meaningful molecular substructures.
The hierarchical representation enforced by LaPool, which preserves the original graph structure improves model interpretability.
5

---

Under review as a conference paper at ICLR 2020
Learning meaningful substructures can be leveraged to construct generative model which leads to more realistic and feasible molecules.
Throughout our experiments, we use the same architecture for all models to ensure an even comparison across all pooling layers: 2 layer GNNs with 128 filters each before the optional pooling layer, followed by 2 GNNs with 64 filters each and two fully connected layers including the output layer. Detailed information on architectural tuning and pooling-specifte hyper-parameter search are provided in Supplemental Section A.
4.1 BENCHMARK ON MOLECULAR GRAPH UNDERSTANDING
DiffPool and Graph U-Net models have been shown to outperform standard graph convolution networks on several graph benchmark datasets (Ying et al.] 2018 Gao and Ji, 2018). Although not explicitly stated, both methods are most effective when the graph signal is smooth. In such cases where adjacent nodes tend to be similar, the DiffPool procedure will cluster together nodes in the same neighborhood, maintaining the overall graph structure, while Graph U-net will select nodes in the same neighborhood and will not create isolated components that no longer exchange information. On molecular graphs. however, the graph signal is rarely smooth. Therefore, we expect these two methods to be less effective at identifying the important molecular substructures.gived that they do not explicitly consider structural relationships. We demonstrate this empirically by extracting known molecular substructure information from publicly available molecular datasets and evaluating performance in identifying these structures. We use a subset of approximately 17,000 molecules extracted from the ChEMBL database (Li et al. 2018c) and benchmark all methods on different types of substructures to verify the robustness of the comparison.
As shown in Table I the capture of structural relationships translates into superior performance of LaPool, as measured across standard metrics on various substructure prediction tasks. Indeed, we find that for predicting the presence of both 86 molecular fragments arising purely from structural information, as well as 55 structural alerts associated with molecule toxicity. LaPool globally outperforms other pooling models and GIN for the F1 and ROC-AUC metrics iniero-averaged to deal with high class imbalance). In particular, on the harder and extremely imbalanced molecular alerts prediction task, all models performed poorly compared LaPool, suggesting that the hierarchical representation learned by LaPool helps to achieve a better understanding of the molccular graphs.
Table I: Molecular fragment prediction results on ChEMBL dataset (5-fold cross vross-aaliittion..
Fragments
Alerts
Fl
ROC-AUC
F1
ROC-AUC
GIN
99.436 + 0.545
99.991 + 0.004
31.759 + 3.728
82.495 + 8.429
DiffPool
97.961 + 0.384
99.967 + 0.025
48.638
9.916
76.537 0.241
Graph U-net
95,469 + 1.414
99.962 + 0.033
37.585 = 2.978
85.124 + 8.447
LaPool
98.980 + 0.506
99.994 + 0.000
78.592 7.217
94.164 + 1.784
4.2 EXPERIMENTS ON STANDARD GRAPH CLASSIFICATION BENCHMARKS
In addition to evaluating molecular structural understanding of the pooling models, we benchmark our model on molecular toxicity prediction using the TOX21 dataset (Council al. 2007). We further conduct experiments on non-molecular benchmark graph datasets (DD, PROTEINS, FRANKEN- STEIN), which usually contain larger and often denser graphs compared molecular graphs (see Supplemental section C for dataset statistics). For TOX21, we report the test ROC-AUC averaged over 5-folds (following the 80- 10-10 split proportion used Wu et al. (0188). while we follow prior work (Ying et al. 2018) on the remaining datasets by reporting the best accuracy 10-fold cross-validation.
As shown in Table LaPool outperforms all other approaches on the well known TOX21 dataset and on the PROTEINS and FRANKENSTEIN, both of which contain non-molecue con-molecular graphs with size similar the TOX21 molecules. In particular, the PROTEINS dataset LaPool achieved an

---

Under review as a conference paper at ICLR 2020
accuracy of 83.83, representing significant gap relative DiffPool, its closest competitor, at 77.25. This suggests that the LaPool method is not restricted to molecular data but has broad applicability, especially in the context of sparse graph classification.
Table 2: Performance evaluation on standard benchmark graphs
TOX21
DD
PROTEINS
FRANKENSTEIN
GIN
82.90 + 0.69
77.97
66.47
68.20
DiffPool
82.37 0.90
85,88
77.25
69.12
Graph U-net
81.41 0.60
77.40
74.50
66.16
LaPool
83.42 0.97
81.36
83.83
69.74
4.3 MOLECULAR GENERATION
We aim to showcase LaPool's utility in drug discovery by demonstrating that it can be leveraged to generate molecules. In previous work, GANs and VAEs were used generate either string or graph representations of molecules. Here, we use the GAN- --sedded Wasserstein Auto-Encoder recently proposed in (Tolstikhin al., 2017) model the data distribution of molecules (see Figure 1 in Supplemental Material). For the encoder, we use similar network architecture as in our supervised experiments. The decoder and discriminator are simple MLPs, with complete architectural details provided in Supplemental Section A.4. Although the encoder is permutation invariant, the decoding process may not be. To force the decoder to learn a single graph ordering. we use a canonicalization algorithm (Schneider et al. 2015) that reorders atoms to ensure unique graph for each molecule. We further improve the robustness of our generative model to node permutations by computing the embedding, parameterized by GIN, both the
reconstruction loss using permutation-invariae input and reconstructed graphs (see Supplemental Section A.4.2). We find that such formulation improves the reconstruction loss and increases the ratio of valid molecules generated.
Dataset and Baseline Models Following previous work on molecular generation, we evaluate our generative model with an encoder enhanced by the LaPool layer (referred to as WAE-LaP) on the QM9 molecular dataset (Ramakrishnan et al. (014)). This dataset contains 133,885 small drug organic compounds with up to 9 heavy atoms (C. O. F). We compare WAE-LaP alternatives within our WAE framework where either no pooling is used (WAE-GNN) or where DiffPool is used as the pooling layer (WAE-Diff). Our results are also compared to previous results on the same dataset, including Grammar- VAE, GraphVAE, and MolGAN.
Evaluation Metries We measure the performance of the generative model using metrics standard in the field: validity (proportion of valid molecules from generated samples), uniqueness (proportion of unique molccules generated), and novelty (proportion of generated samples not found in the training set). All metrics were computed
set of 10,000 generated molecules.
Table 3:
Performance comparison of generative models on QM9, Values are reported percentages and haseline results are taken from (De Cao and Kipi 2018).
WAE-GNN
WAE-Diff
WAE-LaP
Grammar- VAE
GraphVAE
MolGAN
% Valid
96.8
97.2
98.8
60.2
91.0
98.1
% Unique
50.0
29.3
65.5
9.3
24.1
10.4
% Novel
78.9
78.9
78.4
80.9
61.0
94.2
As shown in Table WAE-LaP generated the most valid and unique molecules compared to all WAE- based generative models with slightly lower but similar novelty. Although MolGAN performed best on the novelty metric, it has among the lowest percentage of unique molecules. We hypothesize that the decrease in novelty observed with LaPool might be a result its pooling mechanism encouraging fragment novelty during sampling, thus limiting novelty resulting from rearrangement at the atom level. Nevertheless, as WAE-based methods produced similar proportions of novel molecules, results still suggest that combining LaPool with other generative approaches could improve the uniqueness and validity of generated compounds. We therefore conclude that the pooling performed by LaPool improve molecular graph representation, which is crucial in generative setting.
7

---

Under review as a conference paper at ICLR 2020
4.4 IMPROVED INTERPRETABILITY
To better understand the insights provided by LaPool, we conduct model interpretability experiments on molecular fragment prediction. Here we refer interpretability as the degree to which a human (in this context, medicinal chemist) can understand the cause of the model's decision (Miller) 2018). This explains our focus on fragment prediction since in that setting an "interpretable model" that achieves high performance would need to first understand the graph structure and the relationship between nodes.
LaPool (ours)
DiffPool
Graph U-net
Figure 2: Visualization of node selection and/or clustering performed by DifiPool, Graph U-net and LaPool for structural alert prediction. Dynamic clustering was used for LaPool, while the cluster size (k) resulting in the the best model used for DiffPool(k 7) and Graph U-net (A: 5). The top graph the pooled graph. The bottom graph is the original molecule, with the pic-charts representing the cluster affinity cach node, For LaPool and Graph U-net, the bold nodes respectively represent the chosen centroids and selected nodes.
We first investigate the behavior of LaPool and DiffPool by analyzing the clustering made at the pooling layer level. This comparison was limited to DiffPool since Graph U-net does not perform node clustering. We argue that an interpretable pooling layer should preserve the overall graph structure after pooling and produce meaningful clusters that could provide insight into the contribution of each molecular subgraph from the perspective of an expert chemist. While defining what is meaningful inherently subjective, we attempt to shed light on these models by observing their behavior in the drug discovery domain, using understanding of chemical structure reference.
We show in Figure 2 that LaPool is able to coarsen the molecular graphs into sparsely connected graphs, which can be interpreted as the skeleton of the molecules. Indeed, the data-driven dynamic segmentation it performed is akin to chemical fragmentation (Gordon et al.) 2011). In contrast, DiffPool's cluster assignment is more uniform across the graph, leading to densely connected coarsened graphs which are less interpretable from a chemical viewpoint. In particular, fails in the presence of molecular symmetry, to the same clusters. This is illustrated in both example (c) which shows how DiffPool creates fully connected graph from an originally disconnected graph, and example (b) which shows how symmetric elements, despite being far from each other, are assigned identically. On the other hand, we observe that Graph U-net ignores the graph structure, typically disconnecting it. also appears very biased toward selecting atoms in similar environment to ones already selected. Such failures are not present when using LaPool, since the dynamic centroid selection and the subsequent distance regularization enforce preservation of the molecular graph structure. A typical failure case for LaPool seen in (e) and corresponds to a missing centroid node in a given region of the graph, which results in soft assignment of the region multiple clusters. However, this behavior is inherent to most DiffPool samples since the fixed number of clusters and the inability to consider node distance cannot account for the diversity of molecular datasets.
encourages the mapping of nodes with similar features
8

---

Under review as a conference paper at ICLR 2020
GIN
DIPPOOL
GRAPH U-Net
LAPOOL
GROUND TRUTH
Score -.81
Grore -0.59
#cor--10
SCOrE -
SOOW 0.00
SCOnE OR1
SOOR - 3.17P
SCOPE
SCCPE 1.D
BOOKS
BOOTm 1.00
Score 1.0
SCONE DJDO
SCOVE 0.30
SCCIE - 0.222
SOUT
Figure Example of node importance for GIN, DiftPool. Graph U-Net and LaPool models. The ground truth shown on the right and nodes are highlighted according to their predicted importance. computed interpretability score indicated below each molecules and the hest performing models on each example are highlighted in gray. The computed interpretability score is shown for each model.
In addition to assessing the quality of clustering performed by LaPool and DiffPool, we attempt to directly target interpretability by computing explanation of each model decision and comparing to a ground truth. We design a simple experiment in which we predict the presence either Epoxide, Thioepoxide, Aziridine substructures (denoted by the molecular pattern "CI(O,S,NJCI"', that are indicative of molecular toxicity. Interpretability is therefore defined as the accuracy of the importance attributed by each model to relevant substructures of the input molecules. given the presence of the underlying ground truth fragment we wish to predict. Similar to (Pope et al. 2019). adapt an existing explainability method for CNNs to GNNs. Specifically, the integrated gradient (Sundararajan et al., 2017) over the input node features due to its stability and robustness in the presence of zero-value features (see Supplemental section C for discussion and alternate approach). Next, we derive an importance score for each node using the L1-norm of the feature attribution map for the node. By both qualitatively observing samples from the data (Figure 3. and by measuring the PR-AUC over the computed importance values given the ground truth to assess the ability to distinguish between important and non-important nodes (Table we find LaPool to more robustly identify the salient structure, resulting in improved overall interpretability. An interesting outcome of this experiment the performance of Graph U-net, ranked second best. Such performance is direct result of using cluster size large enough cover the toxic fragment size.
choose compute
Table 4: Comparison of prediction explainability based on average PR-AUC over node attribution
GIN
DiffPool
Graph U-Net
LaPool
avg. PR-AUC 0.876
0.799
0.879
0.906
5 CONCLUSION
In this work, we have proposed LaPool, a differentiable and robust pooling operator for molecular and sparse graphs that considers both node information and graph structure. In doing so, have proposed a method which able to identify important substructures graph by leveraging the graph Laplacian. In contrast with previous work, this method retains the connectivity structure and

---

Under review as a conference paper at ICLR 2020
feature information of the graph during the coarsening procedure, while encouraging nodes belonging to the same substructure to be mapped together in the coarsened graph.
Incorporating the proposed pooling layer into cxisting graph neural networks, we have demonstrated that the enforced hicrarchization allows for the capture of richer and more relevant set of features at the graph-level representation. We discussed the performance of LaPool relative existing graph pooling layers and demonstrated on both molecular graph classification and generation benchmarks that LaPool outperforms existing graph pooling modules and produces more interpretable results. In particular, we argue that the molecular graph segmentation performed by LaPool provides greater insight into molecular activity and that the associated properties can be leveraged in drug discovery. Finally, we show that although LaPool was designed for molecular graphs, it generalizes well to other graph types. In future work, we aim investigate how additional sources of information such as edge features could be incorporated into the graph pooling process.
REFERENCES
Rim Assouel, Mohamed Ahmed, Marwin H. Segler, Amir Saffari, and Yoshua Bengio. DEFactor: Differentiable Edge Factorization-baaee Probabilistic Graph Generation. 1811x [cs], November 2018. URL
arXiv: 1811,09766.
Siheng Chen. Rohan Varma. Aliaksei Sandryhaila, and Jelena Kovačević. Discrete Signal Processing Graphs: Sampling Theory. IEEE Transactions Signal Processing, 63164))651--665,,, December 2015a. ISSN 1053- 587X, 1941-0476. doi: (0.1109/SP..211522222 24669645.. URL http:///pp.....c/rgg///.oo abs abl.aa/s.115555111 arXiv: 1503.05432.
Siheng Chen, Rohan Varma, Aarti Singh, and Jelena Kovačević. Signal representations on graphs: Tools and applications. arXiv preprint 15X.:::512.0544: 2015b.
Siheng Chen, Rohan Varma, Aarti Singh, and Jelena Kovačević. Signal Representations on Graphs: Tools and Applications. arXiv: 152.::55//..540000 [cs, math), December 2015c. [RLL 05406 arXiv: 1512.05406.
National Research Council et al. Toxicity testing in the 2 century: a vision and a strategy. National Academies Press, 2007.
Shahaf Dafna and Carlos Guestrin. Learning Thin Junction Trees via Graph Cuts. In Artificial Intelligence and Statistics, pages 113-120, April 2009. URL. httt:://proseeiiiii wttps/s...pp.eeeaaaaaaesaa. daa.s///a//a/aaaaaaa html
ml
Nicola De Cao and Thomas Kipf. MolGAN: An implicit generative model for small molecular graphs. ICML 2018 workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018.
Michael Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks graphs with fast localized spectral filtering. Advances in neural information processing systems, pages 3844-3852, 2016.
Hongyang Gao and Shuiwang Ji. Graph U-Net. under review, September 2018 URL httt::////peeeeeieee meet/Eorumiid-HeePPPP
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals. and George E. Dahl. Neural Message Passing for Quantum Chemistry. arXiv:1704.01212 [cs]. April 2017. URL/attp://arxiv.com abs/ arXiv: 1704.01212.
4704.01212
lan Goodfellow, Jean Pouget-Abadis Abadie, Mehdi Mirza. Bing Xu, David Warde-Farley, Sherjil Ozair. Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
Mark S Gordon, Dmitri G Fedorov, Spencer R Pruitt, and Lyudmila V Slipchenko. Fragmentation methods: A route accurate calculations large systems. Chemical reviews, 112(1):632 672, 2011.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arXiv preprint arXiv:1410.5401, 2014.
Mikael Henaff, Joan Bruna, and Yann LeCun. Deep Convolutional Networks Graph-Structured Data. 106.v//50.00566 (esi, June 2015.
arXiv: 1506.05163.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Junction Tree Variational Autoencoder for Molecular Graph Generation, / 1802.04364 fes, stat], February 2018. URI http::htttt///i/eeeee/eeaaaatttt www.aarwww.oorg//.ooo 8028 04364 arXiv: 1802,04364.
10

---

Under review as a conference paper at ICLR 2020
Artur Kadurin, Sergey Nikolenko, Kuzma Khrabrov, Alex Aliper, and Alex Zhavoronkov. drugan: advanced generative adversarial autoencoder model for novo generation new molecules with desired molecular properties in silico. Molecular pharmaceutics, 14(9):3098 3104, 2017.
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of computer-aided molecular design, 30(8):955 608, 2016.
Diederik Kingma and Jimmy Ba. Adam: A method stochastic optimization. arXiv preprint arXiv: 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 13Xivv:112..61144 2013.
Matt Kusner, Brooks Paige, and José Miguel Hernández-Lobate Grammar variational autoencoder. In Proceedings of the 34th International Conference on Machine Learning-Voll Volume 70, pages 1945-1954. JMLR. org, 2017.
Anirban Laha, Saneem Ahmed Chemmengath, Priyanka Agrawal, Mitesh Khapra, Karthik Sankaranarayanan, and Harish G Ramaswamy. On Controllable Sparse Alternatives to Softmax. page 11, February 2016.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper Insights into Graph Convolutional Networks for Semi- Supervised Learning- arXiv:1801.001 0766007066600 [cs, stat]. January 2018a. htLp:///////// abb/ 1301. 07606 arXiv: 1801.07606.
Yiho Li, Liangren Zhang, Zhenming Liu. Multi-ohjective novo drug design with conditional graph generative model. Journal of Cheminformatics, 10(1):33, dec 2018b. ISSN 1758-2946. doi: 1186/
springeropenn.n.mmm URL htttp::// httpp://..cccctttt.pseeeeee..nnm com// art/aarriicesssse
028/-6
Yibo Li, Liangren Zhang, and Zhenming Liu. Multi-objective de novo drug design with conditional graph generative model. Journal of cheminformatics, 10(1):33, 2018c.
Yujia Li, Daniel Tarlow, Mare Broekschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv: 1..05499, 2015.
Yujia Li, Oriol Vinyals. Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning Deep Generative Models doi: 10. 10.1146/annurev
of Graphs. 2018d. ISSN 2326-8298.
(oo.144//auure-stttiiiisssss/s 0tatssiess-010814-200222 URL pptpt
//arxiv.on abs/ 80.3.03..0000 03324
Yao Ma, Suhang Wang, Charu C. Aggarwal, and Jiliang Tang. Graph Convolutional Networks with EigenPool- ing. arXiv:1904.13107 [cs, stat), April 2019. 1304.13107.
arXiv:
André F. T. Martins and Ramón Fernandez Astudillo. From Softmax to Sparsemax: Sparse Model of Attention and Lulti-Label Classification. vrXXiv1602.000 02068 [cs, stat), February 2016. URL/http://arxiv..om arXiv: 1602.02068.
abs / 602.02.00 02068
Tim Miller. Explanation artificial intelligence: Insights from social sciences. Artificial Intelligence, 2018.
Marcus Olivecrona, Thomas Blaschke, Ola Engkvist, and Hongming Chen. Molecular de-oovo design through deep reinforcement learning. Journal of Cheminformattic, 9(1):48, September 2017. ISSN 1758-2946. doi:
10. 10001166613321--07-22255 URL h
Phillip E Pope, Soheil Kolouri, Mohammad Rostami, Charles Martin, and Heiko Hoffmann. Explainability methods for graph convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1072,117711,011 2019.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp. and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1:140022, 2014.
David Rogers and Mathew Hahn. detended-cconeettie fingerprints. Journal of chemical information and modeling, 50(5):742 75,-744447....... 2010.
Nadine Schneider, Roger A Sayle, and Gregory A Landrum. Get your atoms in order an open-source implementation of a novel and robust molecular canonicalization algorithm. Journal of chemical information and modeling, 55(10):2111 2111022111122200 2015.
David I Shuman, Sunil K Narang, Pascal Frossard, Antonio Ortega. and Pierre Vandergheynst. The emerging field of signal processing on graphs: Extending high-dimensional data analysis to networks and other irregular domains. arXiv preprint 12 2012.
11

---

Under review as a conference paper at ICLR 2020
Martin Simonovsky and Nikos Komodakis. GraphVAE: Towards Generation of Small Graphs Using Variational Autoencoders. Technical report.
03480, pdf
Sangeetha Subramaniam, Monica Mehrotra, and Dinesh Gupta. Virtual high throughput screening (vhts)-a perspective. Bioinformation, 3(1):14, 2008.
Mukund Sundararajan, Ankur Taly, and Qigi Yan. Axiomatic attribution for deep networks. abs/1703.01365, 2017. URL http:// /ttp///axvvvvvee//////e.....v.v wttp./orgg/a//////../ 01365
CoRR,
Πya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. arXiv preprint arXiv:1711.015550 2017.
Ulrike Luxburg. A Tutorial Spectral Clustering. arXiv:0711.0189 [cs], November 2007. URL http:/ wtwpiv.iiiv..///i///eee/e/aae. org/ attp//bb./.........../.. 3189 arXiv: 0711.0189.
Zhengin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing. and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, (2):513 530, 2018.
Zonghan Wu. Shirui Pan, Fengwen Chen, Guodong Long, Chenggi Zhang, and Philip Yu. A comprehensive survey on graph neural networks. arXiv preprint arXiv:1901.005%, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? arXiv:
arXiv:1810.0083 [cs, stat]. October 2018. URL http:://axxv..rrg otgpab// 1810./110,... 00826 1810.00826.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. Hierarchical Graph Representation Learning with Differentiable Pooling. In Proceedings of the 32Nd International Conference Neural Information Processing Systems, NIPS' 18, pages 4805 4815, USA, 2018. Curran Associates Inc. URL http:: acm. www.nnotootitttt.....oooooo wtt.::oo....mmmmmmm.mmm33mmm 3327383 Montréal, Canada.
event-place:
Jiaxuan You, Bowen Liu, Zhitao Ying. Vijay Pande, and Jure Leskovec. Graph convolutional policy network for goal-directed molecular graph generation. In Advances in Neural Information Processing Systems, pages 6410-6421, 2018:t
Jiaxuan You, Rex Ying, Xiang Ren, William Hamilton, and Jure Leskovec. Graphrnn: Generating realistic graphs with deep auto-regressive models. arXiv preprint arXiv: 1802.0877, 2018h.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen, An end-to-ent deep learning architecture for graph classification. In Thirty- Secoon AAAI Conference on Artificial Intelligence, 2018.
12

---

Under review as a conference paper at ICLR 2020
A NETWORK ARCHITECTURE AND TRAINING PROCEDURE
Below, we describe the network architecture and the training procedure used for both supervised and generative experiments.
A.1 EDGE ATTRIBUTES
Part of the work presented assumes the absence of edge attributes the graphs. However, in molecular graphs, the nature of bond between atoms plays important role regarding activity and property. As such, edge types should be considered, especially in generative models. To consider this, add to our network an initial Edge-GC layer described in the following.
Let G (V, E, X) be an undirected molecular graph, such that E = [E,.... Ea] E {0, (0,1)xxnx0 where is the number of nodes in the graph and is the number of possible edge. We have that
Σ P A 13
1siSe
where A is the adjacency matrix of the graph. The Edge GC layer is simply defined as
Y Me, (EI, ... |M,, (Ec,
where || is the concatenation operator on the node feature dimension and Me, are graph neural networks parameterized learn different features for each edge type. A new graph defined as G² = (V, Y) can then be feed into the subsequent layers of the network.
A.2 ATOM AND EDGE FEATURES
In our experiments, the initial node feature tensor is represented by one-hot encoding of atoms (ignoring hydrogens) within the respective datasets and additional properties such as the atom implicit valence, its formal charge, number of radical electrons and whether it in molecular ring. For edge attributes, we consider the single, double and triple bond, which were enough to cover all molecules in the datasets, given that feature extraction was preceded by kekulization molecules.
A.3 SUPERVISED EXPERIMENTS
In all of our supervised experiments, we use graph convolution module consisting of two graph convolutional layers of 128 channels each with ReLU activation; followed by an optional hierarchical graph pooling layer; then two additional graph convolution layers (64) with skip connection to introduce jumping knowledge and gated global graph pooling layer (Li et al., 2015) to yield a graph-level representation. This is further followed by one fully connected layers (128) with batch normalization and ReI.U activation, finalized by a linear output layer with appropriate activation for the task readouts. Notice that used one pooling layer, since noticeable improvement was observed when using more in our experimental setting.
For DiffPool, performed hyperparameter search to find the optimal number of clusters (12.5% 25%, 50% of the maximum number of nodes in the batch (Ying et al. 2018)). A similar search is also performed for the Graph U-net pooling layer. For LaPool, we consider the same number of clusters and the dynamic node seelction. We also performed a grid search over the window size k used as regularization prevent nodes from mapping to centroids that are more than k-hop away as an alternative to the distance-regularizee version. The grid search was performed for k E (1,2,3).
For the supervised experiments, we use batch size of 64 and train the networks for 100 epochs, with early stopping.
A.4 GENERATIVE MODELS A.4.1
WAE MODEL
We use a Wasserstein Auto-Encoder (WAE) as our generative model (see Figure ST The WAE minimizes penalized form of the Wasserstein distance between model distribution and target
13

---

Under review as a conference paper at ICLR 2020
a) Adversarial autoencoder (AAE)
b) Encoder network
E
@0 (2)))
T
E-GINN Layer
GNN Layer
GNN Layer
Graph Pooling
G-SumPor
Layer
0M
D
La
C
Pv(x/z)
Figure $1; Model architecture for the generative model. (a) We use a WAE, in which a generator (auto-encoder) progressively learns the true molecular data distribution. (b) Architecture used for the encoder network.
distribution, and has been shown to improve learning stability. As described in (Tolstikhin al. 2017). we aim to minimize the following objective:
inf Ex (q(z))))))))))))))) (co/tt GZ)) ADz(Qz,, +D2(22 ()E
where any nonparametric set probabilistic encoders, D2 the Jensen-Shannon divergence between the learned latent distribution Qz and prior Pz, and X 0 is a hyperparameter. Dz is estimated using adversarial training (discriminator).
For generative model, the encoder follows similar structure as the network used for supervised experiments, with the exception being that the network now learns a continuous latent space qu (2[9) given set of imput molecular graphs 9 = [G1,... G.). More precisely, it consists of one edge graph layer, followed by two GCs (32 channels each), an optional hierarchical graph pooling, additional GC layers (64, 64), then global sum pooling step (128) and two fully connected layers (128), meaning the molecular graphs are embedded into a latent space of dimension 128. Instead of modeling the node/edge decoding with an autoregressive framework as done in recent works You et al. 20186 Assouel et al. 2018 Li et al.) 2018d) to capture the interdependency between them, we used a simple MLP that takes the latent code as input an pass through two fully connected layers (128, 64). The output of those layers is used as shared embedding for two networks: one predicting the upper triangular entries of the edge tensor, and the second predicting node features tensor. This results in faster convergence.
For the discriminator, we use simple MI.P that predicts whether the latent code comes from a normal prior distribution z (O, 1). This MLP is constituted by two stacked FCLs (64, 32) followed by an output layer with sigmoid activation. As in (Kadurin et al. 2017). we do not use natch-normalization. since it resulted in mismatch between the discriminator and the generator.
All models use the same basic generative architecture, with the only difference being the presence of pooling- llyeer and its associated parameters. For DiffPool, we fixed the number of cluster to three, while for LaPool, we use the distance-based regularization.
A.4.2 RECONSTRUCTION LOSS
For each input molecular graph G (V, E, X), the decoder reconstruct graph Ō (V, Since we use canonical ordering (available in RDKit) to construct G from the SMILES representa- tion of molecules, the decoder is forced to learn how to generate a graph under this order. Therefore, the decoding process is not necessarily able to consider permutations on the vertices set, and genera- tion of isomorphic graphs will be heavily penalized in the reconstruction loss. (Simonovsky and Komodakis), the authors argue that it suffices to compute the reconstruction loss on (yG) and r()), where y is permutation invariant embedding function. As a heuristic, we used a Graph Isomorphism Network (GIN), with weights fixed 1, in order to approximate the Weisfeiler-Lehmme graph isomorphism test (see (Xu
expensive graph matching procedure overcome that limitation. We
14

---

Under review as a conference paper at ICLR 2020
et al. 2018) for more details). In particular, edge-aware GIN layer (see section A.1 to embed both G and G. The reconstruction loss is then defined as:
Crec
1 [ [6(G))
-6(0))~7())))
-
(10)
Our experiments show that this loss function was able to produce a higher number of valid molecules, although we speculate that such a heuristic might prove harder to optimize on datasets with larger graphs.
A.4.3 TRAINING PROCEDURE
The QM9 dataset was split into train (60%), valid (20%) and hold-out test dataset (20%). Note that only 25% of the training set is sampled during each epoch (batch size 32). The generator network (encoder-decoder) and the discriminator network are trained independently, using the Adam optimizer Kingma and Ba (0014) with an initial learning rate of le for the generator and le - for the discriminator. During training, generator, on platcau. To stabilize the learning process and prevent the discriminator from becoming "too good" at distinguishing the true data distribution from the prior, train the generator times more often.
slowly reduce the learning rate by factor of 0.5, for the
B GENERATED MOLECULES
Below, we highlight a few molecules generated by WAE-LaP on the QM9 dataset.
Figure S2: Example molecules generated by WAF-LaP. Hydrogen atoms are not shown for simplicity.
C DATASET STATISTICS
Table 5: Dataset statistics and properties.
TOX21
DD
PROTEINS
FRANKENSTEIN
Avg. nodes
18.51
284.32
39.05
16.83
Avg. edges
19.23
715.66
72.82
17.88
#Graphs
8014
1178
1113
4337
#Classes
D NODE IMPORTANCE INTERPRETABILITY SCORE
In addition to assessing the quality of clustering performed by LaPool and DiffPool, attempt to measure the interpretability of their predictions. consider a setting in which the goal is to predict the presence of either Epoxides, Thioepoxides Aziridines substructures in molecular graphs. These
15

---

Under review as a conference paper at ICLR 2020
three fragments correspond to structural alerts that are often indicative of molecular toxicity. We attempt to identify the relational inductive bias used by each model during prediction. In our setting, define the interpretability of model as its ability to focus nodes that are directly relevant to the structural alerts and leverage that information for its prediction. In other words, we expect the most important nodes for the model prediction correspond to nodes that part of the structural alerts. We measure the importance of each atom toward the model prediction using the Integrated gradient method. Briefly, spectrum, then integrate the gradient of each of the model loss with respect to both the perturbed adjacency matrix and node features. Similar saliency maps, we then take the sum of the absolute integrated gradients over each node as an approximate attribution score for the nodes. Finally, we compute the interpretability score using the Precision- Recall AUC between measured importance and ground truth which is defined by binary mask of nodes that are part of the structural alerts. The PR-AUC allows us to assess the node importance separation capacity of each model while taking imbalance into account. We only focus positive predictions for each model. As an alternative to the Integrated Gradient, we also measure the interpretability score using Guided Baopgaatnn (see Table 6
compute perturbation of node and edge attributes over continuous
Table 6: Comparison of prediction explainability based on average PR-AUC over node attrihution using various explainability framework.
GIN
DifiPool
Graph U-net
LaPool
Integrated Gradient
0.876
0.799 0.739
0.879 0.835
0.906 0.857
Guided BacpPropnn
0.819
E SIGNAL PRESERVATION THROUGH LAPLACIAN MAXIMA
We illustrate here on a 1-d signal S, how using the Laplacian maxima serves to retain the most prominent regions of the graph signal, after smoothing (Figure S3). We measure the energy conser- vation after downsampling: 5g = ESS) - E(S of the T-d signal energy to highlight why selecting the Laplacian maxima allow reconstructing the signal with low error when compared to the minimum Laplacian (which focuses on low frequencies). The energy Es of a discrete signal yi is defined in ( and similar the energy of a wave in a physical system (without the constants).
Es =
Σ y/
(11)
Signal intensity
Random smooth 1D signal 25 nodes, Ey 71.1
4
Maximum Laplacian pooling 8 leaders, Eş = 72.5 Minimum Laplacian pooling leaders, Eş = 76.3
2
0
5
10
15
20
25
Node number
Figure S3: Comparison maximun/minimum Laplacian pooling random and smoothed signal on ID graph with 25 nodes. The graph energy E's is indicated.
To mimic the molecular graph signal at the pooling stage, the given signal is built from an 8-terms random Fourier series with added Gaussian noise, then smoothed with 2 consecutive neighbor average smoothing. For the pooling methods, a linear interpolation is used to cover the same signal space before computing Es. As expected, the maxima Laplacian selection requires fewer number of samples for signal reconstruction and energy preservaton. It also significantly outperforms minima selection.
16


Reviews:
[{"review_id": "rkxeUi4AYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n\nThe authors propose a new pooling layer, LaPool, for hierarchical graph representation learning (Ying et al., 2019) by clustering nodes around centroids that are selected based on \"signal intensity variation\". The signal intensity variation of node x is defined as sum_{y in HOP(x, h)} ||x - y||  where HOP(x, h) is the set of nodes reachable from x within h hops. Once top k maximizers are selected as centroids (k can be predetermined or dynamically chosen), a sparse cluster assignment distribution is computed for each node using sparsemax (Laha et al., 2018), and the affinity matrix and the node embeddings are coarsened as in Ying et al. (2019). The authors show that LaPool can improve performance in various graph-related tasks over baselines and generate interpretable clusters. \n\n\nStrengths\n\n- Explicit centroid selection based on signal intensity variation seems like an intuitive idea worth investigating. \n\n- LaPool seems to be empirically effective, in particular outperforming Graph U-Net which is probably the most relevant baseline (also k-max pooling for hierarchical graph representation learning). But I'm not an expert on the considered tasks, so I cannot judge how significant these results are. \n\n\nWeaknesses: \n\n- The paper has issues with clarity. There seem to be many sloppy notations as well as unclear (possibly wrong) arguments.\n\n1. Isn't equation (1) true for unnormalized graph Laplacian (D - A), not normalized? \n\n2. In the proof of Theorem 1, why is it that C C' X = X (i.e., X is in range(C))? This is a crucial step that I'm not sure why is true. I might be missing something, but it's not clear to me in the current version. \n\n3. Sloppy notations. What do \"A^h\", \"||L X||_{R^d}\", \"top_k(V|L S)\" mean in equation (4)? I can infer their meaning, but do I have to? This also interferes with my other confusions: back to question #1, is it true that S = ||L X||_{R^d}? \n\n4. Is the number of clusters fixed or dynamic for these experiments? Figure 2 seems to be dynamic based on the main text, but I cannot tell if it's the case for other experiments. Based on the provided code, it seems the default number of clusters is always 10% of the number of nodes.\n\n- The paper doesn't do a good job of contextualizing itself. Graph U-Net seems to be the most relevant previous work, but there's no discussion of how this work relates and why it's better. For instance, is it the case that Graph U-Net is not interpretable because there's no explicit clustering? It'd be much clearer to spell out such differences. \n\n- It's unclear why certain design choices are made and why they're better. For instance, is swapping entropy minimization with sparsemax necessarily better? I understand not every design decision should be (or needs to be) justified, but it's helpful if it is to understand what helps and what doesn't.\n\n\nSummary\n\nThe core idea of the paper, clustering nodes based on signal intensity for hierarchical graph representation learning, is interesting and seems to be useful in practice, but the paper has issues with clarity and the rest of the framework is a bit limited in novelty (DiffPool + sparsemax + GIN).", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxrWMtptH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new pooling approach \"Laplacian pooling\" for graph neural networks, which the authors claim is able to better preserve information about the local structure, and to provide interpretability.  Namely, the pooling approach is based on finding centroids (nodes having high signal variation compared to their neighbors, via graph-Laplacian) and assigning other nodes to be \"followers\" based on a soft-attention mechanism. The authors add these new pooling layers to existing GNN architectures and show improved performance on problems of classification and generative modeling of molecular graphs. The paper also extends CNN interpretability techniques (integrated gradients) to GNNs.\n\nI am borderline on the paper -- I'll give a weak accept rating for now. The proposed Laplacian-pooling ideas could be interesting, and the results encouraging -- but I found the mathematical motivation to be not very convincing, and has various (mostly correctable issues). The paper can be viewed as more of an engineering effort, which attempts to find practical tricks aimed at modeling molecular structures. What I like about the paper is that the authors make an earnest attempt to model the domain (biochemistry) -- for example they realize that a graph formalism for molecular structures is rather simplistic, and misses many important details -- such as different types of bonds (which require different types of edges). The authors also realize that typical neighborhood smoothing (diffusion) that makes sense for say spatial or social network graphs may not make sense for molecular graphs, where specific substructures (e.g. presence of a benzene ring) may be highly indicative for some classification tasks. The paper also contains a collection of interesting practical heuristics and observations (mainly in appendix) to help train GNN models for classification and generative modeling -- which researchers in the field may find valuable. I am not sure if the idea of coarsening (via hierarchical pooling) can be meaningfully applied to a wide-variety of natural molecules - but it does seem to make sense for some organic molecules -- e.g. protein chains. \n\nDetailed comments: \n1. The paper makes an analogy between band-pass filtering and the proposed approach. In my opinion the analogy is rather weak -- while it may carry over to spatial graphs (e.g. grid graphs) but may not apply to more complex graphs -- e.g. graphs where each node is at most a few hops away from any other node. It's not clear in what sense (3) corresponds to high-pass filtering. Can you show that it's somehow related to filtering-out the large (low-pass) eigenvalues of the graph-laplacian?\n\n2. There is a typo in equation (1) -- equality of the quadratic form f'L f requires an unnormalized definition of the graph laplacian D - A, instead of I - D^{-1} A. \n\n3. Notation in equation (3) -- is unclear and undefined.  What is ||L X||_{R^d} ? is that a norm (giving a scalar), or concatenation?  What is Top_k(V | L*S) -- the readers have to guess. You may be relying on notation from existing papers -- but still need to set up notation to be self-consistent. Subtle comment: \"signal intensity variation\" sounds like the variation of signal intensity -- e.g. something like difference of signal norms. Perhaps intensity of signal variation is a better term.  Sparsemax is also undefined. What is A^h -- the h-hop adjacency matrix? \n\n4. I do not understand what do you mean by \"information preservation\" after pooling -- and I did not understand the importance of the \"structure-aware feature content\" definition, and the value of theorem 1.  Matrix C in the derivation is undefined. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByxxBKYotB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a graph down-sampling component named LaPool. It uses the graph signal to dynamically select some \"important\" centroids and learn a sparse assignment matrix for clustering the remaining nodes.\n\nThis paper should be rejected due to the following reasons.\n\n1. Strange new results compared with its previous version\n\nThe paper has a previous arxiv version. While the method does not change, the performance in this current submission has dramatic change compared with previous version: the proposed model seems much improved, while some important baselines (that outperform the proposed model) only have 50% of it previous performance in this submission.\n\nFor example, for Table 1 in the current version, the proposed model reached almost perfect scores in both F1 and AUC. However in previous version for the same setting and experiment (in Table 2 and 3 of its arxiv version), the performance are much lower especially the Structural alert prediction results.  Their previous results, which show the model does not perform better than DiffPool .\n\nFor the structural alert prediction results on DiffPool, below I copied and pasted the results from the arxiv version\n                                  \n\nTable 3: Structural alert prediction results \n------------------------------------------------------------------------------------------------------------\n                                             Tox21                   |               ChEMBL\n                       F1-macro F1-micro ROC-AUC F1-macro F1-micro ROC-AUC\n------------------------------------------------------------------------------------------------------------\nGIN                     78.9        68.3             72.6           93.6         76.7          59.2\nDiffPool              79.2        68.0             75.6           94.5        83.3          59.3\nGraph U-net      71.1        47.6             67.9           92.9        68.1          59.3\nLaPooldistance 80.6        74.2             73.5           95.2        81.3          59.5\nLaPoolunreg     81.3        72.8             74.1           94.1        75.8          58.9\nLaPool3hop       79.1        71.6             74.8           93.8        75.0          59.1\n------------------------------------------------------------------------------------------------------------\nOne example: \n(1) in the ICLR version, The F1 score for DiffPool is only 48.638 ± 9.916 on ChEMBL data (about 50% of its previous level) but F1 for the proposed method is improved. Why is that? \n(2) same as baseline  Graph U-net , the ICLR version reports F1 37.585 ± 2.978, why is that?\n(3) same as GIN, in the ICLR version, F1 is only 31.759 ± 3.728, less than 50% of its previous level in arxiv version.\n\nThe author needs to justify this dramatic change.\n\n2. Although LaPool can dynamically select centroids, for a dense graph such as a complete graph, only one centroid will remain there since there is only one node which has larger signal variation than all its neighbors, as shown in Eq. 4. This consequently hurts the model performance on more dense graphs. That may be why in Table 2, on dense data \"DD\", LaPool performs much worse than baseline DiffPool. Also on another dense data \"FRANKENSTEIN\", LaPool does not performs significantly better than DiffPool.\n \n\n3, the evaluation of interpretability is not convincing. This paper considers  \"interpretability as the degree to which a human (in this context, a medicinal chemist) can understand the cause of the model’s decision\". Therefore, the conclusion that this model is more interpretable is based on only one person's subjective judgement.  Even so, from the scores the model does not outperform baseline \"GIN\" that much.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new pooling approach "Laplacian pooling" for graph neural networks and applies this to molecular graphs. While the paper has been substantially improved from its original form, there are still various concerns regarding performance and interpretability that remain unanswered. In its current form the paper is not ready for acceptance to ICLR-2020.
Decision: Reject