Structural Landmarking and Interaction Modelling: on Resolution Dilemmas in Graph Classification

1 INTRODUCTION . Complex systems are ubiquitous in natural and scientific disciplines , and how the relation between component parts gives rise to global behaviour of a system is a central research topic in many areas such as system biology ( Camacho et al. , 2018 ) , neural science ( Kriegeskorte , 2015 ) , and drug and material discoveries ( Stokes et al. , 2020 ; Schmidt et al. , 2019 ) . Recently , graph neural networks provide a promising architecture for representation learning on graphs – the structural abstraction of a complex system . State-of-the-art performances are observed in various graph mining tasks ( Bronstein et al. , 2017 ; Defferrard et al. , 2016 ; Hamilton et al. , 2017 ; Xu et al. , 2019 ; Velickovic et al. , 2017 ; Morris et al. , 2019 ; Wu et al. , 2020 ; Zhou et al. , 2018 ; Zhang et al. , 2020 ) . However , due to the non-Euclidean nature , important challenges still exist in graph classification . For example , in order to generate a fixed-dimensional representation for a graph of arbitrary size , graph pooling is typically adopted to summarize the information from each each node . In the pooled form , the whole graph is squeezed into a “ super-node ” , in which the identities of the constituent sub-graphs and their interconnections are mixed together . Is this the best way to generate graph-level features ? From a complex system ’ s view , mixing all parts together might make it difficult for interpreting the prediction results , because properties of a complex system arise largely from the interactions among its components ( Hartwell et al. , 1999 ; Debarsy et al. , 2017 ; Cilliers , 1998 ) . The choice of the “ collapsing ” -style graph pooling roots deeply in the lack of natural alignment among graphs that are not isomorphic . Therefore pooling sacrifices structural details for feature ( dimension ) compatibility . Recent years , substructure patterns1 draw considerable attention in graph mining , such as motifs ( Milo et al. , 2002 ; Alon , 2007 ; Wernicke , 2006 ; Austin R. Benson , 2016 ) and graphlets ( Shervashidze et al. , 2009 ) . It provides an intermediate scale for structure comparison or counting , and has been applied to node embedding ( Lee et al. , 2019 ; Ahmed et al. , 2018 ) , deep graph kernels ( Yanardag & Vishwanathan , 2015 ) and graph convolution ( Yang et al. , 2018 ) . However , due to combinatorial nature , only substructures of very small size ( 4 or 5 nodes ) can be considered ( Yanardag 1Informally , substructure in this paper means a connected subgraph and will be used interchargeably with it . & Vishwanathan , 2015 ; Wernicke , 2006 ) , greatly limiting the coverage of structural variations ; also , handling substructures as discrete objects makes it difficult to compensate for their similarities , and so the risk of overfitting may rise in supervised learning scenarios ( Yanardag & Vishwanathan , 2015 ) . We view these intrinsic difficulties as related to resolution dilemmas in graph-structured data processing . Resolution is the scale at which measurements can be made and/or information processing algorithms are conducted , and here we will discuss two types of resolution and related dilemmas : the spatial resolution ( dilemma ) and the structural resolution ( dilemma ) . Spatial resolution relates to the geometrical scale of the “ component ” that can be identified from the final representation of a graph ( based on which the prediction is performed ) . In GNN , since graph pooling compresses the whole graph into a single vector , node and edge identities are mixed together and the spatial resolution drops to the lowest . We call this vanishing spatial resolution ( dilemma ) . Structural resolution is the fineness level in differentiating between two substructures . Currently practice of exact matching makes it computationally intractable to handle the exponentially many sub-graph instances , and the risk of overfitting may also rise as observed in deep graph kernels ( Yanardag & Vishwanathan , 2015 ) and dictionary learning ( Marsousi et al. , 2014 ) . We will call this over-delicate substructure profiling an exploding structural resolution ( dilemma ) . In fact , these two resolution dilemmas are not isolated . They have a causal relation and the origin is the way we perform identification and comparison of discrete substructures ( more in Section 2.3 ) . Our contribution . Inspired by the well-studied science of complex systems , and in particular the importance of the interacting relation between component parts of a system , we propose a simple neural architecture called “ Structural Landmarking and Interaction Modelling ” - or SLIM . It allows graphs to be projected onto a set of end-to-end optimizable , well-aligned structural landmarks , so that identities of graph substructures and their interactions can be captured explicitly to explain the complexity and improve graph classification . We show that , by resolving the two resolution dilemmas , and subsequently respecting the structural organization of complex systems , SLIM can be empirically very promising and offers new possibilities in graph representation learning . In the rest of the paper , we will first define the resolution dilemmas of graph classification in Section 2 , together with the discussion of related works . We then cover in Section 3 , 4 and 5 the design , analysis , and performance of SLIM , respectively . Finally , the last section concludes the paper . 2 RESOLUTION DILEMMAS IN GRAPH CLASSIFICATION . A complex system is often composed of many parts interacting with each other in a non-trivial way . Since graphs are structural abstraction of complex systems , accurate graph classification depends on how global properties of a system relate to its structure . It is believed that property of a complex system arises from interactions among its components ( Debarsy et al. , 2017 ; Cilliers , 1998 ) . Consequently , accurate interaction modelling should benefit prediction . However , it is non-trivial due to resolution dilemmas , as described in the following subsections . 2.1 SPATIAL RESOLUTION DIMINISHES IN GRAPH POOLING . Graph neural networks ( GNN ) for graph classification typically involves two key blocks , graph convolution and graph pooling ( Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Xu et al. , 2019 ) , at significantly different spatial resolutions . The goal of convolution is to pass information among neighboring nodes in the general form of hv = AGGREGATE ( { hu , u ∈ Nv } ) , where Nv is the neighbors of v ( Hamilton et al. , 2017 ; Xu et al. , 2019 ) . Here , the spatial resolution is controlled by the number of convolution layers : more layers capture lager substructures/sub-trees and can lead to improved discriminative power ( Xu et al. , 2019 ) . In other words , the spatial resolution in the convolution stage can be controlled easily , and multiple resolutions may be even combined together via CONCATENATE function ( Hamilton et al. , 2017 ; Xu et al. , 2019 ) for improved modelling . The goal of graph pooling is to generate compact , graph- or subgraph-level representations that are compatible across graphs . Due to the lack of natural alignment between non-isomorphic graphs , graph pooling typically “ squeezes ” a graph G into a single vector ( or “ super-node ” ) in the form of hG = READOUT ( { f ( hv ) , ∀v ∈ V } ) , where V is the vertex set of G. The readout functions include : max-pooling ( Cangea et al. , 2018 ) , sumpooling ( Xu et al. , 2019 ) , some other pooling functions ( Hamilton et al. , 2017 ) , or deep sets ( Zaheer et al. , 2017 ) ; attention has been used to evaluate node importance in attention pooling ( Lee et al. , 2019 ) and gPool ( Gao & Ji , 2019 ) ; hierarchical pooling has also been investigated ( Ying et al. , 2018 ) . The spatial resolution drops significantly in graph pooling , as shown in Figure 1 . Since all the nodes ( and their representation ) are mixed into one vector , subsequent classifier can no longer identify any individual substructure regardless of the spatial resolution used in the convolution stage . We call this “ diminishing spatial resolution ” . A diminishing spatial resolution will mix the identity of sub-structures ( e.g. , functional modules of a molecule ) , making it non-trivial to trace the behaviour of the classifier back to meaningful parts of the graph for interpretation . 2.2 STRUCTURAL RESOLUTION EXPLODES IN SUBSTRUCTURE IDENTIFICATION . Substructures are the basic unit to accommodate interacting relations . A global criterion to identify and align substructures is the key to preserving substructure identities and comparing the inherent interactions across graphs . Again , the granularity in determining whether two substructures are “ similar ” or “ different ” is subject to a wide spectrum of choices , which we call “ structural resolution ” . We illustrate the concept in Figure 2 . The right end denotes the finest resolution in differentiating substructures : exact matching , as we manipulate motif/graphlet ( Milo et al. , 2002 ; Alon , 2007 ; Wernicke , 2006 ; Yang et al. , 2018 ; Shervashidze et al. , 2009 ) . The exponential configuration of subgraphs will finally lead to an “ exploding ” structural resolution , because maintaining a large number of unique substructures is computationally infeasible , and easily overfits ( Yanardag & Vishwanathan , 2015 ) . The left end of the spectrum treats all substructures the same and underfits the data . We are interested in a medium structural resolution , where similar substructures are mapped to the same identity , which we believe can benefit the generalization performance ( empirical evidence in Fig . 5 ) . 2.3 RELATION BETWEEN SPATIAL AND STRUCTURAL RESOLUTION DILEMMAS . The two resolution dilemmas have a causal relation , and the logic chain is as follows . 1 . Due to difficulty of characterizing discrete subgraphs , exact matching is typically adopted . 2 . As a result , an exploding structural resolution ( dilemma ) is caused . 3 . Such an over-delicate granularity makes it infeasible to compare substructure across graphs . 4 . As a result , a collapsing-style graph pooling has to be adopted that summarizes the whole graph into a single vector , serving as compatible graph-level features . 5 . As a result , a vanishing spatial resolution ( dilemma ) caused finally . Namely , the exploding structural resolution makes ( collapsing-style ) graph pooling an inevitable choice , which in turn leads to diminishing spatial resolution . Since the root cause of exploding structural resolution is how we typically manipulate discrete sub-structures , i.e. , exact matching , we will replace it with structural landmarking in SLIM , so that both dilemmas are coordinately solved . 3 STRUCTURAL LANDMARKING AND INTERACTION MODELLING ( SLIM ) . The key idea of SLIM is to compute landmarks ( or representatives ) from the distribution of substructures ( embedded in a continuous space ) across different graphs . By doing this , identification and comparison of sub-structures become much easier , and so an identity-preserving graph pooling becomes applicable that explicitly models interaction between component parts of a graph . Problem Setting . Give a set of labeled graphs { Gi , yi } ’ s for i = 1 , 2 , ... , n , with each graph defined on the node/edge set Gi = ( Vi , Ei ) with adjacency matrix Ai ∈ Rni×ni where ni = |Vi| , and yi ∈ { ±1 } . Assume that nodes are drawn from c categories , and the node attribute matrix for Gi is Xi ∈ Rni×c . Our goal is to train an inductive model to predict the labels of the testing graphs . The SLIM network has three main steps : ( 1 ) substurcture embedding , ( 2 ) substructure landmarking , and ( 3 ) identity-preserving graph pooling , as shown in Figure 3 . Detailed discussion follows .

Summary: The proposed SLIM algorithm organizes graph neural networks around substructures surrounding "landmarks" in the graph.  In addition to presenting the three steps of the SLIM algorithm (sub-structure embedding, sub-structure landmarking, and "identity-preserving" graph pooling), the authors compare to other approaches on a graph classification problem.  A large part of the paper is also given over to a high-level discussion of "resolution dilemmas."