Topological Relational Learning on Graphs

1 Introduction . Node classification is one of the most active research areas in graph learning . The target here is , given a single attributed graph G and a small subset of nodes with prior label information , to predict labels of all remaining unlabelled nodes . Applications of node classification are very diverse , ranging from customer attrition analytics to tracking corruption-convictions among politicians . Graph neural networks ( GNNs ) offer a powerful machinery for addressing such problems on large heterogeneous networks , resulting in an emerging field of geometric deep learning ( GDL ) which adapts deep learning to non-Euclidean objects such as graphs [ 9 , 51 , 54 , 16 ] . Although GDL achieves a highly competitive performance in various graph-based classification and prediction tasks , similarly to the image domain deep learning on graphs is often found to be vulnerable to graph perturbations and adversarial attacks [ 43 , 50 , 26 ] . In turn , most recent results [ 42 , 19 ] suggest that local graph information may be invaluable for robustifying GDL against graph perturbations and adversarial attacks . Also , as shown by [ 36 , 4 , 53 ] in conjunction with network community learning , local algorithms ( i.e. , algorithms based only on a small radius neighborhood around the node ) may demonstrate superior performance if coupled with side information in the form of a node labeling positively correlated with the true graph structure . Inspired by these results , the ultimate goal of this 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . paper is to introduce the idea of local algorithms with local topological side information on similarity of node neighborhood shapes into GNN . By shape here , we broadly understand data characteristics which are invariant under continuous transformations such as stretching , bending , and compressing , and to study such graph shapes properties , we invoke the machinery of topological data analysis ( TDA ) [ 21 , 11 ] . Topological Relational Inference : from Matchmaking to Adversarial Graph Learning and Beyond In particular , to capture more complex graph properties and enhance model robustness , we introduce the concept of topological relational inference ( TRI ) and propose two novel options for information passing which rely on the local topological structure of each node : ( i ) Inject a new topology-induced multiedge between two nodes based on shape similarity of their local neighborhoods ; ( ii ) Enrich each individual node features by harnessing local topological side information from its neighbors . The rationale behind our idea is multi-fold . First , we assess not only global graph topology and relationships between the feature sets of two individual nodes , as with the current diffusion mechanisms and random walks on graphs , but we also examine important interactions between shapes of the feature sets of the node neighborhoods . As a result , our new topology-induced multigraph representation ( TIMR ) of graph G in ( i ) strengthens the relationship among nodes which might not be ( yet ) connected by a visible ( or `` tangible '' ) edge in G but whose higher order intrinsic characteristics are very similar . The intuitive analogy here might be with a matchmaking agency who aims to connect two people on a blind date , based on a careful assessment of interplay among similarities of their interests , socio-demographics as well as that of their close friends , rather than bringing in two individuals together through a random walk among all available profiles of potential candidates . Another example is the customer churn and retention analytics based on peer effects [ 15 ] . That is , the goal here is to classify the node as potential churn customer based not only on the individual attributes but on interactions among its neighbor attributes . Second , such a new multiedge in ( i ) , based on shape similarity of node local neighborhoods may assist not only the matchmaking agency in coupling the most appropriate individuals ( or link prediction in less romantic and more technical terms ) , but to enhance resilience of the graph structure and associated graph learning . Indeed , a new multiedge in ( i ) can be used as an essential remedial ingredient in graph defense mechanisms as criteria to clean the perturbed graph , i.e. , to reconstruct edges removed by the attacker or to suppress edges induced by the attacker , depending on the strength of the topology-induced link . Third , by learning shape properties within each local node neighborhood , our TRI allows for systematic recovering of intrinsic higher-order interactions among nodes well beyond a level of pair-wise connectivity , and such local topological side information in ( ii ) enhances accuracy in node classification tasks even on clean graphs . Significance of our contributions are the following : • We propose a novel perspective to graph learning with GNN – topological relational inference , based on the idea of similarity among shapes of local node neighborhoods . • We develop a new topology-induced multigraph representation of graphs which systematically accounts for the key local information on the attributed graphs and serves as an important remedial ingredient against graph perturbations and attacks . • We derive theoretical stability guarantees for the new local topological representation of the graph and discuss their implications for the graph algebraic connectivity . • Our expansive node classification experiments show that TRI-GNN outperforms 14 state-of-theart baselines on 6 out 7 graphs and delivers substantially higher robustness ( i.e. , up to 10 % in performance gains under noisy scenarios ) than baselines on all 7 datasets . 2 Related Work . Graph Neural Networks Inspired by the success of the convolution mechanism on image-based tasks , GNNs continue to attract an increasing attention in the last few years . Based on the spectral graph theory , [ 10 ] introduced a graph-based convolution in Fourier domain . However , complexity of this model is very high since all Laplacian eigenvectors are needed . To tackle this problem , ChebNet [ 18 ] integrated spectral graph convolution with Chebyshev polynomials . Then , Graph Convolutional Networks ( GCNs ) of [ 30 ] simplified the graph convolution with a localized first-order approximation . More recently , there have been proposed various approaches based on accumulation of the graph information from a wider neighborhood , using diffusion aggregation and random walks . Such higher-order methods include approximate personalized propagation of neural predictions ( APPNP ) [ 31 ] , higher-order graph convolutional architectures ( MixHop ) [ 3 ] , multi-scale graph convolution ( N-GCN ) [ 2 ] , and Lévy Flights Graph Convolutional Networks ( LFGCN ) [ 14 ] . In addition to random walks , other recent approaches include GNNs on directed graphs ( MotifNet ) [ 35 ] , graph convolutional networks with attention mechanism ( GAT , SPAGAN ) [ 48 , 52 ] , and graph Markov neural network ( GMNN ) [ 39 ] . Most recently , Liu et al . [ 34 ] consider utilizing information on the node neighbors ’ features in GNN , proposing Deep Adaptive Graph Neural Network ( DAGNN ) . However , DAGNN does not account for the important information on the shapes of the node neighborhoods . Persistent Homology for Graph Learning Machinery of TDA and persistent homology ( PH ) is increasingly widely used in conjunction with graph classification , that is , when the goal is to predict a label for the entire graph rather than for individual nodes . Such tools for graph classification with persistent topological signatures include kernel-based methods [ 46 , 40 , 33 , 55 , 32 ] and neural networks [ 24 , 12 ] . All of the above methods consider the task of classifying graph labels and are based on assessing ‘ global ’ graph topology , while our focus is node classification and our approach is based on evaluating local topological graph properties ( i.e. , shape of individual node neighborhoods ) . Integration of PH to node classification is virtually unexplored . To the best of our knowledge , the closest result in this direction is PEGN-RC [ 56 ] . However , the key idea of PEGN-RC is distinctly different from our approach . PEGN-RC reweights only each existing edge , based on the topological information within its edge vicinity and , in contrast to TRI-GNN , neither compares any shapes , nor creates new or removes existing edges . Importantly , PEGN-RC does not integrate topology of both graph and node attributes , while TRI-GNN does . 3 Preliminaries on Topological Data Analysis and Persistent Homology . The machinery of topological data analysis ( TDA ) and , particularly , persistent homology offer a mathematically rigorous and systematic framework of tools to evaluate shape properties of the observed data , that is , intrinsic data characteristics which are invariant under continuous deformations such as stretching , compressing , and bending [ 58 , 21 , 37 ] . The main premise is that the observed data which can be , as in our case , a graph G or a point cloud in a Euclidean or any finite metric space constitute a discrete sample from some unknown metric spaceM . Our goal is then to recover information on some essential properties of M which has been lost due to sampling . Persistent homology addresses this reconstruction task by counting occurrences of certain patterns , e.g. , loops , holes , and cavities , within shape ofM . Such pattern counts and functions thereof , called topological signatures are then used to characterize intrinsic properties of G. The approach is implemented in two main steps . We start from associating G with some nested sequence of subgraphs G1 ⊆ G2 ⊆ . . . ⊆ Gm = G. Then we monitor evolution of pattern occurrences ( e.g. , cycles , cavities , and more generally n-dimensional holes ) in this nested sequence of subgraphs . To ensure a systematic and computationally efficient manner of pattern counting , we construct a simplicial complex Ci ( e.g. , a clique complex ) induced by Gi . The sequence { Gi } induces a filtration : nested sequence of simplicial complexes C1 ⊆ C2 ⊆ . . . ⊆ Cm = C. Now we can not only track patterns but also evaluate the lifespan of each topological feature . Let bσ be the index of the simplicial complex Cbσ at which we first record ( i.e. , birth ) the n-dimensional topological feature σ ( n-cycle ) , while simplicial complex Cdσ be the first complex we observe its disappearance ( i.e. , death ) . Then lifespan or persistence of the topological feature σ is dσ − bσ . To evaluate all topological features together , we consider a persistence diagram ( PD ) where the multi-set Dn = { ( bσ , dσ ) ∈ R2 : dσ > bσ } ∪∆ records the birth and deaths of all n-cycles in the filtration { Ci } . Here , ∆ = { ( t , t ) |t ∈ R } is the diagonal set containing points in PD , counted with infinite multiplicity . Different persistent diagrams can be compared based on the cost of the optimal matching between points of the two diagrams , while avoiding topological noise near ∆ [ 13 ] . Depending on the question at hand , we can construct different suitable filtrations relevant to the problem . In this paper , we consider two different filtrations . First , we consider the sublevel filtration based on a node degree function f : V 7→ N. As degree is an integer valued function , so are our thresholds { αi } ⊂ N. Our sublevel filtration is then defined as follows . Let Vαi = { u ∈ V | f ( u ) ≤ αi } . Then , Gαi is the subgraph generated by Vαi . In particular , the edge euv ∈ E is in Gαi if both u and v are in Vαi . We call this degree based filtration . In addition , we consider a second filtration defined by the edge-weight function on the graph where edge weights are induced by similarity of node attributes . We call this attribute based filtration ( See Section 4 ) . Note that degree based filtration is induced only by the graph properties , while attribute based filtration is constructed by using the features coming from the observed data .

Summary: To tackle the over-smoothing problem in GNNs, this work proposes to use topological relational inference for systematically learning local graph structures. The similarity between two local graphs are measured by the persistent homology. A “multiedge” is added between two nodes whose local graphs are similar enough.  The local topological side information is added enrich the features of each node. Theoretical stability guarantees were derived for the above new graph representation. The experimental results provide empirical evidence to support the proposed approach.