GIR Framework: Learning Graph Positional Embeddings with Anchor Indication and Path Encoding

1 INTRODUCTION . Graph , as an important data structure , is a powerful tool to represent ubiquitous relationships in the real world . Learning vector representations for graph data , benefits many downstream tasks on the graph such as node classification ( Kipf & Welling , 2017 ) and link prediction ( Zhang & Chen , 2018 ) . Many graph representation learning methods have been proposed recently , among those , Graph Neural Networks ( GNNs ) , inheriting the merits of neural networks , have shown superior performance and become a much popular choice . Existing GNN models mainly follow the message passing neural network ( MPNN ) ( Gilmer et al. , 2017 ) pattern , which stacks message passing layers that aggregate information from neighborhoods and then update representations for each node . Typical MPNNs are limited by 1-Weisfeiler-Lehman test ( Xu et al. , 2019 ) , and lack of ability to capture the position information within the graph ( You et al. , 2019 ) , without distinguishable node/edge attributes , nodes in a different part of the graph with topologically equivalent neighborhood structures or even with different substructures may be embedded into the identical representation by typical MPNNs alone ( You et al. , 2019 ; Li et al. , 2020 ) , as shown in figure 1 ( b ) , A1 and B2 can not be distinguished with MPNNs and no distinguishable attributes ( when ignoring colored anchor nodes ) . Researchers have developed methods to alleviate this issue . Some earlier works adopt one-hot encodings as extended node attributes ( Kipf & Welling , 2017 ) . More recent methods utilize graph distance information to get position-aware embeddings . Anchor-based GNNs ( You et al. , 2019 ; Liu et al. , 2019 ) select anchor nodes as positioning bases , and use position information related to anchors to break the structural symmetry ( Figure 1 ( c ) ) . Distance encoding ( Li et al. , 2020 ) utilizes distance information to push typical MPNNs beyond the 1-Weisfeiler-Lehman test limitation ( Figure 1 d ) . Those existing position-aware GNN methods mainly explicitly pre-compute the distance between node pairs , utilizing them as node attributes or controlling of message passing steps , and then perform message passing encoding . Due to the design choices for what and how to utilize distance information , those existing methods consider graph structure and attributes information encoded by GNN propagation and position information in a relatively separate manner , potentially useful interaction between intermediate results of the distance computing and encoding stages may be ignored . In this paper , we follow the anchor-based GNN strategy and aim at performing a more flexible interaction with graph structure , positioning and attribute information . With limited fixed anchor positioning base selected , abundant position information could be assigned , boosting the performance of MPNNs in the position aware scenarios in a trackable way . Besides the explicit assigning strategy , here we mainly focus on conveying position information implicitly , in order to retain the potential possibility for more flexible usage of the graph data , as some information beyond pre-performed graph algorithms ( e.g . shortest path distance algorithm ) may potentially useful to be captured . Our motivation comes from the structural similarity between GNN message passing and distance relaxation process of the Bellman-Ford shortest path algorithm ( Bellman , 1958 ) , we show that with a simple indication of anchors and appropriate design for message passing functions , MPNNs can keep track of the shortest paths from anchors . Further , inspired by a relaxation order improved variant of Bellman-Ford algorithm ( Moore , 1959 ) , we propose a generalized MPNN architecture , termed Graph Inference Representation ( GIR ) , as a more specialized model for the position-aware scenario . The GIR model propagates messages from anchors along paths to each node , and outputs of the k-th layer encode k-step representations related to anchors . On an unweighted graph with no distinguishable node attributes , the proposed strategies provide natural ways to help break the structural symmetry as the previous anchor-based GNNs could ( Figure 1 ( b & e ) ) . Our contributions are summarized as follows : 1 . We propose a general anchor labeling strategy , enabling MPNNs for mimicking the Bellman-Ford algorithm and getting position-aware embeddings implicitly under anchorbased GNN pattern and discuss theoretical implications and experimental realizations . 2 . We propose the Graph Inference Representation ( GIR ) model that contains more specialized structure for learning position-aware embeddings . 3 . Empirically , we evaluate the performance of proposed methods for tasks in position-aware datasets , experimental results show that our position-aware GIRs achieve generally higher performance . The rest of the paper is organized as follows . Section 2 reviews related works . Section 3 introduces notations and definitions , theoretical implications are discussed . Section 4 details the proposed GIR framework . Section 5 presents the experiment and gives a discussion of the results . Finally , section 6 presents our conclusion . 2 RELATED WORKS . Our work follows the anchor-based GNN pattern from the position-aware GNN literature , with the design inspired by the Bellman-Ford algorithm . 2.1 POSITION-AWARE GNNS . Position-aware GNNs leverage distance information in the message propagating encoding process . Distance Encoding ( Li et al. , 2020 ) follows the MPNN propagation design and extends it with preassigned graph distance from the propagation target to each node ( Figure 1 , ( d ) ) . For each message propagation step on a target node v , only distance from v are utilized , potentially useful distance from other nodes are not considered . Anchor-based GNNs use selected anchor nodes as a positioning base , existing methods mainly follow the two-stage pattern : select anchor nodes first , and then encode the information related to anchors . Position-aware GNN ( PGNN ) ( You et al. , 2019 ) selects anchor node sets randomly before running every forward of the model to get a low distortion embedding capturing global position information , a PGNN layer directly propagates message from the selected anchor node set to each target node , weighted by the pairwise distance . The random anchor selecting strategy of PGNNs leads to unstable limitation , some more recent work uses fixed anchor nodes instead to overcome this issue . AGNN ( Liu et al. , 2019 ) pre-selects fixed anchors by minimum point cover nodes algorithm ; GraphReach ( Nishad et al. , 2020 ) follows the fix anchor setting and adopts random walk reachability estimations instead of the shortest path distance . Those methods adopt the strategy that ignores graph structure when performing message propagation . Our GIR framework follows the anchor-based GNN pattern and differs from previous works in the position information encoding and utilizing strategy ( depicted in figure 1 ) . GIRs are capable of encoding position information implicitly , taken intermediate results in computing shortest path distance into consideration , and under specific settings , corresponding distances related to anchors and the graph structure are utilized together . 2.2 BELLMAN-FORD ALGORITHM . The Bellman-Ford algorithm ( Bellman , 1958 ) computes the shortest path distance from a single source node to each node in a weighted digraph . For a graph G = ( V , E ) , source node s , the algorithm performs |V| − 1 relaxation iterations , maintaining the shortest path from s with at most i edges in the i-th iteration . A relaxation step iterate over all edges , correct distance to better ones . The relaxation of distance on i-th iteration and edge 〈u , v〉 with weight w is defined as , distiv = min ( dist i−1 v , dist i−1 u + w ) i > 0 ( 1 ) where dist0 is initialized as , dist0v = { 0 v = s ∞ otherwise ( 2 ) A variant of Bellman-Ford algorithm ( Moore , 1959 ) notices that if the distance value of node v has not been changed since its last relaxed , edges out of v are no need to perform relaxation . Learning graph algorithm including the Bellman-Ford algorithm with MPNN has been experimented in Velikovi et al . ( 2020 ) , they show the effectiveness of adopting max-pooling as aggregator , and propose to learn with intermediate results of graph algorithms . Our work goes further toward the implication from the Bellman-Ford algorithm , and proposes a more specialized message passing strategy beyond typical MPNN for this . In addition , experiment setting of Velikovi et al . ( 2020 ) focus on a single specific source node , and in the experiment on synthetic datasets , we generalize this to adapt the anchor-based GNN settings . 3 PRELIMINARIES . 3.1 NOTATIONS . A graph can be represented as G = ( V , E ) , where V = { v1 , · · · , vn } is the node set and E = { 〈vi , vj〉|vi , vj ∈ V } is the edge set . Nodes are augmented with the feature matrix X , which is either input attributes if available or placeholders . Edges may augmented with the optional weights W . In-neighborhoods of node v are represented as N ( v ) . Notations are summarized in table 1 . 3.2 POSITION-AWARE EMBEDDINGS . One goal of the anchor-based GNN model is to utilize anchors as bases to encode position-aware information for each node . To capture this intuition , PGNNs ( You et al. , 2019 ) view embeddings as position-aware if the shortest path distance between node pairs could be reconstructed from their embeddings , which is hard for models with fixed anchors to satisfy . Here we define position-aware embeddings related to anchors ( Definition 1 ) . Considering that the shorest path distance to a node set has different definitions , such as the shorest path distance to all anchor nodes or to any node in the anchor node set , or whether to focus on shorest path with limited hop . Here in the definition 1 , we focus on the limited hop shorest path distance ( for the alignment with MPNN ) to any node in specific node sets , for retaining a high generality , and the union of those node sets need to be the anchor node set , for consisting with the position-aware embeddings related to anchors definition . Definition 1 ( Position-aware Embeddings ) . For A ⊆ P ( A ) , ⋃ A = A , where P ( A ) is the power set of node set A , the node embedding Z = { zi = f ( vi ) |∀vi ∈ V } , where f is a graph encoder that maps a node to its embedding , is k-hop A -position-aware if there exists functions { gA′ ( · ) |∀A′ ∈ A } such that gA′ ( zi ) = dsp ( vi , A′ ) , where dsp ( vi , A′ ) is the k-hop shortest path distance between node vi and any node in the node set A′ in the graph G. If all elements in A are singleton , brakets of sets inner A could be omitted . As examples for definition 1 , we could reconstruct k-hop shorest path distance from any node in the anchor set A from k-hop { A } -position-aware embeddings , and k-hop shorest path distances from all anchor nodes from k-hop A-position-aware embeddings , these two settings will be discussed most commonly in following sections . 3.3 INDICATABILITY . Here we discuss the theoretical implicitly of our GIR framework design . We begin with our motivation of mimicking the Bellman-Ford algorithm with MPNNs , the concept of indicatability is introduced as an effective tool for the design of the GIR framework . A natural idea for an MPNN to mimic the Bellman-Ford algorithm is to perform neural relaxation functions and enable the capability of keeping track of the intermediate limited hop shortest path distances . The motivation of mimicking the Bellman-Ford algorithm and getting position-aware embeddings implies the capability for reconstructing k-hop shortest path distance from k-th layer . To capture this intuition , we introduce the concept of indicatability that implies the existence and constructibility of functions that reconstruct specific information related to a node set from a vector representation . Definition 2 ( Indicatability ) . For A′ ⊆ A , the function set f is A′-indicatable over function f if there exists a constructable function fA′ such that for all vi ∈ A , fA′ ( zi ) = f ( vi , A′ ) . The indicatability ( Definition 2 ) is defined over a pre-defined function f that maps a node vi and a node set A′ to a real value ( e.g . the multi-souce shortest path distance function ) , which could be mimicked by a neural network module that takes zi , the embedding of vi as input and could generate outputs include f ( zi , A′ ) , as the target neural network module f is capable for mimicking f over A′ , we say that f is A′-indicatable over f . Moreover , the k-hop A -position-aware definition ( Definition 1 ) can be restated as the existence of f-dist with A′-indicatability over k-hop limited multi-souce shorest path distance function , for all A′ ∈ A .

Summary: This paper proposes a new type of anchor-based GNN by implicitly exploiting node positioning within customized message passing steps of MPNN. The framework consists of an anchor labeling strategy and a specified propagation path with the utilization of Bellman-Ford algorithm, to enable its positional awareness. The empirical results on small synthetic and real datasets show the effectiveness of implicit positional encoding, and achieve comparable results against other anchor-based models.