GIR Framework: Learning Graph Positional Embeddings with Anchor Indication and Path Encoding

1 INTRODUCTION . Graph , as an important data structure , is a powerful tool to represent ubiquitous relationships in the real world . Learning vector representations for graph data , benefits many downstream tasks on the graph such as node classification ( Kipf & Welling , 2017 ) and link prediction ( Zhang & Chen , 2018 ) . Many graph representation learning methods have been proposed recently , among those , Graph Neural Networks ( GNNs ) , inheriting the merits of neural networks , have shown superior performance and become a much popular choice . Existing GNN models mainly follow the message passing neural network ( MPNN ) ( Gilmer et al. , 2017 ) pattern , which stacks message passing layers that aggregate information from neighborhoods and then update representations for each node . Typical MPNNs are limited by 1-Weisfeiler-Lehman test ( Xu et al. , 2019 ) , and lack of ability to capture the position information within the graph ( You et al. , 2019 ) , without distinguishable node/edge attributes , nodes in a different part of the graph with topologically equivalent neighborhood structures or even with different substructures may be embedded into the identical representation by typical MPNNs alone ( You et al. , 2019 ; Li et al. , 2020 ) , as shown in figure 1 ( b ) , A1 and B2 can not be distinguished with MPNNs and no distinguishable attributes ( when ignoring colored anchor nodes ) . Researchers have developed methods to alleviate this issue . Some earlier works adopt one-hot encodings as extended node attributes ( Kipf & Welling , 2017 ) . More recent methods utilize graph distance information to get position-aware embeddings . Anchor-based GNNs ( You et al. , 2019 ; Liu et al. , 2019 ) select anchor nodes as positioning bases , and use position information related to anchors to break the structural symmetry ( Figure 1 ( c ) ) . Distance encoding ( Li et al. , 2020 ) utilizes distance information to push typical MPNNs beyond the 1-Weisfeiler-Lehman test limitation ( Figure 1 d ) . Those existing position-aware GNN methods mainly explicitly pre-compute the distance between node pairs , utilizing them as node attributes or controlling of message passing steps , and then perform message passing encoding . Due to the design choices for what and how to utilize distance information , those existing methods consider graph structure and attributes information encoded by GNN propagation and position information in a relatively separate manner , potentially useful interaction between intermediate results of the distance computing and encoding stages may be ignored . In this paper , we follow the anchor-based GNN strategy and aim at performing a more flexible interaction with graph structure , positioning and attribute information . With limited fixed anchor positioning base selected , abundant position information could be assigned , boosting the performance of MPNNs in the position aware scenarios in a trackable way . Besides the explicit assigning strategy , here we mainly focus on conveying position information implicitly , in order to retain the potential possibility for more flexible usage of the graph data , as some information beyond pre-performed graph algorithms ( e.g . shortest path distance algorithm ) may potentially useful to be captured . Our motivation comes from the structural similarity between GNN message passing and distance relaxation process of the Bellman-Ford shortest path algorithm ( Bellman , 1958 ) , we show that with a simple indication of anchors and appropriate design for message passing functions , MPNNs can keep track of the shortest paths from anchors . Further , inspired by a relaxation order improved variant of Bellman-Ford algorithm ( Moore , 1959 ) , we propose a generalized MPNN architecture , termed Graph Inference Representation ( GIR ) , as a more specialized model for the position-aware scenario . The GIR model propagates messages from anchors along paths to each node , and outputs of the k-th layer encode k-step representations related to anchors . On an unweighted graph with no distinguishable node attributes , the proposed strategies provide natural ways to help break the structural symmetry as the previous anchor-based GNNs could ( Figure 1 ( b & e ) ) . Our contributions are summarized as follows : 1 . We propose a general anchor labeling strategy , enabling MPNNs for mimicking the Bellman-Ford algorithm and getting position-aware embeddings implicitly under anchorbased GNN pattern and discuss theoretical implications and experimental realizations . 2 . We propose the Graph Inference Representation ( GIR ) model that contains more specialized structure for learning position-aware embeddings . 3 . Empirically , we evaluate the performance of proposed methods for tasks in position-aware datasets , experimental results show that our position-aware GIRs achieve generally higher performance . The rest of the paper is organized as follows . Section 2 reviews related works . Section 3 introduces notations and definitions , theoretical implications are discussed . Section 4 details the proposed GIR framework . Section 5 presents the experiment and gives a discussion of the results . Finally , section 6 presents our conclusion . 2 RELATED WORKS . Our work follows the anchor-based GNN pattern from the position-aware GNN literature , with the design inspired by the Bellman-Ford algorithm . 2.1 POSITION-AWARE GNNS . Position-aware GNNs leverage distance information in the message propagating encoding process . Distance Encoding ( Li et al. , 2020 ) follows the MPNN propagation design and extends it with preassigned graph distance from the propagation target to each node ( Figure 1 , ( d ) ) . For each message propagation step on a target node v , only distance from v are utilized , potentially useful distance from other nodes are not considered . Anchor-based GNNs use selected anchor nodes as a positioning base , existing methods mainly follow the two-stage pattern : select anchor nodes first , and then encode the information related to anchors . Position-aware GNN ( PGNN ) ( You et al. , 2019 ) selects anchor node sets randomly before running every forward of the model to get a low distortion embedding capturing global position information , a PGNN layer directly propagates message from the selected anchor node set to each target node , weighted by the pairwise distance . The random anchor selecting strategy of PGNNs leads to unstable limitation , some more recent work uses fixed anchor nodes instead to overcome this issue . AGNN ( Liu et al. , 2019 ) pre-selects fixed anchors by minimum point cover nodes algorithm ; GraphReach ( Nishad et al. , 2020 ) follows the fix anchor setting and adopts random walk reachability estimations instead of the shortest path distance . Those methods adopt the strategy that ignores graph structure when performing message propagation . Our GIR framework follows the anchor-based GNN pattern and differs from previous works in the position information encoding and utilizing strategy ( depicted in figure 1 ) . GIRs are capable of encoding position information implicitly , taken intermediate results in computing shortest path distance into consideration , and under specific settings , corresponding distances related to anchors and the graph structure are utilized together . 2.2 BELLMAN-FORD ALGORITHM . The Bellman-Ford algorithm ( Bellman , 1958 ) computes the shortest path distance from a single source node to each node in a weighted digraph . For a graph G = ( V , E ) , source node s , the algorithm performs |V| − 1 relaxation iterations , maintaining the shortest path from s with at most i edges in the i-th iteration . A relaxation step iterate over all edges , correct distance to better ones . The relaxation of distance on i-th iteration and edge 〈u , v〉 with weight w is defined as , distiv = min ( dist i−1 v , dist i−1 u + w ) i > 0 ( 1 ) where dist0 is initialized as , dist0v = { 0 v = s ∞ otherwise ( 2 ) A variant of Bellman-Ford algorithm ( Moore , 1959 ) notices that if the distance value of node v has not been changed since its last relaxed , edges out of v are no need to perform relaxation . Learning graph algorithm including the Bellman-Ford algorithm with MPNN has been experimented in Velikovi et al . ( 2020 ) , they show the effectiveness of adopting max-pooling as aggregator , and propose to learn with intermediate results of graph algorithms . Our work goes further toward the implication from the Bellman-Ford algorithm , and proposes a more specialized message passing strategy beyond typical MPNN for this . In addition , experiment setting of Velikovi et al . ( 2020 ) focus on a single specific source node , and in the experiment on synthetic datasets , we generalize this to adapt the anchor-based GNN settings . 3 PRELIMINARIES . 3.1 NOTATIONS . A graph can be represented as G = ( V , E ) , where V = { v1 , · · · , vn } is the node set and E = { 〈vi , vj〉|vi , vj ∈ V } is the edge set . Nodes are augmented with the feature matrix X , which is either input attributes if available or placeholders . Edges may augmented with the optional weights W . In-neighborhoods of node v are represented as N ( v ) . Notations are summarized in table 1 . 3.2 POSITION-AWARE EMBEDDINGS . One goal of the anchor-based GNN model is to utilize anchors as bases to encode position-aware information for each node . To capture this intuition , PGNNs ( You et al. , 2019 ) view embeddings as position-aware if the shortest path distance between node pairs could be reconstructed from their embeddings , which is hard for models with fixed anchors to satisfy . Here we define position-aware embeddings related to anchors ( Definition 1 ) . Considering that the shorest path distance to a node set has different definitions , such as the shorest path distance to all anchor nodes or to any node in the anchor node set , or whether to focus on shorest path with limited hop . Here in the definition 1 , we focus on the limited hop shorest path distance ( for the alignment with MPNN ) to any node in specific node sets , for retaining a high generality , and the union of those node sets need to be the anchor node set , for consisting with the position-aware embeddings related to anchors definition . Definition 1 ( Position-aware Embeddings ) . For A ⊆ P ( A ) , ⋃ A = A , where P ( A ) is the power set of node set A , the node embedding Z = { zi = f ( vi ) |∀vi ∈ V } , where f is a graph encoder that maps a node to its embedding , is k-hop A -position-aware if there exists functions { gA′ ( · ) |∀A′ ∈ A } such that gA′ ( zi ) = dsp ( vi , A′ ) , where dsp ( vi , A′ ) is the k-hop shortest path distance between node vi and any node in the node set A′ in the graph G. If all elements in A are singleton , brakets of sets inner A could be omitted . As examples for definition 1 , we could reconstruct k-hop shorest path distance from any node in the anchor set A from k-hop { A } -position-aware embeddings , and k-hop shorest path distances from all anchor nodes from k-hop A-position-aware embeddings , these two settings will be discussed most commonly in following sections . 3.3 INDICATABILITY . Here we discuss the theoretical implicitly of our GIR framework design . We begin with our motivation of mimicking the Bellman-Ford algorithm with MPNNs , the concept of indicatability is introduced as an effective tool for the design of the GIR framework . A natural idea for an MPNN to mimic the Bellman-Ford algorithm is to perform neural relaxation functions and enable the capability of keeping track of the intermediate limited hop shortest path distances . The motivation of mimicking the Bellman-Ford algorithm and getting position-aware embeddings implies the capability for reconstructing k-hop shortest path distance from k-th layer . To capture this intuition , we introduce the concept of indicatability that implies the existence and constructibility of functions that reconstruct specific information related to a node set from a vector representation . Definition 2 ( Indicatability ) . For A′ ⊆ A , the function set f is A′-indicatable over function f if there exists a constructable function fA′ such that for all vi ∈ A , fA′ ( zi ) = f ( vi , A′ ) . The indicatability ( Definition 2 ) is defined over a pre-defined function f that maps a node vi and a node set A′ to a real value ( e.g . the multi-souce shortest path distance function ) , which could be mimicked by a neural network module that takes zi , the embedding of vi as input and could generate outputs include f ( zi , A′ ) , as the target neural network module f is capable for mimicking f over A′ , we say that f is A′-indicatable over f . Moreover , the k-hop A -position-aware definition ( Definition 1 ) can be restated as the existence of f-dist with A′-indicatability over k-hop limited multi-souce shorest path distance function , for all A′ ∈ A .

Summary: In this paper, the authors propose the anchor-based framework for position encoding.   It is well known that the message-passing framework inherently has a limitation to encode graph structure and sometimes fails to discriminate isomorphic subgraphs.  Graph Inference Representation exploits the distance relaxation process of the Bellman-Ford shortest path algorithm.  The authors show that the MPNN framework can keep track of the shortest paths using anchors.  To this end, the authors introduce the anchor labeling strategy for MPNNs mimicking the Bellman-Ford algorithm.  It has a more specialized structure and outperforms baseline models.