On Locality in Graph Learning via Graph Neural Network

1 INTRODUCTION . Graph Neural Network ( GNN ) is a family of machine learning ( ML ) models tailored for learning from graph-structured data ( Duvenaud et al. , 2015 ; Li et al. , 2017b ; Gilmer et al. , 2017 ; You et al. , 2019 ) . Recently , great success has been shown using models such as GCN ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2018 ) and GAT ( Veličković et al. , 2017 ) on tackling various graphrelated problems in domains such as chemistry ( Gilmer et al. , 2017 ) , biology ( Duvenaud et al. , 2015 ) , social networking ( Hamilton et al. , 2018 ; Chen et al. , 2018 ; 2017b ) and traffic networks ( Li et al. , 2017a ) . Despite their practicality and potential , theoretical understanding of the learning process of GNNs is still underexplored , especially given their difference in data dependency from other ML families . In most ML settings , data samples are assumed to be created independently and identically ( I.I.D . ) from a certain distribution . Because of this , uniform sampling is effective in many data selection processes such as training/validation/test set partition , batch sampling and initial labelling set selection ( Settles , 2009 ; Ren et al. , 2020 ) . For learning from graph-structured data , inherent dependencies exist among data samples , as captured by the graph structure . Taking training/validation/test set partition for example , different random partitions of the labelled nodes significantly impact the performance of the GNN model ( Shchur et al. , 2019 ) , especially when the size of the training set is small relative to the data population . Fig . 1 illustrates this phenomenon on the Cora data set ( Sen et al. , 2008 ) . This calls for a better understanding of the relation between the dependence among data samples and the learning process of GNN . Network homophily ( McPherson et al. , 2001 ) is a fundamental principle that describes the dependence among data samples in many real-world networks . Network homophily implies that nodes close by in ( graph ) distance tend to have similar features and labels . Most existing GNN designs assume strong homophily in the graph data ( Zhu et al. , 2020 ) and exploit it by propagating features and aggregating them within various graph neighbourhoods using different mechanisms ( Hamilton et al. , 2018 ; Veličković et al. , 2017 ; Kipf & Welling , 2017 ) . The capability to capture topological information from the underlying graph structure is the key to contribute to the success of GNN . The following question is raised : can we combine this network principle and topology awareness of GNN to derive useful insights for more effective graph learning via GNN ? Most existing efforts focus on the ( vertex ) sampling strategy to accelerate GNN training ( Chen et al. , 2018 ) or to conserve memory during GNN training ( Ying et al. , 2018 ) . There is little understanding or work on the underlying principle between the dependency of the data samples and the performance of GNN models Ma et al . ( 2021 ) . To address this question , we theoretically and experimentally investigate how and why different random partitions of training sets affect the learning outcome ( test performance ) of GNN in node-level tasks . We focus on graph data sets where the network homophily property holds . We formally show the spatial locality in the inference ability of GNN ( under certain conditions ) , i.e. , GNN predicts better on vertexes close to the training set . Our main contributions are summarized as follows . 1 . We prove that GNN predicts better on vertexes that are closer to the training set under certain assumptions . This implies that better coverage of the training set over the rest of the graph ( in terms of neighbourhoods of a small radius ) can translate into better generalization performance of the GNN model in the semi-supervised setting . 2 . We experimentally validate our theory and assumptions on Cora , Citeseer and PubMed data sets with prevailing GNN models such as GCN , GAT and GraphSAGE . 3 . Using the observation and theory as guidance , we formulate two optimization problems for studying ( 1 ) the “ cold start ” problem in active learning ( Grimova et al. , 2018 ) , and ( 2 ) minimal size of the labelled data set with respect to the model performance . The first optimization provides a strategy for selecting the most “ economic ” initial data set to be labelled and the other gives insights on how many vertexes should be labelled initially for the desired learning outcome of a GNN model . This work represents our first attempt to unwrap a better understanding of the learning process of GNN . It is novel to combine the network principle of the dependence among data samples and topology awareness of GNN to formally infer useful insights ( based on graph structure ) for GNN learning . The results suggest that the topology awareness of GNN is not only an important property to improve GNN performance , as shown in ( Li et al. , 2020 ; You et al. , 2019 ; Xu et al. , 2018 ; Nishad et al. , 2021 ) , but also useful guidance for making informed decisions in the learning process of GNN . 2 RELATED WORK . Expressive Power of GNN . Modern GNN architectures are inspired by the Weisfeiler–Lehman ( WL ) isomorphism test ( Leman & Weisfeiler , 1968 ) which uses the graph structure to propagate information ( Kipf & Welling , 2017 ) . One idea in characterizing the power of GNN is to measure its ability in differentiating different graph structures , referred to as the expressive power of GNN ( we refer to Sato ( 2020 ) for a survey on this line of works ) . In particular , Xu et al . ( 2018 ) proved that the expressive power of GNNs is no more than the WL test . This has inspired a number of studies on improving the expressive power of GNN by enabling it to differentiate graph structures that can not be distinguished by the WL-test ( Morris et al. , 2020 ) . The studies in ( You et al. , 2019 ; Li et al. , 2020 ; Nishad et al. , 2021 ; Zhou et al. , 2020 ) show that combining graph topology information ( distance ) into the learning process of GNN ( e.g. , for feature encoding , aggregation , normalization ) can improve the performance of GNN on various tasks . Generalization of GNN . Studies aiming to understand generalization performance of GNNs on node-level tasks are rather limited ( Ma et al. , 2021 ) . To our best knowledge , there are two very recent studies that investigate/demonstrate similar observations as ours . Zhu et al . ( 2021 ) studied the problem in GNN learning that the test set and the training set are generated from different distributions . They found the test error of GNNs is inversely related to their defined metric ( distance ) between test set and training set , which is similar to our discovery here . However , their work focused on designing a regularization mechanism to allow the learnt distribution to approximate the distribution in the test set . Ma et al . ( 2021 ) studied a similar problem as ours by extending the PACBayesian analysis for I.I.D . data to analyzing the generalization performance of GNNs . They prove that given a fixed training set , GNN generalization performance varies among different subgroups of the test population defined by a feature distance . Our study complements their study in the case that graph distance metric is used . Our analysis provides a different and more direct perspective for understanding how graph structure information influences GNN learning performance . 3 PRELIMINARIES . 3.1 GRAPH AND NOTATION . Let G = ( V , E ) be the input graph with node feature vector Xv for all v ∈ V . In this paper , we assume G is connected and focus our analysis on the node classification task where a label yv is associated with vertex v , and the objective is to learn a representation vector hv of v such that v ’ s label can be predicted as yv = f ( hv ) , where f is the prediction function . We use d ( x , y ) to denote the distance between x and y : ( 1 ) d ( . , . ) is defined to be the shortest path distance if x and y are vertexes in the graph ; ( 2 ) d ( . , . ) is the Euclidean distance if x and y are representations in the embedding space ; ( 3 ) d ( . , . ) is the distance between x and the closest element in Y , if Y is a set . For example , let u be a vertex and D be a set of vertexes , and then d ( u , D ) : = minv∈D { d ( u , v ) } . Let N and R denote the set of natural numbers and real numbers , respectively . R+ is the set of non-negative real numbers . 3.2 GRAPH NEURAL NETWORK . GNN combines the graph structure and node features Xv to learn a representation vector hv of node v. Modern GNNs adopt a neighborhood aggregation scheme , where the representation of a node is updated iteratively by aggregating representations of its neighbors . After k iterations of aggregation , a node ’ s representation captures the structural information within its k-hop neighborhood in the graph . LetH ⊆ Rm be the embedding space of the learnt representations of vertexes and m ∈ N is the dimension . Mθ : V 7→ H denotes the GNN model with parameter θ that maps a vertex v ∈ V into a representation vector hv in the embedding space H , following a neighborhood aggregation scheme . Let f : H 7→ Rc be the prediction function that maps an embedding vector hv to a vector in Rc , in which c ∈ N is the number of classes and each entry represents the probability of belonging to the respective class . The GNN prediction process can be viewed as composition of f ◦Mθ : V 7→ Rc . Let L : Rc 7→ R+ denote the loss function which evaluates how accurate the prediction is . As the graph structure ( topology ) among vertexes provides important information , it is important for GNN to preserve as much topological information as possible for embedding . Indeed , it has been formally and experimentally shown in ( Li et al. , 2020 ; You et al. , 2019 ; Nishad et al. , 2021 ) that improving the ability of GNN to preserve topological information such as shortest path distance can improve the expressive power of GNN . Here , we explore how to combine the topology awareness of GNN with the network homophily principle to derive useful insight for GNN learning . In particular , we focus on the ability of GNN to preserve distances information among vertexes , i.e. , to have a low distortion between graph distance and embedding distance . The distortion of a function between two metric spaces is defined as follows . Definition 1 ( distortion ) . Given two metric spaces ( E , d ) and ( E ′ , d′ ) and a mapping f : E 7→ E ′ , f is said to have distortion α , if there exists a constant r > 0 such that ∀u , v ∈ E , rd ( u , v ) ≤ d′ ( f ( u ) , f ( v ) ) ≤ αrd ( u , v ) You et al . ( 2019 ) ; Li et al . ( 2020 ) ; Nishad et al . ( 2021 ) have presented effective ways to increase the awareness of GNNs to vertex distances , i.e , to decrease the distortion rate α . For example , one can encode the shortest path distance ( SPD ) or other graph structure information to be part of the node feature for each vertex ( Li et al. , 2020 ) . Based on Bourgains ’ theorem ( Bourgain , 1985 ) , You et al . ( 2019 ) propose a vertex-distance-aware mechanism by selecting a set of anchor vertexes to provide positional information , and use the positional information to adjust the intermediate activation in GNN learning process . It is also discussed in ( You et al. , 2019 ) that the commonly used GNNs are a local version of the vertex-distance-aware mechanism they proposed , as the commonly used GNNs aggregate information for the training node from its neighbourhood . This neighbourhood aggregation mimics the existence of anchor vertexes and can provide local position information . This implies that common GNNs are already equipped with some ability to capture vertex distances ( low distortion ) , at least between neighbours whose representations are aggregated . We conduct experiments in Sec . 4.3 to further validate this .

Summary: This paper considers the problem of training set selection for graph neural network training. It shows that the generalization from the training to the test set in a well-trained graph neural network is closely tied to the shortest path distance in the graph, which motivates training strategies that "cover" the graph in this sense. The authors apply these ideas in a subset selection program, and verify their results with numerical experiments.