Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods

1 Introduction . Graph learning methods generate predictions by leveraging complex inductive biases captured in the topology of the graph [ 7 ] . A large volume of work in this area , including graph neural networks ( GNNs ) , exploits homophily as a strong inductive bias , where connected nodes tend to be similar to each other in terms of labels [ 46 , 3 ] . Such assumptions of homophily , however , do not always hold true . For example , malicious node detection , a key application of graph machine learning , is known to be non-homophilous in many settings [ 55 , 13 , 25 , 11 ] . Further , while new GNNs that work better in these non-homophilous settings have been developed [ 82 , 44 , 81 , 17 , 15 , 73 , 36 , 35 , 9 , 54 ] , their evaluation is limited to a few graph datasets used by Pei et al . [ 58 ] ( collected by [ 61 , 66 , 48 ] ) that have certain undesirable properties such as small size , narrow range of application areas , and high variance between different train/test splits [ 82 ] . Consequently , method scalability has not been thoroughly studied in non-homophilous graph learning . In fact , many non-homophilous techniques frequently require more parameters and computational resources [ 82 , 1 , 17 ] , which is neither evident nor detrimental when they are evaluated on very small datasets . Even though scalable graph learning techniques do exist , these methods generally can not ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . be directly applied to the non-homophilous setting , as they oftentimes assume homophily in their construction [ 71 , 32 , 20 , 10 ] . Non-homophily in graphs also degrades proven graph learning techniques that have been instrumental to strong performance in scalable graph learning . For instance , label propagation , personalized PageRank , and low-pass graph filtering have been used for scalable graph representation learning models , but these methods all assume homophily [ 71 , 32 , 20 , 10 ] . Moreover , we give empirical evidence that existing minibatching techniques in graph learning [ 16 , 77 ] significantly degrade performance in non-homophilous settings . In response , we develop a novel model , LINKX , that addresses these concerns ; LINKX outperforms existing graph learning methods on large-scale nonhomophilous datasets and admits a simple minibatching procedure that maintains strong performance . To summarize , we demonstrate three key areas of deficiency as mentioned above , namely : ( 1 ) that there is a lack of large , high-quality datasets covering different non-homophilous applications , ( 2 ) that current graph minibatching techniques and scalable methods do not work well in non-homophilous settings , and ( 3 ) that prior non-homophilous methods are not scalable . To these ends , this paper makes the following contributions : Dataset Collection and Benchmarking . We collect a diverse series of large , non-homophilous graph datasets and define new node features and tasks for classification . These datasets are substantially larger than previous non-homophilous datasets , span wider application areas , and capture different types of complex label-topology relationships . With these proposed datasets , we conduct extensive experiments with 14 graph learning methods and 3 graph minibatching techniques that are broadly representative of the graph machine learning model space . Analyzing Scalable Methods and Minibatching . We analyze current graph minibatching techniques like GraphSAINT [ 77 ] in non-homophilous settings , showing that they substantially degrade performance in experiments . Also , we show empirically that scalable methods for graph learning like SGC and C & S [ 71 , 32 ] do not perform well in non-homophilous settings — even though they achieve state-of-the-art results on many homophilous graph benchmarks . Finally , we demonstrate that existing non-homophilous methods often suffer from issues with scalability and performance in large non-homophilous graphs , in large part due to a lack of study of large-scale non-homophilous graph learning . LINKX : a strong , simple method . We propose a simple method LINKX that achieves excellent results for non-homophilous graphs while overcoming the above-mentioned minibatching issues . LINKX works by separately embedding the adjacency A and node features X , then combining them with multilayer perceptrons and simple transformations , as illustrated in Figure 1 . It generalizes node feature MLP and LINK regression [ 79 ] , two baselines that often work well on non-homophilous graphs . This method is simple to train and evaluate in a minibatched fashion , and does not face the performance degradation that other methods do in the minibatch setting . We develop the model and give more details in Section 4 . 2 Prior Work . Graph Representation Learning . Graph neural networks [ 28 , 38 , 69 ] have demonstrated their utility on a variety of graph machine learning tasks . Most GNNs are constructed by stacking layers that propagate transformed node features , which are then aggregated via different mechanisms . The neighborhood aggregation used in many existing GNNs implicitly leverage homophily , so they often fail to generalize on non-homophilous graphs [ 82 , 6 ] . Indeed , a wide range of GNNs operate as low-pass graph filters [ 53 , 71 , 6 ] that smooth features over the graph topology , which produces similar representations and thus similar predictions for neighboring nodes . Scalable methods . A variety of scalable graph learning methods have been developed for efficient computation in larger datasets [ 77 , 16 , 75 , 28 , 71 , 32 , 20 , 10 ] . Many of these methods explicitly make use of an assumption of homophily in the data [ 71 , 32 , 20 , 10 ] . By leveraging this assumption , several simple , inexpensive models are able to achieve state-of-the-art performance on homophilic datasets [ 71 , 32 ] . However , these methods are unable to achieve comparable performance in non-homophilous settings , as we show empirically in Section 5 . Graph sampling . As node representations depend on other nodes in the graph , there are no simple minibatching techniques in graph learning as there are for i.i.d . data . To scale to large graphs , one line of work samples nodes that are used in each layer of a graph neural network [ 28 , 75 , 14 ] . Another family of methods samples subgraphs of an input graph , then passes each subgraph through a GNN to make a prediction for each node of the subgraph [ 16 , 76 , 77 ] . While these methods are useful for scalable graph learning , we show that they substantially degrade performance in our non-homphilous experiments ( see Section 5 ) . Non-Homophilous methods . Various GNNs have been proposed to achieve higher performance in low-homophily settings [ 82 , 44 , 81 , 17 , 15 , 73 , 36 , 35 ] . Geom-GCN [ 58 ] introduces a geometric aggregation scheme , MixHop [ 1 ] proposes a graph convolutional layer that mixes powers of the adjacency matrix , GPR-GNN [ 17 ] features learnable weights that can be positive and negative in feature propagation , GCNII [ 15 ] allows deep graph convolutional networks with relieved oversmoothing , which empirically performs better in non-homophilous settings , and H2GCN [ 82 ] shows that separation of ego and neighbor embeddings , aggregation in higher-order neighborhoods , and the combination of intermediate representations improves GNN performance in low-homophily . There are several recurring design decisions across these methods that appear to strengthen performance in non-homophilous settings : using higher-order neighborhoods , decoupling neighbor information from ego information , and combining graph information at different scales [ 82 ] . Many of these design choices require additional overhead ( see Section 4.3 ) , thus reducing their scalability . Datasets . The widely used citation networks Cora , Citeseer , and Pubmed [ 62 , 74 ] are highly homophilous ( see Appendix A ) [ 82 ] . Recently , the Open Graph Benchmark [ 31 ] has provided a series of datasets and leaderboards that improve the quality of evaluation in graph representation learning ; however , most of the node classification datasets tend to be homophilous , as noted in past work [ 82 ] and expanded upon in Appendix A.2 . A comparable set of high-quality benchmarks to evaluate non-homophilous methods does not currently exist . 3 Datasets for Non-Homophilous Graph Learning . 3.1 Currently Used Datasets . The most widely used datasets to evaluate non-homophilous graph representation learning methods were used by Pei et al . [ 58 ] ( and collected by [ 61 , 66 , 48 ] ) ; see our Table 1 for statistics . However , these datasets have fundamental issues . First , they are very small — the Cornell , Texas , and Wisconsin datasets have between 180-250 nodes , and the largest dataset Actor has 7,600 nodes . In analogy to certain pitfalls of graph neural network evaluation on small ( homophilic ) datasets discussed in [ 63 ] , evaluation on the datasets of Pei et al . [ 58 ] is plagued by high variance across different train/test splits ( see results in [ 82 ] ) . The small size of these datasets may tend to create models that are more prone to overfitting [ 21 ] , which prevents the scaling up of GNNs designed for non-homophilous settings . Peel [ 57 ] also studies node classification on network datasets with various types of relationships between edges and labels . However , they only study methods that act on graph topology , and thus their datasets do not necessarily have node features . We take inspiration from their work , by testing on Pokec and Facebook networks with node features that we define , and by introducing other year-prediction tasks on citation networks that have node features . 3.2 An Improved Homophily Measure . Various metrics have been proposed to measure the homophily of a graph . However , these metrics are sensitive to the number of classes and the number of nodes in each class . Let G = ( V , E ) be a graph with n nodes , none of which are isolated . Further let each node u 2 V have a class label ku 2 { 0 , 1 , . . . , C 1 } for some number of classes C , and denote by Ck the set of nodes in class k. The edge homophily [ 82 ] is the proportion of edges that connect two nodes of the same class : h = | { ( u , v ) 2 E : ku = kv } | |E| . ( 1 ) Another related measure is what we call the node homophily [ 58 ] , defined as 1|V | P u2V d ( ku ) u du , in which du is the number of neighbors of node u , and d ( ku ) u is the number of neighbors of u that have the same class label . We focus on the edge homophily ( 1 ) in this work , but find that node homophily tends to have similar qualitative behavior in experiments . The sensitivity of edge homophily to the number of classes and size of each class limits its utility . We consider a null model for graphs in which the graph topology is independent of the labels ; suppose that nodes with corresponding labels are fixed , and include edges uniformly at random in the graph that are independent of node labels . Under this null model , a node u 2 V would be expected to have d ( ku ) u /du ⇡ |Cku |/n as the proportion of nodes of the same class that they connect to [ 3 ] . For a dataset with C balanced classes , we would thus expect the edge homophily to be around 1C , so the interpretation of the measure depends on the number of classes . Also , if classes are imbalanced , then the edge homophily may be misleadingly large . For instance , if 99 % of nodes were of one class , then most edges would likely be within that same class , so the edge homophily would be high , even when the graph is generated from the null model where labels are independent of graph topology . Thus , the edge homophily does not capture deviation of the label distribution from the null model . We introduce a metric that better captures the presence or absence of homophily . Unlike the edge homophily , our metric measures excess homophily that is not expected from the above null model where edges are randomly wired . Our metric does not distinguish between different non-homophilous settings ( such as heterophily or independent edges ) ; we believe that there are too many degrees of freedom in non-homophilous settings for a single scalar quantity to be able to distinguish them all . Our measure is given as : ĥ = 1 C 1 C 1X k=0 hk |Ck| n + , ( 2 ) where [ a ] + = max ( a , 0 ) , and hk is the class-wise homophily metric hk = P u2Ck d ( ku ) uP u2Ck du . ( 3 ) Note that ĥ 2 [ 0 , 1 ] , with a fully homophilous graph ( in which every node is only connected to nodes of the same class ) having ĥ = 1 . Since each class-wise homophily metric hk only contributes positive deviations from the null expected proportion |Ck|/n , the class-imbalance problem is substantially mitigated . Also , graphs in which edges are independent of node labels are expected to have ĥ ⇡ 0 , for any number of classes . Our measure ĥ measures presence of homophily , but does not distinguish between the many types of possibly non-homophilous relationships . This is reasonable given the diversity of non-homophilous relationships . For example , non-homophily can imply independence of edges and classes , extreme heterophily , connections only among subsets of classes , or certain chemically / biologically determined relationships . Indeed , these relationships are very different , and are better captured by more than one scalar quantity , such as the compatibility matrices presented in the appendix . Further discussion is given in Appendix A .

Summary: The paper collects graph datasets that are not as homophile as the regular benchmarks for GNNs. Furthermore, it shows that a rather simple new baseline achieves quite good results compared to much more involved models / the SOTA. The paper presents important contributions to the community, is well and thoroughly written, related work is covered adequately, the experiments are overall thorough, and I have few to criticise. Altogether, I suggest to accept the paper.