Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods

1 Introduction . Graph learning methods generate predictions by leveraging complex inductive biases captured in the topology of the graph [ 7 ] . A large volume of work in this area , including graph neural networks ( GNNs ) , exploits homophily as a strong inductive bias , where connected nodes tend to be similar to each other in terms of labels [ 46 , 3 ] . Such assumptions of homophily , however , do not always hold true . For example , malicious node detection , a key application of graph machine learning , is known to be non-homophilous in many settings [ 55 , 13 , 25 , 11 ] . Further , while new GNNs that work better in these non-homophilous settings have been developed [ 82 , 44 , 81 , 17 , 15 , 73 , 36 , 35 , 9 , 54 ] , their evaluation is limited to a few graph datasets used by Pei et al . [ 58 ] ( collected by [ 61 , 66 , 48 ] ) that have certain undesirable properties such as small size , narrow range of application areas , and high variance between different train/test splits [ 82 ] . Consequently , method scalability has not been thoroughly studied in non-homophilous graph learning . In fact , many non-homophilous techniques frequently require more parameters and computational resources [ 82 , 1 , 17 ] , which is neither evident nor detrimental when they are evaluated on very small datasets . Even though scalable graph learning techniques do exist , these methods generally can not ⇤Equal contribution . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . be directly applied to the non-homophilous setting , as they oftentimes assume homophily in their construction [ 71 , 32 , 20 , 10 ] . Non-homophily in graphs also degrades proven graph learning techniques that have been instrumental to strong performance in scalable graph learning . For instance , label propagation , personalized PageRank , and low-pass graph filtering have been used for scalable graph representation learning models , but these methods all assume homophily [ 71 , 32 , 20 , 10 ] . Moreover , we give empirical evidence that existing minibatching techniques in graph learning [ 16 , 77 ] significantly degrade performance in non-homophilous settings . In response , we develop a novel model , LINKX , that addresses these concerns ; LINKX outperforms existing graph learning methods on large-scale nonhomophilous datasets and admits a simple minibatching procedure that maintains strong performance . To summarize , we demonstrate three key areas of deficiency as mentioned above , namely : ( 1 ) that there is a lack of large , high-quality datasets covering different non-homophilous applications , ( 2 ) that current graph minibatching techniques and scalable methods do not work well in non-homophilous settings , and ( 3 ) that prior non-homophilous methods are not scalable . To these ends , this paper makes the following contributions : Dataset Collection and Benchmarking . We collect a diverse series of large , non-homophilous graph datasets and define new node features and tasks for classification . These datasets are substantially larger than previous non-homophilous datasets , span wider application areas , and capture different types of complex label-topology relationships . With these proposed datasets , we conduct extensive experiments with 14 graph learning methods and 3 graph minibatching techniques that are broadly representative of the graph machine learning model space . Analyzing Scalable Methods and Minibatching . We analyze current graph minibatching techniques like GraphSAINT [ 77 ] in non-homophilous settings , showing that they substantially degrade performance in experiments . Also , we show empirically that scalable methods for graph learning like SGC and C & S [ 71 , 32 ] do not perform well in non-homophilous settings — even though they achieve state-of-the-art results on many homophilous graph benchmarks . Finally , we demonstrate that existing non-homophilous methods often suffer from issues with scalability and performance in large non-homophilous graphs , in large part due to a lack of study of large-scale non-homophilous graph learning . LINKX : a strong , simple method . We propose a simple method LINKX that achieves excellent results for non-homophilous graphs while overcoming the above-mentioned minibatching issues . LINKX works by separately embedding the adjacency A and node features X , then combining them with multilayer perceptrons and simple transformations , as illustrated in Figure 1 . It generalizes node feature MLP and LINK regression [ 79 ] , two baselines that often work well on non-homophilous graphs . This method is simple to train and evaluate in a minibatched fashion , and does not face the performance degradation that other methods do in the minibatch setting . We develop the model and give more details in Section 4 . 2 Prior Work . Graph Representation Learning . Graph neural networks [ 28 , 38 , 69 ] have demonstrated their utility on a variety of graph machine learning tasks . Most GNNs are constructed by stacking layers that propagate transformed node features , which are then aggregated via different mechanisms . The neighborhood aggregation used in many existing GNNs implicitly leverage homophily , so they often fail to generalize on non-homophilous graphs [ 82 , 6 ] . Indeed , a wide range of GNNs operate as low-pass graph filters [ 53 , 71 , 6 ] that smooth features over the graph topology , which produces similar representations and thus similar predictions for neighboring nodes . Scalable methods . A variety of scalable graph learning methods have been developed for efficient computation in larger datasets [ 77 , 16 , 75 , 28 , 71 , 32 , 20 , 10 ] . Many of these methods explicitly make use of an assumption of homophily in the data [ 71 , 32 , 20 , 10 ] . By leveraging this assumption , several simple , inexpensive models are able to achieve state-of-the-art performance on homophilic datasets [ 71 , 32 ] . However , these methods are unable to achieve comparable performance in non-homophilous settings , as we show empirically in Section 5 . Graph sampling . As node representations depend on other nodes in the graph , there are no simple minibatching techniques in graph learning as there are for i.i.d . data . To scale to large graphs , one line of work samples nodes that are used in each layer of a graph neural network [ 28 , 75 , 14 ] . Another family of methods samples subgraphs of an input graph , then passes each subgraph through a GNN to make a prediction for each node of the subgraph [ 16 , 76 , 77 ] . While these methods are useful for scalable graph learning , we show that they substantially degrade performance in our non-homphilous experiments ( see Section 5 ) . Non-Homophilous methods . Various GNNs have been proposed to achieve higher performance in low-homophily settings [ 82 , 44 , 81 , 17 , 15 , 73 , 36 , 35 ] . Geom-GCN [ 58 ] introduces a geometric aggregation scheme , MixHop [ 1 ] proposes a graph convolutional layer that mixes powers of the adjacency matrix , GPR-GNN [ 17 ] features learnable weights that can be positive and negative in feature propagation , GCNII [ 15 ] allows deep graph convolutional networks with relieved oversmoothing , which empirically performs better in non-homophilous settings , and H2GCN [ 82 ] shows that separation of ego and neighbor embeddings , aggregation in higher-order neighborhoods , and the combination of intermediate representations improves GNN performance in low-homophily . There are several recurring design decisions across these methods that appear to strengthen performance in non-homophilous settings : using higher-order neighborhoods , decoupling neighbor information from ego information , and combining graph information at different scales [ 82 ] . Many of these design choices require additional overhead ( see Section 4.3 ) , thus reducing their scalability . Datasets . The widely used citation networks Cora , Citeseer , and Pubmed [ 62 , 74 ] are highly homophilous ( see Appendix A ) [ 82 ] . Recently , the Open Graph Benchmark [ 31 ] has provided a series of datasets and leaderboards that improve the quality of evaluation in graph representation learning ; however , most of the node classification datasets tend to be homophilous , as noted in past work [ 82 ] and expanded upon in Appendix A.2 . A comparable set of high-quality benchmarks to evaluate non-homophilous methods does not currently exist . 3 Datasets for Non-Homophilous Graph Learning . 3.1 Currently Used Datasets . The most widely used datasets to evaluate non-homophilous graph representation learning methods were used by Pei et al . [ 58 ] ( and collected by [ 61 , 66 , 48 ] ) ; see our Table 1 for statistics . However , these datasets have fundamental issues . First , they are very small — the Cornell , Texas , and Wisconsin datasets have between 180-250 nodes , and the largest dataset Actor has 7,600 nodes . In analogy to certain pitfalls of graph neural network evaluation on small ( homophilic ) datasets discussed in [ 63 ] , evaluation on the datasets of Pei et al . [ 58 ] is plagued by high variance across different train/test splits ( see results in [ 82 ] ) . The small size of these datasets may tend to create models that are more prone to overfitting [ 21 ] , which prevents the scaling up of GNNs designed for non-homophilous settings . Peel [ 57 ] also studies node classification on network datasets with various types of relationships between edges and labels . However , they only study methods that act on graph topology , and thus their datasets do not necessarily have node features . We take inspiration from their work , by testing on Pokec and Facebook networks with node features that we define , and by introducing other year-prediction tasks on citation networks that have node features . 3.2 An Improved Homophily Measure . Various metrics have been proposed to measure the homophily of a graph . However , these metrics are sensitive to the number of classes and the number of nodes in each class . Let G = ( V , E ) be a graph with n nodes , none of which are isolated . Further let each node u 2 V have a class label ku 2 { 0 , 1 , . . . , C 1 } for some number of classes C , and denote by Ck the set of nodes in class k. The edge homophily [ 82 ] is the proportion of edges that connect two nodes of the same class : h = | { ( u , v ) 2 E : ku = kv } | |E| . ( 1 ) Another related measure is what we call the node homophily [ 58 ] , defined as 1|V | P u2V d ( ku ) u du , in which du is the number of neighbors of node u , and d ( ku ) u is the number of neighbors of u that have the same class label . We focus on the edge homophily ( 1 ) in this work , but find that node homophily tends to have similar qualitative behavior in experiments . The sensitivity of edge homophily to the number of classes and size of each class limits its utility . We consider a null model for graphs in which the graph topology is independent of the labels ; suppose that nodes with corresponding labels are fixed , and include edges uniformly at random in the graph that are independent of node labels . Under this null model , a node u 2 V would be expected to have d ( ku ) u /du ⇡ |Cku |/n as the proportion of nodes of the same class that they connect to [ 3 ] . For a dataset with C balanced classes , we would thus expect the edge homophily to be around 1C , so the interpretation of the measure depends on the number of classes . Also , if classes are imbalanced , then the edge homophily may be misleadingly large . For instance , if 99 % of nodes were of one class , then most edges would likely be within that same class , so the edge homophily would be high , even when the graph is generated from the null model where labels are independent of graph topology . Thus , the edge homophily does not capture deviation of the label distribution from the null model . We introduce a metric that better captures the presence or absence of homophily . Unlike the edge homophily , our metric measures excess homophily that is not expected from the above null model where edges are randomly wired . Our metric does not distinguish between different non-homophilous settings ( such as heterophily or independent edges ) ; we believe that there are too many degrees of freedom in non-homophilous settings for a single scalar quantity to be able to distinguish them all . Our measure is given as : ĥ = 1 C 1 C 1X k=0 hk |Ck| n + , ( 2 ) where [ a ] + = max ( a , 0 ) , and hk is the class-wise homophily metric hk = P u2Ck d ( ku ) uP u2Ck du . ( 3 ) Note that ĥ 2 [ 0 , 1 ] , with a fully homophilous graph ( in which every node is only connected to nodes of the same class ) having ĥ = 1 . Since each class-wise homophily metric hk only contributes positive deviations from the null expected proportion |Ck|/n , the class-imbalance problem is substantially mitigated . Also , graphs in which edges are independent of node labels are expected to have ĥ ⇡ 0 , for any number of classes . Our measure ĥ measures presence of homophily , but does not distinguish between the many types of possibly non-homophilous relationships . This is reasonable given the diversity of non-homophilous relationships . For example , non-homophily can imply independence of edges and classes , extreme heterophily , connections only among subsets of classes , or certain chemically / biologically determined relationships . Indeed , these relationships are very different , and are better captured by more than one scalar quantity , such as the compatibility matrices presented in the appendix . Further discussion is given in Appendix A .

Summary: This paper proposes 7 new large heterophily graph datasets (up to 2 million nodes) and also provides a simple, scalable yet effective baseline LINKX which combines the information from topology and node features. The paper extensively evaluates the datasets on various baselines, including typical GNNs and GNNs specifically designed for heterophily graphs. They find that many baselines cannot compete with the simple baseline LINKX and some of them run out of memory. It also points out the minibatching problem and provides thorough results for fullbatch and minibatch training.