SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We define a characteristic measuring the informativeness of a neighbor's label for a node's label. For example, in Figure 1a, the neighbor's label uniquely defines the node's label. Thus, the node classification task is simple on this dataset, and we want our informativeness to be maximal for such graphs. Let us formalize this idea. Assume that we sample an edge  $(\xi, \eta) \in E$  (from some distribution). The class labels of nodes  $\xi$  and  $\eta$  are then random variables  $y_{\xi}$  and  $y_{\eta}$ . We want to measure the amount of knowledge the label  $y_{\eta}$  gives for predicting  $y_{\xi}$ . The entropy  $H(y_{\xi})$  measures the 'hardness' of predicting the label of  $\xi$  without knowing  $y_{\eta}$ . Given  $y_{\eta}$ , this value is reduced to the conditional entropy  $H(y_{\xi} | y_{\eta})$ . In other words,  $y_{\eta}$  reveals  $I(y_{\xi}, y_{\eta}) = H(y_{\xi}) - H(y_{\xi} | y_{\eta})$  information about the label. To make the obtained quantity comparable across different datasets, we say that label informativeness is the normalized mutual information of  $y_{\xi}$  and  $y_{\eta}$ :
$$
\mathrm {L I} := I \left(y _ {\xi}, y _ {\eta}\right) / H \left(y _ {\xi}\right). \tag {3}
$$
We have  $\mathrm{LI} \in [0,1]$ . If the label  $y_{\eta}$  allows for unique reconstruction of  $y_{\xi}$ , then  $\mathrm{LI} = 1$ . If  $y_{\xi}$  and  $y_{\eta}$  are independent,  $\mathrm{LI} = 0$ .
Depending on the distribution used for sampling an edge  $(\xi, \eta)$ , one can obtain several variants of LI. For instance, if the edges are sampled uniformly at random (which is a natural approach), the mutual distribution  $(y_{\xi}, y_{\eta})$  for a randomly sampled edge is  $p(c_1, c_2) = \sum_{(u, v) \in E} \frac{\mathbb{1}\{y_u = c_1, y_v = c_2\}}{2|E|}$ . Then, the marginal distribution of  $y_{\xi}$  (and  $y_{\eta}$ ) is the degree-weighted distribution  $\bar{p}(c)$ . Thus, (3) becomes:
$$
\mathrm {L I} _ {e d g e} = - \frac {\sum_ {c _ {1} , c _ {2}} p (c _ {1} , c _ {2}) \log \frac {p (c _ {1} , c _ {2})}{\bar {p} (c _ {1}) \bar {p} (c _ {2})}}{\sum_ {c} \bar {p} (c) \log \bar {p} (c)} = 2 - \frac {\sum_ {c _ {1} , c _ {2}} p (c _ {1} , c _ {2}) \log p (c _ {1} , c _ {2})}{\sum_ {c} \bar {p} (c) \log \bar {p} (c)}.
$$
For brevity, we further denote  $\mathrm{LI}_{edge}$  by LI and focus on this version of the measure. However, note that another natural approach to edge sampling is to first sample a random node and then sample a random edge incident to this node. For a discussion of this approach, we refer to Appendix C.2.
To claim that LI is a suitable graph characteristic, we need to show that it is comparable across different datasets. For this, we need to verify two properties: maximal agreement and asymptotic constant baseline. Recall that LI is upper bounded by one and equals one if and only if the neighbor's class uniquely reveals the node's class. This property can be considered as a direct analog of the maximal agreement defined in Section 2.2. The following theorem shows that LI satisfies the asymptotic constant baseline; see Appendix C.1 for the proof.
Theorem 2. Assume that  $|E| \to \infty$  as  $n \to \infty$  and that the entropy of  $\bar{p}(\cdot)$  is bounded from below by some constant. Let  $\bar{p}_{min} = \min_k \bar{p}(k)$  and assume that  $\bar{p}_{min} \gg C / \sqrt{|E|}$  as  $n \to \infty$ . Then, for the random configuration model, we have  $\mathrm{LI} = o(1)$  with high probability.
In summary, LI is a simple graph characteristic suitable for comparing different datasets. We note that LI is not a measure of homophily. LI naturally complements homophily measures by distinguishing different types of heterophilous graphs.
4 Empirical illustrations
In this section, we first characterize some existing graph datasets in terms of homophily and LI to see which structural patterns are currently covered. Then, we show that LI, despite being a very simple graph characteristic, much better agrees with GNN performance than homophily.
4.1 Characterizing real graph datasets
We first look at the values of homophily and label informativeness for existing graph datasets. For this analysis, we choose several node classification datasets of different sizes and properties. Statistics of these datasets and values of all the measures discussed in this paper are provided in Table 5 in Appendix E, while Table 2 shows selected results.
Recall that both node and edge homophily are sensitive to the number of classes and class size balance. Indeed, they may indicate high homophily levels for some heterophilous datasets. An
5 Implementations of all the graph measures discussed in the paper and examples of their usage are provided in this Colab notebook.
extreme example is questions:  $h_{edge} = 0.84$ , while more reliable adjusted homophily shows that the dataset is heterophilous:  $h_{adj} = 0.02$ . In fact, all binary classification datasets in Table 5 could be considered homophilous if one chooses  $h_{edge}$  or  $h_{node}$  as the measure of homophily. In contrast,  $h_{adj}$  satisfies the constant baseline property and shows that most of the considered binary classification datasets are heterophilous.
It is expected that datasets with high homophily  $h_{adj}$  also have high LI since homophily implies informative neighbor classes. For medium-level homophily, LI can behave differently: for instance, ogbn-arxiv and twitter-hate have similar homophily levels, while the neighbors in ogbn-arxiv are significantly more informative. For heterophilous datasets, LI can potentially be very different, as we demonstrate in Section 4.2 on synthetic and semi-synthetic data. However, most existing heterophilous datasets have  $\mathrm{LI} \approx 0$ . This issue is partially addressed by new heterophilous datasets recently proposed in [33]. For the proposed roman-empire dataset,  $\mathrm{LI} = 0.11$  and  $h_{adj} = -0.05$ . Thus, while being het
Table 2: Characteristics of some real graph datasets, see Table 5 for the full results
Dataset	C	hedge	hadj	LI
lastfm-asia	18	0.87	0.86	0.74
cora	7	0.81	0.77	0.59
ogbn-arxiv	40	0.65	0.59	0.45
twitter-hate	2	0.78	0.55	0.23
wiki	5	0.38	0.15	0.06
Twitch-gamers	2	0.55	0.09	0.01
actor	5	0.22	0.00	0.00
questions	2	0.84	0.02	0.00
roman-armpire	18	0.05	-0.05	0.11
erophilous, this dataset has non-zero label informativeness, meaning that neighboring classes are somewhat informative. We believe that datasets with more interesting connectivity patterns will be collected in the future.
4.2 Correlation of LI with GNN performance
Recently, it has been shown that standard GNNs can sometimes perform well on non-homophilous datasets [22, 24]. We hypothesize that GNNs can learn more complex relationships between nodes than just homophily, and they will perform well as long as the node's neighbors provide some information about this node. Thus, we expect LI to better correlate with the performance of GNNs than homophily. To illustrate this, we use carefully designed synthetic and semi-synthetic data. First, it allows us to cover all combinations of homophily levels and label informativeness. Second, we can control that only a connection pattern changes while other factors affecting the performance are fixed.
Synthetic data based on SBM model To start with the most simple and controllable setting, we generate synthetic graphs via a variant of the stochastic block model (SBM) [13]. In this model, the nodes are divided into  $C$  clusters, and for each pair of nodes  $i,j$ , we draw an edge between them with probability  $p_{c(i),c(j)}$  independently of all other edges. Here  $c(i)$  is a cluster assignment for a node  $i$ , which in our case corresponds to the node label  $y_i$ .
We set the number of classes to  $C = 4$  and the class size to  $l = n / 4$ . We define the probabilities as follows:  $p_{i,j} = p_0K$  if  $i = j$ ,  $p_{i,j} = p_1K$  if  $i + j = 5$ , and  $p_{i,j} = p_2K$  otherwise. Here  $K > 0$  and we require  $p_0 + p_1 + 2p_2 = 1$ . Note that the expected degree of any node is (up to a negligibly small term)  $p_0Kl + p_1Kl + 2p_2Kl = Kl$ .
This model allows us to explore various combinations of dataset characteristics. Indeed,  $p_0$  directly controls the homophily level, while the relation between  $p_1$  and  $p_2$  enables us to vary LI. To see this, we note that the condition  $i + j = 5$  gives two pairs of classes: (1,4) and (2,3). Thus, if  $p_2 = 0$  and  $p_1 > 0$ , knowing the label of any neighbor from another class, we can uniquely restore the node's label. In contrast, for given  $p_0$ , the case  $p_1 = p_2$  gives the smallest amount of additional information. The following proposition characterizes the covered combinations of LI and homophily; the proof follows from the construction procedure.
Proposition 1. As  $n \to \infty$ , the dataset characteristics of the proposed model converge to the following values (with high probability):  $h_{adj} = \frac{4}{3} p_0 - \frac{1}{3}$ ,  $\mathrm{LI} = 1 - \frac{H(p_0, p_1, p_2, p_2)}{\log 4}$ , where  $H(x) = -\sum_{i} x_i \log (x_i)$ .
Thus,  $h_{adj}$  ranges from  $-1/3$  to 1 and LI can be between 0 and 1. If  $\mathrm{LI} = 0$ , then we always have  $h_{adj} = 0$ ; if  $h_{adj} = 1$ , then  $\mathrm{LI} = 1$ . However, if  $\mathrm{LI} = 1$ , then either  $h_{adj} = -1/3$  or  $h_{adj} = 1$ .
(a) Synthetic SBM graphs
(b) Semi-synthetic cora graphs from [24]
Figure 2: Accuracy of GraphSAGE on synthetic and semi-synthetic graphs
We generated graphs according to the procedure described above with the expected node degree of 10 and various combinations of  $p_0, p_1, p_2$ . Given the class labels, the features are taken from the four largest classes in the cora dataset [38, 27, 46, 25]. We use the obtained graphs to train four popular GNN models: GCN [16], GraphSAGE [10], GAT [43], and Graph Transformer (GT) [40]. In total, we run more than 20000 experiments on synthetic datasets for each of the four models. A detailed description of data generation, training setup, and hyperparameters used is provided in Appendix F.1.
Figure 2a shows the results for GraphSAGE. It can be seen that the performance is much more correlated with LI than with homophily. In particular, when LI is high, GraphSAGE achieves good performance even on strongly heterophilous graphs with negative homophily. We refer to Appendix F.2 for additional visualizations.
The Spearman correlation coefficient between accuracy and LI is equal to 0.93, while between accuracy and adjusted homophily it equals 0.05. For other considered models, the difference in correlation is also significant; see Table 3. Note that due to the balanced classes and degrees, all homophily measures give the same correlation, so we report only  $h_{adj}$ .
Table 3: Spearman correlation between model accuracy and characteristics of synthetic SBM datasets
Model	hadj	LIedge
GCN	0.19	0.76
GraphSAGE	0.05	0.93
GAT	0.17	0.77
GT	0.17	0.77
Semi-synthetic data from [24] Ma et al. [24] also argue that standard GNNs can perform well on certain heterophilous graphs. They construct semi-synthetic graphs by adding inter-class edges following different patterns to several real-world graphs, thus obtaining several sets of graphs with varying levels of homophily. Ma et al. [24] run experiments on these graphs and note that a standard GNN achieves strong performance on some heterophilous graphs.
We train GCN, GraphSAGE, GAT, and GT on the same modifications of the cora and citeseer graphs used in [24] and find that the models achieve strong performance when the graphs have high label informativeness. Training setup and hyperparameters used in these experiments are the same as above and are described in Appendix F.1. The performance of GraphSAGE on cora is shown in Figure 2b. In Appendix F.3, we also show the performance of GraphSAGE on the citeseer dataset. Table 4 shows the Spearman correlation coefficients between accuracy and various homophily measures or LI for all
Table 4: Spearman correlation between model accuracy and characteristics of semi-synthetic datasets from [24]
Model	hedge	hnode	hclass	hadj	LIedge
cora
GCN	-0.31	-0.31	-0.31	-0.31	0.72
GraphSAGE	-0.24	-0.24	-0.24	-0.24	0.78
GAT	-0.24	-0.25	-0.24	-0.24	0.77
GT	-0.23	-0.24	-0.23	-0.23	0.79
citeseer
GCN	-0.24	-0.24	-0.24	-0.24	0.76
GraphSAGE	-0.53	-0.53	-0.54	-0.54	0.51
GAT	-0.27	-0.27	-0.27	-0.27	0.75
GT	-0.19	-0.19	-0.19	-0.19	0.80
models and both datasets. The correlation coefficients for homophily measures are all negative, while for LI they are positive and sufficiently large. This again confirms that LI indicates whether graph structure is helpful for GNNs.
Additionally, we conduct experiments with other datasets: Appendix F.4 describes the results for the LFR benchmark [17] and Appendix F.5 presents an experiment on synthetic data from [22].
In this paper, we discuss how to characterize graph node classification datasets. First, we revisit the concept of homophily and show that commonly used homophily measures have significant drawbacks preventing the comparison of homophily levels between different datasets. For this, we formalize properties desirable for a good homophily measure and prove which measures satisfy which properties. Based on our analysis, we conclude that adjusted homophily is a better measure of homophily than the ones commonly used in the literature. We believe that being able to properly estimate the homophily level of a graph is essential for the future development of heterophily-suited GNNs: we need a characteristic that reliably differentiates homophilous and heterophilous graphs.
Then, we argue that heterophilous graphs may have very different structural patterns and propose a new property called label informativeness (LI) that allows one to distinguish them. LI characterizes how much information a neighbor's label provides about a node's label. Similarly to adjusted homophily, this measure satisfies important properties and thus can be used to compare datasets with different numbers of classes and class size balance. Through a series of experiments, we show that LI correlates well with the performance of GNNs.
To conclude, we believe that adjusted homophily and label informativeness will be helpful for researchers and practitioners as they allow one to easily characterize the connectivity patterns of graph datasets. We also hope that new realistic datasets will be collected to cover currently unexplored combinations of  $h_{adj}$  and LI. Finally, our theoretical framework can be helpful for the further development of reliable graph characteristics.
**Limitations** We advise using adjusted homophily as a reliable homophily measure since it is the only existing measure having constant baseline and thus it dominates other known alternatives in terms of desirable properties. However, it still violates minimal agreement, and monotonicity is guaranteed only for sufficiently large values of  $h_{adj}$ . It is currently an open question whether there exist measures dominating adjusted homophily in terms of the satisfied properties.
Regarding LI, we do not claim that this measure is a universal predictor of GNN performance. We designed this measure to be both informative and simple to compute and interpret. For instance, LI considers all edges individually and does not account for the node's neighborhood as a whole. As a result, LI can be insensitive to some complex dependencies. Such dependencies can be important for some tasks, but taking them into account is tricky and would significantly complicate the measure. However, we clearly see that despite its simplicity, LI correlates with GNN performance much better than homophily.
Let us also note that our analysis of both homophily and LI is limited to graph-label interactions. In future work, it would be important to also analyze node features. Indeed, node features may have non-trivial relations with both graph and labels. For example, a graph can be heterophilous in terms of labels but homophilous in terms of node features or vice versa. These interactions may allow one to understand the properties and performance of GNNs even better. However, analyzing feature-based homophily or informativeness can be much more difficult since the features can differ in nature, scale, and type.
We thank Yao Ma for sharing the semi-synthetic datasets from Ma et al. [24] that we used in Section 4.2. We also thank Andrey Ploskonosov for thoughtful discussions.
[38] P. Sen, G. Namata, M. Bilgic, L. Getoor, B. Galligher, and T. Eliassi-Rad. Collective classification in network data. AI Magazine, 29(3):93-93, 2008.
[39] O. Shchur, M. Mumme, A. Bojchevski, and S. Gunnemann. Pitfalls of graph neural network evaluation. Relational Representation Learning Workshop, 2018.
[40] Y. Shi, Z. Huang, S. Feng, H. Zhong, W. Wang, and Y. Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509, 2020.
[41] S. Suresh, V. Budde, J. Neville, P. Li, and J. Ma. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
[42] J. Tang, J. Sun, C. Wang, and Z. Yang. Social influence analysis in large-scale networks. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 807-816, 2009.
[43] P. Velicković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph Attention Networks. International Conference on Learning Representations, 2018.
[44] M. Wang, D. Zheng, Z. Ye, Q. Gan, M. Li, X. Song, J. Zhou, C. Ma, L. Yu, Y. Gai, et al. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv:1909.01315, 2019.
[45] K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[46] Z. Yang, W. Cohen, and R. Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning, pages 40-48. PMLR, 2016.
[47] H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna. GraphSAINT: Graph sampling based inductive learning method. In International Conference on Learning Representations, 2020.
[48] J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33:7793-7804, 2020.
[49] J. Zhu, R. A. Rossi, A. Rao, T. Mai, N. Lipka, N. K. Ahmed, and D. Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11168-11176, 2021.
A Random configuration model
Numerous random graph models have been proposed to reflect and predict important quantitative and topological aspects of real-world networks [3]. The simplest model is the Erdős-Rényi random graph, i.e., we assume that  $G$  is sampled uniformly at random from the set of all graphs with  $n$  nodes and  $|E|$  edges. However, this model is known to be not a good descriptor of real-world networks since its Poisson degree distribution significantly differs from heavy-tailed degree distributions observed in real-world networks. A standard solution is to consider a random graph with a given degree sequence [26]: a graph is sampled uniformly from the set of all graphs with a given degree sequence.
A configuration model, which we assume throughout the paper, is defined as follows. To generate a graph, we form a set  $A$  containing  $d(v)$  distinct copies of each node  $v$  and then choose a random matching of the elements of  $A$ . In this case, self-loops and multiple edges may appear. Under some conditions, the obtained graph is simple (i.e., does not contain self-loops and multiple edges) with probability  $1 - o(1)$  [26].
Let us also note that there is another model that is similar to the one discussed above but can be easier to analyze. To obtain a graph with given expected degree sequence [5], we take the degree sequence from the observed graph and say that the number of edges between  $i$  and  $j$  follows a Poisson distribution with the mean  $\frac{d(i)d(j)}{2|E|}$  if  $i \neq j$  and the expected number of self-loops for a node  $i$  is  $\frac{d(i)^2}{4|E|}$ . This model does not preserve the exact degree sequence but has it in expectation. Note that usually  $d(i)d(j) \ll 2|E|$ , so multiple edges rarely appear. Asymptotically, this model is similar to the configuration model, but a graph with a given expected degree sequence is easier to analyze theoretically. In our analysis, we assume the configuration model, so we have to track the error terms carefully.
B Analysis of homophily
B.1 Class homophily
Recall that class homophily is defined as [20]:
$$
h _ {c l a s s} = \frac {1}{C - 1} \sum_ {k = 1} ^ {C} \left[ \frac {\sum_ {v : y _ {v} = k} | \{u \in N (v) : y _ {u} = y _ {v} \} |}{\sum_ {v : y _ {v} = k} d (v)} - \frac {n _ {k}}{n} \right] _ {+}.
$$
Here the first term inside the brackets is the fraction of edges that go from a particular class  $k$  to itself. The second term  $n_k / n = p(k)$  is the null expected fraction. We note that  $p(k)$  is the expected fraction if we assume the Erdős-Rényi model. Indeed, for this model, all edges have equal probability, and thus the expected fraction of neighbors of class  $k$  is proportional to the size of this class. However, as mentioned above, the Erdős-Rényi null model has certain disadvantages since it does not take into account node degrees. This may lead to incorrect estimates of null probabilities for degree-imbalanced classes, as shown in the example below.<sup>6</sup>
Proposition 2. Assume that we have two classes of size  $n / 2$ . Further, assume that the expected degrees of nodes in the first class are equal to  $d$ , while nodes in the second class have expected degrees  $ld$  for some  $l > 1$ . Then, if edges are added independently of the classes, the expected value of  $h_{\text{class}}$  is
$$
\mathbb {E} h _ {\text {c l a s s}} = \frac {l}{l + 1} - \frac {1}{2}.
$$
Thus, for randomly connected nodes we get  $\mathbb{E}h_{class}\to 1 / 2$  as  $l\rightarrow \infty$
Proof. We need to compute the expected number of intra-class edges for each class. For the first class, we multiply the number of nodes by the degree of each node and by the probability of a particular
For simplicity, in these statements, we use the random graph model with a given expected degree sequence as it allows for simpler illustrations. For the configuration model, the error terms can be tracked similarly to the proofs in Sections B.4 and C.1.
edge to go to the same class:  $\frac{n}{2} \cdot d \cdot \frac{nd}{n(l + 1)d} = \frac{nd}{2(l + 1)}$ . Normalizing by the sum of the degrees, we get  $\frac{1}{(l + 1)}$ . Similarly, for the second class, the normalized number of the intra-class edges is  $\frac{l}{(l + 1)}$ . Hence,
$$
\mathbb {E} h _ {\text {c l a s s}} = \left[ \frac {1}{l + 1} - \frac {1}{2} \right] _ {+} + \left[ \frac {l}{l + 1} - \frac {1}{2} \right] _ {+} = \frac {l}{l + 1} - \frac {1}{2}.
$$
This proposition shows that class homophily does not satisfy the constant baseline property.
As discussed in Section 2.3, if we remove the  $[ \cdot ]_{+}$  operation from the definition of class homophily, we obtain a measure that does satisfy the asymptotic constant baseline. Following Section 2.5, we call this measure balanced adjusted homophily as it adjusts balance homophily for chance:
$$
h _ {b a l} ^ {a d j} = \frac {1}{C - 1} \sum_ {k = 1} ^ {C} \left(\frac {\sum_ {v : y _ {v} = k} | \{u \in N (v) : y _ {u} = y _ {v} \} |}{\sum_ {v : y _ {v} = k} d (v)} - \frac {n _ {k}}{n}\right) = \frac {C h _ {b a l} - 1}{C - 1}.
$$
Proposition 3. Assuming that there are  $C$  classes and the graph is generated according to the model with given expected degrees, we have  $\mathbb{E}h_{bal}^{adj} = 0$ .
Proof. Let us denote the sizes of the classes by  $n_1, \ldots, n_C$ . Recall that we use the notation  $D_k = \sum_{v: y_v = k} d(v)$ . It is easy to see that