SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Oleg Platonov*
HSE University
Yandex Research
olegplatonov@yandex-team.ru
Skoltech
dkuznedelev@yandex-team.ru
Artem Babenko
artem.babenko@phystech.edu
Liudmila Prokhorenkova*†
ostroumova-la@yandex-team.ru
Homophily is a graph property describing the tendency of edges to connect similar nodes; the opposite is called heterophily. It is often believed that heterophilous graphs are challenging for standard message-passing graph neural networks (GNNs), and much effort has been put into developing efficient methods for this setting. However, there is no universally agreed-upon measure of homophily in the literature. In this work, we show that commonly used homophily measures have critical drawbacks preventing the comparison of homophily levels across different datasets. For this, we formalize desirable properties for a proper homophily measure and verify which measures satisfy which properties. In particular, we show that a measure that we call adjusted homophily satisfies more desirable properties than other popular homophily measures while being rarely used in graph machine learning literature. Then, we go beyond the homophily-heterophily dichotomy and propose a new characteristic that allows one to further distinguish different sorts of heterophily. The proposed label informativeness (LI) characterizes how much information a neighbor's label provides about a node's label. We prove that this measure satisfies important desirable properties. We also observe empirically that LI better agrees with GNN performance compared to homophily measures, which confirms that it is a useful characteristic of the graph structure.
Graphs are a natural way to represent data from various domains such as social networks, citation networks, molecules, transportation networks, text, code, and others. Machine learning on graph-structured data has experienced significant growth in recent years, with Graph Neural Networks (GNNs) showing particularly strong results. Many variants of GNNs have been proposed [16, 10, 43, 45], most of them can be unified by a general Message Passing Neural Networks (MPNNs) framework [7]. MPNNs combine node features (attributes) with graph topology to learn complex dependencies between the nodes. For this, MPNNs iteratively update the representation of each node by aggregating information from the previous-layer representations of the node itself and its neighbors.
In many real-world networks, edges tend to connect similar nodes: users in social networks tend to connect to users with similar interests, and papers in citation networks mostly cite works from the same research area. This property is usually called homophily. The opposite of homophily is heterophily: for instance, in social networks, fraudsters rarely connect to other fraudsters, while in dating networks, edges often connect the opposite genders. Early works on GNNs mainly focus on homophilous graphs. However, it was later discovered that classic GNNs typically do not perform well on heterophilous graphs, and new GNN architectures have been developed for this setting [1, 4, 32, 48, 49].
To measure the level of homophily, several homophily measures are used in the literature [1, 20, 32, 48], but these measures may significantly disagree with each other. In this work, we start by addressing the problem of how to properly measure the homophily level of a graph. For this, we formalize some desirable properties of a reasonable homophily measure and check which measures satisfy which properties. One essential property is called constant baseline and, informally speaking, it requires a measure to be not biased towards particular numbers of classes or their size balance. Our analysis reveals that commonly used homophily measures do not satisfy this property and thus cannot be compared across different datasets. In contrast, a measure that we call adjusted homophily (a.k.a. assortativity coefficient) satisfies most of the desirable properties while being rarely used in graph ML literature. Based on our theoretical analysis, we advise using adjusted homophily as a better alternative to the commonly used measures.
Then, we note that heterophilous datasets may have various connectivity patterns, and some of them are easier for GNNs than others. Motivated by that, we propose a new graph property called label informativeness (LI) that allows one to further distinguish different sorts of heterophily. LI characterizes how much information the neighbor's label provides about the node's label. We analyze this measure via the same theoretical framework and show that it satisfies the constant baseline property and thus is comparable across datasets. We also observe empirically that LI better agrees with GNN performance than homophily measures. Thus, while being very simple to compute, LI intuitively illustrates why GNNs can sometimes perform well on heterophilous datasets — a phenomenon recently observed in the literature. While LI is not a measure of homophily, it naturally complements adjusted homophily by distinguishing different heterophily patterns.
In summary, we propose a theoretical framework that allows for an informed choice of suitable characteristics describing graph connectivity patterns in node classification tasks. Based on this framework, we suggest using adjusted homophily to measure whether similar nodes tend to be connected. To further characterize the datasets and distinguish different sorts of heterophily, we propose a new measure called label informativeness.
2 Homophily measures
Assume that we are given a graph  $G = (V, E)$  with nodes  $V$ ,  $|V| = n$ , and edges  $E$ . Throughout the paper, we assume that the graph is simple (without self-loops and multiple edges) and undirected. Each node  $v \in V$  has a feature vector  $\mathbf{x}_v \in \mathbb{R}^m$  and a class label  $y_v \in \{1, \dots, C\}$ . Let  $n_k$  denote the size of  $k$ -th class, i.e.,  $n_k = |\{v : y_v = k\}|$ . By  $N(v)$  we denote the neighbors of  $v$  in  $G$  and by  $d(v) = |N(v)|$  the degree of  $v$ . Also, let  $D_k := \sum_{v : y_v = k} d(v)$ . Let  $p(\cdot)$  denote the empirical distribution of class labels, i.e.,  $p(k) = \frac{n_k}{n}$ . Then, we also define degree-weighted distribution as  $\bar{p}(k) = \frac{\sum_{v : y_v = k} d(v)}{2|E|} = \frac{D_k}{2|E|}$ .
2.1 Popular homophily measures
Many GNN models implicitly make a so-called homophily assumption: that similar nodes are connected. Similarity can be considered in terms of node features or node labels. Usually, label homophily is analyzed, and we also focus on this aspect, leaving feature homophily for further studies. There are several commonly used homophily measures in the literature. Edge homophily [1, 48] is the fraction of edges that connect nodes of the same class:
$$
h _ {e d g e} = \frac {\left| \left\{\{u , v \} \in E : y _ {u} = y _ {v} \right\} \right|}{\left| E \right|}. \tag {1}
$$
<sup>3</sup>We further denote (unordered) edges by  $\{u, v\}$  and ordered pairs of nodes by  $(u, v)$ .
Node homophily [32] computes the fraction of neighbors that have the same class for all nodes and then averages these values across the nodes:
$$
h _ {n o d e} = \frac {1}{n} \sum_ {v \in V} \frac {\left| \left\{u \in N (v) : y _ {u} = y _ {v} \right\} \right|}{d (v)}.
$$
These two measures are intuitive but have the downside of being sensitive to the number of classes and their balance, which makes them hard to interpret and incomparable across different datasets [20]. For example, suppose that each node in a graph is connected to one node of each class. Then, both edge homophily and node homophily for this graph will be equal to  $\frac{1}{C}$ . Thus, these metrics will produce widely different values for graphs with different numbers of classes, despite these graphs being similar in exhibiting no homophily. To fix these issues, Lim et al. [20] propose another homophily measure sometimes referred to as class homophily [22]. Class homophily measures excess homophily compared to a null model where edges are independent of the labels. More formally,
$$
h _ {c l a s s} = \frac {1}{C - 1} \sum_ {k = 1} ^ {C} \left[ \frac {\sum_ {v : y _ {v} = k} | \{u \in N (v) : y _ {u} = y _ {v} \} |}{\sum_ {v : y _ {v} = k} d (v)} - \frac {n _ {k}}{n} \right] _ {+},
$$
where  $[x]_{+} = \max \{x,0\}$ . The factor  $\frac{1}{C - 1}$  scales  $h_{class}$  to the interval [0, 1]; larger values indicate more homophilous graphs and non-homophilous ones are expected to have close to zero values.
However, there are still some issues with class homophily. First, when correcting the fraction of intra-class edges by its expected value, class homophily does not consider the variation of node degrees. Indeed, if nodes of class  $k$  have, on average, larger degrees than  $2|E| / n$ , then the probability that a random edge goes to that class can be significantly larger than  $n_k / n$ . Second, only positive deviations from  $n_k / n$  contribute to class homophily, while classes with heterophilous connectivity patterns are neglected. Let us illustrate these drawbacks of class homophily with a simple example.
Example Let us construct non-homophilous graphs for which class homophily is significantly larger than zero. First, we take a clique of size  $r$  with all nodes belonging to the red class; then, for each node in the clique, connect it to  $r - 1$  leaves, all of which belong to the blue class (example for  $r = 4$  is shown on the right). Note that all blue nodes are strictly heterophilous (i.e., only connect to nodes of the opposite class), while all red nodes are class-agnostic (i.e., have the same number of neighbors of both classes). Such graphs are clearly non-homophilous, and a meaningful homophily measure should not produce a value significantly greater than zero for them. However, class homophily for such graphs is positive and can become as large as  $\frac{1}{2}$ :  $h_{\text{class}} = \frac{1}{2} - \frac{1}{r} \rightarrow \frac{1}{2}$  as  $r \rightarrow \infty$ .
2.2 Desirable properties for homophily measures
Above, we discussed some disadvantages of existing homophily measures. In this section, we formalize and extend this discussion: we propose a list of properties desirable for a good homophily measure. Our analysis is motivated by recent studies of clustering and classification performance measures [8, 9], but not all their properties can be transferred to homophily measures. For instance, we do not require symmetry — a property that a measure does not change when we swap the compared objects — since homophily compares entities of different nature (a graph and a labeling). For the same reason, the distance property (requiring a measure to be linearly transformed to a metric distance) cannot be defined. On the other hand, some of our properties are novel.
Maximal agreement This property requires that perfectly homophilous graphs achieve a constant upper bound of the measure. Formally, we say that a homophily measure  $h$  satisfies maximal agreement if for any graph  $G$  in which  $y_{u} = y_{v}$  for all  $\{u,v\} \in E$  we have  $h(G) = c_{\max}$ . For all other graphs  $G$ , we require  $h(G) < c_{\max}$ .
Minimal agreement We say that a homophily measure  $h$  satisfies minimal agreement if  $h(G) = c_{\min}$  for any graph  $G$  in which  $y_u \neq y_v$  for all  $\{u, v\} \in E$ . For all other graphs  $G$ , we require  $h(G) > c_{\min}$ . In other words, if all edges connect nodes of different classes, we expect to observe a constant minimal value.
Constant baseline This property ensures that homophily is not biased towards particular class size distributions. Intuitively, if the graph structure is independent of labels, we would expect a low homophily value. Moreover, if we want a measure to be comparable across datasets, we expect to observe the same low value in all such cases. There are several ways to formalize the concept of independence, and we suggest the one based on the so-called configuration model.
Definition 1. Configuration model is defined as follows: take  $n$  nodes, assign each node  $v$  degree  $d(v)$ , and then randomly pair edge endpoints to obtain a graph. $^4$
Assuming that we are given  $n$  labeled nodes and the graph is constructed according to the configuration model (independently from the labels), we expect to observe a fixed (small) homophily independently of the number of classes and class size balance. We formalize this property as follows and refer to Appendix B.2 for other possible definitions.
Definition 2. A homophily measure  $h$  has asymptotic constant baseline if for  $G$  generated according to the configuration model and for any  $\varepsilon > 0$  with probability  $1 - o(1)$  we have  $|h(G) - c_{base}| < \varepsilon$  for some constant  $c_{base}$  as  $n \to \infty$ .
In combination with maximal agreement, asymptotic constant baseline makes the values of a homophily measure comparable across different datasets: the maximal agreement guarantees that perfectly homophilous graphs have the same value, while constant baseline aligns the uninformative cases with neither strong homophily nor strong heterophily.
Empty class tolerance Since homophily measures are used to compare different graph datasets, they have to be comparable across datasets with varying numbers of classes. For this, the following property is required.
Definition 3. A measure is tolerant to empty classes if it is defined and it does not change when we introduce an additional dummy label that is not present in the data.
For instance, edge homophily and node homophily are empty class tolerant, while class homophily is not. Empty class tolerance is a new property that was not discussed in [8, 9] since classification and clustering evaluation measures are used within one given dataset, see Appendix B.3 for more details.
Monotonicity As we discuss in Appendix B.3, it can be non-trivial to define monotonicity for homophily measures and there can be different possible options. In this paper, we use the following definition that aligns especially well with edge-wise homophily measures discussed below in Section 2.5.
Definition 4. A homophily measure is monotone if it is empty class tolerant, increases when we add an edge between two nodes of the same class (except for perfectly homophilous graphs) and decreases when we add an edge between two nodes of different classes (except for perfectly heterophilous graphs).
In contrast to Gösgens et al. [8, 9], our notion of monotonicity requires the property to hold across graphs with different numbers of classes. This is caused by the empty class tolerance property.
2.3 Properties of popular homophily measures
Below we briefly discuss the properties of popular homophily measures. In most cases, the proofs are straightforward or follow from Section 2.5 below. Table 1 summarizes the results.
Edge homophily satisfies maximal and minimal agreement and is empty class tolerant and monotone. However, it does not satisfy asymptotic constant baseline, which is a critical drawback: one can get misleading results in settings with imbalanced classes.
Node homophily satisfies maximal and minimal agreement. It is empty class tolerant, but not monotone: adding an edge between two perfectly homophilous nodes of the same class does not change node homophily. Similarly to edge homophily, node homophily does not satisfy the asymptotic constant baseline and thus is incomparable across different datasets.
See Appendix A for additional discussions about the model.
Table 1: Properties of homophily measures: maximal agreement (Max), minimal agreement (Min), asymptotic constant baseline (ACB), empty class tolerance (ECT), monotonicity (Mon).  $X^{*}$  denotes that the property is not satisfied in general, but holds for large  $h_{adj}$  (see Theorem 1).
Measure	Max	Min	ACB	ECT	Mon
Edge homophily	✓	✓	✗	✓	✓
Node homophily	✓	✓	✗	✓	✗
Class homophily	✓	✗	✗	✗	✗
Adjusted homophily	✓	✗	✓	✓	✗*
Class homophily satisfies maximal agreement with  $h_{\text{class}} = 1$ , but minimal agreement is not satisfied: not only perfectly heterophilous graphs may have  $h_{\text{class}} = 0$ . Class homophily is not empty class tolerant and thus is not monotone. Additionally, it does not have the asymptotic constant baseline. See Appendix B.1 for the proofs and discussions. Interestingly, removing the  $[\cdot]_+$  operation from the definition of class homophily solves the problem with the asymptotic constant baseline, but minimal agreement, empty class tolerance, and monotonicity are still violated.
2.4 Adjusted homophily
Let us discuss a much less known homophily measure that by construction satisfies two important properties — maximal agreement and constant baseline. To derive this measure, we start with edge homophily and first enforce the constant baseline property by subtracting the expected value of the measure. Under the configuration model, the probability that a given edge endpoint will be connected to a node with a class  $k$  is (up to a negligible term)  $\frac{\sum_{v: y_v = k} d(v)}{2|E|}$ . Thus, the adjusted value becomes  $h_{edge} - \sum_{k=1}^{C} \frac{D_k^2}{4|E|^2}$ . Now, to enforce maximal agreement, we normalize the measure as follows:
$$
h _ {a d j} = \frac {h _ {e d g e} - \sum_ {k = 1} ^ {C} \bar {p} (k) ^ {2}}{1 - \sum_ {k = 1} ^ {C} \bar {p} (k) ^ {2}}, \tag {2}
$$
where we use the notation  $\bar{p}(k) = \frac{D_k}{2|E|}$ . This measure is known in graph analysis literature as assortativity coefficient [28]. While assortativity is a general concept that is often applied to node degrees, it reduces to (2) when applied to categorical node attributes on undirected graphs. Unfortunately, this measure is rarely used in graph ML literature (Suresh et al. [41] is the only work we are aware of that uses it for measuring homophily), while our theoretical analysis shows that it satisfies many desirable properties. Indeed, the following theorem holds (see Appendix B.4 for the proof).
Theorem 1. Adjusted homophily satisfies maximal agreement, asymptotic constant baseline, and empty class tolerance. The minimal agreement is not satisfied. Moreover, this measure is monotone if  $h_{adj} > \frac{\sum_i \bar{p}(i)^2}{\sum_i \bar{p}(i)^2 + 1}$  and we note that the bound  $\frac{\sum_i \bar{p}(i)^2}{\sum_i \bar{p}(i)^2 + 1}$  is always smaller than 0.5. When  $h_{adj}$  is small, counterexamples to monotonicity exist.
While adjusted homophily violates some properties, it still dominates all other measures and is comparable across different datasets with varying numbers of classes and class size balance. Thus, we recommend using it as a measure of homophily in further works.
2.5 Edge-wise homophily vs classification evaluation measures
We conclude the analysis of homophily measures by establishing a connection between them and classification evaluation measures [8]. For this, let us first define edge-wise homophily measures. We say that a homophily measure is edge-wise if it is a function of the class adjacency matrix that we now define. Since we consider undirected graphs, each edge  $\{u,v\} \in E$  gives two ordered pairs of nodes  $(u,v)$  and  $(v,u)$ . We can define a class adjacency matrix  $\mathcal{C}$  as follows: each matrix element  $c_{ij}$  denotes the number of edges  $(u,v)$  such that  $y_u = i$  and  $y_v = j$ . Since the graph is undirected, the matrix  $\mathcal{C}$  is symmetric. Note that monotonicity can be naturally put in terms of the class adjacency matrix: adding an edge between two nodes of the same class  $i$  corresponds to incrementing the
(a) Informative neighbors
(b) Less informative neighbors
Figure 1: Non-homophilous graphs with different connection patterns
diagonal element  $c_{ii}$  by two, and adding an edge between two nodes of different classes  $i$  and  $j$  corresponds to incrementing  $c_{ij}$  and  $c_{ji}$  by one.
Now, for each edge  $(u,v)$ , let us say that  $y_{u}$  is a true label (for some object) and  $y_{v}$  is a predicted label. Then, any classification evaluation measure (e.g., accuracy) applied to this dataset is a measure of homophily. Based on that, we get the following correspondence.
Clearly, accuracy corresponds to edge homophily  $h_{edge}$ .
Interestingly, both Cohen's Kappa and Matthews correlation coefficient correspond to adjusted homophily. As argued in [8], the Matthews coefficient is one of the best classification evaluation measures in terms of its theoretical properties. Our extended analysis confirms this conclusion for the corresponding homophily measure: we prove a stronger version of asymptotic constant baseline and also establish a stronger variant of monotonicity in the interval of large values of  $h_{adj}$ . The latter result is essential since in [8] it was only claimed that monotonicity is violated when  $C > 2$ .
Another measure advised by [8] is symmetric balanced accuracy. Since in our case the class adjacency matrix is symmetric, it gives the same expression as balanced accuracy:  $h_{bal} = \frac{1}{C}\sum_{k=1}^{C}\frac{|(u,v):y_u=y_v=k|}{D_k}$ . The obtained measure satisfies the maximal and minimal agreement properties. However, it is not empty class tolerant and thus is not monotone. The asymptotic constant baseline is also not satisfied: the value  $c_{base} = 1/C$  depends on the number of classes. Thus, despite this measure being suitable for classification evaluation, it cannot be used as a homophily measure. This difference is caused by the fact that homophily measures have to be comparable across datasets with different numbers of classes, while for classification evaluation it is not required.
Finally, note that similarly to our derivations in Section 2.4, the value  $h_{bal}$  can be adjusted to have both maximal agreement and constant baseline. Interestingly, this would lead to a slightly modified class homophily with  $[\cdot]_+$  operation removed. As discussed in Section 2.3, the obtained measure satisfies only the maximal agreement and constant baseline.
To conclude, there is a correspondence between edge-wise homophily measures and classification evaluation measures. Adjusted homophily corresponds to both Cohen's Kappa and Matthews coefficient. In terms of the satisfied properties, adjusted homophily dominates all other measures derived from this correspondence.
Finally, we also note that homophily measures can be directly related to community detection evaluation measures including the well-established characteristic in graph community detection literature called modularity [30]. See Appendix B.5 for a detailed discussion.
3 Label informativeness
In the previous section, we discussed in detail how to properly measure the homophily level of a graph. While homophily indicates whether similar nodes are connected, heterophily is defined as the negation of homophily. Thus, heterophilous graphs may have very different connectivity patterns. In this section, we characterize such patterns.
To give an example, among strictly heterophilous graphs in which nodes never connect to nodes of the same class, there can be those where edges are drawn between particular pairs of classes (Figure 1a) and those where the class of a node cannot be derived from the class of its neighbor (Figure 1b). While adjusted homophily correctly captures the absence of homophily in these graphs, it is not designed to identify which type they belong to. However, distinguishing such graphs is practically important: informative neighbors can be very useful for models accounting for the graph structure.