SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {E} h _ {b a l} ^ {a d j} = \frac {1}{C - 1} \sum_ {i = 1} ^ {C} \left(\frac {D _ {i}}{\sum_ {j} D _ {j}} - \frac {n _ {i}}{\sum_ {j} n _ {j}}\right) = \frac {1 - 1}{C - 1} = 0.
$$
B.2 Constant baseline
There are several possible ways to formalize the constant baseline property. In the literature [8, 9], this property is often formalized as follows: assuming some randomized model, the expected value of a measure should be equal to some constant. For homophily measures, this corresponds to the following definition.
Definition 5. A homophily measure  $h$  has constant baseline if for  $G$  generated according to the configuration model we have  $\mathbb{E}h(G) = c_{\text{base}}$  for some constant  $c_{\text{base}}$ .
This property is very strict as minor variations in the definition of the model (e.g., different alternatives of the configuration model discussed in Appendix A) may lead to negligibly small error terms preventing us from getting exactly the same constant  $c_{\text{base}}$ . Thus, in the main text, we use the alternative Definition 2. This definition is weaker in the sense that we allow asymptotically negligible deviations from the constant  $c_{\text{base}}$ . On the other hand, our definition is somewhat stronger since we also require the concentration of the value around its expectation. In that sense, our definition is stronger than the asymptotic constant baseline defined in [8].
B.3 Monotonicity property
In this section, we discuss how one can define monotonicity for homophily measures.
First, let us revisit how monotonicity is defined for measures used in other areas. In general, the monotonicity property is defined as follows: if an object  $B'$  is definitely more similar to an object  $A$  than  $B$ , then the similarity between  $B'$  and  $A$  should be larger than between  $B$  and  $A$ . The only problem is to define what it means to be "definitely more similar". In [9], the monotonicity property was introduced for cluster similarity measures. Such measures evaluate how close two partitions of the same set of items are. To formally define the concept of more similar partitions, Gösgens et al. [9] use the concept of perfect splits and perfect merges — such transformations of a partition  $B$  that make it more similar to  $A$ . Later, monotonicity was defined for classification evaluation measures that compare the predicted labels for a set of items with the reference ones [8]. Here it is easier to
formulate what it means that a labeling  $B'$  is definitely more similar to the reference labeling than  $B$ . Indeed, if we consider a labeling  $B$  and change one incorrect label to the correct one, then it becomes closer to the reference labeling.
Now, let us return to homophily measures. Such measures evaluate the agreement between a graph and a labeling of its nodes. To define monotonicity, we need such transformations that make a graph and a labeling definitely more similar. Since the problem is not symmetric, we can either rewire edges or relabel nodes. Regarding relabeling of nodes, we may say that taking a perfectly heterophilous node (that is not connected to any node of its class) and relabeling it in such a way that it becomes perfectly homophilous (which is only possible when all its neighbors belong to one class) should make the graph and labeling more similar to each other. Regarding the edges, we may say that taking an edge that connects two nodes of different classes and rewiring it to connect two nodes of the same class should increase the measure. However, rewiring edges may also affect the graph structure in a non-trivial way (degree distribution, diameter, etc.).
In this paper, we follow the edge rewiring approach, and thus our definition aligns well with edge-wise homophily measures and with previous works on classification and clustering performance measures [8, 9].
If we consider edge-wise measures (that can be defined in terms of the class adjacency matrix) and directly follow the definition of monotonicity in [8, 9], our definition would be as follows: a homophily measure is monotone if it increases when we decrement  $c_{ij}$  and  $c_{ji}$  by one and increment either  $c_{ii}$  or  $c_{jj}$  by two for  $i \neq j$ . This condition corresponds to taking an edge with  $y_u \neq y_v$  and changing  $y_u$  to  $y_v$  or vice versa.
However, an important property that has to be taken into account when discussing monotonicity for homophily measures is the fact that such measures are used to compare different graph datasets, in contrast to classification evaluation measures that compare the agreement between predicted labelings with a fixed given reference labeling. This means that for classification measures, monotonicity and constant baseline are critical for fixed numbers of classes and elements. In contrast, homophily measures have to be comparable across datasets with different sizes and numbers of classes. For instance, balanced accuracy has a constant baseline with the expected value of  $\frac{1}{C}$ , which is sufficient for classification evaluation but is a drawback for measuring homophily. Because of this, we do not consider the standard definition of monotonicity as it is restricted to a fixed number of classes only. Instead, we introduce tolerance to empty classes, and based on that, we introduce a stronger version of monotonicity where we allow separate edge additions or deletions. In fact, our definition of monotonicity (Definition 4) is similar to strong monotonicity in [8, 9], but also requires empty class tolerance since comparing measures across different numbers of classes is crucial.
B.4 Proof of Theorem 1
The fact that adjusted homophily is empty class tolerant directly follows from its definition: an empty class does not contribute to the numerator or denominator of  $h_{adj}$ .
Maximal agreement is also straightforward: we have  $h_{edge} - \sum_{k=1}^{C} D_k^2 / (2|E|)^2 \leq 1 - \sum_{k=1}^{C} D_k^2 / (2|E|)^2$  with equality if and only if all edges are homophilous.
Minimal agreement is not satisfied since the value  $\frac{-\sum_{k=1}^{C} D_k^2 / (2|E|)^2}{1 - \sum_{k=1}^{C} D_k^2 / (2|E|)^2}$  can be different for different datasets. Thus, perfectly heterophilous datasets may get different values, which also causes some monotonicity violations for small values of  $h_{adj}$ .
Asymptotic constant baseline of adjusted homophily Now, let us formulate and prove the constant baseline property.
Proposition 4. Let  $G$  be a graph generated according to the configuration model. Assume that the degree-weighted distribution of classes is not very unbalanced, i.e., that  $1 - \sum_{k} \bar{p}(k)^2 \gg 1 / \sqrt{E}$ . Then,  $|h_{adj}| \leq \phi$  with probability  $1 - o(1)$  for some  $\phi = \phi(|E|) \to 0$  as  $|E| \to \infty$ .
Proof. Let us first analyze the numerator of  $h_{adj}$  which we denote by  $h_{mod}$  (since it corresponds to the network's modularity — see Appendix B.5).
Let  $\xi$  denote the number of intra-class edges in a graph constructed according to the configuration model. We may write:
$$
\xi = \frac {1}{2} \sum_ {k = 1} ^ {C} \sum_ {i = 1} ^ {D _ {k}} \mathbb {1} _ {i, k},
$$
where  $\mathbb{1}_{i,k}$  indicates that an endpoint  $i$  of some node with class  $k$  is connected to another endpoint of class  $k$ . Note that the probability of this event is  $\frac{D_k - 1}{2|E| - 1}$ . Thus, we have
$$
\mathbb {E} h _ {m o d} = \frac {\xi}{| E |} - \sum_ {k = 1} ^ {C} \frac {D _ {k} ^ {2}}{4 | E | ^ {2}} = \frac {1}{| E |} \sum_ {k = 1} ^ {C} \frac {D _ {k} (D _ {k} - 1)}{2 (2 | E | - 1)} - \sum_ {k = 1} ^ {C} \frac {D _ {k} ^ {2}}{4 | E | ^ {2}} = O \left(\frac {1}{| E |}\right). \qquad (4)
$$
Now, let us estimate the variance of the number of intra-class edges. We may write:
$$
\begin{array}{l} \operatorname {V a r} (2 \xi) = \mathbb {E} (2 \xi) ^ {2} - (\mathbb {E} (2 \xi)) ^ {2} = \mathbb {E} \left(\sum_ {k = 1} ^ {C} \sum_ {i = 1} ^ {D _ {k}} \mathbb {1} _ {i, k}\right) ^ {2} - (\mathbb {E} 2 \xi) ^ {2} \\ = \mathbb {E} (2 \xi) + \sum_ {k = 1} ^ {C} 2 \cdot \binom {D _ {k}} {2} \cdot \left(\frac {1}{2 | E | - 1} + \frac {D _ {k} - 2}{2 | E | - 2} \cdot \frac {D _ {k} - 3}{2 | E | - 3}\right) \\ + \sum_ {k = 1} ^ {C} \sum_ {l = k + 1} ^ {C} 2 D _ {k} D _ {l} \cdot \frac {D _ {k} - 1}{2 | E | - 1} \cdot \frac {D _ {l} - 1}{2 | E | - 3} - (\mathbb {E} 2 \xi) ^ {2} \\ = \mathbb {E} (2 \xi) + \sum_ {k = 1} ^ {C} \frac {D _ {k} ^ {4} + O \left(D _ {k} ^ {3}\right)}{4 | E | ^ {2}} \\ + \sum_ {k = 1} ^ {C} \sum_ {l = k + 1} ^ {C} \frac {2 D _ {k} ^ {2} D _ {l} ^ {2} + O (D _ {k} ^ {2} D _ {l} + D _ {k} D _ {l} ^ {2})}{4 | E | ^ {2}} - \left(\sum_ {k = 1} ^ {C} \frac {D _ {k} ^ {2} + O (D _ {k})}{2 | E |}\right) ^ {2} \\ = \mathbb {E} (2 \xi) + \sum_ {k = 1} ^ {C} \frac {O \left(D _ {k} ^ {3}\right)}{4 | E | ^ {2}} + \sum_ {k = 1} ^ {C} \sum_ {l = k + 1} ^ {C} \frac {O \left(D _ {k} ^ {2} D _ {l} + D _ {k} D _ {l} ^ {2}\right)}{4 | E | ^ {2}} \\ = \mathbb {E} (2 \xi) + O (| E |) = O (| E |). \\ \end{array}
$$
Let  $\varphi = \varphi(|E|)$  be any function such that  $\varphi \to \infty$  as  $|E| \to \infty$ . Using the Chebyshev's inequality and (4), we get:
$$
\begin{array}{l} \mathrm {P} \left(\left| h _ {m o d} \right| \geq \frac {\varphi}{\sqrt {E}}\right) = \mathrm {P} \left(\left| h _ {m o d} - \mathbb {E} h _ {m o d} \right| > \frac {\varphi}{\sqrt {E}} + O \left(\frac {1}{\left| E \right|}\right)\right) \\ = O \left(\frac {\operatorname {V a r} (\xi) | E |}{| E | ^ {2} \varphi^ {2}}\right) = O \left(\frac {1}{\varphi^ {2}}\right) = o (1). \\ \end{array}
$$
Recall that  $h_{adj} = \frac{h_{mod}}{1 - \sum_{k=1}^{C} \bar{p}(k)^2}$ . Since we have  $1 - \sum_{k=1}^{C} \bar{p}(k)^2 \gg 1 / \sqrt{E}$  and  $|h_{mod}| < \frac{\varphi}{\sqrt{E}}$  with probability  $1 - o(1)$ , we can choose such slowly growing  $\varphi$  that  $h_{adj} < \frac{\varphi}{\sqrt{E}\left(1 - \sum_{k=1}^{C} \bar{p}(k)^2\right)} = o(1)$  with probability  $1 - o(1)$ .
Monotonicity of adjusted homophily Finally, let us analyze the monotonicity of adjusted homophily and finish the proof of Theorem 1.
Recall that a homophily measure is monotone if it is empty class tolerant, and it increases when we increment a diagonal element by 2 (except for perfectly homophilous graphs) and decreases when we increase  $c_{ij}$  and  $c_{ji}$  by one for  $i \neq j$  (except for perfectly heterophilous graphs).
Empty class tolerance is clearly satisfied for adjusted homophily, so let us now analyze what happens when we increment a diagonal element or two (symmetric) off-diagonal elements.
Let us denote  $N := 2|E|$ ,  $a_i := \sum_j c_{ij}$ . Then, we have  $\bar{p}(i) = \frac{a_i}{N}$ . Thus, we have to prove that the measure is monotone when  $h_{adj} > \frac{\sum_i a_i^2}{\left(\sum_i a_i^2 + n^2\right)}$ .
Using the notation with a class adjacency matrix, adjusted homophily can be written as follows:
$$
h _ {a d j} = \frac {N \sum_ {i} c _ {i i} - \sum_ {i} a _ {i} ^ {2}}{N ^ {2} - \sum_ {i} a _ {i} ^ {2}}.
$$
To check whether the measure increases when we increment diagonal elements, let us compute the derivative w.r.t.  $c_{kk}$  for some  $k$ :
$$
\frac {\partial h _ {a d j}}{\partial c _ {k k}} = \frac {\left(\sum_ {i} c _ {i i} + N - 2 a _ {k}\right) \left(N ^ {2} - \sum_ {i} a _ {i} ^ {2}\right)}{\left(N ^ {2} - \sum_ {i} a _ {i} ^ {2}\right) ^ {2}} - \frac {\left(2 N - 2 a _ {k}\right) \left(N \sum_ {i} c _ {i i} - \sum_ {i} a _ {i} ^ {2}\right)}{\left(N ^ {2} - \sum_ {i} a _ {i} ^ {2}\right) ^ {2}}.
$$
Let us simplify the numerator:
$$
\begin{array}{l} \left(\sum_ {i} c _ {i i} + N - 2 a _ {k}\right) \left(N ^ {2} - \sum_ {i} a _ {i} ^ {2}\right) - (2 N - 2 a _ {k}) \left(N \sum_ {i} c _ {i i} - \sum_ {i} a _ {i} ^ {2}\right) \\ = N ^ {3} - 2 N ^ {2} a _ {k} - \sum_ {i} c _ {i i} \sum_ {i} a _ {i} ^ {2} - N ^ {2} \sum_ {i} c _ {i i} + 2 a _ {k} N \sum_ {i} c _ {i i} + N \sum_ {i} a _ {i} ^ {2}. \\ \end{array}
$$
For monotonicity, we need the derivative to be positive, i.e.,
$$
N ^ {3} + 2 a _ {k} N \sum_ {i} c _ {i i} + N \sum_ {i} a _ {i} ^ {2} > 2 N ^ {2} a _ {k} + \sum_ {i} c _ {i i} \sum_ {i} a _ {i} ^ {2} + N ^ {2} \sum_ {i} c _ {i i}.
$$
Let us denote by  $\bar{D}$  the sum of all off-diagonal elements, i.e.,  $\bar{D} := N - \sum_{i} c_{ii}$ . Then, we can rewrite the above condition as follows:
$$
N ^ {2} \bar {D} + \bar {D} \sum_ {i} a _ {i} ^ {2} > 2 N a _ {k} \bar {D},
$$
$$
N ^ {2} + \sum_ {i} a _ {i} ^ {2} > 2 N a _ {k}.
$$
The latter equality holds since
$$
N ^ {2} + \sum_ {i} a _ {i} ^ {2} > N ^ {2} + a _ {k} ^ {2} \geq 2 N a _ {k}.
$$
Thus,  $h_{adj}$  increases when we increment a diagonal element.
To see whether the measure decreases when we increment the off-diagonal elements, let us compute the derivative of the measure w.r.t.  $c_{kl}$  (which is equal to  $c_{lk}$ ) for  $k \neq l$ :
$$
\frac {\partial h _ {a d j}}{\partial c _ {k l}} = \frac {(2 \sum_ {i} c _ {i i} - 2 a _ {k} - 2 a _ {l}) (N ^ {2} - \sum_ {i} a _ {i} ^ {2})}{(N ^ {2} - \sum_ {i} a _ {i} ^ {2}) ^ {2}} - \frac {(4 N - 2 a _ {k} - 2 a _ {l}) (N \sum_ {i} c _ {i i} - \sum_ {i} a _ {i} ^ {2})}{(N ^ {2} - \sum_ {i} a _ {i} ^ {2}) ^ {2}}.
$$
The numerator is:
$$
\begin{array}{l} \left(2 \sum_ {i} c _ {i i} - 2 a _ {k} - 2 a _ {l}\right) \left(N ^ {2} - \sum_ {i} a _ {i} ^ {2}\right) - (4 N - 2 a _ {k} - 2 a _ {l}) \left(N \sum_ {i} c _ {i i} - \sum_ {i} a _ {i} ^ {2}\right) \\ = - 2 (a _ {k} + a _ {l}) N ^ {2} - 2 \sum_ {i} c _ {i i} \sum_ {i} a _ {i} ^ {2} - 2 N ^ {2} \sum_ {i} c _ {i i} + 2 (a _ {k} + a _ {l}) N \sum_ {i} c _ {i i} + 4 N \sum_ {i} a _ {i} ^ {2}. \\ \end{array}
$$
For monotonicity, we need the following inequality:
$$
\left(a _ {k} + a _ {l}\right) N ^ {2} + \sum_ {i} c _ {i i} \sum_ {i} a _ {i} ^ {2} + N ^ {2} \sum_ {i} c _ {i i} > \left(a _ {k} + a _ {l}\right) N \sum_ {i} c _ {i i} + 2 N \sum_ {i} a _ {i} ^ {2}. \tag {5}
$$
This inequality is not always satisfied. Indeed, both  $a_{k} + a_{l}$  and  $\sum_{i} c_{ii}$  can be small. Let us note, however, that the inequality is satisfied if  $h_{adj}$  is large enough. Indeed, the sufficient condition for (5) is
$$
\sum_ {i} c _ {i i} \left(\sum_ {i} a _ {i} ^ {2} + N ^ {2}\right) > 2 N \sum_ {i} a _ {i} ^ {2}.
$$
This is equivalent to the following inequality for  $h_{adj}$ :
$$
\begin{array}{l} h _ {a d j} = \frac {N \sum_ {i} c _ {i i} - \sum_ {i} a _ {i} ^ {2}}{N ^ {2} - \sum_ {i} a _ {i} ^ {2}} > \frac {N \sum_ {i} \frac {2 N \sum_ {i} a _ {i} ^ {2}}{\left(\sum_ {i} a _ {i} ^ {2} + N ^ {2}\right)} - \sum_ {i} a _ {i} ^ {2}}{N ^ {2} - \sum_ {i} a _ {i} ^ {2}} \\ = \frac {N ^ {2} \sum_ {i} a _ {i} ^ {2} - \left(\sum_ {i} a _ {i} ^ {2}\right) ^ {2}}{\left(N ^ {2} - \sum_ {i} a _ {i} ^ {2}\right) \left(\sum_ {i} a _ {i} ^ {2} + N ^ {2}\right)} = \frac {\sum_ {i} a _ {i} ^ {2}}{\left(\sum_ {i} a _ {i} ^ {2} + N ^ {2}\right)}. \\ \end{array}
$$
Thus,  $h_{adj}$  is monotone if its values are at least  $\frac{\sum_i a_i^2}{\left(\sum_i a_i^2 + N^2\right)}$ . Note that we have  $\frac{\sum_i a_i^2}{\left(\sum_i a_i^2 + N^2\right)} < 0.5$ , so when  $h_{adj} > 0.5$  it is always monotone.
However, non-monotone behavior may occur when the numerator of  $h_{adj}$  is small. This undesirable behavior is somewhat expected: due to the normalization,  $h_{adj}$  satisfies constant baseline and maximal agreement but violates minimal agreement. Since minimal agreement does not hold, one can expect monotonicity to be violated for small values of  $h_{adj}$ .
For instance, we can construct the following counter-example to monotonicity. Assume that we have four classes  $(0,1,2,3)$  and non-zero entries of the class adjacency matrix are  $c_{23} = c_{32} = M$ ,  $c_{33} = 2$ . Then, the adjusted homophily is:
$$
h _ {a d j} = \frac {2 (2 M + 2) - M ^ {2} - (M + 2) ^ {2}}{(2 M + 2) ^ {2} - M ^ {2} - (M + 2) ^ {2}} = \frac {- 2 M ^ {2}}{2 M ^ {2} + 4 M} = \frac {- M}{M + 2}.
$$
Now, we increment the entries  $c_{01}$  and  $c_{10}$  by 1. The new adjusted homophily is:
$$
h _ {a d j} ^ {\prime} = \frac {2 (2 M + 4) - M ^ {2} - (M + 2) ^ {2} - 2}{(2 M + 4) ^ {2} - M ^ {2} - (M + 2) ^ {2} - 2} = \frac {- 2 M ^ {2} + 2}{2 M ^ {2} + 1 2 M + 1 0} = \frac {1 - M}{M + 5}.
$$
We disprove monotonicity if we have  $h_{adj}^{\prime} > h_{adj}$ , i.e.,
$$
\frac {1 - M}{M + 5} > \frac {- M}{M + 2},
$$
$$
2 - M - M ^ {2} > - M ^ {2} - 5 M,
$$
$$
2 > - 4 M,
$$
which holds for all  $M \geq 1$ .
B.5 How homophily and modularity are related
Modularity Modularity is arguably the most well-known measure of goodness of a partition for a graph. It was first introduced in [30] and is widely used in community detection literature: modularity is directly optimized by some algorithms, used as a stopping criterion in iterative methods, or used as a metric to compare different algorithms when no ground truth partition is available. The basic idea is to consider the fraction of intra-community edges among all edges of  $G$  and penalize it for avoiding trivial partitions like those consisting of only one community of size  $n$ . In its general form and using the notation adopted in this paper, modularity is
$$
\frac {1}{| E |} \left(\left| \left\{\{u, v \} \in E: y _ {u} = y _ {v} \right\} \right| - \gamma \mathbb {E} \xi\right),
$$
where  $\xi$  is a random number of intra-class edges in a graph constructed according to some underlying random graph model;  $\gamma$  is the resolution parameter which allows for varying the number of communities obtained after maximizing modularity. The standard choice is  $\gamma = 1$ , which also guarantees that the expected value of modularity is 0 if a graph is generated independently of class labels according to the underlying model.
Usually, modularity assumes the configuration model. In this case, we have  $\mathbb{E}\xi = \frac{\sum_k D_k(D_k - 1)}{2(2|E| - 1)} \approx \frac{1}{4|E|} \sum_k D_k^2$ , giving the following expression:
$$
h _ {m o d} = \frac {\left| \left\{\{u , v \} \in E : y _ {u} = y _ {v} \right\} \right|}{| E |} - \sum_ {k = 1} ^ {C} \frac {D _ {k} ^ {2}}{4 | E | ^ {2}}.
$$
We refer to Newman [29], Prokhorenkova and Tikhonov [34] for more details regarding modularity and its usage in community detection literature.
Relation to homophily While modularity measures how well a partition fits a given graph, homophily measures how well graph edges agree with the partition (class labels). Thus, they essentially measure the same thing, and modularity can be used as a homophily measure. Indeed, it is easy to see that modularity coincides with the numerator of adjusted homophily, see (2). Hence, adjusted homophily can be viewed as a normalized version of modularity. Note that for modularity, normalization is not crucial as modularity is usually used to compare several partitions of the same graph. In contrast, homophily is typically used to compare different graphs, which is why normalization is essential.
C Analysis of label informativeness
In this section, we prove Theorem 2. Let us give a formal statement of this theorem.
Theorem 2. Assume that  $|E| \to \infty$  as  $n \to \infty$ . Assume that the entropy of  $\bar{p}(\cdot)$  is bounded from below by some constant. Let  $\bar{p}_{min} = \min_k \bar{p}(k)$ . Assume that  $\bar{p}_{min} \gg C / \sqrt{|E|}$  as  $n \to \infty$ . Let  $\varepsilon = \varepsilon(n)$  be any function such that  $\varepsilon \gg \frac{C}{p_{min} \sqrt{|E|}}$  and  $\varepsilon = o(1)$  as  $n \to \infty$ . Then, with probability  $1 - o(1)$ , we have  $|\mathrm{LI}| \leq K\varepsilon = o(1)$  for some constant  $K > 0$ .
$$
\mathrm {L I} = - \frac {\sum_ {c _ {1} , c _ {2}} p (c _ {1} , c _ {2}) \log \frac {p (c _ {1} , c _ {2})}{\bar {p} (c _ {1}) \bar {p} (c _ {2})}}{\sum_ {c} \bar {p} (c) \log \bar {p} (c)}.
$$
Thus, for each pair  $k,l$  ,we need to estimated  $p\left( {k,l}\right)$  . Let us denote:
$$
E (k, l) := \sum_ {u = 1} ^ {n} \sum_ {v = 1} ^ {n} \mathbb {1} \left\{\left\{u, v \right\} \in E, y _ {u} = k, y _ {v} = l \right\},
$$
so we have  $p(k, l) = \frac{E(k, l)}{2|E|}$ .
As before, we use the notation  $D_{k} = \sum_{v:y_{v} = k}d(v)$ . First, consider the case  $k\neq l$ . Let us compute the expectation of  $E(k,l)$ :
$$
\mathbb {E} E (k, l) = \frac {D _ {k} D _ {l}}{2 | E | - 1} \text {f o r} k \neq l,
$$
$$
\mathbb {E} E (k, k) = \frac {D _ {k} (D _ {k} - 1)}{2 | E | - 1}.
$$
Now, we estimate the variance. Below,  $\mathbb{1}_{i,j}$  indicates that two endpoints are connected.
$$
\begin{array}{l} \operatorname {V a r} E (k, l) = \mathbb {E} \left(\sum_ {i = 1} ^ {D _ {k}} \sum_ {j = 1} ^ {D _ {l}} \mathbb {1} _ {i, j}\right) ^ {2} - (\mathbb {E} E (k, l)) ^ {2} \\ = \mathbb {E} E (k, l) + \frac {D _ {k} D _ {l} (D _ {k} - 1) (D _ {l} - 1)}{(2 | E | - 1) (2 | E | - 3)} - (\mathbb {E} E (k, l)) ^ {2} \\ = \mathbb {E} E (k, l) + O \left(\frac {D _ {k} D _ {l} (D _ {k} + D _ {l})}{| E | ^ {2}}\right) = O \left(\frac {D _ {k} D _ {l}}{| E |}\right). \\ \end{array}
$$
$$
\operatorname {V a r} E (k, k) = \mathbb {E} \left(\sum_ {i = 1} ^ {D _ {k}} \sum_ {j = 1} ^ {D _ {k}} \mathbb {1} _ {i, j}\right) ^ {2} - (\mathbb {E} E (k, k)) ^ {2} = \mathbb {E} E (k, k) + O \left(\frac {D _ {k} ^ {3}}{| E | ^ {2}}\right) = O \left(\frac {D _ {k} ^ {3}}{| E |}\right).
$$
From Chebyshev's inequality, we get:
$$
\mathrm {P} \left(| E (k, l) - \mathbb {E} E (k, l) | > \varepsilon \mathbb {E} E (k, l)\right) = O \left(\frac {| E |}{D _ {k} D _ {l} \varepsilon^ {2}}\right) = O \left(\frac {1}{\bar {p} _ {\min} ^ {2} | E | \varepsilon^ {2}}\right) = o \left(\frac {1}{C ^ {2}}\right).
$$
Thus, with probability  $1 - o(1)$ ,  $\mathrm{P}\left(|E(k,l) - \mathbb{E}E(k,l)| < \varepsilon \mathbb{E}E(k,l)\right)$  for all pairs of classes. In this case,
$$
\mathrm {L I} = - \frac {\sum_ {k , l} p (k , l) \log (1 + O (\varepsilon))}{\sum_ {l} \bar {p} (k) \log \bar {p} (k)} = O (\varepsilon).
$$
C.2 Alternative definition
Recall that in Section 3 we define the label informativeness in the following general form:  $\mathrm{LI} := I(y_{\xi}, y_{\eta}) / H(y_{\xi})$ . Then, to define  $\mathrm{LI}_{edge}$ , we say that  $\xi$  and  $\eta$  are two endpoints of an edge sampled uniformly at random. Another possible variant is when we first sample a random node and then sample its random neighbor. The probability of an edge becomes
$$
\bar {p} (c _ {1}, c _ {2}) = \sum_ {(u, v) \in E} \frac {\mathbb {1} \left\{y _ {u} = c _ {1} , y _ {v} = c _ {2} \right\}}{n   d (u)}.
$$
In this case,  $H(y_{\xi})$  is the entropy of the distribution  $p(c)$ ,  $H(y_{\eta})$  is the entropy of  $\bar{p}(c)$ . Thus, we obtain:
$$
\mathrm {L I} _ {n o d e} = - \frac {\sum_ {c _ {1} , c _ {2}} \bar {p} (c _ {1} , c _ {2}) \log \frac {\bar {p} (c _ {1} , c _ {2})}{p (c _ {1}) \bar {p} (c _ {2})}}{\sum_ {c} p (c) \log p (c)}.
$$
In this paper, we mainly focus on  $\mathrm{LI}_{edge}$  and refer to it as LI for brevity. First, this measure is conceptually similar to adjusted homophily discussed above: they both give equal weights to all edges. Second, in our analysis of real datasets, we do not notice a substantial difference between  $\mathrm{LI}_{edge}$  and  $\mathrm{LI}_{node}$  in most of the cases, see Table 5. However, these measures can potentially be different, especially for graphs with very unbalanced degree distributions (in  $\mathrm{LI}_{node}$ , averaging is over the nodes, so all nodes are weighted equally, while in  $\mathrm{LI}_{edge}$ , averaging is over the edges, so more weight is given to high-degree nodes). The choice between  $\mathrm{LI}_{edge}$  and  $\mathrm{LI}_{node}$  may depend on a particular application.
D Additional related work