SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In the aspect of characterizing graph datasets, our work is conceptually similar to a recent paper by Liu et al. [21]. In this paper, the authors empirically analyze what aspects of a graph dataset (e.g., node features or graph structure) influence the GNN performance. Liu et al. [21] follow a data-driven approach and measure the performance change caused by several data perturbations. In contrast, we follow a theoretical approach to choosing graph characteristics based only on the label-structure relation. Additionally, our characteristics are very simple, model agnostic, and can be used for general-purpose graph analysis (beyond graph ML). Similarly to [21], we believe that the proposed characteristics can help in the selection and development of diverse future graph benchmarks.
Similarly to our work, several papers note that homophily does not always reflect the simplicity of a dataset for GNNs, and standard GNNs can work well on some heterophilous graphs. To address this problem, Luan et al. [22] propose a metric called aggregation homophily that takes into account both graph structure and input features. We note that the aggregation homophily is based on a particular aggregation scheme of the GCN model. In contrast, the proposed LI is a simple and intuitive model-agnostic measure. An additional advantage of LI is that it is provably unbiased and can be compared across datasets with different numbers of classes and class size balance. Another feature-based metric was very recently proposed by Luan et al. [23]. It is called Kernel Performance Metric (KPM) and reflects how beneficial is a graph structure for prediction. Ma et al. [24] also observe that some heterophilous graphs are easy for GNNs. To analyze this problem, they propose a measure named cross-class neighborhood similarity defined for pairs of classes. While this measure is an informative tool to analyze a particular dataset, it does not give a single number to easily compare different datasets.
Finally, let us note that Suresh et al. [41] use (2) as a measure of graph assortativity (homophily). The authors show that the homophily level varies over the graph, and the prediction performance of GNNs correlates with the local homophily. They use these insights to transform the input graph and
get an enhanced level of homophily. In future work, it would be interesting to see whether additional benefits can be obtained using label informativeness instead of homophily.
E Characterizing real graph datasets
Cora, citeSeer, and pubmed [6, 25, 38, 27, 46] are three classic paper citation network benchmarks. For cora and citeSeer labels correspond to paper topics, while for pubmed labels specify the type of diabetes addressed in the paper. Coauthor-cs and coauthor-physics [39] are co-authorship networks. Nodes represent authors, and two nodes are connected by an edge if the authors co-authored a paper. Node labels correspond to fields of study. Amazon-computers and amazon-photo [39] are co-purchasing networks. Nodes represent products, and an edge means that two products are frequently bought together. Labels correspond to product categories. Lastfm-asia is a social network of music streaming site LastFM users who live in Asia [35]. Edges represent follower relationships, and labels correspond to user's nationality. In the facebook [37] graph nodes correspond to official Facebook pages, and links indicate mutual likes. Labels represent site categories. In the github [37] graph, nodes represent GitHub users and edges represent follower relationships. A binary label indicates that a user is either a web or a machine learning developer. Ogbn-arxiv and Ogbn-products [14] are two datasets from the recently proposed Open Graph Benchmark. Ogbn-arxiv is a citation network graph with labels corresponding to subject areas. ogbn-products is a co-purchasing network with labels corresponding to product categories. Actor [42, 32] is a popular dataset for node classification in heterophilous graphs. The nodes correspond to actors and edges represent co-occurrence on the same Wikipedia page. The labels are based on words from an actor's Wikipedia page. Flickr [47] is a graph of images with labels corresponding to image types. Deezer-europe [35] is a user network of the music streaming service Deezer with labels representing a user's gender. Twitch-de and twitch-pt [37] are social network graphs of gamers from the streaming service Twitch.<sup>7</sup> The labels indicate if a streamer uses explicit language. Genius [19], twitch-gamers [36], arxiv-year [14], snap-patents [18], and wiki [20] are recently proposed large-scale heterophilous datasets. For the wiki dataset we remove all isolated nodes. Roman-emperor, amazon-ratings, minesweeper, workers, and questions [33] are recently proposed mid-scale heterophilous datasets. We additionally construct one more binary classification graph — twitter-hate. This graph is based on data from Hateful Users on Twitter dataset on Kaggle.<sup>8</sup> The labels indicate if a user posts hateful messages or not. We remove all the unlabeled nodes from the graph and use the largest connected component of the resulting graph.
We transform all the considered graphs to simple undirected graphs and remove self-loops.
E.2 Dataset characteristics
Dataset characteristics are shown in Table 5. This table extends Table 2 in the main text. It can be seen that the typically used homophily measures —  $h_{edge}$  and  $h_{node}$  — often overestimate homophily levels, since they do not take into account the number of classes and class size balance. This is particularly noticeable for datasets with two classes. If fact, according to these measures all binary classification datasets in our table are homophilous. In contrast,  $h_{adj}$  corrects for the expected number of edges between classes and shows that most of the considered binary classification datasets are actually heterophilous (github and twitter-hate being the exceptions).
As for label informativeness, it can be seen that on real heterophilous datasets it is typically very low (with the exception of roman-empire dataset). This is in contrast to synthetic datasets used for experiments in [22, 24], which sometimes exhibit a combination of low homophily and high label informativeness. High label informativeness of these datasets leads to strong GNN performance on them despite low homophily levels.
Let us note that edge and node label informativeness differ in how they weight high/low-degree nodes. For node label informativeness, averaging is over the nodes, so all nodes are weighted equally. For
${}^{7}$  Twitch-de and twitch-pt are subgraphs of a larger dataset twitch-gamers. We report characteristics for all of them since they have different sizes, edge density, and may have different structural properties.
Table 5: Dataset characteristics, more homophilous datasets are above the line
Dataset	n	|E|	C	hedge	hnode	hclass	hadj	LIedge	LInode
cora	2708	5278	7	0.81	0.83	0.77	0.77	0.59	0.61
citeSeer	3327	4552	6	0.74	0.72	0.63	0.67	0.45	0.45
pubmed	19717	44324	3	0.80	0.79	0.66	0.69	0.41	0.40
coauthor-cs	18333	81894	15	0.81	0.83	0.75	0.78	0.65	0.68
coauthor-physics	34493	247962	5	0.93	0.92	0.85	0.87	0.72	0.76
amazon-computers	13752	245861	10	0.78	0.80	0.70	0.68	0.53	0.62
amazon-photo	7650	119081	8	0.83	0.85	0.77	0.79	0.67	0.72
lastfm-asia	7624	27806	18	0.87	0.83	0.77	0.86	0.74	0.68
facebook	22470	170823	4	0.89	0.88	0.82	0.82	0.62	0.74
github	37700	289003	2	0.85	0.80	0.38	0.38	0.13	0.15
twitter-hate	2700	11934	2	0.78	0.67	0.50	0.55	0.23	0.51
ogbn-arxiv	169343	1157799	40	0.65	0.64	0.42	0.59	0.45	0.53
ogbn-products	2449029	61859012	47	0.81	0.83	0.46	0.79	0.68	0.72
actor	7600	26659	5	0.22	0.22	0.01	0.00	0.00	0.00
flickr	89250	449878	7	0.32	0.32	0.07	0.09	0.01	0.01
deezer-europe	28281	92752	2	0.53	0.53	0.03	0.03	0.00	0.00
twitch-de	9498	153138	2	0.63	0.60	0.14	0.14	0.02	0.03
twitch-pt	1912	31299	2	0.57	0.59	0.12	0.11	0.01	0.02
twitch-gamers	168114	6797557	2	0.55	0.56	0.09	0.09	0.01	0.02
genius	421961	922868	2	0.59	0.51	0.02	-0.05	0.00	0.17
arxiv-year	169343	1157799	5	0.22	0.29	0.07	0.01	0.04	0.12
snap-patents	2923922	13972547	5	0.22	0.21	0.04	0.00	0.02	0.00
wiki	1770981	242605360	5	0.38	0.28	0.17	0.15	0.06	0.04
roman-empire	22662	32927	18	0.05	0.05	0.02	-0.05	0.11	0.11
amazon-ratings	24492	93050	5	0.38	0.38	0.13	0.14	0.04	0.04
minesweeper	10000	39402	2	0.68	0.68	0.01	0.01	0.00	0.00
workers	11758	519000	2	0.59	0.63	0.18	0.09	0.01	0.02
questions	48921	153540	2	0.84	0.90	0.08	0.02	0.00	0.01
Figure 3: Dependence of GraphSAGE accuracy on homophily and label informativeness for synthetic SBM graphs
edge label informativeness, averaging is over the edges, which implies that high-degree nodes make a larger contribution to the final measure. It is natural to expect that for high-degree nodes the amount of information from each individual neighbor is lower than for low-degree nodes since neighbors of high-degree nodes are expected to be more diverse and closer to the "average" distribution. Thus, it is natural to expect edge label informativeness to often be smaller than node label informativeness, which agrees with Table 5. Note that for most of the real datasets, the values of these two measures are rather close, with twitter-hate, genius, and arxiv-year being the exceptions.
F Correlation of LI with GNN performance
F.1 Experimental setup
For our experiments on synthetic data, we select 208 combinations of homophily and LI. For each combination, we generate 10 random graphs with the corresponding homophily and LI values using the SBM-based model described in Section 4.2. Each graph has 1000 nodes (250 nodes for each class) and the expected node degree of 10. Node features are taken from the four largest classes in the cora dataset (each of these four classes is mapped to one class in the synthetic data, and node features are sampled randomly from the corresponding class). For each synthetic graph, we create 10 random  $50\% / 25\% / 25\%$  train/validation/test splits. Thus, for each model, we make 10 runs per graph or 100 runs per homophily/LI combination, totaling 20800 runs.
We use GCN [16], GraphSAGE [10], GAT [43] and Graph Transformer (GT) [40] as representative GNN architectures for our experiments. For GraphSAGE, we use the version with the mean aggregation function and do not use the node sampling technique used in the original paper. For all models, we add a two-layer MLP after every graph neighborhood aggregation layer and further augment all models with skip connections [11], layer normalization [2], and GELU activation functions [12]. For all models, we use two graph neighborhood aggregation layers and hidden dimension of 512. We use Adam [15] optimizer with a learning rate of  $3 \cdot 10^{-5}$  and train for 1000 steps, selecting the best step based on the validation set performance. We use a dropout probability of 0.2 during training. Our models are implemented using PyTorch [31] and DGL [44].
F.2 Synthetic data based on SBM model
In Figure 2a of the main text, we show the results for GraphSAGE. In Figure 3, we additionally plot the dependence of GraphSAGE accuracy on homophily and label informativeness. It can be seen that the model's accuracy is much more correlated with label informativeness than with homophily. The results for GCN, GAT, and GT are presented in Figures 4, 5, and 6, respectively. As can be seen, the performance of all models generally follows the same pattern, and LI is a better predictor of model performance than homophily. In particular, when LI is high, all models achieve high accuracy even if homophily is negative. Recall that we provide Spearman correlation coefficients between model accuracy and adjusted homophily or LI in Table 3.
F.3 Semi-synthetic data from [24]
We use the same experimental setting described above for training GCN, GraphSAGE, GAT, and GT on modifications of the cora and citeseer graphs from [24]. The results for GraphSAGE and cora are provided in Figure 2b of the main text. The results for GraphSAGE and citeseer are shown in Figure 7.
F.4 Synthetic data based on LFR model
We also experiment with the LFR random graph model, which is a well-known benchmark for community detection (we treat communities as node labels). We generated 39 LFR graphs with the mixing parameter values evenly spaced between 0.025 and 0.975. Each graph has 1000 nodes and 5 communities. The maximum community size is 350, and the minimum community size is 100. The average node degree is 10, and the maximum node degree is 50. Note that homophily monotonically changes with the mixing parameter, while LI is  $U$ -shaped. Similarly to our experiments with SBM, we measure the Spearman correlation coefficient between accuracy and LI /  $h_{adj}$ , see Table 6. We see that LI is a better predictor of GNN performance than homophily.
Table 6: Spearman correlation between model accuracy and characteristics of LFR graphs
Model	hadj	LIedge
GCN	0.79	0.96
GraphSAGE	0.68	0.99
GAT	0.77	0.97
GT	0.77	0.97
Figure 4: Accuracy of GCN on synthetic SBM graphs
Figure 5: Accuracy of GAT on synthetic SBM graphs
Figure 6: Accuracy of GT on synthetic SBM graphs
Figure 7: Accuracy of GraphSAGE on semi-synthetic citeseer graphs from [24]
F.5 Synthetic data from [22]
Luan et al. [22] have shown that GNNs can achieve strong performance on certain heterophilous graphs. Again, this phenomenon can be explained by the high label informativeness of the heterophilous graphs used for these experiments.
The authors investigate how different levels of homophily affect GNN performance. They find that the curve showing the dependence of GNN performance on edge homophily (as well as on node homophily) is  $U$ -shaped: GNNs show strong results not only when edge homophily is high, but also when edge homophily is very low. Our label informativeness explains this behavior. We use the same data generating process as in [22] and find that the curve of label informativeness depending on edge homophily is also  $U$ -shaped (see Figure 8). Thus, on the datasets from [22], GNNs perform well exactly when label informativeness is high regardless of edge homophily. The  $U$ -shape of the label informativeness curve is not surprising: when edge homophily is very low, knowing that a node has a neighbor of a certain class provides us with information that this node probably does not belong to this class.
Figure 8: Label informativeness depending on edge homophily on synthetic graphs from [22]
Figure 9: Performance of GAT and GraphSAGE on real graphs vs  $\mathrm{LI}_{edge}$
Table 7: Spearman correlation coefficients between ROC AUC score and characteristics of real-world datasets from Table 5
Model	hedge	hnode	hclass	hadj	LIedge	LInode
GraphSAGE	0.64	0.62	0.67	0.61	0.78	0.82
GAT	0.69	0.65	0.74	0.69	0.85	0.84
F.6 Correlation of GNN performance with homophily and LI on real datasets
Analyzing which measures better agree with GNN performance on real data is non-trivial. Indeed, if we consider diverse node classification problems, there are many factors affecting the GNN performance: the size of the datasets and fraction of training examples, the edge density, the number of features and their informativeness, the number of classes and their balance, and so on. However, as requested by the reviewers, we analyze how well different homophily measures and LI correlate with GNN performance on the datasets in Table 5. In this experiment, we measure the ROC AUC score since some of the datasets are highly unbalanced, and thus accuracy is not a suitable performance measure. For the datasets with more than two classes, we report the macro-averaged ROC AUC score. The relation between ROC AUC and LI for GAT and GraphSAGE is shown in Figure 9. The Spearman correlation for all the measures is reported in Table 7. We see that the largest correlation is achieved by  $\mathrm{LI}_{edge}$  and  $\mathrm{LI}_{node}$ . Let us note that this table does not aim to compare homophily measures with each other since a good homophily measure is expected to properly capture the tendency of edges to connect similar nodes. Thus, we compare homophily measures using the proposed theoretical framework in Section 2.