SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4 New Evaluation Setting
In this section, we introduce a new setting for evaluating link prediction methods. We first discuss the unrealistic nature of the current evaluation setting in Section 4.1. Based on this, we present our new evaluation setting in Section 4.2, which aims to align better with real-world scenarios. Lastly, in Section 4.3, we present and discuss the results based on our new evaluation setting.
4.1 Issues with the Existing Evaluation Setting
The existing evaluation procedure for link prediction is to rank a positive sample against a set of  $K$  randomly selected negative samples. The same set of  $K$  negatives are used for all positive samples
$^{2}$ https://github.com/melifluos/subgraph-sketching
<sup>3</sup>https://github.com/seongjunyun/Neo-GNNs
(a) Negative sample generation for one positive sample.
(b) Process of determining negative samples that contain a node  $a$
Figure 2: Pipeline for generating the hard negative samples for a positive sample (a, b).
(with the exception of ogbl-citation2 which uses 1000 per positive sample). We demonstrate that there are multiple issues with this setting, making it difficult to properly evaluate the effectiveness of current models.
Issue 1: Non-Personalized Negative Samples. The existing evaluation setting uses the same set of negative samples for all positive samples (outside of ogbl-citation2). This strategy, referred to as global negative sampling [37], is not a commonly sought objective. Rather, we are often more interested in predicting links that will occur for a specific node. Take, for example, a social network that connects users who are friends. In this scenario, we may be interested in recommending new friends to a user  $u$ . This requires learning a classifier  $f$  that assigns a probability to a link existing. When evaluating this task, we want to rank links where  $u$  connects to an existing friend above those where they don't. For example, if  $u$  is friends with  $a$  but not  $b$ , we hope that  $f(u, a) > f(u, b)$ . However, the existing evaluation setting doesn't explicitly test for this. Rather it compares a true sample  $(u, a)$  with a potentially unrelated negative sample, e.g.,  $(c, d)$ . This is not aligned with the real-world usage of link prediction on such graphs.
Issue 2: Easy Negative Samples. The existing evaluation setting randomly selects negative samples to use. However given the large size of most graphs (see Table 7 in Appendix C), randomly sampled negatives are likely to choose two nodes that bear no relationship to each other. Such node pairs are trivial to classify. We demonstrate this by plotting the distribution of common neighbors (CN), a strong heuristic, for all positive and negative test samples in Figure 1. Almost all the negative samples contain no CNs, making them easy to classify. We further show that the same problem afflicts even the smaller datasets in Figure 3 in Appendix A.
These observations suggest that a more realistic evaluation strategy is desired. At the core of this challenge is which negative samples to use during evaluation. We discuss our design for solving this in the next subsection.
4.2 Heuristic Related Sampling Technique (HeaRT)
In this subsection, we introduce new strategy for evaluating link prediction methods. To address the concerns outlined in Section 4.1, we design a new method for sampling negatives during evaluation. Our strategy, HeaRT, solves these challenges by: (a) personalizing the negatives to each sample and (b) using heuristics to select hard negative samples. This allows for the negative samples to be directly related to each positive sample while also being non-trivial. We further discuss how to ensure that the negative samples are both personalized and non-trivial for a specific positive sample.
From our discussion in Section 4.1, we are motivated in personalizing the negatives to each positive sample. Since the positive samples in the current datasets are node pairs, we seek to personalize the negatives to both nodes in the positive sample. Extending our example in Section 4.1, this is analogous to restricting the negatives to contain one of the two users from the original friendship pair. As such, for a positive sample  $(u,a)$ , the negative samples will belong to the set:
$$
S (u, a) = \{(u ^ {\prime}, a) \mid u ^ {\prime} \in \mathcal {V} \} \cup \{(u, a ^ {\prime}) \mid a ^ {\prime} \in \mathcal {V} \}, \tag {3}
$$
where  $\mathcal{V}$  is the set of nodes. This is similar to the setting used for knowledge graph completion (KGC) [38] which uses all such samples for evaluation. However, one drawback of evaluating each
Table 4: Results on Cora, CiteSeer, and Pubmed (%) under HeaRT. Highlighted are the results ranked first, second, and third.
Models	Cora	Citeseer	Pubmed
MRR	Hits@10	MRR	Hits@10	MRR	Hits@10
Heuristic	CN	9.78	20.11	8.42	18.68	2.28	4.78
AA	11.91	24.10	10.82	22.20	2.63	5.51
RA	11.81	24.48	10.84	22.86	2.47	4.9
Shortest Path	5.04	15.37	5.83	16.26	0.86	0.38
Katz	11.41	22.77	11.19	24.84	3.01	5.98
Embedding	Node2Vec	14.47 ± 0.60	32.77 ± 1.29	21.17 ± 1.01	45.82 ± 2.01	3.94 ± 0.24	8.51 ± 0.77
MF	6.20 ± 1.42	15.26 ± 3.39	7.80 ± 0.79	16.72 ± 1.99	4.46 ± 0.32	9.42 ± 0.87
MLP	13.52 ± 0.65	31.01 ± 1.71	22.62 ± 0.55	48.02 ± 1.79	6.41 ± 0.25	15.04 ± 0.67
GNN	GCN	16.61 ± 0.30	36.26 ± 1.14	21.09 ± 0.88	47.23 ± 1.88	7.13 ± 0.27	15.22 ± 0.57
GAT	13.84 ± 0.68	32.89 ± 1.27	19.58 ± 0.84	45.30 ± 1.3	4.95 ± 0.14	9.99 ± 0.64
SAGE	14.74 ± 0.69	34.65 ± 1.47	21.09 ± 1.15	48.75 ± 1.85	9.40 ± 0.70	20.54 ± 1.40
GAE	18.32 ± 0.41	37.95 ± 1.24	25.25 ± 0.82	49.65 ± 1.48	5.27 ± 0.25	10.50 ± 0.46
GNN+Pairwise Info	SEAL	10.67 ± 3.46	24.27 ± 6.74	13.16 ± 1.66	27.37 ± 3.20	5.88 ± 0.53	12.47 ± 1.23
BUDDY	13.71 ± 0.59	30.40 ± 1.18	22.84 ± 0.36	48.35 ± 1.18	7.56 ± 0.18	16.78 ± 0.53
Neo-GNN	13.95 ± 0.39	31.27 ± 0.72	17.34 ± 0.84	41.74 ± 1.18	7.74 ± 0.30	17.88 ± 0.71
NCN	14.66 ± 0.95	35.14 ± 1.04	28.65 ± 1.21	53.41 ± 1.46	5.84 ± 0.22	13.22 ± 0.56
NCNC	14.98 ± 1.00	36.70 ± 1.57	24.10 ± 0.65	53.72 ± 0.97	8.58 ± 0.59	18.81 ± 1.16
NBFNet	13.56 ± 0.58	31.12 ± 0.75	14.29 ± 0.80	31.39 ± 1.34	>24h	>24h
PEG	15.73 ± 0.39	36.03 ± 0.75	21.01 ± 0.77	45.56 ± 1.38	4.4 ± 0.41	8.70 ± 1.26
positive sample against the entire set of possible corruptions is the high computational cost. To mitigate this issue we consider only utilizing a small subset of  $S(u, a)$  during evaluation.
The key challenge is how to generate a subset of  $S(u, a)$ . If we randomly sample from  $S(u, a)$ , we risk only utilizing easy negative samples. This is one of the issues of the existing evaluation setting (see Issue 2 in Section 4.1), whereby randomly selecting negatives, they unknowingly produce negative samples that are too easy. We address this by selecting the negative samples via a combination of multiple heuristics. Since heuristics typically correlate well with performance, we ensure that the negative samples will be non-trivial to classify. This is similar to the concept of candidate generation [39, 40], which only ranks a subset of candidates that are most likely to be true.
An overview of the generation process is given in Figure 2. For each positive sample, we generate  $K$  negative samples. To allow personalization to both nodes in the positive sample equally, we sample  $K/2$  negatives with each node. For the heuristics, we consider RA [29], PPR [41], and feature similarity. A more detailed discussion on the negative sample generation is given in Appendix G. It's important to note that our work centers specifically on negative sampling during the evaluation stage (validation and test). This is distinct from prior work that concerns the negatives sampled used during the training phase [42, 43]. As such, the training process remains unaffected under both the existing and HeaRT setting.
In this subsection we present our results when utilizing HeaRT. We follow the parameter ranges introduced in Section 2.3. For all datasets we use  $K = 500$  negative samples per positive sample during evaluation. Furthermore for ogbl-ppa we only use a small subset of the validation and test positive samples (100K each) for evaluation. This is because the large size of the validation and test sets (see Table 7 in Appendix C) makes HeaRT prohibitively expensive.
The results are shown in Table 4 (Cora, CiteSeer, Pubmed) and Table 5 (OGB). For simplicity, we only include the MRR and Hits@10 for Cora, CiteSeer, Pubmed, and the MRR and Hits@20 for OGB. Additional results for other metrics can be found in Appendix I. We note that most datasets, outside of ogbl-ppa, exhibit much lower performance than under the existing setting. This is though we typically use much fewer negative samples in the new setting, implying that the negative samples produced by HeaRT are much harder. We highlight the main observations below.
**Observation 1:** Better Performance of Simple Models. We find that under HeaRT, "simple" baseline models (i.e., heuristic, embedding, and GNN methods) show a greater propensity to outperform their counterparts via ranking metrics than under the existing setting. Specifically, we focus on MRR in Table 1, 4, and 5, and the corresponding ranking-based metrics in Table 2. Under the existing
Table 5: Results on OGB datasets (%) under HeaRT. Highlighted are the results ranked first, second, and third.
Models	ogbl-collab	ogbl-ddi	ogbl-ppa	ogbl-citation2
MRR	Hits@20	MRR	Hits@20	MRR	Hits@20	MRR	Hits@20
CN	12.60	27.51	6.71	38.69	25.70	68.25	17.11	41.73
AA	16.40	32.65	6.97	39.75	26.85	70.22	17.83	43.12
RA	28.14	41.16	8.70	44.01	28.34	71.50	17.79	43.34
Shortest Path	46.71	46.56	0	0	0.54	1.31	>24h	>24h
Katz	47.15	48.66	6.71	38.69	25.70	68.25	14.10	35.55
Node2Vec	12.10 ± 0.20	25.85 ± 0.21	11.14 ± 0.95	63.63 ± 2.05	18.33 ± 0.10	53.42 ± 0.11	14.67 ± 0.18	42.68 ± 0.20
MF	26.86 ± 1.74	38.44 ± 0.07	13.99 ± 0.47	59.50 ± 1.68	22.47 ± 1.53	70.71 ± 4.82	8.72 ± 2.60	29.64 ± 7.30
MLP	12.61 ± 0.66	23.05 ± 0.89	N/A	N/A	0.98 ± 0.00	1.47 ± 0.00	16.32 ± 0.07	43.15 ± 0.10
GCN	18.28 ± 0.84	32.90 ± 0.66	13.46 ± 0.34	64.76 ± 1.45	26.94 ± 0.48	68.38 ± 0.73	19.98 ± 0.35	51.72 ± 0.46
GAT	10.97 ± 1.16	29.58 ± 2.42	12.92 ± 0.39	66.83 ± 2.23	OOM	OOM	OOM	OOM
SAGE	20.89 ± 1.06	33.83 ± 0.93	12.60 ± 0.72	67.19 ± 1.18	27.27 ± 0.30	69.49 ± 0.43	22.05 ± 0.12	53.13 ± 0.15
GAE	OOM	OOM	3.49 ± 1.73	17.81 ± 9.80	OOM	OOM	OOM	OOM
SEAL	22.53 ± 3.51	36.48 ± 2.55	9.99 ± 0.90	49.74 ± 2.39	29.71 ± 0.71	76.77 ± 0.94	20.60 ± 1.28	48.62 ± 1.93
BUDDY	32.42 ± 1.88	45.62 ± 0.52	12.43 ± 0.50	58.71 ± 1.63	27.70 ± 0.33	71.50 ± 0.68	19.17 ± 0.20	47.81 ± 0.37
Neo-GNN	21.90 ± 0.65	38.40 ± 0.29	10.86 ± 2.16	51.94 ± 10.33	21.68 ± 1.14	64.81 ± 2.26	16.12 ± 0.25	43.17 ± 0.53
NCN	17.51 ± 2.50	37.07 ± 2.97	12.86 ± 0.78	65.82 ± 2.66	35.06 ± 0.26	81.89 ± 0.31	23.35 ± 0.28	53.76 ± 0.20
NCNC	19.02 ± 5.32	35.67 ± 6.78	>24h	>24h	33.52 ± 0.26	82.24 ± 0.40	19.61 ± 0.54	51.69 ± 1.48
NBFNet	OOM	OOM	>24h	>24h	OOM	OOM	OOM	OOM
PEG	15.68 ± 1.10	29.74 ± 0.95	12.05 ± 1.14	50.12 ± 6.55	OOM	OOM	OOM	OOM
setting, such methods only rank in the top three for any dataset a total of 5 times. However, under HeaRT this occurs 10 times. Furthermore, under the existing setting only 1 "simple" method ranks best overall while under HeaRT there are 4. This suggests that recent advanced methods may have benefited from the easier negative samples in the existing setting.
Another interesting observation is that on ogbl-collab, heuristic methods are able to outperform more complicated models by a large margin. Specifically, we find that Katz is the best ranked method, Shortest Path the second, and RA the fourth. Furthermore, the MRR gap between the second ranked method (Shortest Path) and the third (BUDDY) is very large at 14.29 points. We observe that this result is caused by the dynamic nature of the graph, where node pairs that are linked in the training data may also be present as positive samples in the test. We further expound on this observation in Appendix H.
**Observation 2: Lower Model Standard Deviation.** We observed earlier that, under the existing evaluation setting, the model variance across seeds was high (see observation 3 in Section 3). This complicates model comparison as the model performance is unreliable. Interestingly, we find that HeaRT is able to dramatically reduce the variance for all datasets. We demonstrate this by first calculating the mean standard deviation across all models on each individual dataset. This was done for both evaluation settings with the results compared. As demonstrated in Table 6, the mean standard deviation decreases for all datasets. This is especially true for Cora, CiteSeer, and Pubmed, which each decrease by over  $85\%$ . Such a large decrease in standard deviation is noteworthy as it allows for a more trustworthy and reliable comparison between methods.
We posit that this observation is caused by a stronger alignment between the positive and negative samples under our new evaluation setting. Under the existing evaluation setting, the same set of negative samples is used for all positive samples. One consequence of this is that a single positive sample may bear little to no relationship to the negative samples (see Section 4.1 for more discussion). However, under our new evaluation setting, the negatives for a positive sample are a subset of the corruptions of that sample. This allows for a more natural comparison via ranking-based metrics as the samples are more related and can be more easily compared.
Table 6: Mean model standard deviation for the existing setting and HeaRT. We use Hits@20 for ogbl-ddi, Hits@50 for ogbl-collab, Hits@100 for ogbl-ppa, and MRR otherwise.
Dataset	Existing	HeaRT	% Change
Cora	5.19	0.79	-85%
CiteSeer	5.94	0.88	-85%
Pubmed	4.14	0.35	-92%
ogbl-collab	1.49	0.96	-36%
ogbl-ppa	2.13	0.36	-83%
ogbl-ddi	6.77	3.49	-48%
ogbl-citation2	1.39	0.59	-58%
Observation 3: Lower Model Performance. We observe that the majority of datasets exhibit a significantly reduced performance in comparison to the existing setting. For example, under the existing setting, models typically achieve a MRR of around 30, 50, and 30 on Cora, Citeseer, and Pubmed (Table 1), respectively. However, under HeaRT the MRR for those datasets is typically around 20, 25, and 10 (Table 4). Furthermore for ogbl-citation2, the MRR of the best performing model falls from a shade under 90 on the existing setting to slightly over 20 on HeaRT. Lastly, we note that the performance on ogbl-ppa actually increases. This is because we only utilize a small subset of the total test set when evaluating on HeaRT, nullifying any comparison between the two settings.
These outcomes are observed despite HeaRT using much fewer negative samples than the original setting. This suggests that the negative samples generated by HeaRT are substantially more challenging than those used in the existing setting. This underscores the need to develop more advanced methodologies that can tackle harder negatives samples like in HeaRT.
In this work we have revealed several pitfalls found in recent works on link prediction. To overcome these pitfalls, we first establish a benchmarking that facilitates a fair and consistent evaluation across a diverse set of models and datasets. By doing so, we are able to make several illuminating observations about the performance and characteristics of various models. Furthermore, based on several limitations we observed in the existing evaluation procedure, we introduce a more practical setting called HeaRT (Heuristic Related Sampling Technique). HeaRT incorporates a more real-world evaluation setting, resulting in a better comparison among methods. By introducing a more rigorous and realistic assessment, HeaRT could guide the field towards more effective models, thereby advancing the state of the art in link prediction.
This research is supported by the National Science Foundation (NSF) under grant numbers CNS 2246050, IIS1845081, IIS2212032, IIS2212144, IOS2107215, DUE 2234015, III-2212145, III-2153326, DRL2025244 and IOS2035472, the Army Research Office (ARO) under grant number W911NF-21-1-0198, the Home Depot, Cisco Systems Inc, Amazon Faculty Award, Johnson&Johnson, JP Morgan Faculty Award and SNAP.
[1] Linyuan Lu and Tao Zhou. Link prediction in complex networks: A survey. Physica A: statistical mechanics and its applications, 390(6):1150-1170, 2011.
[2] Nur Nasuha Daud, Siti Hafizah Ab Hamid, Muntadher Saadoon, Firdaus Sahran, and Nor Badrul Anuar. Applications of link prediction in social networks: A review. Journal of Network and Computer Applications, 166:102716, 2020.
[3] István A Kovács, Katja Luck, Kerstin Spirohn, Yang Wang, Carl Pollis, Sadie Schlabach, Wenting Bian, Dae-Kyum Kim, Nishka Kishore, Tong Hao, et al. Network-based prediction of protein interactions. Nature communications, 10(1):1240, 2019.
[4] Zan Huang, Xin Li, and Hsinchun Chen. Link prediction approach to collaborative filtering. In Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, pages 141-142, 2005.
[5] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 32(1):4-24, 2020.
[6] Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Gunnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018.
[43] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pages 452-461, 2009.
[44] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[45] JC de Borda. M'emoire sur les' elections au scrutin. Histoire de l'Académie Royale des Sciences, 1781.
7 Checklist
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Appendix K.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Appendix L.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] See https://github.com/Juanhui28/HeaRT.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Yes. See Section 2.3 and Appendix C.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix C.2.
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] See Appendix J.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] See https://github.com/Juanhui28/HeaRT.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]