SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We list the statistics of the real-world knowledge graph datasets in Table 6 and the synthetic Kinship dataset in Table 7. We present detailed descriptions for each dataset below.
CodeX. The CodeX dataset, recently proposed for knowledge graph completion tasks, is a comprehensive collection extracted from both Wikidata and Wikipedia. This challenging dataset comes in three versions: small (S), medium (M), and large (L), allowing for comprehensive evaluation.
YAGO3-10. YAGO3-10 is a subset of YAGO3 (Suchanek et al., 2007), a large knowledge base completion dataset, with the majority of triples describing attributes of persons, including their citizenship, gender, and profession.
WN18. WordNet 18 (WN18) dataset is one of the most commonly used subsets of WordNet.
WN18RR. WN18RR is a modified version of WN18 designed to be more challenging for knowledge graph reasoning algorithms by removing reverse relations in the knowledge graph.
Kinship. A synthetic dataset, widely used (Zhang et al., 2020; Fang et al., 2023) for evaluating the statistical relational learning ability and the scalability of reasoning algorithms. We use five different sizes of the dataset for evaluating its run time efficiency and parameter scalability, namely Kinship-S1/S2/S3/S4/S5, respectively.
Table 6: Statistics of real-world knowledge base datasets.
Dataset	#Ent	#Rel	#Train/Valid/Test	#Rules
CodeX-s	2,034	42	32,888/1,827/1,828	35
CodeX-m	17,050	51	185,584/10,310/10,311	52
CodeX-1	77,951	69	551,193/30,622/30,622	57
YAGO3-10	123,182	37	1,079,040/5,000/5,000	22
WN18	40,943	18	141,442/ 5,000/ 5,000	140
WN18RR	40,943	11	86,835/ 3,034/ 3,134	51
Table 7: Statistics for Kinship datasets of varied sizes (S1-S5).
S1	S2	S3	S4	S5
Number of rules containing 1 premise atom	12	12	12	12	12
Number of rules containing 2 premise atoms	9	9	9	9	9
Number of predicates	15	15	15	15	15
Number of entities	52	106	158	202	267
B.2 Probabilistic logic reasoning on Kinship Dataset
We assess performance on the Kinship dataset across five different sizes. Due to the full confidence of rules, we only perform inference in this experiment and do not need to update weights. We include
Table 8: Comparative evaluation of reasoning performance on the Kinship dataset.
Algorithms	Ground iteration	AUC-ROC
S1	S2	S3	S4	S5
PSL	-	.976±.011	.980±.005	.991±.003	.982±.005	.972±.004
ExpressGNN	-	.957±.002	.921±.001	.959±.004	.940±.001	.989±.004
1	.841±.005	.895±.001	.922±.001	.901±.001	.903±.000
DiffLogic-RotatE	2	.931±.005	.994±.001	.998±.001	.985±.001	.993±.001
3	.937±.005	.987±.001	.995±.001	.978±.001	.989±.001
1	.567±.099	.537±.041	.507±.024	.503±.018	.504±.014
DiffLogic-MLP	2	.956±.032	.997±.002	.999±.003	.999±.001	.999±.000
3	.982±.014	.997±.001	.999±.001	.999±.000	.999±.000
DiffLogic using two different embedding models, i.e., RotatE and MLP, and evaluate their reasoning performance using RGIG with varied iterations (i.e., 1, 2, 3) for grounding. We include PSL and ExpressGNN as baselines, but we exclude pLogicNet due to its inability to utilize handcrafted rules. Given that the Kinship dataset lacks a validation set, we run each model ten times and report the AUC-ROC statistics from the final epoch of each run on the test set. The results are presented in Table 8, with the best results shown in bold.
B.3 Comparing inference time on Kinship
We evaluate the inference time on the Kinship dataset across five different sizes. We include models in Appendix B.2 for this experiment. For two DiffLogic variants, we only evaluate their inference time when using 3 iterations of RGIG for grounding. All the runtime experiments are conducted in the same machine with configurations as in Table 9. All of these models are implemented in Python, thereby ensuring a fair comparison. The inference time results are displayed in Table 10, with the best results shown in bold.
Table 9: Machine configuration.
Component	Specification
GPU	NVIDIA GeForce RTX 3090
CPU	Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz
Table 10: Comparison of runtime of inference on Kinship.
Algorithms	Grounding iteration	Runtime
S1	S2	S3	S4	S5
PSL	-	~3.6min	~7.9min	~12.9min	~13.5min	~32min
ExpressGNN	-	~18.4min	~19.1min	~18.9min	~19.4min	~20.2min
DiffLogic-RotateE	3	37s	~1.5min	~3.2min	~3.6min	~4min
DiffLogic-MLP	3	21.8s	41.5s	45s	54.4s	~1.2min