# Measuring the Inferential Values of Relations in Knowledge Graphs
## 5 Experimental
### Dataset Construction  
**Dataset.** This work conducts the triple classification experiments on three commonly used knowledge graph benchmark datasets: FB15K-237, NELL-995, and YAGO3-10 [39]. They are common knowledge graph datasets that are applied to knowledge graph embedded learning tasks. FB15K-237 is extracted from the Freebase knowledge graph project, eliminating redundant relations and the type of relation is relatively basic, focusing on direct relations between entities. YAGO3-10 is a subset of the dataset YAGO3, covering parts of Wikipedia, Wikidata, and WordNet, and contains the entities associated with at least ten different relations. The relations are richer than FB15K-237, especially in the academic field and in the field of time and historical events, and are suitable for more complex reasoning tasks. NELL-995 is derived from the dataset \"The Never Ending Language Learner (NELL) System\", representing a subset of NELL's 95 th iteration. The entities in NELL-995 are more diverse, with relations related to Internet information and social behavior being more common, covering multiple domains of knowledge automatically learned from the Internet. The statistics of the dataset are shown in Table 2.  
**Construction of Information Table.** It must be mentioned that since our study is the first exploration about the inferential value measurement of relations in knowledge graphs, there is still no specialized dataset released on this research, as of the time of our submission. Therefore, the dataset of knowledge graph inference for experiments has to be built on the generalized datasets. Considering the logical inference chains in the datasets of knowledge graphs are usually implicit and require manual annotations, which is costly, this work utilizes the shortest paths as the logical inference chains to construct the logical relation information table. The attribute values in the information table represent the frequencies of occurrence of relations on the shortest paths, i.e., the logical inference chains. Based on this, the relation entropy is calculated. Generally, there are a large number of entities in a knowledge graph, and computing the shortest path between every pair of entities is very time-consuming, especially for the large-scale knowledge graphs. Fortunately, as long as there are enough logical inference chains and they are diverse enough, RIMIE can work effectively. Therefore, we only need to extract a sufficient number of shortest paths, rather than the shortest paths between all entities, to construct the information table.  
**Relation Combinations.** The experiment of triple classification on LRIT is designed to verify the impact of different relations on the completeness of the knowledge graph. This work mainly compares the inferential value of three different sets of relations:  
* Relation-Entropy: the top@\(k\) relations selected based on relation entropy.
* Relation-Max: the top@\(k\) relations with the highest occurrences in the knowledge graph.
* Relation-Random: the \(k\) relations selected randomly.  
**Construction of Test Set.** The triple classification experiment also requires constructing the test set, denoted as \(F_{test}\), containing the logical inference chains. Each sample in the test set is a logical inference chain. Subsequently, the triplets are sampled from these logical inference chains, and a given number of negative examples are constructed for each triplet for use in the classification experiment.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Dataset** & **Entity** & **Relation** & **Train** & **Valid** & **Test** \\ \hline FB15K-237 & 14,541 & 237 & 272,115 & 17,535 & 20,466 \\ NELL-995 & 75,492 & 200 & 149,678 & 543 & 3992 \\ YAGO3-10 & 123,182 & 37 & 1,079,040 & 5000 & 5000 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The dataset statistics, in which the numbers of entities and relations and the volumes of training, validation, and test sets, are listed, respectively.