# Measuring the Inferential Values of Relations in Knowledge Graphs
## 5 Experimental
### Experimental Configuration and Implementation  
**Triplet Classification.** This study employs three embedding models, TransE [40], ComplEx [41], and VLP [42], to obtain the vector representations of entities and relations of knowledge graphs. The introduction to the models is displayed in Appendix A. Three sets of relations are selected: Relation-Entropy, Relation-Max and Relation-Random, introduced above. A new training set is generated by removing the triplets containing these relations from the original training set. Correspondingly, four different training sets are generated: the original complete training set, and three training sets obtained by deleting the relations from each of the three selected groups.  
**Remark 3**: _We are actually aware that the triplet classification experiments on TransE/ComplEx, compared with the real-vold long-chain reasoning (such as crime investigating and suspense stories), are far from exciting. We still construct these experiments for two reasons, as follows. Firstly, as the classical task and model, respectively, although triplet classification and TransE/ComplEx are no longer so fashionable, they are still the simplest and most popular forms of knowledge reasoning and embedding, respectively. Their basic idea, especially that of TransE, has been embedded in almost all knowledge graph embedding tasks. Secondly, as of the time of our submission, there has not been any released long-chain reasoning task dataset to be used. Fortunately, it is known that scaling the triplet-based classification is the lowest common denominator in scaling more complex reasoning applications. Since any long-chain reasoning on knowledge graphs is composed of a series of triplets, the correct classification or not of each triplet is crucial for the whole reasoning process. To sum up, scaling the triplet classification experiments on TransE/ComplEx is therefore the lowest common denominator in scaling more complex knowledge reasoning. Our work provides a baseline for further research on more complex knowledge reasoning._  
**Implementation.** The logical relation information table is constructed, and the relation entropy \(r_{e}\) for each relation is calculated by using RMIE. This work needs to compare the effects of different relation combinations on the experimental results. Given a relation set \(R\), we remove the triplets containing relation \(r\in R\) from the original training set \(F\), and this results in a new training set \(F_{f_{R}}^{-}\). The knowledge graph embedding models are used to obtain the vector representations of entities and relations on the new training set \(F_{f_{R}}^{-}\). Then, the triplet classification experiments are performed by using the obtained entity and relation vectors on the test set \(F_{test}\). Finally, the impacts of different relation combinations are observed and compared on the triplet classification experiments. In this process, to ensure the credibility of results, the same test set is used for each group of experiments.  
**Evaluation Criteria.** To assess the experimental results, this work employs the classical evaluation metrics from classification tasks: Precision-Recall (PR) curve and Receiver Operating Characteristic (ROC) curve, as well as the area under the PR curve (PR AUC) and the area under the ROC curve (ROC AUC). PR and ROC curves not only assess the classification ability of the model but also demonstrate its generalization ability and the impact of different classification thresholds on the models. Therefore, PR and ROC curves are used as the evaluation metrics.  
**Experimental Environment.** All experiments are conducted on the PyTorch deep learning framework with a Linux cluster (CPU: Intel(R) Core(TM) i7-7800X CPU @ 3.50 GHz, GPU:NVIDIA GeForce RTX 3090 Ti).