Venue: EMNLP 2023

# Relation-aware Ensemble Learning for Knowledge Graph Embedding

 Ling Yue\({}^{1}\) Yongqi Zhang\({}^{2}\) Quanming Yao\({}^{1}\) Yong Li\({}^{1}\)

Xian Wu\({}^{3}\) Ziheng Zhang\({}^{3}\) Zhenxi Lin\({}^{3}\) Yefeng Zheng\({}^{3}\)

###### Abstract

Knowledge graph (KG) embedding is a fundamental task in natural language processing, and various methods have been proposed to explore semantic patterns in distinctive ways. In this paper, we propose to learn an ensemble by leveraging existing methods in a relation-aware manner. However, exploring these semantics using relation-aware ensemble leads to a much larger search space than general ensemble methods. To address this issue, we propose a divide-search-combine algorithm RelEns-DSC that searches the relation-wise ensemble weights independently. This algorithm has the same computation cost as general ensemble methods but with much better performance. Experimental results on benchmark datasets demonstrate the effectiveness of the proposed method in efficiently searching relation-aware ensemble weights and achieving state-of-the-art embedding performance. The code is public at [https://github.com/LARS-research/RelEns](https://github.com/LARS-research/RelEns). 1

Footnote 1: L. Yue and Y. Zhang made equal contributions, Correspondence is to Q. Yao.

\({}^{1}\)Department of Electronic Engineering, Tsinghua University, Beijing, China

\({}^{2}\)4Paradigm Inc., Beijing, China

\({}^{3}\)Jarvis Research Center, Tencent YouTu Lab, Shenzhen, China

lingyue/qyaoaa/liyong@7tsinghua.edu.cn, yzhangee@connect.ust.hk, kevinxwu/zihengzhang/chalerislin/yefengzheng@tencent.com

## 1 Introduction

Knowledge graph (KG) embedding is a popular method for inferring latent features and making predictions in incomplete KGs Ji et al. (2021). This technique involves transforming entities and relations into low-dimensional vectors and using a scoring function Bordes et al. (2013); Wang et al. (2017) to assess the plausibility of a triplet (consisting of a head entity, a relation, and a tail entity). Well-known scoring functions, such as TransE Bordes et al. (2013), ComplEx Trouillon et al. (2017), ConvE Dettmers et al. (2018), and CompGCN Vashishth et al. (2020), have demonstrated remarkable success in learning from KGs.

Ensemble learning is a well-known technique that improves the performance of machine learning tasks by combining and reweighting the predictions of multiple models Breiman (1996); Wolpert (1992); Dietterich (2000). Its effectiveness has also been verified in KG embedding by previous studies Krompass and Tresp (2015); Wang et al. (2022b); Rivas-Barragan et al. (2022).

While designing different scoring functions to model various relation properties Ji et al. (2021); Sun et al. (2019); Li et al. (2022), such as symmetry, inversion, composition and hierarchy, is crucial for achieving good performance, existing ensemble methods do not reflect the relation-wise characteristics of different models. This motivates us to propose specific ensemble weights for different relations, named as RelEns problem, in this paper. By doing so, different KG embedding models can specialize in different relations, leading to improved performance. However, the number of parameters to be searched will linearly increase, which can significantly complicate the ensemble construction process especially for KGs with many relations. To alleviate the difficulty of searching for relation-wise ensemble weights, we propose DSC, an algorithm that Divide the overall ensemble objective into multiple sub-problems, Search for the ensemble weights for each relation independently, and then Combine the results. This approach significantly reduces the size of the search space and evaluation cost for individual sub-problems, compared to the overall objective.

In summary, we propose RelEns-DSC, a novel relation-aware ensemble learning method for KG embedding that searches different ensemble weights independently for different relations, using a divide-concur strategy. Empirically, RelEns-DSC significantly improves the performance on three benchmark datasets (WN18RR, FB15k-237, NELL-995) and achieves the first place on the large-scale leaderboards ogbl-biokg and ogbl-wikikg2. Our approach is more effective than general ensemble techniques, and it is more efficient with the divide-concur strategy under parallel computing.





## 2 Proposed Method

Denote a KG as \({\cal G}=({\cal V,R,D})\), where \({\cal V}\) contains \(V\) entities (nodes), \({\cal R}\) contains \(R\) types of relations between entities, and \({\cal D}=\{(h,r,t):h,t\in{\cal V},r\in{\cal R}\}\) contains the triplets (edges). \({\cal D}\) is split into three disjoint sets \({\cal D}_{\text{tra}},{\cal D}_{\text{val}},{\cal D}_{\text{ts}}\) for training, validation and testing, respectively.

The learning objective of a KG embedding model is to rank positive triplets higher than negative triplets, in order to accurately identify the potential positive triplets missed in the current graph Wang et al. (2017); Ji et al. (2021).

Specifically, formulated as a tail prediction problem2, the KG embedding model aims to rank the tail entity \(t\) of a given triplet \(x=(h,r,t)\), which belongs to either \({\cal D}_{\text{val}}\) or \({\cal D}_{\text{st}}\), higher than a set of negative entities. The set of negative entities is defined as \({\cal N}_{t}=\{e\in{\cal V}:(h,r,e)\notin{\cal D}\}\). The model \(F(x)\) computes a score vector \(s\) for each entity \(e\in\{t\}\cup{\cal N}_{t}\), which indicates the degree of plausibility that the triplet \((h,r,e)\) is true.

Footnote 2: Head prediction is conducted in the same way with negative entities \({\cal N}_{h}=\{e\in{\cal V}:(e,r,t)\notin{\cal D}\}\). For simplicity, we only use tail prediction as an example to introduce our method.

A ranking function \(\Gamma\big{(}s\big{)}\) is used to convert the scores \(s\) into a ranking list \(\mathbf{p}=(p_{1},\ldots,p_{C})\) for the \(C\!=\!1+|{\cal N}_{t}|\) entities. A smaller rank value implies the higher prediction priority. Following Bordes et al. (2013); Trouillon et al. (2017); Sun et al. (2019); Vashishth et al. (2020), we adopt mean reciprocal ranking (MRR) as the evaluation metric. Larger MRR indicates better performance.

### Relation-wise Ensemble Problem

We observe that embedding models may exhibit varying strengths in modeling different types of relations (see Appendix A.2 for details). To account for this, we propose a novel approach that learns distinct weights for each relation, based on the performance of models on validation set \({\cal D}_{\text{val}}\). Specifically, given \(N\) trained KG embedding models, i.e., \(F_{1},F_{2},\ldots,F_{N}\), and a set of relations \({\cal R}\). We introduce a weight \(\alpha^{r}_{i}\geq 0\) assigned to model \(F_{i}\) for relation \(r\) and \(M(\mathbf{p},x)=1/p_{t}\) for the reciprocal ranking of a given data point \(x=(h,r,t)\). Let \({\cal D}^{r}_{\text{val}}\) denote as the subset of validation triplets whose relations are \(r\). The objective of relation-wise ensemble can be written as follows:

\[\max_{\{\alpha^{r}_{i}\}_{i=1,\ldots,N}^{r}}\sum\nolimits_{r\in{ \cal R}}\sum\nolimits_{x_{j}\in{\cal D}^{r}_{\text{val}}}M\big{(}\Gamma(\mathbf{ p}^{r}_{j}),x^{r}_{j}\big{)},\] (1) s.t. \[\mathbf{p}^{r}_{j}=-\sum\nolimits_{i=1}^{N}\alpha^{r}_{i}\Gamma\big{(} F_{i}(x^{r}_{j})\big{)},\alpha^{r}_{i}\geq 0.\]

For each triplet \(x^{r}_{j}\) with relation \(r\), we apply the ensemble weights \(\alpha^{r}_{i}\) to the ranking list \(\Gamma(F_{i}(x^{r}_{j}))\) generated by the \(i\)-th model. The scales of scores vary significantly. Optimizing scores directly may be more challenging. Additionally, since ranks have similar scales, the searched weights can better indicate the importance of the corresponding base model. Specifically, we obtain the ensembled score \(\mathbf{p}^{r}_{j}=-\sum_{i=1}^{N}\alpha^{r}_{i}\Gamma\big{(}F_{i}(x^{r}_{j})\big{)}\), where "\(-\)" turns the ranks to scores, indicating higher prediction priority with a higher value in \(\mathbf{p}^{r}_{j}\).

In particular, if the ensemble weights assigned for each model \(F_{i}\) for all relations are identical, i.e., \(\alpha^{1}_{i}=\alpha^{2}_{i}=\cdots=\alpha^{R}_{i}\) for \(i=1,\dots,N\), the objective in equation (1) (denoted as _RelEns-Basic_) reduces to the general ensemble method (denoted as _SimpleEns_). By optimizing the values of \(\alpha^{r}_{i}\), the goal is to achieve higher MRR performance on the validation set \({\cal D}_{\text{val}}=\sum_{r\in{\cal R}}{\cal D}^{r}_{\text{val}}\).

### Divide Search and Combine

Comparing with _SimpleEns_, _RelEns-Basic_ requires searching for \(NR\) parameters. As MRR is a non-differential metric, zero-order optimization techniques, like random search and Bayesian optimization Bergstra et al. (2011), are often used to solve Eq. (1). However, these algorithms usually involve sampling candidates in the search space, the complexity of which can grow exponentially with the search dimension due to the curse of dimensionality Koppen (2000). As a result, optimizing Eq. (1) can be challenging. To address this issue, we propose Proposition 1, which enables the separation of the big problem Eq. (1) into \(R\) independent sub-problems. In the divided problem \(r\), there are only \(N\) parameters \(\{\alpha^{r}_{i}\}_{i=1,\dots,N}\) to be searched.

**Proposition 1** (separable optimization problem).: _The optimal values of \(\{\alpha^{r}_{i}\}_{i=1,\dots,N}^{r=1,\dots,R}\) that are searched on \({\cal D}_{\text{val}}\) in (1) can be equated to the values of \(\{\alpha^{r}_{i}\}_{i=1,\dots,N}\) that are independently optimized on \({\cal D}^{r}_{\text{val}}\) for each \(r\in{\cal R}\) via the following problem_

\[\max_{\{\alpha^{r}_{i}\}_{i=1,\dots,N}^{r}}\sum\nolimits_{x_{j} \in{\cal D}^{r}_{\text{val}}}M(\Gamma(\mathbf{p}^{r}_{j}),x^{r}_{j}),\] (2) s.t. \[\mathbf{p}^{r}_{j}=-\sum\nolimits_{i=1}^{N}\alpha^{r}_{i}\Gamma\big{(} F_{i}(x^{r}_{j})\big{)},\alpha^{r}_{i}\geq 0.\]

The complete divide-search-and-combine procedures are outlined in Algorithm 1. By separably searching the divided problems, we can determine the optimal values of \(\{\alpha_{i}^{r}\}_{i=1,...,N}\) for each \(r\) on the validation data \(\mathcal{D}_{\text{val}}^{r}\).

Finally, we combine the searched values of \(\{\alpha_{i}^{r}\}_{i=1,...,N}^{r=1,...,R}\) to compute the scores \(\mathbf{p}_{j}^{r}=-\sum_{i=1}^{N}\alpha_{i}^{r}\Gamma\big{(}F_{i}(x_{j}^{r})\big{)}\) for \(x_{j}^{r}\in\mathcal{D}_{\text{st}}\) in order to evaluate the performance.

```
0: Base models \(F_{1},\ldots,F_{N}\), ensemble parameters \(\{\alpha_{i}^{r},i=1,\ldots,N,r\in\mathcal{R}\}\), dataset \(\mathcal{D}_{\text{val}}\), ranking functions \(\Gamma(\cdot)\) and \(M(\cdot,\cdot)\).
1:Divide: divide \(\mathcal{D}_{\text{val}}\) and \(\{\alpha_{i}^{r},i\!=\!1,\dots,N,r\!\in\!\mathcal{R}\}\) into \(\left\{\mathcal{D}_{\text{val}}^{r},\{\alpha_{i}^{r},i\!=\!1,\dots,N\}\!, r\!\in\!\mathcal{R}\right\}\).
2:Search:
3:for\(r\) in \(\mathcal{R}\) (can work in parallel) do
4: search the values of \(\{\alpha_{i}^{r}\}_{i=1,...,N}\) by solving (2);
5:endfor
6:Combine: Combine the optimal values of \(\{\alpha_{i}^{r}\}_{i=1,...,N}^{r=1,...,R}\) for evaluation;
7:returnMRR and Hit@\(k\) on testing data \(\mathcal{D}_{\text{lst}}\).
```

**Algorithm 1** RelEns-DSC: Divide-search-combine algorithm for relation-wise ensemble.

### Complexity Analysis

Assuming that the evaluation cost of \(\Gamma(\cdot)\) and \(M(\Gamma(\cdot),x)\) on a single data sample \(x\) is a constant, the time complexity of ensemble learning is determined by two factors: (i) the number of data samples to be evaluated; (ii) the number of ensemble parameters to be sampled. For _SimpleEns_, the complexity is \(O(|\mathcal{D}_{\text{val}}|e^{N})\). On the other hand, _RelEnsBasic_ in Eq. (1) requires \(O(|\mathcal{D}_{\text{val}}|e^{RN})\) since the sampling complexity increases exponentially with the search dimension. In comparison, the complexity of _RelEns-DSC_ in Algorithm 1 is \(O(|\mathcal{D}_{\text{val}}|e^{N})\), which is on par with _SimpleEns_.

## 3 Experiments

The experiments were implemented using Python and run on a 24GB NVIDIA GTX3090 GPU.

As the ranking function \(\Gamma(\cdot)\) and MRR are non-differentiable, We chose the widely used Bayesian optimization technique, Tree-Parzen Estimator (TPE) Bergstra et al. (2015), to solve the maximization problems in Eq. (1) and Eq. (2), the details of which are provided in Appendix B.3.

### Datasets

We conduct experiments on commonly studied datasets for KG, including WN18RR Dettmers et al. (2018), FB15k-237 Toutanova and Chen (2015), and NELL-995 Xiong et al. (2017). Additionally, we apply the _RelEns-DSC_ on OGB Hu et al. (2020) datasets ogbl-biokg and ogbl-wikikg2. Details of statistics are in Appendix B.1.

### Experimental Setup

Base models.We select some representative embedding models as our base models \(F_{i}\), including: (i) translational distance models TransE Bordes et al. (2013), RotatE Sun et al. (2019), HousE Li et al. (2022); (ii) bilinear model ComplEx Trouillon et al. (2017); (iii) neural network model ConvE Dettmers et al. (2018); and (iv) GNN based model CompGCN Vashishth et al. (2020). For the OGB datasets, we select the top-3 methods from the OGB leaderboard3 up to October 1st in 2023. Footnote 3: [https://ogb.stanford.edu/docs/leader_linkprop/](https://ogb.stanford.edu/docs/leader_linkprop/).

Evaluation Metric.Four evaluation metrics (MRR and Hit@{1,3,10}) are reported for the benchmarks WN18RR, FB15k-237 and NELL-995. For OGB datasets ogbl-biokg and ogbl-wikikg2, we report MRR results to keep consistent with the leaderboard. Hyperparameters.To compare on the general benchmarks, we use the fine-tuned hyperparameters reported by KGTuner Zhang et al. (2022). For top three methods on OGB leaderboard, we use their reported hyperparameters. Details of these settings are in Appendix B.2.

### Performance Comparison

Table 1 and Table 2 present the testing performance comparison. _SimpleEns_ is the variant introduced in Section 2.1. We observe that _SimpleEns_ consistently outperforms the base models by weighting different models according to their learning ability. The proposed method _RelEns-DSC_ surpasses _SimpleEns_ by a large margin, verifying the effectiveness of considering relation-specific ensemble weights for KG embedding.

The top models on ogbl-biokg are more diverse than ogbl-wikikg2. On ogbl-biokg, AutoBLM and ComplEx are bilinear models, while TripleRE is a translational model. The training framework of 

the three models are also different. In comparison, the top three methods on ogbl-wikikkg2 are all translational models with similar approaches of sharing entity embeddings. As a result, the variations of relation-wise performance of the top three models on ogbl-biokg are larger than ogbl-wikikg2 (with std 0.0452 vs. 0.0261). This can explain why the relation-wise ensemble is more significant on ogbl-biokg than ogbl-wikikg2.

Furthermore, we illustrate the ensemble weights of _SimpleEns_ and _RelEns-DSC_ on the WN18RR dataset in Figure 1, which shows that _RelEns-DSC_ learns relation-specific ensemble weights, which contributes to its superior performance.

### Efficiency Comparison

We compare the learning curves (highest MRR yet searched vs. running time) of _SimpleEns_, _RelEnsBasic_, and _RelEns-DSC_ on NELL-995 in Figure 2 (the curves of other datasets are in Appendix C.1). The ensemble weights for all three methods are initialized as \({}^{1/N}\). We denote the number of parameter configurations for TPE to search as \(Q\), and show the results of \(Q=50\) and \(Q=100\).

Based on the results, _RelEns-Basic_ is much worse than _SimpleEns_, since the search complexity of _RelEns-Basic_ increases exponentially. At the beginning of searching, _RelEns-DSC_ is inferior to both _SimpleEns_ and _RelEns-Basic_ since it only

Figure 1: Ensemble weights of _SimplEns_ and _RelEns-DSC_ on the WN18RR dataset.

 

searches the weights on a few relations, while others are unchanged. Over time, the performance of _RelEns-DSC_ has improved significantly as more and more relations have found their optimal values. Increasing \(Q\) from 50 to 100 did not lead to any improvement in the performance of _SimpleEns_ and _RelEns-Basic_. However, _RelEns-DSC_ was able to achieve better overall performance since increasing \(Q\) allows the sub-problems to be more sufficiently solved with more iterations. In addition, _RelEns-DSC_ can be benefited by parallel computing on the relation level, further improving efficiency.

### Ablation Study

Table 3 shows the performance comparison of multiple variants of _RelEns-DSC_ on the three benchmark datasets. Due to space limit, results of Hit@{1,3} and the implementation details of the variants are provided in Appendix B.2. The stacking method (_Stacking_), arithmetic mean method (_Mean_) and MRR-based weighted mean method (_MRR-Mean_) have poorer performance compared to _RelEns-DSC_. This indicates the importance of searching for ensemble weights with TPE technique. _Stacking_ performs the worst since the non-differentiable metric MRR cannot directly optimized. In particular, considering relation-specific ensemble weights, _RelEns-DSC_ can lead to better performance than the general ensemble methods.

## 4 Conclusion

This paper introduces a novel ensemble method, Relation-aware Ensemble with Divide-Search-Combine (RelEns-DSC) for KG embedding. The proposed RelEns-DSC learns relation-specific ensemble weights for different models and efficiently searches the weights using the divide-concur strategy. Empirical results demonstrate that our proposed method outperforms existing ensemble methods for KG embedding, in both effectiveness and efficiency.

Limitations.The proposed method mainly addresses the ensemble problem for entity prediction tasks in knowledge graph completion. However, it does not effectively address the other graph learning tasks, such as entity/node classification, relation prediction, and graph classification. In addition, the significance of RelEns-DSC is under the case of multi-relational graphs like knowledge graph and heterogeneous graph, thus is not well adapted to homogeneous graph with single edge type.

#### Acknowledgements

The work was performed when L. Yue was an research engineer in LARS group. Q. Yao was in part sponsored by NSFC (No. 92270106) and CCF-Tencent Open Research Fund.

Figure 2: Learning curves of different ensemble methods on NELL-995. _RelEns-DSC2/5_ indicates the number of threads _(2/5)_ used for parallel computing.

 



## References

* J. Bergstra, R. Bardenet, Y. Bengio, and B. Kegl (2011)Algorithms for hyper-parameter optimization. In NIPS, pp. 2546-2554. Cited by: SS3.
* J. Bergstra, B. Komer, C. Eliasmith, D. Yamins, and D. D. Cox (2015)Hyperopt: a python library for model selection and hyperparameter optimization. Comput. Sci. Discov.8 (1), pp. 014008. External Links: Document, ISSN 0978-321X Cited by: SS1.
* A. Bordes, N. Usunier, A. Garcia-Duran, J. Weston, and O. Yakhnenko (2013)TransLating embeddings for modeling multi-relational data. In NeurIPS, pp. 2787-2795. Cited by: SS1.
* L. Breiman (1996)Bagging predictors. Machine Learning24, pp. 123-140. Cited by: SS1.
* Y. Chen, P. Minervini, S. Riedel, and P. Stenetorp (2021)Relation prediction as an auxiliary training objective for improving multi-relational graph representations. arXiv:2110.02834. Cited by: SS1.
* T. Dettmers, P. Minervini, P. Stenetorp, and S. Riedel (2018)Convolutional 2d knowledge graph embeddings. In AAAI, Vol. 32. Cited by: SS1.
* T. G Dietterich (2000)Ensemble methods in machine learning. In MCS, pp. 1-15. Cited by: SS1.
* W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec (2020)Open graph benchmark: datasets for machine learning on graphs. NeurIPS. Cited by: SS1.
* S. Ji, S. Pan, E. Cambria, P. Marttinen, and S. Yu Philip (2021)A survey on knowledge graphs: representation, acquisition, and applications. IEEE transactions on neural networks and learning systems33 (2), pp. 494-514. Cited by: SS1.
* M. Koppen (2000)The curse of dimensionality. In 5th Online World Conference on Soft Computing in Industrial Applications, Vol. 1, pp. 4-8. Cited by: SS1.
* D. Krompass and V. Tresp (2015)Ensemble solutions for link-prediction in knowledge graphs. In PKDD ECML 2nd Workshop on Linked Data for Knowledge Discovery, Cited by: SS1.
* H. Li, X. Gao, L. Feng, Y. Deng, and Y. Yin (2022)Stargraph: knowledge representation learning based on incomplete two-hop subgraph. Technical report Cited by: SS1.
* R. Li, J. Zhao, C. Li, D. He, Y. Wang, Y. Liu, H. Sun, S. Wang, W. Deng, Y. Shen, et al. (2022)House: knowledge graph embedding with household parameterization.. In ICML, pp. 13209-13224. Cited by: SS1.
* F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al. (2011)Scikit-learn: machine learning in python. the Journal of machine Learning research12, pp. 2825-2830. Cited by: SS1.
* D. Rivas-Barragan, D. Domingo-Fernandez, Y. Gadiya, and D. Healey (2022)EnSEMDS of knowledge graph embedding models improve predictions for drug discovery. Briefings in Bioinformatics23 (6), pp. bbaac481. Cited by: SS1.
* J. Snoek, H. Larochelle, and R. P. Adams (2012)Practical bayesian optimization of machine learning algorithms. NIPS25. Cited by: SS1.
* Z. Sun, Z. Deng, J. Nie, and J. Tang (2019)Rotate: knowledge graph embedding by relational rotation in complex space. In ICLR, Cited by: SS1.
* K. Toutanova and D. Chen (2015)Observed versus latent features for knowledge base and text inference. In CVSC, pp. 57-66. Cited by: SS1.
* T. Trouillon, C. R. Dance, J. Welbl, S. Riedel, E. Gaussier, and G. Bouchard (2017)Knowledge graph completion via complex tensor factorization. JMLR18 (1), pp. 4735-4772. Cited by: SS1.
* S. Vasishith, S. Sanyal, V. Nitin, and P. Talukdar (2020)Composition-based multi-relational graph convolutional networks. ICLR. Cited by: SS1.
* B. Wang, Q. Meng, Z. Wang, D. Wu, W. Che, S. Wang, Z. Chen, and C. Liu (2022)InterHT: knowledge graph embeddings by interaction between head and tail entities. arXiv:2202.04897. Cited by: SS1.
* Q. Wang, Z. Mao, B. Wang, and L. Guo (2017)Knowledge graph embedding: a survey of approaches and applications. TKDE29 (12), pp. 2724-2743. Cited by: SS1.
* Y. Wang, Y. Chen, Z. Zhang, and T. Wang (2022)A probabilistic ensemble approach for knowledge graph embedding. Neurocomputing500, pp. 1041-1051. Cited by: SS1.
* D. H Wolpert (1992)Stacked generalization. Neural Networks5 (2), pp. 241-259. Cited by: SS1.
* W. Xiong, T. Hoang, and W. Yang Wang (2017)DeepPath: a reinforcement learning method for knowledge graph reasoning. arXiv:1707.06690. Cited by: SS1.
* L. Yao, J. Peng, Q. Liu, H. Cai, S. Ji, F. He, and X. Cheng (2023)Technical report for ogb link property prediction: ogblik-wikig2. Cited by: SS1.
* L. Yu, Z. Luo, H. Liu, D. Lin, H. Li, and Y. Deng (2022)TripleRE: knowledge graph embeddings via tripled relation vectors. arXiv:2209.08271. Cited by: SS1.
* Y. Zhang, Q. Yao, and J. T. Kwok (2022)Bilinear scoring function search for knowledge graph learning. TPAMI. Cited by: SS1.
* Y. Zhang, Z. Zhou, Q. Yao, and Y. Li (2022)KGUnner: efficient hyper-parameter search for knowledge graph learning. In ACL,

Supplementary Materials for the Method

### Relation-wise Ensemble

An overview of the relation-wise ensemble problem is provided

in Figure 3. First, the dataset \({\cal D}\) is split into multiple sub-sets \({\cal D}^{1},{\cal D}^{2},\ldots,{\cal D}^{R}\) according to the relations. For each sample \(x_{j}^{r}\) from \({\cal D}^{r}\), the models \(F_{1},F_{2},\ldots,F_{N}\) output the scores and the ranking function \(\Gamma(\cdot)\) provides ranking lists for the \(C\) entities according to their scores. The relation-wise ensemble weights \(\alpha_{1}^{r},\alpha_{2}^{r},\ldots,\alpha_{N}^{r}\) re-weight the rank lists as the new scores \(\mathbf{p}_{j}^{r}\) of \(x_{j}^{r}\) and re-rank the new scores to evaluate the performance.

Two metrics are used in this paper: (i) Mean reciprocal ranking (MRR):

\[\text{MRR}=\frac{1}{2\left|{\cal D}\right|}\sum_{(h,r,t)\in{\cal D}}\big{(} \frac{1}{p_{h}}+\frac{1}{p_{t}}\big{)};\]

and (ii) Hit@\(k\): ratio of ranks no larger than \(k\), i.e.,

where \(\mathbb{I}(a)=1\) if a is true, otherwise 0, \(p_{h}\) is the rank of head entity \(h\) in the head-prediction sub-task (the same for \(p_{t}\) and \(t\)). The larger the MRR or Hit@\(k\), the better is the embedding.

### Relation Properties

In the main text, we claim that the different models work properly for different relations. In this part, we summarize the types of relations that different models can handle in Table 4.

In addition, we show the performance of various base models for specific relations on WN18RR.

Among the total of eleven relations considered, we present the results based on the following four representative relations:

Figure 3: An overview of the relation-wise ensemble problem.

 

* membe_meronym: translational models such as TransE and RotatE exhibit the highest performance.
* synset_domain_topic_of: bilinear models like ComplEx achieve the best results.
* has_part: while traditional scoring functions perform well on this relation, neural network-based models such as ConvE and CompGCN exhibit suboptimal performance.
* verb_group: in contrast to has_part, neural network models such as ConvE and CompGCN perform better, whereas traditional scoring functions show inferior performance.

These results demonstrate that KG embedding models may specialize in different relations, leading to significant variation in their performance across relations.

## Appendix B Supplementary Materials for the Experimental Settings

### Statistics of Datasets

We use the following datasets for evaluation: (i) WN18RR is a link prediction dataset which is a subset of WordNet [20]; (ii) FB15k-237 contains triplets of knowledge base relationships and textual mentions of Freebase entity pairs [46]; (iii) NELL-995 is a dataset built from the web via an intelligent agent called Never-Ending Language Learner that reads the web over time [21]; (iv) ogbl-biokg is a KG, which was created using data from a large number of biomedical data repositories [22]; and (v) ogbl-wikikg2 is a KG extracted from the Wikidata knowledge base [22]. Statistics of these datasets are provided in Table 5.

\begin{table}
\begin{tabular}{c|r r r r r} \hline \hline Dataset & \multicolumn{1}{l|}{\(|\mathcal{V}|\)} & \multicolumn{1}{l|}{\(|\mathcal{R}|\)} & \multicolumn{1}{l|}{\(|\mathcal{D}_{\text{tra}}|\)} & \multicolumn{1}{l|}{\(|\mathcal{D}_{\text{val}}|\)} & \multicolumn{1}{l}{\(|\mathcal{D}_{\text{stl}}|\)} \\ \hline WN18RR & 40,943 & 11 & 86,835 & 3,034 & 3,134 \\ FB15k-237 & 14,541 & 237 & 272,115 & 17,535 & 20,466 \\ NELL-995 & 74,536 & 200 & 149,678 & 543 & 2,818 \\ \hline ogbl-biokg & 93,773 & 51 & 4,762,678 & 162,886 & 162,870 \\ ogbl-wikikg2 & 2,500,604 & 535 & 16,109,182 & 429,456 & 598,543 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Statistics of the datasets.

Figure 4: MRR of selected base models for specific relations on the WN18RR dataset.

 



### Hyperparameter Setting

We list the hyperparameters for base models in KGTuner (Zhang et al., 2022b)4 on the WN18RR, FB15k-237 and NELL-995 datasets in Table 6 and Table 7.

Footnote 4: [https://github.com/LARS-research/KGTuner](https://github.com/LARS-research/KGTuner)

For CompGCN (Vashishth et al., 2020)5, we use 200-dimensional embeddings for node and relation embeddings and apply the standard binary cross entropy loss with label smoothing. The number of GCN layers is 2, and the score function used in CompGCN is ConvE, the learning rate is set to 0.001, the batch size is 128, and the dropout rate is 0.1.

Footnote 5: [https://github.com/malllabiisc/CompGCN](https://github.com/malllabiisc/CompGCN)

For HousE (Li et al., 2022)6, we used the default hyperparameters specified in the original paper. Both node and relation embeddings were set to 800 dimensions. The learning rate was set to 0.0005, and the batch size was 1000.

Footnote 6: [https://github.com/rui9812/HousE](https://github.com/rui9812/HousE)

For the top three methods on the OGB leaderboard 7, since their code has been officially made public by OGB, we used their code directly with their corresponding hyperparameters.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline HP/Model & ComplEx & ConvE & TransE & RotatE \\ \hline \# negative samples & 32 & 512 & 128 & 2048 \\ \hline loss function & BCE_mean & BCE\_adv & BCE\_adv & BCE\_adv \\ gamma & 2.29 & 12.16 & 3.50 & 3.78 \\ adv. weight & 0.00 & 0.78 & 1.14 & 1.66 \\ \hline regularizer & NUC & DURA & FRO & FRO \\ reg. weight & \(1.21\times 10^{-3}\) & \(9.79\times 10^{-3}\) & \(4.19\times 10^{-4}\) & \(5.13\times 10^{-8}\) \\ dropout rate & 0.28 & 0.02 & 0.00 & 0.00 \\ \hline optimizer & Adam & Adam & Adam & Adam \\ learning rate & \(6.08\times 10^{-4}\) & \(6.88\times 10^{-4}\) & \(1.02\times 10^{-4}\) & \(1.24\times 10^{-3}\) \\ initializer & x\_uni & x\_uni & norm & norm \\ \hline batch size & 1024 & 512 & 512 & 512 \\ dimension size & 2000 & 1000 & 1000 & 1000 \\ inverse relation & False & False & False & False \\ \hline \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \end{tabular}
\end{table}
Table 6: Hyperparameters for the WN18RR dataset.

\begin{table}
\begin{tabular}{c|c|c|c|c} \hline \hline HP/Model & ComplEx & ConvE & TransE & RotatE \\ \hline \# negative samples & 512 & 512 & 512 & 128 \\ \hline loss function & BCE\_adv & BCE\_sum & BCE adv & BCE\_adv \\ gamma & 13.05 & 14.52 & 6.76 & 14.46 \\ adv. weight & 1.93 & 0.00 & 1.99 & 1.12 \\ \hline regularizer & DURA & DURA & FRO & NUC \\ reg. weight & \(9.75\times 10^{-3}\) & \(6.42\times 10^{-3}\) & \(2.16\times 10^{-4}\) & \(2.99\times 10^{-4}\) \\ dropout rate & 0.22 & 0.07 & 0.02 & 0.01 \\ \hline optimizer & Adam & Adam & Adam & Adam \\ learning rate & \(9.70\times 10^{-4}\) & \(2.09\times 10^{-4}\) & \(2.66\times 10^{-4}\) & \(5.89\times 10^{-4}\) \\ initializer & uni & norm & x\_norm & norm \\ \hline batch size & 1024 & 1024 & 512 & 1024 \\ dimension size & 2000 & 500 & 1000 & 2000 \\ inverse relation & False & False & False \\ \hline \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \\ \end{tabular}
\end{table}
Table 7: Hyperparameters for the FB15k-237 and NELL-996 datasets.





### Details of Tree-structured Parzen Estimator (TPE)

The TPE (Tree-structured Parzen Estimator) algorithm is a Bayesian optimization [14] method that aims to efficiently optimize black-box functions with a limited budget of function evaluations. It was introduced by [30] as a part of the Hyperopt framework [26], which focuses on hyperparameter optimization. Bayesian optimization is a sequential model-based optimization technique that leverages prior knowledge and data to intelligently search for the optimal solution. It uses a probabilistic surrogate model, typically a Gaussian process or a tree-based model, to model the unknown objective function. This model is iteratively updated as new observations are made, providing an estimate of the function's behavior and uncertainty.

The TPE algorithm improves upon traditional Bayesian optimization by employing a novel method for modeling and sampling from the posterior distribution of the objective function. It uses a tree-structured Parzen estimator to model the distribution of good and bad parameter configurations. The algorithm maintains two density functions: \(p\left(x|y\right)=\ell(x)\) if \(y<y^{*}\) otherwise \(p\left(x|y\right)=g(x)\), where \(\ell(x)\) is the density formed by using observations \(\{x^{(i)}\}\) from past evaluations, such that the corresponding loss (i.e., the performance metric for the model) is less than some threshold \(y^{*}\) that lead to good results, and \(g(x)\) is the density formed by using the remaining observations that lead to bad results.

At each iteration, the TPE algorithm samples promising configurations from the good density and less promising configurations from the bad density. The algorithm balances the exploration-exploitation trade-off by dividing the sampled configurations into two groups based on their relative performance. The better-performing configurations are used to update the density function for good configurations, while the less successful ones update the density function for bad configurations. This process aims to guide the search towards promising regions of the parameter space. By iteratively updating the density functions and adaptively sampling configurations, the TPE algorithm efficiently explores the parameter space, gradually converging towards the optimal solution. It has been shown to be effective in hyperparameter optimization for machine learning models and other optimization tasks.

Overall, the TPE algorithm, as a Bayesian optimization method, offers a principled and efficient approach for optimizing black-box functions with limited resources, making it particularly useful in scenarios where function evaluations are time-consuming or costly.

## Appendix C Supplementary Materials for Experimental Results

### Full Results of Learning Curves

We provide the learning curves on WN18RR, FB15k-237 and NELL-995 datasets in this part. Figure 5 confirms that the observations in Section 3.4 are consistent with the results presented here. While _RelEns-Basic_ exhibits a superior modeling ability compared to _SimpleEns_ by learning relation-specific weights, enabling different models to specialize in different relations. However, the search complexity of _RelEns-Basic_ increases exponentially with the number of relations.

* On WN18RR with only 11 relations, _RelEns-Basic_ can slightly outperform _SimpleEns_ since the complexity is not increased much and the relation-wise ensemble problem can have better performance than the general ensemble problem.
* Nevertheless, on datasets containing hundreds of relations such as FB15k-237 and NELL-995 _RelEns-Basic_ exhibits significantly poorer performance compared to _SimpleEns_ due to high sampling complexity in the relation-wise ensemble problem

Figure 5: Learning curves of different ensemble methods for _Ens-DSC2/5_ indicates the number of threads used for parallel computing.

 



### Ablation Study on Ensemble methods

All the variants use the same way to ensemble predictions of base models. The same as mentioned in Section 2.1, we use the ensemble weights on rankings of base models as the ensemble score, i.e., \({\mathbf{p}_{j}^{r}=-\sum_{i=1}^{N}\alpha_{i}^{r}\Gamma\big{(}F_{i}(x_{j}^{r})\big{)}}\). The implementation details of the ensemble variants are provided as follows:

* Mean: This is the most basic ensemble method which directly takes the arithmetic mean of the predictions made by each individual model.
* Stacking [35]: The stacking ensemble variant is a more sophisticated approach that involves training a meta-model to learn how to combine the predictions of multiple base models. Specifically, the rank lists outputted by the base models are used as features to train the meta-model. Here, we use a logistic regression model implemented by scikit-learn [40] as the meta-model, with a maximum iteration of 300. Since the ranking metrics on the validation data is non-differentiable, we use the same loss function during training on the validation data to optimize the parameters of meta-model.
* SimpleEns: This variant is a degenerated problem of (1) by setting identical ensemble weights of different relations on a single mode, i.e., \(\alpha_{i}^{1}=\alpha_{i}^{2}=\cdots=\alpha_{i}^{R}\) for \(i=1,\ldots,N\). In order to keep consistent with RelEns-Basic and RelEns-DSC, these weights are searched by optimizing (1) with TPE technique.
* MRR-Mean: The MRR-Mean variant incorporates the Mean Reciprocal Rank (MRR) of the base model as a weighting factor. In contrast to the Mean variant, it assigns proportionally greater weight to the superior individual base model.

We conducted an ablation study on the WN18RR, FB15k-237, and NELL-995 datasets, and the results are shown in Table 8. MRR can provide a general indication of the importance of different models, but higher MRR does not always correlate with higher model importance due to the crucial role of diversity among the base models in the ensemble strategy. Additionally, MRR-weight may not be optimal weights, necessitating further weight searches to enhance performance. The results consistently demonstrate the superiority of searched methods (RelEns-DSC) over MRR-weighted methods (MRR-Mean).