DynaSemble: Dynamic Ensembling of Textual and Structure-Based Models for Knowledge Graph Completion (2024)

Abstract: We consider two popular approaches to Knowledge Graph Completion (KGC): textual models that rely on textual entity descriptions, and structure-based models that exploit the connectivity structure of the Knowledge Graph (KG). Preliminary experiments show that these approaches have complementary strengths: structure-based models perform exceptionally well when the gold answer is easily reachable from the query head in the KG, while textual models exploit descriptions to give good performance even when the gold answer is not easily reachable. In response, we propose DynaSemble, a novel method for learning query-dependent ensemble weights to combine these approaches by using the distributions of scores assigned by the models in the ensemble to all candidate entities. DynaSemble achieves state-of-the-art results on three standard KGC datasets, with up to 6.8 pt MRR and 8.3 pt Hits@1 gains over the best baseline model for the WN18RR dataset.

heading: 1 Introduction
text: The task of Knowledge Graph Completion (KGC) can be described as inferring missing links in a Knowledge Graph (KG) based on given triples (h, r, t), where r is a relation that exists between the head entity h and the tail entity t. Several KGC approaches, such as NBFNet (Zhu et al., 2021) and RGHAT (Zhang et al., 2020), exploit the underlying graph structure, often using GNNs. On the other hand, textual models such as SimKGC (Wang et al., 2022) and HittER (Chen et al., 2021) leverage pre-trained large language models (LLMs) such as BERT (Devlin et al., 2019) to utilize textual descriptions of the KG entities and relations for KGC.
Our preliminary experiments suggest that when the gold answer t for query (h, r, ?) is reachable from h via a path of reasonable length in the KG, structure-based models tend to outperform textual models. In contrast, textual models use textual de-
scriptions to perform better than structure-based models when t is not easily reachable from h. Motivated by our findings, we seek to explore how ensembling, an approach currently underrepresented in KGC literature (see Jain et al. (2018b) for an example), can effectively harness the complementary strengths of these models.
Consequently, we propose DynaSemble: a novel, simple, model-agnostic and lightweight method for learning ensemble weights such that the weights are (i) query-dependent and (ii) learned from statistical features obtained from the distribution of scores assigned by individual models to all candidate entities. This approach results in a new state-of-the-art baseline when applied on two strong KGC models: SimKGC and NBFNet, which are textual and structure-based in nature, respectively.
On three KGC datasets, we find that applying DynaSemble to SimKGC and NBFNet consistently improves KGC performance, outperforming best individual models by up to 6.8 pt MRR and 8.3 pt Hits@1 on the WN18RR dataset. To the best of our knowledge, our results are state of the art for all three datasets. Further experiments (including a fourth dataset to which NBFNet does not scale) show that DynaSemble generalises to ensembling with another KG embedding model, RotatE (Sun et al., 2019), with similar gains. We also demonstrate that DynaSemble outperforms conventional model-combination techniques such as static ensembling (where the ensemble weight is a tuned constant hyperparameter) and re-ranking. We release all code1 to guide future research.

heading: 2 Background and Related Work
text: Task: We are given an incomplete KG K = (E ,R, T ) consisting of entities E , relation set R and set of triples T = {(h, r, t)} (where h, t ∈ E and r ∈ R). The goal of KGC is to answer queries
1https://github.com/dair-iitd/KGC-Ensemble
205
of the form (h, r, ?) or (?, r, t) to predict missing links, with corresponding answers t and h. We model (?, r, t) as (t, r−1, ?) queries in this work. Overview of Related Work: We focus on three types of KGC models. The first type consists of Graph Neural Network (GNN) based models such as NBFNet (Zhu et al., 2021), RGHAT (Zhang et al., 2020) that leverage neighborhood information to train distinct GNN architectures. The second type contains textual models such as KGBERT (Yao et al., 2019), HittER (Chen et al., 2021) and SimKGC (Wang et al., 2022) which fine-tune a pre-trained LLM on textual descriptions of entities and relations for KGC. The third type involves models such as RotatE (Sun et al., 2019) and ComplEx (Trouillon et al., 2016; Jain et al., 2018a) that learn low-dimensional embeddings for entities and relations and compose them by employing unique scoring functions. Unification of these approaches has not been extensively studied in KGC literature. VEM2L (He et al., 2022) proposes a method to encourage multiple KGC models to learn from each other during training. KGT5 (Saxena et al., 2022) finds that their textual model struggles when the query has a large number of correct answers in the training set and routes those queries to a structurebased model as a consequence, exhibiting some performance gains. Since our main experiments are based on NBFNet, SimKGC, HittER and RotatE, we describe these next.
NBFNet: Neural Bellman-Ford Network (NBFNet) is a path-based link prediction model that introduces neural functions into the Generalized Bellman-Ford (GBF) Algorithm (Baras and Theodorakopoulos, 2010), which in turn models the path between two nodes in the KG through generalized sum and product operators. This formulates a novel GNN framework that learns entity representations for each candidate tail t conditioned on h and r for each query (h, r, ?). The score of any candidate t is then computed by applying an MLP to its embedding.
SimKGC: SimKGC is an LLM-based KGC model that employs a bi-encoder architecture to generate the score of a given triple (h, r, t). The model considers two pre-trained BERT (Devlin et al., 2019) models. The first model is finetuned on a concatenation of textual descriptions of h and r to generate their joint embedding ehr and the second model is finetuned on the textual description of t to generate the embedding et. The score for the triple is the
cosine similarity between ehr and et. HittER: HittER proposes a hierarchical transformer-based approach for jointly learning entity and relation embeddings by aggregating information from the graph neighborhood. A transformer provides relation-dependent entity embeddings for the neighborhood of an entity, which are then aggregated by another transformer. These embeddings are trained using a joint masked entity prediction and link prediction task. RotatE: RotatE is a KG Embedding model that maps entities and relations to a complex vector space and models each relation r as a complex rotation from the head r to the tail t for triple (h, r, t). More specifically, the scoring function of RotatE is ∥h ◦ r− t∥ where h, t ∈ Ck are the complex embedding of h and t, and ◦ is the Hadamard product.

heading: 3 DynaSemble
text: Our goal is to dynamically ensemble k KGC models Mi, which may be textual or structure-based, to maximize performance. Each model Mi assigns a score Mi(h, r, t) to all candidate tails t ∈ E for query q = (h, r, ?). These models are trained independently and their parameters are frozen before ensembling. We formulate the ensemble E as:
E(h, r, t) = k∑
i=1
wi(q)Mi(h, r, t)
where E(h, r, t) is the ensemble score for t given query q = (h, r, ?). We first normalize these scores as described below. Normalization: To bring the distribution of scores assigned by each model Mi over all t ∈ E in the same range for each query, we max-min normalize the scores obtained from all models Mi separately:
Mi(h, r, t)← Mi(h, r, t)− min t′∈E Mi(h, r, t′)
Mi(h, r, t)← Mi(h, r, t)
maxt′∈E Mi(h, r, t′) The scores obtained after normalization lie in the range [0,1] for all models. We next describe the simple model used to learn the query-dependent ensemble weights wi. Model: We extract the following features from the score distribution of each model Mi:
f(Mi, q) = mean t′∈E (Mi(h, r, t′))||var t′∈E (Mi(h, r, t′))
In the above equations, mean() and var() are the standard mean and variance functions respectively,
whose outputs are concatenated to obtain the feature. This choice is driven by the insight that the variance and mean of the distribution of scores computed by any model over E is correlated to the model confidence. A more detailed discussion, along with an exploration of other possible feature sets can be found in Appendix C.
Next, we concatenate these features for all Mi to obtain a final feature vector that is passed to an independent 2-layer MLP (MLPi) for each model Mi to learn query-dependent wi:
wi(q) = MLPi(f(M1, q)||f(M2, q)||...||f(Mk, q))
Intuitively, this concatenation informs each MLP about the relative confidence of all models regarding their predictions, enhancing the ensemble weight computation for corresponding models. Note that our approach is agnostic to models Mi.
Our experiments in this paper mostly involve only one textual model. Therefore, we learn the ensemble weights for the other models with respect to this textual model, which is assigned a fixed weight of 1. This decreases the parameter count while still being as expressive as learning distinct ensemble weights for all models. The method for learning these other weights is unchanged. Loss Function: We train DynaSemble on the validation set (traditionally used to tune ensemble weights) using margin loss between the score of the gold entity and a set of negative samples. The train set is not used since all models are likely to give high-confidence predictions on its triples (Appendix D). If the gold entity is t∗ and the set of negative samples is N, the loss function L for query q = (h, r, ?) is:
L = ∑
t∈N max (E(h, r, t)− E(h, r, t∗) + m, 0)
where m is the margin hyperparameter. This hyperparameter ensures that the generated ensemble weights stay numerically stable during training. In practice, we find that this loss function can be substituted for a cross-entropy loss as well.

heading: 4 Experiments
text: Datasets: We use four datasets for evaluation: WN18RR (Dettmers et al., 2018), FB15k237 (Toutanova and Chen, 2015), CoDexM (Safavi and Koutra, 2020) and YAGO310 (Mahdisoltani et al., 2015). For each triple in the test set, we answer queries (h, r, ?) and (t, r−1, ?) with answers t and h. We report the Mean Reciprocal Rank (MRR) and Hits@k (H@1, H@10) under the filtered measures (Bordes et al., 2013). Details and data statistics are in Appendix A.
Baselines: We use SimKGC ([Sim] in tables) and HittER ([Hit] in tables) as strong textual model baselines. NBFNet ([NBF] in tables) serves as a strong structure-based model baseline. We also present results with RotatE ([RotE] in tables) to showcase the generalisation of our method to KG embedding models. We have reproduced the numbers published by the original authors for these baselines, and use model checkpoints published by the authors2 for [Hit] on the WN18RR and FB15k-237 datasets. Since [NBF] does not scale up to YAGO3-10 with reasonable hyperparameters on our hardware, we omit those results. We represent DynaSemble of models by + in tables.
Experimental Setup: All baseline models are frozen after training using optimal configurations. Ensemble weights are trained on the validation split, using Adam as the optimizer with a learn-
2https://github.com/microsoft/HittER
ing rate of 5.0e-5. We use 10,000 negative samples per query. MLP hidden dimensions are set to 16 and 32 for ensemble of 2 and 3 models respectively. MLP weights are initialized uniformly in the range [0, 2]. DynaSemble training converges in a single epoch, making our method fast and efficient.
Results: We report DynaSemble results in Table 1 (more details in Appendix B). We observe a notable increase in performance after ensembling with [Sim] and [Hit] for both [NBF] and [RotE], which shows that our approach is performant for the ensembling of textual models with both structurebased and KG embedding models. In particular, we obtain 6.8 pt MRR and 8.4 pt Hits@1 improvement with [Sim] + [NBF] over [Sim] on WN18RR. Ensembling with [Sim] results in substantial performance gains even when it is outperformed by structure-based models (on FB15k-237, CoDex-M and YAGO3-10 datasets).
We find that ensembling of [NBF] and [RotE] with [Sim] results in larger improvements than with [Hit] (notably with a 16.4 pt MRR and 15.2 pt Hits@1 gap between [Sim] + [NBF] and [Hit] + [NBF]). Even on the FB15k-237 dataset, where [Hit] outperforms [Sim] by 5.1 pt MRR and 4.6 pt Hits@1, [Sim] + [NBF] narrowly outperforms [Hit] + [NBF] by 0.6 pt MRR and 0.6 pt Hits@1. These observations suggest that [Sim] leverages the textual information in the knowledge graph more effectively than [Hit], thus acting as a better complement to the structure-based models.
On YAGO3-10, where [RotE] outperforms [Sim] by 33.5 pt MRR and 29.9 pt Hits@1, we still obtain 1.3 pt MRR and 1.4 pt Hits@1 gain with [Sim] + [RotE] over [RotE]. Results for [Sim] + [NBF] + [RotE] show that ensembling with [RotE] results in marginal gains over [Sim] + [NBF], obtaining up to 1.1 pt MRR and 0.7 pt Hits@1 gain on CoDex-M. We hypothesize that the gains are marginal due to [RotE]’s ability to implicitly capture and exploit structural information (explored in more detail in Appendix E and F), making it somewhat redundant in the presence of [NBF]. To the best of our knowledge, our best results on the WN18RR, FB15k-237 and CoDex-M datasets are state-of-the-art.

heading: 5 Analysis
text: We perform four further analyses to answer the following questions: Q1. How does the behavior of textual and structure-based models vary with reachability? Q2. Do the weights learned by
DynaSemble follow expected trends with reachability? Q3. Does DynaSemble improve performance over conventional model-combination techniques? Q4. How does DynaSemble of a textual and structure-based model compare to DynaSemble of two textual or structure-based models?
Reachability Ablation: To answer Q1, we divide the test set for each dataset into ‘reachable’ and ‘unreachable’ splits. A triple (h, r, t) is part of the reachable split if t can be reached from h with a path of length at most l (= 2) in the KG. If not, it is put in the unreachable split. We present splitwise results for [NBF], [Sim] and [Sim]+[NBF] on the WN18RR and FB15k-237 datasets in Table 2.
We observe that [Sim] outperforms [NBF] on the unreachable split (by up to 25.8 pt MRR and 24.0 pt Hits@1 for WN18RR), while [NBF] outperforms [Sim] on the reachable split (by up to 13.3 pt MRR and 12.6 pt Hits@1 for FB15k-237). This is because [NBF] can easily exploit knowledge of the KG structure to perform well on the reachable split, while [Sim] can instead use BERT to leverage textual descriptions to perform better on the unreachable split. The performance gap between [Sim] and [NBF] on the unreachable split is notably larger for WN18RR than for FB15k-237, which can be attributed to the sparsity of the WN18RR dataset, the unreachable split for which also has several entities unseen in the training data. In such cases, [Sim] achieves reasonable performance, whereas [NBF] lacks any paths for reasoning. Our ensemble obtains substantial gains over best individual models on both splits, with 4.2 pt MRR and 4.9 pt Hits@1 gain on the reachable split and 5.0 pt MRR and 4.7 pt Hits@1 gain on the unreachable split for WN18RR. More details in Appendix E.
Analysis of Ensemble Weights: To answer Q2, we study the mean of the ensemble weight w2 for [Sim] + [NBF] over the queries in the reachable and unreachable splits of the datasets we use. We observe that this mean is consistently larger (by a margin of up to 17% for WN18RR) on the reachable split
than the unreachable split. This is because [NBF] tends to give better performance on the reachable split, and a larger w2 gives it more importance in the ensemble. More details and numbers are in Appendix G, including results analyzing the nontrivial standard deviation of w2. Comparison with Conventional Techniques: To answer Q3, we present results for static ensembling and re-ranking using [Sim] and [NBF] for WN18RR and FB15k-237 datasets in Table 3. ‘Static ensembling’ involves manually tuning the ensemble weight as a constant on the validation set. For [NBF]-[Sim] re-ranking (Han et al., 2020), we consider the top 100 entities by score from [NBF] for each query and re-rank them according to their [Sim] score. The rest of the entities are ranked according to [NBF]. We present results for [Sim][NBF] re-ranking as well for comparison. We also include results for the ensembling heuristic used in KGT5 (Saxena et al., 2022) (KGT5 Ensemble), which uses the textual model to answer queries that have no answers in the training set and the structure-based model to answer all other queries.
We find that DynaSemble outperforms reranking, ‘KGT5 ensembling’ and ‘static ensembling’ across datasets. Notably, DynaSemble beats reranking by 9.7 pt MRR and 9.8 pt Hits@1, KGT5 ensembling by 5.6 pt MRR and 8.2 pt Hits@1, and static ensembling by 1.0 pt MRR and 1.4 pt Hits@1 on the WN18RR dataset. This highlights the utility of DynaSemble in comparison to existing model combination heuristics. We also perform a paired student’s t-test to validate the statistical significance of the gains obtained from DynaSemble over "static ensembling", resulting in a t-value of 8.9 (p < 0.001) for the WN18RR dataset and 6.7 (p < 0.01) for the CoDex-M dataset. Further details can be found in Appendix H. Impact of Types of Ensembled Models: To an-
swer Q4, we contrast results for [Sim] + [NBF] (DynaSemble of a textual and structure-based model) against [Sim] + [Hit] (DynaSemble of two textual models) and [NBF] + [RotE] (DynaSemble of two structure-based models) for WN18RR and FB15k-237 datasets in Table 10.
DynaSemble achieves 1.7 pt MRR and 1.7 pt Hits@1 improvements over best individual models ([Sim]) for [Sim] + [Hit] on the WN18RR dataset and 1.8 pt MRR and 1.8 pt Hits@1 improvements over best individual models ([NBF]) for [NBF] + [RotE] on the FB15k-237 dataset, showing that DynaSemble generalizes to these settings. We further note that [Sim] + [NBF] outperforms [Sim]+[Hit] by 5.1 pt MRR and 5.7 pt Hits@1 and [NBF]+[RotE] by 17.8 pt MRR and 16.9 pt Hits@1 on the WN18RR dataset. This trend persists for the FB15k-237 dataset, where [Sim]+[NBF] marginally outperforms [NBF] + [RotE] despite [RotE] outperforming [Sim] by 1.6 pt MRR and 0.8 pt Hits@1 individually. These observations are in line with our insights regarding the complementary strengths of textual and structure-based KGC approaches, which results in larger gains when models corresponding to different approaches are ensembled.

heading: 6 Conclusion and Future Work
text: We present DynaSemble: a simple, novel, modelagnostic and lightweight dynamic ensembling approach for KGC, while also highlighting the complementary strengths of textual and structure-based KGC models. Our state-of-the-art results for a DynaSemble of SimKGC and NBFNet over three standard KGC datasets (WN18RR, FB15k-237 and CoDex-M) creates a new competitive ensemble baseline for the task. We release all code for future research. Future work includes tighter trainingtime unification methods, and extensions to temporal (Jain et al., 2020; Singh et al., 2023) and multilingual KGC models (Chakrabarti et al., 2022).
Limitations
We do not consider Neuro-Symbolic KGC approaches in this work, which have also recently started to give competitive results with other KGC approaches, through models such as RNNLogic (Qu et al., 2021) and extensions (Nandi et al., 2023). Our experiments consider ensembling of one textual model with multiple structural models. This is because most textual models in recent KGC literature are not competitive with SimKGC (Wang et al., 2022), therefore we do not expect large gains by including them along with SimKGC in an ensemble. The ensembling of multiple textual models with multiple structure-based models would be a possible future work. In models with substantial validation splits, learning query embeddings to augment the features we use to compute ensemble weights is also a possibility.
Ethics Statement
We anticipate no substantial ethical issues arising due to our work on ensembling textual and structure-based models for KGC. Our work relies on other baseline models for ensembling. This may propagate any bias present in these baseline models, however ensembling may also reduce these biases.

heading: Acknowledgements
text: This work is supported by IBM AI Horizons Network, grants from Google, Verisk, and Huawei, and the Jai Gupta chair fellowship by IIT Delhi. We thank the IIT-D HPC facility for its computational resources.

heading: A Data Statistics and Evaluation Metrics
text: Table 5 outlines the statistics of the datasets utilized in our experimental section. We utilize the standard train, validation and test splits for all datasets.
Metrics: For each triplet (h, r, t) in the KG, typically queries of the form (h, r, ?) and (?, r, t) are created for evaluation, with corresponding answers t and h. We represent the (?, r, t) query as (t, r−1, ?) with the same answer h, where r−1 is the inverse relation for r, for both training and testing. Given ranks for all queries, we report the Mean Reciprocal Rank (MRR) and Hit@k (H@k, k = 1, 10) under the filtered setting in the main paper and two additional metrics: Mean Rank (MR) and Hits@3 in the appendices.

heading: B Detailed Results on Proposed Ensemble
text: Here we present our experimental setup for the main results presented in Table 1. Since loading both NBFNet and the two BERT encoders from SimKGC into GPU at the same time is too taxing for our hardware, we dump the embeddings of all possible (h, r) and t from SimKGC to disk, and use them for training our ensemble. SimKGC is reliant on textual descriptions for performance. The original authors provide descriptions for WN18RR and FB15k-237, while descriptions for CoDex-M are available as part of the dataset. Since YAGO310 does not contain any descriptions, we treat the entity names as their descriptions. SimKGC also has a structural re-ranking step independent of its biencoder architecture, which we do not utilize as we expect our ensembling method to subsume it.
Next, we present results in Table 6 that are supplementary to results already presented in Table 1. In addition to MRR, Hits@1 and Hits@10 considered in Table 1, we also present numbers for Mean Rank (MR) and Hits@3 in Table 6. As before, the ’+’ sign represents our ensemble approach. We also consider an additional KG embedding model ComplEx (Trouillon et al., 2016) ([Comp] in tables) in this section and present complete results for it.
We observe that for the two new metrics considered in Table 6, we also obtain substantial performance gains on ensembling, notably a gain of 5.2
pt Hits@3 and 67.4% MR on the WN18RR dataset with [Sim] + [NBF] over [Sim]. Further, we observe that [Sim]+[Comp] consistently outperforms both [Sim] and [Comp], (by up to 2.2 pt MRR for CoDex-M). We also present complete numbers for [Sim]+[NBF]+[Comp] and [Sim]+[NBF]+[RotE] here.

heading: C Feature Selection for Ensemble Weight Learning
text: In this section, we justify our choice of features for learning ensemble weights. We focus on NBF and Sim for this purpose. We claim that after our normalization procedure, a model has lower mean and variance when it is confident about the validity of its top predictions. To highlight this, we present distribution of the normalized scores over all candidate entities for NBF and Sim for two queries in the WN18RR dataset: one from the reachable split and the other from the unreachable split. The query for Figure 1a lies in the reachable split while the query for Figure 1b lies in the unreachable split. The entity id of the gold answer is marked with a red vertical line in both cases.
We notice that for the query in the reachable split, [NBF] is very confident about its top prediction. Therefore, it scores the gold answer significantly higher than the other candidates. Upon normalization, this causes the other entities to have comparatively smaller values (mostly in the range [0-0.4]), with a tighter spread. In comparison, for the query in the unreachable split, [NBF] cannot predict the gold answer confidently. This results in a much larger spread of scores across entities, with a lot of extreme values close to 1 indicating that the model is unable to conclusively determine which entity is the correct one. We choose the mean and variance as features because they will be able to distinguish between these two distributions, with their values being substantially smaller in the first case where [NBF] is confident about the predictions.
[Sim] also has these properties, albeit to a lesser degree. This is because SimKGC cannot exploit the KG structure, and therefore has to draw conclusions based on textual descriptions, which can point to several candidate answers of seemingly comparable validity. This results in the score distributions having a higher spread and a lower margin between the score of the top prediction and the other candidates. Therefore, the relative values of these mean and variance features can also inform the MLPs about the relative confidence of the models about their output, allowing them to compute ensemble weights for corresponding models as necessary.
As validation, we present the average of the mean and variance features from [NBF] over all test queries in the reachable and unreachable split for the WN18RR, FB15k-237 and CoDex-M datasets in Table 8.
We find that the average of the mean and variance features is up to 21% lower (for WN18RR) on the reachable split than the unreachable split, allowing the MLPs to distinguish between the splits based on score distribution statistics alone. We finally present results of experiments with other similar sets of features as input to the MLP in Table 9.
We find that features that are created according to the reasoning above (Mean + Var and Mean + Std) perform better as compared to other features (Zip, Top 10) across datasets and metrics.

heading: D Choice of Training Data for Dynamic Ensemble
text: In this section, we expand upon the choice of using the validation split to train the dynamic ensemble weights, which is usually used for manually tuning the constant ensemble weights in static ensembling. We present results for dynamic ensembles trained on three splits of data: i) the full training split (FullTrain) ii) the validation split (Validation, this corresponds to the dynamic ensemble results in the paper) iii) a randomlychosen 1% split of the training data, which is heldout while training the base models before ensembling (Held− OutTrain). We present results for [Sim] + [NBF]) trained on these three splits of the WN18RR and FB15k-237 datasets in Table ??.
We find that FullTrain results in less than 0.1 pt MRR improvement over best individual models for both datasets. This is because both base
models are capable of fitting the training data with near-perfect performance. As a result, both models showcase high confidence about their outputs and the dynamic ensemble is unable to learn any correlations between model confidence and corresponding ensemble weight for the test split. Therefore, the ensemble weights for each model converge rapidly to 0 or 1 during training.
We additionally find that Held− OutTrain results in performance within 0.3 pt MRR of Validation in both datasets. This small drop in performance might be caused by the slightly smaller amount of data being used to train both the base models and the dynamic ensemble, as compared to the original setting. This shows that holding out part of the training data is an effective strategy to train the dynamic ensemble on datasets that do not have a validation split, as the small drop in performance of the base model is amply compensated by the gains from ensembling.

heading: E Detailed Reachability Ablation
text: In this section we discuss further results of the experiment done to answer Q1 in Section 5. The results presented in Table 7 are supplementary to the results presented in Table 2 where in addition to the MRR, Hits@1, Hits@10 metrics already presented in Table 2, we present results over one additional metric, MR. Additionally, we present the results on the ‘reachable’ and ‘unreachable’ split of CoDexM, and for [RotE] and [Comp] on all datasets. We observe that [Sim] has up to 76% lower MR than [NBF] on the unreachable split while [NBF] has up to 83.3% lower MR than [Sim] on the reachable split over all the three datasets (both statistics mentioned are for WN18RR). The ensemble of [Sim]+[NBF] brings the MR down further, notably obtaining a gain of 42.5% on reachable split and 66% on unreachable split over best individual models for the WN18RR dataset. We also observe that [RotE] and [Comp] show similar variation of performance across splits when compared to [NBF], performing notably better on the reachable split as compared to the unreachable split across datasets. This indicates that these KG embedding models are also dependent on KG structure and paths between the head and gold tail to some extent for performance. We investigate this in more detail in Appendix F.

heading: F RotatE as a Structure-Based Model
text: We claim that despite structure not being explicitly involved in the training of [RotE], it is still capable of capturing the structure of the KG to some extent in its relation embeddings by exploiting the compositionality inherent in its scoring function. Consider an example in which (h1, r1, h2), (h2, r2, h3) and (h1, r3, h3) are all present in the KG. Let Tr(h) be the vector obtained after rotating the embedding of h by the complex rotation defined by r. During training, Tr1(h1) will be brought close to the embedding of h2 and Tr2(h2) will be brought close to the embedding of h3. As a result, Tr2(Tr1(h1)) will be brought close to h3. Upon training on (h1, r3, h3), Tr3(h1)) will also be brought close to h3. However, the relative positions of h1 and h3 on the complex plane already contain information about Tr2 ◦ Tr1 , which is used while training Tr3 . As more such examples are seen over multiple epochs, Tr3 will eventually be brought closer to the composed rotation Tr2 ◦ Tr1 . Therefore, when query (h, r3, ?) is seen at test time, the model will be more likely to return candidates t which are connected to h through a path in the KG involving relations r1 and r2, making it structure dependent.
Of course, this phenomenon is not limited to paths of length 2, but can encode paths of longer length as well. We also expect only the most common paths to be captured through this mechanism, since multiple such paths have to be encoded by the same relation embedding. To validate these claims, we perform an experiment where we exhaustively mine the dataset for cases where (h1, r1, h2) is present in the KG, alongside an entity h3 such that (h1, r2, h3) and (h3, r3, h2) are also present in the KG. This essentially considers all the cases where there is a path involving r2 and r3 that is closed by r1. We enumerate all such cases for each triple (r1, r2, r3) and filter out those triples that have less than 20 occurrences in the KG. For each of the remaining triples, we take a random vector and transform it according to Tr2 ◦ Tr3 . We then report the r such that transforming the same vector according to Tr moves it closest to the result obtained on transforming it according to Tr2 ◦ Tr3 . We expect r to be r1 for a majority of the triples based on our claims. We report accuracies obtained through this experiment for [RotE] on the WN18RR, FB15k237 and CoDex-M datasets in Table 11.
We find that the accuracies are substantially better than the random baseline of 1NumberofRelations for all datasets (which is 9.1% for WN18RR, 0.4% for FB15k-237 and 1.4% for CoDex-M). [Sim] is not capable of capturing this notion, since it encodes (h, r) together using BERT, not as a composition of h and r embeddings. Therefore, we find that its behavior is independent of the split in which the query under consideration is present.

heading: G Reachability Trends of Ensemble Weights
text: The aim of this section is to further discuss the results of the experiment done to answer Q2 in Section 5. The results in Table 13 present the mean and standard deviation of ensemble weights w2 over the queries in the reachable and unreachable split for the WN18RR, CoDex-M and FB15k-237 datasets. The weight discussed in these tables is w2 in the ensemble defined as w1[Sim] + w2[NBF] (with w1 = 1) according to Section 3. We observe that across all datasets, the average weight for reachable split is higher than the weight of unreachable split (up to 17% higher for WN18RR), thus reinforcing the fact that our approach gives more weightage to [NBF] on the reachable split across datasets. The standard deviation of w2 is also non-trivial on all splits of all datasets, showing that our approach is capable of adjusting it as required by individual queries.
To investigate why our ensemble weights are not binary and are quite consistent with each other for each split, we contrast [Sim] + [NBF] with a model that selects NBFNet on the reachable split and SimKGC on the unreachable split: Split− Select. We present results in Table 14.
We find that dynamic ensembling performs better than the oracle by 4.8 pt MRR. This is because structure-based models tend to rank more connected tails higher, while text-based models rank tails based solely on their textual descriptions. Therefore, a soft ensemble can take advantage of both structural and textual information to perform better than a mixture-of-experts model that simply selects one of the base models based on expected performance trends.

heading: H Significance of Improvements with Dynamic Ensembling
text: We first perform a paired student’s t-test on the MRR over a 5-fold split for [Sim] + [NBF] to confirm that the gains obtained by our approach over static ensembling are statistically significant. We present the results in Table 12.
We obtain a t-value of 8.9 for WN18RR and 6.7 for CoDex-M. With a p-value of 0.05, the reference value is 2.78. Therefore, the gains obtained by our model over static ensembling are indeed statistically significant.
The performance of an ensemble is ultimately dependent on the performance of the individual models. To obtain an estimate of the best possible performance that can be obtained from model fusion, we present results in Table 15 with [Sim] + [NBF] for a model that selects the most performant model for each query (BEST).
We find that the results for our dynamic ensemble are only up to 2.3 MRR pts behind a theoretical oracle that always knows the best model for each query, indicating that most of the potential for improvement through late fusion techniques has been obtained through dynamic ensembling.

heading: Future Work
text: We release all code for future research. Future work includes tighter trainingtime unification methods, and extensions to temporal (Jain et al. , 2020; Singh et al. , 2023) and multilingual KGC models (Chakrabarti et al. , 2022). Limitations
We do not consider Neuro-Symbolic KGC approaches in this work, which have also recently started to give competitive results with other KGC approaches, through models such as RNNLogic (Qu et al. , 2021) and extensions (Nandi et al. , 2023). Our experiments consider ensembling of one textual model with multiple structural models. This is because most textual models in recent KGC literature are not competitive with SimKGC (Wang et al. , 2022), therefore we do not expect large gains by including them along with SimKGC in an ensemble. The ensembling of multiple textual models with multiple structure-based models would be a possible future work. In models with substantial validation splits, learning query embeddings to augment the features we use to compute ensemble weights is also a possibility