Croppable Knowledge Graph Embedding
Authors: Yushan Zhu, Wen Zhang, Zhiqiang Liu, Mingyang Chen, Lei Liang, Huajun Chen
Accepted at: acl25

Croppable Knowledge Graph Embedding

Yushan Zhu♣◊, Wen Zhang♣◊, Zhiqiang Liu♣◊, Mingyang Chen♣,
Lei Liang♣, Huajun Chen♣◊◊*
♣ Zhejiang University *Ant Group
◊ Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph
◊◊ Zhejiang Key Laboratory of Big Data Intelligent Computing
{yushanzhu, zhang.wen, zhiqiangliu, mingyangchen, huajunsir}@zju.edu.cn

Abstract
Knowledge Graph Embedding (KGE) is a common approach for Knowledge Graphs (KGs) in AI tasks. Embedding dimensions depend on application scenarios. Requiring a new dimension means training a new KGE model from scratch, increasing cost and limiting efficiency and flexibility. In this work, we propose a novel KGE training framework MED. It allows one training to obtain a croppable KGE model for multiple scenarios with different dimensional needs. Sub-models of required dimensions can be directly cropped and used without extra training. In MED, we propose a mutual learning mechanism to improve the low-dimensional sub-models and make high-dimensional sub-models retain the low-dimensional sub-models’ capacity, an evolutionary improvement mechanism to promote the high-dimensional sub-models to master the triple that the low-dimensional sub-models can not, and a dynamic loss weight to adaptively balance the multiple losses. Experiments on 4 KGE models across 4 standard KG completion datasets, 3 real-world scenarios using a large-scale KG, and extending MED to the BERT language model demonstrate its effectiveness, high efficiency, and flexible extensibility.

1 Introduction
Knowledge Graphs (KGs) consist of triples in the form of (head entity, relation, tail entity), abbreviated as (h, r, t). KGs are widely used in recommendation systems (Zhu et al., 2021), information extraction (Daiber et al., 2013), question answering (Diefenbach et al., 2018), etc. A common KG application way is knowledge graph embedding (KGE) (Bordes et al., 2013; Sun et al., 2019b), which maps KG entities and relations into continuous vector spaces for various tasks.

Higher-dimensional KGEs have more expressive power and better performance but come with more parameters, demanding more storage and computing resources (Zhu et al., 2022; Sachan, 2020). The suitable KGE dimensions vary by device or scenario. As Fig. 1 shows, large remote servers can support high-dimensional KGEs, while small-to-medium terminal devices like in-vehicle systems or smartphones can only handle low-dimensional ones due to resource limits. Thus, people prefer training high-quality KGEs with appropriate dimensions. However, a new KGE must be trained from scratch when a new dimension is needed. Especially for low-dimensional KGEs, model compression techniques like knowledge distillation (Hinton et al., 2015; Zhu et al., 2022) are required to ensure performance. This raises training costs and restricts KGE’s efficiency and flexibility across scenarios.

Thus a new concept "croppable KGE" is proposed and we are interested in the research question that is it possible to train a croppable KGE, with which KGEs of various required dimensions can be cropped out of it and directly used without extra training, and perform well?

In this work, our croppable KGE learning idea is to train an entire KGE with multiple different-dimensional sub-models sharing embedding parameters and trained simultaneously. The aim is for low-dimensional sub-models to benefit from high-dimensional ones, and high-dimensional sub-models to retain low-dimensional ones’ ability and learn what they can’t. Based on this, we propose a croppable KGE training framework MED, with three main modules: Mutual learning mechanism, Evolutionary improvement mechanism, and Dynamic loss weight. Specifically, the mutual learning mechanism, based on knowledge distil-

![A diagram showing Knowledge Graph, KGEs, and Devices/Scenes, with the question "Can we train a 'croppable KGE' with elastic dimension that suitable for various scenes?"](page_760_320_480_240.png)

Figure 1: Diverse KGE dimensions for a KG.
lation, enables neighbor sub-models to learn from each other, improving low-dimensional sub-model performance and helping high-dimensional ones retain low-dimensional abilities. The evolutionary improvement mechanism helps high-dimensional focus on and master triples that low-dimensional ones can’t correctly predict. The dynamic loss weight adaptively balances sub-models’ losses according to dimension to enhance overall performance.

We evaluate the effectiveness of MED by implementing it on four typical KGE methods and four standard KG datasets. We prove its practical value via a real-world large-scale KG and downstream tasks, and demonstrate its extensibility on BERT (Devlin et al., 2019) and GLUE (Wang et al., 2019) benchmarks. Experimental results show that: (1) MED trains a croppable KGE for various dimensional needs, with high-performing, parameter-shared sub-models ready for direct use. (2) MED has far higher training efficiency than independent training or knowledge distillation. (3) MED can be extended to other neural networks and performs well. (4) The three modules in MED are crucial for optimal performance. In summary, our contributions are as follows:

• We introduce the research question and task of training croppable KGE, allowing direct use of differently-dimensional KGEs.

• We propose a novel framework MED, including a mutual learning mechanism, an evolutionary improvement mechanism, and a dynamic loss weight, to ensure the overall performance of all sub-models.

• We show that MED’s sub-models perform well, with low-dimensional ones outperforming KGEs trained by SOTA distillation methods. MED also shows good performance and extensibility in real-world applications and other types of neural networks.

2 Related Work

This work is to achieve a croppable KGE for different dimensional needs. A common way to get a good-performance KGE of the target dimension is using knowledge distillation with a high-dimensional powerful teacher KGE. So, we focus on two relevant research areas: knowledge graph embedding and knowledge distillation.

2.1 Knowledge Graph Embedding

Knowledge graph embedding (KGE) maps KG entities and relations into continuous vector spaces for downstream tasks. TransE (Bordes et al., 2013), a representative translation-based KGE, treats relations as translations between entities. Its variants include TransH(Wang et al., 2014), TransR(Lin et al., 2015), TransD(Ji et al., 2015), etc. RESCAL(Nickel et al., 2011), based on vector decomposition, was followed by improvements like DistMult(Yang et al., 2015), ComplEx(Trouillon et al., 2016), and SimplE(Kazemi and Poole, 2018). RotatE(Sun et al., 2019b), a rotation-based method, views relations as rotations between entities, similar to QuatE(Zhang et al., 2019) and DihEdral(Xu and Li, 2019). PairRE(Chao et al., 2021) uses two relation vectors for complex pattern encoding. With neural network development, KGEs based on graph neural networks (GNNs) (Dettmers et al., 2018; Nguyen et al., 2018; Schlichtkrull et al., 2018; Vashishth et al., 2020) also emerged. While KGEs are simple and effective, a key challenge remains: Different scenarios demand different KGE dimensions based on device resources. Training a new KGE model from scratch for each dimension requirement hikes training costs and restricts flexibility in serving diverse scenarios.

2.2 Knowledge Distillation

High-dimensional KGEs possess strong expressive power thanks to numerous parameters. However, they demand substantial storage and computing resources, rendering them unsuitable for all scenarios, particularly those involving small devices. To address this, a prevalent approach is to compress a high-dimensional KGE into target dimension via knowledge distillation (Hinton et al., 2015; Mirzadeh et al., 2020) and quantization (Bai et al., 2021; Stock et al., 2021) technology.

Quantization replaces continuous vector representations with lower-dimensional discrete codes. TS-CL (Sachan, 2020) was the first to apply quantization in KGE compression. LightKG (Wang et al., 2021a) uses a residual module to create diversity in codebooks. Yet, as quantization doesn’t boost inference speed, it’s still not suitable for devices with limited computing resources.

Knowledge distillation (KD) is widely used in Computer Vision (Mirzadeh et al., 2020) and Natural Language Processing (Devlin et al., 2019; Sun et al., 2019a) to shrink model size and boost infer-
ence speed. Its core is using a large teacher model’s output to guide a small student model’s training. DualDE (Zhu et al., 2022) is a representative KD-based work to transfer knowledge from high- to low-dimensional KGE, considering teacher-student mutual influences. MulIDE (Wang et al., 2021b) transfers knowledge from multiple hyperbolic KGE models to one student. ISD (Zhou et al., 2022b) improves low-dimensional KGE by playing teacher and student roles alternatively. IterDE (Liu et al., 2023) introduces iterative distillation for smooth knowledge transfer. Other KG-related distillation works include PMD (Fan et al., 2024), which applies distillation to pre-trained language models for KG completion, IncDE (Liu et al., 2024), using distillation between same-dimensional models at different times for incremental learning, and SKDE (Xu et al., 2024), which proposes self-knowledge distillation to avoid a complex teacher model. Among these, DualDE (Zhu et al., 2022) and IterDE (Liu et al., 2023) are most relevant to our work as they compress high-dimensional teacher into low-dimensional student. In this work, we propose a novel KD-based KGE training framework MED, one training can obtain a croppable KGE that meets multiple dimensional requirements.

3 Preliminary

<table>
  <tr>
    <th>KGE method</th>
    <th>Scoring Function \( f(h, r, t) \)</th>
  </tr>
  <tr>
    <td>TransE (Bordes et al., 2013)</td>
    <td>\( -\| h + r - t \| \)</td>
  </tr>
  <tr>
    <td>Simile (Kazemi and Poole, 2018)</td>
    <td>\( \frac{1}{2}(<h^H, r^T> + <t^H, r^{-1}, r^T>) - \| h + r - t \| \)</td>
  </tr>
  <tr>
    <td>RotatE (Sun et al., 2019b)</td>
    <td>\( -\| h \circ r^T - t \circ r^T \| \)</td>
  </tr>
  <tr>
    <td>PairRE (Chao et al., 2021)</td>
    <td>\( -\| h \circ r^T - t \circ r^T \| \)</td>
  </tr>
</table>

Table 1: Score functions.

Knowledge graph embedding (KGE) methods use a scoring function \( f \) to represent entities and relations in a continuous vector space. Given a KG \( G = (\mathcal{E}, \mathcal{R}, \mathcal{T}) \) where \( \mathcal{E} \), \( \mathcal{R} \) and \( \mathcal{T} \) are sets of entities, relations and observed triples, for a triple \( (h, r, t) \) (\( h \in \mathcal{E}, r \in \mathcal{R}, t \in \mathcal{E} \)), we use the triple scoring function \( s_{(h, r, t)} = f(h, r, t) \) (with entity and relation embeddings as input) to measure triple plausibility in the embedding space. Table 1 summarizes scoring functions of some popular KGE methods, where \( \circ \) is the Hadamard product, \( <x^1, ..., x^k> = \sum_i x^1_i ... x^k_i \) is the generalized dot product. Higher triple scores mean the model is more likely to consider triples true. The optimization objective of the KGE model is

\[
L_{KGE} = - \sum_{(h, r, t) \in \mathcal{T} \cup \mathcal{T}^-} y \log \sigma(s_{(h, r, t)}) \\
+ (1-y) \log(1-\sigma(s_{(h, r, t)})),
\]

where \( \mathcal{T}^- = \mathcal{E} \times \mathcal{R} \times \mathcal{E} \setminus \mathcal{T} \) is negative triple set, \( \sigma \) is Sigmoid function, \( y \) is ground-truth label of \( (h, r, t) \), \( y = 1 \) for positive and 0 for negative one.

4 MED Framework

As in Fig. 2, our croppable KGE framework MED contains \( n \) sub-models of different dimensions, \( M_i (i = 1, 2, ..., n) \) with dimension of \( d_i \). Each \( M_i \) consists of the first \( d_i \) dimensions of the full embedding. The triple \( (h, r, t) \) score from \( M_i \) is \( s^i_{(h, r, t)} = f([h[0:d_i], r[0:d_i], t[0:d_i]]) \), where \( [h[0:d_i]] \) are the first \( d_i \) elements of vector \( h \). The parameters of sub-model \( M_i \) are shared by higher-dimensional \( M_j (i < j \leq n) \). The number of sub-models \( n \) and each sub-model’s dimension \( d_i \) can be set per actual needs. For low-dimensional sub-models, we aim to maximize performance. High-dimensional sub-models should not only replicate the capabilities of low-dimensional ones but also learn what low-dimensional ones can’t, that is, correctly predicting triples mispredicted by low-dimensional sub-models. MED is based on knowledge distillation (Hinton et al., 2015; Tang et al., 2019; Devlin et al., 2019) technique, where student fits hard (ground-truth) label and soft label from teacher simultaneously. In MED, we first introduce a mutual learning mechanism. This helps low-dimensional sub-models learn from high-dimensional ones for better performance, and vice versa, so high-dimensional sub-models retain the

![Overview of MED.](page_1016_172_388_312.png)

Figure 2: Overview of MED.
capabilities of low-dimensional ones. Next, we present an evolutionary improvement mechanism to let high-dimensional sub-models acquire knowledge that low-dimensional ones can’t learn well. Finally, we train MED with dynamic loss weight to adaptively balance multiple optimization goals of sub-models.

4.1 Mutual Learning Mechanism

We consider each sub-model \( M_i \) as the student of its higher-dimensional neighbor \( M_{i+1} \) for better performance, as high-dimensional KGEs with more parameters are more expressive (Sachan, 2020; Zhu et al., 2022). We also make \( M_i \) the student of its lower-dimensional neighbor \( M_{i-1} \), enabling the higher-dimensional sub-model to review and retain the lower-dimensional one’s capabilities. So, pairwise neighbor sub-models are both teacher and student, learning from each other. Mutual learning loss between each pair of neighbor sub-models is

\[
L_{ML}^{i-1,i} = \sum_{(h,r,t) \in \mathcal{T} \cup \mathcal{T}^-} d_\delta \left( s_{(h,r,t)}^{i-1}, s_{(h,r,t)}^i \right), \quad 1 < i \leq n,
\]

where \( s_{(h,r,t)}^i \) is the score of triple \( (h, r, t) \) from sub-model \( M_i \), indicating the triple’s existence likelihood. \( d_\delta \) is Huber loss (Huber and Peter, 1964) with \( \delta = 1 \), often used in KGE knowledge distillation (Zhu et al., 2022). In MED, each sub-model learns only from its neighbor sub-models. This not only cuts down training computational complexity but also keeps the dimension gap between each teacher-student pair relatively small. A large dimension gap between them can undermine the distillation effect (Mirzadeh et al., 2020; Zhu et al., 2022), making our approach crucial and effective.

4.2 Evolutionary Improvement Mechanism

In knowledge distillation (Hinton et al., 2015), the hard (ground-truth) label is a key supervision signal during training. High-dimensional sub-models need to master triples that low-dimensional sub-models can not, that is correctly predicting positive (negative) triples mispredicted as negative (positive) by low-dimensional sub-models. In MED, for a triple \( (h, r, t) \), sub-model \( M_i \)'s optimization weight for it depends on triple score from \( M_{i-1} \).

For a positive triple, \( M_i \)'s optimization weight is negatively correlated with the score from \( M_{i-1} \). If \( M_{i-1} \) gives a high score (correctly identifying it as positive), \( M_i \)'s optimization weight for it is low. If \( M_{i-1} \) gives a low score (misidentifying it as negative), \( M_i \)'s optimization weight is high as \( M_{i-1} \) can’t predict it well. The optimization weight of \( M_i \) for the positive triple is

\[
pos_{h,r,t}^i = \begin{cases}
    1/|\mathcal{T}_{batch}|, & i = 1 \\
    \frac{\exp w_1 s_{(h,r,t)}^{i-1}}{\sum_{(h,r,t) \in \mathcal{T}_b} \exp w_1 s_{(h,r,t)}^{i-1}}, & 1 < i \leq n,
\end{cases}
\]

where \( s_{(h,r,t)}^{i-1} \) is the score for triple \( (h, r, t) \) from \( M_{i-1} \), \( \mathcal{T}_b \) is positive triple set in a batch, and \( w_1 \) is a learnable scaling parameter. Conversely, for a negative triple, \( M_i \)'s optimization weight is positively correlated with the score from \( M_{i-1} \), as also used in (Sun et al., 2019b). The optimization weight of \( M_i \) for negative triple is

\[
neg_{h,r,t}^i = \begin{cases}
    1/|\mathcal{T}_{batch}^-|, & i = 1 \\
    \frac{\exp w_2 s_{(h,r,t)}^{i-1}}{\sum_{(h,r,t) \in \mathcal{T}_b^-} \exp w_2 s_{(h,r,t)}^{i-1}}, & 1 < i \leq n,
\end{cases}
\]

where \( \mathcal{T}_b^- \) is negative triple set in a batch, and \( w_2 \) is a learnable scaling parameter. The evolutionary improvement loss of the sub-model \( M_i \) is

\[
L_{EI}^i = - \sum_{(h,r,t) \in \mathcal{T} \cup \mathcal{T}^-} pos_{h,r,t}^i \cdot y \log \sigma(s_{(h,r,t)}^i) \\
+ neg_{h,r,t}^i \cdot (1-y) \log(1-\sigma(s_{(h,r,t)}^i)),
\]

In each sub-model, different hard-label loss weights are set for different triples, and high-dimensional sub-models focus more on triples low-dimensional ones can’t learn well.

4.3 Dynamic Loss Weight

As MED optimizes multiple sub-models, we use dynamic loss weights during training. At first, low-dimensional sub-models prioritize learning from high-dimensional ones to improve performance, relying more on soft label information. Thus, for them, evolutionary improvement loss should be less than mutual learning loss. Conversely, high-dimensional sub-models should focus more on what low-dimensional ones can’t correctly predict and reduce the impact of low-quality low-dimensional outputs. They rely more on hard label information, so their evolutionary improvement loss should exceed mutual learning loss. For a teacher-student pair, the mutual learning loss affects both models equally. We set different evolutionary improvement loss weights for sub-models,
and the final training loss function of MED is

\[
L = \sum_{i=2}^{n} L_{ML}^{i-1,i} + \sum_{i=1}^{n} \exp(\frac{w_3 \cdot d_i}{d_n}) \cdot L_{EI}^i,
\]

where \( w_3 \) is a learnable scaling parameter, and \( d_i \) is the dimension of the \( i \)th sub-model.

5 Experiment

We evaluate MED on typical KGE and GLUE benchmarks and address these research questions:
(RQ1) Can MED train a croppable KGE at once, with multiple differently-dimensional sub-models cropped from it, all performing well? (RQ2) Can MED yield parameter-efficient KGE models? (RQ3) Does MED work in real-world applications? (RQ4) Can MED be extended to non-KGE neural networks?

5.1 Experiment Setting

5.1.1 Dataset and KGE methods

MED is universal and applicable to any KGE method with a triple score function. We take 4 common KGE methods in Table 1 as examples: TransE (Bordes et al., 2013), SimplE (Kazemi and Poole, 2018), RotatE (Sun et al., 2019b) and PairRE (Chao et al., 2021).

<table>
  <tr>
    <th>Dataset</th>
    <th>#Ent.</th>
    <th>#Rel.</th>
    <th>#Train</th>
    <th>#Valid</th>
    <th>#Test</th>
  </tr>
  <tr>
    <td>WN18RR</td>
    <td>40,943</td>
    <td>11</td>
    <td>86,835</td>
    <td>3,034</td>
    <td>3,134</td>
  </tr>
  <tr>
    <td>FB15K237</td>
    <td>14,541</td>
    <td>237</td>
    <td>272,115</td>
    <td>17,535</td>
    <td>20,466</td>
  </tr>
  <tr>
    <td>CoDE-X-L</td>
    <td>77,951</td>
    <td>69</td>
    <td>551,193</td>
    <td>30,622</td>
    <td>30,622</td>
  </tr>
  <tr>
    <td>YAGO3-10</td>
    <td>123,143</td>
    <td>37</td>
    <td>1,079,040</td>
    <td>4,978</td>
    <td>4,982</td>
  </tr>
  <tr>
    <td>SKG</td>
    <td>6,974,959</td>
    <td>15</td>
    <td>50,775,620</td>
    <td>-</td>
    <td>-</td>
  </tr>
</table>

Table 2: Statistics of datasets.

We conduct comparison experiments on on four KG completion benchmark datasets: two common ones, WN18RR (Toutanova et al., 2015) and FB15K237 (Dettmers et al., 2018), and two larger-scale ones, CoDE-X-L (Safavi and Koutra, 2020) and YAGO3-10 (Mahdisoltani et al., 2015). In real-world scenarios, we apply MED to a large-scale e-commerce social knowledge graph (SKG) from Taobao, which has over 50 million social record triples from about 7 million users. Table 2 presents dataset statistics.

5.1.2 Evaluation Metric

For link prediction task, we adopt standard metrics MRR and Hit@k (\( k = 1, 3, 10 \)) with filter setting (Bordes et al., 2013). For a test triple \((h, r, t)\), we construct candidate triples by replacing \( h \) with all entities and calculate the triple score rank of \((h, r, t)\) among candidate triples as the head prediction rank \( rank_h \). Likewise, we get the tail prediction rank \( rank_t \). We average \( rank_h \) and \( rank_t \) as \((h, r, t)\)’s final rank. MRR is the mean reciprocal rank of all test triples, Hit@k is the percentage of test triples with rank \( \leq k \). We use Effi=MRR/\#P (\#P is the parameter number) (Chen et al., 2023) to measure models’ parameter efficiency. We use f1-score and accuracy for user labeling task, and normalized discounted cumulative gain ndcg@k(\( k = 5, 10 \)) for product recommendation task.

5.1.3 Implementation

For the link prediction task, we set \( d_n = 640 \) for the highest-dimensional sub-model \( M_n \) and \( d_1 = 10 \) for the lowest-dimensional sub-model \( M_1 \). We set \( n = 64 \) and the dimension gap 10 for every pair of neighbor sub-models. There are a total of 64 available sub-models of different dimensions from 10 to 640 in our croppable KGE model. The dimension of \( M_i (i = 1, 2..., 64) \) is \( 10 \times i \). For the user labeling and product recommendation task, we set \( n = 3 \) and train the croppable KGE containing 3 sub-models: \( M_1 \) with \( d_1 = 10 \) for mobile phone (MB) terminals that are limited by storage and computing resources, \( M_2 \) with \( d_2 = 100 \) for the personal computer (PC), and \( M_3 \) with \( d_3 = 500 \) for the platform’s servers. We initialize the learnable scaling parameters \( w_1, w_2 \) and \( w_3 \) in (3), (4) and (6) to 1. We implement MED by extending OpenKE (Han et al., 2018), an open-source KGE framework based on PyTorch. We set the batch size to 1024 and the maximum training epoch to 3000 with early stopping. For each positive triple, we generate 64 negative triples by randomly replacing its head or tail entity with another entity. We use Adam (Kingma and Ba, 2015) optimizer with a linear decay learning rate scheduler and perform a search on the initial learning rate in \{0.0001, 0.0005, 0.001, 0.01\}. We train all sub-models simultaneously by optimizing the uniformly sampled sub-models from the full Croppable model in each step.

5.1.4 Baselines

For each required dimension \( d_r \), we extract the first \( d_r \) dimensions from our croppable KGE as the target model and compare it to the KGE models obtained by 8 baselines of the following 3 types:

• Directly training the target KGE model of requirement dimension \( d_r \), referred to as **1) DT**.
The directly trained highest-dimensional KGE model (\( d_r = d_n \)) is denoted as \( M_{max}^{DT} \).

• Extracting the first \( d_r \) dimensions from \( M_{max}^{DT} \) as the target model, called **2) Ext**. We update \( M_{max}^{DT} \) by arranging 640 dimensions in descending order based on their importance before extracting as (Molchanov et al., 2017; Voita et al., 2019): **3) Ext-L**, the importance for each dimension of \( M_{max}^{DT} \) is the variation of KGE loss on validation set after removing it; and **4) Ext-V**, the importance for each dimension is the average absolute of parameter weights of all entities and relations.

• Distilling the target KGE by KD methods: **5) BKD** (Hinton et al., 2015) is the most basic one by minimizing the KL divergence of the output distributions of teacher and student; **6) TA** (Mirzadeh et al., 2020) uses a medium-size teaching assistant (TA) model as a bridge for size gap, where TA model has the same dimension as the directly trained one whose MRR is closest to the average MRR of teacher and student. We also compare with two KD methods proposed for KGE, which have similar configurations to ours, i.e. compressing high-dimensional teacher into low-dimensional student: **7) Du-alIDE** (Zhu et al., 2022) considers the mutual influences between teacher and student and optimizes them simultaneously; **8) IterDE** (Liu et al., 2023) enables the KGE model to alternately act as student and teacher so that knowledge can be transferred smoothly between high-dimensional teacher and low-dimensional student. In these baselines, \( M_{max}^{DT} \) is the teacher, and other settings including hyperparameters are the same as their original papers.

**5.2 Performance Comparison**

We report the link prediction results of some representative dimensions in Table 3, more results of other dimensions and metrics are in Appendix A and the ablation studies are in Appendix B.

MED outperforms baselines in most settings, especially at extremely low dimensions. On WN18RR with \( d=10 \), MED achieves an improvement of **14.9%** and **15.1%** on TransE, **8.4%** and **6.6%** on RotatE, **29.4%** and **10.6%** on PairRE than the best MRR and Hit@10 of baselines. A similar trend is seen on FB15K237. This success stems from MED’s rich knowledge sources for low-dimensional models: For sub-model \( M_i \), \( M_{i+1} \) is a direct teacher, and \( M_{i+2} \) can indirectly influence \( M_i \) via \( M_{i+1} \). In theory, all higher-dimensional sub-models can transfer knowledge to lower-dimensional ones through stepwise propagation. Although this propagation may harm high-dimensional models with low-quality knowledge from low-dimensional ones, MED’s evolutionary improvement mechanism mitigates the damage, allowing high-dimensional models to compete with directly trained KGEs (Fig. 3). We also note that Ext-based methods are highly unstable, performing worse than DT in most cases, suggesting that dimension importance alone doesn’t ensure sub-model performance.

![Results of different dimensions for PairRE on WN18RR (left) and FB15K237 (right).](page_1012_1042_377_246.png)

Figure 3: Results of different dimensions for PairRE on WN18RR (left) and FB15K237 (right).

**5.3 Parameter efficiency of MED**

In Table 4, we compare our sub-models of suitable low dimensions to parameter-efficient KGEs especially proposed for large-scale KGs including NodePiece (Galkin et al., 2022) and EARL (Chen et al., 2023). With a similar number of model parameters, MED’s sub-models outperform specialized parameter-efficient KGE methods, showing their parameter efficiency. More crucially, it can offer different-sized parameter-efficient models for applications.

**5.4 MED in real applications**

We apply the trained croppable KGE with TransE on SKG to three real-world applications: user labeling on servers and product recommendation on PCs and mobile phones. Table 5 shows that our crop-pable user embeddings outperform all baselines, including directly trained (DT), the best baseline Du-alIDE, and a common dimension reduction method in industry principal components analysis (PCA) on \( M_{max}^{DT} \). Notably, the strong performance in the mobile phone task (constrained by storage and computing resources to a max embedding dimension of 10) highlights the great practical value of MED. More application details are in Appendix C.

**5.5 Extend MED to Neural Networks**

To test our method’s extensibility to other neural networks, we use BERT (Devlin et al., 2019) as
Table 3: MRR and Hit@10 (H10) of some dimensions on WN18RR (WN) and FB15K237 (FB).

<table>
  <tr>
    <th rowspan="2">KGE</th>
    <th colspan="6">WN18RR</th>
    <th colspan="6">FB15K237</th>
  </tr>
  <tr>
    <th>Method</th>
    <th>MRR</th>
    <th>H10</th>
    <th>Hit@10</th>
    <th>Method</th>
    <th>MRR</th>
    <th>H10</th>
    <th>Hit@10</th>
    <th>Method</th>
    <th>MRR</th>
    <th>H10</th>
    <th>Hit@10</th>
  </tr>
  <tr>
    <td rowspan="10">TransE</td>
    <td>DT</td><td>0.121</td><td>0.298</td><td>0.214</td><td>0.496</td><td>0.233</td><td>0.531</td><td>0.237</td><td>0.537</td><td>0.115</td><td>0.285</td><td>0.199</td><td>0.358</td>
  </tr>
  <tr><td>Ext-L</td><td>0.125</td><td>0.308</td><td>0.224</td><td>0.497</td><td>0.236</td><td>0.534</td><td>0.237</td><td>0.537</td><td>0.119</td><td>0.294</td><td>0.283</td><td>0.462</td></tr>
  <tr><td>Ext-V</td><td>0.139</td><td>0.315</td><td>0.224</td><td>0.497</td><td>0.236</td><td>0.534</td><td>0.237</td><td>0.537</td><td>0.139</td><td>0.256</td><td>0.237</td><td>0.396</td></tr>
  <tr><td>TA</td><td>0.144</td><td>0.335</td><td>0.226</td><td>0.512</td><td>0.234</td><td>0.533</td><td>-</td><td>-</td><td>0.175</td><td>0.246</td><td>0.303</td><td>0.484</td></tr>
  <tr><td>DualDE</td><td>0.148</td><td>0.337</td><td>0.225</td><td>0.514</td><td>0.235</td><td>0.533</td><td>-</td><td>-</td><td>0.179</td><td>0.301</td><td>0.306</td><td>0.483</td></tr>
  <tr><td>IterDE</td><td>0.150</td><td>0.338</td><td>0.223</td><td>0.518</td><td>0.236</td><td>0.529</td><td>0.237</td><td>0.537</td><td>0.196</td><td>0.341</td><td>0.308</td><td>0.486</td></tr>
  <tr><td>MED</td><td>0.170</td><td>0.388</td><td>0.232</td><td>0.518</td><td>0.236</td><td>0.529</td><td>0.237</td><td>0.537</td><td>0.196</td><td>0.341</td><td>0.308</td><td>0.505</td></tr>
  <tr>
    <td rowspan="10">SimpleE</td>
    <td>DT</td><td>0.061</td><td>0.126</td><td>0.316</td><td>0.389</td><td>0.409</td><td>0.459</td><td>0.421</td><td>0.481</td><td>0.097</td><td>0.179</td><td>0.236</td><td>0.390</td><td>0.285</td><td>0.458</td><td>0.295</td><td>0.472</td>
  </tr>
  <tr><td>Ext-L</td><td>0.005</td><td>0.007</td><td>0.160</td><td>0.249</td><td>0.357</td><td>0.401</td><td>0.421</td><td>0.481</td><td>0.037</td><td>0.068</td><td>0.090</td><td>0.144</td><td>0.229</td><td>0.372</td><td>0.295</td><td>0.472</td></tr>
  <tr><td>Ext-V</td><td>0.005</td><td>0.006</td><td>0.169</td><td>0.244</td><td>0.398</td><td>0.454</td><td>0.421</td><td>0.481</td><td>0.045</td><td>0.059</td><td>0.083</td><td>0.146</td><td>0.196</td><td>0.316</td><td>0.205</td><td>0.472</td></tr>
  <tr><td>BKD</td><td>0.075</td><td>0.156</td><td>0.343</td><td>0.399</td><td>0.414</td><td>0.468</td><td>-</td><td>-</td><td>0.113</td><td>0.204</td><td>0.244</td><td>0.412</td><td>0.287</td><td>0.463</td></tr>
  <tr><td>TA</td><td>0.089</td><td>0.189</td><td>0.368</td><td>0.418</td><td>0.415</td><td>0.472</td><td>-</td><td>-</td><td>0.124</td><td>0.221</td><td>0.254</td><td>0.416</td><td>0.290</td><td>0.465</td></tr>
  <tr><td>DualDE</td><td>0.090</td><td>0.190</td><td>0.370</td><td>0.419</td><td>0.416</td><td>0.469</td><td>-</td><td>-</td><td>0.120</td><td>0.215</td><td>0.257</td><td>0.427</td><td>0.293</td><td>0.465</td></tr>
  <tr><td>IterDE</td><td>0.077</td><td>0.162</td><td>0.375</td><td>0.419</td><td>0.416</td><td>0.469</td><td>-</td><td>-</td><td>0.120</td><td>0.215</td><td>0.257</td><td>0.427</td><td>0.293</td><td>0.465</td></tr>
  <tr><td>MED</td><td>0.111</td><td>0.224</td><td>0.385</td><td>0.431</td><td>0.418</td><td>0.477</td><td>0.421</td><td>0.482</td><td>0.143</td><td>0.267</td><td>0.261</td><td>0.427</td><td>0.291</td><td>0.466</td><td>0.294</td><td>0.470</td></tr>
  <tr>
    <td rowspan="10">RotatE</td>
    <td>DT</td><td>0.172</td><td>0.418</td><td>0.456</td><td>0.556</td><td>0.471</td><td>0.537</td><td>0.476</td><td>0.575</td><td>0.232</td><td>0.424</td><td>0.312</td><td>0.495</td><td>0.322</td><td>0.506</td><td>0.325</td><td>0.515</td>
  </tr>
  <tr><td>Ext-L</td><td>0.206</td><td>0.277</td><td>0.399</td><td>0.487</td><td>0.445</td><td>0.541</td><td>0.476</td><td>0.575</td><td>0.135</td><td>0.243</td><td>0.221</td><td>0.365</td><td>0.280</td><td>0.453</td><td>0.325</td><td>0.515</td></tr>
  <tr><td>Ext-V</td><td>0.261</td><td>0.377</td><td>0.337</td><td>0.471</td><td>0.416</td><td>0.532</td><td>0.476</td><td>0.575</td><td>0.160</td><td>0.281</td><td>0.238</td><td>0.393</td><td>0.288</td><td>0.485</td><td>0.325</td><td>0.515</td></tr>
  <tr><td>BKD</td><td>0.175</td><td>0.434</td><td>0.456</td><td>0.556</td><td>0.470</td><td>0.570</td><td>-</td><td>-</td><td>0.277</td><td>0.442</td><td>0.314</td><td>0.503</td><td>0.322</td><td>0.510</td></tr>
  <tr><td>TA</td><td>0.172</td><td>0.438</td><td>0.456</td><td>0.556</td><td>0.470</td><td>0.570</td><td>-</td><td>-</td><td>0.277</td><td>0.442</td><td>0.314</td><td>0.503</td><td>0.322</td><td>0.510</td></tr>
  <tr><td>DualDE</td><td>0.179</td><td>0.440</td><td>0.462</td><td>0.559</td><td>0.473</td><td>0.573</td><td>-</td><td>-</td><td>0.282</td><td>0.449</td><td>0.315</td><td>0.502</td><td>0.322</td><td>0.512</td></tr>
  <tr><td>IterDE</td><td>0.176</td><td>0.436</td><td>0.459</td><td>0.560</td><td>0.471</td><td>0.569</td><td>-</td><td>-</td><td>0.276</td><td>0.445</td><td>0.317</td><td>0.504</td><td>0.323</td><td>0.512</td></tr>
  <tr><td>MED</td><td>0.326</td><td>0.544</td><td>0.485</td><td>0.645</td><td>0.544</td><td>0.645</td><td>0.476</td><td>0.574</td><td>0.326</td><td>0.544</td><td>0.485</td><td>0.645</td><td>0.326</td><td>0.544</td><td>0.324</td><td>0.514</td></tr>
  <tr>
    <td rowspan="10">PairRE</td>
    <td>DT</td><td>0.220</td><td>0.321</td><td>0.415</td><td>0.473</td><td>0.449</td><td>0.534</td><td>0.455</td><td>0.544</td><td>0.182</td><td>0.314</td><td>0.284</td><td>0.452</td><td>0.319</td><td>0.503</td><td>0.352</td><td>0.522</td>
  </tr>
  <tr><td>Ext-L</td><td>0.152</td><td>0.209</td><td>0.334</td><td>0.463</td><td>0.419</td><td>0.526</td><td>0.453</td><td>0.544</td><td>0.148</td><td>0.222</td><td>0.217</td><td>0.353</td><td>0.294</td><td>0.469</td><td>0.332</td><td>0.522</td></tr>
  <tr><td>Ext-V</td><td>0.162</td><td>0.220</td><td>0.363</td><td>0.442</td><td>0.437</td><td>0.533</td><td>0.453</td><td>0.544</td><td>0.150</td><td>0.249</td><td>0.219</td><td>0.333</td><td>0.309</td><td>0.489</td><td>0.332</td><td>0.522</td></tr>
  <tr><td>BKD</td><td>0.228</td><td>0.336</td><td>0.421</td><td>0.483</td><td>0.451</td><td>0.536</td><td>-</td><td>-</td><td>0.198</td><td>0.332</td><td>0.288</td><td>0.453</td><td>0.321</td><td>0.509</td></tr>
  <tr><td>TA</td><td>0.245</td><td>0.340</td><td>0.426</td><td>0.487</td><td>0.452</td><td>0.537</td><td>-</td><td>-</td><td>0.208</td><td>0.346</td><td>0.292</td><td>0.453</td><td>0.323</td><td>0.509</td></tr>
  <tr><td>DualDE</td><td>0.253</td><td>0.336</td><td>0.428</td><td>0.495</td><td>0.453</td><td>0.540</td><td>-</td><td>-</td><td>0.207</td><td>0.342</td><td>0.293</td><td>0.456</td><td>0.326</td><td>0.512</td></tr>
  <tr><td>IterDE</td><td>0.253</td><td>0.336</td><td>0.428</td><td>0.495</td><td>0.453</td><td>0.540</td><td>-</td><td>-</td><td>0.207</td><td>0.342</td><td>0.293</td><td>0.456</td><td>0.326</td><td>0.512</td></tr>
  <tr><td>MED</td><td>0.317</td><td>0.376</td><td>0.433</td><td>0.502</td><td>0.451</td><td>0.541</td><td>0.451</td><td>0.542</td><td>0.239</td><td>0.384</td><td>0.303</td><td>0.466</td><td>0.324</td><td>0.510</td><td>0.330</td><td>0.520</td></tr>
</table>

Table 4: Link prediction results on WN18RR, FB15K237, CoDEx-L and YAGO3-10.

<table>
  <tr>
    <th rowspan="2">Dim</th>
    <th colspan="3">WN18RR</th>
    <th colspan="3">CoDEx-L</th>
    <th colspan="3">YAGO3-10</th>
  </tr>
  <tr>
    <th>#PMJ</th>
    <th>MRR</th>
    <th>H10</th>
    <th>Eps</th>
    <th>#PMJ</th>
    <th>MRR</th>
    <th>H10</th>
    <th>Eps</th>
    <th>#PMJ</th>
    <th>MRR</th>
    <th>H10</th>
    <th>Eps</th>
  </tr>
  <tr>
    <td>RotatE</td><td>100</td><td>2.9</td><td>0.296</td><td>0.473</td><td>100</td><td>4.1</td><td>0.411</td><td>0.429</td><td>100</td><td>2.5</td><td>0.196</td><td>0.322</td><td>0.052</td><td>20</td><td>4.8</td><td>0.121</td><td>0.262</td><td>0.025</td>
  </tr>
  <tr>
    <td>+ NodePiece</td><td>100</td><td>3.2</td><td>0.286</td><td>0.420</td><td>100</td><td>4.4</td><td>0.403</td><td>0.515</td><td>0.092</td><td>100</td><td>3.6</td><td>0.190</td><td>0.313</td><td>0.061</td><td>100</td><td>4.1</td><td>0.247</td><td>0.488</td><td>0.061</td>
  </tr>
  <tr>
    <td>+ EAG</td><td>150</td><td>1.8</td><td>0.177</td><td>0.177</td><td>150</td><td>2.0</td><td>0.171</td><td>0.171</td><td>0.171</td><td>150</td><td>2.0</td><td>0.171</td><td>0.171</td><td>0.171</td><td>150</td><td>2.0</td><td>0.171</td><td>0.171</td><td>0.171</td>
  </tr>
  <tr>
    <td>+ MED</td><td>40</td><td>1.2</td><td>0.318</td><td>0.504</td><td>40</td><td>3.2</td><td>0.466</td><td>0.561</td><td>0.146</td><td>20</td><td>3.1</td><td>0.243</td><td>0.385</td><td>0.078</td><td>20</td><td>4.9</td><td>0.313</td><td>0.528</td><td>0.064</td>
  </tr>
</table>

Table 5: Results on SKG.

<table>
  <tr>
    <th>User Labeling</th>
    <th colspan="2">Product Recommendation</th>
  </tr>
  <tr>
    <th>server (500k)</th>
    <th>PC terminal (100d)</th>
    <th>MP terminal (10d)</th>
  </tr>
  <tr>
    <td>Method</td>
    <td>train time</td>
    <td>acc.</td>
    <td>fl</td>
    <td>ndcg@5</td>
    <td>ndcg@10</td>
    <td>ndcg@5</td>
    <td>ndcg@10</td>
  </tr>
  <tr>
    <td>DT</td>
    <td>10h</td>
    <td>0.899</td>
    <td>0.874</td>
    <td>0.411</td>
    <td>0.447</td>
    <td>0.344</td>
    <td>0.361</td>
  </tr>
  <tr>
    <td>PCA</td>
    <td>19h</td>
    <td>-</td>
    <td>-</td>
    <td>0.423</td>
    <td>0.456</td>
    <td>0.404</td>
    <td>0.433</td>
  </tr>
  <tr>
    <td>DualDE</td>
    <td>19h</td>
    <td>-</td>
    <td>-</td>
    <td>0.431</td>
    <td>0.465</td>
    <td>0.422</td>
    <td>0.451</td>
  </tr>
  <tr>
    <td>MED</td>
    <td>53h</td>
    <td>0.893</td>
    <td>0.879</td>
    <td>0.431</td>
    <td>0.465</td>
    <td>0.422</td>
    <td>0.451</td>
  </tr>
</table>

an example. We use distillation methods based on Hugging Face Transformers (Wolf et al., 2020) as baselines. As in previous works(Sun et al., 2019a; Tang et al., 2019; Jung et al., 2023; Zhou et al., 2022a), we perform distillation at the fine-tuning stage. See Appendix D for more details. Table 6 shows the results on the GLUE development set(Wang et al., 2019). We compare MED with other KD models of similar speedup or parameter number. MED performs competitively on most tasks against BERT-specialized KD methods. Compared to HAT (Wang et al., 2020a), which has a similar architecture to ours, MED’s sub-models outperform HAT across three parameter levels. Sub-models with 54M, 17.5M, and 6.36M parameters show average improvements of 16.3%, 21.7% and 19.7% respectively.

5.6 Analysis of MED

5.6.1 Training efficiency

![Sub-models’ MRR during training on WN18RR with RotatE.](page_1092_1342_324_246.png)

Figure 4: Sub-models’ MRR during training on WN18RR with RotatE.

Table 7 reports the training time of getting 64 models of various sizes (d=10, 20, ..., 640). Figure 4 shows MRR change of MED’s sub-models during training. For fair comparison, all training is done on a single NVIDIA Tesla A100 40GB GPU. For DT, the training time is the sum of directly training 64 KGE models sequentially. Ext-based baselines have the same training time, equal to that of training a \( d_n \)-dimensional KGE model as dimension arrangement time is negligible. KD-based baselines’ training time is the sum of training a \( d_n \)-dimensional teacher model and distilling 63 student models. On FB15K237, we don’t train all 63 sizes of student models for TA, DualDE, and IterDE, estimated to take over 400 hours per KGE
<table>
  <tr>
    <th>Method</th>
    <th>#P(M)</th>
    <th>Speedup</th>
    <th>MNLI-m acc.</th>
    <th>MNLI-mm acc.</th>
    <th>MRPC acc.</th>
    <th>QNLI acc.</th>
    <th>QQP acc.</th>
    <th>RTE acc.</th>
    <th>STS-2 acc.</th>
    <th>STS-B acc.</th>
    <th>best/Speedup</th>
  </tr>
  <tr>
    <td>BERT<sub>base</sub></td>
    <td>110</td>
    <td>1.0×</td>
    <td>84.4</td>
    <td>85.3</td>
    <td>88.6/84.1</td>
    <td>89.7</td>
    <td>89.6/91.1</td>
    <td>67.5</td>
    <td>92.5</td>
    <td>88.8/88.5</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-BKD</sub></td>
    <td>66</td>
    <td>2.0×</td>
    <td>82.2</td>
    <td>82.9</td>
    <td>86.2/80.8</td>
    <td>88.5</td>
    <td>88.0/91.0</td>
    <td>65.4</td>
    <td>90.9</td>
    <td>88.2/87.8</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-PKD</sub></td>
    <td>66</td>
    <td>2.0×</td>
    <td>82.3</td>
    <td>82.6</td>
    <td>86.4/81.0</td>
    <td>88.6</td>
    <td>87.9/91.0</td>
    <td>63.9</td>
    <td>90.8</td>
    <td>88.5/86.8</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-MetaDistil</sub></td>
    <td>66</td>
    <td>2.0×</td>
    <td>82.3</td>
    <td>82.6</td>
    <td>84.6/78.8</td>
    <td>87.8</td>
    <td>87.1/91.5</td>
    <td>61.5</td>
    <td>90.7</td>
    <td>88.7/87.5</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-FSD</sub></td>
    <td>66</td>
    <td>2.0×</td>
    <td>82.4</td>
    <td>82.6</td>
    <td>86.8/81.8</td>
    <td>88.9</td>
    <td>87.1/91.2</td>
    <td>61.5</td>
    <td>91.0</td>
    <td>88.4/86.8</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-FSD</sub></td>
    <td>66</td>
    <td>2.0×</td>
    <td>82.4</td>
    <td>83.0</td>
    <td>87.1/82.2</td>
    <td>89.0</td>
    <td>88.1/91.2</td>
    <td>66.6</td>
    <td>91.0</td>
    <td>88.7/88.3</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-BKD</sub></td>
    <td>55</td>
    <td>2.9×</td>
    <td>80.5</td>
    <td>80.9</td>
    <td>87.2/83.1</td>
    <td>87.5</td>
    <td>86.6/90.4</td>
    <td>65.2</td>
    <td>90.2</td>
    <td>84.5/84.2</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-PKD</sub></td>
    <td>55</td>
    <td>2.9×</td>
    <td>80.9</td>
    <td>81.3</td>
    <td>87.0/82.9</td>
    <td>87.7</td>
    <td>86.8/90.5</td>
    <td>66.1</td>
    <td>90.5</td>
    <td>84.3/84.0</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-MetaDistil</sub></td>
    <td>55</td>
    <td>2.9×</td>
    <td>82.4</td>
    <td>82.7</td>
    <td><b>88.4/84.8</b></td>
    <td>88.5</td>
    <td>87.8/90.8</td>
    <td><b>67.8</b></td>
    <td>91.8</td>
    <td>86.3/86.0</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-FSD</sub></td>
    <td>54</td>
    <td>2.0×</td>
    <td>80.8</td>
    <td>71.6</td>
    <td>81.5/78.8</td>
    <td>85.3</td>
    <td>86.7/90.1</td>
    <td>59.3</td>
    <td>90.1</td>
    <td>83.9/80.1</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT<sub>T-FSD</sub></td>
    <td>54</td>
    <td>2.0×</td>
    <td>82.7</td>
    <td>83.3</td>
    <td>88.0/84.0</td>
    <td>86.8</td>
    <td><b>89.1/90.7</b></td>
    <td>67.2</td>
    <td><b>91.9</b></td>
    <td>87.6/87.2</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT-HAT<sup>†</sup></td>
    <td>17.5</td>
    <td>4.7×</td>
    <td>63.6</td>
    <td>64.2</td>
    <td>68.4/78.4</td>
    <td>61.1</td>
    <td>69.0/79.7</td>
    <td>47.2</td>
    <td>82.9</td>
    <td>74.1/75.8</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT-MED</td>
    <td>17.5</td>
    <td>4.7×</td>
    <td>81.2</td>
    <td>82.4</td>
    <td>86.1/82.0</td>
    <td>86.4</td>
    <td>83.8/86.2</td>
    <td>64.6</td>
    <td>88.2</td>
    <td>86.1/86.4</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT-HAT<sup>†</sup></td>
    <td>6.36</td>
    <td>5.2×</td>
    <td>59.9</td>
    <td>60.0</td>
    <td>66.5/77.3</td>
    <td>60.1</td>
    <td>66.5/77.1</td>
    <td>46.2</td>
    <td>81.7</td>
    <td>71.9/70.4</td>
    <td></td>
  </tr>
  <tr>
    <td>BERT-MED</td>
    <td>6.36</td>
    <td>5.2×</td>
    <td>72.6</td>
    <td>73.7</td>
    <td>84.1/78.1</td>
    <td>86.0</td>
    <td>79.6/82.7</td>
    <td>61.7</td>
    <td>86.9</td>
    <td>82.8/81.6</td>
    <td></td>
  </tr>
</table>

Table 6: Results on the dev set of GLUE. The results of knowledge distillation methods for BERT<sub>4</sub> and BERT<sub>6</sub> are reported by (Jung et al., 2023; Zhou et al., 2022a) and the † results reported by us.

<table>
  <tr>
    <th></th>
    <th>TransE</th>
    <th>SimpE</th>
    <th>RotatE</th>
    <th>PairRE</th>
  </tr>
  <tr>
    <td>DT</td>
    <td>74.0 (9.39×)</td>
    <td>68.0 (12.14×)</td>
    <td>141.0 (11.18×)</td>
    <td>67.4 (10.06×)</td>
  </tr>
  <tr>
    <td>Ext-based</td>
    <td>1.5 (0.19×)</td>
    <td>1.3 (0.23×)</td>
    <td>2.5 (0.20×)</td>
    <td>1.6 (0.24×)</td>
  </tr>
  <tr>
    <td>BKD</td>
    <td>91.5 (11.73×)</td>
    <td>72.0 (12.86×)</td>
    <td>163.0 (12.83×)</td>
    <td>87.5 (13.06×)</td>
  </tr>
  <tr>
    <td>WN TA</td>
    <td>172.0 (22.05×)</td>
    <td>142.0 (25.36×)</td>
    <td>272.0 (21.42×)</td>
    <td>166.0 (24.78×)</td>
  </tr>
  <tr>
    <td>DualDE</td>
    <td>151.0 (19.36×)</td>
    <td>133.0 (23.75×)</td>
    <td>240.0 (18.90×)</td>
    <td>133.0 (19.85×)</td>
  </tr>
  <tr>
    <td>IterDE</td>
    <td>140.9 (18.06×)</td>
    <td>118.0 (21.07×)</td>
    <td>216.0 (17.01×)</td>
    <td>124.0 (18.51×)</td>
  </tr>
  <tr>
    <td>MED</td>
    <td>7.8 (1.00×)</td>
    <td>5.6 (1.00×)</td>
    <td>12.7 (1.00×)</td>
    <td>6.7 (1.00×)</td>
  </tr>
  <tr>
    <td>DT</td>
    <td>218.0 (10.23×)</td>
    <td>179.0 (10.65×)</td>
    <td>381.0 (10.73×)</td>
    <td>179.0 (9.37×)</td>
  </tr>
  <tr>
    <td>Ext-based</td>
    <td>4.7 (0.22×)</td>
    <td>5.1 (0.27×)</td>
    <td>9.5 (0.27×)</td>
    <td>3.7 (0.19×)</td>
  </tr>
  <tr>
    <td>BKD</td>
    <td>21.3 (1.00×)</td>
    <td>16.8 (1.00×)</td>
    <td>35.5 (1.00×)</td>
    <td>19.1 (1.00×)</td>
  </tr>
  <tr>
    <td>MED</td>
    <td>21.3 (1.00×)</td>
    <td>16.8 (1.00×)</td>
    <td>35.5 (1.00×)</td>
    <td>19.1 (1.00×)</td>
  </tr>
</table>

Table 7: Training time (hours).

<table>
  <tr>
    <th>n</th>
    <th>train time</th>
    <th>10d MRR H10</th>
    <th>40d MRR H10</th>
    <th>160d MRR H10</th>
    <th>640d MRR H10</th>
  </tr>
  <tr>
    <td>64</td>
    <td>12.7h</td>
    <td>0.324 0.469</td>
    <td>0.466 0.561</td>
    <td>0.471 0.574</td>
    <td>0.476 0.574</td>
  </tr>
  <tr>
    <td>16</td>
    <td>6.2h</td>
    <td>0.322 0.467</td>
    <td>0.465 0.561</td>
    <td>0.473 0.575</td>
    <td>0.477 0.576</td>
  </tr>
  <tr>
    <td>4</td>
    <td>3.3h</td>
    <td>0.319 0.463</td>
    <td>0.463 0.561</td>
    <td>0.475 0.577</td>
    <td>0.480 0.578</td>
  </tr>
</table>

Table 8: Results of different n.

method. Compared to DT, MED speeds up by up to 10× for 4 KGE methods. Ext-based baselines have the shortest training time but poor performance and low practical value. Except for BKD, other KD-based methods need to optimize both student and teacher, increasing train cost.

5.6.2 Effect of the number of sub-models
We set the number of different sub-models (n = 64, 16, 4) for RotatE on WN18RR. Table 8 shows that reducing the number of sub - models boosts high-dimensional (d=160 and 640) model performance but lowers that of low-dimensional (d=10 and 40) models. Training efficiency is nearly linearly related to the number of models.

5.6.3 Whether high-dimensional sub-models cover low-dimensional ones’ capabilities
A high-dimensional model retaining lower-dimensional ones’ ability should correctly predict all triples the latter can. We calculate the percentage of test-set triples where if the smallest sub-model correctly predicting a triple is \( M_i \), all higher-dimensional ones (\( M_{i+1},\ M_{i+2},\ ..., \ M_n \)) do too.

This result is the ability retention ratio (ARR). We use Hit@10 to judge correct prediction: \( M_i \) predicts a triple correctly if it scores in the top 10 among candidates. Fig. 5 shows that MED’s ARR is much higher than baselines, especially on FB15K237. This means MED’s high-dimensional sub-models cover low-dimensional ones’ power, thanks to the mutual learning mechanism for knowledge review. This advantage of MED provides a simple way to judge triple difficulty for KGE methods: triples low-dimensional sub-models predict may be easy as high-dimensional ones can too, while those only predicted by high-dimensional sub-models are difficult.

![Bar chart comparing ARR for different methods](page_1092_1202_340_246.png)

Figure 5: The ability retention ratio (ARR).

6 Conclusion

In this work, we propose a novel KGE training framework, MED, which trains a croppable KGE in one go. Sub-models of various dimensions can be cropped from it and used directly without extra training. In MED, we introduce the mutual learning mechanism to improve low-dimensional sub-models and make the high-dimensional sub-models retain low-dimensional ones’ ability, the evolutionary improvement mechanism to prompt high-dimensional sub-models to learn what low-dimensional ones can’t, and dynamic loss weights to balance multiple losses adaptively. Experimental results demonstrate MED’s effectiveness, high efficiency, and flexible extensibility.
7 Limitations

As a research paper, we are unable to conduct experiments on all knowledge graph embedding (KGE) methods. To demonstrate the generality of the method proposed in this paper, we selected representative models from different types of KGE methods, including distance-based TransE, rotation-based RotatE, semantic matching-based SimplE, and subrelation encoding-based PairRE. In fact, there are many KGE methods that outperform those used in our experiments. However, the method proposed in this paper is applicable to any KGE method with a triple scoring function.

8 Acknowledgements

This work is founded by National Natural Science Foundation of China (NSFCU23B2055/NSFCU19B2027/NSFC62306276), Zhejiang Provincial Natural Science Foundation of China (No. LQ23F020017), Yongjiang Talent Introduction Programme (2022A-238-G), and Fundamental Research Funds for the Central Universities (226-2023-00138). This work was supported by AntGroup.

References

Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jin Jin, Xin Jiang, Qun Liu, Michael R. Lyu, and Irwin King. 2021. Binarybert: Pushing the limit of BERT quantization. In ACL/IJCNLP (1), pages 4334–4348. Association for Computational Linguistics.

Antoine Bordes, Nicolas Usunier, Alberto García-Durán, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787–2795.

Linlin Chao, Jianshan He, Taifeng Wang, and Wei Chu. 2021. Pairre: Knowledge graph embeddings via paired relation vectors. In ACL/IJCNLP (1), pages 4360–4369. Association for Computational Linguistics.

Mingyang Chen, Wen Zhang, Zhen Yao, Yushan Zhu, Yang Gao, Jeff Z. Pan, and Huajun Chen. 2023. Entity-agnostic representation learning for parameter-efficient knowledge graph embedding. In AAAI, pages 4182–4190. AAAI Press.

Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).

Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In I-SEMANTICS, pages 121–124. ACM.

Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowledge graph embeddings. In AAAI, pages 1811–1818. AAAI Press.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1), pages 4171–4186. Association for Computational Linguistics.

Dennis Diefenbach, Kamal Deep Singh, and Pierre Maret. 2018. Wdaqua-core1: A question answering service for RDF knowledge bases. In WWW (Companion Volume), pages 1087–1091. ACM.

William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005. Asian Federation of Natural Language Processing.

Cunhang Fan, Yujie Chen, Jun Xue, Yonghui Kong, Jianhua Tao, and Zhao Lv. 2024. Progressive distillation based on masked generation feature method for knowledge graph completion. In AAAI, pages 8380–8388. AAAI Press.

Mikhail Galkin, Etienne G. Denis, Jiapeng Wu, and William L. Hamilton. 2022. Nodepiece: Compositional and parameter-efficient representations of large knowledge graphs. In ICLR. OpenReview.net.

Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. Openke: An open toolkit for knowledge embedding. In EMNLP (Demonstration), pages 139–144. Association for Computational Linguistics.

Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017. Neural collaborative filtering. In WWW, pages 173–182. ACM.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. CoRR, abs/1503.02531.

Huber and J. Peter. 1964. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35(1):73–101.

Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Knowledge graph embedding via dynamic mapping matrix. In ACL (1), pages 687–696. The Association for Computer Linguistics.

Hee-Jun Jung, Doyeon Kim, Seung-Hoon Na, and Kangil Kim. 2023. Feature structure distillation with centered kernel alignment in BERT transferring. Expert Syst. Appl., 234:120980.
Seyed Mehran Kazemi and David Poole. 2018. Simple embedding for link prediction in knowledge graphs. In NeurIPS, pages 4289–4300.

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR (Poster).

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. In AAAI, pages 2181–2187. AAAI Press.

Jiajun Liu, Wenjun Ke, Peng Wang, Ziyu Shang, Jin-hua Gao, Guozheng Li, Ke Ji, and Yanhe Liu. 2024. Towards continual knowledge graph embedding via incremental distillation. In AAAI, pages 8759–8768. AAAI Press.

Jiajun Liu, Peng Wang, Ziyu Shang, and Chenxiao Wu. 2023. Iterde: An iterative knowledge distillation framework for knowledge graph embeddings. In AAAI, pages 4488–4496. AAAI Press.

Farzaneh Mahdisoltani, Joanna Biega, and Fabian M. Suchanek. 2015. YAGO3: A knowledge base from multilingual wikipedias. In CIDR. www.cidrdb.org.

Seyed-Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. 2020. Improved knowledge distillation via teacher assistant. In AAAI, pages 5191–5198. AAAI Press.

Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. 2017. Pruning convolutional neural networks for resource efficient inference. In ICLR (Poster). OpenReview.net.

Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Q. Phung. 2018. A novel embedding model for knowledge base completion based on convolutional neural network. In NAACL-HLT (2), pages 327–333. Association for Computational Linguistics.

Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. In ICML, pages 809–816. Omnipress.

Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. 2019. Relational knowledge distillation. In CVPR, pages 3967–3976. Computer Vision Foundation / IEEE.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383–2392. The Association for Computational Linguistics.

Mrinmaya Sachan. 2020. Knowledge graph embedding compression. In ACL, pages 2681–2691. Association for Computational Linguistics.

Tara Safavi and Danai Koutra. 2020. Codex: A comprehensive knowledge graph completion benchmark. In EMNLP (1), pages 8328–8350. Association for Computational Linguistics.

Michael Sejr Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In ESWC, volume 10843 of Lecture Notes in Computer Science, pages 593–607. Springer.

Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA. A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1631–1642. ACL.

Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Hervé Jégou, and Armand Joulin. 2021. Training with quantization noise for extreme model compression. In ICLR. OpenReview.net.

Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019a. Patient knowledge distillation for BERT model compression. In EMNLP/IJCNLP (1), pages 4322–4331. Association for Computational Linguistics.

Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019b. Rotate: Knowledge graph embedding by relational rotation in complex space. In ICLR (Poster). OpenReview.net.

Raphael Tang, Yao Lu, Lingling Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from BERT into simple neural networks. CoRR, abs/1903.12136.

Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In EMNLP, pages 1499–1509. The Association for Computational Linguistics.

Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pages 2071–2080. JMLR.org.

Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha P. Talukdar. 2020. Composition-based multi-relational graph convolutional networks. In ICLR. OpenReview.net.

Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In ACL (1), pages 5797–5808. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.

Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. 2020a. HAT: hardware-aware transformers for efficient natural language processing. In ACL, pages 7675–7688. Association for Computational Linguistics.

Haoyu Wang, Yaqing Wang, Defu Lian, and Jing Gao. 2021a. A lightweight knowledge graph embedding framework for efficient inference and storage. In CIKM, pages 1909–1918. ACM.

Kai Wang, Yu Liu, Qian Ma, and Quan Z. Sheng. 2021b. Mulde: Multi-teacher knowledge distillation for low-dimensional knowledge graph embeddings. In WWW, pages 1716–1726. ACM / IW3C2.

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.

Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. 2014. Knowledge graph embedding by translating on hyperplanes. In AAAI, pages 1112–1119. AAAI Press.

Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2020 - Demos, Online, November 16-20, 2020, pages 38–45. Association for Computational Linguistics.

Canran Xu and Ruijiang Li. 2019. Relation embedding with dihedral group in knowledge graph. In ACL (1), pages 263–272. Association for Computational Linguistics.

Haotian Xu, Yuhua Wang, and Jiahui Fan. 2024. Self-knowledge distillation for knowledge graph embedding. In LREC/COLING, pages 14595–14605. ELRA and ICCL.

Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. In ICLR (Poster).

Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. 2019. Quaternion knowledge graph embeddings. In NeurIPS, pages 2731–2741.

Wen Zhang, Chi Man Wong, Gangqiang Ye, Bo Wen, Wei Zhang, and Huajun Chen. 2021. Billion-scale pre-trained e-commerce product knowledge graph model. In ICDE, pages 2476–2487. IEEE.

Wangchunshu Zhou, Canwen Xu, and Julian J. McAuley. 2022a. BERT learns to teach: Knowledge distillation with meta learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 7037–7049. Association for Computational Linguistics.

Zhehui Zhou, Defang Chen, Can Wang, Yan Feng, and Chun Chen. 2022b. Improving knowledge graph embedding via iterative self-semantic knowledge distillation. CoRR, abs/2206.02963.

Yushan Zhu, Wen Zhang, Mingyang Chen, Hui Chen, Xu Cheng, Wei Zhang, and Huajun Chen. 2022. Dualde: Dually distilling knowledge graph embedding for faster and cheaper reasoning. In WSDM, pages 1516–1524. ACM.

Yushan Zhu, Huaixiao Zhao, Wen Zhang, Gangqiang Ye, Hui Chen, Ningyu Zhang, and Huajun Chen. 2021. Knowledge perceived multi-modal pretraining in e-commerce. In ACM Multimedia, pages 2744–2752. ACM.

A More Results of link prediction

More results of link prediction are shown in Table 9 and Table 10 for WN18RR, and Table 11 and Table 12 for FB15K237. All comparison results of sub-models of MED to the directly trained KGEs (DT) of 10- to 640-dimension are shown in Fig. 6.

B Ablation Study

We conduct ablation studies to evaluate the effect of three modules in MED: the mutual learning mechanism (MLM), the evolutionary improvement mechanism (EIM), and the dynamic loss weight (DLW). Table 13 shows the MRR and Hit@k (\( k = 1, 3, 10 \)) of MED removing these modules respectively on WN18RR and TransE.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">10d</th>
    <th colspan="2">20d</th>
    <th colspan="2">40d</th>
    <th colspan="2">80d</th>
    <th colspan="2">160d</th>
    <th colspan="2">320d</th>
    <th colspan="2">640d</th>
  </tr>
  <tr>
    <th>MRR</th><th>Hit@10</th>
    <th>MRR</th><th>Hit@10</th>
    <th>MRR</th><th>Hit@10</th>
    <th>MRR</th><th>Hit@10</th>
    <th>MRR</th><th>Hit@10</th>
    <th>MRR</th><th>Hit@10</th>
    <th>MRR</th><th>Hit@10</th>
  </tr>
  <tr>
    <td>DT</td><td>0.121</td><td>0.287</td><td>0.176</td><td>0.453</td><td>0.214</td><td>0.496</td><td>0.227</td><td>0.524</td><td>0.233</td><td>0.531</td><td>0.235</td><td>0.534</td><td>0.237</td><td>0.537</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.125</td><td>0.298</td><td>0.172</td><td>0.423</td><td>0.199</td><td>0.468</td><td>0.213</td><td>0.495</td><td>0.225</td><td>0.515</td><td>0.226</td><td>0.521</td><td>0.237</td><td>0.537</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.139</td><td>0.315</td><td>0.196</td><td>0.461</td><td>0.224</td><td>0.497</td><td>0.232</td><td>0.516</td><td>0.236</td><td>0.534</td><td>0.236</td><td>0.535</td><td>0.237</td><td>0.537</td>
  </tr>
  <tr>
    <td>TransE</td>
    <td>Ext-V</td><td>0.139</td><td>0.309</td><td>0.198</td><td>0.458</td><td>0.222</td><td>0.494</td><td>0.234</td><td>0.525</td><td>0.236</td><td>0.532</td><td>0.236</td><td>0.536</td><td>0.237</td><td>0.537</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.141</td><td>0.322</td><td>0.207</td><td>0.480</td><td>0.226</td><td>0.513</td><td>0.232</td><td>0.527</td><td>0.233</td><td>0.531</td><td>0.236</td><td>0.533</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.144</td><td>0.333</td><td>0.213</td><td>0.488</td><td>0.234</td><td>0.530</td><td>0.235</td><td>0.533</td><td>0.238</td><td>0.535</td><td>-</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.148</td><td>0.337</td><td>0.213</td><td>0.488</td><td>0.225</td><td>0.514</td><td>0.234</td><td>0.530</td><td>0.235</td><td>0.533</td><td>0.238</td><td>0.535</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.143</td><td>0.332</td><td>0.211</td><td>0.484</td><td>0.224</td><td>0.511</td><td>0.232</td><td>0.528</td><td>0.236</td><td>0.531</td><td>0.237</td><td>0.533</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td>0.170</td><td>0.388</td><td>0.219</td><td>0.491</td><td>0.232</td><td>0.518</td><td>0.232</td><td>0.523</td><td>0.236</td><td>0.529</td><td>0.237</td><td>0.536</td><td>0.237</td><td>0.537</td>
  </tr>
  <tr>
    <td>Method</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
  </tr>
  <tr>
    <td>DT</td><td>0.061</td><td>0.126</td><td>0.257</td><td>0.371</td><td>0.316</td><td>0.389</td><td>0.382</td><td>0.446</td><td>0.409</td><td>0.459</td><td>0.417</td><td>0.474</td><td>0.421</td><td>0.481</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.004</td><td>0.007</td><td>0.041</td><td>0.077</td><td>0.049</td><td>0.249</td><td>0.119</td><td>0.237</td><td>0.401</td><td>0.407</td><td>0.451</td><td>0.421</td><td>0.481</td><td>0.481</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.005</td><td>0.006</td><td>0.048</td><td>0.078</td><td>0.169</td><td>0.244</td><td>0.369</td><td>0.435</td><td>0.398</td><td>0.454</td><td>0.417</td><td>0.481</td><td>0.421</td><td>0.481</td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.004</td><td>0.006</td><td>0.047</td><td>0.076</td><td>0.246</td><td>0.317</td><td>0.368</td><td>0.402</td><td>0.398</td><td>0.461</td><td>0.413</td><td>0.472</td><td>0.421</td><td>0.481</td>
  </tr>
  <tr>
    <td>SimplE</td>
    <td>BKD</td><td>0.075</td><td>0.156</td><td>0.285</td><td>0.381</td><td>0.343</td><td>0.399</td><td>0.394</td><td>0.450</td><td>0.414</td><td>0.468</td><td>0.418</td><td>0.475</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.089</td><td>0.189</td><td>0.316</td><td>0.386</td><td>0.368</td><td>0.418</td><td>0.405</td><td>0.456</td><td>0.415</td><td>0.472</td><td>0.421</td><td>0.481</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.083</td><td>0.175</td><td>0.328</td><td>0.388</td><td>0.386</td><td>0.423</td><td>0.407</td><td>0.454</td><td>0.419</td><td>0.475</td><td>0.422</td><td>0.482</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.077</td><td>0.162</td><td>0.321</td><td>0.378</td><td>0.375</td><td>0.419</td><td>0.404</td><td>0.452</td><td>0.416</td><td>0.469</td><td>0.421</td><td>0.482</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td>0.111</td><td>0.224</td><td>0.335</td><td>0.395</td><td>0.385</td><td>0.431</td><td>0.407</td><td>0.457</td><td>0.418</td><td>0.477</td><td>0.421</td><td>0.481</td><td>0.421</td><td>0.482</td>
  </tr>
  <tr>
    <td>Method</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
  </tr>
  <tr>
    <td>DT</td><td>0.172</td><td>0.418</td><td>0.409</td><td>0.504</td><td>0.456</td><td>0.556</td><td>0.465</td><td>0.564</td><td>0.471</td><td>0.567</td><td>0.474</td><td>0.573</td><td>0.476</td><td>0.575</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.299</td><td>0.378</td><td>0.379</td><td>0.464</td><td>0.437</td><td>0.516</td><td>0.458</td><td>0.544</td><td>0.467</td><td>0.549</td><td>0.471</td><td>0.552</td><td>0.476</td><td>0.575</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.206</td><td>0.277</td><td>0.336</td><td>0.424</td><td>0.399</td><td>0.487</td><td>0.423</td><td>0.515</td><td>0.445</td><td>0.541</td><td>0.466</td><td>0.564</td><td>0.476</td><td>0.575</td>
  </tr>
  <tr>
    <td>RotatE</td>
    <td>Ext-V</td><td>0.261</td><td>0.377</td><td>0.304</td><td>0.433</td><td>0.337</td><td>0.471</td><td>0.366</td><td>0.497</td><td>0.416</td><td>0.532</td><td>0.451</td><td>0.561</td><td>0.476</td><td>0.575</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.175</td><td>0.434</td><td>0.424</td><td>0.540</td><td>0.457</td><td>0.556</td><td>0.471</td><td>0.565</td><td>0.472</td><td>0.570</td><td>0.474</td><td>0.572</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.177</td><td>0.438</td><td>0.424</td><td>0.542</td><td>0.459</td><td>0.558</td><td>0.470</td><td>0.567</td><td>0.473</td><td>0.572</td><td>0.474</td><td>0.572</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.179</td><td>0.440</td><td>0.425</td><td>0.542</td><td>0.462</td><td>0.559</td><td>0.471</td><td>0.567</td><td>0.473</td><td>0.573</td><td>0.475</td><td>0.573</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.176</td><td>0.436</td><td>0.421</td><td>0.538</td><td>0.459</td><td>0.560</td><td>0.470</td><td>0.567</td><td>0.471</td><td>0.569</td><td>0.474</td><td>0.572</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td>0.324</td><td>0.460</td><td>0.456</td><td>0.543</td><td>0.466</td><td>0.561</td><td>0.471</td><td>0.568</td><td>0.471</td><td>0.574</td><td>0.476</td><td>0.573</td><td>0.476</td><td>0.574</td>
  </tr>
  <tr>
    <td>Method</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
    <td>MRR</td><td>Hit@10</td>
  </tr>
  <tr>
    <td>DT</td><td>0.220</td><td>0.321</td><td>0.342</td><td>0.381</td><td>0.415</td><td>0.472</td><td>0.435</td><td>0.516</td><td>0.449</td><td>0.534</td><td>0.452</td><td>0.542</td><td>0.453</td><td>0.544</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.152</td><td>0.209</td><td>0.261</td><td>0.379</td><td>0.334</td><td>0.463</td><td>0.375</td><td>0.493</td><td>0.419</td><td>0.526</td><td>0.438</td><td>0.545</td><td>0.453</td><td>0.544</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.162</td><td>0.220</td><td>0.281</td><td>0.360</td><td>0.363</td><td>0.442</td><td>0.417</td><td>0.495</td><td>0.437</td><td>0.523</td><td>0.446</td><td>0.544</td><td>0.453</td><td>0.544</td>
  </tr>
  <tr>
    <td>PairRE</td>
    <td>Ext-V</td><td>0.172</td><td>0.261</td><td>0.306</td><td>0.374</td><td>0.389</td><td>0.456</td><td>0.420</td><td>0.498</td><td>0.441</td><td>0.529</td><td>0.446</td><td>0.543</td><td>0.453</td><td>0.544</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.228</td><td>0.336</td><td>0.375</td><td>0.413</td><td>0.421</td><td>0.483</td><td>0.443</td><td>0.525</td><td>0.451</td><td>0.536</td><td>0.453</td><td>0.542</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.245</td><td>0.349</td><td>0.381</td><td>0.426</td><td>0.468</td><td>0.448</td><td>0.534</td><td>0.452</td><td>0.537</td><td>0.453</td><td>0.543</td><td>-</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.232</td><td>0.336</td><td>0.377</td><td>0.424</td><td>0.462</td><td>0.459</td><td>0.485</td><td>0.534</td><td>0.452</td><td>0.537</td><td>0.453</td><td>0.543</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.235</td><td>0.336</td><td>0.379</td><td>0.423</td><td>0.426</td><td>0.495</td><td>0.459</td><td>0.533</td><td>0.450</td><td>0.538</td><td>0.452</td><td>0.543</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td>0.317</td><td>0.376</td><td>0.408</td><td>0.467</td><td>0.433</td><td>0.502</td><td>0.449</td><td>0.537</td><td>0.451</td><td>0.541</td><td>0.451</td><td>0.542</td><td>0.451</td><td>0.542</td>
  </tr>
</table>

Table 9: MRR and Hit@10 of some representative dimensions on WN18RR.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">10d</th>
    <th colspan="2">20d</th>
    <th colspan="2">40d</th>
    <th colspan="2">80d</th>
    <th colspan="2">160d</th>
    <th colspan="2">320d</th>
    <th colspan="2">640d</th>
  </tr>
  <tr>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>DT</td><td>0.202</td><td>0.011</td><td>0.291</td><td>0.016</td><td>0.385</td><td>0.018</td><td>0.401</td><td>0.025</td><td>0.403</td><td>0.027</td><td>0.407</td><td>0.033</td><td>0.412</td><td>0.034</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.201</td><td>0.016</td><td>0.285</td><td>0.023</td><td>0.338</td><td>0.023</td><td>0.364</td><td>0.028</td><td>0.384</td><td>0.033</td><td>0.388</td><td>0.028</td><td>0.412</td><td>0.034</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.218</td><td>0.029</td><td>0.317</td><td>0.025</td><td>0.361</td><td>0.039</td><td>0.403</td><td>0.046</td><td>0.405</td><td>0.036</td><td>0.408</td><td>0.033</td><td>0.412</td><td>0.034</td>
  </tr>
  <tr>
    <td>TransE</td>
    <td>0.218</td><td>0.029</td><td>0.314</td><td>0.045</td><td>0.391</td><td>0.051</td><td>0.407</td><td>0.047</td><td>0.408</td><td>0.036</td><td>0.411</td><td>0.027</td><td>0.412</td><td>0.034</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.216</td><td>0.035</td><td>0.331</td><td>0.040</td><td>0.392</td><td>0.033</td><td>0.401</td><td>0.031</td><td>0.404</td><td>0.030</td><td>0.407</td><td>0.032</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.224</td><td>0.040</td><td>0.343</td><td>0.043</td><td>0.395</td><td>0.037</td><td>0.408</td><td>0.030</td><td>0.407</td><td>0.030</td><td>0.410</td><td>0.034</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.226</td><td>0.037</td><td>0.346</td><td>0.043</td><td>0.394</td><td>0.037</td><td>0.408</td><td>0.031</td><td>0.408</td><td>0.031</td><td>0.411</td><td>0.034</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.217</td><td>0.032</td><td>0.345</td><td>0.044</td><td>0.392</td><td>0.036</td><td>0.407</td><td>0.030</td><td>0.408</td><td>0.031</td><td>0.407</td><td>0.033</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.269</b></td><td><b>0.040</b></td><td><b>0.369</b></td><td><b>0.045</b></td><td><b>0.399</b></td><td><b>0.038</b></td><td><b>0.404</b></td><td><b>0.042</b></td><td><b>0.407</b></td><td><b>0.037</b></td><td><b>0.410</b></td><td><b>0.033</b></td><td><b>0.412</b></td><td><b>0.031</b></td>
  </tr>
  <tr>
    <td>Method</td>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>DT</td><td>0.061</td><td>0.028</td><td>0.297</td><td>0.193</td><td>0.361</td><td>0.289</td><td>0.406</td><td>0.343</td><td>0.420</td><td>0.382</td><td>0.428</td><td>0.386</td><td>0.433</td><td>0.391</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.003</td><td>0.001</td><td>0.055</td><td>0.023</td><td>0.181</td><td>0.114</td><td>0.249</td><td>0.168</td><td>0.377</td><td>0.329</td><td>0.422</td><td>0.381</td><td>0.433</td><td>0.391</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.004</td><td>0.003</td><td>0.051</td><td>0.031</td><td>0.187</td><td>0.128</td><td>0.389</td><td>0.333</td><td>0.413</td><td>0.365</td><td>0.429</td><td>0.384</td><td>0.433</td><td>0.391</td>
  </tr>
  <tr>
    <td>SimplE</td>
    <td>0.004</td><td>0.002</td><td>0.050</td><td>0.029</td><td>0.269</td><td>0.205</td><td>0.378</td><td>0.349</td><td>0.409</td><td>0.372</td><td>0.426</td><td>0.382</td><td>0.433</td><td>0.391</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.077</td><td>0.034</td><td>0.331</td><td>0.225</td><td>0.384</td><td>0.311</td><td>0.415</td><td>0.358</td><td>0.426</td><td>0.371</td><td>0.431</td><td>0.385</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.093</td><td>0.042</td><td>0.349</td><td>0.269</td><td>0.375</td><td>0.349</td><td>0.412</td><td>0.384</td><td>0.425</td><td>0.388</td><td>0.431</td><td>0.389</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.086</td><td>0.038</td><td>0.361</td><td>0.285</td><td>0.391</td><td>0.368</td><td>0.416</td><td>0.383</td><td>0.427</td><td>0.389</td><td>0.434</td><td>0.392</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.079</td><td>0.033</td><td>0.355</td><td>0.279</td><td>0.382</td><td>0.356</td><td>0.415</td><td>0.379</td><td>0.424</td><td>0.383</td><td>0.433</td><td>0.389</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.119</b></td><td><b>0.048</b></td><td><b>0.366</b></td><td><b>0.292</b></td><td><b>0.395</b></td><td><b>0.359</b></td><td><b>0.419</b></td><td><b>0.380</b></td><td><b>0.429</b></td><td><b>0.389</b></td><td><b>0.435</b></td><td><b>0.391</b></td><td><b>0.434</b></td><td><b>0.390</b></td>
  </tr>
  <tr>
    <td>Method</td>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>DT</td><td>0.304</td><td>0.005</td><td>0.436</td><td>0.357</td><td>0.475</td><td>0.393</td><td>0.487</td><td>0.420</td><td>0.489</td><td>0.423</td><td>0.491</td><td>0.428</td><td>0.493</td><td>0.429</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.315</td><td>0.257</td><td>0.399</td><td>0.335</td><td>0.452</td><td>0.395</td><td>0.472</td><td>0.415</td><td>0.480</td><td>0.413</td><td>0.470</td><td>0.418</td><td>0.493</td><td>0.429</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.224</td><td>0.166</td><td>0.359</td><td>0.288</td><td>0.420</td><td>0.352</td><td>0.441</td><td>0.373</td><td>0.461</td><td>0.396</td><td>0.481</td><td>0.417</td><td>0.493</td><td>0.429</td>
  </tr>
  <tr>
    <td>RotatE</td>
    <td>0.289</td><td>0.197</td><td>0.336</td><td>0.234</td><td>0.377</td><td>0.263</td><td>0.402</td><td>0.293</td><td>0.442</td><td>0.357</td><td>0.467</td><td>0.397</td><td>0.493</td><td>0.429</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.312</td><td>0.009</td><td>0.452</td><td>0.361</td><td>0.479</td><td>0.403</td><td>0.487</td><td>0.421</td><td>0.490</td><td>0.424</td><td>0.492</td><td>0.425</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.314</td><td>0.010</td><td>0.452</td><td>0.363</td><td>0.481</td><td>0.408</td><td>0.489</td><td>0.420</td><td>0.488</td><td>0.422</td><td>0.492</td><td>0.425</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.320</td><td>0.011</td><td>0.452</td><td>0.364</td><td>0.483</td><td>0.412</td><td>0.489</td><td>0.423</td><td>0.488</td><td>0.426</td><td>0.491</td><td>0.425</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.311</td><td>0.013</td><td>0.439</td><td>0.356</td><td>0.479</td><td>0.407</td><td>0.484</td><td>0.423</td><td>0.488</td><td>0.425</td><td>0.493</td><td>0.424</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.354</b></td><td><b>0.277</b></td><td><b>0.476</b></td><td><b>0.409</b></td><td><b>0.486</b></td><td><b>0.418</b></td><td><b>0.490</b></td><td><b>0.422</b></td><td><b>0.492</b></td><td><b>0.424</b></td><td><b>0.493</b></td><td><b>0.427</b></td><td><b>0.495</b></td><td><b>0.428</b></td>
  </tr>
  <tr>
    <td>Method</td>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>DT</td><td>0.271</td><td>0.174</td><td>0.368</td><td>0.313</td><td>0.428</td><td>0.384</td><td>0.450</td><td>0.399</td><td>0.463</td><td>0.405</td><td>0.462</td><td>0.406</td><td><b>0.464</b></td><td><b>0.407</b></td>
  </tr>
  <tr>
    <td>Ext</td><td>0.165</td><td>0.120</td><td>0.292</td><td>0.198</td><td>0.366</td><td>0.267</td><td>0.398</td><td>0.314</td><td>0.437</td><td>0.364</td><td>0.452</td><td>0.388</td><td>0.464</td><td>0.407</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.175</td><td>0.129</td><td>0.302</td><td>0.237</td><td>0.383</td><td>0.319</td><td>0.431</td><td>0.377</td><td>0.450</td><td>0.395</td><td>0.455</td><td>0.400</td><td>0.464</td><td>0.407</td>
  </tr>
  <tr>
    <td>PairRE</td>
    <td>0.194</td><td>0.124</td><td>0.323</td><td>0.269</td><td>0.407</td><td>0.352</td><td>0.435</td><td>0.379</td><td>0.452</td><td>0.398</td><td>0.458</td><td>0.400</td><td>0.464</td><td>0.407</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.279</td><td>0.180</td><td>0.384</td><td>0.345</td><td>0.372</td><td>0.452</td><td>0.405</td><td>0.460</td><td>0.452</td><td>0.405</td><td>0.463</td><td>0.408</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.203</td><td>0.107</td><td>0.387</td><td>0.332</td><td>0.417</td><td>0.280</td><td>0.410</td><td>0.344</td><td>0.452</td><td>0.409</td><td>0.463</td><td>0.408</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.281</td><td>0.175</td><td>0.389</td><td>0.314</td><td>0.437</td><td>0.381</td><td><b>0.463</b></td><td><b>0.409</b></td><td>0.463</td><td>0.410</td><td>0.465</td><td>0.410</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.285</td><td>0.172</td><td>0.390</td><td>0.331</td><td>0.435</td><td>0.377</td><td>0.461</td><td>0.405</td><td>0.463</td><td>0.411</td><td>0.464</td><td>0.410</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.314</b></td><td><b>0.259</b></td><td><b>0.426</b></td><td><b>0.367</b></td><td><b>0.443</b></td><td><b>0.392</b></td><td><b>0.462</b></td><td><b>0.405</b></td><td><b>0.464</b></td><td><b>0.406</b></td><td><b>0.465</b></td><td><b>0.407</b></td><td>0.464</td><td>0.406</td>
  </tr>
</table>

Table 10: Hit@3 and Hit@1 of some representative dimensions on WN18RR.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">10d</th>
    <th colspan="3">20d</th>
    <th colspan="3">40d</th>
    <th colspan="3">80d</th>
    <th colspan="3">160d</th>
    <th colspan="3">320d</th>
    <th colspan="3">640d</th>
  </tr>
  <tr>
    <th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th><th>MRR</th><th>Hit@10</th>
  </tr>
  <tr>
    <td>DT</td><td>0.150</td><td>0.235</td><td>0.277</td><td>0.440</td><td>0.299</td><td>0.477</td><td>0.313</td><td>0.484</td><td>0.315</td><td>0.499</td><td>0.318</td><td>0.501</td><td>0.322</td><td>0.508</td>
  </tr>
  <tr>
    <td>Ext</td><td>0.115</td><td>0.211</td><td>0.191</td><td>0.324</td><td>0.236</td><td>0.392</td><td>0.266</td><td>0.436</td><td>0.286</td><td>0.462</td><td>0.299</td><td>0.479</td><td>0.322</td><td>0.508</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.109</td><td>0.194</td><td>0.175</td><td>0.293</td><td>0.232</td><td>0.381</td><td>0.263</td><td>0.424</td><td>0.285</td><td>0.462</td><td>0.301</td><td>0.484</td><td>0.322</td><td>0.508</td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.139</td><td>0.256</td><td>0.200</td><td>0.348</td><td>0.231</td><td>0.396</td><td>0.270</td><td>0.437</td><td>0.293</td><td>0.466</td><td>0.308</td><td>0.488</td><td>0.322</td><td>0.508</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.176</td><td>0.293</td><td>0.279</td><td>0.446</td><td>0.303</td><td>0.480</td><td>0.315</td><td>0.500</td><td>0.315</td><td>0.501</td><td>0.320</td><td>0.502</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.179</td><td>0.301</td><td>0.281</td><td>0.443</td><td>0.306</td><td>0.483</td><td>0.316</td><td>0.502</td><td>0.319</td><td>0.505</td><td>0.322</td><td>0.508</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.179</td><td>0.301</td><td>0.281</td><td>0.443</td><td>0.306</td><td>0.483</td><td>0.316</td><td>0.502</td><td>0.319</td><td>0.505</td><td>0.322</td><td>0.508</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.176</td><td>0.285</td><td>0.276</td><td>0.446</td><td>0.307</td><td>0.482</td><td>0.315</td><td>0.503</td><td>0.317</td><td>0.505</td><td>0.319</td><td>0.505</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.196</b></td><td><b>0.341</b></td><td><b>0.290</b></td><td><b>0.472</b></td><td><b>0.308</b></td><td><b>0.486</b></td><td><b>0.317</b></td><td><b>0.502</b></td><td><b>0.320</b></td><td><b>0.505</b></td><td><b>0.321</b></td><td><b>0.507</b></td><td><b>0.322</b></td><td><b>0.507</b></td>
  </tr>
  <tr>
    <td>DT</td><td>0.097</td><td>0.179</td><td>0.176</td><td>0.321</td><td>0.236</td><td>0.390</td><td>0.217</td><td>0.431</td><td>0.285</td><td>0.458</td><td>0.291</td><td>0.467</td><td><b>0.295</b></td><td><b>0.472</b></td>
  </tr>
  <tr>
    <td>Ext</td><td>0.037</td><td>0.068</td><td>0.069</td><td>0.107</td><td>0.090</td><td>0.144</td><td>0.159</td><td>0.258</td><td>0.202</td><td>0.372</td><td>0.269</td><td>0.432</td><td>0.293</td><td>0.472</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.045</td><td>0.059</td><td>0.056</td><td>0.062</td><td>0.083</td><td>0.146</td><td>0.114</td><td>0.205</td><td>0.196</td><td>0.316</td><td>0.258</td><td>0.421</td><td>0.293</td><td>0.472</td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.049</td><td>0.069</td><td>0.066</td><td>0.101</td><td>0.105</td><td>0.149</td><td>0.138</td><td>0.224</td><td>0.224</td><td>0.369</td><td>0.261</td><td>0.414</td><td>0.295</td><td>0.472</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.113</td><td>0.204</td><td>0.182</td><td>0.315</td><td>0.244</td><td>0.412</td><td>0.275</td><td>0.439</td><td>0.287</td><td>0.463</td><td>0.293</td><td>0.470</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.124</td><td>0.221</td><td>0.192</td><td>0.329</td><td>0.254</td><td>0.416</td><td>0.276</td><td>0.448</td><td>0.290</td><td>0.465</td><td>0.295</td><td>0.471</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.120</td><td>0.213</td><td>0.195</td><td>0.346</td><td>0.258</td><td><b>0.429</b></td><td>0.279</td><td>0.443</td><td><b>0.293</b></td><td><b>0.466</b></td><td><b>0.296</b></td><td><b>0.468</b></td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.120</td><td>0.215</td><td>0.193</td><td>0.338</td><td>0.257</td><td>0.427</td><td>0.281</td><td>0.440</td><td>0.293</td><td>0.465</td><td>0.297</td><td>0.468</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.143</b></td><td><b>0.267</b></td><td><b>0.233</b></td><td><b>0.384</b></td><td><b>0.261</b></td><td><b>0.427</b></td><td><b>0.279</b></td><td><b>0.448</b></td><td><b>0.291</b></td><td><b>0.466</b></td><td><b>0.293</b></td><td><b>0.468</b></td><td>0.294</td><td>0.470</td>
  </tr>
  <tr>
    <td>DT</td><td>0.254</td><td>0.424</td><td>0.297</td><td>0.477</td><td>0.312</td><td>0.495</td><td>0.317</td><td>0.502</td><td>0.322</td><td>0.506</td><td>0.323</td><td>0.510</td><td><b>0.325</b></td><td><b>0.515</b></td>
  </tr>
  <tr>
    <td>Ext</td><td>0.138</td><td>0.245</td><td>0.203</td><td>0.340</td><td>0.251</td><td>0.410</td><td>0.276</td><td>0.443</td><td>0.291</td><td>0.465</td><td>0.305</td><td>0.485</td><td>0.325</td><td>0.515</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.135</td><td>0.243</td><td>0.188</td><td>0.319</td><td>0.221</td><td>0.365</td><td>0.246</td><td>0.402</td><td>0.280</td><td>0.453</td><td>0.299</td><td>0.477</td><td>0.325</td><td>0.515</td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.160</td><td>0.281</td><td>0.198</td><td>0.340</td><td>0.238</td><td>0.393</td><td>0.265</td><td>0.427</td><td>0.288</td><td>0.458</td><td>0.302</td><td>0.478</td><td>0.325</td><td>0.515</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.277</td><td>0.442</td><td>0.305</td><td>0.485</td><td>0.314</td><td>0.503</td><td>0.321</td><td>0.508</td><td>0.322</td><td>0.510</td><td>0.323</td><td>0.509</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.280</td><td>0.447</td><td>0.306</td><td>0.485</td><td>0.313</td><td>0.501</td><td>0.319</td><td>0.507</td><td>0.323</td><td>0.510</td><td>0.323</td><td>0.509</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.282</td><td>0.449</td><td>0.307</td><td>0.486</td><td>0.315</td><td>0.502</td><td>0.318</td><td>0.507</td><td>0.322</td><td><b>0.512</b></td><td><b>0.324</b></td><td><b>0.514</b></td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.276</td><td>0.445</td><td>0.306</td><td>0.482</td><td>0.317</td><td>0.504</td><td>0.319</td><td>0.508</td><td>0.323</td><td>0.512</td><td>0.324</td><td>0.513</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.288</b></td><td><b>0.459</b></td><td><b>0.311</b></td><td><b>0.492</b></td><td><b>0.318</b></td><td><b>0.504</b></td><td><b>0.322</b></td><td><b>0.509</b></td><td><b>0.323</b></td><td><b>0.510</b></td><td><b>0.324</b></td><td><b>0.512</b></td><td>0.324</td><td>0.514</td>
  </tr>
  <tr>
    <td>DT</td><td>0.182</td><td>0.314</td><td>0.243</td><td>0.395</td><td>0.284</td><td>0.452</td><td>0.307</td><td>0.476</td><td>0.319</td><td>0.505</td><td>0.328</td><td>0.518</td><td><b>0.332</b></td><td><b>0.522</b></td>
  </tr>
  <tr>
    <td>Ext</td><td>0.148</td><td>0.222</td><td>0.177</td><td>0.289</td><td>0.217</td><td>0.353</td><td>0.259</td><td>0.416</td><td>0.294</td><td>0.469</td><td>0.321</td><td>0.506</td><td>0.332</td><td>0.522</td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.150</td><td>0.249</td><td>0.196</td><td>0.294</td><td>0.219</td><td>0.333</td><td>0.271</td><td>0.436</td><td>0.309</td><td>0.489</td><td>0.326</td><td>0.513</td><td>0.332</td><td>0.522</td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.176</td><td>0.277</td><td>0.192</td><td>0.303</td><td>0.229</td><td>0.374</td><td>0.279</td><td>0.450</td><td>0.311</td><td>0.490</td><td>0.329</td><td>0.513</td><td>0.332</td><td>0.522</td>
  </tr>
  <tr>
    <td>BKD</td><td>0.198</td><td>0.332</td><td>0.251</td><td>0.407</td><td>0.288</td><td>0.453</td><td>0.311</td><td>0.487</td><td>0.321</td><td>0.508</td><td>0.330</td><td>0.521</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>TA</td><td>0.206</td><td>0.346</td><td>0.263</td><td>0.430</td><td>0.294</td><td>0.455</td><td>0.314</td><td>0.493</td><td>0.323</td><td>0.509</td><td>0.332</td><td>0.521</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.207</td><td>0.342</td><td>0.261</td><td>0.437</td><td>0.293</td><td>0.461</td><td>0.314</td><td>0.494</td><td>0.324</td><td>0.512</td><td>0.324</td><td>0.524</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.205</td><td>0.340</td><td>0.264</td><td>0.431</td><td>0.293</td><td>0.462</td><td>0.314</td><td>0.494</td><td>0.324</td><td>0.512</td><td>0.324</td><td>0.522</td><td>-</td><td>-</td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.239</b></td><td><b>0.384</b></td><td><b>0.274</b></td><td><b>0.437</b></td><td><b>0.303</b></td><td><b>0.466</b></td><td><b>0.314</b></td><td><b>0.495</b></td><td><b>0.324</b></td><td><b>0.510</b></td><td><b>0.329</b></td><td><b>0.521</b></td><td>0.330</td><td>0.520</td>
  </tr>
</table>

Table 11: MRR and Hit@10 of some representative dimensions on FB15K237.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">10d</th>
    <th colspan="3">20d</th>
    <th colspan="3">40d</th>
    <th colspan="3">80d</th>
    <th colspan="3">160d</th>
    <th colspan="3">320d</th>
    <th colspan="3">640d</th>
  </tr>
  <tr>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>DT</td><td>0.169</td><td>0.102</td><td>0.301</td><td>0.190</td><td>0.327</td><td>0.212</td><td>0.340</td><td>0.218</td><td>0.348</td><td>0.222</td><td>0.353</td><td>0.224</td><td>0.358</td><td>0.228</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext</td><td>0.123</td><td>0.065</td><td>0.211</td><td>0.122</td><td>0.264</td><td>0.156</td><td>0.296</td><td>0.180</td><td>0.320</td><td>0.197</td><td>0.331</td><td>0.208</td><td>0.358</td><td>0.228</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.118</td><td>0.065</td><td>0.192</td><td>0.115</td><td>0.256</td><td>0.157</td><td>0.292</td><td>0.180</td><td>0.316</td><td>0.198</td><td>0.333</td><td>0.210</td><td>0.358</td><td>0.228</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.150</td><td>0.081</td><td>0.222</td><td>0.126</td><td>0.265</td><td>0.156</td><td>0.301</td><td>0.185</td><td>0.325</td><td>0.205</td><td>0.341</td><td>0.217</td><td>0.358</td><td>0.228</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>TransE</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>BKD</td><td>0.178</td><td>0.106</td><td>0.308</td><td>0.198</td><td>0.336</td><td>0.208</td><td>0.349</td><td>0.222</td><td>0.349</td><td>0.223</td><td>0.354</td><td>0.226</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>TA</td><td>0.188</td><td>0.112</td><td>0.307</td><td>0.200</td><td>0.336</td><td>0.212</td><td>0.348</td><td>0.220</td><td>0.353</td><td>0.225</td><td>0.355</td><td>0.223</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.193</td><td>0.115</td><td>0.307</td><td>0.201</td><td>0.337</td><td>0.216</td><td>0.351</td><td>0.223</td><td>0.354</td><td>0.226</td><td>0.356</td><td>0.227</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.187</td><td>0.112</td><td>0.299</td><td>0.185</td><td>0.333</td><td>0.214</td><td>0.351</td><td>0.222</td><td>0.353</td><td>0.223</td><td>0.354</td><td>0.224</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.215</b></td><td><b>0.122</b></td><td><b>0.321</b></td><td><b>0.199</b></td><td><b>0.338</b></td><td><b>0.218</b></td><td><b>0.347</b></td><td><b>0.223</b></td><td><b>0.351</b></td><td><b>0.226</b></td><td><b>0.356</b></td><td><b>0.227</b></td><td>0.358</td><td>0.227</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">10d</th>
    <th colspan="3">20d</th>
    <th colspan="3">40d</th>
    <th colspan="3">80d</th>
    <th colspan="3">160d</th>
    <th colspan="3">320d</th>
    <th colspan="3">640d</th>
  </tr>
  <tr>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>DT</td><td>0.103</td><td>0.055</td><td>0.193</td><td>0.105</td><td>0.256</td><td>0.161</td><td>0.297</td><td>0.191</td><td>0.314</td><td>0.197</td><td>0.323</td><td>0.208</td><td><b>0.324</b></td><td><b>0.211</b></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext</td><td>0.039</td><td>0.019</td><td>0.071</td><td>0.047</td><td>0.091</td><td>0.057</td><td>0.171</td><td>0.109</td><td>0.251</td><td>0.159</td><td>0.294</td><td>0.187</td><td>0.324</td><td>0.211</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.043</td><td>0.035</td><td>0.048</td><td>0.037</td><td>0.111</td><td>0.040</td><td>0.131</td><td>0.093</td><td>0.216</td><td>0.134</td><td>0.281</td><td>0.177</td><td>0.324</td><td>0.211</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.047</td><td>0.036</td><td>0.074</td><td>0.043</td><td>0.097</td><td>0.077</td><td>0.145</td><td>0.109</td><td>0.248</td><td>0.156</td><td>0.289</td><td>0.189</td><td>0.324</td><td>0.211</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>SimpLE</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>BKD</td><td>0.123</td><td>0.064</td><td>0.20</td><td>0.115</td><td>0.261</td><td>0.164</td><td>0.299</td><td>0.191</td><td>0.308</td><td>0.202</td><td>0.318</td><td><b>0.213</b></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>TA</td><td>0.133</td><td>0.073</td><td>0.210</td><td>0.123</td><td>0.276</td><td>0.175</td><td>0.302</td><td>0.195</td><td>0.318</td><td>0.203</td><td>0.323</td><td>0.211</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.130</td><td>0.071</td><td>0.224</td><td>0.115</td><td>0.279</td><td>0.175</td><td>0.305</td><td>0.196</td><td>0.324</td><td><b>0.208</b></td><td><b>0.326</b></td><td>0.211</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.132</td><td>0.069</td><td>0.217</td><td>0.118</td><td>0.276</td><td>0.174</td><td>0.303</td><td>0.192</td><td><b>0.326</b></td><td>0.204</td><td>0.324</td><td>0.212</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.164</b></td><td><b>0.073</b></td><td><b>0.254</b></td><td><b>0.158</b></td><td><b>0.288</b></td><td><b>0.177</b></td><td><b>0.305</b></td><td><b>0.196</b></td><td>0.319</td><td>0.205</td><td>0.318</td><td>0.209</td><td>0.322</td><td>0.209</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="3">10d</th>
    <th colspan="3">20d</th>
    <th colspan="3">40d</th>
    <th colspan="3">80d</th>
    <th colspan="3">160d</th>
    <th colspan="3">320d</th>
    <th colspan="3">640d</th>
  </tr>
  <tr>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
    <th>Hit@3</th><th>Hit@1</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>DT</td><td>0.284</td><td>0.168</td><td>0.330</td><td>0.207</td><td>0.346</td><td>0.223</td><td>0.352</td><td>0.224</td><td>0.353</td><td>0.229</td><td>0.357</td><td>0.230</td><td><b>0.363</b></td><td><b>0.234</b></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext</td><td>0.152</td><td>0.080</td><td>0.225</td><td>0.129</td><td>0.278</td><td>0.170</td><td>0.304</td><td>0.190</td><td>0.322</td><td>0.203</td><td>0.335</td><td>0.217</td><td>0.363</td><td>0.234</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext-L</td><td>0.147</td><td>0.078</td><td>0.209</td><td>0.121</td><td>0.247</td><td>0.146</td><td>0.275</td><td>0.166</td><td>0.312</td><td>0.193</td><td>0.333</td><td>0.209</td><td>0.363</td><td>0.234</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Ext-V</td><td>0.174</td><td>0.097</td><td>0.218</td><td>0.126</td><td>0.264</td><td>0.159</td><td>0.293</td><td>0.182</td><td>0.319</td><td>0.201</td><td>0.336</td><td>0.213</td><td>0.363</td><td>0.234</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>RotatE</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>BKD</td><td>0.306</td><td>0.193</td><td>0.338</td><td>0.214</td><td>0.352</td><td>0.224</td><td>0.354</td><td>0.230</td><td>0.356</td><td>0.230</td><td>0.358</td><td>0.231</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>TA</td><td>0.308</td><td>0.196</td><td>0.339</td><td>0.216</td><td>0.353</td><td>0.225</td><td>0.358</td><td>0.229</td><td>0.359</td><td>0.229</td><td>0.358</td><td>0.231</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>DualDE</td><td>0.311</td><td>0.197</td><td>0.341</td><td>0.216</td><td>0.353</td><td><b>0.227</b></td><td><b>0.360</b></td><td>0.230</td><td>0.361</td><td>0.232</td><td>0.361</td><td>0.233</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>IterDE</td><td>0.307</td><td>0.195</td><td>0.342</td><td>0.215</td><td>0.355</td><td>0.225</td><td>0.359</td><td><b>0.232</b></td><td><b>0.363</b></td><td>0.233</td><td>0.362</td><td><b>0.234</b></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>MED</td><td><b>0.324</b></td><td><b>0.201</b></td><td><b>0.344</b></td><td><b>0.216</b></td><td><b>0.355</b></td><td>0.225</td><td>0.357</td><td>0.231</td><td>0.358</td><td>0.233</td><td>0.362</td><td>0.233</td><td>0.362</td><td>0.232</td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
</table>

<table>
  <tr>
    <th rowspan="2">dim</th>
    <th colspan="4">MED</th>
    <th colspan="4">MED w/o MLM</th>
    <th colspan="4">MED w/o EIM</th>
    <th colspan="4">MED w/o DLW</th>
  </tr>
  <tr>
    <th>MRR</th><th>Hit@10</th><th>Hit@3</th><th>Hit@1</th>
    <th>MRR</th><th>Hit@10</th><th>Hit@3</th><th>Hit@1</th>
    <th>MRR</th><th>Hit@10</th><th>Hit@3</th><th>Hit@1</th>
    <th>MRR</th><th>Hit@10</th><th>Hit@3</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>10</td><td>.170</td><td>.388</td><td>.269</td><td>.036</td><td>.149</td><td>.335</td><td>.234</td><td>.032</td><td>.169</td><td>.388</td><td>.267</td><td>.037</td><td>.171</td><td>.387</td><td>.268</td><td>.035</td>
  </tr>
  <tr>
    <td>20</td><td><b>.219</b></td><td><b>.491</b></td><td><b>.369</b></td><td>.042</td><td>.197</td><td>.437</td><td>.323</td><td>.032</td><td>.217</td><td>.488</td><td>.366</td><td>.044</td><td>.218</td><td>.487</td><td>.367</td><td>.039</td>
  </tr>
  <tr>
    <td>40</td><td><b>.232</b></td><td><b>.518</b></td><td><b>.399</b></td><td>.048</td><td>.224</td><td>.496</td><td>.379</td><td>.029</td><td>.232</td><td>.517</td><td>.403</td><td>.042</td><td>.232</td><td>.517</td><td>.402</td><td>.037</td>
  </tr>
  <tr>
    <td>80</td><td><b>.232</b></td><td><b>.523</b></td><td><b>.404</b></td><td>.042</td><td>.228</td><td>.521</td><td>.399</td><td>.033</td><td>.235</td><td>.529</td><td>.408</td><td>.037</td><td>.234</td><td>.523</td><td>.410</td><td>.041</td>
  </tr>
  <tr>
    <td>160</td><td><b>.236</b></td><td><b>.529</b></td><td><b>.407</b></td><td>.034</td><td>.234</td><td>.525</td><td>.406</td><td>.034</td><td>.234</td><td>.527</td><td>.405</td><td>.032</td><td>.235</td><td>.527</td><td>.405</td><td>.032</td>
  </tr>
  <tr>
    <td>320</td><td><b>.237</b></td><td><b>.536</b></td><td><b>.410</b></td><td>.033</td><td>.236</td><td>.532</td><td>.409</td><td>.035</td><td>.233</td><td>.530</td><td>.398</td><td>.031</td><td>.234</td><td>.533</td><td>.405</td><td>.029</td>
  </tr>
  <tr>
    <td>640</td><td><b>.237</b></td><td><b>.537</b></td><td><b>.412</b></td><td>.031</td><td><b>.238</b></td><td>.535</td><td>.412</td><td>.042</td><td>.232</td><td>.528</td><td>.402</td><td>.029</td><td>.233</td><td>.530</td><td>.396</td><td>.025</td>
  </tr>
</table>

Table 12: Hit@3 and Hit@1 of some representative dimensions on FB15K237.

Table 13: Ablation study on WN18RR with TransE.
(a) TransE on WN18RR    (b) SimplE on WN18RR    (c) RotatE on WN18RR    (d) PairRE on WN18RR
(e) TransE on FB15K237   (f) SimplE on FB15K237   (g) RotatE on FB15K237   (h) PairRE on FB15K237

Figure 6: Performance of sub-models of MED and the directly trained (DT) KGEs of dimensions from 10 to 640.

B.1 Mutual Learning Mechanism (MLM)
We remove the mutual learning mechanism from MED and keep the other parts unchanged, where (6) is rewritten as
\[
L = \sum_{i=1}^{n} \exp \left( \frac{w_3 \cdot d_i}{d_n} \right) \cdot L_{EI}^i.
\]
From the result of “MED w/o MLM” in Table 13, we find that after removing the mutual learning mechanism, the performance of low-dimensional sub-models deteriorates seriously since the low-dimensional sub-models can not learn from the high-dimensional sub-models. For example, the MRR of the 10-dimensional sub-model decreased by 12.4%, and the MRR of the 20-dimensional sub-model decreased by 10%. While the performance degradation of the high-dimensional sub-model is not particularly obvious, and the MRR of the highest-dimensional sub-model (\(dim = 640\)) is not worse than that of MED, which is because to a certain degree, removing the mutual learning mechanism also avoids the negative influence to high-dimensional sub-models from low-dimensional sub-models. On the whole, this mechanism greatly improves the performance of low-dimensional sub-models.

B.2 Evolutionary Improvement Mechanism (EIM)
In this part, we replace evolutionary improvement loss \(L_{EI}^i\) in (6) with the regular KGE loss \(L_{KGE}^i\):
\[
L_{KGE}^i = \sum_{(h,r,t)\in T\cup T^-} y \log \sigma(s_{(h,r,t)}^i) \\
+(1-y) \log(1-\sigma(s_{(h,r,t)}^i)).
\]
From the result of “MED w/o EIM” in Table 13, we find that removing the evolutionary improvement mechanism mainly degrades the performance of high-dimensional sub-models. While due to the existence of the mutual learning mechanism, the low-dimensional sub-model can still learn from the high-dimensional sub-model, so as to ensure the certain performance of the low-dimensional sub-model. In addition, we also find that as the dimension increases to a certain extent, the performance of the sub-model does not improve, and even begins to decline. We guess that this is because the mutual learning mechanism makes every pair of neighbor sub-models learn from each other, resulting in some low-quality or wrong knowledge gradually transferring from the low-dimensional sub-models to the high-dimensional sub-models, and when the evolutionary improvement mechanism is removed, the high-dimensional sub-models can no longer correct the wrong information from the low-dimensional sub-models. The higher the dimension of the sub-model, the more the accumulated error, so the performance of the high-dimensional sub-models is seriously damaged. On the whole, this mechanism mainly helps to improve the effect of high-dimensional sub-models.

B.3 Dynamic Loss Weight (DLW)
To study the effect of the dynamic loss weight, we fix the ratio of all mutual learning losses to all evolutionary improvement losses as 1 : 1, and (6) is rewritten as
\[
L = \sum_{i=2}^{n} L_{ML}^{i-1,i} + \sum_{i=1}^{n} L_{EI}^i.
\]
According to the result of “MED w/o DLW” in
Table 13, the overall results of “MED w/o DLW” are in the middle of the results of “MED w/o MLM” and “MED w/o EIM”: the performance of the low-dimensional sub-model is better than that of “MED w/o MLM”, and the performance of the high-dimensional sub-model is better than that of “MED w/o EIM”. On the whole, its results are more similar to “MED w/o EIM”, that is, the performance of the low-dimensional sub-model does not change much, while the performance of the high-dimensional sub-model decreases more significantly. We believe that for the high-dimensional sub-model, the proportion of mutual learning loss is still too large, which makes it more negatively affected by the low-dimensional sub-model. This result indicates that the dynamic loss weight plays a role in adaptively balancing multiple losses and contributes to improving overall performance.

C Details of applying the trained KGE by MED to real applications

The SKG is used in many tasks related to users, and injecting user embeddings trained over SKG into downstream task models is a common and practical way.

User labeling is one of the common user management tasks that e-commerce platforms run on backend servers. We model user labeling as a multiclass classification task for user embeddings with a 2-layer MLP:

\[
\mathcal{L} = - \frac{1}{|\mathcal{U}|} \sum_{i=1}^{|\mathcal{U}|} \sum_{j=1}^{|\mathcal{CLS}|} y_{ij} \log(\mathrm{MLP}(u_i)),
\]

where \( u_i \) is the \( i \)-th user’s embedding, the label \( y_{ij} = 1 \) if user \( u_i \) belongs to class \( cls_j \), otherwise \( y_{ij} = 0 \).

The product recommendation task is to properly recommend items to users that users will interact with a high probability and it often runs on terminal devices. Following PKGM (Zhang et al., 2021), which recommends items to users using the neural collaborative filtering (NCF) (He et al., 2017) framework with the help of pre-trained user embeddings as service vectors, we add trained user embeddings over SKG as service vectors to NCF. In NCF, the MLP layer is used to learn item-user interactions based on the latent feature of the user and item, that is, for a given user-item pair \( user_i - item_j \), the interaction function is

\[
\phi_1^{MLP}(p_i, q_j) = \mathrm{MLP}([p_i; q_j]),
\]

where \( p_i \) and \( q_j \) are latent feature vectors of user and item learned in NCF. We add the trained user embedding \( u_i \) to NCF’s MLP layer and rewrite Equation (11) as

\[
\phi_1^{MLP}(p_i, q_j, u_i) = \mathrm{MLP}([p_i; q_j; u_i]),
\]

and the other parts of NCF stay the same as in PKGM (Zhang et al., 2021).

We train entity and relation embeddings for SKG based on TransE (Bordes et al., 2013) and input the trained entity (user) embedding into Equation (10) and Equation (12).

D Details of extending MED to language model BERT-base

D.1 Dataset and Evaluation Metric

For the experiments extending MED to BERT, we adopt the common GLUE (Wang et al., 2019) benchmark for evaluation. To be specific, we use the development set of the GLUE benchmark which includes four tasks: Paraphrase Similarity Matching, Sentiment Classification, Natural Language Inference, and Linguistic Acceptability. For Paraphrase Similarity Matching, we use MRPC (Dolan and Brockett, 2005), QQP and STS-B (Conneau and Kiela, 2018) for evaluation. For Sentiment Classification, we use SST-2 (Socher et al., 2013). For Natural Language Inference, we use MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), and RTE for evaluation. In terms of evaluation metrics, we follow previous work (Devlin et al., 2019; Sun et al., 2019a). For MRPC and QQP, we report F1 and accuracy. For STS-B, we consider Pearson and Spearman correlation as our metrics. The other tasks use accuracy as the metric. For MNLI, the results of MNLI-m and MNLI-mm are both reported separately.

D.2 Baselines

For comparison, we choose Knowledge Distillation (KD) models and Hardware-Aware Transformers (Wang et al., 2020a) (HAT) customized for transformers as baselines. For the KD models, we compare MED with Basic KD (BKD) (Hinton et al., 2015), Patient KD (PKD) (Sun et al., 2019a), Relational Knowledge Distillation (RKD) (Park et al., 2019), Deep Self-attention Distillation (MiniLM) (Wang et al., 2020b), Meta Learning-based KD (MetaDistill) (Zhou et al., 2022a) and Feature Structure Distillation (FSD) (Jung et al., 2023). For the comparability of the results, we
choose 4-layer BERT (BERT_4) or 6-layer BERT (BERT_6) as the student model architectures, which guarantees that the number of model parameters (\#P(M)) or *speedup* is comparable. For HAT, we use the same model architecture as our MED for training and show the results of sub-models with three parameter scales.

D.3 Implementation

To implement MED on BERT, for the word embedding layer, all sub-models share the front portion of embedding parameters in the same way as in KGE, and for the transformer layer, all sub-models share the front portion of weight parameters as in HAT (Wang et al., 2020a). Specifically, assuming that the embedding dimension of the largest BERT model \( B_n \) is \( d_n \), and the embedding dimension of the sub-model \( B_i \) is \( d_i \), for any parameter matrix with the shape \( x \times y \) in \( B_n \), the front portion sub-matrix of it with the shape \( \frac{d_i}{d_n} x \times \frac{d_i}{d_n} y \) is the parameter matrix of the corresponding position in \( B_i \). Finally, it just need to replace the triple score \( s_{(h,r,t)} \) in Equation (2), Equation (3), Equation (4), and Equation (5) with the logits output for the corresponding category of the classifier in the classification task.

We set \( n = 4 \) for BERT applying MED, and 4 sub-models have the following settings: [768, 512, 256, 128] for embedding dim and [768, 512, 256, 128] for hidden dim, [12, 12, 6, 6] for the head number in attention modules, 12 for encoder layer number.

E Visual analysis of embedding

We select four primary entity categories (‘organization’, ‘sports’, ‘location’, and ‘music’) that contain more than 300 entities in FB15K237, and randomly select 250 entities for each. We cluster these entities’ embeddings of 3 different dimensions (\( d=10, 100, 600 \)) by the t-SNE algorithm, and the clustering results are visualized in Fig. 7. Under the same dimension, the clustering result of MED is always the best, followed by DualDE, while the result of Ext-V is generally poor, which is consistent with the conclusion in Section 5.2. We also find some special phenomenons for MED when dimension increases: 1) the nodes of the ‘sports’ gradually become two clusters meaning MED learns more fine-grained category information as dimension increases, and 2) the relative distribution among different categories hardly changes and shows a trend of “inheritance” and “improvement”. This further proves MED achieves our expectation that high-dimensional sub-models retain the ability of low-dimensional sub-models, and can learn more knowledge than low-dimensional sub-models.

![Clustering on FB15K237 with RotatE.](page_97_120_1257_496.png)

Figure 7: Clustering on FB15K237 with RotatE.

Reviewer paper summaries: This paper addresses the issue that different edge devices require knowledge graphs of varying scales (dimensions). It proposes a method to train multiple knowledge graphs of different dimensions simultaneously, aiming to improve deployment efficiency and reduce the cost of repetitive training. The core idea is to use a high-dimensional model to guide low-dimensional models through knowledge distillation and introduce a mutual learning mechanism.
Summary of strengths: - It tackles the pain point in practical applications where different devices need knowledge graphs of different scales, which has strong practical relevance.
- By using high-dimensional models to guide low-dimensional ones, it achieves model compression while maintaining performance. The method is relatively simple and straightforward.
- The overall idea is innovative, proposing the concept of training models of multiple dimensions in a single process, thus reducing the workload of model training and maintenance.
Summary of weaknesses: - Experimental results indicate that performance improvements mainly stem from the distillation of the high-dimensional model to the low-dimensional ones. There is a lack of adequate experimental verification and quantitative analysis on the actual effect and necessity of the mutual learning mechanism.
- The core method's description is not thorough enough, lacking detailed explanations of algorithm specifics, model parameters, and the training process, which affects the credibility of the results and the reproducibility of the work.
-  The paper lacks comparative studies to fully demonstrate the advantages of the proposed method over existing ones. The experimental metrics are limited, and the analysis is not comprehensive.
- The structure and language expression of the article need improvement. The quality of some figures (such as vector diagrams) is not high, which affects the professionalism and readability of the paper.
Comments, suggestions and typos: - Can you provide more detailed experiments to quantify the actual contribution of the mutual learning mechanism and compare it with the effects of using only the distillation method?
- Could you include a detailed description of the algorithm, including model architecture, training strategies, and parameter settings, to enhance the method's transparency and reproducibility?

This paper presents a promising solution to a significant problem in practical applications, and the overall concept exhibits a degree of novelty. However, the current manuscript lacks depth in method exposition, experimental validation, and scholarly writing. The effectiveness of the mutual learning mechanism has not been fully demonstrated, and the absence of methodological details and insufficient experimentation undermines the paper's persuasiveness.
Overall assessment scores: 3.5
Metareview score: 3
Metareview: The paper proposes MED, an innovative framework for training croppable knowledge graph embeddings (KGEs) that support multiple embedding dimensions within a single model. The paper claims that the proposed approach can addresses practical deployment needs across devices with different resource constraints, offering efficiency and flexibility.
Summary of reasons to publish: The paper proposed an interesting approach to obtain multiple KG embeddings with a single model, which seems interesting.