You CAN Teach an Old Dog New Tricks! On Training Knowledge Graph Embeddings
Authors: Daniel Ruffinelli
Samuel Broscheit
Rainer Gemulla
Venue:  (2020)
Keywords: knowledge graph embeddings
hyperparameter optimization

Abstract: Knowledge graph embedding (KGE) models learn algebraic representations of the entities and relations in a knowledge graph. A vast number of KGE techniques for multi-relational link prediction have been proposed in the recent literature, often with state-of-the-art performance. These approaches differ along a number of dimensions, including different model architectures, different training strategies, and different approaches to hyperparameter optimization. In this paper, we take a step back and aim to summarize and quantify empirically the impact of each of these dimensions on model performance. We report on the results of an extensive experimental study with popular model architectures and training strategies across a wide range of hyperparameter settings. We found that when trained appropriately, the relative performance differences between various model architectures often shrinks and sometimes even reverses when compared to prior results. For example, RESCAL~\citep{nickel2011three}, one of the first KGE models, showed strong performance when trained with state-of-the-art techniques; it was competitive to or outperformed more recent architectures. We also found that good (and often superior to prior studies) model configurations can be found by exploring relatively few random samples from a large hyperparameter space. Our results suggest that many of the more advanced architectures and techniques proposed in the literature should be revisited to reassess their individual benefits. To foster further reproducible research, we provide all our implementations and experimental results as part of the open source LibKGE framework.

Published as a conference paper at ICLR 2020
YOU CAN TEACH AN OLD DOG NEW TRICKS! ON TRAINING KNOWLEDGE GRAPH EMBEDDINGS
Daniel Ruffinelli*
Samuel Broscheit
Data and Web Science Group University of Mannheim, Germany
Data and Web Science Group
University of Mannheim, Germany
Rainer Gemulla
Data and Web Science Group University of Mannheim, Germany
ABSTRACT
Knowledge graph embedding (KGE) models learn algebraic representations of the entities and relations a knowledge graph. vast number of KGE techniques for multi-relational link prediction have been proposed in the recent literature, of- ten with state-of-the-art performance. These approaches differ along a number of dimensions, including different model architectures, different training strategies, and different approaches to hyperparameter optimization. In this paper, we take step back and aim to summarize and quantify empirically the impact of each of these dimensions on model performance. We report on the results of an extensive experimental study with popular model architectures and training strategies across a wide range of hyperparameter settings. We found that when trained appropri- ately, the relative performance differences between various model architectures often shrinks and sometimes even reverses when compared to prior results. For example, RESCAL (Nickel et al.) (011), one of the first KGE models, showed strong performance when trained with state-of-the-arr techniques; it was compet- itive to or outperformed more recent architectures. We also found that good (and often superior to prior studies) model configurations can be found by exploring relatively few random samples from a large hyperparameter space. Our results suggest that many of the more advanced architectures and techniques proposed in the literature should be revisited to reassess their individual benefits. To foster fur- ther reproducible research, provide all our implementations and experimental results as part of the open source LibKGE framework.
INTRODUCTION
Knowledge graph embedding (KGE) models learn algebraic representations, termed embeddings, of the entities and relations in a knowledge graph. They have been successfully applied to knowledge graph completion (Nickel et al. 2015) as well as in downstream tasks and applications such as recommender systems (Wang et al. (017) or visual relationship detection (Baier et al. 2017).
vast number of different KGE models for multi-relational link prediction have been proposed in the recent literature; e.g., RESCAL (Nickel et al.] 2011), TransE (Bordes et al. 2013). DistMult, ComplEx (Truillon et al. 2016). ConvE (Dettmers et al. 2018)) TuckER (Balazevic et al. 2019), RotatE (Sun al.] 2019a), SACN (Shang et al. 2019), and many more. Model architectures gen- erally differ in the way the entity and relation embeddings are combined to model the presence triple) in the knowledge graph; they include factorization models (e.g., RESCAL, DistMult, ComplEx, TuckER), translational mod- els (TransE, RotatE), and more advanced models such as convolutional models (ConvE). In many cases, the introduction of new models went along with new approaches for training these models- e.g., new training types (such as negative sampling or vsslll scoring), new loss functions (such as
absence of an edge (more precisely, pubjjcct-redicate--bo
"Contributed equally. broscheitttttt /iinfornatti ik, uni -ma-mmmaiii......... im I Irgemul 1 i -mannhe de

---

Published as a conference paper at ICLR 2020
Publication
Model
Loss
Training
Regularizer
Optimizer Reciprocal
Nickel et al. (2011)
RESCAL
MSE
Full
L2
ALS
No
Bordes et al.
(2013
TransE
MR
NegSamp
Normalization
SGD
No
Yang et al. 2015
DistMult
MR
NegSamp
Weighted 1.2
Adagrad
No
Trouillon et al. (2016)
ComplEx
BCE
NegSamp
Weighted L2
Adagrad
No
Kadlec et al. (207)
DistMult
CE
NegSamp
Weighted L2
Adam
No
Dettmers et al. (2018
ConvE
BCE
KvsAll
Dropout
Adam
Yes
Lacroix et al.
2018)
ComplEx
CE
1vsAll
Weighted L3
Adagrad
Yes
MSE mean squared error, MR margin ranking, BCE binary cross entropy, CE cross entropy
Table 1: Selected KGE models and training strategies from the literature. Entries marked in bold were introduced (or first used) in the context of KGE in the corresponding publication.
pairwise margin ranking or binary cross entropy), new forms of regularization (such as unweighted and weighted L2), or the use of reciprocal relations (Kazemi & Poole 2018 Lacroix et al.) 2018 and ablation studies were not always performed. Table I shows training techniques along with the publications that introduced them.
ssowss an overview of selected models and
The diversity in model training makes it difficult to compare performance results for various model architectures, especially when results are reproduced from prior studies that used different ex- perimental setup. Model hyperparameters are commonly tuned using grid search on small grid involving hand-crafted parameter ranges or settings known to "work well" from prior studies. A grid suitable for one model may be suboptimal for another, however. Indeed, it has been observed that newer training strategies can considerably improve model performance (Kadlee et al., 2017 Lacroix al. 2018 Salehi al. 2018).
In this paper, we take a step back and aim to summarize and quantify empirically the impact of different model architectures and different training strategies model performance. We performed an extensive set of experiments using popular model architectures and training strategies in a com- mon experimental setup. In contrast most prior work, we considered many training strategies as well as a large hyperparameter space, and we performed model selection using quasi-random search (instead of grid search) followed by Bayesian optimization. We found that this approach was able to find good (and often superior to prior studies) model configurations with relatively low effort. Our study complements and expands on the results of Kotnis & Nastase (2018) (focus on negative sampling) and Mohamed et al. (2019) (focus on loss functions) as well as similar studies in other ar- eas, including language modeling (Melis et al. 2017), generative adversarial networks (Lucic al.) 2018), or sequence tagging (Reimers & Gurevych 2017).
We found that when trained appropriately, the performance of a particular model architecture can by far exceed the performance observed in older studies. For example, RESCAL (Nickel et al., 2011). which constitutes one of the first KGE models but is rarely considered in newer work, showed very strong performance in study: it was competitive to outperformed more recent architectures such as ConvE (Dettmers et al., 2018) and TuckER (Balazevic et al., 2019). More generally, we found that the relative performance differences between various model architectures often shrunk and sometimes even reversed when compared to prior results. This suggests that (at least currently) training strategies have significant impact on model performance and may account for substantial fraction of the progress made in recent years. We also found that suitable training strategies and hyperparameter settings vary significantly across models and datasets, indicating that a small search grid may bias results on model performance. Fortunately, as indicated above, large hyperparameter spaces can be (and should be) used with little additional training effort. To facilitate such efforts, we provide implementations of relevant training strategies, models, and evaluation methods part of the open source LibKGE framework [ which emphasizes reproducibility and extensibility.
Our study focuses solely on pure KGE models, which do not exploit auxiliary information such as textual data or logical rules (Wang al.] 2017). Since many of the studies on these non-pure models did not (and, to be fair, could not) use current training strategies and consequently underestimated the performance of pure KGE models, their results and conclusions need to be revisited.
https://gitthie com.b...ommuue uma www.uuaaaappllaaae
2

---

Published as a conference paper at ICLR 2020
2 KNOWLEDGE GRAPH EMBEDDINGS: MODELS, TRAINING, EVALUATION
The literature on KGE models expanding rapidly. We review selected architectures, training methods, and evaluation protocols; see T The table examplarily indicates that new model ar- chitectures are sometimes introduced along with new training strategies (marked bold). Reasonably recent survey articles about KGE models include|Nickel et al. (2015) and W et al. (2017).
rulti-relational link prediction. KGE models are typically trained and evaluated in the context of nulti-relational link prediction for knowledge graphs (KG). Generally, given a set & of entities and a set R of relations, a knowledge graph K. C 3 x R x E is a set of (spo) triples. The goal of multi-relational link prediction is to "complete the KG", i.e., to predict true but unobserved triples based on the information in IC. Common approaches include rule-based methods (Galarraga et al.; [0033] Meilicke et al. 2019), KGE methods (Nickel et al., 2011 Bordes et al. Trouillon et al., 2016; Dettmers et al. 2018). and hybrid methods (Guo et al. 2018).
Knowledge graph embeddings (KGE), A KGE model associates with each entity i E E and re- lation & E RR an embedding e; E gd and Th E Rd in low-dimensional vector space, respec- tively; here de, d, E N+ are hyperparameters for the embedding size. Each particular model uses a scoring function $ : 3 x R x 3 + R to associate a score s(i. k.j) with each potential triple (i, j, E & X R x E. The higher the score of a triple, the more likely it is considered to be true by the model. The scoring function takes form s(i, j) s(es, rk ez), i.e., depends on i, k; and only through their respective embeddings. Here f, which represents the model architecture, may be either fixed function or learned (e.g.. f may be parameterized function).
The most common evaluation task for KGE methods is entity ranking, which is a form of question answering. The available data is partitioned into set of training, validation, and test triples. Given a test triple (i, j) (unseen during training), the entity ranking task is to determine the correct answer i.e., the missing entity j and resp.- to questions (i, k. ?) and (?, j). To do so, potential answer triples are first ranked by their score in descending order. All triples but (4, j) test data are subsequently filtered out that other triples
Evaluation.
that occur in the training, validation, known to be true do not affect the ranking. Finally, metrics such as the mean reciprocal rank (MRR) of the true answer or the average HITS @ are computed; see the appendix.
KGE models. KGE model architectures differ in their scoring function. We can roughly clas- sify models as decomposable or monolithie: the former only allow element- interactions be- tween trelation-specifid subject and object embeddings, whereas the latter allow arbitrary in- teractions. More specifically, decomposable models use scoring functions of form s(i, k) J(E: (ei, er.) o hz (er, e))]=),, where o is any element-wise function (e.g., multiplication), hn and lz are functions that obtain relation-specific subject and object embeddings, resp., and g and f are scalar functions (c.g.. identity sigmoid). The most popular models in this category are perhaps RESCAL (Nickel et al. 2011). TransE (Bordes et al. 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), and ConvE (Dettmers et al.] 2018). RESCAL's scoring function is bilinear in the entity embeddings: it uses s(2, k) = e, R.e,, where R& E R.d.xd xd, is a matrix formed from the entries of rk E Rd. (where d, = d.). DistMult and ComplEx can be seen constrained variants of RESCAL with smaller relation embeddings (d, d.). TransE is vranslation-basel model and uses negative distance -Ile; + r eillp between e; + r and e, as score, commonly using the LI norm (p- 1) or the L2 norm (p- = 2). Finally, ConvE uses 2D con- volutional layer and a large fully connected layer to obtain relation-specific entity embeddings (i.e.. in by above). Other recent examples for decomposable models include TuckER (Balazevic et al.) 2019), RotatE (Sun et al. 2019a), and SACN (Shang et al. 2019). Decomposable models are gen- erally fast to use: once the relation-specifee embeddings are (pre-)compptee score computations are cheap. Monolithic models-suee as ConvKB or KBGAT-do do not decompose into celation-speciffe embeddings: they take form a(i, j) = f(e,, Tk; e,). Such models are more flexible, but they are also considerably more costly to train and use. It is currently unclear whether monolithic models can achieve comparable or better performance than decomposable models Sun et al. (2019b).
Training type. There are three commonly used approaches to train KGE models, which dif- fer mainly in the way negative examples are generated. First, training with negative sampling (NegSamp) (Bordes et al. 2013)) obtains for each positive triple t (i, k. j) from the training data a set of (eseudo-)negatton triples obtained by randomly perturbing the subject, relation, or ob- ject position in (and optionally verifying that the so-obtained triples do not exist in the KG). An
3

---

Published as a conference paper at ICLR 2020
alternative approach (Lacroix et al., 2018). which we term /vsAll, is to omit sampling and take all triples that can be obtained by perturbing the subject and object positions as negative examples for t (even if these tuples exist in the KG). IvsAll is generally more expensive than NegSamp, but it is feasible (and even surprisingly fast in practice) if the number of entities is not excessively large. Finally, Dettmers et al. (0118) proposed training type that term KvsA! this approach (i) con- structs batches from non-empty rows (2, k *) or (*, k, instead of from individual triples, and (ii) labels all such triples as either positíve (occurs in training data) or negative (otherwise).
Loss functions. Several loss functions for training KGBs have been introduced so far. RESCAL originally used squared error between the score of each triple and its label (positive or negative). TransE used pairwise margin ranking with hinge loss (MR), where each pair consists of positive triple and one of its negative triples (only applicable to NegSamp and IvsAll) and the margin v) is a hyperparameter. Trouillon et al. (2016) proposed to use binary cross entropy (BCE) loss: it applies a sigmoid the score of each (positive or negative) triple and uses the cross entropy between the resulting probability and that triple's label as loss. BCE is suitable for multi-class and multi-label classification. Finally, Kadlec et al. (2017) used cross entropy (CE) between the model distribution (softmax distribution over scores) and the data distribution (labels of corresponding triples, normal- ized to sum to 1). CE is more suitable for multi-class classification (as in NegSamp and IvsAll), but it has also been used in the multi-label setting (Kv) Mohamed al.] (2019) found that the choice of loss function can have a significant impact on model performance, and that the best choice data and model dependent. Our experimental study provides additional evidence for this finding.
Reciprocal relations. Kazemi & Poole (2018) and Lacroix et al. (2018) introduced the technique of reciprocal relations into KGE training. Observe that during evaluation and also most training methods discussed above, the model is solely asked to score subjects (for questions of form (?, k,j)) or objects (for questions of form (i. k, ?)). The idea of reciprocal relations use separate scoring functions Suhh and saj for each of these tasks, resp. Both scoring functions share entity embeddings, but they do not share relation embeddings: each relation thus has two embeddings The use of reciprocal relations may decrease the computational cost (as in the case of ConvE), and it may also lead to better model performance [acroix et al. (2018) (e.g., for relations in which one direction easier to predict). On the downside, the use of reciprocal relations means that a model does not provide single triple score s(i, k) anymore; generally, Syub (i, k. F Soty(i, (i, j). Kazemi & Poole 2018) proposed to take the average of the two triple scores and explored the resulting models.
Regularization. The most popular form of regularization in the literature is L2 regularization on the embedding vectors, either unweighted or weighted by the frequency of the corresponding entity or relation (Yang et al. 2015). Lacroix al. (018)) proposed use L3 regularization. TransE normal- ized the embeddings to unit norm after each update. ConvE used dropout (Srivastava et al., (201 in its hidden layers (and only in those). In our study, we additionally consider LI regularization and the use of dropout on the entity and/or relation embeddings.
Hyperparameters. Many more hyperparameters have been used in prior work. This includes, for example, different methods to initialize model parameters, different optimizers. different optimizer parameters such as the learning rate or batch size, the number of negative examples for NegSamp, the regularization weights for entities and relations, and so on. To deal with such a large search space, most prior studies favor grid search over a small grid where most of these hyperparameters remain fixed. As discussed before, this approach may lead to bias in the results, however.
3 EXPERIMENTAL STUDY
In this section, we report on the design and results of our experimental study, We focus on the most salient points here; more details can be found in the appendix.
NNte that the KvsAll strategy is called 1-N scoring in Dettmers et al. (018).
AAleernatieeyy, we can view such approach as only predicting objects, but doing for both each original relation (k) and new reciprocal relation formed by switching subject and object (k- '
4

---

Published as a conference paper at ICLR 2020
3.1 EXPERIMENTAL SETUP
Datasets. We used the FB15K-237 (Toutanova & Chen 2015)
(extracted from Freebase) and WNRR (Dettmers et al. 2018) (extracted from WordNet) datasets our study. We chose these datasets because (i) they are frequently used in prior studies, (ii) they are "hard" datasets that have been designed specifically to evaluate multi-relational link prediction techniques, (iii) they are di- verse in that relative model performance often differs, and (iv) they are of reasonable size for large study. Dataset statistics are given in Table in the appendix.
Models. We selected RESCAL (Nckel et al. 2011). TransE (Bordes et al. 2013). DistMult (Yang (et al. 2015). ComplEx (Trouillon et al. (016)) and ConvE (Dettmers et al. 22018) for our study. These models are perhaps the most well-known KGE models and include both early and more recent models. We did not consider monolithic models due to their excessive training cost.
Evaluation. We report filtered MRR (%) and filtered HITS@ 10 (%); see Sec. and the appendix for details. We use test data to compare final model performance (Table 2 and validation data for our more detailed analysis.
Hyperparameters. We used a large hyperparameter space to ensure sure that suitable hyperparam- eters for each model are not excluded priori. We included all major training types (NegSamp, IvsAll, KvsAll), use of reciprocal relations, loss functions (MR, BCE, CE). regularization tech- niques (n0ne/L.1/L.2/L.3, dropout), optimizers (Adam, Adagrad), and initialization methods (4 in total) used in the KGE community as hyperparameters. We considered three embeddings sizes (128, 256, 512) and used separate weights for dropouu/regularizatin for entity and relation embeddings. Table 5 in the appendix provides additional details. To the best of our knowledge, prior study used such large hyperparameter search space.
Training. All models were trained for maximum of 400 epochs. We validated models using filtered MRR (on validation data) every five epochs and performed carly stopping with a patience of 50 epochs. To keep search tractable, stopped training models that did not reach > 5% filtered MRR after 50 epochs; in our experience, such configurations did not produce good models.
Model selection. Model selection was performed using filtered MRR validation data. We used the Ax framework (https: //a dev/) to conduct quasi-random hyperparameter search via a Sobol sequence. Quasi-random search methods aim to distribute hyperparameter settings evenly and try avoid "clumping" effects (Bergstra & Bengio 2012). More specifically, for each dataset and model, we generated 30 different configurations per valid combination of training type and loss function (2 for TransE, which only supports NegSamp+MR and NegSamp+CE: 7 for all other models). After the quasi-random hyperparameter search, we added a short Bayesian optimization phase (best configuration so far 20 new trials, using expected improvement; also provided by Ax) to tune the numerical hyperparameters further. Finally, we trained five models with the so-obtained hyperparameter configuration and selected the best-performing model according to validation MRR as the final model. The standard deviation of the validation MRR was relatively low; see Table 9
Reproducibility. We implemented all models, training strategies, evaluation, and hyperparameter search in the LibKGE framework. The framework emphasizes reproducibility and extensibility and is highly configurable. We provide all configurations used in this study as well as the detailed data of our experiments for further analysis We hope that these resources facilitate the evaluation of KGE models in a comparable environment.
3.2 COMPARISON OF MODEL PERFORMANCE
the filtered MRR and filtered Hits @ 10 on test data of various models both from prior studies and the best models (according filtered validation MRR) found in our study.
First reported performance vs. our observed performance. We compared the first reported per- formance on FB15K-237 and WNRR ("First" block of Table (2) with the performance obtained in our study ("Ours" block). We found that the performance of single model can vary wildly across studies. For example, ComplEx was first run on FB15K-237 by Dettmers al. (2018), where achieved filtered MRR of 24.7%. This is relatively low number by today's 's standards. In our
https://github. http://.oooommmmaap///////----- wttpppppp///pppppe
5

---

Published as a conference paper at ICLR 2020
FBI5K-237
WNRR
MRR
Hits@ 10
MRR
Hits @ 10
RESCAL WWag al. 2019)
27.0
42.7
42.0
44.7
TransE (N guyen el al. 2018
29.4
46.5
22.6
50.1
First
DistMult Dettmerrs al. 2018
24.1
41.9
43.0
49.0
ComplEx DDttmers et al. 2018
24.7
42.8
44.0
51.0
ConvE DDttmmers al. 2018)
32.5
50.1
43.0
52.0
RESCAL
35.7
54.1
46.7
51.7
Ours
TransE
31.3
49.7
22.8
52.0
DistMult
34.3
53.1
45.2
53.1
ComplEx
34.8
53.6
47.5
54.7
ConvE
33.9
52.1
44.2
50.4
Large Recent
TuckER (Balazevic et al.
2019)
35.8
54.4
47.0
52.6
RotatE (un et al. 2019a)
33.8
53.3
47.6
57.1
SACN
Shang et al.
2019
35.0
54.0
47.0
54.4
DistMult (Salehi al.
2018
35.7
54.8
45.5
54.4
ComplEx N3
Lacroix et al. 2018
37.0
56.0
49.0
58.0
Table 2: Model performance in prior studies and our study (as percentages, on test data). First:
first reported performance on the respective datasets (oldest models first); Ours: performance in our study; Recent: best performance results obtained in prior studies of selected recent models; Large: best performance achieved in prior studies using very large models (not part our search space). Bold numbers indicate best performance in group. References indicate where the performance num- ber was reported.
study, ComplEx achieved a competitive MRR 34.8%, which is large improvement over the first reports. Studies that report the lower performance number of ComplEx (i.e., 24.7%) thus do not ad- equately capture its performance. Similar remarks hold for RESCAL and DistMult as well as (albeit to smaller extent) ConvE and TransE. To the best of our knowledge, our results for RESCAL and ConvE constitute the best results for these models obtained so far.
Relative model performance (our study). Next, we compared the performance across the mod- els models used our study ("Ours" block). We found that the relative performance differences between model architectures often shrunk and sometimes reversed when compared to prior results ("First" block). For example, ConvE showed the best overall performance in prior studies, but is consistently outperformed by RESCAL, DistMult, and ComplEx in study. As another example, RESCAL NNicel et al., 201T). which constitutes one of the first KGE models but rarely consid- ered in newer work, showed strong performance and outperformed all models except ComplEx. This suggests that (at least currently) training strategies have a significant impact on model performance and may account for large fraction of the progress made in recent years.
Relative model performance (overall). Table@aadiitionll] shows the best performance results ob- tained in prior studies of selected recent models ("Recent" block) and very large models with very good performance ("Large" block). We found that TuckER (Balazevic et al. 2019), RotatE (Sun) al. 2019a), and SACN (Shang al. 2019) all achieve state-of-the-art performance, but the per- formance difference to the best prior models ("Ours" block) in terms of MRR is small or even vanishes. Even for HITS 10, which we did consider for model selection, the advantage of more recent models is often small, if present. The models in the last block ("Large"") show the best performance numbers we have seen in the literature. For DistMult and ComplEx, these numbers have been obtained using very large embedding sizes (up to 4000). Such large models were not part of our search space.
Limitations. We note that all models used in our study are likely to yield better performance when hyperparameter tuning is further improved. For example, we found a ComplEx configuration (of size 100) that achieved 49.0% MRR and 56.5% Hits @ 10 WNRR, and RESCAL configuration that achieved 36.3% MRR and 54.6% Hits@ 10 on FB15K-237 preliminary experiments, [Snn al. (2019a) report TransE configuration that achieved 33.3% MRR (52.2% Hits 10)

---

Published as a conference paper at ICLR 2020
FB15K-237
WNRR
Validation MRR
30
20
0
RESCAL TransE DistMult ComplEx ConvE
RESCAL TransE DistMult ComplEx ConvE
Figure I: Distribution of filtered MRR (%) on validation data over the quasi-random hyperparameter configurations explored in our study.
FB15K-237. Since the focus of this study is to compare models in a common experimental setup and without manual intervention, we do not further report on these results.
3.3 IMPACT OF HYPERPARAMETERS
Anatomy of search space. Figure shows the distribution of filtered MRR for each model and dataset (on validation data, Sobol configurations only). Each distribution consists of roughly 200 different quasi-random hyperparameter configurations (except TransE, for which 60 configurations have been used). Perhaps unsurprisingly, we found that all models showed a wide dispersion on both datasets and only very few configurations achieved ofate-of-the-art results. There are, however, notable differences across datasets and models. On FB15K-237, for example, the median ConvE configuration best; on WNRR, DistMult and Complex have much higher median MRR. Gener- ally. the impact of the hyperparameter choice is more pronounced on WNRR (higher variance) than on FB FB15--237.
Best configurations (quasi-random search). The hyperparameters of the best performing models during our quasi-random search are reported in Table 3 (selected hyperparameters) and Tables 7 (in the appendix, all hyperparameters). First, we found that the optimum choice of hyperpa- rameters often model and dataset dependent: with the exception of the loss function (discussed below), almost hyperparameter setting was consistent across datasets and models. For exam- ple, the NegSample training type was frequently selected on FB15K-233 but not on WNRR, where KvsAll more prominent. Our findings provide further evidence that grid search on small search space not suitable to compare model performance because the result may be heavily influ- enced by the specific grid points being used. Moreover, the budget that used for quasi-random search was comparable to grid search over a small (or at least not very large) grid: we tested merely 30 different model configurations for every combination of training type and loss function. It is therefore both feasible and beneficial to work with large search space.
Best configurations (Bayesian optimization). We already obtained good configurations solely from quasi-random search. After Bayesian optimization (which we used to tune the numerical hy- perparameters), the performance on test data almost always improved only marginally. The best hyperparameters after the Bayesian optimization phase are reported in Table in in the appendix.
Impact specific hyperparameters. is difficult asses the impact of each hyperparameter in- dividually. As proxy for the importance of each hyperparameter setting, we report in Table and and Tables and 7 (appendix) the best performance in terms of filtered MRR on validation data for different choice of value for each hyperparameter (difference in parenthesis). For example, the best configuration during quasi-random search for RESCAL on FB15K 1B155-233 237 did not use reciprocal rela- tions. The best configuration that did use reciprocal relations had an MRR that (35.6 instead of 1). This does not mean that the gap is explained by the use of the reciprocal re- lations alone, other hyperparameters may also be different, but it does show the best performance
0.5 points lower

---

Published as a conference paper at ICLR 2020
RESCAL
TransE
DistMult
ComplEx
ConvE
Valid. MRR
36.1
31.5
35.0
35.3
34.3
Emb. size
128 (-0.5)
512 (-3.7)
256 (-0.2)
256 (-0.3)
256 (-0.4)
FB15K-237
Batch size
512 (-0.5)
128 (-7.1)
1024 (-0.2)
1024 (-0.3)
1024 (-0.4)
Train type
IvsAll (-0.8) NegSamp
NegSamp (-0.2) NegSamp (-0.3) IvsAll (-0.4)
Loss
CE (-0.9)
CE (-7.1)
CE (-3.1)
CE (-3.8)
CE (-0.4)
Optimizer
Adam (-0.5) Adagrad (-3.7)
Adagrad (-0.2)
Adagrad (-0.5) Adagrad (-1.5)
Initializer
Normal (-0.8) XvNorm (-3.7)
Unif. (-0.2)
Unif. (-0.5) XvNorm (-0.4)
Regularizer
None (-0.5)
1.2 (-3.7)
(-0.2)
L.3 (-0.3)
L3 (-0.4)
Reciprocal
No (-0.5)
Yes (-9.5)
Yes (-0.3)
Yes (-0.3)
Yes
Valid. MRR
46.8
22.6
45.4
47.6
44.3
Emb. size
128 (-1.0)
(-5.1)
512 (-1.1)
128 (-1.0)
512 (-1.2)
Batch size
128 (-1.0)
128 (-5.1)
1024 (-1.1)
512 (-1.0)
1024 (-1.3)
WNRR
Train type
KvsAll (-1.0) NegSamp
KvsAll (-1.1)
1vsAll (-1.0)
KvsAll (-1.2)
Loss
CE (-2.0)
CE (-5.1)
CE (-2.4)
CE (-3.5)
CE (-1.4)
Optimizer
Adam (-1.2) Adagrad (-5.8)
Adagrad (-1.5)
Adagrad (-1.5)
Adam (-1.4)
Initializer
Unif. (-1.0)
XvNorm (-5.1)
Unif. (-1.3)
Unif. (-1.5) XvNorm (-1.4)
Regularizer
L3 (-1.2) L2 (-5.1) L3 (-1.1) L2 (-1.0) LI (-1.2)
Reciprocal
Yes (-1.0)
Yes (-5.9)
Yes (-1.1) No (-1.0)
Yes
Table 3: Hyperparameters of best performing models after quasi-random hyperparameter search and Bayesian optimization filtered MRR (on validation data). For each hyperparameter, also give the reduction in filtered MRR for the best configuration that does not use this value of the hyperparameter (in parenthesis).
RESCAL
TransE
DistMult
ComplEx
ConvE
Validation MRR
40
F815K-237
30
10
Validation MRR
40
WNRR
30
10
NegSamp-as MMR
NBgSamp+BCE
1vSAB-BCE
KvsAlI+BCE
NegSamp+CE
1vsABI+CE
KvsAI+CC
Figure 2: Distribution filtered MRR (%) on validation data over the quasi-random hyperparameter configurations for different training type and loss functions.
obtained when enforcing reciprocal relations. We can see that most hyperparameters appear to have moderate impact on model performance: many hyperparameter settings did not produce a sig- nificant performance improvement that others did not also produce. A clear exception here is the of the loss function, which is consistent across models and datasets (always CE), partly with large performance improvements alternative loss functions.
8

---

Published as a conference paper at ICLR 2020
FB15K-237
WNRR
Validation MRR
40
40
RESCAL
TransE
35
35
DistMult
ComplEx
30
30
ConvE
25
20
20
100
200
300
400
100
200
300
400
Epoch
Epoch
Figure 3: Best filtered MRR (%) on validation data achieved during quasi-random search as a func- tion of the number of training epochs. Changes in the corresponding "best-so-far" hyperparameter configuration are marked with a dot.
Impact of loss function. In order to further explore the impact of the loss function, we show the distribution of the filtered MRR validation data for every (valid) combination of training type and loss function in Figure I0 We found that the use of CE (the three rightmost bars) generally lead to better configurations than using other losses. This surprising (and somewhat unsatisfying) that we use the CE loss, which is multi-class loss, for a multi-label task. Similar observations have been made for computer vision tasks (Joulin al.) 2016 Mahajan al. 2018) though. Note that the combination of KvsAll with CE (on the very right in figure) has not been explored previously.
3.4 MODEL SELECTION
We briefly summarize the behavior of various models during model selection. Figure 3 shows the best validation MRR (over all quasi-random hyperparameter configurations) achieved by each model when trained for the specified number of epochs. The corresponding hyperparameter configuration may change models were trained longer; in the figure, marked such changes with dot. We found that a suitable hyperparameter configuration as well as a good model can be found using significantly less than the 400 epochs used in our study. Nevertheless, longer training did help in many settings and some models may benefit when trained for even more than 400 epochs.
CONCLUSION
We reported the results of an extensive experimental study with popular KGE model architectures and training strategies across wide range hyperparameter settings. We found that when trained appropriately, the relative performance differences between various model architectures often shrunk and sometimes even reversed when compared prior results. This suggests that (at least currently) training strategies have a significant impact on model performance and may account for a substan- tial fraction of the progress made recent years. To enable an insightful model comparison, we recommend that new studies compare models in common experimental setup using sufficiently large hyperparameter search space. Our study shows that such an approach is possible with reason- able computational cost, and provide implementation as part of the LibKGE framework to facilitate such model comparison. Moreover, we feel that many the more advanced architectures and techniques proposed in the literature need to be revisited reassess their individual benefits.

---

Published as a conference paper at ICLR 2020
REFERENCES
Stephan Baier, Yunpu Ma, and Volker Tresp. Improving visual relationship detection using semantic modeling of scene descriptions. Proceedings of the 16th International Semantic Web Confer- ence (ISWC), 2017.
Ivana Balazevic, Carl Allen, and Timothy Hospedales. TuckER: Tensor factorization for knowledge graph completion. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
James Bergstra and Yoshua Bengio. Random search for Machine Learning Research, 2012.
optimization. Journal of
Antoine Bordes, Nicolas Usumier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Proceedings of Advances in Neural Information Processing Systems (NIPS), 2013.
Tim Dettmers Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2D) knowledge graph embeddings. In Proceedings of the Thirty-Secone AAAI Conference on Arti- ficial Intelligence (AAAI), 2018.
Luis Antonio Galarraga, Christina Teffioudi, Katja Hose, and Fabian M. Suchanck. AMIE: associ- ation rule mining under incomplete evidence in ontological knowledge bases. In Proceedings of The Web Conference (WWW), 2013.
Shu Guo, Quan Wang. Lihong Wang, Bin Wang, and Li Guo. Knowledge graph embedding with iterative guidance from soft rules. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI), 2018.
Armand Joulin, Laurens van der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual fea- tures from large weakly supervised data. In Proceedings the European Conference on Computer Vision (ECCV), 2016.
Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. Knowledge base completion: Baselines strike back. In Proceedings of the 2nd Workshop on Representation Learning for NLP (Rep4NI.P@ACI) ACI.), 2017.
Seyed Mehran Kazemi and David Poole. Simple embedding for link prediction in knowledge graphs In Proceedings of Advances in Neural Information Processing Systems (NIPS), 2018.
Bhushan Kotnis and Vivi Nastase. Analysis of the impact of negative sampling on link prediction in knowledge graphs. In Proceedings of the 1st Workshop on Knowledge Base Construction, Reasoning and Mining (KBCOM@ WSDM), 2018.
Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. Canonical tensor decomposition for knowledge base completion. In Proceedings of the 35th International Conference Machine Learning (ICML.,, 2018. 2018.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are GANs created equal? A large-scale study. Proceedings of Advances in Neural Information Processing Systems (NIPS), 2018.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. Anytime bottom-up rule learning for knowledge graph completion. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence (IJCAI), 2019.
Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2017.
10

---

Published as a conference paper at ICLR 2020
Sameh Mohamed, Vt Novek, Pierre- Yves Vandenbussche, and Emir Muoz. Loss functions in knowl- edge graph embedding models. In Proceedings of the Workshop on Deep Learning for Knowledge Graphs (DLAKG2019), 2019.
Dai Quoc Nguyen, Tu Dinh Nguyen, Dat Quoc Nguyen, and Dinh Phung. novel embedding model for knowledge base completion based on convolutional neural network. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2018.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way model for collective learn- ing on multi-relational data. In Proceedings of the 28th International Conference on Machine Learning (ICML), 2011.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 2015.
Nils Reimers and Iryna Gurevych. Reporting score distributions makes a difference: Performance study of LetM-networks for sequence tagging. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing (EMNLP), 2017.
Farnood Salchi, Robert Bamler, and Stephan Mandt. Probabilistic knowledge graph embeddings. In Proceedings of the Ist Symposium on Advances in Approximate Bayesian Inference (AABI), 2018.
Chao Shang, Yun Tang, Jing Huang, Jinbo Bi, Xiaodong He, and Bowen Zhou. End-to-end structure- aware convolutional networks for knowledge base completion. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 2014.
Zhiqing Sun, Zhi-Hun Hong Deng, Jian- Yun Nie, and Jian Tang. RotatE: Knowledge graph embedding by relational rotation in complex space. In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019a.
Zhiqing Sun, Shikhar Vashishth, Soumya Sanyal, Partha Talukdar, and Yiming Yang. A re- evaluation of knowledge graph completion methods. arXiv preprint arXiv:1911.03903 2019b.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Compositionality (CVSC@ACL) 2015.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning (ICML), 2016.
2016.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 2017.
Yanjie Wang. Daniel Ruffinelli, Rainer Gemulla, Samuel Broscheit, and Christian Meilicke. On evaluating embedding models for knowledge base completion. In Proceedings of the 4th Work- shop on Representation Learning for NLP (Rep4NLP@ACL), 2019.
Bishan Yang, Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
11

---

Published as a conference paper at ICLR 2020
APPENDIX
Evaluation metrics. We formally define the evaluation metries used our study. Given a triple (i, ()), denote by rank(jji, k) the filtered rank of object j, i.e., the rank of model score s(i, j) among the collection of all pseudo-negative object scores
{i) J E & and (i, j') does not occur in training, validation, or test data }.
If there are ties, we take the mean rank of all triples with score s(i, k, Define rank(i]k::)) likewise. Denote by Keee the set of test triples. Then
Σ
MRR
2/ACway
rrak((ik,,))
rank(jjū,
Hits@ K-
[(rank(i]kk.)) s K) (rank(jjli,) k) K)), (E,K)))A
Σ
2 K
where indicator 1(E) is 1 if condition E is true, else 0.
12

---

Published as a conference paper at ICLR 2020
Dataset
Entities
Relations
Train
Validation
Test
FB- -B7
14.005
237
272 115
17555
20 466
WNRR
40.559
11
86
3 034
3134
Table 4: Dataset statistics
Hyperparameter
Quasi-random search
Bayesian optimization
Embedding size
{128, 256, 512]
Value from Tab. 6
Training type
[NegSamp, 1vsAll, KvsAll} (True, False]
Value from Tab. 6
Reciprocal
Value from Tab. 6
No. subject samples (NegSamp)
1. 1000), log scale
11 1000). log scale
No. object samples (NegSamp)
1, 1000, log scale
1, 1000.. log scale
Label smoothing (KvsAll)
0.0. 0.3]
[0.0, 0.3]
Loss
(BCE, MR, CE}
Value from Tab. 6
Margin (MR)
0, 10]
[0, Value from Tab. 6
Ly-norm (TransE)
{1,2]
Optimizer
(Adam, Adagrad}
Value from Tab.
Batch size
{128, 256, 512, 1024)
Value from Tab.
Learning rate
10-1. 1]. log scale 0.10
[10-1 1]. log scale
LR scheduler patience
[0]
Lp regularization
[L,, L3, None]
Value from Tab. 6
Entity emb. weight
[10-20,00 10-
[10-20, 10-51
Relation emb. weight
[10-20.10 10-22010000
20 [10-20,10-1 10-6)
Frequency weighting
(True, False]
Value from Tab.
Embedding normalization (TransE)
Entity
(True, False) (Tre, False]
Value from Tab.
Relation
Value from Tab.
Dropout
Entity embedding
0.00, 0.5
[0.0, 0.5
Relation embedding
[0.0, 0.5
[0.0, 0.5
Feature map (ConvB)
0.00, 0.5
0.5
Projection (ConvB)
0.0, 0.5
(0.0, 0.5
Embedding initialization
[Normal, Unif, XvNorm, XvUnif)
Value from Tab. 6
Std. deviation (Normal)
1.0
10-5. 1.0
Interval (Unif)
1.00
-1.0,
Gain (XvNorm)
1.0
1.0
Gain (XvUnif)
1.0
1.0
Table 5: Hyperparameter search space used in our study. Settings that apply only to certain config- urations are indicated in parenthesis.
13

---

Published as a conference paper at ICLR 2020
Table 6: Hlpeeppraameeess best performing models found with quasi-randon hyperparameter optimization on FB 5K-237. Settings that apply only to certain configurations are indicated in parenthesis. For each hyperparameter, we also give the reduction in filtered MRR for the best configuration that does not use this value of the nyperparameter (in parenthesis).
34.3
256 (-0.4)
1 (-0.4)
CE (-0.4)
(-0.5) Adagrad (-1.5)
1024 (-0.4)
(-0.4)
L3 (-0.4)
Yes (-1.5)
Unif. (-0.5) XvNorm (-0.4)
ConvE
Yes
0.00373
1.55 11
-11
0.00
0.10
0.19
0.49
7.91
35.3
256 (-0.3)
NegSamp (-0.3)
Yes (-0.3)
CE (-3.8)
(-0.3)
9 (-0.3)
L3 (-0.3)
Yes (-0.3)
ComplEx
557
367
Adagrad
1024
0.14118
1.55 10-10
3.93 12--15
0.46
0.36
[-0.85, 0.85]
33.0
(-0.2)
NegSamp (-0.2)
Yes (-0.3)
CE (-3.1)
Adagrad (-0.2)
(-0.2)
(-0.2)
L3 (-0.2)
Yes (-0.3)
Unil. (-0.2)
DistMult
256
557
367
1024
0.14118
1.55 10
3.93 12--1
0.46
0.36
[-0.85, 0.85]
31.5
(-3.7)
Yes (-9.5)
CE (-7.1)
(-3.7)
(-7.1)
(-3.7)
(-3.7)
No (-11.0)
XvNorm (-3.7)
Trans
512
IvsAll (-0.8) NegSamp
56
L2
Adagrad
128
0.04122
L2
1.32 -07
3.72 -1
No
No
0.00
0.00
36./ 128 (-0.5)
No (-0.5)
CE (-0.9)
Adam (-0.5)
512 (-0.5)
1 (-0.5)
None (-0.5)
Yes (-0.5)
Normal (-0.8)
RESCAL
0.00063
0.37
0.28
0,80620
Embedding normalization (TransE)
No. subject samples (NegSamp)
No. object samples (NegSamp)
Label Smoothing (KvsAll)
Std. deviation (Normal)
Relation cmb. weight
Frequency weighting
Feature map (ConvE)
Embedding initialization
Lp norm (TransE)
Scheduler patience
Entity emb. weight
Entity embedding
Relation embedding
Projection (ConvE)
Margin (MR)
L Learning Rate
Lp regularization
Interval (Unif)
Training type
Reciprocal
Batch size
Mean MRR
Embedding
Optimizer
Relation
Dropout
Entity
Loss
14

---

Published as a conference paper at ICLR 2020
Table 7: Hyperparameters best performing models found with quasi-rante raaddmm hyperparameter optimization on WNRR. Settings that apply only certain config- urations are indicated in parenthesis. For each hyperparameten wc also give the reduction in filtered MRR for the best configuration that does not use this value of the hyperparameter (in parenthesis).
44.3
(-1.2)
KysAll (-1.2)
CE (-1.4)
Adam (-1.4)
1024 (-1.3)
(-1.2)
L1 (-1.2)
No. (-1.4)
Unif. (-1.5) XvNorm (-1.4)
ConvE
512
Yes
-0.29
0.00160
3.70 12
6.56 6.56-10 10
0.00
0.23
0.09
0.42
47.6
128 (-1.0)
IvsAll (-1.0)
No (-1.0)
CE (-3.5)
Adagrad (-1.5)
512 (-1.0)
7 (-1.0)
L2 (-1.0)
No (-1.0)
ComplEx
0.50338
1.44-18
0.05
0.44
0.31] 0.311,
1.48
[-0.31,
45.4
512 (-1.1)
KvsAll (-1.1)
Yes (-1.1)
CE (-2.4)
Adagrad (-1.5)
1024 (-1.1)
6 (-1.1)
L3 (-1.1)
Yes (-1.1)
Unif. (-1.3)
DistMult
0.21
0.25575
1.34-10
6.38 16
0.12
0.36
[-0.81, 0.81]
22.6
512 (-5.1)
Yes (-5.9)
CE (-5.1)
(-5.8)
128 (-5.1)
(-5.1)
L. (-5.1)
No (-7.4)
XvNorm (-5.1)
Trans
NegSamp
2
56
L2
Adagrad
0.04122
1.32 07
3.72 18
No
No
0.00
0.00
46.8
128 (-1.0)
KvsAll (-1.0)
Yes (-I.0)
CE (-2.0)
Adam (-1.2)
128 (-1.0)
(-1.0)
L3 (-1.2)
No (-1.0)
Unil. (-1.0)
RESCAL
0.30
0.00160
3.82 20
7.47 05
0.00
0.00
|-0.31.0.31]
Embedding normalization (TransE)
No. subject samples (NeySamp)
No. object samples (NegSamp)
Label Smoothing (Kls
Std. deviation (Normal)
Relation cmb. weight
Frequency weighting
Feature map (ConvE)
Embedding initialization
Relation embedding
Ly norm (TransE)
Scheduler patience
Entity emb. weight
Entity embedding
Projection (ConvE)
Embedding size
Margin (MR)
Learning Rate
Lp regularization
Interval (Unif)
Training type
Reciprocal
Batch size
Mean MRR
Optimizer
Relation
Dropout
Entity
Loss
WNRA
15

---

Published as a conference paper at ICLR 2020
RESCAL
TransE
DistMult
ComplEx
ConvE
Learning rate
0.00074
0.00030 5
0.15954
0.18255
0.00428
Scheduler patience
FBISK-237
Entity reg. weight
1.41 -09
6.70 09
4.30
Relation reg. weight
4.10-15
4.13-11
1.12-14
Entity emb. dropout
0.43
0.00
0.42
0.50
0.00
Relation emb. dropout
0.16
0.00
0.41
0.23
0.12
Projection dropout (ConvE)
0.50
Feature map dropout (ConvE)
0.49
Learning rate
0.00085
0.25327
0.33127
0.52558
0.00162
Scheduler patience
8
5
5
Entity reg. weight
1.37
1.06 07
1.25 - 12
4.52 06
1.08 08
WNRR
Relation reg. weight
2.57-15
4.50-13
1.53-14 14
4.19-10
9.52-11
Entity emb. dropout
0.00
0.25
0.37
0.36
0.00
Relation emb. dropout Projection dropout (ConvE) Feature map dropout (ConvE)
0.24
0.00
0.50
0.31
0.23
0.15
0.33
Table 8: Hyperparameters of best performing models found after quasi-random hyperparameter optimization and Bayesian optimization. All other hyperparameters the same as in Tables (FB15K-237) and 7 (WNRR).
RESCAL
TransE
DistMult
ComplEx
ConvE
FBI5K-237
MRR
36.1=0.3
31.5++0.1
35.0+0.0
35.3+0.11
34.3=0.1
Hits
54.3=0.5
49.8-102
53.540.1
53,940.2
52.4.0.2
WNRR
MRR
46.8=0.2
22.610.0
45.410.1
47.610.1
44.3-0.1
Hits @ 10
51.8=0.2
51.5+0.1
52.4+0.3
54.140.4
50.4
Table 9: Mean and standard deviation of the validation data performance of each model's best hyperparameter configuration when trained from scratch five times.
16

---

Published as a conference paper at ICLR 2020
RESCAL
TransE
DistMult
ComplEx
ConvE
40
Validation MRR
FB15K-237
20
10
Validation MRR
30
WNRR
20
10
128
256
512
128
256
512
128
256
512
128
256
512
128
256
512
Figure 4: Distribution of filtered MRR (on validation data, quasi-random search only) for different embedding sizes (top FB FB15K-237, bottom row: WNRR)
RESCAL
TransE
DistMult
ComplEx
ConvE
40
Validation MRR
F815K-237
20
Validation MRR
30
WNRR
10
128
256
512
1024
128
256
512
128
256
512
1024
128
256
1024
128
256
512
1024
Figure 5: Distribution of filtered MRR (on validation data, quasi-random search only) for different batch sizes (top row: FB FB15K-1 15K72377 bottom row: WNRR)
17

---

Published as a conference paper at ICLR 2020
RESCAL
TransE
DistMult
ComplEx
ConvE
40
Validation MRR
F815K-237
20
Validation MRR
30
WNRR
10
Adagrad
Adam
Adagrad
Adam
Adagrad
Adam
Adagrad
Adam
Adagrad
Adam
Figure 6: Distribution of filtered MRR (on validation data, quasi-random search only) for different optimizers (top row: FB15K-237, bottom row: WNRR)
RESCAL
TransE
DistMult
ComplEx
ConvE
40
Validation MRR
FB15K-237
20
Validation MRR
30
WWRRR
10
Normal Unif. XvNorm XvUnif
Normal
Unif. XvNorm XvUnif
Normal
Unif. XvNorm XvUnif
Normal
Unif. XuNorm XvUnif
Normal Unif. XvNorm XvUnif
Figure 7: Distribution of filtered MRR (on validation data, quasi-random search only) for different initializers (top row: FB15K-237, bottom row: WNRR)
18

---

Published as a conference paper at ICLR 2020
RESCAL
TransE
DistMult
ComplEx
Validation MRR
FB15K- 237
30
20
10
0
40
Validation MRR
30
WNRR
20
10
No Reciprocal
Reciprocal
No Reciprocal
Reciprocal
No Reciprocal
Reciprocal
No Reciprocal
Reciprocal
Figure 8: Distribution of filtered MRR (on validation data, quasi-random search only) with and
without reciprocal relations (top row: FB15K-237, bottom row: WNRR)
RESCAL
TransE
DistMult
ComplEx
ConvE
40
Validation MRR
FB15K-237
20
10
Validation MRR
30
WNRR
20
10
None
L1
L2
L.
None
L1
L2
L3
None
L1
L2
None
L1
L2
L3
None
L1
12
L3
Figure 9: Distribution of filtered MRR (on validation data, quasi-random search only) for different penalty terms in the loss function (top row: FB15K-237, bottom row: WNRR)
19

---

Published as a conference paper at ICLR 2020
RESCAL
TransE
DistMult
ComplEx
ConvE
40
Validation MRR
FB15K-237
20
Validation MRR
30
WNRR
10
No Dropout
Dropout
No Dropout
Dropout
No Dropout
Dropout
No Dropout
Dropout
No Dropout
Dropout
Figure 10: Distribution of filtered MRR (on validation data, quasi-random search only) with and without dropout (top row: FB15K-237, bottom row: WNRR)
20


Reviews:
[{"review_id": "Syljyis0Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n========\nThe paper conducts a thorough analysis of existing models for constructing knowledge graph embeddings. It focuses on attempting to remove confounding aspects of model features and training regime, in order to better assess the merits of KGE models. The paper describes the reimplementation of five different KGE models, re-trained with a common training framework which conducts hyperparameter exploration. The results show surprising insights, e.g., demonstrating that a system from 2011, despite being the earliest of the KGE models analyzed, demonstrates competitive results over a more recent (2017) published model.\n\nOverall Comments\n===============\nThe paper, and the described software release specifically, represent a solid contribution to the area of knowledge graph embeddings. I agree with the basic premise of this paper’s analysis: in order to accelerate research in a maturing field (like knowledge graphs), it is important to be able to properly compare with older systems, removing artifacts that are due to general improvements in training and optimization techniques, from modeling specific changes. The report of the strong results from the RESCAL system, along with others, drive the point through. Furthermore, the paper is well-written and easy to follow, and should become a good reference for future works on KGEs.\n\nDetailed comments\n===============\nBelow are some detailed comments about specific parts of the paper, in order of importance:\n\n1. The paper mentions disregarding “monolithic” models in the current analysis, primarily due to the expensive training of these models. It may, however, be the case that the future state-of-the-art models will be larger and slower to train (and, perhaps, of the monolithic type). Are there any limitations to the proposed experimental framework that would prevent running monolithic/large models?\n\n2. Regarding the item above, if one were to look at the training curves for the exploration of the current 5 KGE models, is it possible that verify winning hyperparameter configurations earlier than the full training is complete. In my experience, it is often the case that with fewer than 1/10th steps of full training (well before convergence), it is possible to compare model configurations (relatively). For example, “Population-base training” (https://arxiv.org/abs/1711.09846, https://arxiv.org/abs/1902.01894) is one framework where fewer training steps are used to quickly learn good hyperparameter configurations. I’m wondering whether the KGE hyperparameter exploration training curves display similar early trends. Could a shortened training procedure produce sufficient information for learning good parameters, and potentially deal with larger/slower models?  In addition: would adopting population-based training be applicable to the proposed framework?\n\n3. In Section 3.2, “Limitations”, there is a surprising comment that performance can be improved with further hyperparameter tuning. It is not clear how the authors found the configurations that produced the improved results. It would be helpful to clarify why the hyperparameter exploration proposed in the paper did not discover these improved configurations. Were the improved configurations outside of the range of considered values? Or would the exploration require more points to find the improved configuration?\n\n4. In Section 3.3 “Best configurations (quasi-random search)”, specifically Table 3, the paper presents an ablation of independent hyperparameters, over the best configuration for each of the 5 models. This is a very interesting section. One further suggestion, however, is whether the paper could include the performance of each of the models on the _average_ best configuration. Although the paper describes losses for switching individual parameters to their second best values, it is unlikely that the losses are cumulative. So, for example, if we can take the average/majority best value for each parameter (embedding size = 512, batch size = 1024, training type = 1vsall, loss = CE, etc.), and collect results for that configuration. I think it would be interesting to know the difference between a model trained on a “collectively known good” set of parameters vs. a model and task specific tuned set of parameters.\n\n5. In Section 2, “Evaluation”, HITS@k is not formally defined. Unfortunately, I have encountered slight variants of this metrics (e.g: (1) given a SINGLE correct label, HITS@k is the average rate of the label being present in the top k scored results, or (2) given ALL possible correct labels, HITS@k is the percentage of correct labels present within the top k scored results, etc.). It would be nice to precisely describe HITS@k in this work.\n\n6. Caption for Table 2 does not contain a description for the “Recent” super-column.\n\n7. In Section 3.3 “Best configuration (quasi-random search)” Space missing at “... Tables 6 and 7(in …”, between 7 and (.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SkemmQtecS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors did an extensive experimental study over neural link prediction architectures that was never done before, in such a systematic way, by other works in this space. Their findings suggest that some hyperparameters, such as the loss being used, can provide substantial improvements to some models, and can be the reason of the significant improvements in neural link prediction accuracy the community observed in recent months.\n\nThis is a really interesting paper, and can really shine some light on what was going on in neural link prediction over recent years. It also provides a great overview of the field -- in terms of architectures, loss functions, regularizers, sampling strategies, data augmentation strategies etc. -- that is really needed right now in the field.\n\nOne concern I have is that the hyperparameter tuning strategy is not really described -- authors just say something along the lines of \"we use av.dev\", but for those unfamiliar with this specific hyperparameter optimiser this does not provide much information (e.g. what is a Sobol sequence? I had to look it up).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1gAye5RtB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an experimental study about some KGE methods. It argues that papers often propose changes in several different dimensions, such as model, loss, training, regularizer, etc., at once without providing a sufficient investigation about the individual components' contributions. The experimental study considers two datasets (FB15k-237 and WNRR) and five different models (RESCAL, TransE, DistMult, ComplEx, ConvE). The models were selected using a quasi-random hyperparameter search, followed by a short Bayesian optimization phase to fine-tune the parameters. The performance of the best models found during this hyperparameter search are compared to first published results for the same model, as well as to a small selection of recent papers. To analyse the influence of single hyperparameters, the best found configuration is compared to the best configuration which does not use this specific value for the given hyperparameter.\n\nOverall, the paper adresses an important problem, as papers about new KGE methods often lack a clear separation of the individual changes' contribution. The experimental results show that older, simpler can compete with recently proposed models when trained properly. The intra-model comparison lacks statistical rigorousity, yet hints a few directions to further explore.\n\nThe experiments are based on a quasi-random hyperparameter search. While it is necessary for efficient exploration of larger search spaces [1], and should be the standard methodology for hyperparameter search of a new method, the interpretability of the comparison of two runs suffers. For the comparison between the trained models and previously published results, the sample size might be sufficient to draw the conclusions. However, the intra-model comparison, e.g. in Figure 2, are now comparing subsets of the runs which only comprise approx. (200/6) runs. Furthermore, the influence of random initialization is not accounted for. Another place where this can be witnessed is Table 3. Here, for some ablations, e.g. TransE + Reciprocal, no reduction is given. If I understood it correctly, this is due to not having a configuration which uses TransE and reciprocal relations. Also for the other ablations, it is unclear how statistically significant the reduction is.\n\n\nFurther Comments:\n1. Please add the best published results for a specific model-dataset combination to table 2.\n2. Do the plots in Figure 1 include the runs which were stopped after 50 epochs due to insufficient MRR?\n3. Could you elaborate on the combination of KvsAll and CE?\n4. The combination of subject and object triple scores has for instance been used in SimplE [2].\n\n\n[1] Bergstra, James, and Yoshua Bengio. \"Random search for hyper-parameter optimization.\" Journal of Machine Learning Research 13.Feb (2012): 281-305.\n[2] Kazemi, Seyed Mehran, and David Poole. \"Simple embedding for link prediction in knowledge graphs.\" Advances in Neural Information Processing Systems. 2018.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors analyze knowledge graph embedding models for multi-relational link predictions. Three reviewers like the work and recommend acceptance. The paper further received several positive comments from the public. This is solid work and should be accepted.
Decision: Accept (Poster)