Beyond Link Prediction: On Pre-Training Knowledge Graph Embeddings
Authors: Daniel Ruffinelli
Rainer Gemulla
Venue: Submitted to ICLR 2023 (2023)

Abstract: Knowledge graph embeddings (KGE) models provide low-dimensional representations of the entities and relations in a knowledge graph (KG). Most prior work focused on training and evaluating KGE models for the task of link prediction; the question of whether or not KGE models provide useful representations more generally remains largely open. In this work, we explore the suitability of KGE models (i) for more general graph-structure prediction tasks and (ii) for downstream tasks such as entity classification. For (i), we found that commonly trained KGE models often perform poorly at structural tasks other than link prediction. Based on this observation, we propose a more general multi-task training approach, which includes additional self-supervised tasks such as neighborhood prediction or domain prediction. In our experiments, these multi-task KGE models showed significantly better overall performance for structural prediction tasks. For (ii), we investigate whether KGE models provide useful features for a variety of downstream tasks. Here we view KGE models as a form of self-supervised pre-training and study the impact of both model training and model selection on downstream task performance. We found that multi-task pre-training can (but does not always) significantly improve performance and that KGE models can (but do not always) compete with or even outperform task-specific GNNs trained in a supervised fashion. Our work suggests that more research is needed on how to pre-train KGE models and on their suitability for downstream applications.

Under review as a conference paper at ICLR 2023
BEYOND LINK PREDICTION: ON PRE-TRAININ KNOWLEDGE GRAPH EMBEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge graph embeddings (KGE) models provide low-dimensional represen- tations of entities and relations in knowledge graph (KG). Most prior work fo- cuses on training and evaluating KGE models for the task of link prediction; the question of whether not KGE models provide useful representations more gen- erally remains largely open. In this work, we explore the suitability of KGE mod- els (i) for more general graph-structun prediction tasks and (ii) for downstream tasks such as entity classification. For (i), we found that commonly trained KGE models often perform poorly structural tasks other than link prediction. Based on this observation, we propose more general multi-task training approach, which includes additional self-supervised tasks such as neighborhood prediction domain prediction. In our experiments, these multi-task KGE models showed significantly better overall performance for structural prediction tasks. For (ii), we investigate whether KGE models provide useful features for variety of down- stream tasks. Here we view KGE models as form of self-supervised pre-training and study the impact of both model training and model selection downstream task performance. We found that multi-task pre-training can (but does not always) significantly improve performance and that KGE models can (but do not always) compete with or even outperform task-specific GNNs trained in supervised fash- ion. Our work suggests that more research is needed on the relation between pre- training KGE models and their suitability for downstream applications.
INTRODUCTION
Knowledge graph embeddings (KGE) provide low-dimension representations of entities and rela- tions of knowledge graph (KG). Although large number of KGE models have been proposed in the literature- see for example the surveys of Nickel et al. 2013) Wang et al. (2017) and et al. (2021) most prior work focuses the task of link prediction, L.e., answering questions such as (Austin, capitalOf. ?) by reasoning over an incomplete KB. In addition to link prediction, it is often argued that KGEs can provide representations that capture semantic properties of the entities and, indeed, pre-trained KGE models have been used to inject structured knowledge into language models (He al.] 2020 Zhang et al., 2019). visual models (Baier al., 2017). recommender sys- tems (il-Kishky et al. [022]] Wang et 2018). question answering systems (lyyas et al. 2022 and
other types of downstream models (Wang et al., 2017).
The question of whether pre-trained KGE models provide generally useful representations remains largely open. Likewise, not well-understood how choices taken in model training and model selection affect these representations. In this work, we shed light onto these questions from multiple directions.
First, we study the suitability of out-of-the-box KGE models for basic graph-structure prediction tasks beyond link prediction. In particular, we consider the tasks of predicting the relation a triple as suggested by Chang et al. (2020) (e.g., the relationship between Austin and Texas), the domain and range of a relation (c.g., whether Austin is a capital), as well as entity and relation neighborhood of each entity (c.g., which other entities are related to Austin). Perhaps surprisingly. found that commonly trained KGE models often performed poorly on such tasks, challenging the intuition that KGE models capture graph structure well.

---

Under review as a conference paper at ICLR 2023
Second, we investigate whether KGE models suitable pre-trained representations for node-level downstream tasks such as entity classification (c.g., the profession of a person) or regression (e.g,, the average rating of movie). To do so, we conducted an empirical study using 27 downstream tasks on two different KGs. We found that out-of-the-box KGE models often perform decent on these tasks and, in fact, the best KGE models can (but do not always) exceed the performance of recent graph neural networks such as KE-GCN (Yu et al., (021). However, the KGE models with best downstream task performance were often not the best-performing models for link prediction. For example, we found that the basic TransE model (Bordes et al., 2013) may be superior to KGE models more suited link prediction such as ComplEx (Trouillon et 20. or RotatE (Sun al.) 2019). This suggests that link prediction performance is not necessarily indicative of downstream task performance.
Both of these findings suggest that the focus on link prediction tasks is too narrow for pre-training KGE models, i.e., to provide generally useful features.. We thus explore whether the performance of KGE models for both graph-structure prediction and downstream tasks can be improved by better pre-training and model selection. Inspired by multi-task approaches other areas-such as natu- ral language processing (Aribandi et al., 2022, Sanh et al., 2022) computer vision (Doersch & Zisserman 2017] we included the graph-structure prediction tasks discussed above as additional training objectives and as evaluation measures during model selection. In particular, we propose multi-task training (MTT) and a multi-task ranking (MTR) approach that both can be used along with an arbitrary KGE model class and without a substantial increase computational cost. In our experimental study, the resulting multi-task KGE models had significantly better overall perfor- mance for graph-structure prediction tasks and often (but not always) also led better downstream task performance. We also found that downstream task performance could be further improved by using a smaller set of pre-training tasks. The results suggest that the optimal choice of tasks depends on the dataset, KGE model class, and downstream task and may be difficult to determine in practice.
In summary, the contributions of this paper are as follows: (i) We show empirically that commonly trained KGE models fail at basic graph-structure prediction tasks beyond link prediction. (ii) We propose novel multi-task training and ranking approaches that address this shortcoming. (iii) We explore the impact of standard and multi-task training well as different approaches for model selection on downstream task performance. (iv) We contextualize KGE model performance with results obtained from recent graph neural networks, which in contrast to KGE models are trained directly on each downstream task. Although our work makes a step toward improved pre-training of KGE models, it also suggests that more research is needed on the relation between pre-training KGE models and their general suitability for downstream applications.
PRELIMINARIES AND RELATED WORK
We briefly describe KGE models, training and evaluation methods for link prediction, well as prior work on other tasks. A more comprehensive discussion can be found in surveys such as (Nickel al. 2015 Wang et al. 2017 et al. 202T).
Link prediction. A knowledge graph G E X E is collection of (subject, predicate, object)- triples over set & of entities and set of relations. Triples represent known facts such as (Austin, capitalOf, Texas). In the KGE literature, the link prediction task is the task of inferring the subject or object to questions of form (?, capitalOf, Texas) and (Austin, capitalOf. 2), respectively.
KGE models. KGE models (Sun et al. 2019 Trouillon et al. 2016; Bordes et al.. 2013) represent each entity and each relation of a KG with a low- dineensnsto embedding, commonly a real or complex vector. KGE models have an associated scoring function E x R x - IR that associates each triple with a real-valued score. Intuitively, high scores indicate plausible triples, low scores implausible triples. Commonly, the scoring function depends the input triple only through the embeddings of its arguments. For example, TransE (Bordes et al. 2013) is a translation-bart model with s(i, j) -Ile, + T - eell. where e, ERR and r I denote entity and relation embeddings of dimensionality d > O, respectively. Scoring functions can be more involved, e.g., based on convolutional neural networks (Dettmers et al. 2018) or transformers (Chen et al., 2021aa.
Standard training. KGE models are commonly trained on the link prediction task. We only give a high-level description here. For each triple (s, p, o) in the training data Guain KGE models are trained
2

---

Under review as a conference paper at ICLR 2023
such that the score s(s, p.o) is high (a positive) but, for certain choices of E E such that (s. & Guain the score of s(s. p, o') is low (a negative); similarly for subjects s' E E with (s', P, o) & Guain- The actual cost function varies across training types (e.g,, sampled negatives or all negatives), loss function (e.g, cross entropy), ) and more generally the choice of hyperparameters; see (Ali et al. 202T Ruffinelli et al.] 2020) for more detailed diseussion and experimental comparison.
Standard evaluation. The most commonly used evaluation protocol for KGE models is entity ranking (ER), which is also based on link prediction. Given a test triple (s, p, & Grain. the model is used to answer the link prediction queries (s, ?)? and (?, P, o). In particular, the scores of all possible answers that do not already occur in the training data are computed. The model evaluated based on the rank of the test answers s and o, respectively. Common metrics are the mean reciprocal rank (MRR) and Hits@ K. The reliability of entity ranking in assessing model performance was studied and questioned, e.g., in (Safavi & Koutra 2020 Tiwari et al. 2002 Zhou et al. 2022 Wang et al. 2019). In contrast, our focus is mostly on other evaluation tasks.
Other training approaches. RESCAL (Nickel et al. 2011), one of the earliest KGE models, trained on the reconstruction task. Such tasks aim to to construct the entire training data using cost functions such as E.p.o [ p, o) E Guainl s(8, P, o) 0) where /[-] is 0/1 indicator. A similar approach was explored by Li et al. (2021). We do not consider such methods further because training costs are excessive (at least unless squared error is used) and the empirical performance reported by Li et al. (2021) is generally far behind KGE models trained with link prediction. Chen et al. 2202 lb) proposed to augment the link prediction task with relation prediction during training (but not eval- uation). We expand upon this work by considering additional pre- pre-training tasks and by focusing on graph-structure prediction and downstream task performance instead.
Other evaluation approaches. In early (and rarely in recent) work, KGE models were evaluated using triple classification (Socher et al.] 2013] Wang et al. [0144] [in al. 2015 Wang et al. 2022). We do not consider this task in this work because performance estimates are typically overly opti- mistic and misleading unless hard negatives are used (Safavi & Koutra 2020); such hard negatives are generally not available. Chang et al. (2020) evaluated KGE models on the relation prediction task, which we also consider as one of the evaluation tasks in this work. There is also work on explaining or interpreting KGE models (Meilicke al., 2018 Allen al., 2021 Rim et al., 2021), whereas our focus is on studying whether such models provide useful representations in the first place. As mentioned in the introduction, pre-trained KGE models have been used as a components in language models (He et al. 2020) Zhang et al.] 2019). visual models (Baier et al. 2017), recom- mender systems (EI-Kishky et al. 2022 Wang et al. [0188) or question answering systems ([lyas) et al.] 2022). Likewise, (Pezeshkpour et al. 2018 Jain et al. 2021 evaluated pre-trained KGE mod- els for entity classification or regression tasks, as we do. We expand on this line of work by using a larger set of tasks (graph-struche) structure prediction and more downstream tasks), by proposing improved pre-training methods, and by studying the impact of pre-training on downstream task performance.
GRAPH-STRUCTUR PREDICTION
In addition link prediction, explore the suitability of KGE models for other basic graph- structure prediction tasks. An example and summary is given in T We describe the form of the queries for each task as a triple such as (s, ?, *), where or o denote input entities, p denotes an input relation, ? denotes the prediction target, and * acts as a wildcard. Using this notation, we consider the following tasks and queries:
Link prediction (LP): Given relation and a subject, predict the object (denoted (s.p,?)). Like- wise, given relation and an object, predict the subject (denoted (?,p, o)).
Relation prediction (REL, Chang et al. (2020); Chen et al. (2021b); Given two entities s and p, predict the relation between them (denoted (s, o)).
Domain prediction (DOM): Given a relation, predict its domain (denoted (?,p, +)) or its range (denoted (*,p,?) ?)).
Entity neighborhood prediction (NBE): Given a subject entity, predict related objects (denoted
(s, *, ?)). Likewise, given object, predict related subjects (denoted (2, o)).
Relation neighborhood prediction (NBR): Given a entity, predict the relations where it occurs subject (denoted (s, ?, +)) and where it occurs as object (denoted (x, ?,)).
3

---

Under review as a conference paper at ICLR 2023
Knowledge graph
Task
Example query
Some answers
(Dallas, locatedIn, Texas)
Link (LP)
(Austin, locatedIn, ?)
Texas, USA
(Texas, locatedIn, USA)
(?, locatedIn, Texas)
Austin, Dallas locatedIn, capitalOs
(Austin, capitalOf, Texas)
Relation (REL)
(Austin, ?, Texas) (%, locatedIn, ?)
(Austin, locatedIn, Texas)
Domain (DOM)
Texas, USA, North A.
(Arkansas, borders, Texas)
(?, locatedIn, *)
Dallas, Texas, USA
(USA, locatedIn, North A.) Entity neighb. (NBE)
(Austin, ?)
Texas, USA
(Austin, locatedIn, USA)
(2, *, Texas)
Dallas, Arkansas capitalOf. locatedIn
Relation neighb. (NBR)
(Austin, ?. ()
(%, ?, Texas)
borders, capitalOf
Table 1: Graph-structure prediction tasks used for self-superviser pre- pre-training and evaluation along with example queries. Here denotes the prediction target and * acts as a wildcard.
Note that we the wildcard to denote existential quantification. For example, given a ground-truth KG & and domain prediction query (?, *), an entity s E & is a correct answer if there exists an entity E & such that (s. p. o) E G.
We chose this particular set of tasks because they are simple, they capture basic information about the graph structure beyond link prediction, and they only have one prediction target (an entity or a relation). The latter property allows efficient pre- pre-training and evaluation, as discussed below. For this reason, we exclude tasks such as entity-pair prediction (Wang et al.) 2019) (denoted p? ?) in our notation) or reconstruction (Nickel et al. 2011) 1) (denoted ? In our experimental study, we also found that the exclusion of some of the above pre- tr-trrainig tasks (c.g., LP) can further im- prove downstream task performance. The optimal choice of tasks depends on dataset, KGE model, and downstream task, however. We leave the exploration of task selection as well as on exploring additional pre-training tasks to future work.
Multi-task ranking (MTR). To evaluate the performance of KGE models on the graph-structure prediction tasks, generalize the entity ranking (ER) protocol for link prediction. Intuitively, for each of the nine tasks (REL as well as LP/DOM/NBE/NBR for both subject targets and for object targets), construct query from each test triple [ or relation) that do not already occur in the training data, and then use metrics such as MRR or Hits @ K. The final MTR metric is given by the miero-average over all nine tasks.
obtain ranking of the prediction targets (entity
We now describe how to obtain task-specific rankings. First, for a REL query of form (s,?,o), we proceed as in (Chang et al., 2020) and rank all r' E R such that (s.r'. o) & Guain descending order of their scores s(s, r' o). For the other tasks, which involve wildcards, it is not immediately clear how to perform prediction using KGE model. We first discuss scoring and ranking, then filtering of training data. Consider for example the NBR query (s,?, *). where our goal is to rank relations. The perhaps simplest approach to obtain a relation ranking is to first rank all triples of form (s,r', o'), where r' E R and o' E E, and then rank relations by their first appearance (c.g., the relation of the highest-scoring triple is ranked at the top). More generally, we make use of an extended score function that accepts wildcards. The approach just described corresponds to using s(s, r', = max r(s,r', ,o)), Although other aggregation functions are feasible, we only consider does not make any additional assumptions on the scoring function. To filter training data during model evaluation,
o.), i.e, the score of relation r' is the score of its most plausible triple.
aggrregattnn because
remove all relations r' such that (s,r', ro',''))) E Guain for some o' E & i.e., we remove all prediction targets that are already implied by the training data. We proceed similarly for all other tasks involving wildcards. Note that the number of score computations needed to predict entity targets for queries without wildcards is O(EE). whereas the one for queries with wildcards is ((E We discuss below how the latter cost can be reduced to O(EE)
Multi-fask training (MTT). We now generalize standard KGE model training to all of the graph- structure prediction tasks. Our goal is to be able to improve KGE model performance at these tasks, while at the same time keeping training and prediction cost low. We do this by constructing task- specific cost function for each individual task first; the final cost function is then given as a weighted
'The nine queries for test triple (s. P.P.O o) are precisely the ones given in the task descriptions.
4

---

Under review as a conference paper at ICLR 2023
linear combination of the task-specific costs (and additional regularization terms), where the weights are hyperparameters.
The task-specific cost functions for link prediction and relation prediction are obtained as in standard training (Sec. 2): For each positive triple (s, p. o) E G. we construct a set of negatives according to the query (i.e., by perturbing the position of the prediction target) and then apply the loss function (e.g., cross entropy). For the other tasks, which involve wildcards, we proceed differently. Instead of performing some form of (costly) score aggregation during training, wildcards into tasks without wildcards. To do so, we make use of three virtual wildcard entities-one for subjects (anys)) one for relations (anyR), and one for objects (anyo) and learn embeddings for these entities. During training, we conceptually replace wildcards by their corresponding wildcard entity and proceed as before. For example, for training triple (s, p, o) and NBR query (s, ?, *), we consider the virtual triple (s, P. anyo) along with query (s, ? anzo). By doing so, we converted the NBR task into REL. task. We also use the so-obtained wildcard embeddings during prediction time in the same fashion; e.g., set s(s, aggregation, the model thus directly learns extended scores.
"convert" tasks with
s(s, rnyo))) Instead of performing score
The advantage of the MTT approach is that (i) the prediction costs remain stable, i.e., the cost of graph-structure prediction or downstream task prediction is unaffected by the number or choice of pre-training tasks, and (ii) the pre-training costs increase only linearly in the number of tasks.
Note that the wildcard embeddings are not used for entity-level downstream tasks. Nevertheless, us- ing wildcard entities during training affects all other entities well. This is because the embedding of each entity occurs in all graph-structure prediction tasks. The entity embeddings of a good KGE model thus needs to be suitable for all these tasks, not just for link prediction.
EXPERIMENTAL STUDY
We conducted large experimental study. Our goals were (i) assess KGE model performance for graph-structure prediction, (ii) to assess performance of pre-trained KGE models on downstream tasks, (iii) to assess the effect of multi-task training both graph-structure prediction and down- stream task performance, and (iv) to contextualize these results by comparing them to results ob- tained by recent graph neural networks (GNNs).
4.1 EXPERIMENTAL SETUP
Datasets, code, and scripts to reproduce all experimental -Link-provided in- finallliiii lonnal-versee version?
results
available
Knowledge graphs. We used three commonly used benchmark datasets for evaluating KGE models: FB15K-237 237 (Toutanova & Chen 2015). WNRR (Dettmers et al., 2018), and YAGO3-10 (Mahdis- oltani et al. 2014). Each dataset associated with training, validation and test split. FB15K- 237 and WNRR are designed to be harder benchmarks for link prediction. YAGO3-10 is not, but it considerably larger. Dataset statistics are summarized in Table 5 in the appendix.
KGE models. We considered four popular, representative KGE models: TransE (Bordes et al, 2013) and DistMult (Yang al.) (2015) (basic translational and factorization models, resp.) well as RotatE (Sun et al. 2019) and ComplEx (Trouillon et al. 2016) (SOTA translational and factoriza- tion models). RotatE and ComplEx are the methods of choice for low-cost embeddings with good prediction performance (Ruffinelli et al. 2020 Sun et al. size and/or training cost (Lacroix et al.) 2018: Chen al.
2019), and an increase in model 20216 can perform as well as SOTA
models of other KGE model types such as the transformer- based HitER model (Chen et al. 2021a
KGE training. We used LibKGE (Broscheit et al., 2020) for STD training (L.P only) as a baseline and added MTT/MTR model training/evaluation. All KGE models were trained for maximum of 200 epochs with early stopping on validation MRR checked every 10 epochs. We used cross-entropy as loss function, as it systematically outperformed other losses in most prior studies. We used /vsAll training with FB 181152237 and WNRR (to achieve good results) and NegSamp with YAGO3-10 (to scale to this larger dataset). Since we interested in pre-trained KGE models, no information from downstream tasks is used for KGE model training and selection; e.g.. the same KGE model used for all downstream tasks in each experiment. In particular, models were selected w.r.t. performance
5

---

Under review as a conference paper at ICLR 2023
(MRR) the validation data. Unless stated otherwise, models trained with STD training the LP task, models trained with MTT training use MTR. Further improvements may be made by using downstream tasks during training (Aribandi et al. 2022); we leave such exploration to future work.
KGE evaluation. We evaluate KGE models with respect to each of the five graph-structure predic- tion tasks of See. (LP, REL, DOM, NBE, NBR) using filtered MRR on test data. We also aggregate these metrics into the multi-task ranking MRR (MTR).
KGE hyperparameters. We closely follow the approach of the experimental study of Ruffinelli et al. (0200) to perform hyperparameter selection. We performed 30 random trials using SOBOL sampling (Bergstra & Bengio, 2012) over a large search space tune several hyperparameters, e.g. regularization, embedding size, batch size, dropout, initialization, and task weights (each in [0.1, 10.0). log scale). To keep our study feasible, we reduced the maximum batch and embedding size for larger datasets and expensive models. The full search space can be found in Table g
Downstream tasks. We collected or created data for 27 downstream tasks on FB15K-237 or YAGO3-10. This includes the datasets of Jain et al. (2021) for entity classification on FB15K-237 and YAGO3-I0, which aims to predict the types of entities at different granularities. For regres- sion, we use the datasets of PPezeshkpour et al. (2018) for YAGO3-10, which consist of temporal prediction tasks (e.g.. the year an event took place), and the dataset of Huang et al. (202T) for node importance prediction. We also created several regression tasks for FB15K-25 from the multi- modal data of Garciaa -uránn et al. (2018) by predicting literals associated to entities (c.g., a date, a person's 's height, the rating of movie). Datasets statistics are given in Tables an the appendix.
Downstream models. We use scikit-learn (Pedregosa et al,, 2011) using only the node embedding of the pre-trained KG model as input. For classification. we use multilayer perceptrons (MLP), logistic regression, KNN, and random forests. For regression, we use MIP and linear regression.
Downstream training. Each model was trained using 5-fold cross validation and selected based on mean validation performance across folds (see below). We then retrained the selected model on the union of the training and validation split (if present). To tune hyperparameters, we use 10 trials of random search with SOBOL sampling for each downstream model. The search space is given in Table Note that we treat the choice of downstream model hyperparameter as well.
Downstream evaluation. For entity classification, we report weighted Fl, as in Jain et al. 2021), aggregated across all classification tasks (denoted EC). For regression, we chose relative squared er- ror (RSE) because it is interpretable and allows meaningful averaging across the different regression tasks (denoted REG). An RSE value of is equivalent to the performance of a model that predicts the average of the dependent variable in the evaluation data; lower values are better. For each metric, we report the mean and standard deviation over 3 training runs of the downstream model.
Downstream baselines. We consider multiple baseline models to contextualize the results from pre- trained KGE models. In contrast to KGEs, the baselines are directly trained on the downstream task (i.e.. no pre-training) and need to access the KG to perform predictions. We include KE-GCN (Yu) ctt al. 2021). a recent GNN with state-of-the-arr results for graph alignment and entity classification. For regression tasks, we hyperparameters using 30 SOBOL trials (as for KGE models); the search space is shown in Table 9 For training, evaluation, and model selection, we follow the approach for downstream models (e.g.. 5-fold CV). We also consider selected SOTA results of other downstream models; see Sec. 4.5
linear layer after the final convolutional layer of KE-GCN We tune
4.2 GRAPHE-STRUCTURE PREDICTION
In Table we report test MRR of all graph-structure prediction tasks from Table for KGE models using standard training and link prediction for model selection (STD) and our proposed multi-task training and model selection (MTD) Bold entries show best performance per metric and evaluation method. For easier comparison between STD and MTT, underlined entries highlight the best perfor- mance compared to the entry with the same corresponding KGE model on the same dataset, but that uses the other training method. The columns labeled Downstream Tasks are discussed in Sec. 4.3
II our experiments, this to hetter performance than using single dimensional output in the final convolution layer as done by Huang et al. (2021).
DDu space constraints, we report results on WNRR in Table 11 the appendix.

---

Under review as a conference paper at ICLR 2023
Gruph-structure prediction (t)
Downstream tasks
LP
REL DOM NBE NBR
MTR
EC (1)
REG (U)
ComplEx
STD
.347
.805
.098
.011
.041
.200
844+.008
447---051
MTT
.331
.76
.159
.048
.672
.378
.388.002
4411.0000
DistMult
STD .342
.388
.032
.007
.033
.135
883++0009
.551+-062
FB15K-237
MTT .327
.957
.160
.034
.670
.371
.862+0001
.490+-0552
RotatE
STD .287
9199
.114
.024
.106
.225
8644.005
.6184.037
MTT .295
.965
.158
.047
_6588
.370
.83
.54-+.081
TransE
STD .300
.000
.111
.018
.049
.113
.81
.628+.045
MTT .291
.633
.15
.062
.641
368
.855.0003
.9911.147
KE-GCN
.829 4.226
5511+.001
ComplEx
STD .550
.890
.001
.050
.386
.318
.712+.008
-889+0023
MTT .510
.943
.045
.071
.713
.403
.729+
5394-0037
DistMult
STD
.539
.877
.004
.069
.434
.330
..38
.5191.019
YAGO3-10
MTT .538
.941
.046
.061
.740
.412
.75
.4764.055
RotatE"
STD .429
.693..002
.745+.030
MTT .324
.931
.032
.079
.638
.343
.74
553-1.074
TransE'
STD
.490
7411..001
4844.0033
MTT
.263
.959
.037
.080
.617
.330
.755
.326..022
KE-GCN
700+:.223
.388
Not evaluated on new graph-structure prediction tasks due to high cost. GCN-based model by Yu et al. [202] trained directly downstream tasks.
Table 2: Performance on test data graph-structure prediction and downstream tasks with STD and MTT training, well KE-GCN. For graph-structure prediction, report MRR (higher better), for entity classification (EC) we report weighted F1 (higher is better), and for regression (REG) show relative squared error (lower is better). Bold entries show best performance per task. Underlined entries show best performance between STD and MTT.
The results show that across all datasets and KGE models, STD training performed poorly on all graph-structure tasks, except LP and (often) REL. The performance for these tasks improved signif- icantly with MTT training in almost all cases; these tasks have been introduced as auxiliary training objectives. This suggests that models trained solely using link prediction fail to capture graph struc- ture more generally. Also note that MTT models had slightly lower performance on LP, but the decrease often small and outweighted by significantly improved performance over the other tasks (often 2x-4x, up to 10x, depending on model, task and dataset). A notable exception was NBE, which the only task that uses wildcard embeddings for relations. Here STD occasionally outperformed MTT (on YAGO-10 using DistMult and often on WNRR; see Tab. TTin the appendix). Generally, however, MTT improved significantly on STD for graph structure prediction and can thus be used to improve KGE's ability to learn multiple graph tasks simultancously.
4.3 DOWNSTREAM TASKS
Table 2 also shows mean performance across all downstream tasks for each benchmark dataset. As before, bold entries show best performance per metric and evaluation method, and underlined entries facilitate performance comparisons across the different training approaches. We report performance for each individual downstream task in table to 16 in the appendix.
The best overall downstream task performance across all KGE and KE-GCN models was achieved by MTT in all cases. The margin compared to STD was sometimes small (e.g., EC on FB15K- 237) and sometimes large (e.g., REG on YAGO3-10). The margin compared to KE-GCN, which trains directly on each task, was large. Nevertheless, STD training occasionally performed better than MTT (e.g., EC tasks for FB15K-237). This suggests that capturing wider variety of graph structures does not necessarily translate to better downstream task performance. We explore this further in Section 4.4 where consider subsets of the MTT tasks. Ultimately, conclude that

---

Under review as a conference paper at ICLR 2023
Graph-structure prediction (t)
Downstream tasks
LP
REL DOM NBE NBR
MTR
EC (1)
REG (1)
ComplEx STD
.347
.805
.098
.011
.041
.200
8444.008
.447.0051
MTT
.331
.966
.159
.048
.672
.378
.388.002
441=
w/o LP
.201
.972
.159
.065
.676
.356
822:1.006
.718=.040
w/o DOM
.302
.967
.153
.023
.677
.372
.883.0009
.50=.031
w/o NBE
.308
.967
.161
.003
.677
.371
8744.008
.512=.033
w/o NBR
.299
.971
.155
.034
.487
.331
8800.0008
.506=.099
TransE
STD
.300
.900
.111
.018
.049
.213
881
.628..04
MTT
.291
.963
.155
.062
.641
.368
855$.003
.991=.147
w/o LP
.249
.968
.160
.034
.667
.359
8701.000
.456-.034
w/o DOM
.294
.665
.151
.033
.672
.370
.82++000
5515
w/o NBE
.299
.966
.159
.009
.667
.366
881#.004
.466 .0033
w/o NBR
.296
.964
.156
.059
.572
.354
.8594.002
.603 150
Table 3: Performance on FB15K-237 of graph-structure prediction and downstream tasks of STD and various forms of multi-task training of KGE models test data. Metrics and format follow those of T 2 The objective w/o LP is an MTT objective with all tasks in Table exceept for LP.
the choice of pre- pre-training objective clearly has impact on downstream performance, although currently unclear how to make this choice.
Our results also suggest that-perhaps surprisingly- with weaker performance during pre- training with both STD and MTT often performed competitively in downstream tasks and some- times outperformed models with stronger pre-true vre-training performance. For example, ComplEx considerably outperformed RotatE and TransE on FB15K-237 on LP and MTR, but both models outperformed ComplEx on the EC tasks for that dataset. Similar observations can be made about both EC and REG tasks on YAGO3-10. The REG tasks on FB15K-237 were an exception though; here higher performance during pre- pre-training translated to better performance on downstream tasks. Generally, these results are problematic, as they suggest that LP and MTR are often inadequate to guide the choice of the KGE model class, problem that needs further exploration in future work.
4.4 IMPACT OF TASK SELECTION AND MODEL SELECTION
Next, we explored the impact of task selection and, in particular, whether all proposed MTT tasks beneficial. To keep computational costs feasible, focused FB15K-237 with ComplEx and TransE. We explored performance using STD, MTT, and MTT without either the LP, DOM, NBE, or NBR pre-training task. Our results summarized in Tab.
We found that for graph structure predictions, excluding a task generally led to lower performance on that task, expected. may also, however, lead boost in performance other tasks. For example, the best NBE performance for ComplEx is obtained when LP is excluded.
For downstream tasks, we observe that the choice of training tasks can have significant impact and that good choices differ between KGE models and downstream tasks. For example, compared to full MTT training, using a subset of tasks led to large improvements for ComplEx on EC and for TransE on REG. In both cases, as well as with TransE on EC, the best performance is obtained by removing one of the tasks during training. This reinforces our previous observation that including more tasks during pre-training does not necessarily lead to higher downstream performance, but it also provides more evidence that STD training is not enough for good downstream task performance, fact, good models can be obtained without including the link prediction tasks: e.g., the best performance for TransE on REG was obtained when LP was excluded.
We also explored the impact of model selection methods. Table 4 reports performance on FB 9B1K-- 237 of some KGE models using both training approaches across different types of model selection methods: selecting LP (the standard approach), selecting on MTR and selecting directly the metric used to evaluate the downstream task. We found that STD training performed best com- bination with L. model selection. MTT performance on downstream tasks improved consistently
8

---

Under review as a conference paper at ICLR 2023
Selection Method
EC Weighted F1 (*)
REG RSE (J)
LP
MTR
Weighted Fl
LP
MTR
RSE
FB15K-237
ComplEx
STD
.844
.830
.850
.447
.654
.437
MTT
.858
.838
.827
.394
.441
.393
DistMult
STD
.873
.825
.846
.550
.677
.5339
MTT
.865
.661
.864
.471
.489
.476
Table 4: Performance on FB15K-237 downstream tasks for different KGE model training (STD/MTT) and selection approaches (LPgMTR/weighted FI/RSE).
Weighted FI/RSE use downstream tasks data for model selection.
when using LP instead of MTR for model selection, however. Model selection with the downstream task metric provides only marginal benefits for both STD and MTT and can in fact be detrimental, likely due to overfitting on validation data. This indicates that model selection without information about downstream tasks-i.e., using LP MTR-is suitable. The combination that performed best in our study was MTT training and LP model selection.
Overall, found that full MTT training and MTR for model selection (as used our main results of Tab. ( was suitable choice, but further improvements are possible by dataset-, model- and task- spsk-spec choices of pre-training task and validation objective.
4.5 COMPARISON PASK-SPECIFIC MODELS
We compared the performance of pre-trained ComplEx model (using MTT) to best results for additional downstream tasks from the literature. These prior results were obtained by task-specific models and were not reproduced by us; see Sec. Se3. for description of tasks and detailed results. We found that in most cases, this pre-trained ComplEx model did not reach the performance of SOTA task-specific models (which in some cases leveraged additional information). More explo- ration is needed to whether and when pre-trained KGE models are preferable (e.g., as in the tasks of Tab. 2 and on the effectiveness-cost trade-off of alternative approaches.
CONCLUSION
In this work, we explored methods pretrain KGE models for tasks beyond link prediction. First, we showed empirically that commonly trained KGE models fail at basic graph-structure prediction tasks and proposed novel multi-task training and ranking approaches. These multi-task KGE mod- els led to substantially better performance, i.e, their embeddings captured more information about graph structure. Second, we explored downstream task performance for number of entity classifi- cation and regression tasks. Here multi-task training generally led to the best overall performance, margin was sometimes small. Our ablation studies suggest that pre-training can be further improved by a data- and model-specific selection of both pre-training tasks and model selection met- ric. Generally, more research needed on how to make these choices and, more generally, on the relation between pre-training KGE models and their general suitability for downstream applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mehdi Ali, Max Berrendorf, Charles Tapley Hoyt, Laurent Vermue, Sahand Sharifzadeh, Volker Tresp, and Jens Lehmann. Pykeen L.0: A python library for training and evaluating knowledge graph embeddings. J. Mach. Learn. Res., 22(82): 2021.
Carl Allen, Ivana Balazevic, and Timothy Hospedales. Interpreting knowlege graph relation repre- sentation from word embeddings. In Ninth International Conference on Learning Representations 2021, 2021.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, et al. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022.
Stephan Baier, Yunpu Ma, and Volker Tresp. Improving visual relationship detection using seman- tic modeling of scene descriptions. In Claudia d' Amato, Miriam Fernandez, Valentina Tamma, Freddy Lecue, Philippe Cudré-Mauroux, Juan Sequeda, Christoph Lange, and Jeff Heflin (eds.), ISWC, pp. 53-68, 2017.
James Bergstra and Yoshua Bengio. Random search for machine learning research, 13(2), 2012.
optimization. Journal of
Peter Bloem, Xander Wilcke, Lucas van Berkel, and Victor de Boer. kgbench: collection of knowledge graph datasets for evaluating relational and multimodal machine learning. In European Semantic Web Conference, pp. 614-630. Springer, 2021.
Antoine Bordes, Nicolas Usumier, Alberto Garcia-Duran Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In Advances in neural information processing systems, 2013.
Samuel Broscheit, Daniel Ruffinelli, Adrian Kochsiek, Patrick Betz, and Rainer Gemulla. Libkge- knowledge graph embedding library for reproducible research. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp- 165-174, 2020.
David Chang, Ivana Balažević, Carl Allen, Daniel Chawla, Cynthia Brandt, and Richard Andrew Taylor. Benchmark and best practices for biomedical knowledge graph embeddings. Proceed- ings of the conference. Association for Computational Linguistics. Meeting, volume 2020, pp. 167. NIH Public Access, 2020.
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. Hitter: Hier- archical transformers for knowledge graph embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10395-10407, 2021a
Yihong Chen, Pasquale Minervini, Sebastian Riedel, and Pontus Stenetorp. Relation prediction as an auxiliary training objective for improving multiorelational graph representations. 3rd Conference on Automated Knowledge Base Construction, 2021b.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE international conference computer vision, pp. 2051-2060, 2017.
Ahmed Ki-Kishky, Thomas Markovich, Serim Park, Chetan Verma, Backjin Kim, Ramy Eskan- der, Yury Malkov. Frank Portman, Sofía Samaniego, Ying Xiao, and Aria Haghighi. Twhin: Embedding the twitter heterogeneous information network for personalized recommendation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2852-2850..20 2022.
Alberto Garcia-Durán, Sebastijan Dumancic, and Mathias Niepert. Learning sequence encoders for temporal knowledge graph completion. In EMNLP, 2018
10

---

Under review as a conference paper at ICLR 2023
Bin He, Di Zhou, Jinghui Xiao, Xin Jiang. Qun Liu, Nicholas Jing Yuan, and Tong Xu. Bert-mk: Integrating graph contextualized knowledge into pre-trained language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp.
2020.
Han Huang. Leilei Sun, Bowen Du, Chuanren Liu, Weifeng Lv, and Hui Xiong. Representation learning on knowledge graphs for node importance estimation, In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
Ihab F Ilyas, Theodoros Rekatsinas, Vishnu Konda, Jeffrey Pound, Xiaoguang Qi, and Mohamed Soliman. Saga: A platform for continuous construction and serving of knowledge at scale. 2022.
Nitisha Jain, Jan-Christoph Kalo, Wolf-Tilo Balke, and Ralf' Krestel. Do embeddings actually cap- ture knowledge graph semantics? In European Semantic Web Conference, pp. 143-159. Springer, 2021.
Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and S Yu Philip. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions Neural Networks and Learning Systems, 2021.
Timothée Lacroix, Nicolas Usunier, and Guillaume Obozinski. Canonical tensor decomposition for knowledge base completion. In International Conference on Machine Learning, pp. 2863-2872.
PMLR, 2018.
Zelong Li, Jianchao Ji, Zuohui Fu, Yingqiang Ge, Shuyuan Xu, Chong Chen, and Yongfeng Zhang. Efficient non-sampling knowledge graph embedding. Proceedings of the Web Conference 2021, pp. 1727-1736, 2021.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Twenty-ninth AAAI conference on artificial intelligence, 2015.
Farzaneh Mahdisoltani, Joanna Biega, and Fabian Suchanek. Yago3: knowledge base from mul- tilingual wikipedias. In 7th biennial conference innovative data systems research. CIDR Con- ference, 2014.
Christian Meilicke, Manuel Fink, Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. Fine-grained evaluation of rule-and embedding-basse systems for knowledge graph completion. In International semantic web conference, pp. 3-20. Springer, 2018.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-way way model for collective learning on multi-relational data. In Icml, 2011.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEF, 2015.
Pedregosa, G. Varoquaux, Gramfort, V. Michel, Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning Python. Journal of Machine Learning Research, 2011.
Pouya Pezeshkpour, Liyan Chen, and Sameer Singh. Embedding multimodal relational data for knowledge base completion. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Wiem Ben Rim, Carolin Lawrence, Kiril Gashteovski, Mathias Niepert, and Naoaki Okazaki. Be- havioral testing of knowledge graph embedding models for link prediction. In 3rd Conference on Automated Knowledge Buse Construction, 2021.
Daniel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. You teach an old dog tricks! on training knowledge graph embeddings. In International Conference on Learning Representations, 2020.
11

---

Under review as a conference paper at ICLR 2023
Tara Safavi and Danai Koutra. CoDEx: A Comprehensive Knowledge Graph Completion Bench- mark. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 8328-8350, 2020.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai. Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representa- tions, 2022.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg. Ivan Titov, and Max Welling, Modeling relational data with graph convolutional networks. In European semantic web conference, pp. 593-607. Springer, 2018.
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural ten- sor networks for knowledge base completion. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahra- mani, and K.Q. Weinberger (eds.), NIPS, volume 26. Curran Associates, Inc., 2013.
Zhiqing Sun, Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations, 2019.
Sudhanshu Tiwari, Iti Bansal, and Carlos R Rivero. Revisiting the evaluation protocol of knowledge graph completion methods for link prediction. In Proceedings of the Web Conference 2021, pp- 809-820, 2021.
Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pp. 57-66, 2015.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Com- plex embeddings for simple link prediction. In International conference on machine learning, pp. 2071-2080. PMLR, 2016.
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. Composition-baseed multi- relational graph convolutional networks. In International Conference Learning Representa- tions, 2020.
Hongwei Wang, Fuzheng Zhang, Xing Xie, and Minyi Guo. Dkn: Deep knowledge-aware network for news recommendation. In Proceedings of the 2018 world wide web conference, pp. 1835- 1844, 2018.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 2017.
Xintao Wang, Qianyu He, Jiaqing Liang, and Yanghua Xiao. Language models as knowledge em- beddings. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI 2022), 2022.
Yanjie Wang, Daniel Ruffinelli, Rainer Gemulla, Samuel Broscheit, and Christian Meilicke. On evaluating embedding models for knowledge base completion. In RepLANLPe ACL, 2019.
Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by trans- lating on hyperplanes. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 28, 2014.
Bishan Yang. Scott Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference knowledge bases. In Proceedings of the International Conference on Learning Representutions (ICLR) 2015, 2015.
Donghan Yu, Yiming Yang, Ruohong Zhang, and Yuexin Wu. Knowledge embedding based graph convolutional network. In Proceedings of the Web Conference 2021, pp. 1619-1628, 2021.
12

---

Under review as a conference paper at ICLR 2023
Zhengyan Zhang. Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1441-1451,
Ying Zhou, Xuanang Chen, Ben He, Zheng Ye, and Le Sun. Re-thinking knowledge graph comple- tion evaluation from an information retrieval perspective. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022.
13

---

Under review as a conference paper at ICLR 2023
Dataset
Entities
Relations
Train
Validation
Test
EC Tasks
REG Tasks
FB15K-237
14 505
237
272 115
17555
20 466
4
10
YAGO3-10
123 182
37
1 0779 040
5000
5.000
8
5
WNRR
40.559
86 835
3034
134
0
Table 5: Statisties of benchmark datasets for pre-training KGEs, including number of entity classi- fication (EC) tasks and regression (REG) tasks.
A APPENDIX
A.1 DATASET STATISTICS
Benchmark
Name
Num. Classes
Train
Validation
Test
FB15K-237
Entity Type
3 (person, org, ...)
6719
680
Profession
5 (artist, writer, ...)
2537
635
Organization Type
(NGO, musical org. org....)
342
86
Writer Type
2 (journalist, poet,
136
34
YAGO3-10
Entity Type
(person, org, ...)
69
17398
Player Type
5 (soccer, hockey, (artists, politician, ...)
33928
8483
Profession
14480
3
Writer Type
7 (poet, novelist,
4870
1218
Scientist Type
10 (physicist, biologist, ...)
2041
511
Organization Type
4 (institution, party,... ...)
1 1248
312
Artists Type
5 (painter, sculptor
520
130
Waterbody Type
5 (lake, occan....
195
49
Table Statistics datasets for entity classification downstream tasks used to evaluate pre-trained KGEs. All datasets created by Jain et al. (2021), they split into trainining and test only and each consists predicting entity types at different levels of the entity hierarchy.
Benchmark Name
Task
Train
Validation
Test
FB15K-233 237
Node Importance
(Entity, Wikipedia page views)
9 877
1 380
2 823
Birth Year
(People, year of birth) (Location, latitude) (Location, longitude) (Person, height in meters)
3538
442
444
Latitude
2 2568
321
322
Longitude
2 260
320
322
Person Height
2 295
287
288
Size Area
(Location, area)
1 731
216
218
Population
(Location, population)
1 1433
193
193
Film Release Year
(Film, release year) (Organization, year founded)
1 1493
186
188
Org Year Founded
985
123
124
Film Rating
(User, rated film to 100) (Person, year of birth)
591
73
75
YAGO3-10 Born on Year
60409
6730
Created on Year
(Entity, year of creation) (Person, year of death) (Entity, year of end/death) (Event, year took place)
23 896
2638
Died on Year
13.582
1513
Destroyed on Year Happened on Year
1 630
186
749
73
Table 7: Statistics of datasets for regression downstream tasks used to evaluate pre-trained KGEs. YAGO3-10 datasets were created by [Pezeshkpour et al. (2018). All FB15K-237 datasets were cre- ated by us, except the node importance task, which was created by Huang et al. (2021).
14

---

Under review as a conference paper at ICLR 2023
A.2
EXPERIMENTAL SETTINGS
Hyperparameter
Values
Embedding size'
{128, 256,512)
Training type
(NegSamp (YAGO3-10), IvsAll (FB15K, WNRR))
Task Weights (MTT)
[0.1, 10). log scale
No. subject samples (NegSamp)
11 1000000], log scale
No. object samples (NegSamp)
1. 1,10000]. log scale
Optimizer
(Adam, Adagrad}
Batch size
{128. 256,5124 512, 1024(except on YAGO3-10))
Learning rate
10-4 1). log scale
LR scheduler patience
0, 10
Lp regularization
L2, L3, None]
Entity emb. weight
[10-20, 10 51
Relation emb. weight
[10-20 10
Frequency weighting
{True, False)
Embedding normalization (TransE)
Entity
(True, False) (Tree, False)
Relation
Dropout
Entity embedding
0.00, 0.5]
Relation embedding
0.00. 0.5)
Embedding initialization
(Normal, Unif, XvNorm, XvUnif)
Std. deviation (Normal)
[10-5,1.0
Interval (Unif)
1.00
Gain (XvNorm)
1.0
Gain (XvUnif)
1.0
For RotatE, embedding size is fixed 128 on WNRR and set to either 128 or 256 for YAGO3-10, 10, For
Transe, this is to either 128 256 for FB15K-22 237 and fixed to 128 for WNRR and 1024 for
YAGO3-10.
RotatE, batch size fixed 256 in YAGO3-10 to 128 on FB15K-237 and WNRR. Transe, this set to either 128 or 256 on YAGO3-10.
Table 8: Hyperparameter search space for pre-training KGE models. Restrictions specific to RotatE and TransE are due to high memory and runtime cost when training these models.
Model
Hyperparameter
Values
MLP
Hidden Layer Sizes
{(100, ), (10, ), (100, 100), (10, 10)} 0.00001, 0.001]
Alpha
Learning Rate Init Solver
0.001, 0.01]
Adamm LBFGS]
Logistic Regression
C
[100, 100000]
KNN
n_neighbors
[1
Random Forest
num estimators
[10, 50, 100, 200)
Linear Regression
Alpha
[0.00001, 0.001]
KE-GCN
Dimension
{16, 32, 64)
Additional Layers
{0, 1, 2) {0.001, 0.005, 0.01, 0.05, 0.13 {0.3, 0.5}
Learning Rate
Alpha
Table Hyperparameter search space for training downstream models. All hyperparameters except those of KE-GCN follow the semantics by scikit-learn.
15

---

Under review as a conference paper at ICLR 2023
A.3 COMPARISON TO SOTA RESULTS
In Table TO we compare downstream task performance of a pre-trained ComplEx model using the MTT approach with state-of-the-art task-specific models from the literature. Note that did train these models ourselves and experimental setups used in prior work may be very different from ours. In these experiment, we did not reach otate-of-the-arr performance with KGE models. Aside from training directly for the task, this may be because (i) some of the datasets (AIFB, MUTAG) are very small so that KGE models do not learn much before overfitting, (ii) some knowledge graphs are multi-modal (MDGENRE, DMGFULL), but we do not leverage this information On MDGENRE, however, the additional modalities do not play a significant role, as not only does our pre-trained KGE achieve comparative performance, but so do the non-multimoda baselines from Bloem et al. (202T). This is not the case with the DMGFULL datasets, however, which contains a significantly higher number of triples from different modalities.
Dataset
Task Metric
Their model Their performance Ours (Selection method)
MDGENRE* EC
Accuracy
Features
0.66
0.66:10.01 (MTR)
0.68+00.00 (Acc)
DMGFULL*
EC
Accuracy
MR-GCN
0.76
0.51:00.00 (MTR)
0.67:10.02 (Acc)
FB15K
NIE NDCG@ 100 RGTN
0.95
0,41#0.00 (MTR)
0.42:00.00 (NDCG)
Spearman
RGTN
0.82
0.75++0.00 (MTR)
0.76:00.00 (Spearman)
MUTAG;
EC
Accuracy
CompGCN
0.85
0.75+00.0 (MTR)
0.75+-0.01 (Acc)
AIFB$
EC
Accuracy
R-GCN
0.95
0.88+00.00 (MTR)
0.88+-0.00 (Acc)
Bloem al. 2021). Huang al. 2021 Vashishth al. (2020)
Schlichtkrull al. 2018)
Table 10: Comparison of entity classification (EC) and node importance estimation (NIE) between best previously published models (directly trained on task) and pre-trained KGEs (MTT, selection with MTR or downstream metric).
"There are multi-modal KGE models, however, e.g. Pezeshkpour al. 2018
16

---

Under review as a conference paper at ICLR 2023
A.4
ADDITIONAL EXPERIMENTAL RESULTS
Graph-structure prediction (t)
Downstream tasks
L.
REL DOM
NBE
NBR
MTR
EC (t)
REG (4)
ComplEx
STD
.474
.782
.001
.150
.578
.354
MTT
.460
.833
.024
.181
.791
.16
DistMult
STD
.447
.767
.001
.169
.605
.356
WNRR
MTT
.435
.897
.025
.190
.781
.417
RotatE
STD
.460
.794
.001
.210
.713
.398
MTT
.422
.874
.024
.152
.790
.408
TransE
STD
.174
.707
.001
.199
.307
.212
MTT
.175
.837
.021
.109
.747
.327
Table 11: Performance on graph-structure prediction and downstream tasks of STD and multi-task KGE models as well as KE-GCN on test data. For graph-structure prediction, report MRR (higher is better), for entity classification (EC) we report weighted Fl (higher is better), and for regression (REG) we show relative squared error (lower is better). Bold entries show best performance per task. Underlined entries show best performance between STD and MTT training.
Average training epoch time in seconds FB15K-237
YAGO3-10
WNRR
ComplEx STD
04.92
097.88
2.32
MTT
10.83
137.13
8.13
DistMult STD
04.29
095.82
02.10
MTT
09.27
222.37
10.98
Table 12: Average training epoch time in seconds over first epochs of best models with STD and MTT training. All tests were done with an 1tth gen. Intel Core 17-11700K, 64GB of RAM and an NVIDIA GeForce RTX 3090.
17

---

Under review as a conference paper at ICLR 2023 3
Table 13: Weighted F1 on test data of downstream classifiers (MLP. Logistic Regression, KNN and Random Forest) that use pre- re--triiee KGE embedkiigs. as input to solve entily classification tasks about entities in FB 15K- 237: and KE-GCN Yu et al. [021]. a GCN that trains directly on the downstream data. Bold entries show best performance per task. Underlined entries show best performance per task compared to same KGE model and same downstream model. Datasets are sorted by decreasing size of the training set from left to right.
.661 + 000
.656_1.00 000
.785 020
17291.000 000
.781 000
.828 028:1000 000
.810 .000
.683 005
685 020
Writer
higher is better)
Organization
921 02.
900+:.00 000
.912 009
.299 .000
.895 012
.933 004
.935 005
.937 00.
.90 002
(Weighted Fl
FBISK-233
Profession
19:::01. 011
.811 007
811+:007 .007
8. .00
798=.007 007
.797
.794 7977
007
.811:003
738 000
.808
Entiry Classification Type
ComplEx STD .986 001
.9861 000
DistMult STD .9841: 000
996 16611.00 .000
STD .983 000
MTT 9871.000 .00
001
98811001 .011
.98 100
STD .974
MTT
MTT
MTT
KE-GCN
RotatE
TransE
18

---

Under review as a conference paper at ICLR 2023
Table 14: Weighted F1 on test data of downstream classifiers (MLP. Logistic Regression, KNN and Random Forest) that use pre- rre-trained KGE embedkligg. as input to solve entity classification tasks about entities in YAGO3- 10; and KE-GCN (Yu al. [021]. GCN that traims directly on the downstream data. Bold entries
show best performance per task. Underlined entries show best performance per task compared to same KGE model and same downstream model. Datasets are sorted by decreasing size of the training set from left to right.
Waterbody
6674+.013 115
.664 000
.0000
722-1.000 000
.593 000
.6255.013
.6704 000
.673 000
4881.014
.738
.722
.738
.480 018
.4761.005 .00
5355.0077
4961.014
.470 000
531
4991.013
.5431.000
4633.014
Artist
Organization
.7891 0000
.867 .015
.814H 001
.8831 .005
.730 013
.8161 007
.8334 0000
.8531 0008
.8534 .006
higher is better)
.518 013
.538 002
.529 003
.545 000
013 .58 4.013
.630 .000
.600+.0144 0114
.610 000
Scientist
33611.015
(Weighted Fl
.575 7006
.607 +000 000
.5771.000
:00
.618 61811002 023
000
.628 000
6231.000 000
.6661.000 .00
.582..003 005
YAGO3-10
Writer
.606
Entiry Classification
Profession
.753+ 004
.7611.003 003
.764 003
.7901.00 000
7092.002
.709 002
7611 100
76224.000
000
78811002 003
77091000 000
.988 000
.918 .00
.199 000
9191.001 000
918 002
.914..011
.920 001
.23 001
.896 001
Player
.994 000
.96 .00
.994 000
.96 99611.00 1..00
9974 000
993 000
.992+.00M 000
995 000
0.996-10.000
Type
ComplEx STD
MTT
DistMult STD
MTT
STD
MTT
STD
MTT
KE-GCN
RotatE
TransE
19

---

Under review as a conference paper at ICLR 2023 -
Table 15: Relative squared enror (RSE) on test data of downstream models (MLP and Linear Regression) that use pre- traineeind KGE embeddings as inpul lo solve regression tasks about entities in FB FB155-23 -37: and KE-GCN (Yu et al. 2021] a GCN that trains directly on the downstream data. Bold entries show best performance per task. Underlined entries show best performance per task compared to same KGE model and same downstream model. Models with RSE above I are considered unsatisfactory. Datasets are sorted by decreasing size of the training set from left to right.
Film Rel. Year Date Founded Film Rating
.736 046
.727 029
8113 002
907 148
.798 059
.7761.066 062
.778+.066 .04
740
1.344 .691
.494 0.2
.440 083
6277:003
.5841. 004
6811.000
048
.722 134
.51 123
.3331.00
498
.68
.156-:01 016
.16- .004
.152 .003
113 1331..004 .004
.175 -000
0.1501.011
1714.027
163 007
.1441.00 .008
Population
4424.077
6524:.03
.114:.09 09.
.4981. 1.005
.364..125
125
764112248
1.702+.055
8800 347
.644.0051
Size Area
.2341.018
.457 1.266
4124.318
18917005
911
4281.219
.432.003
5.9021.946
7544.0180
Regression IRSE lower better)
089
Longitude Person Height
FB1SK-237
.678 010
.6561.015 015
003 .669
+003
.6741. 006
-00 000
.7221.020 020
730+.021
021
.7801.010
.7881.002
.657
0894.010 010
191
+.05
.05
10941.000 .008
2894.032
032
1661.011
00791005
.0411.002
.1131.003
.088
Latitude
.172+.013
2111.010
.1821. 031
016
.200 044
3111.028
1584.
003
.122..015
8044.005 37641.035 035 .2181.023
Birth Year
.6011.23 239
04.8
042
.722 091
027
88224 067
110
6681.005 057
.302+.044
8444.006
.756
9904
Node Imp.
ComplEx STD .870 048
7771:.014
STD 8074.00 023
MTT 9941. 066
STD 8774 026
MTT 8461 005
STD .191.000
MTT 8331.015
MTT
DistMult
KE-GCN
RotatE
TransE
20

---

Under review as a conference paper at ICLR 2023
Table 16: Relative squared error (RSE) on test data of downstream models (MLP and Linear Regression, that use pre ce--rrined KGE embeddings as inpul to solve regression tasks about entities in YAGO3- 10; and KE- GCN et al. 2021]] GCN that trains directly on the downstream data. Bold entries show best performance per task. Underlined entries show best performance
task compared to same KGE model and same downstream model. Models with RSE above I are considered unsatisfactory Datasets are sorted by decreasing size of the training set from left to right.
Happened Date
.324 006
.3761.093
.311 030
3651..068
.314 014
.2241.008 008
.3764.112 125
09111.008
.6671.001 701000
Destroyed Date
872100000
.7291.038 036
7733:.04 004
.725 7251.018
1.040 090
.7501 060
.6174.063 .033
.3571.041
.6571.045
Regression (RSE lower is better)
Died on Date
.555 014
.520 031
.466 025
.39 3981L.01 1.018
.849 000
480 .02
-3641.025 .28
22941.013
2991.011
YAGO3-10
Created Date
.672. .033
.619 .055
.6121. 004
.560 .500 .012
.828 005
1.097:220 266
.6874. 034
.6191.05 034
6111:008
Born on Date
.519 000
.440 1
011
STD .432
33311.013 0013
STD 691
019
4124.011
STD 365
2681.013
.256 009
ComplEx STD
MTT
MTT
MTT
MTT
DistMult
KE-GCN
RotatE
TransE
21


Reviews:
[{"review_id": "h88igodc1B8", "reviewer": "Reviewer_1Rac", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to explore the suitability of various Knowledge Graph Embeddings (KGE) for graph structure prediction tasks such as entity and relation neighborhood prediction, type/range of a given relation, and relation prediction. This text also investigates the performance of these models on several downstream tasks. Building upon these two investigations, the authors then propose a Multi-Task Training (MTT) objective to potentially improve the performance on graph structure prediction tasks and downstream applications without further task-specific training. ", "summary_of_the_review": "Overall, I feel this is a good piece of work with a well-known and well-founded research problem. Authors have done extensive evaluation (with some minor weaknesses) to explore the use of Multi-Task Training objectives to improve the generalizability of Knowledge Graph Embeddings methods. To that extent, I vote to accept the paper. ", "strength_and_weaknesses": "Strengths\n\n+ Motivation for the work is well-founded in literature and concretely defined in the work. \n+ Authors introduce several new basic graph structure prediction tasks (Neighborhood and Domain prediction). They also evaluate existing models on these tasks showing the poor generalizability of current KGE methods to such tasks.\n+ Further, this work proposes an MTT objective that includes the well-known Link and Relation Prediction tasks in conjunction with other graph structure prediction tasks introduced in this work.\n+ Evaluation shows that MTT training does help improve the existing KGE model's performance on graph structure prediction tasks and downstream tasks. \n\nWeaknesses / Questions\n\n+ Authors should provide an ablation study in order to better understand the role of each task introduced in MTT. \n+ In my opinion, this method loses the Efficiency vs Performance tradeoff. It takes 2x-4x (worst) more time to train but seems to marginal gain in performance on downstream tasks.\n+ It would have been interesting to see the results of KE-GCN on basic graph structure prediction tasks as well (treating these tasks as downstream and training on them). \n+ Instead of YAGO3-10, why didn't the authors use one of the [OGB](https://ogb.stanford.edu/docs/linkprop/) datasets for large-scale experiments? I would like to know if this is a conscious design choice by the authors.\n+ \"We also use the so-obtained wildcard embeddings for prediction in the same fashion; e.g., we set $$s(s, r^{'}, *) = s(s, r^{'}, any_{O})$$\". I am not sure which prediction the authors are talking about here. \n+ Please provide the code used for all the experiments. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Sh-ny5th7bq", "reviewer": "Reviewer_x7Bp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a multi-task training strategy for improving the performance of the knowledge graph embedding (KGE) model in downstream tasks, e.g., entity classification or regression. In addition to the conventional link prediction training of KGE, the proposed model is pre-trained with 4 extra tasks, e.g., relation prediction, entity neighboring, etc. The experiments on FB15K-237, YAGO3-10, and WRNN show the multi-task trained model achieves better MRR on the pretraining tasks but lags behind LP than the single training (as expected). The performance on downstream tasks are marginally better for certain KGE model, e.g., RotatE on FB15K-237. Overall, the performance of graph-structure prediction tasks (e.g., NBE) is not stunning as dedicatedly trained. The performance of downstream tasks is also limited.", "summary_of_the_review": "1.\tThe results of applying multi-task training are not stunning at all. The model trained on the single link prediction task (STD) still has the best performance on the link prediction task (Table 3). And the model trained on extra tasks (MTT) no doubt performs well on the extra tasks. I would suggest authors train models ablative, i.e., for the unseen test tasks. In addition, the performance improvements on downstream tasks are also limited. E.g., On the FB15K-237 dataset, the STD strategy excels the proposed MTT for nearly all settings of EC (except for RotatE). In this regard, the effectiveness of the proposed multi-task training strategy is nor promising.\n\n2.\tThe domain and range predictions seem not widely used tasks for KGE. Are there any reasons for choosing these two tasks? E.g., do they have any potential for benefiting downstream applications?\n\n3.\tAs the SOTA performances of KGC are achieved by pretrained LM-based models, such as KG-BERT (Yao et al., 2019), KGT5 (Saxena et al., ACL’22), and KG-S2S (Chen, et al., COLING’22). The applicability of the proposed MTT strategy should be further discussed.\n\n4.\tIt is not clear how the wildcard entities (MTT part) approach the max-aggregation (MTR part).  The max-aggregation search for the best relation over entire entity set, while the wildcard entities make use of an abstract \"wildcard\" embedding to get the result. Although using the wildcard entity would be easier as it reduces the computational cost for searching, but I believe the two methods The methodologies for the two methods are conceptionally different and may have different results.\n\n5.\tWhy STD has better results than the dedicated MTT on NBE task (0.21 v.s. 0.152, Table 3) needs further explanation. \n\nMinor comments:\n(1) Authors should give the description of abbreviation STD (i.e., the model trained with single task, i.e., LP.) when it is first introduced in Section 4.1.\n(2) .. replace wildcards by their the corresponding .. -> remove the (page 5)\n", "strength_and_weaknesses": "Strength:\n1.\tSome observations are interesting, e.g., KGE performs poorly on graph-structure prediction tasks other than link prediction. \n2.\tThe paper is clearly written. \n\nWeakness:\n1.\tThe results are not stunning. The multi-task training does not lead to consistently better performance on all graph-structure prediction tasks. The performance improvement on downstream tasks is not effective. \n2.\tSome experimental settings are not rigorously designed. E.g., MTT should test on unseen tasks. The experimental results could be further discussed or explained, i.e., the variation of model behavior under different datasets/settings.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NKNpVs-tRp4", "reviewer": "Reviewer_RUJy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a thorough empirical investigation of the performance of knowledge graph embedding models on a variety of graph structure prediction tasks, including link prediction, relation prediction, domain prediction, and neighborhood prediction. The results show that these models do not perform well on the tasks except link prediction. The authors go on to propose and test multi-task learning to solve this problem. Multi-task learning can also help with KG downstream tasks like entity classification and regression.", "summary_of_the_review": "Overall, this paper sheds some light on both KG embedding models and their applications. However, most of the findings, in my opinion, are common knowledge in the related field. Some recent representative KG embedding models and in-depth analysis of the reasons for the performance are lacking in the empirical study. So, I think the contribution of the paper is not enough.", "strength_and_weaknesses": "This paper presents some \"interesting\" findings regarding KG embedding models and applications. It would motivate us to reconsider the suitability and generalizability of KG embedding models. However, I have the following concerns.\n\nFirst, as a KG embedding expert, the finding does not surprise me that a model trained on the link prediction task performs poorly on other structure prediction tasks. There is a significant gap between the training and evaluation tasks in this experiment. It is also not surprising that multi-task learning can improve the model's performance on some structure prediction tasks, because, just as stated by the authors in Section 4.2, \"these tasks have been introduced as auxiliary training objectives\". So, I think this paper is not so attractive to the KG embedding researchers, although I appreciate the authors' hard work. It lacks in-depth analysis of the reasons behind the performance.\n\nSecond, the selected four KG embedding models in the empirical study, i.e., TransE, DistMult, ComplEx, and RotatE, are old. I think these four models could not adequately represent the large and fast-growing family of KG embedding models. In Section 4.1, the authors state that ComplEx is the SOTA factorization model. But in my view, the SOTA factorization-based KG embedding model is TuckER [1]. Also, some recent GNN-based [2] or Transformer-based KG embedding models are ignored in this study. So, a natural question is, whether the recent KG embedding models still suffer from the claimed issue in this paper.\n\n[1] Ivana Balazevic, Carl Allen, and Timothy Hospedales. 2019. TuckER: Tensor Factorization for Knowledge Graph Completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5185–5194, Hong Kong, China. Association for Computational Linguistics.\n\n[2] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal A. C. Xhonneux, Jian Tang: Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS 2021: 29476-29490\n\n[3] Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. 2021. HittER: Hierarchical Transformers for Knowledge Graph Embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10395–10407, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject