Question: What does the assessment task include?
Evidence: We consider three tasks, link prediction, triple classification, and predicting missing relation types BIBREF24 :
Link Prediction A common task for knowledge graph completion is link prediction, aiming to predict the missing entity when the other two parts of a triplet (h, r, t) are given. In other words, we need to predict t given (h, r) – tail-entity prediction or predict h given (r, t) – head entity prediction. In-line with prior work BIBREF6 , we report results on link prediction in terms of Mean Reciprocal Rank (MRR), Mean Rank (MR) and Hits@10 on the FB15K-237 dataset in the filtered setting on both sub-tasks: (a) head entity prediction and (b) tail entity prediction. Our results are shown in Table 2 . Note that the present state-of-the-art model, ConvKB achieves an MRR of $(0.375, 0.487)$ on the head and tail link prediction tasks. Observe that simply incorporating Dolores significantly improves the head and tail entity prediction performance by $3.10\%$ and $7.90\%$ respectively. Similar improvements are also observed on other metrics like Mean Rank (MR: lower is better) and Hits@10 (higher is better).
Triple Classification The task of triple classification is to predict whether a triple (h, r, t) is correct or not. Triple classification is a binary classification task widely explored by previous work BIBREF2 , BIBREF9 , BIBREF10 . Since evaluation of classification needs negative triples, we choose WN11 and FB13, two benchmark datasets and report the results of our evaluation in Table 3 . We note that the present state of the art is the ConvKB model. When we add Dolores to the ConvKB model with our embeddings, observe that we improve the average performance of the state-of-the-art model ConvKB slightly by $0.20$ points (from $88.20$ to $88.40$ ). We believe the improvement achieved by adding Dolores is smaller in terms of absolute size because the state-of-the-art model already has excellent performance on this task ( $88.20$ ) suggesting a much slower improvement curve.
Missing Relation Types The goal of the third task is to reason on the paths connecting an entity pair to predict missing relation types. We follow Das et al. ( BIBREF24 ) and use the same dataset released by Neelakantan, Roth, and McCallum ( BIBREF23 ) which is a subset of FreeBase enriched with information from ClueWeb. The dataset consists of a set of triples ( $e_1$ , r, $e_2$ ) and the set of paths connecting the entity pair ( $e_1$ , $e_2$ ) in the knowledge graph. These triples are collected from ClueWeb by considering sentences that contain the entity pair in Freebase. Neelakantan et al. ( BIBREF23 ) infer the relation type by examining the phrase between two entities. We use the same evaluation criterion as used by Das et al. ( BIBREF24 ) and report our results in Table 4 . Note that the current state-of-the-art model from Das et al. ( BIBREF24 ) yields a score of $71.74$ . Adding Dolores to the model improves the score to $74.42$ yielding a $\textbf {9.5\%}$ improvement.
Altogether, viewing the results on various tasks holistically, we conclude that simply incorporating Dolores into existing state-of-the-art models improves their performance and advances the state of the art on each of these tasks and suggests that our embeddings can be effective in yielding performance gains on a variety of predictive tasks.
We consider three tasks, link prediction, triple classification, and predicting missing relation types.
Answer: Link prediction, triple classification, and predicting missing relation types.