Are Embedded Potatoes Still Vegetables? On the Limitations of WordNet Embeddings for Lexical Semantics
Authors: Xuyou Cheng
Michael Sejr Schlichtkrull
Guy Emerson
Venue: EMNLP 2023 Main (2023)
Keywords: Knowledge Base Embedding
Lexical Semantics
WordNet
Link Prediction

Abstract: Knowledge Base Embedding (KBE) models have been widely used to encode structured information from knowledge bases, including WordNet. However, the existing literature has predominantly focused on link prediction as the evaluation task, often neglecting exploration of the models' semantic capabilities. In this paper, we investigate the potential disconnect between the performance of KBE models of WordNet on link prediction and their ability to encode semantic information, highlighting the limitations of current evaluation protocols. Our findings reveal that some top-performing KBE models on the WN18RR benchmark exhibit subpar results on two semantic tasks and two downstream tasks. These results demonstrate the inadequacy of link prediction benchmarks for evaluating the semantic capabilities of KBE models, suggesting the need for a more targeted assessment approach.

Are Embedded Potatoes Still Vegetables?
On the Limitations of WordNet Embeddings for Lexical Semanties
Xuyou Cheng xc341 @cantab. ac. uk
Michael Schlichtkrull University of Cambridge mss840cam. ac. uk
Guy Emerson
University of Cambridge
gate20cam. ac uk
Abstract
Similarity
Analogy
POS
NER
Knowledge Base Embedding (KBE) models have been widely used encode structured information from knowledge bases, including WordNet. However, the existing literature has predominantly focused on link prediction as the evaluation task, often neglecting exploration of the models' semantic capabilities. this paper, we investigate the potential disconnect between the performance of KBE models of WordNet link prediction and their ability to encode semantic information, highlighting the limitations of current evaluation protocols. Our findings reveal that some top- ap-performing KBE models on the WNISRR benchmark exhibit subpar results on two semantic tasks and two downstream tasks. These results demonstrate the inadequacy of link prediction benchmarks for evaluating the semantic capabilities of KBE models. suggesting the need for more targeted assessment approach.
MRR
-0.749
-0.539
-0.645
-0.757
H@1 1
-0.843
-0.650
-0.717
-0.716
H@ 10
-0.355
-0.138
-0.220
-0.705
Table I: Pearson correlation between link prediction metries and other evaluation tasks, for the results in Table 2. For all metrics, higher is better, but the link pre- diction metrics are anticorrelated with the other tasks.
link prediction for an artificially selected subgraph. For instance, the WN18 benchmark (Bordes et al., 2014) is created from WordNet by excluding rare synsets and relations, and randomly splitting the data into training and test sets. Further, follow- ing issues with test set leakage (Toutanova and Chen, 2015; Dettmers al., 2018), the subsequent revision WNISRR meticulously filters the splits. removing c.g. explicit inverse relations.
1 Introduction
While link prediction may be a natural task for datasets such as FreeBase, where incompleteness is a major problem (Min et al., 2013), the Princeton WordNet is intended to have complete annotations of lexical relations. Indeed, it has often been used as the basis for developing WordNets for languages other than English (Bond and Foster, 2013). The careful filtering of WNI 8RR might result in fairer benchmark for link prediction, but this evaluation sctup deviates from the objective of learning a se- mantically valuable model.
Knowledge Base Embedding (KBE) models aim to encode the structured information contained in knowledge bases as vectors. These models are of- ten claimed to possess rich semantic information, and have been applied to various downstream se- mantic tasks, such as entity typing (Moon et al.. 2017; Zhao al., 2020), entity alignment (Sun al., 2020), rule mining (Yang et al., 2015; Chen et al., 2020), and conceptual clustering -ara-B et al., 2020). the knowledge base encodes seman- tic relations (hypernymy, meronymy, and so on), a KBE model should represent these relations. The assumption is that this knowledge should be help- ful for more general semantic modelling. A further assumption is that evaluating the training task, i.e. relational link prediction, is suitable proxy for other tasks. Our aim is assumptions.
Some of the semantic information in WordNet is implicit. For example, hypernymy (i.e., the "is a" relation) is transitive, but antonymy (the "op- posite of"" relation) is not; nevertheless, they are both encoded synsets having arcs to their di- rect hypernyms antonyms (Miller, 1995). That is, while hypernym(potato, solanaceous vegetable) and expernyn(solanaceon vegetable, vegetable) are relations in WordNet, hypernym(potato, veg- etable) is not. It is supposed to be inferred by the
these implicit
We focus on models of the Princcton WordNet (Miller, 1995). Typically, KBEs are evaluated on

---

user's knowledge of hypernymy. However, link prediction models are neither trained nor tested for this; rather, they are trained to exclude indi- rect hypemym-relations like inpernym(potato, veg- etable), as they are not present in the training or the test splits. Indeed, to KBE model, hypernymy and antonymy look very similar.
TransE embeds both nodes and relations as real vectors, with scores defined by translation: - 0, -21.. DistMult also uses real vectors, but instead defines scores by componentwise mul- tiplication: "r,
Some KBE models have been designed to meet specific geometric and linguistic criteria (Vendrov et al., 2016; Nickel and Kiela, 2017). For the mul- tirelational setting, Balažević et al. (2019) intro- duce MuRP, which encodes relations as Möbius transformations, with scores defined using the dis- tance between transformed points. Allen et al. (2021) analyse the performance of different scoring functions on different categories of relations. Their findings underscore that models should be selected for specific predictive needs: DistMult performs better for some relation types, MuRP for others. Similarly, Chen (2021) proposed to use different scoring functions for different WordNet relations, motivated by theoretical work on Functional Distri- butional Semantics (Emerson, 2018). 8). Building on this, we introduce FuncE, which embeds node real-valued functioo f RR + [0. 1]. and uses a different scoring function for different categories of relation, according to linguistic criteria: hypernym- based, synonym-based, or other (see Appendix A).
Putting together the heavy filtering of common datasets and the indirectness of some information in WordNet, it is unclear the degree to which KBE models trained on these datasets contain the seman- tic information WordNet is designed include. Indeed, we find that well-performing KBE models on WN18RR are anticorrelated with performance on downstream semantic tasks: the best models of WN18RR perform the worst on semantic bench- marks (see Table 1). This aligns with findings for other types of embedding, where discrepancy in model performance between intrinsic and extrinsic metrics has been observed (see $3).
We provide evidence for two related claims: I) performance on link prediction is not predictive of semantic capability; and II) models developed for link prediction fail to capture the semantics of WordNet. Specifically, we show that models with the best performance on the WN18RR benchmark exhibit poor performance on relatively simple se- mantic tasks (as seen in Table 1, and discussed in $5). We further show that well-performing KBE models fail to properly encode the most basic se- mantic relationship in WordNet, i.e. hypernymy (as discussed in $6). Our findings emphasize the need for careful design of both evaluation bench- marks and model architectures: evaluation must target desired functionality, and models must be theoretically capable of that functionality.
Such a function can be interpreted as the mem- bership function for fuzzy region (Zadeh, 1965). distributional semantics, Erk (2009b,a) has argued in favour of such representations (for a survey, see: Emerson, 2020). Membership functions can nat- urally represent tree hierarchies (like hyperbolic embeddings) but can also naturally represent mul- tiple inheritance hierarchies. Using a real-valued space makes casier switch scoring functions for different types of relation.
2
KBE Models
More precisely, is a hypernym of if f(x) 2 g(x) for all z. For simplicity, we embed nodes as logistic regression classifiers, defined by a weight vector e and bias b. so f(x) = sigmoid + b). This allows hypernymy to be calculated easily: re- stricting the input to the unit sphere, f is a hyper- nym of if by by - - d'gl 2 0. We can use this expression as score for the hypernymy relation. Numerically, this is the same hypemymy condition as for ball embeddings (Dong et al., 2018), if we take v to be the centre of the ball and to be the radius. For synonym-based relations, we use the DistMult score function, and for other relations, we use the TransE score function.
A Knowledge Base consists of triples (s, r, o). de- noting the subject node s being connected by the relation r to the object node o. KBE models aim to learn embedding for each node and relation, that it output higher scores for valid triplets and lower scores for invalid ones.
In this paper, we compare several common mod- els, including TransE (Bordes et al., 2013), Dist- Mult (Yang al., 2015) and MuRP (Balažević et al., 2019), as well as function- based model, FuncE, adapted from Chen (2021). We also discuss GNN-based and distributional approaches.

---

We further experiment with recent works that apply Graph Neural Networks (GNNs) as node en- coders, which can potentially leverage the struc- tural context of each node (Schlichtkrull et al., 2018). In particular, we consider KBGAT (Nathani et al., 2019) and rGAT (Chen et al., 2021), both of which were found to have high performance on link prediction for WN18RR.
found that performance on link benchmarks did not reflect performance on their tests. Together, these studies suggest that link prediction row evaluation metric, even when considering tasks defined on the knowledge base itself. Our work dif- fers from these studies as we primarily focus on external evaluation datasets, investigating the lim- itations of WordNet embeddings in representing lexical semantics. Specifically, we apply WordNet KBE models semantic evaluation datasets and downstream tasks.
nar-
Finally, some work in distributional semantics has compared models trained on corpora against models trained on linguistic resources like Word- Net. Faruqui and Dyer (2015) create sparse binary vectors, based on manually defined features, and find that they competitive with distributional vectors various evaluation datasets. Kutuzov al. (2019) learn synset embeddings from Word- Net by utilizing path-length based similarity mea- sures. Saedi et al. (2018) present Wnet2vec, which generates word embeddings following the intuition that semantically similar nodes are connected by more paths and shorter paths. They show that the resulting embeddings outperform several distribu- tional methods word similarity metries. We consider Wnet2vec as additional baseline.
Differences between thining-related and down- stream measures of performance have been found for several NLP tasks. Chiu et al. (2016), Qiu et al. (201 8), and Torregrossa al. (2020) found that per- formance on common benchmarks for word embed- dings do not correlate well with downstream per- formance. Similar results have been demonstrated especially for intrinsic mcasurements bias and fairness, which often struggle to adequately capture downstream failures (Goldfarb- Tarrant et al., 2021; Cao et al., 2022).
4 Experimental Setup
To demonstrate how performance on link prediction benchmarks is inadequate for assessing semantic capabilities, we evaluate common link prediction models on several semantic tasks. We use two tasks directly testing semantic understanding (word similarity, word analogy), and two downstream tasks using the embeddings IPOSttagging, NER). These tasks serve as representative examples challenges that semantic embedding models should be able to tackle effectively.
3 Related Research on Evaluation
A growing body of research advocates a reassess- ment of the evaluation protocol for KBE models, even if link prediction is taken as the ultimate goal. Wang et al. (2019) and Akrami et al. (2018, 2020) argue that current benchmarks contain too many trivial and too few difficult predictions, advocating for more realistic evaluation. Tiwari al. (2021) discuss how random splits alter the graph topology from the original knowledge base, and show that correcting for this allows simple cuuslltton-based models to outperform more complicated ones. Fi- nally, Kadlee et al. (2017) and Ruffinelli et al. (2020) show that performance on link prediction benchmarks is strongly determined by hyperparam- eters and training mechanisms.
4.1 Word Similarity
A common approach for evaluating semantic mod- els is assess their ability predict how similar words are. Typically, this is done by comparing model predictions with numan-annotalle similar- ity scores for selected word pairs. Model perfor- mance is then measured with Spearman correlation. For models that represent words as vectors, cosine similarity is typically employed to measure word similarity. In our analysis, we use the Sim9 benchmark (Hill et al., 2015a), which includes 999 pairs of common words spanning three parts of speech (nouns, verbs, and adjectives). Each sample in the benchmark consists of a pair of words along with a ground-truin truth similarity score assigned by crowd-sourcing (e.g. "coast shore 9.00").
Going beyond link prediction, Jain et al. (2021) challenge the prevalent belief that KBE models can properly capture the semantics of KB entities and relations. By evaluating common KBE models on entity typing and entity clustering benchmarks, the authors discovered that their results were no better than those achieved by simple statistical models. Similarly, Rim al. (2021) designed tests for spe- cific capabilities of KBE models, such as capturing symmetric relations or hierarchical relations. They

---

WN18RR
Similarity
Analogy
POS-tagging
NER
MRR
10
Spearman
@10
acc.
fl score
TransE
34.49
50.94
25.27
48.60
31.98
76.53
49.18
DistMult
42.47
47.40
39.74
28.76
11.60
67.17
48.43
FuncE
25.87
43.66
16.78
51.19
35.30
77.17
54.54
MuRP
48.85
58.64
43.40
38.16
30.59
67.87
43.89
KBGAT
44.45
58.31
36.72
45.05
31.56
76.44
49.45
IGAT
49.98
59.70
45.30
28.92
13.24
71.56
30.70
Wnet2vec
41.56
20.62
59.10
26.29
Table 2: KBE models' performance on WN18RR, SimLex999, BATS, PTB POS taggingg and NER. For SimLex999 and BATS, only subsets of the datasets are used (782 out of 999 word pairs for SimLex999; 17,109 out 98,000 analogy questions for BATS). Best results marked in bold, and the lowest
underlined.
The choice of SimLex999 is motivated by two factors. Firstly, As opposed to other datasets, is designed to measure specifically the similarity of concepts, while excluding other forms of related- ness that we would not expect WordNet capture (e.g. cup and coffee related but not semantically similar). Secondly, the dataset focuses common words, ensuring that representations derived from WordNet have good coverage.
belling tasks: POS- COS-tagging, using the Wall Street Journal sections of Penn Treebank (PTB) (Marcus et al., 1993), and NER of CoNLL'O3 shared task data (Tjong Kim Sang and De Meulder, 2003).
It is important to note that our objective not to develop state-of of-the-art sequence labelling models, but rather to assess the semantic value of KBE mod- els through comparison. To facilitate fair eval- uation of their semantic capabilities, we employ minimal setup for the sequence labelling tasks. This involves using model with single softmax layer upplied sliding window. Although this model may yield subpar performance in the tasks, allows for a reliable comparison between the se- mantics capabilities of different KBE models. Our setup can be seen as form of probing classifier, testing the degree to which semantic embeddings directly contain the necessary information to pre- dict part-of-speech (Belinkov, 2022).
4.2
Word Analogy
The Word Analogy task (Mikolov et al., 2013) is designed to measure the linguistic regularities cap- tured within the semantic space of word represen- tations. Specifically, models are tested to predict analogies, i.c. to answer questions of the form is to a" as is to ? (e.g., man is to king as woman is to queen). The underlying assumption is that high- quality semantic model with regular latent space should be able accurately answer these by lever- aging the geometric structures embedded within the space (c.g., fa lia = Uh Uz). We employ The Bigger Analogy Test Set (BATS) (Gladkova et al., 2016), which categorizes analogy questions into four main categories: Inflection, Derivation, Lexicography, and Encyclopedia. Each category is further divided into ten catecates,,,,s, resulting in total of 98,000 questions.
4.4 From Synsets to Words
Nodes in WordNet represent synsels, not words. As such, to evaluate embeddings of these nodes our benchmarks, we first convert the synset embed- dings obtained from WordNet into word embed- dings. This conversion accomplished by averag- ing the sense (synset) embeddings corresponding to each word. For instance, the word "car" has two different synsets in WordNet, and the embed- dings for these synsets are averaged to obtain the embedding for "car". It is important to note that some words in WordNet appear in multiple synsets with different parts of speech. The word "dog" serves as an example, acting as both noun repre- senting animals in the Canidae family and a verb meaning to chase. such cases, we do not dif-
4.3 Downstream tasks
Another way to understand if embeddings cap- semantic knowledge is assess their perfor- mance on downstream tasks. As demonstrated by Chiu et al. (2016), intrinsic semantic measures and downstream tasks do not necessarily correlate. We also include experiments on two sequence la-

---

ferentiate the words by part of speech. We base this decision on two reasons: (I) the chosen se- mantics benchmark are designed for distributional word embeddings, which do not incorporate part of speech information during evaluation, (2) part of speech information is not provided in the BATS word analogy benchmarks. For GNN models (i.e.. KBGAT (Nathani et al., 2019) and rGAT (Chen et al., 2021)), which consist of GNN encoder and a decoder, we use the synset embeddings directly output by the encoder as ""embeddings".
is evident among the selected semantic tasks; mod- els that excel in one semantic task tend to demon- strate similar prowess across others. Secondly, no clear correlation exists between a model's accuracy in the link prediction benchmark and its efficacy on the semantic tasks. For example, rGAT, despite achieving the highest accuracy on the WNI8RR benchmark, underperforms in areas such as simi- larity, word analogy, POS-tagging, and NER.
The models that showcase top-tier performance on downstream tasks include TransE, FuncE, and KBGAT. A shared characteristic among these mod- els is their reliance on scoring functions motivated by linguistic insights. Both TransE and KBGAT leverage a translational scoring function as de- fined by (Bordes et al., 2013), which demonstrates a strong geometric association with information- theoretic structure first discovered in distributional word embeddings (Allen et al., 2021). FuncE dis- tinguishes itself by using categorized loss func- tions derived from the insights of formal semantics (Emerson, 2018, 2020). This stands in stark con- trast models such as rGAT and DistMult, which lack the underpinning of similar linguistic theories. This discrepancy provides a plausible explanation for their inferior performance on the chosen seman- tic tasks.
5 Many Ways to Skin Potato: Comparing Evaluation Metries
In this section, we evaluate range of models (see $2) on both link prediction and also semantic tasks (see $4).
5.1 Models trained on WN18RR
The most common link prediction dataset for WordNet is WN18RR, a subset developed by Dettmers et al. (2018). The subgraph represented by WNI8RR has been extensively filtered, remov- ing e.g. explicit inverse relations, to produce an excellent link prediction behenmark. The dataset consists of 93,003 triples with 40,943 entities and 11 relation types. To document the KBE models most likely to appear "in the wild", we begin by evaluating embeddings trained on WNISRR.
5.2 Expanded Training
As the WNISRR training set only contains portion of the synsets in WordNet, some words in the semantics benchmark cannot be learned. For BATS, quadruplets containing "missing words" are simply dropped, and the hits-at-10 score on the remaining dataset is reported. For SimLex-999 word pairs containing these "missing words" are assigned the median similarity score obtained from the other word pairs. Despite utilizing subsets of the benchmarks, it's crucial to note that all models are assessed on the exact same selected subsets, ensuring a consistent and fair comparison between them. In terms of Knowledge Base Completion, we use the evaluation protocol defined by Bordes et al. (2013), where the results of the filtered ranking metries are reported as the mean value of head node and tail node predictions.
A possible explanation for the unsatisfying perfor- mance of KBE models on semantic tasks is the heavily filtered subset that is used for training. For WN18RR, triples are "lost" in three ways: by only focusing on the 18 most frequent relations (Bordes et al., 2013), by removing relations with explicit inverses (Detteers et al., 2018), and by setting aside triples for the development and test splits. To investigate, we expand the WN18 subgraph to in- clude a more comprehensive set of synsets in Word- Net. This expansion occurs in two steps: (1) We extend WN18 by incorporating synsets with low in/out degrees while maintaining the same number of relation types. This expanded subgraph contains 112,195 nodes and 217.495 edges, which is referred to as WNI8A. (2) We further extend WNI8A with seven additional relations and their corresponding linked synsets, resulting in a dataset with 1 116,744 nodes and 363,593 edges, referred to as WN25.
Table 2 presents the performance of various KBE models trained on WNISRR, assessed based on a variety of intrinsic and extrinsic tasks. The results allow us to draw two key observations regarding the correlation of metrics. Firstly, a distinct correlation
Results on the extended datasets are shown in Table 3. For training, we retained hyperparame- ters from 2 without carrying out hyperparameter

---

WN18A
WN25
Similarity
Analogy
POS
NER
Similarity
Analogy
POS
NER
TransE
58.00
29.20
79.10
48.64
50.68
36.86
80.05
47.22
DistMult
28.32
5.80
68.58
40.39
21.82
6.35
68.57
39.57
FuncE
59.88
28.37
78.38
49.07
49.37
31.77
78.56
49.24
MuRP
31.33
15.50
74.93
40.89
23.22
28.21
76.34
38.68
rGAT
38.53
2.94
66.84
23.65
36.35
1.66
66.77
25.52
KBGAT 58.32
29.10
79.33
48.91
48.52
35.57
79.63
47.34
Wnet2vec 47.16
14.08
56.09
22.78
37.08
13.82
55.32
14.77
Table 3: KBE models' performance on the semantics benchmarks, with expanded training sets. For SimLex, all the word pairs are covered, while for BATS, 27,343 out of 98,000 quadruplets are covered. Best results are marked in bold, and the lowest ones underlined.
tuming. This decision was grounded the fact that both WN18A and WN25 have no validation set, so carrying out hyperparameter tuning would result in over-fitting. Notably, the results exhibit a similar pattern to those in Table 2: rGAT and DistMult consistently perform poorly across all se- mantic tasks, while models with scoring functions designed with linguistic motivations demonstrate better performance; other words, there is still an inverse correlation between link prediction and downstream results.
crete example of food. To accurately model the semantic information in WordNet, models must encode this hierarchical structure in their latent spaces. Unfortunately, models are typically not designed for this. We find that some of our lected models exhibit unexpected behavior when encoding hypernymy, and almost all of them fail properly capture the transitive property.
6.1 Model Behaviour Contradicts Thesaurus-Base Similarity Measures
It is important to note that some of the results in Table 3 not directly comparable to Table 2, they have different coverage on the benchmarks. This particularly evident in the word analogy task, where the ranking metric is strongly influ- enced by the total number of nodes in the graph. In- terestingly, the performance of rGAT and DistMult is less improved by the dataset extension compared to other KBE models these models exhibit sub- par performance word analogy tasks and show no improvement in the POS tagging benchmark. This suggests that the architecture, rather than e.g. a lack of training data, is the central problem for these models.
Consider an upward path in the hypernym hicrarchy of WordNet (e.g.. potato -> solanaceous vegetable produce + food - solid - matter - physi- cal entity). It is reasonable to expect that concepts closer to each other in the path would have higher similarity scores. This approach is commonly used in thesaurus-based similarity algorithms (Quillian, 1969). However, KBE models not trained exhibit this property. Table 4a presents the cosine similarity between potato and all of its hypernyms. We observe that TransE and FuncE strictly follow the thesaurus bassd similarity idca, with solana- ceous vegetable being the most similar to potato and physical entity being the least similar. In con- trast, DistMult and rGAT give similarity scores that not clearly correlated with the path length.
6 WordNet Potatoes and KBE Model Architectures: A Problematic Mash
The differing behaviour of models on this task reflect their differing motivation. As mentioned in Section 5.1, both TransE and FuncE. are the- oretically motivated, respectively by ideas from the geometry of word embeddings (Allen al., 2021) and from formal semantics (Emersson, 2018. 2020). DistMult and rGAT, on the other hand, are primarily motivated as being good models for link prediction. While they accomplish the latter goal
As a concrete demonstration of how KBE models fail to accurately represent semantics, we consider the case of hypernymy. WordNet encodes tree defined by the hyponymy and hypernymy relations, with concepts near the leaves representing more concrete forms of those occurring near the root. This is transitive relation: A potato is a concrete example of a vegetable, and therefore also a con-

---

Cosine Similarity with 'Potato'
Hypernym Prediction @1
TransE
DistMult
rGAT
TransE
DistMult
FuncE
MuRP
solanaceous veg.
0.7750
-0.0009
0.6927
1-hop
84.41
83,96
86.32
83,45
vegetable
0.5468
0.6390
0.5026
2-hop
2.21
10.08
76.65
9.73
produce
0.2723
-0.3153
0.3991
3-hop
0.81
4.44
47.5
4.03
food
0.1676
0.4571
0.3481
4-hop
0.58
0.41
10.2
1.56
solid
0,0623
-0.3725
0.1821
5-hop
0.56
0.12
3.76
0.53
matter 0,0601
0.3959
0.2130
6-hop
0.38
0.04
1.68
0.32
physical entity
0.0236
-..699
0.4192
7-hop
0.30
0.01
0.84
0.11
Cosine similarity between 'Potato' its hypernyms WordNet.
(b) Predicting multi-hop hypemyms. KBEs are trained and tested on WNISA.
Table 4: Investigation of KBEs' ability to model hyponymy and hypernymy relations.
WN18RR
Similarity
Analogy
POS tagging
NER
MRR
@ 10
@ 10
Spearman p
acc.
fl score
KBGAA--TraNBE
26.03
48.64
20.44
45.05
31.56
76.44
47.36
KBGAT-FuncF:
23.87
42.52
15.03
48.77
25.60
77.03
52.12
iGaT-TransE
22.43
48.03
17.31
46.76
28.60
76.92
48.55
iGAT-FuncE 14.45
39.17
9.56
47.89
30.60
77.22
51.92
Table 5: Results for GNNs with modified scoring functions, to compare against Table 2.
(e.g. terms of MRR on WN18RR, rGAT per- forms the best out of all the models we test), they in turn sacrifice the ability to model the underly- ing semantics of WordNet. This behaviour is also reflected through the comparatively poor perfor- mance of these models on our semantic tasks (see Tables 2 and 3).
fail to predict multi-hop hypernymy. FuncE, which is explicitly designed to model transitivity, is to much greater degree able represent hypernyms further away.
Drawing from these observations, we empha- size the significant potential inherent specialized KBE models tailored for specific kinds of data. We urge future model designers to meticulously con- sider the desired functionalities of their models, ensure that they able to achieve these function- alities in theory, and evaluate whether they achieve them in practice.
6.2 Modelling Transitivity
The hypernymy relation in WordNet is transitive: if (a, hypernym, b) and (b, hypernym, are valid triplets, then (a, hypernym, c) also valid. How- ever, almost none of the KBE models proposed are capable of modelling transitivity. For instance. TransE is trained to enforce that Us + D, & if (s, r, 0) is valid triplet. This method is deficient for modelling transitive relations, since (Ne + 2y se a (v) 01 v.) only imply (n, 2 v.)
6.3 Scoring Functions for GNN-encoders
GNN encoders, e.g. rGAT (Chen et al., 2021), are often designed to work with neural scoring func- tions. As these are designed to maximize link pre- diction performance, rather than capture the latent structure of the semantic space, their correspond- ing embeddings may not work well for downstream tasks. A possible remedy would be to pair these powerful encoders with linguistically motivated de- coders (c.g., TransE or FuncE).
if uy 2 Similarly, bilinear scoring functions and neural decoders do not easily allow the representa- tion transitivity.
Table 4b demonstrates the average accuracy for different models in predicting whether n-hopp hy- pemyyyy (e.g. predicting whether potato is veg- etable, whether it is produce, whether food, and so on). As can be seen, TransE and DistMult
We test this proposal Table pairing rGAT and KBGAT with TransE- and FuncE-decodens We find that, although performance increases for

---

Word Similarity
Word Analogy
Adj.
Noun
Verb
Deri.
Lexi.
Ency.
TransE
70.30
53.32
62.74
44.12
28.50
17.45
DistMult
37.80
27.07
28.32
13.97
2.84
1.39
FuncE
71.10
53.80
67.76
39.40
28.92
15.23
WN18A
MuRP
29.83
34.39
42.15
24.31
12.52
11,41
rGAT
44.04
35.99
46.91
0.26
3.88
5.01
KBGAT
70.45
52.98
63.70
43.78
28.52
16.70
Wnet2vec
64.40
43.13
46.88
16.51
15,40
9.63
TransE
55.45
49.66
47.22
47.40
38.86
19.87
DistMult
30.35
23.15
12.76
12.51
4.03
1.92
FuncE
52.64
50.99
47.09
44.95
34.82
15.92
WN25
MuRP
30.58
39.85
28.04
23.85
15.97
12.56
iGAT
31.17
40.26
29.88
0.27
2.64
2.18
KBGAT
54.52
48.23
46.81
45.99
38.21
18.60
Wnet2vec
41.67
40.94
32.51
22.88
18.29
9.76
Table 6: Results on SimLex999 and BATS, by category (part of speech and dedirssion//letoogggpppeeeyyyyyu Best results are marked bold, and the lowest ones are underlined.
semantic tasks, there is a corresponding drop in MRR; further, the addition of the GNN-encoder to TransE FuncE does not improve the decoder- only variant (see Table 2). This matches existing work pairing GNN encoders with translational scor- ing functions (Cai et al., 2019), which also failed to yield significant improvements over using the factorization model by itself. Indeed, the majority of GNN architectures proposed for link prediction use either bilinear (Schlichtkrull et al., 2018) or neural (Nathani et al., 2019; Chen et al., 2021) scoring functions. As we discuss in Section 6.1. such scoring functions do not represent the hier- archical structure of WordNet well. Our findings suggest that the development a GNN encoder which pairs well with linguistically motivated scor- ing functions - or a linguistically motivated scoring function which pairs well with GNNs - is highly desirable direction for future research.
emerges: models perform better on adjectives than on nouns or verbs. While present for both datasets, this trend is more pronounced for WNI8A than WN25, One possible interpretation that KBE models tend attribute high similarity scores concepts that are highly connected in the training data, i.e. to (potential) edges between vertices denser regions of WordNet. This leads to height- ened similarity scores between concepts that share strong associations, despite having less similarity; contradicting the measurement principles of Sim- Lex (Hill al., 2015b). This particularly promi- nent for noun synsets, as the majority of connec- tions WordNet occur between nouns. In con- trast, there are no connections between adjectives in WN18A, and only a few connections in WN25.
Analogy questions in BATS are classified into four groups according to the original paper: deriva- tional, lexicographical. encyclopedic, and inflec- tional. We exclude the inflectional group due to its limited coverage in WordNet. As per Table 6, KBE models tend to perform the best in the deriva- tional group. This outcome is unexpected, given that large number of analogy questions in the lexicographical group can be answered using rela- tions directly from WordNet (e.g., analogy ques- tions involving hypernym-hyponym part-whole, and meronym-holonym) contrast, derivational relations in WordNet are encoded in more general
7 Further Analysis: Subtasks Under the Spudlight
An interesting question is how uniform our find- ings are across various subtasks of our evaluation tasks. Conveniently, SimLex-999 and BATs allow for such analysis, respectively across POS-tag and question category. Detailed model results for these can be seen in Table 6.
On SimLex-999, a fairly consistent pattern

---

manner, utilizing single 'derivationally related' re- lation. Moreover, the performance of KBE models diverges from the trend observed distributional word embeddings, which typically underperform in the derivational group (Gladkova et al., 2016).
purposefully made simple rather than performant. Our intention was not to develop sfate-of-the- th--th--rtt systems for these tasks, but rather to analyse the embeddings produced by relational link prediction.
Acknowledgements
For both datasets, the ranking of the models by performance is broadly consistent between the data subsets. The impact of the additional relations in WN25 inconsistent, generally improving perfor- mance on BATS but decreasing on SimLex999.
We extend our deepest gratitude to Charles Chen for his pioneering undergraduate thesis, which laid groundwork for the FuncE model. We are also thankful to Tiago Pimentel, Rami Aly, and the anonymous reviewers, whose insightful feedback and constructive criticism was instrumental in fining our manuscript to its present form.
8 Conclusion
Our study has highlighted the substantial discrep- ancy between the performance of KBE models on the WNI8RR benchmark versus their performance in semantics and downstream tasks. Specifically, we have shown that KBE models, when designed absent of linguistic insights, are capable of deliv- ering high performance on link prediction metrics while also underperforming and demonstrating un- expected behaviors semantic analysis. Our find- ings demonstrate the deficiency of the WN18RR benchmark in evaluating specific semantic capabil- ities of KBE models and underscore the potential ad advantages of constructing models tailored for se- mantic tasks. Consequently, we advocate for the development of targeted benchmarks that assess specific semantic abilities of WordNet KBE mod- cls, which enable model designers calibrate and optimize their models alignment with precise semantic requisites.
References
Farahnaz Akrami, Lingbing Guo, Wei Hu, and Chengkai
Li. 2018. Re-evaluating embedding-basee based knowl- edge graph completion methods. In i Proceedings of the 27th ACM International Conference on Informa- tion and Knowledge Management, CIKM '18, page 1779-1788 New York, NY, USA. Association for Computing Machinery.
Farahnaz Akrami, Mohammed Samiul Saeef, Qingheng Zhang, Wei Hu, and Chengkai Li. 2020. Realistic re- evaluation of knowledge graph completion methods: An experimental study. In Proceedings of the 2020 ACM SIGMOD International Conference on Man- agement of Data, SIGMOD '20, page 1995-2010, New York, NY, USA. Association for Computing Machinery.
Carl Allen, Ivana Balazevic, and Timothy Hospedales. 2021. Interpreting knowledge graph relation repre- sentation from word embeddings. In Proceedings of the 9th International Conference on Learning Repre- sentations (ICLR).
Limitations
Our evaluation of the semantic capabilities of KBE models is limited in several ways. We focus on specific aspects of semantics, and two downstream tasks. The semantic tasks we have included are not perfect representations of semantic evaluation, and indeed do not always reflect downstream perfor- mance (Chiu et al., 2016). Further, POS tagging and NER are only two of many options for down- stream tasks requiring some semantic knowledge. As such, it is possible that KBE evaluation accu- rately captures other aspects of semantics. Addi- tionally, we note that our analysis is restricted to one ontology, WordNet, which we chose because it is often used as a benchmark for KBE models. A different ontology, e.g. BabelNet (Navigli and Ponzetto, 2010), may produce different results.
Ivana Balažević, Carl Allen, and Timothy Hospedales. 2019. Multi-relational Poincaré graph embeddings. In Advances in Neural Information Processing Sys- tems, volume 30.
Yonatan Belinkov. 2022. Probing classifiers: Promises. shortcomings, and advances. Computational Linguis- rics, 48(1 1)2007221
Francis Bond and Ryan Foster. 2013. Linking and ex- tending an open multilingual Wordnet. In Proceed- ings of the 51 Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pages 1352-1362, Sofia, Bulgaria. Association for Computational Linguistics.
Antoine Bordes, Xavier Glorot, Jason Weston, and Yoshua Bengio. 2014. A semantic matching energy function for learning with multi-relational data: Ap- plication to word-sense disambiguation. Machine Learning, 94:233-259.
We further note that the models we use in this pa- per, i.e. for semantic tasks and for POS tagging, are

---

Antoine Bordes, Nicolas Usumier, Alberto Gareia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embedddings for modeling multi- relational data. In Advances Neural Information Processing Systems 26, volume 26.
Katrin Erk. 2009a. Representing words as regions in vector space. In Proceedings of the Thirteenth Con- ference on Computational Natural Language Learn- ing (CoNLL-2009), pages 57-65, Boulder, Colorado. Association for Computational Linguistics.
Ling Cai, Bo Yan, Gengehen Mai, Krzysztof Janowicz, and Rui Zhu. 2019. Transgen: Coupling transforma- tion assumptions with graph convolutional networks for link prediction. In Proceedings of the 10th Inter- national Conference on Knowledge Capture, K-CAP '19. page 131-138 New York, NY, USA. Association for Computing Machinery.
Katrin Erk. 2009b. Supporting inferences in semantic space: representing words as regions. In Proceed- ings of the Eight International Conference on Com- putational Semantics, pages 104-115, Tilburg, The Netherlands. Association for Computational Linguis- tics.
Manaal Faruqui and Chris Dyer. 2015. Non- distributional word vector representations. In Pro- ceedings of the 53rd Annual Meeting of the Asso- ciation for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 464- 469, Beijing, China, Association for Computational Linguistics.
Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang. Rahul Gupta, Varun Kumar, Jwala Dhamala, and Aram Galstyan. 2022. On the intrinsic and extrinsic fairness evaluation metrics for contextualized lan- guage representations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 561 561-570, Dublin, Ireland. Association for Computational Lin- guistics.
Mohamed H Gad-Elrab, Daria Stepanova, Trung-Kieh Tran, Heike Adel, and Gerhard Weikum. 2020. Ex- Cut: Explainable embedding- based clustering over knowledge graphs. In The Semantic Web-ISWC 2020: 19th International Semantic Web Conference, Athens, Greece, November 2-6, 2020, Proceedings, Purt I, pages 218-237. Springer.
Meiqi Chen, Yuan Zhang, Xiaoyu Kou, Yuntao Li, and Yan Zhang. 2021. r-GAT: Relational graph attention network for multi-relational graphs. ArXiv preprint 2109.05922.
Tianxiao Chen. 2021. Modelling lexical relations using semantic functions, Undergraduate thesis, University of Cambridge.
Anna Gladkova, Aleksandr Drozd, and Satoshi Mat- suoka. 2016. Analogy- based detection morpholog- ical and semantic relations with word embeddings: what works and what doesn't 't In Proceedings of the NAACL Student Research Workshop. pages 8-15, San Diego, California. Association for Computational Linguisties.
Xiaojun Chen, Shengbin Jia, and Yang Xiang. 2020. A review: Knowledge reasoming over knowledge graph. Expert Systems with Applications, 141:1 12948.
Billy Chiu, Anna Korhonen, and Sampo Pyysalo. 2016. Intrinsic evaluation of word vectors fails to predict ex- trinsic performance. In Proceedings of the 1st Work- shop on Evaluating Vector-Space Representations for NLP. pages 1-6, Berlin, Germany. Association for Computational Linguistics.
Seraphina Goldfarb-Tarrant, Rebecca Marchant, Ri- cardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metries do not correlate with application bias. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 1926-1940, Online. Association for Computational Linguistics.
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp. and Sebastian Riedel. 2018 Convolutional 2D knowledge graph embeddings. In Proceedings of the 32nd AAAI conference on Artificial Intelligence, volume 32.
Tiansi Dong, Chrisitan Bauckhage, Hailong Jin, Juanzi Li, Olaf Cremers, Daniel Speicher, Armin B Cremers, and Joerg Zimmermann. 2018 Imposing category trees onto word-embeddiings using a geometric con- struction. In Proceedings of the 6th International Conference on Learning Representations (ICLR).
Felix Hill, Roi Reichart, and Anna Korhonen. 2015a. SimLex-9999 Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguis- rics, 41(4):665-69)
Felix Hill, Roi Reichart, and Anna Korhonen. 2015b. 9imlex-999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguis- tics, 41 (4):6655-655
Guy Emerson. 2020. What are the goals of distribu- tional semanties? In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pages 7436-7453, Online. Association for Computational Linguistics.
Nitisha Jain, Jan- -hn-Christoph Kalo, Wilf-Tilo Balke, and Ralf Krestel. 2021. Do embeddings actually capture knowledge graph semantics? In The Semantic Web: 18th International Conference, ESWC 2021, Virtual Event, June 6-10, 2021, Proceedings 18, pages 143- 159. Springer.
Guy Edward Toh Emerson. 2018. Functional Distribu- tional Semantics: Learning Linguistically Informed Representations from Precisely Annotated Corpus. Ph.D. thesis, University of Cambridge.

---

Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. 2017. Knowledge base completion: Baselines strike back. In Proceedings of the 2nd Workshop Repre- sentation Learning for NLP, pages 69-74, Vancouver, Canada. Association for Computational Linguistics.
Yuanyuan Qiu, Hongzheng Li, Shen Li, Yingdi Jiang. Renfen Hu, and Lijiao Yang. 2018. Revisiting cor- relations between intrinsic and extrinsic evaluations of word embeddings. In Proceedings of Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annolated Big Data, pages 209-221. Springer.
Andrey Kutuzov, Mohammad Dorgham, Oleksiy Oliynyk, Chris Biemann, and Alexander Panchenko. 2019. Learning graph embeddings from WordNet- based similarity measures. In Proceedings of the Eighth Joint Conference on Lexical and Computa- tional Semanties (*SEM 2019), pages 125-135, Min- neapolis, Minnesota, Association for Computational Linguistics.
M Ross Quillian. 1969. The teachable language com- prehender: A simulation program and theory of lan- guage. Communications of the ACM. 12(8):459-476.
Wiem Ben Rim, Carolin Lawrence, Kiril Gashteovski, Mathias Niepert, and Naoaki Okazaki. 2021. Behav- ioral testing of knowledge graph embedding models for link prediction. In Proceedings of the 3rd Con- ference on Automated Knowledge Base Construction (AKBC).
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor- pus of English: The Penn Treebank. Computational Linguistics, 19(2):313-33)
Damiel Ruffinelli, Samuel Broscheit, and Rainer Gemulla. 2020. You can teach an old dog new tricks! On training knowledge graph embeddings. In Proceedings of the 8th International Conference on Learning Representations (ICLR).
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word represen- tations in vector space. In Proceedings of the 1st International Conference on Learning Representa- tions (ICLR), Workshop Track.
Chakaveh Saedi, António Branco, João António Ro- drigues, and João Silva. 2018 WordNet embeddings. In Proceedings of the Third Workshop on Represen- tation Learning for NLP, pages 121-131, Melbourne, Australia. Association for Computational Linguistics.
George A Miller. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11) 1):39- 41.
Bonan Min, Ralph Grishman, Li Wan, Chang Wang. and David Gondek. 2013. Distant supervision for re- lation extraction with an incomplete knowledge base. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 777-782, Atlanta, Georgia. Association for Computational Linguistics.
Michael Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling, 2018. Modeling relational data with graph convolu- tional networks. In Proceedings of the 15th European Semantic Web Conference, pages 593-607. Springer.
Zequn Sun, Qingheng Zhang, Wei Hu, Chengming Wang. Muhao Chen, Farahnaz Akrami, and Chengkai Li. 2020. A benchmarking study of embedding- entity alignment for knowledge graphs. In Proceed- ings of the VLDB Endowment, volume 13, pages 2326-2340. VL.B Endowment.
Changsung Moon, Paul Jones, and Nagiza F Samatova. 2017. Learning entity type embeddings for knowl- edge graph completion. In Proceedings the 2017 ACM on conference on information and knowledge management. pages 2215-2218.
Sudhanshu Tiwari, Iti Bansal, and Carlos R. Rivero. 2021. Revisiting the evaluation protocol of knowl- edge graph completion methods for link prediction. In Proceedings of the Web Conference 2021. WWW '21, page 809-820, New York. NY, USA. Association for Computing Machinery.
Deepak Nathani, Jatin Chauban, Charu Sharma, and Manohar Kaul. 2019. Learning attention-bast based beddings for relation prediction in knowledge graphs. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational L Linguistics, pages 4710- 4723, Florence, Italy, Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-indes indeeedeed named entity recognition. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
Roberto Navigli and Simone Paolo Ponzetto. 2010. Ba- belNet: Building very large multilingual semantic network. In Proceedings of the 48th Annual Meet- ing of the Association for Computational Linguistics, pages 21 216-225, Uppsala. Sweden. Association for Computational Linguistics.
François Torregrossa, Vincent Claveau, Nihel Kooli, Guillaume Gravier, and Robin Allesiardo. 2020. On the correlation of word embedding evaluation metries. In Proceedings of the Twelfth Language Resowcees and Evaluation Conference, pages 4789-4797, Mar- seille, France. European Language Resources Asso- ciation.
Maximillian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In Advances in Neural Information Processing Sys- lems 30 (NIPS), pages 6338-6347.

---

Kristina Toutanova and Danqi Chen, 2015. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd Workshop on Continuous Vector Space Models and their Composi- lionality, pages 57-66, Beijing, China. Association for Computational Linguistics.
A Relation Categories
Relation Categories
hyponym
instance hyponym hypernym
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-embeddings of images and language. Proceedings of the 4th International Conference Learning Representations (ICLR).
Hypernym-based
instance hypernym
verb group similar to also see derivationally related form
Yanjie Wang. Daniel Ruffinelli, Rainer Gemulla, Samuel Broscheit, and Christian Meilicke. 2019. On evaluating embedding models for knowledge hase completion. In Proceedings of the 4th Workshop Representation Learning for (RepLANLP-2019, pages 104-112, Florence, Italy. Association for Com- putational Linguistics.
Synonym-based
member meronym
has part
member of domain region member of domain usage synset domain topic of member holonym part of synset domain region of
Bishan Yang. Wen-tau Yih. Xiaodong He, Jianfeng Gao, and Li Deng, 2015, Embedding entities and relations for learning and inference in knowledge bases. In Proceedings of the 3rd International Conference on Learning Representations (ICLR).
Other
Lotfi A. Zadeh. 1965. Fuzzy sets. Information and Control, 8(3):338-335
synset domain usage of member of domain topic
Yu Zhao, Anxiang Zhang, Ruobing Xie, Kang Liu, and Xiaojie Wang. 2020. Connecting embeddings for knowledge graph entity typing. Proceedings of the 58th Annual Meeting of the Association for Com-
attribute
cause
entailment
putational Linguistics, pages 6419-6423 Online. As- sociation for Computational Linguistics.
Other (WN25)
pertainym
antonyms
substance holonym
substance meronym
Table 7: WordNet relations grouped into categories. The first 18 relations are used in WN18. The 11 relations used in WNISRR marked in bold.

---

B Hyper-parameters
learning algorithm
learning rate
batch size
dimension
epoch
weight decay
TransE
Adam
le-2
16384
200
400
le-2
DistMult
Adagrad
5c-1
8192
200
400
le-1
FuncE
Adam
le-2
8192
200
400
le-1
MuRP
RiemannianSGD
50
128
200
200
0
rGAT
Adam
le-3
4096
200
800
le-5
KBGAT
Adam
le-3
4086
200
3600
le-5
Wnet2vec
800
Table 8: Slyper-paramee choices for model trained on WNI8RR, WN18A, and WN25
C Benchmark Examples
Benchmark
Sample
SimLex999
Word Pair: (old, new), Similarity Score: 1.58
BATS
Analogy question: man
woman :: king
Answer: queen
PTB POS tagging
Sentence: The quick brown fox jumps over the lazy dog. Tags: DT JJ JJ NN VBZ IN DT JJ NN.
CoNLL-2003 NER
Sentence: Mike lives in London.
Tags: [Mike, B- B-PER] [lives, O] [in, O] [London, B-LOC]
Table 9; Examples from the used benchmarks


Reviews:
[{"review_id": "o55rG6lniQ", "reviewer": "Reviewer_YVuz", "summary": "", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We are happy they found our paper comprehensive, interesting, and important!\n\n**Additional Downstream Task**\n\nOur primary intent with this paper was to investigate the correlation between link prediction and semantic/downstream tasks. As suggested by reviewer 2, NER tagging would be another option for evaluation. However, we expect embedding performance on NER and POS tagging to correlate with each other, as found in previous work (for example: POS tags are highly informative features for NER tagging [1]; out of a set of 33 NLP tasks, there was a high level of transferability between POS tagging and NER [2]), and as such we would not expect to see substantial differences when using NER instead of POS tagging. With that said, we appreciate the suggestion, and agree that adding another task would provide further evidence. Other tasks could include sentiment analysis or text classification. \n\n[1] A Multi-task Approach for Named Entity Recognition in Social Media Data. Aguilar et al., WNUT, at ACL 2017. https://aclanthology.org/W17-4419/\n\n[2] Exploring and Predicting Transferability across NLP Tasks. Vu et al., EMNLP 2020. https://aclanthology.org/2020.emnlp-main.635/\n\n**Possible Solutions**\n\nWe include several quite general recommendations in the paper, i.e. on lines 100-102: *“evaluation must target desired functionality, and models must be theoretically capable of that functionality”*. If we were to theorise more specific suggestions, they would fall along these lines as well. For example, section 6.2 looks specifically at hypernymy – we can expand this discussion in the camera-ready.\n\nFor evaluation, our primary suggested solution is for researchers to not use link prediction benchmarks to estimate performance on semantic or downstream tasks – instead, we recommend directly evaluating on the target task. That way, low correlation between metrics is not an issue. We emphasise that link prediction might still be a useful training task, though.\n\nFor modelling, we recommend thinking carefully about what functionalities a system should be capable of, and what the “shape” of the target task is. In our experiments, we showed that FuncE – a relatively simple model that incorporates *different, hand-crafted* scoring functions for different relation types performs very well on semantic and downstream tasks. We see this as a fruitful direction: specialist models designed for the target task.\n\nWe will expand our discussion of both of these points in the camera ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper evaluates a lot of state-of-the-art KBE models, which show a clear negative correlation between different tasks.\n- The findings in the paper are very interesting and important to further research and real-life application.\n- The authors show a clear and insightful discussion of the main results and findings, which allows to clearly show the problem.  \n", "reasons_to_reject": "- Evaluation of the downstream task is very limited, it could be useful to present a more challenging downstream task, than POS tagging.\n- The authors do not discuss how to further solve this issue.\n", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "The paper provides a comprehensive study of a large amount of KBE models on link prediction and their ability to perform in semantic analysis tasks. The authors show a negative correlation between performance on the WN18RR benchmark of KBE models with two semantic analysis tasks, and one downstream task. These results demonstrate the need for careful design of both evaluation benchmarks and model architectures.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tsJcAYgTTj", "reviewer": "Reviewer_P9Hk", "summary": "", "questions": "- More of a philosophical question regarding the very root motivation of your paper: Why would we use KBE models for word similarity (to some extent I can see why) or analogy and POS tagging? should we just use these types of model for link prediction? What motivated you to investigate this knowledge transfer in the first place? Maybe models specifically designed to predict hypernymy would be better suited for the task?\n- This got me questioned about the usefulness of KBE models if their only usage is for link prediction. Maybe that's their only utility, in the end.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback. We are happy they found our paper to be well-formulated, thorough, and easy to read!\n\n**Additional Downstream Task**\n\nOur primary intent with this paper was to investigate the correlation between link prediction and semantic/downstream tasks. We expect embedding performance on NER and POS tagging to correlate with each other, as found in previous work (for example: POS tags are highly informative features for NER tagging [1]; out of a set of 33 NLP tasks, there was a high level of transferability between POS tagging and NER [2]), and as such we would not expect to see substantial differences when using NER instead of POS tagging. With that said, we appreciate the suggestion, and agree that adding another task would provide further evidence. Other tasks could include sentiment analysis or text classification. \n\n[1] A Multi-task Approach for Named Entity Recognition in Social Media Data. Aguilar et al., WNUT, at ACL 2017. https://aclanthology.org/W17-4419/\n\n[2] Exploring and Predicting Transferability across NLP Tasks. Vu et al., EMNLP 2020. https://aclanthology.org/2020.emnlp-main.635/\n\n**KBE Models for Semantics**\n\nWe appreciate this excellent question, and we hope that our paper will prompt more researchers to think about these underlying issues. KBE models are often motivated as representations of the underlying structure of the knowledge base. If the knowledge base is WordNet, a KBE model should therefore encode the structure of WordNet – things like hypernymy, meronymy, and so on. The intuition is that this knowledge should be helpful for more general semantic modelling. These are implicit assumptions; our aim is to test them. \n\nOn a more practical level, KBEs are used in many applications. We mention a few in the introduction, on lines 27-30: entity typing (Moon et al.,027 2017; Zhao et al., 2020), entity alignment (Sun et al., 2020), rule mining (Yang et al., 2015; Chen et al., 2020), and conceptual clustering (Gad-Elrab et al., 2020).\n\nOur findings suggest that the usefulness of KBEs beyond link prediction is difficult to estimate based on link prediction performance. This doesn’t mean that KBE models shouldn’t be used on other tasks, and in fact even our worst-performing models still perform better than chance (and Wnet2vec, which is not trained as a link prediction model).  However, other metrics are necessary to select the best KBE models for semantic and downstream tasks, and the past years of selecting models by SOTA performance on KBE benchmarks would not have revealed them!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Thorough analysis of KBE pitfalls with respect to semantic and downstream evaluation which does not correlate with their explicit training objective i.e. link prediction.\n- The experiments conducted are well sounded, and every hypothesis/assumption made by the authors is backed by well-formulated arguments.\n- This paper compares several KBE models which strengthened their claim", "reasons_to_reject": "The only weakness I see is that there's only one downstream task i.e. POS tagging. Would NER be a bit more interesting to evaluate?", "questions_to_authors": "- More of a philosophical question regarding the very root motivation of your paper: Why would we use KBE models for word similarity (to some extent I can see why) or analogy and POS tagging? should we just use these types of model for link prediction? What motivated you to investigate this knowledge transfer in the first place? Maybe models specifically designed to predict hypernymy would be better suited for the task?\n- This got me questioned about the usefulness of KBE models if their only usage is for link prediction. Maybe that's their only utility, in the end.", "paper_topic_and_main_contributions": "The paper studies the pitfalls of Knowledge Base Embeddings (KBE) learned representations in various settings;\n- the actual link prediction task, on which these models are trained to do\n- word semantic similarity\n- analogy\n- and part of speech tagging (POS).\n\nThe authors show that link prediction performance does not correlate with semantic performance or the downstream task POS.\nTheir experiments focus on hierarchical relations between different WordNet entities (such as potato -> vegetable -> food).\nIt is clear, from the authors' conclusion, that link prediction performance should not be used as a proxy to assess the downstream performance of KBE.\n\nI would like to add that this paper is as thorough as self-contained, and is really easy to read and understand.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Page 1 Table 1: H@1 is unfamiliar (at least to me). I would have expected P@1 or P@10. It's only at page 5 line 362 that I understood what H@1 meant.\n- Line 353: behcnmark -> benchmark\n- Would that be possible to differentiate the models based on a scoring function from the other ones in the tables? That would improve reading/understanding the results\n- Table 6: Switch second header of word similarity and word analogy (word sim -> spearman, word analogy -> @10)\n- Table 6: Bring table 2 in table 6 so we don't need to do back and forth constently\n- Even though I know BATS and SimLex999, I was sometimes wondering what were they coming from when the authors were analyzing the results. would there be a way to harmonize Word Similarity == SimLex99 and Word Analogy == BATS"}}, {"review_id": "FO40Cpt8fT", "reviewer": "Reviewer_UJC6", "summary": "", "questions": "Question A: The maths for reaching 99,200 questions in BATS should be clarified, starting with the number of entries per sub-category and how they are combined. Is this number still true after excluding the inflection category? Also, does it make sense to include the derivational category when testing semantics?\n\nQuestion B: Why are the reported results computed on a subset of the datasets? If it is due to coverage, is coverage the same for all methods? How does this affect the conclusions?\n\nQuestion C: Is it ok to merge different word senses of the same word in the same node and simply averaging the different sense embeddings?\n\nQuestion D: What were the hyperparameters for the results in tables 2 and 3? Are they not important?\n\nQuestion E: Could you provide more information on pairing the GNN encoders with the linguistically-motivated decoders?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their very positive comments. We are delighted to hear that they find our premise uniquely interesting among NLP papers – we appreciate the support!\n\n**Organisation & Clarity**\n\nWe thank the reviewer for pointing this out. In the camera ready, we will include an example from each dataset to better illustrate the tasks we evaluate on.\n\n**Subsets**\n\nThe models we compare in this paper are all trained on (various subsets of) WordNet. To minimise noise in our evaluation, we evaluated on the subsets of SimLex and BATS covered by WordNet. Coverage, training, and evaluation was the same for all methods (trained on a WordNet fragment, evaluated on the part of SimLex/BATS covered by WordNet). \n\nUsing the entire evaluation datasets would not change our findings – the datasets would just be padded with examples where all methods (being trained on WordNet) would perform at chance; hence, added noise. We will clarify this in the camera ready.\n\nRegarding BATS, 99,200 is the number of examples in the original dataset (Gladkova et al., 2016). After filtering out examples not included in the training set, we are left with 17,109 (WN18RR training set, see caption for Table 2) or 27,343 (full training set, see caption for Table 3). We will also make this clearer in the camera ready, moving the relevant details into the main text of Section 4.2. As WordNet uses lemmas rather than inflected wordforms, the inflected category is almost entirely filtered out. Many derivationally related words are included in WordNet, even if the morphological structure is not indicated. Derivationally related words have semantic relationships, and as seen in Table 7, WordNet-trained models generally perform better on this category than on the other BATS categories.\n\n**Merging Senses**\n\nWe agree that merging all senses by averaging sense embeddings into a single word embedding would not be optimal, if the goal was performance. Probabilistically, this strategy can be seen as assuming a uniform distribution (i.e. equal likelihood) over all senses of a word, and marginalising out the senses. Other distributions might have led to higher numbers – however, as long as we use the *same* marginalisation for all systems, this would not impact the correlations between results on KBE datasets and semantic or downstream tasks.\n\n**Hyperparameters**\n\nWe thank the reviewer for spotting this – we will include all hyperparameters in the camera-ready in an appendix. We are also making our source code available, so that all experiments will be reproducible. The important point about using the same hyperparameters between Tables 2 and 3 is methodological: for Table 2, the WN18RR dataset is split between training, validation, and test, and so hyperparameters can be tuned on the validation set; for Table 3, the training set is expanded to include all of WordNet so there is no validation set. It would be methodologically problematic to tune hyperparameters on the evaluation datasets in Table 3, and so we used the same hyperparameter values as in Table 2. We can clarify this methodological point in the camera-ready.\n\n**Linguistically Motivated GNN Decoders**\n\nThe GNN encoders we experimented with – KBGAT, rGAT – did not, from our findings, function well when using e.g. FuncE or TransE as the decoder. As the decoder (or scoring function) appears to limit quite heavily what performance can be expected, and what phenomena can be modelled, this is a serious limitation of GNN-based KBE models. In order to use GNNs to model semantics (at least in the form of KBE’s of WordNet), this limitation would need to be overcome. This would likely involve an investigation of why some decoders (DistMult) work so much better with GNNs than others (TransE), and potentially the development of new GNNs or decoders that solve the issue. These experiments, while interesting, are a bit outside the scope of our paper – but, we think, a very fruitful direction for future research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper has a very interesting premise, rare in papers that we see published nowadays, followed by an experimentation designed for confirming the previous.", "reasons_to_reject": "Here and there, the paper could be better organised and clearer. For instance, it would be easier to follow if it included a better description of the tasks and datasets, also with examples of their content.\n\nIt is also not clear why the reported results are on subsets of the datasets, and how this impacts the conclusions.\n\nAs the authors mention in the Limitations section, some conclusions can be limited by the number of models and tasks tackled.", "questions_to_authors": "Question A: The maths for reaching 99,200 questions in BATS should be clarified, starting with the number of entries per sub-category and how they are combined. Is this number still true after excluding the inflection category? Also, does it make sense to include the derivational category when testing semantics?\n\nQuestion B: Why are the reported results computed on a subset of the datasets? If it is due to coverage, is coverage the same for all methods? How does this affect the conclusions?\n\nQuestion C: Is it ok to merge different word senses of the same word in the same node and simply averaging the different sense embeddings?\n\nQuestion D: What were the hyperparameters for the results in tables 2 and 3? Are they not important?\n\nQuestion E: Could you provide more information on pairing the GNN encoders with the linguistically-motivated decoders?", "paper_topic_and_main_contributions": "The paper is on a limitation of knowledge graph embeddings: while performance in tasks like link prediction can be good, concepts often lose their semantic properties. \nThis is confirmed by experiments performed in a link prediction dataset and in two \"semantics\" tasks: similarity and analogy.\nTop performances in WN18RR, a link prediction dataset, are obtained by embeddings that perform poorly in the semantic tasks. On the other hand, FuncE, an introduced method that uses different scoring functions for each relation, performs better in the semantic tasks and not so well in link prediction.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Table 1 appears much earlier than mentioned in text, and it mentions Table 2, which is even further ahead, which makes their interpretation difficult.\n\nThe structure of the paper is not provided in the Introduction.\n\nGNN is used before appearing in full.\nNames of the computed metrics are never in full.\n\nWN18 is first mentioned without its contents being described."}}]

Author rebuttal:

Meta-review: The paper tackles the common approach to evaluating of Knowledge-Base Embeddings (KBE) on link prediction task. It shows, comparing several KBE models that performance in this task does not correlate with performance in the downstream task (POS tagging in this case).
All three reviewers agreed that the paper is sound and interesting. The main objection was that the evaluation is performed using only one downstream task, namely POS tagging. During the rebuttal period the authors conducted additional experiments and showed that they support the main claim of the paper. I think these results make the paper stronger, and should be reported in the final version.
Other objections concerned main assumptions and positioning of the paper. I found the authors' responses convincing, and they probably should be reproduced on the extra page.
Decision: Accept-Main