A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems
Authors: Hannah Bast
Matthias Hertel
Natalie Prange
Venue: EMNLP 2023 Main (2023)
Keywords: entity linking
entity linking evaluation
entity linking benchmarks
TL;DR: We provide a fair and in-depth evaluation of a variety of existing end-to-end entity linkers and create two new benchmarks to address problems in existing entity linking benchmarks and to provide a basis for fairer comparison.

Abstract: Existing evaluations of entity linking systems often say little about how the system is going to perform for a particular application. There are two fundamental reasons for this.
One is that many evaluations only use aggregate measures (like precision, recall, and F1 score), without a detailed error analysis or a closer look at the results.
The other is that all of the widely used benchmarks have strong biases and artifacts, in particular:
a strong focus on named entities, an unclear or missing specification of what else counts as an entity mention,
poor handling of ambiguities, and an over- or underrepresentation of certain kinds of entities.

We provide a more meaningful and fair in-depth evaluation of a variety of existing end-to-end entity linkers.
We characterize their strengths and weaknesses and also report on reproducibility aspects.
The detailed results of our evaluation can be inspected under https://elevant.cs.uni-freiburg.de/emnlp2023.
Our evaluation is based on several widely used benchmarks, which exhibit the problems mentioned above to various degrees,
as well as on two new benchmarks, which address the problems mentioned above.
The new benchmarks can be found under https://github.com/ad-freiburg/fair-entity-linking-benchmarks.

A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems
Hannah Bast 14 and Matthias Hertel 12+ and Natalie Prange
Uniierriiyy of Freiburg, Department of Computer Science, Germany Karlsruhe Institute of Technology, Institute for Automation and Applied Informatics, Germany (bast, prange } êcs. uni cei-ffrebburg...
matthias bertellkit.
Abstract
Here is example sentence, with the entity men- tions underlined and the corresponding Wikidata ID in square brackets (and clickable the PDF): American [Q30] athlete Whittington [Q21066526] failed to appear in the 2013-14 season [Q16192072] due to a torn ACL [Q18912826).
Existing evaluations of entity linking systems often say little about how the system is going to perform for particular application. There are two fundamental reasons for this. One is that many evaluations only use aggregate measures (like precision, recall, and Fl score), without a detailed error analysis or a closer look the results. The other is that all of the widely used benchmarks have strong biases and artifacts, in particular: strong focus named entities, unclear missing specification of what else counts as an entity mention, poor handling of ambiguities, and an or underrepresenta- tion of certain kinds of entities.
For research purposes, the problem often split in two parts: entity recognition (ER; identifying the entity mentions) and entiry disambiguation (ED; identifying the correct entity for a mention). practical applications, the two problems almost al- ways occur together. In this paper, we consider the combined problem, calling it entity linking (EL)).
1.1 Problems with existing evaluations
We provide a more meaningful and fair in-depth evaluation of variety of existing end-to-end entity linkers. We characterize their strengths and weaknesses and also report on reproducibility aspects. The detailed results of our evaluation can be inspected under wetpraattessui-fiibbuueddddeeeee
There is a huge body of research entity linking and many systems exist. They usually come with experimental evaluation and comparison other systems. However, these evaluations often say little about how the system will perform in practice, for a particular application. We see the following two fundamental reasons for this.
Our evaluation is based on several widely used benchmarks, which exhibit the problems mentioned above to various degrees, as well as two new benchmarks, which address the problems mentioned above. The new benchmarks can be found under https://github.e http://ggthuu..oom/add..m. cow/aoooooeeeeeeeeeeeeeee freiburg/fair-ent entiity- linking-benchmarks.
Coarse evaluation metrics. Most existing evalua- tions compare systems with respect to their preci- sion, recall, and Fl score; we call these aggregate measures in the following. In particular, the popu- lar and widely used GERBIL platform (Röder et al., 2018) 8) supports only comparisons with respect to (variants of) these measures." 2 What is often miss- ing is a detailed error analysis that compares the linkers along meaningful error categories. This often results in linkers that perform well on the se- lected benchmarks (critically discussed in the next paragraphs), but not in other applications. On top of that, we also had considerable problems with just replicating the reported results.
1 Introduction
Entity linking a problem of fundamental impor- tance in all kinds of applications dealing with nat- ural language. The input is a text in natural lan- guage and a knowledge base of entities, each with a unique identifier, such as Wikipedia or Wikidata,
The task is to identify all sub-sequences in the text that refer to entity, call these entity mentions, and for each identified entity mention determine the entity from the knowledge base which it refers.
We deliberately do not refer these problems as NER, NED and NEL. We omit "N(amed)" because important aspect of our evaluation that we consider non-named entities as well.
"Author contribations are stated in Section 8. M.H. is funded by the Helmholtz Assnciation's Initiative and Net- working Fund through Helmholtz AI.
GGEBBIL. does support separate evaluation of ER and ED, but again with respect to these aggregate measures only.

---

Benchmark artifacts and biases. The following four artifacts and biases are frequent in existing benchmarks. Linkers can exploit these to achieve good results, especially regarding the aggregate measures discussed in the previous paragraph.
several papers in recent years on the meta-topic of a more meaningful evaluation of entity linking systems. We provide a succinct overview of this work in Section 2. However, we have not found any work that has tried to address all of the problems mentioned above. This is what we set out to do in this paper, by providing an in-depth comparison and evaluation of the currently best available entity linking systems existing benchmarks well on two new benchmarks that address the problems mentioned above.
First, all widely used benchmarks have strong focus on named entities, which in the English lan- guage are almost always capitalized and hence easy to recognize. However, many if not most entity- linking applications need to recognize more than just named entities, for example: professions ("ath- lete"), chemical elements ("gold"), diseases ("torn ligament"), genres ("burlesque"), etc.
1.2 Contributions
Second, when going beyond named entities, it is hard to define what counts as an entity mention. Existing benchmarks work around this problem in one of three ways: they contain almost exclusively named entitics, the decision was up to annotators without clear guidelines and without documenta- tion, it is expected from the evaluation that the entity mentions are fixed and only the disambigua- tion is analyzed. Note that it not an option call anything an entity that has an entry in a knowledge base like Wikipedia or Wikidata, because then al- most every word would become part of an entity mention.3
We provide an in-depth evaluation of a variety of ex- isting end-to-end entity linkers, on existing bench- marks as well as on two new benchmarks that we propose in this paper, in order to address the prob- lems pointed out in Section 1.1. More specifically:
We provide a detailed error analysis of these link- ers and characterize their strengths and weaknesses and how well the results from the respective publi- cations can be reproduced. See Table and Figure for an overview of our results, Table 4 and Section for the details, and Section for a concluding summary of the main takeaways. Detailed individ- ual results of our evaluation can be inspected under https://elevant.cs.uu uni- ceai-freiburggdee detpoo/rgdeemmmm2222....3.
Many entity mentions are ambiguous, making it debatable which entity they should be linked to. A typical example the mention American in the sentence above. There is no Wikipedia or Wikidata entry for the property of being American. Instead, there are three closely related entities: the coun- try [Q30]. the language [Q7976], and the citizens [Q846570J. Most existing benchmarks resort to one choice, which punishes systems that make an alternative (but maybe equally meaningful) choice. Several benchmarks have a strong bias towards certain kinds of entities. A prominent example is the widely used AIDA-CoNLL benchmark (Hoffart et al., 2011). It contains many sports articles with many entities of the form France, where the cor- rect entity is the respective sports team and not the country. This invites overfitting, In particular, learning-basee systems are quick to pick up such signals, and even simple baselines can be tuned rel- atively easily to perform well on such benchmarks.
We describe the most widely used existing benchmarks and reveal several artifacts and bi- ases that invite overfitting; see Section 4. We create two new benchmarks that address these problems; see Scction 5. These benchmarks can be found under betry-linking-benchhaaa
2 Related Work
Ling et al. (2015) analyze differences between ver- sions of the entity linking problem that are being tackled by different state-of-the-ard systems. They compare popular entity linking benchmarks and briefly discuss inconsistent annotation guidelines. However, they do not present improved bench- marks. They develop a modular system to analyze how different aspects of an entity linking system af- feet performance. They manually organize linking errors made by this system into six classes to gain better understanding of where linking errors oc- cur. We use the more fine-grained error categories introduced by Bast et al. (2022) for a thorough comparison between linking systems.
We are not the first to recognize these problems or try to address them. In fact, there have been
*For example, there is Wikipedia article for the grammati- cal article the or the general concepts of failure or appearance, all used in our example sentence.

---

System
Overall
ER F1
Disamb. accuracy
Strengths and Weaknesses
Repro- ducibility
F1
ReFinED
73.3%
82.7%
89.2%
very good overall results; particularly strong on metonyms
good
REL
67.7%
82.3%
83.0%
very high ER Fl; often falsely links NIL. mentions
very good
GENRE
64.6%
74.2%
87.4%
sacritices ER recall for high disambiguation accuracy
medioere
Ambiverse
59.0%
76.2%
78.3%
good partial names; detected spans often too short
problematic
Neural 50.6%
73.6%
68.7%
good on demonyms; struggles with partial names
mediocre
Baseline 46.3%
74.0%
63.8%
predicts entity with highest prior probability; ignores context high disambiguation accuracy; poor ER
TagMe
43.0%
54.2%
80.7%
Table 1: Overview of the results of the evaluation. Scores are given as unweighted average over all five benchmarks (that is, the score each benchmark contributes equally to the average, and independent of the number of mentions in that benchmark).
Rosales-Ménder et al. (2019) also aim for fairer comparison between entity linking systems. They create a questionnaire to examine the degree of consensus about certain annotation decisions in the EL community. Based on the results of their questionnaire they create fine- fine-grained annotation scheme and re-annotate three existing benchmarks accordingly. They add new annotations capture as many potential links as possible. Additionally, they annotate some mentions with multiple alter- natives. They define two annotation modes, strict and relaxed, where the former includes only named entities and the latter includes all entities that can be linked to Wikipedia. Their approach is more extreme than in several respects: their relaxed mode contains very many annotations, (because of that) they consider only smaller benchmarks, and their error categories are very fine-grained. Further- more, they evaluate only older linkers.
(2) entity, surface form and mention characteristics and (3) mention annotation characteristics. They suggest to document decisions that are being made while creating the benchmark, which includes an- notation guidelines. Apart from that, they do not provide guidelines suggestions that target the annotation process.
Brasoveanu et al. (2018) argue that in in-tepth qualitative analysis of entity linking errors is nec- essary in order to efficiently improve entity link- ing systems. They categorize EL errors into five categories: knowledge base errors, dataset errors. annotator errors, NIL clustering errors and evalu- ation errors. They select four systems and three benchmarks and manually classify errors into these categories. Their evaluation is very short, and their main result is that most errors annotator errors.
Ortmann (2022) raises the issue of double penal- ties for labeling or boundary errors when comput- ing recall, precision and F1 score in the general context of evaluating labeled spans. Namely, an in- correct label or an incorrect span boundary counts as both a false positive and a false negative while, e.g., a prediction that does not overlap with any ground truth annotation counts as only one false positive even though it is arguably more wrong. Ortmann introduces a new way of computing preci- sion, recall and Fl score where such errors do not count double. We use the standard precision, recall and Fl score for our evaluation, but complemented by fine-grained error categories that show the effect of such errors on the overall score.4
Tha et al. (2017) identify inconsistencies between EL benchmarks and define a set of common anno- tation rules. They derive a taxonomy of common annotation errors and propose semi-automatic tool for identifying these errors in existing bench- marks. They then create improved versions of cur- rent benchmarks and evaluate the effects of their im- provements with 10 different ER and EL systems. However, their annotation rules are made without properly addressing the disagreement about them in the entity linking community. For our benchmark generation, we instead opt to allow multiple alter- native annotations in cases where good argument can be made for any of these linking decisions.
Van Erp et al. (2016) analyze six current entity linking benchmarks and derive suggestions for how to create better benchmarks. They examine dif- ferent benchmark aspects: (1) the document type
"This is in line with our philosophy that single overall score should be taken with grain of salt anyway and that one needs to look at the results more closely to determine strengths and weaknesses of a system.

---

ReFinED
REL
GENRE
Ambiverse
Neural EL
Baseline
TagMe
Overall
ER F1
Disambiguation Accuracy
100
90
90
90
80
80
80
70
70
60
50
50
50
40
40
40
30
30
ANDA-CONLL
KORESO
MSNBC
News-Fair Fais
Wiki Fair
AIDA-CONLL
KORESO
MSNBC
News Fair
Wik-Fair
AIDA-CoNLL
KORESO
MSNBC
News: Fair
Wik-Faair
Figure 1: Overall results of each system on each benchmark; see Table 4 for more fine- aieeggrined results.
3 Metrics
eral mention categories that are difficult to disam- biguate, by only considering ground truth mentions with specific properties. The following categories are analyzed. Demonym: the mention appears in list of demonyms (e-g., German)." Metonymy: the most popular candidate is a location but the ground truth entity is not a location. Partial name: the mention is a part of the ground truth entity's s name but not the full name. Rare: the most popular candidate for the mention is not the ground truth entity. Statisties of the frequencies of these cate- gories across the benchmarks given in Table 3. We also report the disambiguation error rate, which simply one minus the disambiguation accuracy.
We report micro precision, recall and F1 scores, both for the overall EL task and for the ER subtask. Details for how these measures computed are provided in Section A. Additionally, we use the fine-grained error metrics provided by the evalua- tion tool ELEVANT (Bast et al., 2022) to analyze the strengths and weaknesses of the evaluated link- ers in detail:
ER false negatives The following metrics ana- lyze special cases of ER false negatives. Lower- cased: the number of lowercased mentions that not detected. Partially included: the number of mentions where only a part of the mention linked to some entity.
4 Critical review of existing benchmarks
We analyze the performance the entity linking systems included our evaluation on three of the most widely used existing benchmarks It turns out that each of them has its own quirks and biases, as discussed in the following sections. Statistics the annotated entity mentions for each bench- mark are provided in Table 3. See Section A.2 for other popular EL benchmarks that we have ex- cluded from our evaluation due to problems in their design.
ER false positives ER false positives are pre- dicted mentions that do not correspond to ground truth mention or that correspond to ground truth mention annotated with NIL. The following met- rics analyze special cases of ER false positives. Lowercased: the number of falsely predicted men- tions written in lower case. Ground truth NIL: the number of predicted mentions that correspond to a ground truth mention annotated with NIL. Wrong span: the number of predicted mentions that are part of or overlap with ground truth mention of the predicted entity, but the predicted span is not correct.
4.1 AIDA-CONLL
The AIDA-CoNLL dataset (Hoffart et al., 2011) is based on the CoNLL-2003 dataset for entity
Disambiguation The disambiguation accuracy is defined as the correctly linked entities divided by the correctly detected entity mentions. We com- pute fine-grained disambiguation accuracies on sev-
*Our definition demonyms includes cases where the ground truth mention a language or ethnicity rather group of people, as long as the mention word appears the list of demonyms.
Note: we map all entities to Wikidata for our evaluation.

---

recognition which consists of news articles from the 1990s. Hoffart al. manually annotated the existing proper-noun mentions with corresponding entities in the YAGO2 knowledge base. The dataset is split into train, development and test set. For our evaluation, we use the test set which consists of 23 I articles. The benchmark has a strong bias towards sports articles (44% of articles are sports related). This results in large amount of demonym and metonym mentions. The average results achieved by the evaluated systems on AIDA-CoNLL are much higher than the average results on all other benchmarks included our evaluation. Entity men- tions in AIDA-CoNLL are mostly easy-to-detect single or two-word mentions (like names). Only 5.5% of mentions consist of more than two words which makes the ER part particularly easy on this benchmark.
GT mention property
Wiki-Fair
News-Fair
1482
359
Linked to NII.
132
49
Optional
447
84
Has altemative annotation($)
118
22
'able 2: Number of ground truth mentions with the given properties for our two henchmarks (without coref- erences).
eral articles differ from the ones in the original benchmark, but revolve around the same topic.
Our new fair benchmarks
We create two benchmarks to address the short- comings observed in existing entity linking bench- marks. The benchmarks are publicly available through our GitHub repository The first bench- mark, Wiki-Fair, consists of 80 randomly selected Wikipedia articles, the second one, News-Fair, of 40 randomly selected news articles from a web- news crawl (Akhbardch et al., 2021). In each of these articles, three random consecutive paragraphs were manually annotated with Wikidata entities. The rest of the article remains unannotated. This way, large variety of topics is covered with acceptable amount of annotation work while still allowing linkers to use complete articles as con- text. Annotating the benchmarks with Wikidata entities instead of Wikipedia (or DBpedia) entities decreases the likelihood of punishing linker for correctly linking entity that was not contained in the knowledge base during benchmark creation, since the number of entities in Wikidata is order of magnitude larger than in Wikipedia.
4.2 KORE5O
The KORE5O benchmark (Hoffart et al., 2012) con- sists of 50 hand-crafted sentences from five do- mains (celebrities, music, business, sports, polities). The sentences were designed to make entity disam- biguation particularly challenging, mainly by using only partial names when referring to persons, Thus, the benchmark contains lot of partial names and entities of type person. This also entails that, like AIDA-CoNLL, KORE50 contains hardly any men- tions with more than two words. In fact, 91.7% of mentions are easy-to-detect single-word mentions.
4.3 MSNBC
The MSNBC benchmark (Cucerzan, 2007) consists of news articles from 2007. In our evaluation, we use an updated version by Guo and Barbosa (2018) (the results are usually similar to those on the original benchmark). Cucerzan took the top two stories of the ten MSNBC News categories, used them input to his entity linking system and then manually corrected the resulting annotations. Adjectival forms of locations are rarely and incon- sistently annotated in the benchmark The orig- inal dataset contains overlapping annotations for no obvious r This was fixed in the updated version by Guo and Barbosa. They also removed links to no longer existing Wikipedia articles. Sev-
We also annotate non-named entities our benchmarks. In the few existing benchmarks that contain non-named entities, there is typically no discernible rule for which non- non-named entities were annotated such that the annotations seem rather arbitrary. To address this issue, define type whitelist (given in Section A.5) and annotate all entities that have an "instance canstanceooo"""" sf''/'uu path in Wikidata one of these "o
As discussed by Ling et al. (2015), existing entity linking benchmarks differ significantly in which mentions are annotated and with which enti-
"haps://gituu..comaaa ftrigituubboooomddrrrrerrraiiiie -nnt--i-i---tttt-iiiin------ benchmarks
"hille "Iraqi", "German" " "Syrian" are not annotated all. "U.S." the phrase "U.S. builder" is annotated, but not the phrase "U.S. helicopter".
"..g. the word "athlete" in the example sentence in Sec- tion would be annotated in our benchmarks, since the Wiki- data entity for athlete (Q20661 31) is an instance of the type "occupation" which is one of our whitelist types.
FF.g., in the phrase "Frank Blake", both, the entire phrase and "Blake" are annotated separately but with the same entity.

---

Benchmark
mentions lower multiword
NIL demonym metonym partial
rare
person
location
organization
AIDA-CoNLL
5616
0%
37%
15%
11%
18%
31%
53%
KORE50
144
0%
8%
1%
0%
6%
61%
11%
53%
11%
28%
MSNBC
739
1%
43%
12%
0%
2%
33%
7%
32%
24%
40%
News-Fair
24%
36%
13%
15%
21%
13%
26%
Wiki-Fair 1035
18%
4.%
13%
4%
0%
14%
14%
21%
32%
32%
Table 3: Statistics about types of mentions and entities in the benchmarks. mentions: number of (non-optional) ground truth entity mentions. lower: lowercased mentions. multiword: mentions that consist of multiple words. NIL: mentions where the annotation is Unknown. demonym: demonym mentions. melonym: metonym mentions. partial: the mention text is a part the entity's name (but not the full name). rare: the most popular candidate for the entities of type persom/locatiionorgaaazziooo
mention is the ground truth entity. personllcatiotorramaattoo Note that these entity types can sum up more than 100% because some entities have more than one type.
ER false negatives
ER false positives
Disambiguation error
System cased
lower-
partially included
lower- cased
gr. truth
wrong span
demonym metonym
partial name
NIL
ReFinED
39.6
14.6
6.6
121.2
11.4
5.7%
30.8%
16.8%
17.5%
REL
42.4
20.8
0.6
115.4
10.0
19.0%
27.1%
25.3%
30.9%
GENRE
44.4
16.0
1.4
52.2
13.2
2.1%
28.4%
19.5%
15.1%
Ambiverse
43.4
33.6
22.6
121.8
15.6
39.6%
73.9%
29.3%
43.5%
Neural EL
44.4
17.6
0.0
8.0
22.5%
78.1%
54.7%
73.2%
Baseline 41.8
37.2
56.2
110.6
10.2
53.1%
100.0%
65.7%
100.0%
TagMe 27.8
21.4
462.6
70.8
39.4
51.5%
63.4%
23.4%
60.0%
Table 4: Average results over all five benchmarks for the fine- greingr evaluation measures defined in Section 3. Note that the error rate is just one minus the accuracy. For "demonym" and "metonym" error rates, only those benchmarks were considered that contain at least 2% of demonyms or metonyms, respectively.
ties. With our benchmarks, want to introduce basis for fairer comparison of different approaches by giving annotation alternatives in cases where multiple annotations could be considered correct." We found that the averaged Fl scores all evalu- ated linkers are 5.2% lower on Wiki-Fair and 3.7% lower on News-Fair when not providing these alter- natives and only annotating the longer mentions.
ever, for the evaluation in this work, we use ver- sion without coreference mentions.
The total number ground truth mentions is shown in Table 2. The details of our annotation guidelines are given in Section A.4.
Evaluation of existing entity linkers
In the following we analyze six entity linking sys- tems in detail. Our evaluation includes linkers which code or an API are available and functional such that linking results can easily be produced Furthermore, we restrict the set linkers to those that either achieve strong results on popular bench- marks or are popular the entity linking commu- nity. Table 1 gives an overview of the results for all evaluated systems including simple baseline that uses spaCy (Honnibal et al., 2020) for ER and always predicts the entity with the highest prior probability given only the mention text. The two systems with the weakest results in our evaluation (Ncural EL and TagMe) are discussed in detail the appendix (A.3). The appendix also contains
Since there is considerable disagreement about the definition of a named entity, we introduce the concept of optional ground truth annotations, which includes dates and quantities. A prediction that matches optional ground truth annotation will simply be ignored, i.e., the system will not be punished with a false positive, but the prediction does not count as true positive either.
We also annotate coreference mentions. How-
11 For example, both linking the entire phrase "Chatham, New Jersey" to the entity for Chatham and linking just "Chatham" to the entity for Chatham (while linking "New Jersey" to the entity for the state New Jersey) are considered correct on our benchmark. If system predicts the mentions from the latter case, the prediction counts as a single true pos- itive if and only if both mentions were correctly recognized and linked to the correct entities. Otherwise it is counted as single FN. This is to avoid the need for fractional TP or FN. FPs are counted as usual.
This excludes for example Kolitsas et al. (2018), see these GitHuh issues, Ravi et al. (2021) see these GitHub issues or Brascheit (2019).

---

discussion of two systems that we did not in- clude in our table due very weak results and reproducibility issus. The individual results for all evaluated linkers can be examined in full detail in our ELEVANT instance
base the time of the annotation. The same trend can not be observed on our most up-to-date bench- marks, Wiki-Fair and News-Fair.
Disambiguation Even though ReFinED is the best disambiguation system in our evaluation, there is still room for improvement, particularly on metonym mentions, where it has an average er- ror rate of 30.8%. but also on partial name and rare mentions. Given that ReFinED is among the best systems in these categories, conclude that these categories are particularly hard to solve and are worth a closer look when designing new entity linking systems. Especially since they appear fre- quently in many benchmarks, as shown in Table 3.
6.1 ReFinED
Ayoola et al. (2022) developed ReFFnnED, fast end- to-end entity linker based on Transformers. They train linear layer over Transformer token embed- dings to predict BIO tags for the ER task. Mentions are represented by average pooling the correspond- ing token embeddings. They use a separate Trans- former model to produce entity embeddings from the label and description of an entity. The simi- larity between mention and entity embeddings is combined with entity type score and a prior probability to final score.
Reproducibility We were able to reproduce the results reported on ReFinED's GitHub page for the AIDA-CoNLL test set and the updated MSNBC dataset with minor deviations of s 0.6%. We achieved higher results than those reported in the paper on all evaluated benchmarks, since for the pa- per an older Wikipedia version was used (as noted by the authors on their GitHub page).
ReFinED comes two variants: A model trained on Wikipedia only and a model fine-tuned on the AIDA-CoNLL dataset. We report results for the fine-tuned version because it outperforms the Wikipedia version on all benchmarks in our evaluation. Moreover, ReFinED can be used with two different entity candidate sets: 6M Wikidata entities that are also contained in Wikipedia or 33M Wikidata entities. We choose the 6M set because achieves better results on most benchmarks." 14
6.2 REL
Van Hulst et al. (2020) introduce REL (Radboud Entity Linker). REL uses Flair (Akbik et al., 2018) as default ER component which is based on con- textualized word embeddings. For disambiguation, they combine local compatibility, (e.g., prior prob- ability and context similarity), with coherence with other linking decisions in the document using a neural network that trained on the AIDA-CoNLL training dataset. REL comes in two versions: one is based on a Wikipedia dump from 2014 14 and one is based on dump from 2019. We evaluate the 2014 14 version because it outperforms the 2019 version all our benchmarks except Wiki-Fair.
Evaluation summary Of the systems included in our evaluation, ReFinED has the best overall Fl score and is strong both for ER and for disam- biguation. Its closest competitors are GENRE and REL, which are considerably worse regarding ER (GENRE) or disambiguation (REL).
Recognition ReFinED has generally high ER FI score, but the performance difference to the other systems is particularly large on Wiki-Fair and News-Fair. This can at least partly be attributed to the fact that, in contrast to most other systems, ReFinED sometimes links lowercased mentions, which are only annotated on our benchmarks.
Evaluation summary REL achieves high overall
Fl score on all benchmarks and performs partic- ularly well in the ER task. In the disambiguation task, is outperformed by ReFinED and GENRE and performs poorly on Wiki-Fair. In the following we focus on weaknesses we found in the system. Recognition REL has a high number of FPs for mentions where the ground truth entity is NIL. While AIDA-CoNLL this is also duc to out- dated ground truth annotations, the trend is consis- tent across all benchmarks and indicates that REL could benefit from predicting NIL entities.
On AIDA-CONLL, it has the highest numbers of ER FP for mentions where the ground truth entity is NIL. closer inspection shows that in many of these cases, the system's 's predictions are actually correct and the ground truth entity was annotated as NII., probably due to an incomplete knowledge
17 "htps://elevant.co un.. www.rfrrugdd//eemlllleeeeeeeeeee de/emmlp2023/
'On News-Fair and Wiki-Fair (which we annotated with Wikidata entities), the 33M version is slightly better than the 6M version.
REL tends to detect mention spans that are shorter than those annotated in the ground truth;

---

see the "partially included" column in Table 4. Disambiguation REL performs well in the dis- ambiguation task, except on Wiki-Fair, where it just barely outperforms our simple baseline. Many of the disambiguation errors fall into none of our specific error categories (Table 4), which is typi- cally a hint that the true entity was not contained in the system's ' knowledge base and thus could not be predicted. This theory is supported by the fact that the REL version based on a Wikipedia dump from 2019 performs better on Wiki-Fair (and only on Wikii-Fair) than the 2014 version (Wiki-Fair is based on Wikipedia dump from 2020).
Disambiguation GENRE is the best system disambiguating demonyms and is only beaten by REL at disambiguating metonyms. Both kinds of mentions appear often in the AIDA-CoNLL dataset it was fine-tuned on.
Even though GENRE disambiguates metonyms, partial names and rare mentions comparatively well, there is still room for improvement for these hard categories; the respective comment in the discussion of ReFinED.
Reproducibility We could reproduce the result on the AIDA-CoNLAA benchmark with discrepancy of 0.7%. On the other benchmarks, the GENRE model trained on Wikipedia only is reported to give the best results, but performs very poorly in our evaluation; see this GitHub issue.
REL also has trouble disambiguating partial names on Wiki-Fair, but it does not have that prob- lem on the other benchmarks.
Reproducibility We were able reproduce the results reported in the paper for most benchmarks within a margin of error of < 1.0%.
6.4 Ambiverse
Ambiverse uses KnowNER (Seyler et al., 2018) for ER and enhanced version of AIDA (Hoffart et al., 2011) 1) for entity disambiguation. KnowNER uses a conditional random field that is trained on various features such as prior probability and a binary indicator that indicates whether the token is part of a sequence that occurs in a type gazetteer. The AIDA entity disambiguation component uses a graph-bascl method to combine prior probabilities of candidate entities, the similarity between the context of a mention and candidate entity, and the coherence among candidate entities of all mentions.
6.3 GENRE
GENRE (De Cao et al., 2021b) is an autoregres- sive language model that generates text with en- tity annotations. The generation algorithm is con- strained so that the model generates the given input text with annotations from a fixed set of mentions and fixed candidate entities per mention. GENRE comes in two variants: A model that trained Wikipedia only and one that fine-tuned on the AIDA-CoNE dataset. We evaluate the fine- tuned version because it outperforms the Wikipedia version on all benchmarks in our evaluation.
Evaluation summary Ambiverse is outperformed by newer systems, even its "own" benchmark AIDA-CoNLL (created by the makers of Am- biverse). On News-Fair and Wiki-Fair, its overall Fl score is hardly better than the baseline.
Evaluation summary GENRE performs well on all benchmarks, but is typically outperformed by ReFinED and REL. GENRE has a relatively weak ER F1, but strong disambiguation accuracy. This indicates that it tends to annotate only those men- tions for which it is confident that it knows the correct entity.
Recognition Ambiverse's ER component tends to recognize smaller spans than those from the ground 'ruth However, the detected shorter spans are often still linked to the correct entity, as shown by relatively high number of "wrong span" errors on News-Fair and Wiki-Fair.
Recognition GENRE's ER F1, averaged over all benchmarks, is 8.5% worse than that of the best system (ReFinED). Precision is always bet- ter than recall, with an especially large difference on News-Fair and Wiki-Fair. Most other linkers show this discrepancy those two benchmarks, but GENRE trades precision for recall more ag- gressively. Thanks to this, GENRE is among the systems with the lowest number of ER false posi- tives and it is also very good at not linking mentions where the ground truth entity is NIL.
Ambiverse has high number of ER false posi- tives for mentions where the ground truth entity is NIL across all benchmarks, which indicates that the system could benefit from predicting NIL entities.
Disambiguation Ambiverse performs relatively well on partial names on all benchmarks. This shows particularly on KORES0, where 61% % of mentions are partial names. Apart from that, its
For example, in the mention "1936 Summer Olympics", it detects only "Summer Olympics"

---

disambiguation is mediocre, with problems in the "demonym" " and "metonym" category. This shows particularly on AIDA-CoNLL, where these two categories are most strongly represented.
Author Contributions
All three authors conducted the research. N.P. and M.H. annotated the benchmarks. M.H. im- plemented the evaluation of GENRE and Efficient EL, N.P. implemented the evaluation of the other linkers. N.P. is the lead developer of ELEVANT and implemented several extensions needed for the evaluation in this paper. All three authors wrote the paper, with N.P. taking the lead and doing the largest part.
Reproducibility Since Ambiverse uses a modified version of the systems introduced in Seyler et al. (2018) and Hoffart et al. (2011), no direct compari- son to results reported in paper is possible. How- ever, the benchmark on which Ambiverse achieves its highest ranking in our evaluation is KORE50, which is benchmark that was hand-crafted by the same research group that created Ambiverse. On the other hand it also has one of its lowest rank- ings on the LIDA-CONLL test set which was also created by this research group.
Limitations
We only evaluated systems that perform end-to-end entity linking, which consider the most rele- vant use case. However, more systems exist which do only entity recognition or only entity disam- biguation, and these systems could be combined to perform entity linking.
6.5 Neural EL
This section has been moved to the appendix 1A.3.1) due to limited space.
We only evaluated systems with either code and trained models only evaluate the available versions. Our results of- ten deviate from the results reported in the papers, sometimes significantly. For example, the GENRE model trained on Wikipedia is reported to give good results on many benchmarks, but the model pro- vided online performs very poorly. The Efficient EL model was only trained on AIDA-CoNLL. and could benefit from training on larger and more diverse dataset (see Section A.3.4 for detailed evaluation of Efficient EL). Re-implementing or re- training models from the literature out of scope for this paper.
API available, and could
6.6 TagMe
This section has been moved to the appendix (A.3.2) due to limited space.
7
Conclusions
Our in-depth evaluation sheds light on the strengths and weaknesses of existing entity linking systems, well as on problems with existing benchmarks (in particular, the widely used AIDA-CoNLL) and reproducibilty issues. We introduce two new bench- marks with clear annotation guidelines and a fair evaluation as primary goals.
We only considered benchmarks and linkers with knowledge bases that are linkable to Wikidata, such as Wikipedia. However, in other research areas, there exist many knowledge bases and linkers for special use cases, e.g., biology or materials science. Outside of academia, the situation is even more complicated because the data is often proprietary (and sometimes also the employed software).
In particular, we find that even the best systems still have problems with metonym, partial name and rare mentions. All linkers have troubles with non-named entities. They either ignore non-named entities completely or link too many of them. Re- FinED performs best on almost all benchmarks in- cluding our independently designed and fair bench- marks. Several systems have reproducibility issues. The two newest systems, ReFinEE and REL, are significantly better in that respect.
We would like have reported results on more benchmarks, for example, Derezynski (Derezynski et al., 2015) and Reuters- 128 (Röder et al., 201 2014). but had to restrict our analysis due to limited space. We selected the most widely used benchmarks.
Our evaluation was more extensive than what we could fit into nine pages and we identified sev- eral frontiers for going deeper or further: describe more systems in detail, provide even more detailed numbers, include systems which only do disam- biguation, evaluate also by entity type, and consider other knowledge bases; see Section 9.
The evaluation tool ELEVANT by Bast et al. (2022) allows to evaluate and compare the perfor- mance of entity linkers on large selection of entity types (the usual ones: person, location, organiza- tion; but also many others). We limited our analysis

---

the different error categories, which we found more (r)elevant.
Samuel Broscheit. 2019, Investigating entity knowledge in BERT with simple neural end-to-end entity linking. In CoNLJ 2079, pages 677-685.
We evaluate end-to-end entity linking results, which means that the disambiguation performance can only be evaluated on mentions that were cor- rectly detected by a linker. Therefore, each linker's disambiguation performance is evaluated on dif- ferent set of ground truth mentions, thereby limit- ing the comparability of the resulting numbers. For example, a linker that detects only the mentions can disambiguate well would achieve an unrealisti- cally high disambiguation accuracy (at the cost of a low ER recall). A preferable way of evaluating the disambiguation performance would be to dis- entangle the ER and disambiguation components of each linker, and to evaluate the disambiguation component's is accuracy all ground truth mentions. However, this would require major changes to the linkers' code and might not be possible for all link- ers.
Silviu Cucerzan. 2007. ambiguation based on wikipedia data. In EMNLP- MNNLL2007, pages 708-716.
Large-scale named entity dis-
Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy multilingual entity extraction. SEMANTICS 2013, pages 121-124.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 202 la. Highly parallel autoregressive entity linking with dis- criminative correction. In EMNLP 2027 (1), pages 7662-7669.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021b. Autoregressive entity retrieval. In ICLR 2021.
Leon Derczynski, Diana Maynard, Giuseppe Rizzo. Marieke Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. 2015. Analy- sis of named entity recognition and linking for tweets. Inf. Process. Manag. (12:3222-4
References
Paolo Ferragina and Ugo Scaiella. 2010. TAGME: on-the-fly annotation of short text fragments (by wikipedia entities). In CIKM 2010, pages 1625- 1628.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In COLING 2018, pages 1648-1649.
Zhaochen Guo and Denilson Barbosa. 2018 Robust named entity disambiguation with random walks. Se- mantic Web, 94))459 479.
Farhad Akhbardeh, Arkady Arkhangorodsky, Mag- dalena Biesialska, Ondrej Bojar, Rajen Chatter- jee. Vishrav Chaudhary. Marta R. Costa-jussà, Cristina España-Bonet, Angela Fan, Christian Fe- dermann, Markus Freitag, Yvette Graham, Ro- man Grundkiewiez, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakya Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Koemi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshi- aki Nakazawa, Matteo Negri, Santanu Pal. Allah- sera Auguste Tapo. Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021. Findings of the 2021 conference on machine translation (WMT21). In WMT@EMNIA 2021, pages 1-88.
Nitish Gupta, Sameer Singh, and Dan Roth. 2017. En- tity linking via joint encoding of types, descriptions, and context. In EMNLP 2017, pages
Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen. Martin Theobald, and Gerhard Weikum. 2012. KORE: keyphrase overlap relatedness for entity dis- ambiguation. In CIKM 2012, pages 545-554.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal. Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. EMNLP 2011, pages 782-792.
Tom Ayoola, Shubhi Tyagi. Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022. Re- fined: An efficient zero-shot-capable approach to end-to-end entity linking. In NAACL-HLT 2022 (In-
Matthew Honnibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spaCy: Industrial- strength Natural Language Processing in Python.
dustry Papers), pages 209-220.
Kunal Jha Michael Röder, and Axel-Cyrille Ngonga Ngomo. 2017. All that glitters is not gold rule- based curation of reference datasets for named entity recognition and entity linking. In ESWC 2017 (1). volume 10249 of Lecture Notes in Computer Science, pages 125-320.
Hannah Bast, Matthias Hertel, and Natalic Prange. 2022. ELEVANT: A fully automatic fine-grained entity linking evaluation and analysis tool. In EMNLP 2022.
Adrian Brasoveamu, Giuseppe Rizzo, Philipp Kuntschik, Albert Weichselbraun, and Lyndon J. B. Nixon. 2018. Framing named entity linking error types. In LREC 2018.
Nikolaos Kolitsas, Oetavian-Eugen Ganea, and Thomas Hofmann. 2018 End-to-end neural entity linking. CoNLL 2018, pages 519-529.

---

Xiao Ling, Sameer Singh, and Daniel Weld. 2015. Design challenges for entity linking. Trans. Assoc. Comput. Linguistics, 3:3 15-328.
either the span is not annotated in the ground truth or linked with a different entity. False negatives (FN) are ground truth mentions where either the span is not recognized by a system or linked with a wrong entity. A ground truth span that is recog- nized but linked with the wrong entity counts as both false positive and false negative. Optional enti- ties count as neither true positive nor false negative. Unknown entities (i.e. entities that are linked NIL) do not count as false negatives when they are TP not detected. Precision is defined as TP+FF and recall TP FI score is the harmonic mean of precision and recall.
Pablo N. Mendes, Max Jakob, Andrés Garefa-Silva, and Christian Bizer. 2011. I. Dbpedia spotlight: shedding light on the web of documents. In I-SEMANTICS 2011, ACM International Conference Proceeding Se- ries, pages 1-8.
Katrin Ortmann. 2022. Fine-grained error analysis and fair evaluation of labeled spans. In LREC, pages 1400-1407. European Language Resources Associa- tion.
TP
Manoj Prabhakar Kannan Ravi, Kuldeep Singh, Isa- iah Onando Mulang'. Saeedeh Shekarpour, Johannes Hoffart, and Jens Lehmann. 2021. CHOLAN: modular approach for neural entity linking on wikipedia and wikidata. In EACL 2021, pages 504- 514.
We also evaluate the ER capabilities of the sys- tems. Here we only compare the predicted mention spans with the ground truth spans, regardless of the linked entities. Precision, recall and F1 score are defined as above.
Michael Röder, Ricardo Usbeck, Sebastian Hellmann, Daniel Gerber, and Andreas Both. 2014. N col- lection of datasets for named entity recognition and disambiguation in the NLP interchange format. In LREC. pages 3529-3533. European Language Re- sources Association (ELRA).
A.2 Excluded benchmarks
The following benchmark was excluded from our evaluation due to problems in the benchmark de- sign:
Michael Röder, Ricardo Usbeck, and Axel- Cyrille Ngonga Ngomo. 2018. GERBIL
DBPedia Spotlight (Mendes et al., 2011): A small benchmark containing 35 paragraphs from New York Times articles from eight different cate- gories. The annotators were asked to annotate "all phrases that would add information the provided text". The result is a benchmark in which 75% of annotations are non-named entities. The bench- mark contains annotations for words like "curved", "idea", or "house". On the other hand, phrases like "story", "Russian" or "Web language" are not annotated (even though "Web" and "Web pages" are) which makes the annotation decisions seem arbitrary.
benchmarking named entity recognition and linking consistently. Semantic Web, 95):6055625.
Henry Rosales-Méndez, Aidan Hogan, and Barbara Poblete. 2019. Fine-grained evaluation for entity linking. In EMNLP/JJCNPP 2019 (1), pages 718 727.
Dominic Seyler, Tatiana Dembelova, Luciano Del Corro, Johannes Hoffart, and Gerhard Weikum. 2018. A study of the importance of external knowledge in the named entity recognition task. In ACI. 2018(0)) (2), pages 241 146..
Marieke van Erp. Pablo N. Mendes, Heiko Paulheim, Filip Ilievski. Julien Plu, Giuseppe Rizzo, and Jörg Waitelonis. 2016. Evaluating entity linking: An anal- ysis of current benchmark datasets and roadmap for doing a better job. LREC 2016.
A.3 Evaluation of additional systems
A.3.1 Neural EL
Gupta et al. (2017) introduce Neural EL, a neural entity linking system that learns a dense represen- tation for each entity using multiple sources of in- formation (entity description, entity context, entity types). They then compute the semantic similar- ity between mention and the embedding of each entity candidate and combine this similarity with prior probability final score. Neural EL fo- cuses on entity disambiguation. The provided code is however also capable of performing end-to-end entity linking 16 which we are evaluating here.
Johannes M. van Hulst, Faegheh Hasibi, Koen Dereksen, Krisztian Balog, and Arjen P. de Vries. 2020. REL: an entity linker standing on the shoulders of giants. SIGIR 2020. pages 2197-2200.
A Appendix
A.1 Precision, recall, F1 score
We use precision, recall and Fl score to evaluate the entity linking systems. True positives (TP) are the linked mentions where the exact same text span is linked to the same entity in the ground truth. False positives (FP) are the linked mentions where
In the paper, Stanford-NER is used to evaluate the end-to-

---

Evaluation summary Neural EL achieves a low overall Fl score over all benchmarks. Its ER com- ponent performs decent on benchmark that contain only named entities, but weak on News-Fair and Wiki-Fair. Neural EL performs particularly weak in disambiguating partial names but solid in disam- biguating demonyms.
system but beats our baseline on most benchmarks. TagMe leaves it up to the user balance recall and precision with a configurable threshold.
Recognition TagMe has the lowest ER FI scores on all benchmarks with particularly low preci- sion. Recall is low on benchmarks containing only named entities, but decent on News-Fair and Wiki- Fair.
Recognition Neural EL has a relatively high ER precision. Neural EL's ER system is particularly strict with linking only named entities which re- sults in high number of lowercased ER FNs and a generally low performance on all benchmarks containing lowercased entities.
TagMe's ER component has tendeney towards including more tokens in its detected spans than what is annotated in the ground truth, thus achiev- ing good results in the "partially included" category. On AIDA-CoNLL and MSNBC where this effect is most observable, this can however often be as- cribed to erroneous benchmark annotations
Disambiguation Neural EL performs decent on demonyms. On our two benchmarks with a signif- icant number of demonyms, Neural EL ranks 3rd and 4th in the demonym category.
TagMe produces relatively high number of ER FP errors in the "wrong span" category, although sometimes these errors could also be attributed to debatable ground truth spans or missing alternative ground truth spans in the benchmark
On all benchmarks, Neural EL makes high number partial name errors. Only our baseline typically performs worse in this category.
Reproducibility We compare the results achieved on the AIDA-CoNLL development and test set using the publicly available code with the results reported in Gupta et al. (2017). For the com- parison, we provide ground truth mention spans to the system and exclude NIL links in both the ground truth and the predictions. However, we fall short of reproducing the reported results by 4.1% % on the test set (78.8% vs. 82.9% reported in the paper) and by 7% the development set (77.9% vs. 84.9% reported in the paper).
Disambiguation TagMe performs decent in the overall disambiguation category and shows weak disambiguation performance only AIDA- CoNLL. The weak performance AIDA-CONLL can be attributed to high number of metonym er- rors on this benchmark as well as a generally high number of demonym errors. A closer inspection shows that TagMe has a tendency to falsely link demonyms to the corresponding language"
TagMe has relatively low number of dis- ambiguation errors in the "partial name" cate- gory on most benchmarks, especially on KORE50. Since partial names make up 61 % of mentions on KORE50, this results in TagMe being the second- best performing system on KORE50 in the overall disambiguation category. However, it also has the lowest ER recall on KORE50. Comparing the indi- vidual predictions to those of Ambiverse shows that 24 out of 28 partial name mentions that Ambiverse disambiguates wrongly are either not detected by TagMe or also disambiguated wrongly.
A.3.2 TagMe
Ferragina and Scaiella (2010) propose TagMe, an entity linker designed to work well on very short texts like tweets or newsfeed items. They consider Wikipedia hyperlink anchor texts as possible men- tions. For the disambiguation, they compute the relatedness between mention's 's candidate entities and the candidate entities of all other mentions in the text and combine it with the prior probability candidate.
Reproducibility We evaluated TagMe over the WIKI-ANNOW dataset used in the original paper to evaluate end-to-end linking. Since we were un- able to reconstruct the original train and test splits
Evaluation summary TagMe frequently predicts non-named entities. Its overall Fl score is there- fore low on benchmarks that contain only named entities. It achieves decent results in the overall disambiguation category which can partly be ex- plained by the system ignoring mentions that are difficult to disambiguate. When filtering out non- named entity predictions, TagMe remains a weak
FF..g TagMe links the entire phrase in "2003 Kids' Choice Awards" the entity "2003 Kid's Choice Awards" which not actually an error. 16 TagMe links the entire phrase in "London Heathrow airport".
"F.g. in "Polish citizens" linking "Polish" to the Polish language rather than the country.
end entity linking task.

---

of the dataset, we used the entire dataset for eval- uation. However, we fall short of reproducing the Fl score reported in the original TagMe paper by almost 20% using the official TagMe API (57.5% vs. 76.2% reported in the paper).
DBpedia Spotlight achieves the highest ER re- call on News-Fair and the second-highest on Wiki- Fair (only outperformed by ReFinED) due to the low number of undetected lowercase mentions. On all other benchmarks, ER recall is mediocre.
DBpedia Spotlight makes the most ER FNs in the "partially included" category?2 on all bench- marks except KORE50 (REL performs worse).
A.3.3 DBpedia Spotlight
Mendes et al. (2011) propose DBpedia Spotlight, an entity linking system that aims specifically at being able to link DBpedia entities of any type. DBpedia identifies mentions by searching for oc- currences of entity aliases. Candidate entities are determined based on the same alias sets. For the disambiguation, DBpedia entity occurrences are modeled in a Vector Space Model with TF*ICF weights where TF is the term frequency and rep- resents the relevance of a word for a given entity and ICF is the inverse candidate frequency which models the discriminative power of given word. Candidate entities are ranked according the co- sine similarity between their context vectors and the context of the mention. An improved version of the system was introduced in (Daiber et al., 2013).
Disambiguation DBpedia Spotlight performs par- ticularly weak at disambiguating partial names and rare entities. The latter typically indicates that a sys- tem relies heavily on prior probabilities and does not put enough emphasis on the context of the men- 23 t
Reproducibility We tried to reproduce the results reported in the original paper on the DBpedia Spot- light benchmark using the official DBpedia Spot- light API. We were unable to reproduce the results for no configuration which we interpreted as using default parameters (42.4% vs. 45.2% reported in the paper). We were also unable to reproduce the results reported for the best configuration, which we assume corresponds to a confidence threshold of 0.35 and a support of 100 as indicated in the paper (33.6% vs. 56% reported in the paper). How- ever, it is important to note, that the system has undergone many changes since its first publication.
Evaluation summary DBpedia Spotlight is an entity linking system dedicated to linking entities of all types including non-named entities. When adding it to our set of evaluated linkers, it is the weakest performing system on almost all bench- marks including those containing non non-named en- tities. This can mostly be attributed to the weak performance of the ER component, but its disam- biguation results are not convincing either. DBpe- dia Spotlight comes with multiple configurable pa- rameters such as a confidence threshold to balance precision and recall and thus, similar to TagMe, leaves it to the user to find a good parameter set- ting.
A.3.4 Efficient EL
Efficient EL (De Cao et al., 2021a) is genera- tive model with parallelized decoding and an extra discriminative component in the objective. The pro- vided model is only trained on the AIDA-CONLL training data, and the paper evaluates only on the AIDA-CoNI test set.
Evaluation summary When adding it to our set of evaluated linkers, Efficient EL is only outper- formed by ReFinED on AIDA-CONLL but per- forms very poorly on all other benchmarks, since it was only trained on AIDA-CoNLL. We therefore only evaluate its performance on AIDA-CoNLL. On this benchmark, it has the best ER system, but GENRE is better on some disambiguation cate- gories, leaving room for improvement of Efficient EL.
Recognition DBpedia Spotlight has the lowest ER precision on almost every benchmark, mainly be- cause it falsely detects too many lowercased men- tions. While some ER FPs stem from DBpedia Spotlight trying to solve a different task than what most benchmarks were designed fo other er- rors are clearly not what is desired under any task description When filtering out lowercase predic- tions, ER precision improves, but is still among the lowest on all benchmarks.
Recognition Efficient EL is very good at detecting long mentions and has the lowest number of ER
E..g.. in "South Korea" linking both "South" and "Korea" to the corresponding country. "E.g.. in the phrase "in Columbus, Ohio". Columbus is linked to the entity of Christopher Columbus instead of Colum-
E..g. in "sports events" linking "sports" to the entity for "sport".
" "..g.. in "Spanish police" linking "Spanish police" to the entity for Spain.
hus the capital city of Ohio.

---

FPs AIDA-CoNLL.
NIL entities: Annotate entities not in Wikidata with Unknown to evaluate ground truth NIL errors and support coreference resolution evaluation for entities linked to NIL.
Disambiguation Efficient EL's disambiguation ac- curacy on AIDA-CoNLL is close to that of GENRE and REL but it is significantly outperformed by Re- FinED in that category.
A coreference is when the name of entity that appears elsewhere in the doc- ument is not repeated but replaced by pro- noun/description for solely linguistic purposes. E.g., "Barack Obama's wife" should not be an- notated unless Michelle Obama is explicitly men- tioned elsewhere in the document, because only then it's coreference. Otherwise it's a second- order entity linking problem and we're not evaluat- ing that.
Coreferences:
Efficient EL is the best demonym and rare en- tity disambiguator on AIDA-CoNLL but sig- nificantly worse at disambiguating metonyms and partial names then ReFinED, GENRE and REL.
Reproducibility The paper only reports results on the AIDA-CoNLL test set. The result in our evaluation close, not equal to the result the paper (85.0% Fl score compared to 85.5% in the paper).
A.5 Type Whitelist
A.4 Annotation guidelines
To ensure consistent annotation of entities in our benchmark, we annotated all entities that an instance or an instance of subelass of one of the types in a type whitelist. In rare cases where the Wikidata class hierarchy was clearly erroneous, we deviated from this annotation policy. The following is complete list of these whitelist types with their Wikidata QID:
What to annotate: Only annotate entities that are instance of at least one of our whitelist types an instance of subclass of one of the whitelist types.
Quantities and datetimes: Annotate quantities (including ordinals) and datctimes with a special label QUANTITY or DATETIME. Units should not be included in the mention.
Person (Q215627), Fictional Character (Q95074), Geographic Entity (Q27096213),
Demonyms: In general, annotate demonym men- tions with the country. Additionally, annotate the mention with the ethnicity or country-citizens if the culture or ethnicity being referred (e.g., "[American] dish"). The mention should not be annotated with the ethnicity cases like "[Sovict]-
Fictional Location (Q3895768), Organization (Q43229), Creative Work (Q17537576), Product QQ2424552), Event (Q1656682), Brand (Q431289). Genre (Q483394), Languoid (Q17376908), Chemi- cal Entity (Q43460564), Taxon (Q16521), Religion (Q9174), Ideology (Q7257), Position (Q4164871), Occupation (Q12737077), Academic Discipline (Q11862829). Narrative Entity (Q21070598),
backed United Arab Republic" (Soviet refers to (a part of) the government which is better represented by the country) or "[American] movie" (it's s still an American movie if the director decides to migrate another country). Only annotate the mention with the language if it is obvious that the language is being referred to (e.g.. "secctore"" means "sec- tors" [SpanishJ').
Award (Q618779), Disease (Q12136), Religious Identity (Q4392985), Record Chart (Q373899). Government Program (Q22222786). Human Population (Q33829), Color (Q1075), Treatment (Q179661), Symptom (Q169872), Anatomical Structure (Q4936952), Sport (Q349), Animal (Q729).
Spans: Use the Wikipedia title as mention. in doubt, also allow other spans that are aliases for the referenced entity. If an argument could be made for splitting a mention into several, annotate the splitted version alternative (e.g., "[[Louis VIII], [Landgrave of Tesse-Darmstaadee
Optional mentions: Use optional mentions for cases where the entity name and not the entity itself is being referred to, e.g., "known generally as the Istirrup dart moth]".


Reviews:
[{"review_id": "cfPEG3zZbk", "reviewer": "Reviewer_PBJb", "summary": "", "questions": "Question A: How is the problem of imbalanced entity types addressed when constructing new benchmarks? The problem of new benchmarks seems to still exist.\nQuestion B: Why is it not sampled from other sources to enrich the entity types of benchmarks?\nQuestion C: When building new benchmarks, how to sample each category of entity?\nQuestion D: When building new benchmarks, do you consider overlap entities?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. Here is our brief response to each of your questions and criticisms.\n\n**@\"Question A: How is the problem of imbalanced entity types addressed when constructing new benchmarks?\":** We chose the articles at random and just three random paragraphs from each article. Our goal was not to have balanced entity types, but to avoid strong imbalances, as in the existing benchmarks. For example, as you can see from the last three columns of Table 3, in the existing benchmarks almost all entities are of the types person, location, organization. These are indeed common in text, but in our benchmarks, we also have 20 - 40% entities of other types.\n\n**@\"Question B: Why not consider other sources to enrich the entity types\":** We already cover random extracts from random Wikipedia articles and from random news articles. That already gives good coverage and was a lot of annotation work. When considering other sources, one quickly gets into application-specific issues (for example, Tweets or entity linking in biomedical texts). That is certainly worthwhile, too, but a whole new paper and out of scope for this one. Also, it was already very challenging to pack the contents we have now into the available space.\n\n**@\"Question C: When building new benchmarks, how to sample each category of entity?\":** See the answer to Question A.\n\n**@\"Question D: When building new benchmarks, do you consider overlap entities?\":** We do consider alternatives both for the recognition part (ER) and the disambiguation part (ED). But we only do that when the alternatives are more or less equivalent. When one alternative fits clearly better, we pick only the better one. For example, in the mention \"2021 Summer Olympics\", it is clearly better to link the whole mention to that specific event than to link \"Olympics\" to the generic \"Olympic Games\".\n\n**@\"benchmarks proposed seem to be relatively small\":** Our two new benchmarks are not large, but comparable in size to most existing entity-linking benchmarks, with the sole exception of AIDA-CoNNL, which is significantly larger. See Table 3 of our submission, column “mentions”.\n\n**@\"the entity categories covered are not rich enough\":** Telling from the rest of your review, we take it that you mean the entity types we consider. Our entity types cover *all* annotations in the benchmarks we consider (which are the most used ones in existing work on the entity-linking problem) as well as all entities in our new benchmarks that are not too abstract. If you miss an entity type, then the reason is very likely that there is no entity of that type in any of the benchmarks we considered (old and new).\n\n**@\"The writing skills need to be improved\":** We are experienced writers and have written the paper with great care. Without concrete suggestions, we don't know what to make of such criticism.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper systematically analyzes the limitations of the current mainstream benchmarks in the field of entity links, which are crucial for entity linking research. This paper provides a systematic analysis of the research in this field.\n- In view of the limitations of existing benchmarks, this paper proposes two new benchmarks to alleviate these problems and reveal several artifacts and biases.\n- this paper is written structured and easy to follow.\n", "reasons_to_reject": "- The benchmarks proposed in this paper seem to be relatively small, and the entity categories covered are not rich enough.\n- There seems to be no good response to some of the benchmark limitations mentioned above, such as the imbalance of entity types in benchmark.\n- The writing skills of some details need to be improved.\n", "questions_to_authors": "Question A: How is the problem of imbalanced entity types addressed when constructing new benchmarks? The problem of new benchmarks seems to still exist.\nQuestion B: Why is it not sampled from other sources to enrich the entity types of benchmarks?\nQuestion C: When building new benchmarks, how to sample each category of entity?\nQuestion D: When building new benchmarks, do you consider overlap entities?\n", "paper_topic_and_main_contributions": "In response to the problems existing in the evaluation systems of the end-to-end entity linking task, this paper proposes a new evaluation system. For evaluation methods using only general aggregate measures, this paper proposes a detailed error analysis, including: lowercased, partially included, ground truth NIL, wrong span and other indicators. Then, in view of the limitations of the current mainstream benchmarks, two new benchmarks are proposed: Wiki-Fair and News-Fair. Finally, the current entity link system and model are re-evaluated on the new benchmarks. Although the size and variety of newly built benchmarks are limited, they still bring new contributions to the entity link community.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "-\tThe image quality of Figure 1 seems to be not clear enough. It is recommended to use a vector graphic.\n-\tMore details need to be listed in Table 2, such as the distribution of word counts for mentions, distribution of types, and so on.\n-\tSection 6 seems to have too much introduction about the models. The paper should focus on introducing the analysis results of the benchmarks.\n"}}, {"review_id": "PKFlB0H3Dv", "reviewer": "Reviewer_JFFU", "summary": "", "questions": "Q1: Please clarify what is the metric that should replace the F1 metric?\nQ2: Please explain why you have focused only on NER/NEL when framing this issue, given the fact that similar problems and solutions were also proposed for relation extraction and slot filling?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. Here is our brief response to each of your questions and criticisms.\n\n**@\"Q1: Please clarify what is the metric that should replace the F1 metric?\"  and \"it is not clear what the final F1 scores are\":** We are not sure that we fully understand this comment. The main point of our paper is not that the F1 metric should be replaced by another single-valued metric. The point of our paper is to take a much more refined look at the problem, which we do. Having said that, our benchmarks allow for alternative annotations and optional annotations, which indeed requires changes to the F1 score. Footnote 8 explains how we deal with alternative annotations. Lines 362-369 explain how we deal with optional annotations. The F1 score is then computed with the usual formulas mentioned in Appendix A.1.\n\n**@\"Q2: Please explain why you have focused only on NER/NEL\" and \"one should not omit CSSF or SF (Cold Start Slot Filling or simply Slot Filling)\":** Entity linking is a fundamental problem with literally thousands of papers about it. The paper is already packed with content as it is, and we consider it quite reasonable to not make the scope of the paper even wider. Problems like CSSF or SF or relation extraction are distinctly different problems in the context of knowledge bases. Of course, they are related to entity linking, just like many other NLP problems are related to entity linking because entity linking is so fundamental.\n\n**@\"the paper does not seem to adhere to the common terminology in the field [...] NERL and NEL are the well-known names of the tasks\":** We chose the name ER and EL fully deliberately because the N stands for \"named\" and considering non-named entities is an important part of our study. Also, EL is very common in the literature (2200 results in Google Scholar for EL \"entity linking\"), more common even than NEL (620 results for NEL “named entity linking”) and much more common than NERL (only 29 results for NERL \"named entity recognition and linking\").\n\n**@\"This is one of the first papers that actually proposes the idea of fair evaluation for NEL, therefore its omission is not great.\":** We don't understand which \"omission\" you are talking about here. If you mean the terminology, our choice was deliberate and we don't think that criticism is justified, see the comment above.\n\n**@\"no need for multiple sections in the introduction\", \"no need for so many subsections in Section 4\":** We deliberately introduced two subsections in the introduction to give it some structure due to its length of 1.5 pages. Section 4 has one subsection per benchmark in order to follow the same format as Section 6, which has one subsection per evaluated system.\n\n**@\"title of section 3 is problematic\":** We don't claim that this is novel, this is just a needed preliminary for the following sections. See our \"Contributions\" section for what we consider as our contributions. In particular, Section 3 is not mentioned there.\n\n**@\"Section 6 only discusses 4 systems\":** We would have loved to add more, but the space was limited and the paper is already packed with information, given the topic and the breadth of our goal. We therefore discuss the four best systems (see Table 1) in the main text and the others in the appendix.\n\n**@\"the analysis was limited to certain error categories\":** This is a misunderstanding, please read the corresponding sentence again. It doesn’t say that we omitted any error categories. It says that we preferred a breakdown of our analysis by error categories to a breakdown by entity types, for the simple reason that the former is much deeper and more insightful than the latter. We would have loved to include the latter as well, but this was not possible in the limited space.\n\n**@missing references:** Thank you for the (Ortmann, 2022) reference. It is not central, but indeed relevant for a detailed aspect of the ER part of the problem. The reference (Vashishth, 2018) is about a particular method for relation extraction, which is only remotely related to our paper (see our comment on CSSF and SF above).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- a new attempt to create fair evaluations for NER/NEL\n- new metrics for NER/NEL that include false negatives and false positiggbes, as well as disambiguation error rate\n- overall good readability", "reasons_to_reject": "- NER/NEL should not be discussed without the general context of relation extraction / slot filling today\n- some missing bibliography\n- not clear what is the overall metric that should be used from now on in fair evaluations", "questions_to_authors": "Q1: Please clarify what is the metric that should replace the F1 metric?\nQ2: Please explain why you have focused only on NER/NEL when framing this issue, given the fact that similar problems and solutions were also proposed for relation extraction and slot filling?", "paper_topic_and_main_contributions": "The paper is generally well-written and illustrated. Sufficient details are provided to reproduce most of the work even if implementing from scratch. \n\nOne general observation is that the paper does not seem to adhere to the common terminology in the field: e.g., NERL and NEL are the well-known name of the tasks instead of ER and EL. Using other task names than the common ones turns the whole experience of reading the paper into a struggle.\n\nThis is one of the first papers that actually proposes the idea of fair evaluation for NEL, therefore its omission is not great.\n\nThere are several comments that I think need to be addressed: \n1) There is no need for multiple sections in the introduction\n2) There is no need for so many subsections for all the benchmarks in Section 4 - a compact (or dense) writing style is better suited for a scientific publication.\n3) Title of section 3 is problematic as it suggests that the authors were the first to propose this, even though many proposals have been encountered during the years. In particular when treating NER and NEL one should not omit CSSF or SF (Cold Start Slot Filling or simply Slot Filling). If taking the larger field into account there have been many attempts to propose fair evaluations, starting with the TAC-KBP scorers from the TAC challenges.\n4) Not immediately clear what are the final F1-scores of the fair evaluations - or if there is anything equivalent to an improved F1-metric.\n5) Section 6 only discusses 4 systems instead of creating a table in which to synthetize the results from all the systems. Moving some systems in the Appendix creates a problem - as it might look like these other systems were dismissed. \n6) The limitations section mentioned that the analysis was limited to certain error categories which the authors found relevant without mentioning the criteria they used for relevance.\n\nI have read the rebuttal and think they have done a good job of explaining their thinking process. However, I will not modify my review scores.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Generally they should adhere to the terminology and abbreviations of modern NLP: NER (instead of ER), NEL (instead of EL), etc."}, "extra_text": {"Missing_References": "There is one missing reference that I think absolutely needs to be added, and another one that's optional (but I do think it should also be considered - even though it is in the context of slot filling rather than NER/NEL):\n\nOrtmann, K. (2022, June). Fine-Grained Error Analysis and Fair Evaluation of Labeled Spans. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (pp. 1400-1407).\n\nVashishth, S., Joshi, R., Prayaga, S. S., Bhattacharyya, C., & Talukdar, P. (2018). Reside: Improving distantly-supervised neural relation extraction using side information. arXiv preprint arXiv:1812.04361."}}, {"review_id": "2TaPgNNGQG", "reviewer": "Reviewer_wuuS", "summary": "", "questions": "Q. A. 362-369 implies that there are differences in the metric calculation among different systems. Perhaps it is necessary to provide another evaluation that completely disregards date and quantity for fairer comparison?\n\nQ. B. Table 4 only provides average results over all benchmarks. I think it would be beneficial to include metrics on separate benchmarks, allowing readers to make comparisons across various benchmarks, especially when considering newly introduced ones. Additionally, given the variations in the design and scale of these benchmarks, the appropriateness of simply averaging all the results may needs further discussion.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your review. Here is our brief response to each of your questions and criticisms.\n\n**@Question A: \"[...] provide another evaluation that completely disregards date and quantity for fairer comparison?”:** In a sense, that is exactly what we are doing. If one just *omits* the ground truth labels for dates and quantities, a system that regards those as entities would be unfairly punished with false positives. Only by annotating such mentions as optional in our benchmarks, is it possible to not punish such false positives. Similarly, a system that does not regard dates and quantities as entities is not punished with false negatives. Including dates and quantities as optional ground truth annotations is therefore arguably fairer.\n\n**@Question B: ”Lacking evaluation results on separate benchmarks.”:** We absolutely agree that the reader should have access to the non-averaged results. However, space is limited and a table with results for all error categories over all benchmarks for all linkers would be huge. We therefore include these results via a link to an interactive online evaluation tool as noted in line 394-395 and footnote 10 (we only include the link in the final version of the paper for anonymity reasons). This facilitates the inspection of the results for the reader as opposed to including a single huge static table in the paper.\n\n**@\"Metrics section closer to Table 4\":** This conflicts with two other requirements: (1) Table 4 should be close to section 6, since this section revolves around table 4. (2) The Metrics section should precede the Benchmarks section, since we refer to some metrics in the Benchmarks section.\n\n**@”readability of Figure 1”:** Thanks for pointing this out, we will fix that in the final version of the paper.\n\n**@”present baseline and other systems separately”:**  We thought about that, however, we find it more useful to have the systems sorted by overall F1 score. This way it is immediately clear that some systems are even outperformed by a simple baseline.\n\n**@Missing \\item in line 128:** The first sentence is a concise summary of the contributions. We tried to make that clear by adding \"More specifically:\" in line 132, followed by the two bullet points describing our main contributions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The author discussed a widespread phenomenon in current entity linking evaluations, which was overlooked and inadequately discussed.\n2. Rich and in-depth analysis.\n3. New benchmarks with annotation alternatives for fair comparsions.", "reasons_to_reject": "1. Lacking evaluation results on separate benchmarks.  (Please refer to question B)\n2. Some parts of the text might be a little less reader friendly. (Please refer presentation improvements)", "questions_to_authors": "Q. A. 362-369 implies that there are differences in the metric calculation among different systems. Perhaps it is necessary to provide another evaluation that completely disregards date and quantity for fairer comparison?\n\nQ. B. Table 4 only provides average results over all benchmarks. I think it would be beneficial to include metrics on separate benchmarks, allowing readers to make comparisons across various benchmarks, especially when considering newly introduced ones. Additionally, given the variations in the design and scale of these benchmarks, the appropriateness of simply averaging all the results may needs further discussion.", "paper_topic_and_main_contributions": "This paper argues that existing evaluations have limited practical significance in assessing the practical performance of entity linking (EL) systems. as they only provide coarse evaluation metrics and suffer from artifacts and biases. The authors provide an evaluation with two benchmarks and in-depth analysis of existing EL systems and benchmarks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. It would be beneficial to place the Metrics section closer to Table 4, as it would allow readers to refer to the metric explanations more conveniently while examining Table 4.\n2. The readability of Figure 1 is compromised in its printed version.\n3. It might be better to present baseline and other systems separately, possibly by listing the baseline in the first row. \n4. It seems that the beginning of line 128 miss a \\item."}}]

Author rebuttal:

Meta-review: **Summary:**
The authors argue that current evaluations of entity linking (EL) systems lack practical significance due to their reliance on coarse metrics and vulnerability to artifacts and biases. This underscores the necessity for more comprehensive evaluation methods. Accordingly, the paper presents an innovative evaluation system that includes a detailed error analysis, encompassing various indicators. Furthermore, the paper highlights the limitations of traditional benchmarks by introducing two new ones, Wiki-Fair and News-Fair. These newly introduced benchmarks have been utilized to  re-evaluate existing entity linking systems and models. 

**Strengths:**
The reviewers agree on the paper's strengths, which are as follows:
The author address a widespread phenomenon in current entity linking evaluations that has been previously overlooked and insufficiently discussed. This paper systematically analyzes the limitations of mainstream benchmarks, offering a comprehensive examination of research in this field.
The paper presents a novel attempt to create fair evaluations for NER/NEL, introducing new benchmarks and metrics that encompass false negatives and false positives, and disambiguation error rates.

**Weaknesses:**
The paper deviates from common terminology in the field, as NERL and NEL are recognized task names, rather than ER and EL. Additionally, discussing NER/NEL without considering the broader context of relation extraction and slot filling is unusual in current discourse. The proposed benchmarks appear relatively small, with limited coverage of entity categories. Some concerns regarding benchmark limitations, such as entity type imbalance, are not adequately addressed. Consequently, it remains unclear which overall metric should be adopted in fair evaluations, and there is a lack of evaluation results on separate benchmarks. Lastly, certain portions of the text is less reader-friendly. 

**Author-Reviewer discussion and acknowledgment:**
Reviewers provided multiple comments and questions, all of which the authors addressed in their rebuttal response. The authors effectively explain their thought process. All the reviewers acknowledged the authors' rebuttal.

**Conclusion:**
The paper is generally well-written and well-illustrated. It provides sufficient details to reproduce most of the work. However, reviewers recommend that the authors include missing references in the bibliography, and some minor improvements in the writing of certain details would enhance the overall quality of the paper.
Decision: Accept-Main