CleanCoNLL: A Nearly Noise-Free Named Entity Recognition Dataset
Authors: Susanna Rücker
Alan Akbik
Venue: EMNLP 2023 Main (2023)
Keywords: Dataset Relabeling and Evaluation
Label Error Detection and Correction
Named Entity Recognition
CoNLL-03
Entity Linking
TL;DR: We present and evaluate CleanCoNLL, a revised version of the classic CoNLL-03 benchmark, that significantly improves annotation quality and label consistency, and adds a layer of entity linking information.

Abstract: The CoNLL-03 corpus is arguably the most well-known and utilized benchmark dataset for named entity recognition (NER). However, prior works found significant numbers of annotation errors, incompleteness, and inconsistencies in the data. This poses challenges to objectively comparing NER approaches and analyzing their errors, as current state-of-the-art models achieve F1-scores that are comparable to or even exceed the estimated noise level in CoNLL-03. To address this issue, we present a comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our effort adds a layer of entity linking annotation both for better explainability of NER labels and as additional safeguard of annotation quality. Our experimental evaluation finds not only that state-of-the-art approaches reach significantly higher F1-scores (97.1%) on our data, but crucially that the share of correct predictions falsely counted as errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-art models, and that the theoretical upper bound even on high resource, coarse-grained NER is not yet reached.
To facilitate such analysis, we make CleanCoNLL publicly available to the research community.

CLEANCONLL: A Nearly Noise-Free Named Entity Recognition Dataset
Susanna Rücker
Alan Akbik
Humboldt-Universe zu Berlin susanna. ruecker@hu-berlin
Humboldt--nivversitäät zu Berlin alan. www.wkkkbbbbbblinn..oo
Abstract
UK
bookmakers
William Hill
said on Friday they [...]
The CONIL-033 corpus is arguably the most well-known and utilized benchmark dataset for named entity recognition (NER). However, prior works found significant numbers of an- notation errors, incompleteness, and inconsis- tencies in the data. This poses challenges to objectively comparing NER approaches and an- alyzing their errors, as current state-of- tae--offthhhtteee tithhe-ar art models achieve Fl-scores that comparable to or even exceed the estimated noise level in CoNLL-03. address this issue, we present comprehensive relabeling effort assisted by automatic consistency checking that corrects 7.0% of all labels in the English CoNLL-03. Our effort adds layer of entity linking annota- tion both for better explainability of NER labels and additional safeguard of annotation qual- ity. Our experimental evaluation finds not only that state-of-the-art approaches reach signifi- cantly higher Fl Fc-scorss (97.1%) 1%) on data, but crucially that the share of correct predic- tions falsely counted errors due to annotation noise drops from 47% to 6%. This indicates that our resource is well suited to analyze the remaining errors made by state-of-the-ar mod- els, and that the theoretical upper bound even on high resource, coarse-grand coarse-grained NER is not yet reached. To facilitate such analysis, make CLEANCONLL publicly available to the research community'.
CoNLL-03
UK
bookmakers
William Hill said on Friday they [...]
Conmana
CleanCoNLL,
Figure An example from CoNLL-03 in which the named entity "William Hill" is mislabeled person (PER). In CLEANCONLL, we leverage the entity link the Wikipedia article of the company determine that this entity is indeed of type organization (ORG). Further. our entity linking annotation makes more immediately transparent to human inspection why this entity is labeled as ORG.
However, prior works have found significant data quality and consistency issues: Reiss et al. (2020) identified share of 3.8% of all labels as incorrect. with particularly high error rate in the test split (7.5% error ratc). Similarly, Wang et al. (2019) found errors in 5.4% of all sentences of the test split. Qualitatively, various prior works noted inconsis- tencies where different labels given to the same entity throughout the corpus, affecting for instance the names of sports teams when referred to by their geographical name (Wang et al., 2019), sports leagues (Wang et al., 2019), names of deities and religious contexts (Ratinov and Roth, 2009), and the names of ministries (Stanislawek et al., 2019). Further, various prior works noted segmentation errors in which token-, entity- and even sentence- boundaries incorrectly set (Stanislawek et al., 2019; Reiss et al., 2020).
1 Introduction
The CoNLL-03 corpus is arguably the most well-known and utilized dataset for named en- tity recognition (NER). CoNLL-03 is typically the crucial benchmark to evaluate new NER ap- proaches (Schweter and Akbik, 2020; Zhou and Chen, 2021; Wang et al., 2021), language mod- els (Conneau et al., 2020; He et al., 2023), and most recently instruction-tuned LLMs for their few- and zero-shot abilities (Qin al., 2023; Ashok and Lipton, 2023).
We argue that these inconsistencies impair the usefulness of CoNLL-03 for benchmarking state- of-the-art (SOTA) NER approaches, since such ap- proaches generally report Fllscores that lie near (or even above) its estimated noise level. For instance, we find in this paper that nearly half of errors made by state-of ofate-of-the-ar the NER model are in fact correct

---

predictions that are falsely counted as errors due to noisy annotation. We argue that given the maturity of research into high-resource NER, the research community requires a dataset with highly consis- tent, nearly noise-free labels in order to correctly estimate the performance of lfte-of-the-eee models and to better understand their remaining errors. Adding and leveraging entity links. To address this issue, present CLEANCONLL, a revised version of CoNLL-03 that significantly improves annotation quality and consistency. Our main idea is introduce a second layer of annotation to the corpus that adds entity linking information. Entity linking is the task of disambiguating named entities to unique ID in knowledge base, typically the Wikipedia page representing specific entity.
CLEANCONLL, and comparatively evaluate the true error rate of these approaches.
We make the resource publicly available and en- courage the community use and further improve its quality.
Creating CLEANCONLL
CLEANCONLL was created in three distinct phases of annotation, which we describe chrono- logically in the following subsections:
We first (see Sec. 2.1) added a layer of entity linking annotation by merging two existing resources, and developed heuristic to au- tomatically correct NER errors using entity links.
As Figure illustrates with an example sentence, adding entity links serves two purposes: First, en- tity links allow us to define automatic consistency checks on NER labels, since prior work found that NER labels can be (partially) derived from Wikipedia page IDs (Tedeschi et al., 2021). Sec- ond, we find that the disambiguating information provided by entity links makes NER labels more immediately transparent to human inspection, thus making easier for annotators spot mistakes and discuss disagreements.
2.
We then (see Sec. 2.2) performed 3 iterative rounds of cross-checking to identify potential inconsistencies, which we manually examined and corrected whenever necessary".
3. Finally (see Sec. 2.3), we developed spe- cial handling for adjectival affiliation entity names.
2.1 Phase 1: Adding Entity Links
Contributions, In more detail, our contributions are the following:
Base corpus. We use the version from Reiss et al. (2020) as the starting point of our relabeling ef- fort². Reiss et al. (2020) already corrected 3.8% of all labels in CoNLL-03, including several major sentence and entity mention boundary problems. While this version has less noise than the origi- nal corpus, our own analysis (Sec. 3.4) and prior work (Wang and Mueller, 2022) found that large numbers of errors still remain.
We present CLEANCONLL, the result of a comprehensive relabeling effort of CoNLL-03 that includes updated, more consistent NER annotations in all three splits and integrates an additional layer of entity linking annotation.
We detail method for deriving NER annota- tion from Wikipedia links that leverages the category system in Wikidata, and discuss how this approach acts as consistency checks to safeguard annotation quality.
Merging AIDA. For initial corpus creation, we take advantage of the fact that AIDA (Hoffart al., 201 I), a well-known entity linking dataset, shares the same text base as CoNLL-03. Since the entity links in AIDA are expert-annotated by a group distinct of the original CoNLL-03 annotators, they provide layer of high quality annotation of arguably more complex task.
We further describe an automatic approach to flag potential inconsistencies that we use throughout the manual re-annotation process, which refer to as "cross-checking".
However, merging these two corpora was not straightforward since there are discrepancies both in terms of sentence boundary definitions as well
We conduct extensive analyses of CLEAN- CONLL and present both quantitative results of manual assessment of annotation quality in comparison previous versions and qualita- tive examples of error corrections. We train a set of current state-of-the the approaches on
Manual annotation in all phases of the process was carried out by authors of this work. See the limitations section brief discussion.
'Obtained by using their instructions from https: / //gi thub. con/CODAIT/ Identifying Incorrect- Labeltt---aaes-eeeeeeee In-CoLL-----2

---

WINIEDDEA
SOCCER
JAPAN
GET LUCKY WIN,
WIKIFEDIA
Japan nattonal fostball team
Jopan Farional bootball seam
China national football
CHINA
IN SURPRISE DEFEAT.
sargant orgieppinn
nefirna Centi usie
THE Sonal
Costiball Uead
CONLL-AIDA
WIKIDATA
national football (00K01)
WINDATA
Japan national looibell
SoN Lalbad myy munling Progleh Fraetie
Dipect (niin
poacoretin
CRC
ORC
Figure 2: A good portion of named entities in CoNLL-03 have manually assigned Wikipedia URLs in AIDA. In the example sentence "JAPAN" and "CHINA" linked to the article of the referred sports teams from which derive their Wikidata items and categories.
as the coverage of entity annotations. We found that of all 34,941 entities in the Reiss CoNLL ver- sion, 78.9% have an entity link in AIDA, while the remaining 21.1% 1 % (7,369 entities) do not. We found three main reasons for lack of coverage: (7) Rare entities that time of labeling had no correspond- ing Wikipedia article are unlabeled in AIDA. (2) The definition what constitutes a named entity differs in some instances, with CoNLL-03 for in- stance labeling the span "Boeing 707" as entity, while AIDA links only "Boeing". (3) Some com- mon entities like "EU"" for unclear reasons are con- sistently unlabeled in AIDA.
ORG, A full description of our mappings is pro- vided in Appendix A.1. Using these mappings, we automatically mapped 4, 4,163 (74.4%) of the 5,595 unique Wikipedia labels in AIDA to their NER type. However, recall in the development of our manual mappings. 1,432 (25.6%) could not be classified automatically because either none or multiple of our heuristics took effect.
prioritized precision over
Manual annotation. These remaining 1,432 entity links were manually assigned NER types. This annotation was carried out separately for all labels by two experts who as well as the respective Wikipedia and Wikidata information. The inter-annotator agreement was 44, cases of disagreement were subsequently resolved in discussion.
provided with guidelines
Deriving NER labels from entity links. We de- rive NER labels from Wikipedia URL links by leveraging Wikidata", a manually curated knowl- edge graph that connects entities (represented by Wikipedia pages) through a set of relations. We leveraged two of these relations: (1) instance_ _of, which connects instance to the conceptual class it instantiates, (2) subclass_of, which connects conceptual class more general conceptual class.
Using the resulting mappings, we updated all NER labels in the corpus to be consistent with the entity link.
2.2 Phase 2: Consistency Checking
After Phase of our relabeling effort, a number of potential error sources remained:
Refer to Figure/ 2 for an example of this: The mention "CHINA" is linked to the entity "China fational_ teamm, which is Wiki- data instance of "national association football team", which in turn is subelass of "national sports team". With these two relations, we auto- matically select conceptual classes for each linked entity in our corpus.
AIDA entity links, from which we derive NER labels in Phase 1, were missing for sub- set of entities, and may themselves contain noise. For instance, we qualitatively observed instances in which national sports teams were incorrectly linked to the Wikipedia article of the country instead of the team.
For the most common of these classes, we man- ually defined mapping to CoNLL-03 entity types. For instance, we manually defined that the Wiki- data category "sports team" maps to the entity type
Though developed to favor precision, our heuristics of deriving NER labels from Wiki- data categories may lead to an incorrect NER label assignment.
"https: //www. aakkidaa.ooo dagg

---

Leveraging AIDA entities ill-suited cor- rect wrongly spanned mention boundaries or tokenization problems.
You have annotated 75 out of 1920 apans!
spac: England
labela:
gold: LOC.
peed: DR2.
aids: England
hctpo //en wttppoeii..o////i///eeeeeeae ott/wiki//Rggland
and patteen)
2.2.1 Cross-Checking and Correction
To identify and correct these errors, we follow a "cross-checking" approach as introduced in Wang et al. (2019). The main idea is predict labels for small part of the corpus using the entirety of the remaining corpus, but make sure that no entity
iogland narked:
Glaen Reddle called uncapped Everton defender Andy Hinchclitfe Sunday the national squad for the opening world cup qualitier egainst Holdove sext woakend
[9] What to do with this span?: use
Prod: ORG
keep Gold: 10C
use Fred: oas
exactly like last: label: LOC, alda: England
use KIBC
use PER
names are memorized.
label (TO")
add/change (HER can changed abberwards)
To accomplish this, we collect all unique surface forms S of mentions in the corpus and group them randomly into 20 distinct batches S of around 500 surface forms each. For each of these 20 batches of surface forms, we create two splits: Dfeat, which contains all sentences that have one of the surface forms S; of the respective batch. Dirrin, which contains all other sentences, i.c. the sentences that contain only S \ S; (in other words: do not contain any of the surface forms of the respective batch). For each batch, we train state-of-the-ar model on Dirrin and use it to predict Diest, All predic- tions that differ from the annotated NER labels are flagged as potential errors.
soce context
(sors)
neartyy
your asnotacion")
previous
Figure Example of the annotator interface during cross-checking. The mention "England" is annotated as "LOC", but the model predicted "ORG", which correct. The wrong annotation originates from error in AIDA, where the mention is linked to the country's 's article ("Englandd") instead of the football team. Our annotator decided on NER label "ORG" and corrected the entity link to the Wikipedia page "England catindd_atttaa_a__eee _ team".
2.2.2 Iterative Cross-Checking in 3 Rounds
Manual error resolution. All annotations flagged potential errors were then mamually inspected by annotators. They were given the entity span within document context and the different label possibilities including their source: the current gold annotation, the diverging model prediction, as well as the Wikipedia label and its automatically derived NER label. Based on this information, they were asked to a) decide which NER label to use, and b) add or change the current Wikipedia label if necessary. See Figure for illustration of this interface.
We executed 3 rounds of cross-checking, After each round, labels were manually corrected and the improved corpus used to train the next models:
1. Entity detection and classification. The first round of cross-checking directly followed Wang et al. (2019) to train model jointly detect men- tion boundaries as well as classify them using a BIOES sequence tagging approach (Schweter and Akbik, 2020). Over all batches, this resulted in 1,020 flagged annotations, resulting in 521 cor- rected labels after manual inspection.
2. Only entity classification. The second round of cross-checking instead trained a model to only classify already provided entity mentions. See Ap- pendix A.7 for details of this model. This flagged further 709 instances for inspection, resulting in 98 additional corrected NER labels.
Annotation process. In the first round of annota- tion, all spans were independently labeled by two annotators (agreement was 79.3%), after which the guidelines were refined and the rest was done pri- marily by one. Annotators were given the option to mark cases of doubt (2.4%); these were given to the second annotator for independent annotation.
3. Only entity detection. The third round of cross- checking focused only on consistent definitions of entity mentions. To this end, we trained BIOES sequence labeler to predict only entity boundaries (but not their type). Over all batches, we collected 726 sentences with need of manual decision con- cerning mention boundaries (1,720 spans when counting the predicted and gold ones separately
Annotators were further asked to take note of error classes that may go beyond a single instance. For instance, noted general problems with anno- tation of time zones ("GMT", "EST")) Some were labeled as entities and others were not. such cases, we clarified the annotation guidelines and manually corrected all instances in the corpus.

---

Sentence (part)
Mentions
Entity Link (Wikipedia URL)
NER
NER
before
after
Phase
Phase
[...] while Italian team Roberto Baggio did not play due injury.
[Italian]
Italy nalyonationnl football
MISC
[Roberto Baggio]
Roherto Baggio
b) Dick Morris, the Republican politi- cal consultant who reshaped [..]]
[Dick Morris]
Dick_Marris
Morris
PER
PER
[Republican]
Republican, Party_(Unnieed States)
ORG
MISC
Rosati will meet Serbian President Slobodan Milosevic
[Rosati]
Darinsz_ Rosati
PER
PER
[Serbian]
Serbia
LOC
MISC
[Slobodan Milosevicl
Slobodan_ Milolovic
PER
PER
Table 1: Example sentences to illustrate the differences in adjectival affiliation entities. In the original CoNLL-03, affiliations such as "Italian". "Republican" and "Serbian" are marked MISC. However, since we derive NER tags from entity links, they are initially marked as ORG ("Italian" here for instance refers to national football team of Italy) or LOC. Phase 3 of our relabeling reverts these back to MISC usage.
despite their overlaps) that were resolved by man- ual annotation. In 450 of these sentences, different boundaries of one (or more) mentions were set.
CoNLL-03
Reiss ahs
CLEANCONLL
ahs
%
%
abs
%
LOC
10,645
30.3
10,103
28.9
9,399
26.7
ORG
9.323
26.6
9,922
28.4
10,492
29.8
2.3 Phase 3: Handling Adjectival Affiliations
PER 10,059
28.7
9,983
28.6
9.947
28.2
MISC
5,062
14.4
4,933
14.1
5,419
15.4
After Phase 2 of our relabeling effort, a major in- consistency to the original CoNLL-03 definitions became apparent: Adjectival affiliation entities consistently labeled MISC in CoNLL-03, whereas our corpus assigns the NER tags of their nominal usage. As Table illustrates, adjectival entities such as "Italian" may be labeled ORG in our cor- pus if linked to the Wikipedia page of the national football team of Italy.
entities
35,089
34,941
35,257
# sent. 20,744
20,617
20,617
Table 2: Statistics of CLEANCONLL, in comparison to CoNLL-03 and the Reiss version.
tion across the four classes for CLEANCONLL, the original CoNLL-03 and the Reiss variant.
We observe that CLEANCONLL has a slightly higher number ORG and MISC entities than the base versions. Among other reasons, this stems from a more consistent use of ORG labels for sports teams referred to by their geographic name, and more consistent use of MISC for entity types such as named sports leagues and stocks. The appendix provides more information on such cases in Ap- pendix A.6, and discusses qualitative examples Appendix A.5,
To make our annotations consistent with origi- nal CoNLL-03 definitions, semi-automatically revert such cases to the MISC label. Using two prediction models trained on OntoNotes (Hovy et al., 2006), we predict adjectives and entities of OntoNotes type "NORP" (affiliations to "nationali- ties or religious or political groups") to flag entities that should potentially be reverted to MISC. We automatically reverted flagged entities if they were originally labeled MISC in the Reiss version and manually checked all others.
Updated NER labels. Table 3 further examines the extent of label updates introduced by our ap- proach. Compared to CoNLL-03, the labels of 1,832 entity mentions (5.2%) were updated, and 625 (1.8%) mentions were newly added?. In sum, thus modified 7% of labels, affecting 10.3% of sentences We specifically note slightly higher label update rates in the test split (9.4% modified labels, affecting 14.2% of sentences) and the dev
3 Evaluation
We aim to get an understanding of annotation qual- ity of CLEANCONLL. To this end, we discuss its properties (Sec. 3.1) and perform manual evalua- tion of 100 sentences assess and compare anno- tation quality with earlier variants (Sec. 3.2-3.4).
'This includes both completely new ones as well as mentions where boundaries were moved. "Refer to Appendix A.8.3 3.3 for the number of changed lahels before reverting MISC to its original definition in Phase 3 (14.9%).
3.1 Dataset Statistics
Label distribution. Table 2 compares the total count of annotated named entities and the distribu-

---

CLEANCONLL
ment in annotation quality CLEANCONLL com- pared to earlier CoNLL-03 versions.
vs.
CoNLL-03
Reiss
abs
abs
%
3.3
Discussion of Ambiguities
Label unchanged
31,110
91.5
33,488
95.8
Label updated
1,832
5.2
1,205
3.4
Our manual evaluation also showed that cases of ambiguities remain CLEANCONLL that may be debated in our annotation guidelines. For instance, we consistently label named airports as ORG, even in sentences such as "the flight to Atlanta" (where humans might interpret "Alanta" as LOC instead of ORG). We decided to label time zones ("GMT") as entities, but do not label physical units.
New mention
625
1.8
534
1.5
Unknown
690
2.0
30
0.1
updated sent.
2,115
10.3
1,398
6.8
Table 3: NER label updates, comparing CLEAN- CONLL with CoNLL-03 and Reiss. Unknown: Due updated sentence segmentation, for some entities we cannot determine a direct equivalent, so we ex- clude these mentions here. New mention: Includes both completely new and newly bounded mentions.
A particular problem we faced are different pos- sible segmentations of larger entities. The span "Florida Supreme Court" may be seen as one entity, or may be seen to be composed of the LOC-entity "Florida" and the ORG-entity "Supreme Court". In such cases, generally favored the longer entity. One avenue for future work might be to add nested entities that both readings of this span could be reflected in the data.
split (8.4% modified labels, affecting 12.5% of sen- tences). Compared Reiss, we changed 1,205 labels (3.4%) and added 534 mentions (1.5%).
Updated entity links. Our annotation process added 1,955 new entity links over the AIDA base version and corrected 322 links. See Table 10 in Appendix A.8.2 for more details.
3.4 Further Qualitative Discussion
We conduct more extensive qualitative discussion of labeling decisions in CLEANCONLL, and how they differ from CoNLL-03, in Appendix A.5 and Table 7. There, we discuss examples that cover dif- ferent types of annotation problems, like outright wrong labels, missing mentions, boundary prob- lems well inconsistent use of labels similar or even the same instances.
3.2 Evaluation of Annotation Quality
We estimate the annotation quality for four versions of the corpus: The base CoNLL-03 version, the Reiss variant, CLEANCONLL, and variant of our corpus without Phase 3 of annotation, i.e. without reverting adjectival affiliation names MISC. We refer to this variant as CLEANCONLL
Manual evaluation. We gather all sentences in which there is at least one difference in annotation between at least two of the corpora. From these, we sampled 100 sentences for manual evaluation. For all corpora, all 100 sentences were indepen- dently labeled by two expert annotators. Cases of disagreement were discussed and, if possible, resolved. a subset of cases, annotators found contexts ambiguous, meaning that two diffcrent NER interpretations might be seen as valid. These cases were specifically marked.
16
ambiguous
missing_
12
4.01
annotation,
Enittes
6.69
Results. Figure 4 illustrates the results for all variants of the corpus. We find about 10% of men- tions marked as incorrect or missing in the original CoNLL-03, and about 7% of mentions either incor- rect or missing in Reiss. Our CLEANCONLL ver- sion by contrast was found to have only about 1 1% annotation errors in both variants, and missing mentions.
1.00
CoNLL-03
Reiss
CleanCoNILL
CleanCoNIL
Figure Estimation of annotation quality based on manually evaluated sample of 100 sentences for each of the four CoNLL-03 variants. For visual clarity we exclude the correct spans. We find that CLEANCONLL has significantly lower share of annotation errors than prior versions.
These findings indicate significant improve-

---

Name, Citation
Base embeddings
CoNIJ-003
Wang
Reiss
CLEANCONLI CLEANCONI.L
Flair (Akhik et al., 2018)
Flair Embeddings,
92.77 + 0.12
93.75 0.26
92.92 + 0.1 93.27 + 0.22
94.21 + 0.28
Ge embeddings
FLERT (Schweter
Transformer ("xlm-
93.94 +: 0.27
95.28 0.27
94.32 + 0.09 96.78 +: 0.07
96.98 0.05
and Akbik, 2020)
roberta-large")
Biafline (insp. by
Transformer ("xlm- roberta-large") combination of several
93.84 + 0.18
95.0
0.13
94.34 0.17
96.59 0.08
97.08 0.02
al. (2020))
ACE (senience)
93.56 0.15 94.77 0.06
93.48 0.16 95.41 + 0.25
95.89 0.0
(Wang et al., 2021)
Table 4: Comparing standard NER models on the different corpus versions. Averaged Fi-score and standard deviation from three runs (two runs for ACE).
4 Experiments
4.2 Classification of Model Errors
To gain further insight, we sample 100 prediction errors made by a FLERT model trained on each of the four variants of CoNLL-03. We use FLERT since it achieves the best results on average over all CoNLL variants. Our goal is to determine whether errors are "true errors" rather than an artifact of noisy annotation in the test data.
We train and evaluate representative set of NER models on CLEANCONLL, as well on CoNLL- 03 and the versions by Wang et al. (2019) and Reiss et al. (2020), to determine to what extent our relabeling effort affects model performance. We then sample a set of erroneous model predictions for each corpus variant to determine the share of true errors.
Manual evaluation. All 400 errors were indepen- dently evaluated by two expert annotators. They were tasked to determine whether the error is in fact real model error or rather an incorrect annotation. We added the possibility both correct, for cases in which both model and annotation can be argued for or there is ambiguity in the context, making clear decision impossible
We select models from literature that report state- of-the-art F1-scores on CoNLL-03 and/or are easily reproducible with open source libraries:
(1) Flair model as proposed by Akbik et al. (2018), using stacked bidirectional contextual string embeddings on character-level and GloVe embeddings (Pennington et al., 2014).
Results. Figure 5 shows the result of our man- ual evaluation: The share of correct predictions incorrectly classified as errors drops significantly from 47% in the original CoNLL-03 to 31 % in the Reiss variant, to only 6% and 7% in the CLEAN- CONLL variants. This indicates significantly higher annotation quality in CLEANCONLL.
(2) A FLERT model as proposed by Schweter and Akbik (2020), using document-level context fea- tures for fine-tuning a transformer model.
(3) A Biaffine tagging approach, inspired by Yu et al. (2020), in which all possible entity spans up to a maximum length (5) are considered as potential entities and separately classified.
In addition, the increased share of true model rors indicates that CLEANCONLL is well-suited evaluate the remaining errors made by state-of-the- art models, as error classification for the largest part reliable. The fact that most remaining errors are truc errors instead of artifacts of noise indi- cates that the upper bound for coarse- caasse-rriied NER is not yet reached and that higher Fl-scores than the current ofate-of-the-ar of 97% are theoretically feasible.
(4) The ACE model by Wang et (2021) that uses automatic selection and combination a set of pre-trained embeddings?.
4.1 Experimental Results
Table 4 reports averaged F1-scores and standard deviation over three random seeds for each experi- mental setting. As the table shows, observe con- sistently higher F1-scores CLEANCONLL com- pared to all earlier variants of CoNLL-03. This gives indication that our relabeling effort success- fully improved label and mention consistency.
"We use their sentence-level setup, as their document- level setup was computationally too costly for us to repro- duce for all variants and random seeds.
*For both model and annotation emors we distinguished mention boundary and label errors, but for visual simplicity aggregated them here under model or annotation crror.

---

600
Related Work
model Lener
500
baih cormet
As previously mentioned, other works have pointed out poor annotation quality in CoNLL-OS (Wang et al., 2019; Reiss et al., 2020; Wang and Mueller, 2022; Stanislawek et al., 2019). Most relevant for our work are Wang al. (2019) and Reiss et al. (2020), who both published their corrected versions of the corpus.
annotaatio, seror
300
2696
69%
67%
0%
70%9
100
31%s
2090
Wang et al. (2019) propose a method to improve training on noisy datasets via lowering the weights of potentially incorrect instances. They use their method to identify errors, but only focus on the test set. They release a manually corrected test set called CoNLL++. Similarly, Reiss et al. (2020) use majority voting in ensemble of trained models to identify potential errors and release a manually corrected version of CoNLLL-03.
ConL1-03 Retss
CleNCCoNLL'
ClennCoNLL
Figure 5: Estimated share of true vs. annotation errors in prediction errors for model trained each of the four CoNLL-03 variants. We find that the annotation error rate decreases from 47% in original CoNLL- 03 to only 6%-7% for CLEANCONLL, indicating significantly higher annotation quality.
Several other works also touch the topic of la- bel error detection or model error classification in CoNLL-03 (Stanislawek et al., 2019; Fu et al.. 2020; Jain et al., 2022; Chen et al., 2023). Liu and Ritter (2022) do not correct annotation errors but create completely new dataset which they also call CONLL+ (not to be confused with the corrections by Wang al. (2019)), that emulated CoNLL-03 label usage and distribution as well as text domain, but consists of newer articles.
5 Release of CLEANCONLL
We make CLEANCONLL publicly available to the research community through a github repository. We follow CoNLL-03 in releasing only annotations and script to merge them with the Reuters corpus that be obtained free of charge for research purposes' to produce the annotated corpus, see Figure for an excerpt.
Our annotation effort differs in that we (/) first integrate a layer of entity linking information for consistency checking and to assist manual annota- tion, and (2) use trained models for cross-checking over multiple iterations during manual annotation process. Our evaluation indicates that CLEANCONILL has significantly higher annotation quality than all previous versions.
Annotation layers. We release three new layers of unnotation: (1) Our entity linking annotations, (2) the final CLEANCONLL NER tags, and (3) NER tags after Phase 2, i.e. before reverting adjectival affiliations to MISC. We release the latter we believe some researchers might find type- resolved adjectival affiliations useful, and further discuss advantages and disadvantages of both NER label versions Appendix A.4.
Conclusion
Continued improvements by the research com- munity. Taking inspiration from the community- driven process of improvements in the universal dependency treebanks project (Nivre et al., 2020), we set up our open source repository such that re- searchers may file pull requests to further improve annotation quality, and discuss ambiguous cases or underspecification in the annotation guidelines. Our hope is that active participation by researchers in NER and entity linking will over time further im- prove annotation quality and further reduce noise.
We presented CLEANCONLL, corrected version of the classic NER benchmark CoNLL-03, with updated and more consistent NER labels. Our ap- proach adds an additional layer of entity linking annotation to disambiguate entities whenever pos-
sible to their Wikipedia page. We leverage this layer to both identify inconsistencies in annotation, and to make labeling decisions transparent to hu- man inspection. Further, we apply approach of cross-checking throughout our manual annotation process as additional consistency check.
Through our relabeling effort, we updated 7% of all labels of CoNLL-03, and added 5.5% new entity links over the base AIDA resource, Our eval-
For instructions on how to ohtain, refer to / www.clips ips. wantwernrreree Lantwerpen.

---

uation strongly indicates that overall annotation quality and consistency is significantly improved in CLEANCONLL. Sec. 3.2, we estimated a noise share of only around 1% for our sample, whereas CoNLL-03 had an estimated noise share of 10%. We found that current models reach significantly higher Fl-scores on our dataset (Sec. 4.1), and that the share of correct predictions incorrectly classi- fied as errors drops from 47% to 6% (Sec. 4.2).
able ambiguities like "fellow Queenslander Daniel Herbert" referring to the city or sports team) as well as mention boundary decisions ("Thai Commerce Ministry").
Regarding general annotation quality, note that only had two annotators overlapping with the authors of this work. This might have affected la- beling decisions or, in cases of doubt, pushed for correcting instead of keeping the label. However, as the authors worked extensively with the corpus throughout all of this work and had very round view of their annotation guidelines, they favoured consistency instead of "conservatively" keeping and potentially overlooking label errors or edge cases. Providing the annotators with the source of the label suggestions during cross-checking instead of letting them choose completely "blind" was done intentionally to enable them to spot inconsistencies in the gold labels as well as mark cases for spe- cific care which could then be incorporated in the succeeding rounds. However, appointing the same annotators for evaluation (annotation quality assess- ment and classification of model errors, see Sec. 3.2 and 4.2) is indeed not ideal but unfortunately could not be avoided due to time constraints and resources.
These findings indicate the validity of our re- labeling approach, and the usefulness of CLEAN- CONLL for training and evaluating state-of-the. NER models. To enable the research community to do so, we make our new layers of annotation publicly available through an open source repos- itory modeled after the UD treebanks. We hope that active participation by the research community will help us continuously further improve anno- tation quality, with the ultimate goal of an entirely "noise-free" NER dataset.
Limitations
Despite our best efforts to ensure consistency, it is possible that there still is inconsistency of label usage in CLEANCONLL, which enumerate as limitations here. (1) It is possible, that specific instances are wrongly labeled (e.g. because of a wrong or missing Wikipedia label leading to an incorrect NER label that fell through our cross- checking approach).
Concerning generalizability: We acknowledge that the initial round of leveraging entity linking annotations requires precisely such annotations and thus cannot be transferred to every NER dataset and other languages. Furthermore, the applied heuris- tics for deriving NER labels from the Wikidata taxonomy were constructed rather intuitively and would need updating in case of different, maybe more fine-grained NER labels (refer to Appendix A.2 for a discussion) and languages. However, confident that this is feasible, e.g. adding rules for "work of art", "medication", "physical unit". Furthermore, the cross-checking rounds are independent of the named entity linking labels and language of the dataset.
(2) Also - this might be in- evitable some of our annotation guideline deci- sions can be up for discussion
well as some de- cisions about splitting or merging of mentions with sub-entities ("Florida Supreme Court", "the State Council's ' Port Office"). Furthermore (3), we note that the additional named entity labels (Wikipedia links) are not complete, which we hope to change in future versions of CLEANCONLL. (4) Concern- ing tokenization and sentence segmentation, there still are some problems in the data, e.g. the prob- lematic usage of hyphens ("-") inside tokens ("the Syrian- Lebinn-L peace tracks", "the India-South Africa test") which impedes proper labeling of the components ("Syrian", "Lebanese", "India", "South Africa").
Ethics Statement
This work does not raise many ethical problems in our opinion. The major concern is that CleaN-- CONLL still uses the same text base the orig- inal: News articles that are over 20 years old. So both the language use and also the themes might be outdated and might also show signs of bias. Fur- thermore, we only worked on the English part of CoNLL-03.
In general, it might be a promising direction to allow for multiple or nested annotation possibilities. Both concerning label (e.g. there might be irresolv-
('E.g. labeling airports as ORG and deities as MISC deciding on LOC for "Vatican" and between MISC and ORG for "catholic (church)", labeling time zones ("GMT") as MISC but not labeling physical units etc.

---

Acknowledgements
text. Proceedings the 2011 Conference on Em- pirical Methods in Natural Language Processing. pages 782-792, Edinburgh, Scotland, UK. Associa- tion for Computational Linguisties.
We thank the reviewers for their helpful comments. Susanna Rücker is supported by the German Fed- cral Ministry of Economic Affairs and Climate Action (BMWK) as part of the project SEBIRA (KK5148003LB1), Alan Akbik is supported by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's 's Excellence Strategy "Science of Intelligence" (EXC 2002/1, project number 390523135) and the DFG Emmy Noether grant "Eidetic Representations of Natural Language" (project number 448414230).
Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of the Human Lan- guage Technology Conference of the NAACI, Com- panion Volume: Short Papers, pages 57-60, New York City, USA. Association for Computational Lin- guistics.
Sarthak Jain, Varun Manjunatha, Byron Wallace, and Ani Nenkova. 2022. Influence functions for sequence tagging models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 824-839, Abu Dhabi, United Arab Emirates, Associ- ation for Computational Linguistics.
References
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence label- ing. In Proceedings of the 27th International Con- ference on Computational Linguistics. pages 1638- 1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Shuheng Liu and Alan Ritter. 2022. Do CoNLL-2003 named entity taggers still work well in 2023? ArXiv, abs/2212.09747
Joakim Nivre, Marie-Catherine de Marneffe, Filip Gin- ter, Jan Hajic, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers. and Daniel Zeman, 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4034 4043, Marseille, France. European Language Resources Association.
Dhananjay Ashok and Zachary C. Lipton. 2023. PromptNER: Prompting for named entity recogni- tion. ArXiv, abs/2305..1544
Zhijun Chen, Hailong Sun, Haoqian He, and Pengpeng Chen. 2023. Learning from noisy crowd labels with logics. ArXiv, abs/2302.06337.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
N. Chinchor, E. Brown, L. Ferro, and P. Robinson. 1999, Named entity recognition task definition. Technical Report Version 1.4, The MITRE Corporation and SAIC.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 8440- 8451, Online. Association for Computational Lin- guistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is ChatGPT
natural language process- ing task solver? ArXiv, abs/2302.06476
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference Compu- tational Natural Language Learning (CoNIJI-200)), pages 147-155, Boulder, Colorado. Association for Computational Linguistics.
Jinlan Fu, Pengfei Liu, and Qi Zhang. 2020. Rethink- ing generalization of neural models: A named entity recognition case study. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34(05). page 7732-7739.
Frederick Reiss, Hong Xu, Bryan Cutler, Karthik Muthuraman, and Zachary Eichenherger. 2020. Iden- tifying incorrect labels in the CoNI.L-20003 corpus. In Proceedings of the 24th Conference on Computa- tional Natural Language Learning, pages 215-226. Online. Association for Computational Linguistics.
Pengcheng He. Jianfeng Gao, and Weizhu Chen. 2023. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient- disentangled embedding sharing. ArXiv, abs/2111 10/22111.000 10.0543.
Stefan Schweter and Alan Akbik. 2020. FLERT: Document-level features for named entity recogni- tion. ArXiv.
Johannes Hoffart, Mohamed Amir Yosef, larii Bordino, Hagen Fürstenau, Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disanbiguation of named entities in
Tomasz Stanislawek, Anna Wróblewska, Alieja Woj- cicka, Daniel Ziembicki, and Przemyslaw Biecek. 2019. Named entity recognition is there a glass

---

ceiling? In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 624-633, Hong Kong, China. Association for Computational Linguistics.
we get a collection of general categories like "na- tional sports team" or "geographic region". From these collections, we derive the NER label (PER, ORG, LOC) with the following rules:
Simone Tedeschi, Simone Conia, Francesco Cecconi, and Roberto Navigli. 2021. Named Entity Recog- nition for Entity Linking: What works and what's next. In Findings of the Association for Computa- tional Linguisties: EMNLP 2021, pages 2584-2596,
ORG if collection includes one or more of I" ("organization", "political party". "political or- ganization", "confederation", "sports club". "political party", "business", "public com- pany". "type of organisation", "national sports team", "association football club", "sports team", "government organization"]]
Punta Cana, Dominican Republic. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
I
LOC if collection includes one or more of ["state", "country", "city", "classification of human settlements", "human settlement", "ge- ographic entity", "physical location", "U.S. state", "historical country", "island", "geo- graphic region"]
Wei-Chen Wang and Jonas Mueller. 2022. Detect- ing label errors in token classification data. ArXiv, abs/2210.03920.
Xinyu Wang. Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Automated concatenation of embeddings for struc- tured prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages
PER if collection includes one or more of ["hu- man", "person", "Wikimedia human name dis- ambiguation page"]
Manual assignment for dummy MISC. either none or multiple of ORG, LOC or PER was as- signed, we assign the MISC label instead, which for the time being serves as dummy label marker for need of manual assignment. These were then man- ually checked and if necessary changed to ORG, LOC PER.
Online. Association for Computational Linguistics.
Zihan Wang, Jingbo Shang, Liyuan Liu, Lihao Lu, Ji- acheng Liu, and Jiawei Han. 2019. Cross Weigh: Training named entity tagger from imperfect anno- tations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 5154-5163, Hong Kong, China. Association for Com- putational Linguistics.
A.2 Analysis: Revisiting the Automatic NER Mapping
Having used automatic NER labeling through Wikipedia and Wikidata for initial corpus creation (recall Phase 1, Sec. 2.1), but then manually de- tecting and correcting labels through the cross- checking phases (Sec. 2.2), we are interested in the final picture in CLEANCONLL, with respect to NER labels the hand and Wikipedia/Wiki- data items on the other hand. This also gives in- sights into how reliable a purely automatic map- ping approach could be when leveraging other NEL datasets for NER corpus correction or derivation. We conduct two analyses concerning a) automatic mapping correctness and b) alignment of NER type and Wikidata categories.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 6470- 6476, Online. Association for Computational Lin- guistics.
Wenxuan Zhou and Muhao Chen. 2021. Learning from noisy labels for entity-centric information extraction. In Proceedings of the 2021 Conference on Empiri- cal Methods in Natural Language Processing, pages 5381-5392, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
A Appendix
Automatic Mapping Correctness. The AIDA corpus uses 5,595 unique Wikipedia links. As we allowed our annotators to change or add new ones, our corpus version now includes 5,943 unique links. We look at how their current NER labels
A.1 Mapping: Wikidata categories to NER
Through the Wikidata relations instance_of and subclass_of, for each Wikipedia URL in AIDA

---

Wikipedia-NEA mapping in CLEANCONLL compared to
provided by Tedeschi et al. (2021) (NER4NEL) Table 5, who also create NER labels for Wikipedia articles, but use the WordNet taxonomy instead of Wikidata. Since they use more fine- tne-grained NER classes (18), we merge all of their classes other than PER, LOC and ORG to MISC for comparison. We note that our mapping agrees in 88.8%, but differs in 7.4% 12, Looking through these cases of disagrec- ment, there are cases where both NER decisions are understandable: "Metropolitan Museum of is labeled ORG in CLEANCONLL, but LOCC in NER4NEL, newspapers like "The_ Lahe__akartt_PPP Post" are labeled MEDIA in NER4NEL (which for the sake of comparison we merge to MISC) but ORG in CLEANCONLL. However, a good por- tion of the disagreements stem from definitely in- correct labels in NER4NEL, e.g. one sports club Schalke 3ch" being labeled as LOC, others ("Sheffield _aalees" "Gil_ Vicente (icente_F.CC.)) even as PER and "Supreme__e Laatremme_Cuttteee srrttof_Neerr of_Nee__Soull Wales" as TIME (so merged to MISC), where CLEAN- CONLL labeled ORG.
Automatic abs %
NER4NEL abs
%
Wiki label agrees
5,101
85.8
5,278
88.8
Wiki label differs
842
14.2
438
7.4
Dummy-MISC
790
13.3
Real difference
52
0.9
NA
0
0.0
227
3.8
# Wiki labels
5,943
5,943
-ft__ffAArtt' Art"
Table 5: Comparing the final Wikipedia-NER map- ping in CLEANCONLL with the purely automatic one (Round A) and with the NER4NEL mapiigg by Tedeschi et al. (2021).
CLEANCONLL- after several cross-checking re- vision rounds - differ from the initial ones derived via the automatic approach in Phase 1 via Wikidata categories. To enable a full comparison, we rerun the automatic labeling method via Wikidata also for the 953 new Wikipedia links. The results of this comparison are in Table 5.
NER labels and Wikidata categories. Finally, take look at the final picture of NER labels and their respective most common Wikidata cat- egories, i.c. the classes derived via the Wikidata instance_of and subclass_ot relations, see Sec. 2.1. In Table we look at the most frequent Wiki- data classes that mentions with the respective NER label carry in CLEANCONLL (of course, only looking at those that have a Wikipedia label). We see, that the alignments intuitively make sense: For LOC we c.g. find "city", "state", "country" to be under the most frequent ones, for "ORG" we find "football club", "organization" and "business". Par- ticularly interesting are the classes for the - now real MISC label (e.g. "sports competition", "re- curring event" or "(stock) exchange"), which possi- bly could be used to find additional rules for future automatic labeling methods. Reversely, for more specific domains, new NER labels could be con- structed via frequently appearing Wikidata classes (e.g. "work of art", "event").
We note that 85.8% of the final labels agree with their automatically derived one from Phase 1, which overall shows that the automatic labeling process works well. The remaining 14.2% now have a different assigned NER label. This number seems alarmingly large first. However, reall that we designed the rules for automatic NER extrac- tion from Wikidata classes in favour of being more precise than comprehensive and used MISC as the initial dummy label when none (or multiple) of our heuristic rules applied to the item. This dummy MISC then led to the item being manually classi- fied, so it is not surprising that the label now differs from MISC. In fact, not counting these dummy- MISC cases, real difference is only observed in 52 cases (0.9% 11, This shows, that the automatic NER mapping already produced high quality NER labels that nearly needed no revision.
If we were to purely use the automatic labeling approach through heuristics could be further refined depending on the use case and particularly, one could add specific ones for the MISC label or any other additional label, considering the specific domain.
Skkipedia/Wikidarn items, the
A.3 Format of Dataset Distribution
Our resource enables deriving our dataset in CoNLL column format with BIO tagging scheme. Sce the excerpt in Figure 6 as an example: We pro-
We add a comparison of our mapping to the one
"I MMan of these are cases in which we decided upon specific label usages after the heuristics and automatic labeling was already done, e.g. to lahel stock exchanges as MISC or universities and hospitals as ORG.
12ehe remaining 3.8% ("NA") are Wikipedia articles that are not present in the NERANEL mapping, presumably hecause the article did not exist at the time or changed its name.

---

NER label
Wikidata class
frequency
variant (with MISC reverts):
LOC
1. Users of our resource might prefer the original notion of MISC make results compa- rable to the vast research and models trained on the original CoNLL-03.
city
459
big city
350
city town
338
urban
273
213
human settlement
209
political territorial entity country
209
2. A lot of the cases in question are actually cases with real ambiguity in meaning that makes it hard to decide on LOC or ORG and makes inconsistency in labels rather inherent. See for example the following sentences:
sovereign state
capital city
ORG
football club association football club
574
527
sports team
289
(a) the [Chinese] police
juridical person
287
(b) the [ChineseJoRGzoo keeper
organization
275
business
256
(c) the [Chinese] success in the skiing
association foothall
209
World Cup
economic entity enterprise
200
Here, the label depends on the understanding of the entity "Chinese" mcaning (a) the coun- try. (b) the national team or the nationality the individual, or even (c), when there is not one specific team, but still some ORG-like se- mantics in several individuals representing the country. Using MISC (recall that the original CoNLL-03 and the Reiss version would have MISC all of the three) erases the necessity of deciding on one meaning and therefore this specific source of inconsistency.
national sports team
PER
human
2,448
person
2,448
omnivore
2,448
natural person
2,448
mammal
2,448
Homo sapiens
2,284
MISC
sports competition
52
recurring
45
recurring sports
40
sports league tournament. competition
35
29
Experiments comparing both variants of CI.EANCONLL show slightly better model performance on the final version after MISC reverts (see Sec. 4.1), which a strong indi- cator for better label consistency.
25
Table 6: Most frequent Wikidata classes (i.e. derived through instance_ds of and sublass of relation) of mentions with the respective NER label.
However, there are some downsides consider when using the final version with MISC reverts:
vide the original POS tags from (Reiss et al., 2020) and updated NER labels in the two variants (before and after reverting adjectival affiliations to MISC, see Sec. 2.3), as well as the Wikipedia links in the underscored version as they appear in the URL (e.g. Indonesia indonesia_nationall football team). See Appendix A.4 for a discussion pros and cons of both label sets.
The ambiguity problem (the "Chinese" ex- amples above) still remains in non-adjectival cases: In "it was China who dominated the games", the label still needs to be LOC or ORG, depending on the interpretation. Reusing MISC could actually make it harder for models to get round understanding of the real LOC, ORG and MISC semantics and differentiation.
A.4 Discussion: Dataset Variants on Adjectival Affiliations
As mentioned in Sec. 2.3, we distribute CLEAN- CoNLL with two sets of NER labels: One without and one with the reverts to MISC from Phase 3. We here discuss differences and (dis-)advantageer of those two versions.
2. It is questionable apply a purely syntactic criterion such adjectival in an otherwise more semantically influenced labeling task like NER. For illustration, consider the sen- tence "The area patrolled by U.S., French and British planes." " "French" and "British"
There are two main advantages in using the final

---

Text
POS
Wikipedia
CleanCoNLL
name "Florence" is confused with the city (la- beled LOC, corrected in Reiss), or (b), where before Phase
after Phase
Indonesian
JJ
B- G-Indonesia_nane Inddmmsiaa_nttion__ofto.....
B-ORG
B-MISC
keeper
NN
0
Hendro
6-Hendro_Karriko Kartiko
B-PER
B-PER
Kartiko
H-Hnndro Ka
I-PER
I-PER
produced
0
DT
0
string
NN
0
of
IN
0
0
fine JJ
0
0
saves VBZ
0
0
to TO
0
0
prevent
VB
0
0
0
the
DT
0
Koreans NNPS
B-ORG
B-MISC
increasing
VBG
0
0
0
their
PRP$
0
0
0
lead
NN
0
0
0
0
0
Figure 6: An excerpt from CLEANCONLL in column format, including sentence with the original POS tags, Wikipedia labels and the two versions of updated NER annotations in BIO format.
are labeled MISC, whereas "U.S." gets LOC, even if the usage is quite the same. Befor re- verting to MISC, all three of "U.S.", "French" and "British" are labeled LOC which actu- ally is more consistent.
"Green" is spuriously labeled as PER, but fact is just the color or (c), where "Mind Games" labeled as PER (still in Reiss), when in fact it is the name of a racehorse (thus gets MISC from our annotators), In (d), "William Hill" is labeled as PER in CoNLL- 03 and Reiss, but refers to gambling com- pany with the same name (ORG).
3. If in a use case, NER is used as method for collecting sentences or sentence parts that talk about specific content, say, specific (na- tional) sports teams searching for place nationality related elements in documents, it would be harder catch the MISC la- beled entities as well. However, thanks to our Wikipedia labels, this problem might be mitigatable.
Missing mentions. (e), one of three city names ("Manaus") is missing the LOC label CoNLL-03 and Reiss. Similarly in (f), where "Thome" was missed as a person's 's name in both, or (g), where we decided to add label to "Fascist".
We leave it up to the researchers to choose which variant of CLEANCONLL is better fit for their use case.
Boundary problems. In (h), an incorrect sentence boundary leads missing part of the mention ("Hamlet Cup"). Interestingly, Reiss correct the sentence, but still are miss- ing the full mention. In (i), missing spac- ing around the dash is corrupting the labels: "GOETCHL" is person, "ALPINE" should be labeled, though the type is arguable. In (j). we decided extend the mention "Arthur Yates and Co Itd".
A.5 Qualitative Examples for Errors in CoNLL-03
We present examples for errors and inconsistencies found in the previous corpus versions (CoNLL-03 and the version by Reiss et al. (2020)) in compari- son to their annotation in CLEANCONLL - both before and after the reverts to MISC in Phase 3. See Table 7 for the listed examples with their full NER labels. We find:
Label inconsistencies. In both (k) and (I), "Australia" refers to national sports teams. Whereas the CoNLL-03 guidelines specifi-
Wrong labels. Some labels are outright in- correct, like (a), where a person's 's first

---

cally state use LOC in this case (so this is not really an annotation error per se), we dis- ambiguate between references to the country or sport team and use ORG accordingly. The Reiss labeling is inconsistent: LOC in (k) and ORG in (I) introducing inconsistency, very frequent source of noise in their dataset. Simi- larly, both (m) and (n) are structured "[Teaml] at [Teea2]"" While "Team2" is labeled conse- quently LOC in CoNLL-03 (an understand- able ambiguity, since "Team1" is both playing against "Team2" and at "Team2"'ss home site), Reiss introduce inconsistency by changing the label to ORG in some cases (n), but keeping LOC in others (m).
on Technology", "German-born U.S. biolo- gist Max Delbruck", "the current NATO-Ied peace force"). Whereas these cases mostly have MISC before (sometimes they are unla- beled), we decided to consequently use the label that the entity part (" ("Washington", "Ger- man", "NATO") would get on its own.
Oftentimes mentions could be labeled either as one longer or as consisting of several shorter mentions ("the 11th Circuit U.S. Court of Appeals in Atlanta", "the 1994 Lilleham- mer Winter Olympics"), We split them up. if the parts independently still form a frequent mention their own ("Atlanta", "Lilleham- mer"), but merge if the parts are infrequent or hard to interpret ("1 11th Circuit U.S. Court of Appcals").
A.6 Updated label guidelines for CLEANCONLL
During our relabeling phases, we updated some labeling guidelines compared to the label usage in original CoNLL-03, Reiss et al. (2020) and Wang et al. (2019). All of these were done to ensure better consistency throughout the corpus. More concrete:
All of CoNLL-03, the version by Reiss et al. (2020), and the one by Wang al. (2019), use the MISC label for adjectival affiliations ("Uzbek striker Igor Shkvyrin", "the Italian club"). Our initial CLEANCONLL labels use ORG or LOC, according to the referred en- tity. However, in Phase 3 (recall Sec. 2.3) we revert these cases MISC, in final CLEAN- CONLL, the label usage is conform the pre- decessor versions. Note that do distribute the labels before the reverts as well.
National sports teams are frequently referred to by country name ("Japan began the defence of their Asian Cup title"). The original anno- tation guidelines from CoNLL-03 state that in this case the country name should still be labeled as LOC nonetheless (Chinchor et al., 1999). We change this and use ORG in these cases because we deem it more consistent to other (not national) sports team usages ("dur- ing Milan's 2-1 defeat" or "the Colts who played their last home 13
A.7 Details on Entity Classification Model
In the second round of cross-checking (Sec. 2.2.2) we again trained 20 models in cross-checking fashion for spotting inconsistencies, However, this time we simplified the task and let the models clas- sify already given fixed mention spans into NER types, i.e. to perform only Named Entity Classi- fication and not Detection. Instead of sequence tagging like the previous cross-checking round, we used a span tagging model that represents the given mentions as concatenation of the first and last token's 's contextualized embeddings. Putting aside the task of mention detection allows the model really focus on the label semantics. In contrast, recall that in the third round of cross-checking, we instead use a model that only focuses on mention detection and ignores NER elssification.
There are several other cases where we ob- served label variation and decided on a fixed guideline to ensure consistency. Whereas there is label variation between LOC and ORG for c.g. airports, hospitals, schools, uni- versities, label them consistently as ORG, For sports leagues, sport events and stock ex- changes we use MISC consistently.
Several mentions include a compound with an e.g. location and adjective or adverb Consumer Project
("the Washington-basee
Beeis et al. (2020) mostly do so as well, though they do not state this in their paper however not consistently, see examples in Appendix A.5. Wang et al. (2019) do not change these cases.

---

Sentence (part)
Labels in CoNLL 03 (Tjong
Labels in Reiss (Reiss et al., 2020)
Labels in CLEAN- CONLL before Phase 3
Labels in CLEAN- CONLL after Phase 3
Kim Sang and De Meulder. 2003)
(a) 11. Florence Mas-
[Florence LuC: [aanada]]ER [France]LoC
Florence Mas- nada]rER [France]Loc
[Florence Mas-
[Florence Mas- nada]PER [France]Loc
nada (France) 133
nada|PER [France]Loc
(b) Green and black face paint com- pleted his disguise.
[Green]rER
[Green]rER
(c) Favourite: Mind Games (7-4) fin- ished 4th
[Mind Games]PER
[ind GamesJpER
[Mind GamesJMISC
[Mind Games MMSSC
(d) UK bookmakers William Hill said on Friday they have lengthened the odds of Conservative victory
[UK]Loc
[UK]
[UK]uoc
[UK]Loc
[William Hill]PER
[William Hill]PER
[William HillJORG
[William HillJORG
[Conservative |MSC
[Conservative|Misc
[Conservativejori
[Conservative |Msc
(e) the cities of Man- aus. Sao Paulo and Rio de Janeiro.
[Sao Paulo]LoC [Rio de Janeiro]Loc
[Sao Paulo]LoC
[Manaus]Lod
[Manaus]
[Rio de Janeiro]Loc
[Sao Paulo]Loc
[Sao Paulo]Loc
[Rio de Janeiroji.oc
[Rio de Janciro]
(1) With the score tied 1-1 in the ninth, Thome hit 2-2 pitch
[Thome|pER
[Thome JPER
(g) the granddaughter of Italy's 's Fascist dictator Benito
[[tal]]oc [Benito Mus- solini]PER
[Ital]Joc [Benito Mus-
[Italy] Juoc
[ttal]]uce
[Fascist MISC
[Fascis]]MISC
solini]PER
[Benito Mus-
[Benito Mus-
Mussolini
solini Jolini]PER
solini]PER
(h) (Results the Hamlet) Cup tennis tournament
[Cup]MISC
[Cup]MISC
(Hamlet Cup]MISC
[Hamlet Cup]MISC
ALPINE SKIING- GOETCHI WINS
[WORLD CUP]MIS
|WORLD CUPJMISC
[ALPINEJ [GOETCHI_JPE [WORLD
[ALPINE]I oC [GOETCHL.JpER [WORLD CUPIMISC
WORLD CUP DOWNHILL,
CUP]MusC
( NOTE: Arthur
[Arhhrr Yates and CoJORG
|Arthur Yates and CoJORG
Arthur Yates and Co ItdJORG
[Arthrr Yates and Co ItdJORG
Yates and Co ltd is garden products group.
(k) the man who kicked Australia to defeat
[Australia]Loc
[Australia]Loc
[Australia]oRG
[AustraliaJoRG
(1) He will not be considered for a test match against Australia starting on Octoher 10
[Australia]Loc
[AustraliaJorG
[AustraliaJorg
[Australia]ORG
m) LA CLIPPERS AT
ILA CLIP-
ILA CLIP-
ILA CLIP-
[LA CLIP-
NEW YORK
PERSJORG
PERS]ORG
PERS]ORE
PERSJORG
INEW YORK]LoC
INEW YORK]LOC
INEW YORKJORG
INEW YORKJORG
CALGARY AT BOSTON
[CALGARY]ORG [BOSTON]LOC
[CALGARY]ORG [BOSTONJORG
[ALGARY]JRG [BSTOONJRG
[CALGARYJORG [BOSTONJORG
Table 7: Example sentences or sentence parts and their respective labels in CoNLL-03, Reiss and our two CLEANCoNLLversions. 'In CoNLL-03 the sentence is incorrectly split before Cup. While in Reiss this corrected, the label still is only placed on Cup, not on Hamlet Cup. PI CONI.L-03 and Reiss, the spacing around the dash is missing, making ble. In CLEANCONLLwe added the spaces and missing mention.
GOETCHL" one token and therefore making the correct labeling impossi-

---

CoNLL-03
Wang
Reiss abs
CLEANCONLL'
CLEANCONLL
AIDA
abs
%
abs
%
abs
%
abs
abs
10,645
30.3
10.623
30.2
10,103
28.9
12,089
34.3
9,399
ORG
9,323
26.6
9.377
26.7
9,922
28.4
10,701
30.4
10,492
PER
10,059
28.7
10,060
28.6
9,983
28.6
9,947
28.2
9,947
28.2
MISC
5,062
14.4
5,083
14.5
4,933
14.1
2,520
7.1
5,419
15.4
entities
35,089
35,143
34,941
35,257
35,257
27,814
20,744
20,744
20,617
20,617
20,617
20,584
Table Statistics of both label set variants from CLEANCONLL, in comparison to previous versions. 'CLEANCONLL lahels before reverts in Phase 3.
Reiss version
CLEANCONLL
vs. CONLL-03
AIDA
abs
abs
%
Label unchanged
33,340
95.0
URL unchanged
32,699
92.7
Label updated
743
2.1
URL updated
322
0.9
New mention
207
0.6
URL added
1,955
5.5
Unknown
651
1.9
Unknown
281
0.8
# entities
34,941
entities
35,257
entities orig.
35,089
entities orig.
27,814
updated sent.
969
4.7
updated sent.
2,077
10.1
Table 9: NER label updates, comparing mentions in Reiss version with CoNLL-03.
Table 10: Comparing Wikipedia labels in CLEAN- CoNLL to the original ones from AIDA.
Unknown: We map and compare the lahels using sentence Due updated CoNLL-03, some cases we cannot find direct equivalent, we exclude the respective mentions here. New mention: Includes both completely new well newly bounded mentions.
boundarics from original
A.8
Additional Dataset Statistics
CLEANCONLL concerning the Wikipedia URLs: 92.7% of labels are unchanged in comparison AIDA, 0.9% were updated from our annotators, and 5.5% of mentions (1,955) got a newly added Wikipedia label.
A.8.1 Dataset Statistics Full comparison
Table 8 (an extension to Table 2) we present full comparison of label distribution and general dataset statistics, comparing CLEANCONLL to previous versions, as well as AIDA. We include both versions of CLEANCONLL NER labels, so the labels before and after the reverts MISC in Phase 3 (recall Sec. 2.3). Most striking is the decrease in MISC usage before the reverts, and the subsequent increase of the LOC and ORG, coming from labeling adjectival affiliation as either ORG or LOC, according to their original label that we derived through the entity linking approach. In the final version, the distribution is more similar to the previous versions.
A.8.3 Label Updates: Before and after MISC reverts
In Tables 11 and 12 we contrast the label updates introduced by CLEANCONLL, comparing it to the previous versions. We present the state of NER labels before (Table 11) and after (Table 12) the final Phase 3 reverts from Sec. 2.3. Note that the amount of label updates before applying our spe- cial handling of adjectival affiliations (and keeping ORG and LOC instead) is accordingly much higher (14.9% label updates, 20.2% updated sentences) compared to the final 7.0% label updates (10.3% sentences). Refer Appendix A.4 for discussion of which label set use.
A.8.2 Label Updates: Reiss and Wikipedia In Table 9 present the amount of label up- dates that the version by Reiss al. (2020) in- troduced in comparison to CoNLL-03 (see the CLEANCONLL updates in Table for compar- ison).
Table 10 shows the amount of updates from

---

CLEANCONLL (before Phase 3)
vs.
CoNLL-03 abs
Wang abs
Reiss
Label unchanged
29,348
83.6
29,452
83.8
30,671
87.8
Lahel updated
4,595
13.1
4,557
13.0
4,023
11.5
New mention
624
1.8
558
1.6
533
1.5
Unknown
690
2.0
690
2.0
30
# entities
35,257
35,257
35,257
# entities orig. 35,089
35,143
34,941
# updated sentences
4,169
20.2
4,095
19.9
3,550
17.2
Table 11: NER label updates, comparing CLEAN- CONLL before reverts in Phase to the previous versions.
CLEANCONLL (after Phase 3) CoNLL-03 abs
Wang abs
Reiss
abs
%
Label unchanged
32,110
91.5
32,224
91.7
33,488
95.8
Label updated
1,833
5.2
1,785
5.1
1,206
3.5
New mention
624
1.8
558
1.6
533
1.5
Unknown
690
2.0
690
2.0
30
0.1
# entities
35,257
35,257
35,257
# entities orig. 35,089
35,143
34,941
# updated sentences
2,115
10.3
2,027
9.8
1,398
Table 12: NER label updates, comparing CLEAN- CONLL after reverts Phase 3 the previous versions.


Reviews:
[{"review_id": "aIlHqS14qk", "reviewer": "Reviewer_L4cT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your comments, we are very happy that you consider our paper and resource helpful for the community and appreciate your time in writing the review! We hope the dataset can indeed help development and fair comparison of future NER architectures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- New corrected version of the well-known CoNLL-03 dataset.\n- Corrections and additions made to the inconsistencies present in the annotations\n- The corrections improve models trained on the data. \n- The data will be made freely available. ", "reasons_to_reject": "- I cannot see any. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a new version of the well-known CoNLL-03 corpus for NER. The new version, CleanCoNLL, \nhave undergone a comprehensive manual relabeling assisted by automatic consistency checking used during the manual re-annotation process to flag inconsistencies. The authors claim that that 7% of all labels in CoNLL-03 were corrected. The corrections were carried out on all three splits, and the dataset will be released and made available following the same copyright as the original data.\n\nCleanCoNLL also contains more consistent NER annotations and integrated a layer of entity linking annotations. These latter are leveraged from Wikipedia and Wikidata. The authors also carried out three iterative rounds of cross-checking to identify potential inconsistencies. These were subsequently manually examined and corrected if necessary. The authors have also developed and added special handling for adjectival affiliation entity names.\n\nThe authors give an extensive experimental evaluation of the dataset compared to its original version, and using a set of stat-of-the-art (sota) approaches (Flair model, FLERT model, Biaffine tagging approach, and the ACE model (Wang et al., 2021)). They show that using the new clean dataset, sota approaches reach higher F1 scores with less misclassifications due to annotation errors. These types of errors dropped from 47% to 6% which a considerable drop. \n\nThe paper is well written and clear. This is a great contribution which I believe will help develop sota NER models for years to come. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yRuRdQUBRJ", "reviewer": "Reviewer_xzrG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful comments, the pointer to the paper to consider, as well as for your fair criticism concerning the limitations/transferability of the approach!\n\nYou are right, we only consider the English version for now and will follow your suggestion to state this clearly in the abstract of the paper. This follows our initial relabeling round through the Named Entity Linking annotations that are exclusively available for the English CoNLL03 dataset (AIDA). However, though obviously not done in the present work, we do think that the proposed approach of a) leveraging NEL and Wikipedia/Wikidata entities for deriving automatic NER labels, and b) checking consistency via iterative cross-checking is generally applicable for other datasets and languages. The former obviously requires EL annotations while the latter is completely independent and in theory could be seen as a default consistency check for any dataset. We do hope to apply the approach to more datasets and if possible other languages in future work!\n\nRegarding novelty and usefulness, our model evaluation and inspection of model predictions do point to gains in performance and a more fair comparison when using our corrected dataset in training and evaluation compared to the original as well as other corrected versions of the dataset. In particular (see Section 4.2), evaluations on earlier CoNLL-03 versions had a large share of correct predictions falsely counted as errors due to annotation noise, making it difficult to analyze the remaining errors made by state-of-the-art systems. With CleanCoNLL, this noise is greatly reduced, allowing us to analyze the remaining errors, and indicating that the theoretical upper bound even on high resource, coarse-grained NER is not yet reached.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper contributes a new version of relabeled CoNLL NER dataset with fewer annotation errors\n- the paepr conducts comprehensive analysis on the annotation evaluation, result evaluation, etc. readers can understand the statastics of the dataset much better through the thwe analysis\n- it's interesting to see the techniques / tasks of entity linking and cross checking to be applied to fix annotation errors", "reasons_to_reject": "I didn't see major weaknesses regarding the technical aspects. My main doubt or question though, is do we need another version of CoNLL-2003 English. NER people have been fully aware of the annotation error issues from the CoNLL dataset, as it's also mentioned in the paper. From the modeling anfd evaluation perspetive, fixing these errors also have limited ability to distinguish the model performance under the context of CoNLL-2003 English.\nPersonally, I would be more excited to see the labeling efforts and techniques being applied to datasets that are larger or in other langugages. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a relabeling effort for fixing the annotation error from the CoNLL-2003 English dataset. They propose a pipeline that are composed of entity linking to AIDA, cross-checking and fixing adjectival affiliations, which greatly reduce the labeling noise comparing the the original version of the dataset.  ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "it's always a good idea to mention in the abstract that which language the paper is working on (e.g., #BenderRule). For this paper, I would be clear that it's only about CoNLL 2003 English dataset."}, "extra_text": {"Missing_References": "There is another relabeling effort on CoNLL English: https://ojs.aaai.org/index.php/AAAI/article/view/6276 (Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study). It will be nice to add this dataset into comparison as well."}}, {"review_id": "6lFBuoTqDc", "reviewer": "Reviewer_NcfC", "summary": "", "questions": "- general : in this work, were annotators the same persons that conceived the whole correction framework? If so, don't you fear it would introduce a bias (pushing for corrections)?\n- general : most examples given in this paper concerns sports, and especially one phenomenon (LOC/ORG confusion for teams). Does this actually represent most of the improvements in CleanCoNLL, or is it just an example? Could you clarify that?\n- line 255 : source of the Wikipedia label provided to annotators (AIDA? prediction?)\n- line 255 and Figure 3 : the annotators did know which label was the original and which was predicted. Don't you think this might introduce a bias in the correction?\n- line 348 : this line states that there have been boundary corrections, however section 2 implies only label modifications and entity additions, could you clarify at which phases and how boundaries were corrected?\n- Table 3 : are there significant differences in the modification rate between train and test splits? Line 047 implies that this would be possible. If so how would that affect experiments of section 4.2?\n- line 442 : please explain why you chose to examine errors of FLERT whereas Biaffine seems to yield better results?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your elaborate review with a lot of helpful comments and fair questions! We are very happy that you find the paper and methodology convincing and clear, as well as deem our resource helpful and point out our assessments of annotation quality!\n\nAs for your specific questions:\n1. Yes, the annotators do overlap with the authors. You are right that this could influence annotations and pushing for modifications. However, this could be seen also as a good thing because having worked extensively with the dataset and quality assessment, the annotators were very aware of inconsistencies and presumably had a more round view of annotation guidelines which pushes for consistency rather than “conservatively” keeping the labels as is. However, appointing the same annotators for manual evaluation (deciding between annotation and model errors) is indeed not ideal but unfortunately could not be avoided due to time constraints and resources. We will state this more clearly in the final version.\n2. The observation that most of the examples concern sports is interesting! Many articles in CoNLL (especially the test set) are sports themed so these do make up for a considerable portion of changes, but by no means all, as is already shown by Table 1 and Table 7. For example, hospitals/schools show some similar ambiguous handling, as do non-sports governmental organizations such as armed forces. We will make sure to have more diverse examples in the final version! \n3. Yes, the annotators were provided with information as to where a label suggestion comes from, e.g. gold annotation, model prediction or automatically derived via Wikipedia/Wikidata heuristics. \n4. Refer to 3. and 1. above. You are right, this could well have been done “blindly” with some benefits. However, stating the label source more clearly was also helpful: It made inconsistencies in the gold labels more strikingly clear to annotators (as well as could have shown general problems with our automatic NER labels!), and enabled them to mark cases for specific care (see line 265), which could then be incorporated in the succeeding rounds.\n5. As stated in Section 2.2.2, in the third round of cross-checking (see line 291 “3. Only entity detection.”), we specifically mined for boundary problems and corrected them.\n6. This is an interesting idea, thanks! Unfortunately we have not checked for differences of correction rates in the splits but will do so and add them to Table 3.\n7. We used FLERT since the Biaffine approach only achieved the best numbers on CleanCoNLL. On the other CoNLL variants we evaluate, FLERT slightly outperformed or was on par with the Biaffine approach. Using Biaffine would therefore have slightly biased the evaluation to favor CleanCoNLL, which we wanted to avoid for fairness. \n\nWe hope this helps resolve some of your questions! Furthermore, thank you for marking typos and the helpful hint regarding a more inclusive color usage in Figure 3!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Shares a very useful resource\n\n2. Convincing methodology and quality assessment\n\n3. Paper is clearly written, describes thoroughly the motivation, the methodology, the results and the limitations", "reasons_to_reject": "None that I can think of.\n\nAs the authors state in the Section \"Limitations\", the methodology they used is not apply on other corpora, especially domain-specific corpora. However I'd argue that this paper might inspire future dataset creators for quality checking methods.", "questions_to_authors": "- general : in this work, were annotators the same persons that conceived the whole correction framework? If so, don't you fear it would introduce a bias (pushing for corrections)?\n- general : most examples given in this paper concerns sports, and especially one phenomenon (LOC/ORG confusion for teams). Does this actually represent most of the improvements in CleanCoNLL, or is it just an example? Could you clarify that?\n- line 255 : source of the Wikipedia label provided to annotators (AIDA? prediction?)\n- line 255 and Figure 3 : the annotators did know which label was the original and which was predicted. Don't you think this might introduce a bias in the correction?\n- line 348 : this line states that there have been boundary corrections, however section 2 implies only label modifications and entity additions, could you clarify at which phases and how boundaries were corrected?\n- Table 3 : are there significant differences in the modification rate between train and test splits? Line 047 implies that this would be possible. If so how would that affect experiments of section 4.2?\n- line 442 : please explain why you chose to examine errors of FLERT whereas Biaffine seems to yield better results?", "paper_topic_and_main_contributions": "This paper describes an effort for improving the quality of the annotation of the CoNLL corpus. The presented methodology consists in several two-step correction rounds. In the first step the authors leverage automatic methods for highlighting potential errors in the original corpus. In the second step authors mobilized annotators for judging the cases raised in the first step.\nThe authors describe three methods for highlighting potential errors:\n- comparison of CoNLL NE labels with AIDA EL labels\n- cross-checking by training NER tools and label classifiers\n- training a NER model specifically for adjectival affiliation\n\nThe authors also evaluated the quality improvement in two ways:\n- manual reassessment of the corrected annotation\n- comparison of SOTA methods trained on different versions of the dataset", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 047 : of all sentences -> in all sentences\n- Figure 3 : please change background color, green and blue texts on dark red are difficult to read for some people\n- line 270 : Some -> some"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a substantial contribution in the form of a corrected and enhanced version of the CoNLL-03 corpus, which helps improving reliability and accuracy of Named Entity Recognition (NER) evaluation. The main contribution of this work, apart from the valuable new dataset, is the interestingly novel methodology employed for the automatic correction and cleaning of the corpus. Although it’s transferability to other languages still has to be proven experimentally, it appears promising.  

**Pros:** 

- The paper is well written and clear 

- The methodology is convincing and sound 

- Produced a new corrected version of the well-known CoNLL-03 dataset, a very useful resource for the community  

- gives detailed thorough statistics of the dataset, to assess its quality 

- creative / innovative applications of known techniques to the task of annotation correction and cleaning 

- the methodology seems portable to other languages


The work has no important weakness. It is interesting, mature, and the dataset it presents is likely to have a long-term impact on the field.
Decision: Accept-Main