Venue: NAACL 2024

# Cner: Concept and Named Entity Recognition

Giuliano Martinelli\({}^{1}\), Francesco Maria Molfese\({}^{1}\), Simone Tedeschi\({}^{1,2}\)

Alberte Fernandez-Castro\({}^{2,3}\) and Roberto Navigli\({}^{1}\)

\({}^{1}\)Sapienza NLP Group, Sapienza University of Rome \({}^{2}\)Babelscape \({}^{3}\)Roma Tre University {martinelli, molfese, tedeschi, navigli}@diag.uniroma1.it, alb.fernandezcastro@stud.uniroma3.it

###### Abstract

Named entities - typically expressed via proper nouns - play a key role in Natural Language Processing, as their identification and comprehension are crucial in tasks such as Relation Extraction, Coreference Resolution and Question Answering, among others. Tasks like these also often entail dealing with concepts - typically represented by common nouns - which, however, have not received as much attention. Indeed, the potential of their identification and understanding remains underexplored, as does the benefit of a synergistic formulation with named entities. To fill this gap, we introduce Concept and Named Entity Recognition (CNER), a new unified task that handles concepts and entities mentioned in unstructured texts seamlessly. We put forward a comprehensive set of categories that can be used to model concepts and named entities jointly, and propose new approaches for the creation of CNER datasets. We evaluate the benefits of performing CNER as a unified task extensively, showing that a CNER model gains up to +5.4 and +8 macro F1 points when compared to specialized named entity and concept recognition systems, respectively. Finally, to encourage the development of CNER systems, we release our datasets and models at [https://github.com/Babelscape/cner](https://github.com/Babelscape/cner).

## 1 Introduction

In the age of big data, the extraction of valuable knowledge from unstructured text is crucial for a wide range of applications, from Information Retrieval to Text Mining Grishman (2015). Within this context, _"nouns and noun phrases have a special status in describing the concepts that people are interested in searching for"_ Manning et al. (2008). Consequently, to work at their best, Natural Language Processing (NLP) systems should not only leverage the information about named entities, but also that concerning nominal concepts, as the nature of the two is strongly intertwined. A prototypical task in which such interaction is intrinsic to its very nature is Relation Extraction Pawar et al. (2017), where the triples extracted from text often connect named entities with concepts, e.g. (_Carlsen_, plays, _chess_) or (_ChatGPT_, is a, _chatbot_). As a second example, consider Coreference Resolution Sukthanker et al. (2020), where the named entities mentioned in a text (e.g. _Joe Biden_) are paired with subsequent expressions referring to them, including common nouns such as _president_ or _state leader_. Indeed, many tasks have been shown to benefit from the recognition of named entities Molla et al. (2006); Durrett and Klein (2014); Khosla and Rose (2020); Tedeschi et al. (2021); Huguet Cabot et al. (2023), but little attention has been devoted to the identification of concepts.

In this paper, we fill this gap by, first, introducing Concept Recognition (CR), namely, the task of identifying and classifying concepts into pre-defined semantic types, and, second, harmonizing CR with the well-established task of Named Entity Recognitio (NER). As shown in Figure 1, our new

Figure 1: Two examples of CNFER annotations where both nominal concepts and named entities are tagged with the proposed categorization.



joint formulation enables a denser, richer, and more cohesive semantic annotation. Specifically, we put forward the following innovative contributions:

1. We introduce a novel NLP task, namely, Concept and Named Entity Recognition (CNER), and a unified set of categories which are tailored to semantically annotating both nominal concepts and named entities.
2. We put forward an automatic procedure for the creation of CNER, NER and CR training data, and manually produce a dataset of 2,000 sentences for model evaluation.
3. We study the benefits of performing CNER as a unified task compared to performing NER and CR separately.

Finally, to encourage the development of CNER systems, we release our datasets and models to the research community at [https://github.com/Babelscape/cner](https://github.com/Babelscape/cner).

## 2 Related Work

We now review established approaches that deal with the identification and classification of concepts and named entities. Specifically, we observe that the vast majority of works in the literature focus exclusively on either concepts or named entities. This is particularly relevant because, as we will see, the currently available categorizations are not suitable for a unified task that integrates both concept and entity recognition.

### Named Entities

In the last two decades, researchers have devoted significant attention to named entities, with Named Entity Recognition (NER) being a key popular task aimed at locating named entities in unstructured free-form text and then assigning them to predefined semantic types Nadeau and Sekine (2007); Li et al. (2022). Although NER datasets were primarily constructed to classify named entities into a limited set of categories, namely Per, Org, Loc and MiscTjong Kim Sang (2002); Tjong Kim Sang and De Meulder (2003), recent approaches have proposed unwrapping the Misc category - containing miscellaneous entities - into fine-grained categories in order to explicitly identify more specific entity types. Among these, OntoNotes stands out as a well-established resource that identifies 18 categories for named entities Pradhan et al. (2012). However, its annotations do not cover instances of several classes, including food, animals, or celestial bodies, among others. This was instead addressed in subsequent works, i.e. Few-NERD Ding et al. (2021) and MultiCoNER II Fetahu et al. (2023), which proposed named entity categorizations of 66 and 36 categories, respectively. Although these categorizations are more comprehensive and fine-grained, they tend to focus on excessively detailed semantic types. For instance, the category Island is well-suited for classifying named entities like _Lesbos_, but it proves less suitable for nominal concepts, as only a few, such as _islet_, fall within its scope. Other examples of categories that are not suitable for the classification of concepts are Library, Airport and Hotel, _inter alia_.

Ultimately, OntoNotes strikes a good balance in terms of the level of granularity of its categories and, as later discussed in Section 3.2, blends well with a complementary set of abstract semantic types needed for concepts, which includes relations, properties, and feelings, among others.

### Concepts

The closest task to what we have called Concept Recognition is Word Sense Disambiguation Bevilacqua et al. (2021); Navigli (2009, WSD), the main difference being that in WSD a word is tagged with one of the senses it can denote in a given inventory (e.g. WordNet), whereas in CR a word is tagged with a more general category that is independent of the word itself. Assigning fine-grained senses to words has been found to be extremely difficult Izquierdo et al. (2015); Maru et al. (2022), which has led to coarse-grained disambiguation approaches aimed at reducing the number of categories to choose from. Vial et al. (2019) leverage hypernymy relations to reduce WordNet granularity by exploiting its taxonomy graph, releasing a coarser-grained inventory with 39K labels. However, the high number of proposed categories is too specific to be considered as a good candidate categorization for CNER.1Izquierdo Bevia et al. (2007), instead, exploit WordNet relations to automatically extract a set of fundamental senses, called Basic Level Concepts, to which all other senses are mapped. However, depending on a threshold, this approach outputs hundreds or thousands of Basic Level Concepts, which are again too fine-grained.

 

Thanks to their being much closer to the granularity of OntoNotes for entities, WordNet lexicographer files2 can, instead, be used as coarse categories for nominal concepts, where each category contains synsets with the same PoS tag and a general semantic type (e.g. Noun_Feeling). Since this resource is highly reliable and has a strong psycholinguistic grounding, we use it as a starting point for our CNER categorization, as we will discuss in Section 3.2.

Footnote 2: WordNet synsets are organized into 45 lexicographer files based on syntactic categories and logical groupings.

### Entities and Concepts

A categorization that spans both concepts and entities has been proposed in the context of the ultra-fine entity typing3 task Choi et al. (2018); UFET. Here, the objective is to produce a three-level annotation by selecting tags from coarse, fine, and ultra-fine sets of labels consisting of 9, 121 and \(\sim\)10K types, respectively. Nevertheless, the extreme granularity of the ultra-fine labels makes it difficult for systems to output the right categories, as reported by the authors, while the coarse-grained types are too generic to be considered a good candidate for our fine-grained objective. The fine set of categories comprising 121 tags, instead, not only contains categories that are too specific (e.g. Doctor, Coach), but also fails to categorize commonly used concepts, such as relations (e.g. _inferiority_, _brotherhood_), date/times (e.g. _November_, _September 3rd_), feelings (e.g. _love_, _anger_), and psychological features (e.g. _cognition_, _thought_), _inter alia_. Finally, we remark that the proposed ultra-fine entity typing task disregards the identification part. This is in marked contrast to CNER, where each and every span of text denoting concepts or entities has to be identified and tagged.

Footnote 3: Entity Typing is often referred to as the second step of NER, aiming at classifying pre-identified entity mentions.

## 3CNER task

We now formalize the Concept and Named Entity Recognition (CNER) task and then introduce our process to obtain CNER categories.

### Task formulation

Formally, CNF is a sequence labeling task whose goal, given a sequence of tokens \(t=(t_{1},t_{2},...,t_{n})\), is to identify the spans of text \(S\) corresponding to nominal concepts and named entities, and categorize each of them into a predefined set of labels \(L=\{l_{1},l_{2},...,l_{m}\}\). Specifically, a text span \(s_{ij}\in S\) is defined as a contiguous sequence of tokens \(s_{ij}:=(t_{i},t_{i+1},\ldots,t_{j})\), with \(1\leq i\leq j\leq n\).

### Categories

As anticipated in Section 2, we draw upon the robust cognitive foundations of WordNet Miller (1995) and OntoNotes Pradhan et al. (2012) to produce a unified set of categories. Specifically, we leverage the completeness and broad semantic coverage of lexicographer files for nominal concepts, and integrate them with the widely-used semantic categories for named entities available in OntoNotes. Figure 2 shows the resulting 29 CNER categories (inner column) together with their alignment with the OntoNotes and WordNet ones (left and right columns, respectively). In particular, every category in OntoNotes and WordNet has a counterpart in our categorization, whereas the reverse does not hold. Specifically, i) every category highlighted red has a direct link with a WordNet category (e.g. Animal), ii) every category highlighted in yellow corresponds to an OntoNotes category (e.g. Law), and iii) every category in a white cell is grounded on both resources (e.g. Media).

To further assess the validity of such categorization and ensure that each instance is assigned to a reasonable category (i.e. semantically appropriate and neither too general nor too specific), we conducted a manual review of the most prominent WordNet synsets, i.e. the top 500 synsets ranked by their number of descendants in the WordNet taxonomy: a group of three human annotators independently reviewed this set of synsets and either assigned a category that is present in the already available categorization or proposed a new one, i.e. a category whose semantics better defines such an instance and that is well distinguished from the others. The result of this step is twofold. First, we obtained 500 synsets tagged with their CNER category (that we refer to as _seed synsets_), which will be employed as a starting point for our automatic annotation procedure, explained in Section 4.1. Second, we identified six new categories, highlighted in green in Figure 2, that complement the WordNet and OntoNotes ones. The practical implication of the latter result is that, for example, biological concepts such as _molecule_, _protein_ and _organism_ will no longer be tagged with the Object category, but as Biology. Analogously, terms such as _planet_, _quasar_ and _star_, that would all have been included in the Object category, will now 

be assigned to a specific category named Celestial. The same reasoning can be extended to the new categories Culture, Disease, Discipline and Super. For completeness, in Appendix C, we provide a textual description along with instance examples for each CNER category.

## 4 Resources

In the previous section, we described the procedure we used to obtain a new comprehensive categorization specifically designed to jointly capture concepts and named entities. We now provide datasets to enable the training and evaluation of CNER models. Crucially, we note that none of the existing datasets provides full coverage of both concepts and entities, usually expressed by common and proper nouns, respectively.

To fill this gap, we present a methodology for automatically annotating a large corpus of sentences with CNER categories (Section 4.1), which we refer to as \(\textsc{Cner}_{silver}\). Then, we describe the annotation procedure we adopted to produce \(\textsc{Cner}_{gold}\), a manually-curated dataset for model evaluation (Section 4.2). Finally, in Section 4.3 we present a detailed analysis of our two resources compared to the existing ones.

### \(\textsc{Cner}_{silver}\)

We propose an automatic approach to the creation of a large-scale training set for the CNER task. Our goal is, first, to disambiguate concepts and entities mentioned within text, and, second, to transform the resulting annotations into CNER categories. We choose the English Wikipedia as our corpus because it offers a large number of heterogeneous texts spanning all domains of knowledge. To ensure high quality, we restrict ourselves to the subset of articles that the Wikipedia community has deemed to be "good" or "featured" and apply our annotation strategy to these articles.

Importantly, Wikipedia articles contain a few terms that are linked manually to other articles, but many other terms which are left unlinked, leading to an issue of _sparsity_. Therefore, to ensure high density of annotations, we perform three main steps:

* Because the Wikipedia guidelines specify that only the first occurrence of a term should be linked,5 we propagate that link to all other occurrences of the term on the same page; Footnote 5: Wikipedia Guidelines.
* As most of the mentions linked in Wikipedia articles refer to named entities, we ensure coverage and disambiguation of all common nouns with concepts in BabelNet [20, 26] through the application of Word Sense Disambiguation;
* All remaining entity mentions are tagged with standard NER.

Figure 2: Comparison of our CNER categories (center) with the OntoNotes (left) and WordNet ones (right).

 

Once each article has been fully annotated, we transform each tag, be it a Wikipedia hyperlink, a BabelNet concept, i.e. synset, or a NER category, into its CNER category. In what follows, we delve into the details of our _taxonomy-based tagging_ strategy for annotating Wikipedia articles, and discuss the main heuristics for _solving sparsity_.

Taxonomy-based taggingTo annotate each Wikipedia hyperlink in an article \(W_{i}\) with a CNER category \(c\), we first retrieve the corresponding BabelNet synset and map it to one of the 500 seed synsets introduced in Section 3.2. Specifically, given that each Wikipedia hyperlink corresponds to a synset node in the BabelNet taxonomy, in order to assign a CNER category to the hyperlink \(w_{j}\) we start from the corresponding BabelNet node \(n_{j}\) and navigate upward along hypernymy edges within the hierarchy. When a seed synset with category \(c\) is reached, we assign \(c\) to the node \(n_{j}\). In the case of multiple seed synsets reached at the same distance, we prioritize the most frequent category.

Solving sparsityAs a result of the previous step, each hyperlink in a Wikipedia article is annotated with a CNER category. However, as previously mentioned, hyperlinks in Wikipedia are sparse (cf. line 1 in Table 1). To address this problem, we apply a surface matching heuristic where, for each link \(w_{i}\) with an associated category \(c\) and a surface text \(t_{i}\), we propagate \(c\) to all text spans \(s\) in the same document such that \(s=t_{i}\lor s\in syn(w_{i})\), where \(syn(w_{i})\) is the set of synonyms of the BabelNet synset corresponding to \(w_{i}\). However, while this methodology is remarkably effective for named entities, it falls short when it comes to densely annotating concepts. Indeed, after the surface matching heuristic, only 15% of common nouns are tagged, in contrast to 55.3% of proper nouns.

To fill this gap, we complement the above strategy with a state-of-the-art Word Sense Disambiguation model, ESCHER Barba et al. (2021), to disambiguate each common noun in our dataset. As a result, we also obtain BabelNet synsets for the remaining unlinked common nouns, which we classify through the same taxonomy-based technique presented earlier. Finally, in order to annotate the remaining proper nouns with a CNER category, we use the Stanza NLP toolkit Qi et al. (2020). This toolkit produces named entity annotations using OntoNotes categories, which are then mapped to CNER categories by exploiting the one-to-one link between OntoNotes and CNER that was presented in Section 3.2. Remarkably, following the above-described process, we are able to annotate 98.4% of proper nouns, and 97.3% of common nouns, hence obtaining extremely dense annotations. This is in contrast to prior work as will be detailed later, in Section 4.3.

Table 1 reports an ablation study highlighting the distinct contributions of the previously described modules. Importantly, for each annotated span, we include explicit information on whether it is a concept (C) or a named entity (NE), which is essential for the training of the specialized NER and CR models, as will be detailed in Section 5.1. Since BabelNet provides this information for its synsets, we include it for each instance that is annotated via taxonomy-based tagging and WSD. The instances annotated using Stanza NER, instead, are all considered to be named entities except for dates and numbers. We then convert the dataset that has been produced employing the BIO tagging scheme.

### Cner\({}_{gold}\)

To enable a proper and rigorous evaluation of CNER models, we introduce an annotation procedure for constructing Cner\({}_{gold}\), a manually curated dataset. The annotation process started by formulating guidelines through a meticulous examination of examples within our Cner\({}_{silver}\) dataset. However, the task presented multiple challenges related to the annotation of multiword nominal expressions. First, we established that non-compositional expressions such as _white shark_ and _bald eagle_, which represent specific concepts, have to be identified as whole spans, while in compositional expressions, such as _black laptop_, only the noun has to be tagged. Second, in the presence of nested named entities, e.g. _Mr. Smith goes to Washington_, we decided to tag the largest available span denoting an entity, rather than tagging e.g. _Mr. Smith_ and _Washington_ separately. Third, we considered

\begin{table}
\begin{tabular}{l r r} \hline \hline
**Sparsity heuristic** & Noun & Propn \\ \hline Wikilinks & 7.1 & 33.2 \\ + surface matching heuristic & 15.0 & 55.3 \\ + WSD & 92.8 & 56.3 \\ + Stanza NER & 97.3 & 98.4 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Impact of the various modules for solving sparsity in Wikipedia articles. The first row indicates the percentage of common and proper nouns hyperlinked in Wikipedia articles without applying any heuristic.

 

the cases in which the annotator is uncertain about whether a specific span of text is an entity or not (e.g. _One flew over the cuckoo's nest_), and what its potential meanings are (e.g. _manihlapinnatqai_). To help annotators disentangle these cases, we allowed them to use external world knowledge, i.e. giving access to web search, Wikipedia, etc., to properly identify and classify spans of text. We point out that this procedure proved to be effective for the annotation task: in a sample of 100 sentences, annotators, on average, utilized external resources for more than one span per sentence.

To start the annotation task, we randomly sampled 2,000 sentences from \(\textsc{Cner}_{silver}\) and provided the annotators with a simple interface where each row displayed a specific token and its corresponding PoS tag. The annotators were asked to identify concepts and named entities by means of specific C vs NE tags and label them with the corresponding CNER categories by making their choice via a drop-down menu.

Because the dataset was annotated by a single expert linguist with a robust background in the annotation of lexical-semantic tasks, and an author of this paper, we asked two other annotators to label \(10\%\) of the data. We then calculated the inter-annotator agreement on such instances and obtained a Fleiss' \(\kappa\) score of 89.8, indicating optimal agreement. The overall annotation process took 2 months, with a total of 320 hours spent on the task. This process led to the creation of a corpus of 2,000 sentences and more than 56,000 annotated tokens, \(\sim\)22,000 of which were tagged with a CNER category.

### Dataset Statistics

In Table 2, we provide the statistics of our \(\textsc{Cner}_{gold}\) and \(\textsc{Cner}_{silver}\) datasets compared to OntoNotes. We observe that our training set comprises a significantly larger number of annotated sentences, spans and tokens. Furthermore, from the last two columns, we observe a considerably higher annotation density, highlighting that tagging both named entities and nominal concepts leads to a denser and richer annotation compared to focusing on entities only. In Table 2 we also present the statistics of UFET (cf. Section 2.3), but while their formulation encompasses the classification of both named entities and concepts, the annotation density is even lower than that of OntoNotes, as they disregard the identification part.

In addition, the pie charts in Figures 3 and 4 show the category balance in \(\textsc{Cner}_{silver}\) and \(\textsc{Cner}_{gold}\), respectively. Finally, in order to evaluate the quality of the automatic annotation procedure (Section 4.1), we compute the agreement over the set of 2,000 sentences reserved for \(\textsc{Cner}_{gold}\). The Cohen's kappa coefficient over this set of samples is \(\kappa=71.4\), indicating a substantial agreement between the automatic and manual procedures. We provide further dataset statistics in Appendix A.

## 5 Experimental Setup

We now describe the setup, including datasets (Section 5.1) and model architecture (Section 5.2), used to answer the following research questions:

* **(RQ1)** How does a competitive neural model fare on the CNER task?
* **(RQ2)** Can a CNER system perform on par with, or even better than, specialized NER and CR systems on the respective subtasks?

### Datasets

In our experiments we use several versions of our data:

* \(\textsc{Cner}_{silver}\) and \(\textsc{Cner}_{gold}\) are, respectively, the silver- and gold-standard datasets introduced in Section 4 covering both nominal concepts and named entities;

\begin{table}
\begin{tabular}{|p{28.5pt}|c|r|r|r|r|r|r|} \hline \hline
**Dataset** & **Type** & **\# Sentences** & **\# Tokens** & **\# Spans** & **\# Tagged Tokens** & **AVG \# Spans** & **Density \%** \\ \hline \hline OntoNotes 5.0 & Gold & \(76\,714\) & \(1\,388\,973\) & \(104\,151\) & \(190\,310\) & \(1.36\) & \(13.69\) \\ \hline UFET\({}_{crowd}\) & Gold & \(5878\) & \(154\,802\) & \(5994\) & \(17\,627\) & \(1.01\) & \(11.38\) \\ UFET\({}_{silver}\) & Silver & \(2\,272\,421\) & \(59\,500\,651\) & \(3\,121\,857\) & \(7\,016\,134\) & \(1.37\) & \(11.80\) \\ \hline \(\textsc{Cner}_{gold}\) & Gold & \(2000\) & \(56\,843\) & \(14\,730\) & \(22\,461\) & \(7.37\) & \(39.56\) \\ \(\textsc{Cner}_{silver}\) & Silver & \(317\,590\) & \(8\,725\,846\) & \(2\,282\,800\) & \(3\,403\,952\) & \(7.18\) & \(39.00\) \\ \hline \end{tabular}
\end{table}
Table 2: Comparison between our newly-introduced CNF resource, OntoNotes and UFET. AVG # Spans is the average number of tagged spans – either concepts or named entities – per sentence. Density % is the percentage of tagged tokens over the total number of tokens.

 

* \({\textsc{Ner}_{silver}}\) and \({\textsc{Ner}_{gold}}\) are the same datasets as above, in which only entity-related annotations are retained (i.e. concepts are mapped to the O class);
* \({\textsc{Cr}_{silver}}\) and \({\textsc{Cr}_{gold}}\) are the same datasets as above, in which only concept-related annotations are retained (i.e. named entities are mapped to the O class);

Based on these datasets, we train and evaluate the same model architecture under the following four different settings: i) we train on \({\textsc{Cner}_{silver}}\) and test on \({\textsc{Cner}_{gold}}\), ii) we train on \({\textsc{Cner}_{silver}}\) and test on \({\textsc{Ner}_{gold}}\) and \({\textsc{Cr}_{gold}}\), iii) we train on \({\textsc{Ner}_{silver}}\) and test on \({\textsc{Ner}_{gold}}\), and iv) we train on \({\textsc{Cr}_{silver}}\) and test on \({\textsc{Cr}_{gold}}\).6 Differently from setting (i) in which a distinction between concepts and named entities is not required, in setting (ii) we train a CNER model on \({\textsc{Cner}_{silver}}\) which, for each predicted span, provides the type (C or NE) in addition to the CNER category (e.g. for the token _doctor_ the model outputs B-C-Person). This allows us to retain only entity- or concept-related annotations when evaluating on \({\textsc{Ner}_{gold}}\) and \({\textsc{Cr}_{gold}}\), respectively, enabling a fair evaluation and ensuring comparability between the CNER model and the specialized NER and CR systems.7 In fact, an unfiltered evaluation of CNER predictions would lead to uninformative results, since a CNER model would naturally provide dense annotations regarding all the nominal expressions in a sentence, independently of their type (C or NE).

Footnote 6: We split the gold data into half for validation, half for testing (we refer to the latter as \({\textsc{Cne}^{rel}_{gold}}\), \({\textsc{Ner}^{Rest}_{gold}}\) and \({\textsc{Cr}^{test}_{gold}}\)).

Footnote 7: We highlight that NER systems implicitly learn to identify NEs, in the same manner as CR does with concepts.

As evaluation metrics, we adopt the macro F1 score at the token level and the span-based F1 score. Additionally, we also report token-level micro F1 scores, because - differently from NER - the O category is much less frequent, given the high density of CNER annotations (cf. last column of Table 2).

### Architecture

For our experiments, we opted for an architecture that strikes a balance between accuracy and simplicity, offering the robustness required for our task while being efficient and easy to be fine tuned. Specifically, we employ a pretrained transformer encoder, namely DeBERTa-v3 base He et al. (2023), and train a classification head on top of it to predict the category for each token in a sentence. The classification head consists of two linear layers and a normalization layer, with a dropout of 0.1 and the GeLU activation function. Since this architecture has been proven to achieve competitive performance compared to the current state of the art in standard NER benchmarks He et al. (2023), we believe that it constitutes a strong baseline for the CNER task.

Our systems are developed using the PyTorch Lightning framework8 and the HuggingFace models library.9 We train them on a single RTX 4090 Ti, with a patience parameter of 20 and a validation step every 30% of the total number of steps per epoch, resulting in 4 epochs and approximately 2 hours of training time for each model. We use the RAdam optimizer with a learning rate of 5\(\times\)10\({}^{-6}\), and a linear scheduler with a warm-up of 10% of the total steps. We adopt the same architectural

Figure 4: Pie chart showing the \({\textsc{Cner}_{gold}}\) category distribution.

Figure 3: Pie chart showing the \({\textsc{Cner}_{silver}}\) category distribution.

 

setup and hyperparameters to train the CNER, NER and CR models. All models are trained to minimize the cross-entropy loss function. We select the best model based on its macro F1 score on the validation set.

## 6 Results

Rq1The proposed architecture achieves \(87.20\) micro F1, \(59.38\) macro F1 and \(66.72\) span F1 score points, when asked to identify and classify both named entities and concepts with our 29-category tagset (cf. setting (i) in Section 5.1). In Figure 5, we provide the individual span F1 scores for each category in our benchmark. In general, by cross-referencing the category scores with the distribution in Figure 3, we observe that performance has a moderate correlation with the number of training instances (Pearson correlation coefficient \(\rho=0.4\)).

From a closer perspective, our system is affected by two main factors. First, when evaluated only on the identification of spans, the system achieves \(86.83\) span F1 score points, setting an upper bound to the overall system performance. Indeed, the model struggles to identity spans like _Triple J hottest 100 of all time in Australia_ and _A full day's Work_ in their entirety. Second, the confusion matrix in Appendix B shows that the model sometimes mistakes categories that have an intrinsic level of ambiguity (e.g. Animal or Plant with Food). In particular, terms that can be associated with multiple categories (e.g. _chicken_ and _eggplant_) are often difficult to be classified based on the limited sentence context. Additionally, specific terms, often contained in Wikipedia articles, like _Synapids Ophiacodon_ or _metal umlaut_, require technical expertise to be properly classified. We report a detailed error analysis in Appendix B.

Rq2We summarize the results of our experiments in Table 3. Our findings clearly illustrate the significant synergistic benefits of training a single model to simultaneously identify and classify concepts and named entities, compared to restricting the same model to either concepts or named entities. The CNER model exhibits a notable increase of \(\sim\)0.5 micro, \(\sim\)5.4 macro and \(\sim\)2.4 span F1 points over the NER model. A similar improvement is achieved over the CR model with a \(\sim\)1.7 micro, \(\sim\)8.0 macro, and \(\sim\)4.1 span F1 point increase.

These findings underline the advantages of our proposed CNER approach in improving the accuracy and effectiveness of both concept and named entity recognition tasks. In Section 7, we provide a qualitative analysis of our results, and reserve a detailed error analysis for Appendix B.

## 7 Qualitative Analysis

To better understand the behaviour of CNER, NER and CR models, we conduct a qualitative error analysis on Cner\({}^{test}_{gold}\) and report some representative examples in Table 4.

In the first example, we can immediately appreciate the higher annotation density of both concepts and named entities produced by the CNER model compared to the specialized NER and CR alternatives. We also report, in the second example, an instance in which the CNER model correctly

Figure 5: Bar chart with span F1 score (%) for each CNER category sorted in descending order. The support for each category is in parentheses.

 

predicts the label Property for the concept _role_ while CR misclassifies it as an Event. Furthermore, in the same sentence, we have an example of an instance for which there is one correct label, but, based on the given context, other tags could be associated with it. Specifically, both the CNER and CR models misclassify _tobacco_ by tagging it with Food and Substance, respectively, which are less appropriate, but both plausible annotations within the given context.

Finally, in the third example, we present a sentence in which our system shows better generalization capabilities: while the named entity _John_ is misclassified as Per by both the NER and CNER models, the CNER system correctly assigns the label Super to the concept _zombie_, which is misclassified by the CR model with the tag Biology.

## 8 Conclusions

In this paper, we presented the novel task of Concept and Named Entity Recognition (CNER) and introduced a comprehensive set of categories specifically tailored to encompassing the annotation of both proper and common nouns in a unified framework. To fulfill the need for specific resources in our newly introduced task, we proposed an automatic procedure that enabled the creation of the first large-scale training corpus for the CNER task. Our dataset consists of more than 300k annotated sentences, with dense coverage of both proper and common nouns obtained by devising several heuristics for solving sparsity. Moreover, we carried out a manual annotation of 2,000 sentences and used the resulting data for validation and testing.

Our experiments showed that a competitive pre-trained language model was able to successfully learn the CNER task, achieving \(87.20\) micro, \(59.38\) macro and \(66.72\) span F1 score points. Additionally, we also compared the performance of CNER as a joint task rather than separately identifying and classifying nominal concepts and named entities, reporting a remarkable increase in performance on both tasks.

Finally, to encourage the use and development of CNER systems, we publicly release our data and models to the research community and leave to future work the experiments on possible downstream task applications.

\begin{table}
\begin{tabular}{c l|l} \hline \multicolumn{2}{c|}{**Type**} & **Prediction** \\ \hline Gold & The [first]\({}_{\textit{MEASTERE}}\) [town]\({}_{\textit{LOC}}\) to fall into the [Lombars]\({}_{\textit{cancer^{\prime}}}\) [hands]\({}_{\textit{PART}}\) \\  & was [Formu Inlli]\({}_{\textit{LOC}}\) (Cividale del Frituli]\({}_{\textit{LOC}}\)), the [seati]\({}_{\textit{LOC}}\) of the local [magister militum]\({}_{\textit{PER}}\). \\ \cline{2-4} \multirow{3}{*}{**NER**} & NER & The first town to fall into the [Lombars]\({}_{\textit{tumor^{\prime}}}\) hands \\  & was [Formu Inlli]\({}_{\textit{STRECT}}\) ([Cividate del Frituli]\({}_{\textit{LOC}}\)), the seat of the local magister militum. \\ \cline{2-4}  & CR & The [first]\({}_{\textit{MEASTERE}}\) [town]\({}_{\textit{LOC}}\) to fall into the Lombarts’ [hands]\({}_{\textit{PART}}\) \\  & was Forum Itili (Cividate del Frituli), the [seati]\({}_{\textit{LOC}}\) of the local [magister]\({}_{\textit{PER}}\) militum \\ \cline{2-4}  & CNER & The [first]\({}_{\textit{MEASTERE}}\) [town]\({}_{\textit{LOC}}\) to fall into the [Lombars]\({}_{\textit{IGUDY'}}\) [hands]\({}_{\textit{PART}}\) \\  & was [Formu Inlli]\({}_{\textit{STECT}}\) ([Cividate del Frituli]\({}_{\textit{LOC}}\)), the [seati]\({}_{\textit{LOC}}\) of the local [magister]\({}_{\textit{PER}}\) millum \\ \hline \end{tabular} 
\begin{tabular}{c c} \hline \multicolumn{2}{c|}{Gold} & **[Tom McCarthy]\({}_{\textit{PER}}\) highlighted the prominent [role]\({}_{\textit{PPOPERITY}}\) of [ tobacco]\({}_{\textit{PAY}}\) in the [story]\({}_{\textit{MEZHA}}\),** \\ drawing on the [ideas]\({}_{\textit{PSYCH}}\) of [philosphere Jacques Derrida]\({}_{\textit{PER}}\) to suggest the potential [symbolism]\({}_{\textit{MEDA}}\) of this. \\ \cline{2-4} \multirow{3}{*}{**NER**} & NER & **[Tom McCarthy]\({}_{\textit{PER}}\) highlighted the prominent role of tobacco in the story,** \\ drawing on the ideas of [philosphere Jacques Derrida]\({}_{\textit{PER}}\) to suggest the potential symbolism of this. \\ \cline{2-4}  & CR & Tom McCarthy highlighted the prominent [role]\({}_{\textit{EVENT}}\) of [ tobacco]\({}_{\texttt{RTESTANCE}}\) in the [story]\({}_{\textit{MEZHA}}\), \\ drawing on the [idea]\({}_{\textit{PSYCH}}\) of Jacques Derrida to suggest the potential [symbolism]\({}_{\textit{PSYCH}}\) of this. \\ \cline{2-4}  & CNER & **[Tom McCarthy]\({}_{\textit{PER}}\) highlighted the prominent [role]\({}_{\textit{POOPERITY}}\) of [ tobacco]\({}_{\textit{POOO}}\) in the [story]\({}_{\textit{MEZHA}}\),** \\ drawing on the [idea]\({}_{\textit{PSYCH}}\) of [philosphere Jacques Derrida]\({}_{\textit{PER}}\) to suggest the potential [symbolism]\({}_{\textit{PSYCH}}\) of this. \\ \hline \multicolumn{2}{c|}{Gold} & **[John]\({}_{\textit{SPEX}}\) has new [weapons]\({}_{\textit{APFACT}}\); including [holy water]\({}_{\textit{KSENCE}}\); [baial]\({}_{\textit{SIGLOG}}\) for the [undead]\({}_{\textit{SCFPE}}\),** \\  & and a [blunderbus]\({}_{\textit{LAFFTEXT}}\) that uses [nomibelise]\({}_{\textit{TYPER}}\) [parts]\({}_{\textit{APFACT}}\) as [ammonium]\({}_{\textit{LATFACT}}\). \\ \cline{2-4}  & NER & **[John]\({}_{\textit{PER}}\)** has new weapons, including body water, bait for the unhead, and a blunderbus that uses zombie parts as ammunition. \\ \hline \multicolumn{2}{c|}{CR} & John hes new [wagons]\({}_{\textit{ARTFACT}}\), including [holy water]\({}_{\textit{RSSTATE}}\); [baial]\({}_{\textit{ESPNAME}}\) for the [undead]\({}_{\textit{BIOLOG}}\), \\  & and a blunderbus that uses [romibelise]\({}_{\textit{SOPLOG}}\) [parts]\({}_{\textit{ARTTACT}}\) as [ammonium]\({}_{\textit{ALFTACT}}\). \\ \cline{2-4}  & CNER & **[John]\({}_{\textit{PER}}\)** has new [weapons]\({}_{\textit{ARTFACT}}\), including [holy water]\({}_{\textit{SIUSTANCE}}\), [baial]\({}_{\textit{ESINAME}}\) for the undead, \\  & and a blunderbus that uses [romibelise]\({}_{\textit{SUPTER}}\) [parts]\({}_{\textit{ATTRACT}}\) as [ammonium]\({}_{\textit{ALFTACT}}\). \\ \hline \end{tabular}
\end{table}
Table 4: Examples of annotations from the inference of our CNF, NER and CR models over the test split of our \(\textsc{Cner}_{gold}\) dataset. Correct predictions are highlighted in green, while wrong predictions are highlighted in red.

 



## Limitations

The current implementation lacks extension to multiple languages, posing a potential limitation to the broader applicability of the proposed task. Nevertheless, we note that our approach is inherently language agnostic, as Wikipedia hyperlinks are linked to BabelNet synsets. Multilingual data for both NER [21]; Tedeschi and Navigli2022multilingual and WSD [35] are available, but - as regards named entities - an adaptation to our set of categories would be required. We leave this to future work.

Furthermore, although the proposed tagging procedure demonstrates substantial agreement with manual annotation, as indicated by Cohen's \(\kappa=71.4\), we remark that the _Cne\({}_{silver}\)_ dataset is produced automatically, hence it may contain errors. The lack of a large-scale manually annotated gold corpus could represent a limitation to obtaining more accurate CNER models, something that we or the community can address in a future large-scale validation effort.

Finally, the results that our CNER model can achieve on _Cner\({}_{gold}\)_ are bounded. In particular, during the manual annotation process, humans often had to resort to accessing external world knowledge in order to solve the intrinsic ambiguity of numerous spans. A possible solution to this limitation could be to adopt models that rely not only on the input text, but that are also able to exploit external knowledge for their predictions, as is already done by knowledge-augmented pretrained language models.

## Acknowledgements

We gratefully acknowledge the support of the PNRR MUR project PE0000013-FAIR.

Roberto Navigli also gratefully acknowledges the support of the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content), which is funded by the MUR Progetti di Rilevante Interesse Nazionale programme (PRIN 2020). This work has been carried out while Giuliano Martinelli, Francesco Maria Molfese and Simone Teleschi were enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome.

## References

* Barba et al. (2021) Edoardo Barba, Tommaso Pasini, and Roberto Navigli. 2021. ESC: Redesigning WSD with extractive sense comprehension. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4661-4672, Online. Association for Computational Linguistics.
* Bevilacqua et al. (2021) Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_, pages 4330-4338. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
* Choi et al. (2018) Gunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018. Ultra-fine entity typing. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 87-96, Melbourne, Australia. Association for Computational Linguistics.
* Ding et al. (2021) Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitaao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 3198-3213, Online. Association for Computational Linguistics.
* Durrett and Klein (2014) Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. _Transactions of the Association for Computational Linguistics_, 2:477-490.
* Fetahu et al. (2023) Besnik Fetahu, Sudipta Kar, Zhiyu Chen, Oleg Rokhlenko, and Shervin Malmasi. 2023. SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2). In _Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)_. Association for Computational Linguistics.
* Grishman (2015) Ralph Grishman. 2015. Information extraction. _IEEE Intelligent Systems_, 30(5):8-15.
* He et al. (2023) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2023. Debertav3: Improving deberta using electra-style pretraining with gradient-disentangled embedding sharing.
* Cabot et al. (2023) Pere-Lluis Huguet Cabot, Simone Tedeschi, Axel-Cyrille Ngonga Ngomo, and Roberto Navigli. 2023. Red\({}^{\text{fm}}\): a filtered and multilingual relation extraction dataset. In _Proc. of the 61st Annual Meeting of the Association for Computational Linguistics: ACL 2023_, Toronto, Canada. Association for Computational Linguistics.
* Izquierdo et al. (2015) Ruben Izquierdo, Armando Suarez, and German Rigau. 2015. Word vs. class-based word sense disambiguation. _J. Artif. Int. Res._, 54(1):83-122

Ruben Izquierdo Bevia, Armando Suarez Cueto, German Rigau Claramunt, et al. 2007. Exploring the automatic selection of basic level concepts.
* Khosla and Rose (2020) Sopan Khosla and Carolyn Rose. 2020. Using type information to improve entity coreference resolution. In _Proceedings of the First Workshop on Computational Approaches to Discourse_, pages 20-31, Online. Association for Computational Linguistics.
* Li et al. (2022) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2022. A survey on deep learning for named entity recognition. _IEEE Transactions on Knowledge and Data Engineering_, 34(1):50-70.
* Manning et al. (2008) Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schutze. 2008. _Introduction to Information Retrieval_. Cambridge University Press, USA.
* Maru et al. (2022) Marco Maru, Simone Conia, Michele Bevilacqua, and Roberto Navigli. 2022. Nibbling at the hard core of Word Sense Disambiguation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4724-4737, Dublin, Ireland. Association for Computational Linguistics.
* Miller (1995) George A. Miller. 1995. Wordnet: A lexical database for english. _Commun. ACM_, 38(11):39-41.
* Molla et al. (2006) Diego Molla, Menno van Zaanen, and Daniel Smith. 2006. Named entity recognition for question answering. In _Proceedings of the Australasian Language Technology Workshop 2006_, pages 51-58, Sydney, Australia.
* Nadeau and Sekine (2007) David Nadeau and Satoshi Sekine. 2007. A survey of named entity recognition and classification. _Lingvisticae Investigationes_, 30(1):3-26.
* Navigli (2009) Roberto Navigli. 2009. Word sense disambiguation: A survey. _ACM computing surveys (CSUR)_, 41(2):1-69.
* Navigli et al. (2021) Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnini, and Francesco Cecconi. 2021. Ten years of BabelNet: A survey. In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_, pages 4559-4567. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
* Navigli and Ponzetto (2012) Roberto Navigli and Simone Paolo Ponzetto. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. _Artificial Intelligence_, 193:217-250.
* Pasini et al. (2021) Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Xl-wsd: An extra-large and cross-lingual evaluation framework for word sense disambiguation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 13648-13656.
* Pawar et al. (2017) Sachin Pawar, Girish K. Palshikar, and Pushpak Bhattacharyya. 2017. Relation extraction : A survey.
* Pradhan et al. (2012) Sameer Pradhan, Alessandro Moschitit, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Comll-2012 shared task: Modeling multilingual unrestricted coreference in entonotes. In _Joint conference on EMNLP and CoNLL-shared task_, pages 1-40.
* Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_, pages 101-108, Online. Association for Computational Linguistics.
* Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition.
* Sukthanker et al. (2020) Reha Sukthanker, Soujanya Poria, Erik Cambria, and Ramkumar Thirunavukarasu. 2020. Anaphora and coreference resolution: A review. _Information Fusion_, 59:139-162.
* Tedeschi et al. (2021a) Simone Tedeschi, Simone Conia, Francesco Cecconi, and Roberto Navigli. 2021a. Named Entity Recognition for Entity Linking: What works and what's next. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2584-2596, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Tedeschi et al. (2021b) Simone Tedeschi, Valentino Maiorca, Niccolo Campolungo, Francesco Cecconi, and Roberto Navigli. 2021b. WikiNeRuRel: Combined neural and knowledge-based silver data reaction for multilingual NER. In _Findings of the association for Computational Linguistics: EMNLP 2021_, pages 2521-2533, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Tedeschi and Navigli (2022) Simone Tedeschi and Roberto Navigli. 2022. MultiNERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation). In _Findings of the Association for Computational Linguistics: NAACL 2022_, pages 801-812, Seattle, United States. Association for Computational Linguistics.
* Tang and Sang (2002) Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
* Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In _Concluding-02: The 6th Conference on Natural Language Learning at HLT-NAACL 2003_, pages 142-147.
* Vial et al. (2019) Loic Vial, Benjamin Lecouteux, and Didier Schwab. 2019. Sense vocabulary compression through the semantic knowledge of WordNet for neural word sense disambiguation. In _Proceedings of the 10th Global Wordnet Conference_, pages 108-117, Wroclaw, Poland. Global Wordnet Association 

Additional Dataset Statistics

In Table 6 we highlight the overall proportion of concepts (C) and named entities (NE) in our Cner\({}_{silver}\) and Cner\({}_{gold}\) datasets, respectively. Additionally, in Figure 6, we provide the distribution of concept and named entities for each individual category in our datasets.

## Appendix B Additional Results

Rq1 (CNER system)In Figure 5, we presented the results of our fine-tuned CNER system evaluated on the Cner\({}^{test}_{gold}\) test set. For completeness, and to better interpret the obtained results, in Figure 7, we present a confusion matrix of the system's outputs. The matrix shows that, in general, the categories exhibit well-defined boundaries, with some notable exceptions. Specifically, the categories Property, Psych, and Feeling are frequently confused due to their intertwined semantics . Additionally, instances belonging to the Law category, such as _peace treaty_, can occasionally overlap with instances from the Media category, which encompasses general written documents conveying information. Furthermore, the categories Anim and BIO are prone to misclassification, primarily because distinguishing between animals and biological entities can be challenging (e.g. _chephalopods_). Finally, Datetime and Measure are often confused due to the difficulties of discerning time periods when used to explicitly measure time. as opposed to identifying specific periods in time.

Rq2 (CNER vs NER & CR)In Table 3 we showed the benefits of performing CNER as a unified task rather than identifying concepts and named entities separately. In Figure 8, we illustrate the category-wise impact of the CNER system when compared to NER and CR systems, highlighting its positive and negative contributions. Notably, the CNER system consistently exhibits a positive influence over both named entities and concepts. Specifically, categories such as Discipline, Food, and Culture witness particularly positive contributions, with an increase of up to \(40\) span-based F1 score points. In some categories, the CNER system provides positive contributions only for named entities, as observed in Group and Media, or exclusively for concepts, as evidenced in Super and Law.

Evaluation on Standard NER BenchmarkIn order to further evaluate our Cner\({}_{silver}\) data, we present an out-of-domain comparative analysis with a standard NER benchmark, namely CoNLL-2003 Sang and Meulder (2003). In particular, the objective of this analysis is to asses if a model, trained on the Ner\({}_{silver}\) data, is able to attain performance that is comparable to that of directly training on the CoNLL-2003 training corpus. Comparing the two models presents several challenges. First, our span annotation guidelines are different from those of a traditional NER dataset, due to the inclusion of concepts (e.g. _"president George Washington"_ is a whole named entity span, while CoNLL03 would only tag _"George Washington"_). For this reason, in order to compare the two models we decided to evaluate classification accuracy only, without evaluating the span extraction. Another significant problem are the differences in the categorizations: while some of our 29 categories are directly associated with PER, ORG and LOC, the MISC category is not comprehensive because the CoNLL03 test set lacks annotations for the instances of 12 of our categories. For this reason, Table 5 shows classification accuracy on PER, ORG and LOC of three training strategies of the same architecture presented in Section 5.2:

1. The model is trained on Ner\({}_{silver}\) and its outputs are mapped to the three standard NER categories.
2. The model is trained on the CoNLL03 dataset.
3. The model is trained on Ner\({}_{silver}\) with an additional fine-tuning on the CoNLL03 dataset.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dataset** & **NE** & **C** \\ \hline Cner\({}_{silver}\) & 48.3\% & 51.7\% \\ Cner\({}_{gold}\) & 43.6\% & 56.4\% \\ \hline \hline \end{tabular}
\end{table}
Table 6: Distribution of named entities and concepts.

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Training Strategy** & **LOC** & **PER** & **ORG** \\ \hline (1) Ner\({}_{silver}\) & 94.2 & 96.2 & 54.7 \\ (2) CoNLL-2003 & 96.7 & 98.3 & 98.2 \\ \hline \hline (3) Ner\({}_{silver}\)(F) & 96.8 & 98.0 & 98.0 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Classification accuracy (%) of the same architecture trained on three different corpus and tested on the CoNLL2003 benchmark. Ner\({}_{silver}\)(F) is the model pre-trained on Ner\({}_{silver}\)and finetuned on the CoNLL03 training set.

 

Despite the different domains present in \({\textsc{Ner}_{silver}}\) and CoNLL03, namely Wikipedia texts and news articles, respectively, the resulting models obtain comparable performance in labelling PER and LOC instances. However, the same does not hold for the Org instances. Interestingly, we noticed that the low scores of the model trained solely on our data are strongly influenced by the large number of sports articles available in the CoNLL03 test set. In particular, many of the organizations correspond to sports teams whose names often match the name of the corresponding city. Nevertheless, fine-tuning the system on the CoNLL03 training set solves this problem.

## Appendix C CNER Categories

In Table 7, we provide the full list of CNER categories, along with their textual descriptions and instance examples for both named entities and concepts.

Figure 6: Bar chart with the distribution of named entities and concepts for every CNER category in the \({\textsc{Cner}_{gold}}\) and \({\textsc{Cner}_{silver}}\) resources.

 

Figure 7: Bar chart with Span F1 score of the different categories.

 

Figure 8: Bar chart with Span F1 score of the different categories.

 



\begin{table}
\begin{tabular}{p{56.9pt}|p{284.5pt}|p{170.7pt}} \hline
**Category** & **Description** & **Examples** \\ \hline Animal & _Living beings (excluding humans) with the ability to move and perceive their surroundings._ & _dog, cut, manual, carnivore, brown bear, African Wild Dog, Great White Shark, vehicle, software, mouse, data stream, Windows XP, Fiat Panda_ & _nuclide, sdrware, mouse, data stream, Windows XP, Fiat Purchis Loan, Investment in Russia_ \\ \hline Biology & _Biological entities, including living organisms, cells, or biological components_ & _protein, cell, living organism, lipid, Herpes Simplex Virus, Escherichia Coli_ \\ \hline Celestial & _celestal bodies as Planets, stars, asteroids, galaxies and other astronomical objects._ & _comet, nebulae, Sun, Neptune, Asteroid 187 Lamberta, Proxima Centari_ \\ \hline Culture & _Chultural aspects, traditions, customs, and practices associated with specific groups or societies._ & _religion, feminism, socialism, capitalism, anarchism, doctrine, cult, Islam, Buddhism_ \\ \hline Datetime & _Dates and times_ & _18 March, Saturday, 1979, the evening of 19 November, 15:30 am_ \\ \hline Disease & _medical conditions, illnesses, disorders, and health-related issues affecting living organisms._ & _infection, allergy, metastasis, complication, acne, Alzheimer’s Disease, Cystic Fibrosis discipline, sport, football, computer science, anatomy, long jump._ \\ \hline Event & _Events, phenomenon or activities that occur at specific times or places. It includes both significant and everyday occurrences_ & _crime, professorship, temperature change, 2003 Wimbledon Championships, Cannes Film Festival._ \\ \hline Feeling & _Emotions, sensations, and subjective experiences related to human or animal consciousness._ & _affection, attachment, agitation, craving, urge, reputation._ \\ \hline Food & _edible items, dishes, beverages, and culinary products that are consumed for nonishment or enjoyment_ & _beverage, dish, pork, lausagna, Carbonara, Sangiovese, Checkdar Beer Fondue, Pizza Margherita._ \\ \hline Group & _group of people or animals_ & _staff, social group, panel, millila, community, trio, duo, family, genealogy, alliance, nationality, peoples_ \\ \hline Language & _individual language-related items, such as words, phrases, or idiomatic expressions_ & _discourse, context, leucine, morphene, appellation, eponym, nickname, vowel, syllable, headword_ \\ \hline Law & _legal principles, regulations, and rules governing society and various aspects of life_ & _law, civil law, administrative law, martial law, shariah, ordinance, civil right, Magna Carta, Islamic Law_ \\ \hline Loc & geographical locations, such as villages, towns, cities, regions, countries, continents, landmarks, or natural features_ & _space, surface, street, road, town, Rome, Lake Faiku, Mississippi River_ \\ \hline Measure & _units of measurement and quantification used to determine the size, quantity, or quality of various objects or phenomena._ & _day, microsecond, millisecond, two, 35, 45%, first, temperature, length,_ \\ \hline Media & _various forms of communication and entertainment media, such as newspapers, television shows, movies, social media or digital content._ & _soundtrack, report, publication, language, English, Forbes, American Psycho_ \\ \hline Money & _monetary units, currencies, and financial values used in different contexts_ & _monetary unit, dollar, 15 euros, 1116 CHF_ \\ \hline Org & _organizations, institutions, and companies involved in diverse sectors or activities_ & _Industry, commercial enterprise, San Francisco Giants, Google, Democratic Party._ \\ \hline Part & _individual components or sections of larger entities or objects_ & _fringer, chin, head, tail, femur, airplane wing, airplane’s wings, flower's stem_ \\ \hline Per & _individuals or persons, including real people and historical figures_ & _dioctor, historian, professor, musician, Ray Charles, Jessica Alba_ \\ \hline Plant & _Types of trees, flowers, and other plants, including their scientific names._ & _grass, peach tree, Forsythia, Artemisia Martima._ \\ \hline Property & _properties or attributes of objects, entities, or concepts_ & _thickness, height, dimension, shape, age_ \\ \hline Psych & _psychological concepts, mental states, and phenomena related to the human mind and behavior_ & _psychological feature, cognition, attention, necessity_ \\ Relation & _relationships, connections, and associations between entities or concepts_ & _export, competition, comparison, bridge, relatedness, parenting, function, parity, transitivity_ \\ \hline Struct & _physical structures, including buildings, architectural designs, and engineered constructions made by humankind_ & _sh shelter, parrovestone, refuge, tent, loft, San Peter’s Church, Golden Bridge_ \\ \hline Substance & _chemical substances_ & _acid, bactericide, carbonyl, explosive, fertilizer, Zyclton B_ \\ \hline Super & _Mythological and religious entities._ & _Apollo, Persophone, Aphrodite, Saint Peter, Pope Gregory I, Hercules._ \\ \hline \end{tabular}
\end{table}
Table 7: Label, description, and instance examples of each of our CNER categories.