# Semantically-Informed Domain Adaptation for Named Entity Recognition

Mariya Borovikova, Arnaud Ferré, Robert Bossy, Mathieu Roche, Claire Nédellec

► To cite this version:

Mariya Borovikova, Arnaud Ferré, Robert Bossy, Mathieu Roche, Claire Nédellec. Semantically-Informed Domain Adaptation for Named Entity Recognition. ISMIS 2024 - 27. International symposium on methodologies for intelligent systems, Jun 2024, Poitiers, France. pp.55-64, 10.1007/978-3-031-62700-2_6. hal-04998788

HAL Id: hal-04998788

https://hal.science/hal-04998788v1

Submitted on 20 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Semantically-Informed Domain Adaptation for
Named Entity Recognition

Mariya Borovikova¹,³[0000-0003-3245-2741], Arnaud Ferré¹[0000-0001-9115-8222], Robert Bossy¹[0000-0001-6652-9319], Mathieu Roche²,³[0000-0003-3272-8568], and Claire Nédellec¹[0000-0002-0577-0595]

¹ MaIAGE, Université Paris-Saclay, INRAE, Domaine de Vilvert, Jouy-en-Josas
78352, France

² CIRAD, Montpellier F-34398, France

³ TETIS, Univ. Montpellier, AgroParisTech, CIRAD, CNRS, INRAE, Montpellier
34090, France

**Abstract.** Named Entity Recognition (NER) is an important task in Natural Language Processing that involves identifying entities in un-structured text. State-of-the-art NER methods often require extensive manual labeling. To bridge this gap, this paper introduces a domain adaptation technique for NER that leverages semantic information about entity types using Sentence-BERT embeddings of their textual descriptions. We conduct experiments across various datasets from both general and biological domains, evaluating our approach in standard and zero-shot settings. Our experiences demonstrate the effectiveness of our method, outperforming existing zero-shot techniques on certain datasets. Our findings underscore the importance of semantic information in NER. This paper contributes to the advancement of zero-shot domain adaptation for NER and opens avenues for future research in improving NER systems' adaptability and performance across diverse domains.

**Keywords:** Named Entity Recognition · Domain Adaptation

# 1 Introduction

Named Entity Recognition (NER) plays an important role in Natural Language Processing, focusing on the task of identifying named entities within un-structured text (see Figure 1). State-of-the-art NER methods require extensive manual labeling, which is inherently resource-intensive. Unsupervised and semi-supervised approaches, although capable of overcoming this limitation, is often less efficient within domain-specific contexts. Consequently, we have developed a method that leverages textual descriptions of entity types relevant to specific domains or datasets, offering a resource-efficient and time-saving solution.

This paper introduces a domain adaptation technique, i.e. a strategy that involves adjusting a model trained on a source domain to perform effectively on a different target domain. Our adaptation strategy leverages the semantics of entity types for NER. More precisely, we employ textual descriptions of entity
<figure><img src="image_2.png" /><figcaption>**Fig. 1. NER task.** In this text passage, the identified named entities are highlighted.</figcaption></figure>

types, derived from their names, processed by a Sentence-BERT [15] model to process. The resulting semantic representations are seamlessly integrated into our NER model as input features. To validate our approach, we conduct experiments on several datasets from both general and biological domains. Our approach is tested in two settings: (1) the traditional training mode, where the model is trained and tested on distinct dataset partitions, hence on the same entity types, and (2) a mode where the model is trained and tested on different datasets, enabling testing on previously unseen entity types.

## 2 Related work

The state-of-the-art NER models rely on supervised Machine Learning algorithms such as BiLSTM+CRF [22], BERT [3], generative models [17] or prompt engineering [20].

Unsupervised approaches to NER take different forms, including lexicon-based methods [14], clustering-based strategies [19] and generative models [6]. Additionally, prompting engineering strategies are explored in [21,23]. Although not dependent on annotated data, unsupervised approaches, often show lower efficiency compared to supervised methods. Alternatively, they may depend on very large language models, necessitating substantial computational resources, even during the prediction phase.

While unsupervised approaches address challenges associated with annotated data and computational resources, few-shot learning strategies represent a middle ground between unsupervised and supervised methods. These strategies require less data while still maintaining an acceptable level of performance. Few-shot strategies predominantly rely on transfer learning, aiming to leverage knowledge gained during training for application to related tasks. Domain adaptation is a particular case of transfer learning.
Traditional approaches to domain adaptation in NER commonly rely on bootstrapping techniques as seen in [2]. Modern methodologies frequently use difference between domains, as demonstrated in [7]. However, the approach that captured our attention is described in [12]. In this method, the authors leverage two BERT models: one to encode input text and another to encode labels, namely, entity types. The model combines these embeddings using a dot product, and the prediction is based on the maximum value. This approach has proven successful, especially in few-shot learning adaptation. However, it requires at least a small corpus from the target domain and is not fully zero-shot.

Our approach draws inspiration from the concepts outlined in [12]. Similar to the strategy described in that work, we use labels semantics, transforming them through a BERT-based model. However, our approach does not entail model training for labels representation.

# 3 Methodology

The domain adaptation methodology proposed in this paper is based on the following hypothesis: using the semantic representations of entity types can significantly improve the algorithm's overall performance and compensate for the lack of training data. To assess this hypothesis, we used the Sentence-BERT model [15] to semantically represent entity types. This section provides a description of our methodology and the datasets employed. For a visual overview of our approach, please refer to Figure 2.

<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Overview of the Semantic NER approach. Entity types are initially described in textual form and subsequently processed through Sentence-BERT before integration into the NER pipeline.</figcaption></figure>

## 3.1 NER Model Description

The conventional NER approach employs a neural network that takes tokenized text as input and predicts each token's probability of belonging to predefined
classes. However, this model cannot be directly applied to classes unseen during
the training without modification. To address this limitation, we incorporate
the representations of the entity types to be predicted as cues, as described in
section 3.2.

We use the BERT [3] and BioBERT [10] models as the base of the algorithm.
Specifically, we use BERT for general domain datasets and BioBERT for those
of the biological domain. We combine the output from BERT (or BioBERT)
with the output from Sentence-BERT [15] and add three Linear Layers and a
Conditional Random Field (CRF) layer. Before combining the embeddings from
BERT (or BioBERT) and Sentence-BERT, a linear layer is applied to perform
a linear transformation on the input data. This step is necessary because the
(bio)BERT embedding and Sentence-BERT embedding cannot be directly mul-
tiplied due to incompatible dimensions. Subsequently, the combined embeddings
undergo three additional linear transformations, which gradually reduce the di-
mensionality. Finally, the output is passed through a CRF layer to consider the
dependencies between adjacent labels, thereby enhancing the model's ability to
identify entity boundaries. Since texts often contain multiple entity types, we
generate one input sample per entity type for each text. We freeze the weights of
BERT (or BioBERT) and Sentence-BERT and adjusted only the last Linear
and CRF layers. This approach enables transitions between language models
based on the domain of the dataset. We provide an overview of our approach in
Figure 2.

## 3.2 Semantic Entity Type Representation

The process to represent each entity type is the following. We start by taking
an entity type name and attempt to retrieve a corresponding Wikipedia article
with a matching title (e.g., 'location' for the entity type 'location'). In cases
where no such article exists, we select the top result from a Wikipedia search
(e.g., 'calendar date' for the entity type 'date'). Additionally, approximately
62% of articles are manually selected, particularly when the labels of entity
types are ambiguous or lack clarity, as identified through manual review. For
example, we consider an article 'film director' for the 'director' label in the
MIT Movies dataset, and an article 'location' for the 'loc' label in the CoNLL-
2003 dataset. Textual content is then extracted from each selected article and
concatenated into a single document, with a blank line inserted between texts.
This composite text is then transformed using Sentence-BERT to generate latent
representations, enabling the usage of semantic information during NER. An
overview of this stage is presented in Figure 3. The resulting latent representation
is subsequently integrated into the NER pipeline.

## 3.3 Datasets

For our experiences we use datasets from diverse domains to enhance the algo-
rithm's robustness and generalizability, validating its effectiveness beyond spe-
cific niches and confirming its adaptability in addressing a wide range of real-
<figure><img src="image_4.png" /><figcaption>**Fig. 3. Semantic Information Extraction Process.** This figure illustrates the step-by-step process of extracting semantic information for entity types. Beginning with the entity type name (e.g., *Location*, *Movie*, *Microorganism*, etc.), it details the retrieval of related Wikipedia articles and the subsequent generation of latent semantic representations using Sentence-BERT.</figcaption></figure>

world applications. Specifically, we use six datasets, consisting of four from the general domain: CoNLL-2003 [16] (subcorpus in English), MIT movie review semantic corpus [11], MIT restaurant review [11], GeoVirus [5]; along with two from the biological domain: Bacteria Biotope [1] (BB) and NCBI diseases [4]. An overview of datasets is presented in Table 1.

In the pursuit of a zero-shot domain adaptation approach, we execute the following procedure for each dataset (denoted as *n*): creating a training set from all training partitions, excluding the dataset *n*, and subsequently assessing its performance on the test set of dataset *n*.

# 4 Experiments and Results

## 4.1 Experimental setup

As explained in 3.1, we freeze the weights of BERT (or BioBERT) and Sentence-BERT while adjusting only the classifier weights. Since we predict one entity type per iteration, for a thorough evaluation of our model, we generate predictions for each entity for each input text from a corresponding test set. Subsequently, we merge our predictions into a comprehensive output encompassing all entity types, as illustrated in figure 4. This approach enables a fair comparison of our model with existing models. In addition to our primary classifier-based model, we incorporate two baselines. In our first baseline, we use cosine similarity instead of a classifier. Specifically, we calculate the cosine similarity between the entity type embedding and each token embedding, comparing it to a threshold. If the distance is below 0.4, we predict the class with the lowest distance; otherwise, we consider it not belonging to any class. The threshold was chosen experimentally,
<table><caption>**Table 1.** Datasets description</caption><thead><tr><th>Dataset</th><th>Domain</th><th>Entity types</th><th>Size (in tokens)</th><th>Entities</th></tr></thead><tbody><tr><td>CoNLL-2003</td><td>general</td><td>miscellaneous, location, organisation, person</td><td>287553</td><td>34788</td></tr><tr><td>GeoVirus</td><td>general, epidemiology, legal</td><td>location</td><td>258182</td><td>7258</td></tr><tr><td>MIT Movies</td><td>general</td><td>actor, award, character, character name, director, genre, opinion, origin, plot, quote, rating, ratings average, relationship, review, song, soundtrack, title, trailer, year dish, price, restaurant name, cuisine, amenity, location, hours, rating</td><td>125736</td><td>55347</td></tr><tr><td>MIT Restaurants</td><td>general</td><td>phenotype, microorganism, habitat</td><td>81569</td><td>18514</td></tr><tr><td>BB</td><td>biological</td><td>specific disease, modifier, disease class, composite mention</td><td>50003</td><td>3435</td></tr><tr><td>NCBI diseases</td><td>biological</td><td></td><td>251443</td><td>6910</td></tr></tbody></table>

by running one epoch into the datasets with different thresholds, with 0.4 yielding the optimal results. In the second baseline we adopt a conventional training model without integrating entity type semantics.

We evaluated our approaches in both standard and zero-shot modes. In the standard mode, we (1) trained a model and tested it on respective partitions of the same dataset, and (2) trained our model using all datasets and tested it on a single dataset. In the zero-shot mode, our model was trained on all datasets except one, which was reserved for testing purposes.

## 4.2 Results

Tables 2 and 3 present the F-measures of our algorithm, two baselines outlined in section 4.1, and state-of-the-art (SOTA) models. The "Dataset" column identifies the evaluation dataset. The "S" column represents scores obtained using SOTA algorithms. It is important to note that the scores we present are sourced directly from the associated papers. Some datasets lack scores because, to our knowledge, there is no zero-shot algorithm evaluated on these datasets.

In Table 2, the "Same Dataset" column denotes scores achieved by training and testing the model on corresponding partitions of the same dataset, while the "Cross-Datasets" column reflects scores attained by training the model on all training partitions across datasets and testing on the test partition of the respective dataset. In Table 3, the "Cosine Distance" column shows scores acquired with the baseline outlined in section 4.1, while the "Classifier" column
**Fig. 4. Integration of predictions.** This diagram visually represents the process of consolidating entity predictions for multiple entity types into a unified output. This method enables a holistic evaluation of our model's performance across various entity types.

<figure><img src="image_5.png" /></figure>

showcases scores obtained with our algorithm. The "Standard Cross-Dataset" column reflects scores obtained when training our model conventionally, without incorporating entity type semantics. In other words, we used only the BERT (or BioBERT) model with a CRF layer on top.

<table><caption>**Table 2.** F-measures standard training</caption><thead><tr><th>Dataset</th><th>Same Dataset (Intra-Dataset)</th><th>Cross-Datasets (Inter-Dataset)</th><th>SOTA</th></tr></thead><tbody><tr><td>BB</td><td>0.56</td><td>0.58</td><td>0.89 [9]</td></tr><tr><td>GeoVirus</td><td>0.46</td><td>0.78</td><td>0.91 [18]</td></tr><tr><td>CoNLL-2003</td><td>0.88</td><td>0.91</td><td>0.95 [22]</td></tr><tr><td>MIT restaurants</td><td>0.62</td><td>0.75</td><td>0.72 [13]</td></tr><tr><td>MIT movies</td><td>0.58</td><td>0.62</td><td>0.81 [13]</td></tr><tr><td>NCBI diseases</td><td>0.54</td><td>0.42</td><td>0.90 [8]</td></tr></tbody></table>

Our experiments yield several observations: (1) Models trained on all datasets demonstrate superior quality compared to models trained exclusively on individual datasets (with the exception of NCBI Disease dataset); (2) The classifier exhibits higher quality than both baselines (with the exception of GeoVirus dataset); (3) Our model performs better in a standard mode than in an zero-shot mode (with the exception of GeoVirus dataset); (4) Notably, our zero-shot model outperforms the SOTA algorithm on the MIT Restaurants dataset and achieves performance comparable to the zero-shot SOTA score on the MIT Movies dataset.

# 5 Discussion

Based on our findings, we can affirm that incorporating entity type semantics significantly enhances NER results. Upon manual examination of predictions, we observed substantial variations in results among entity types. For instance,
<table><caption>**Table 3.** F-measures zero-shot</caption><thead><tr><th>Dataset</th><th>Classifier (Our Model)</th><th>Cosine Distance (Baseline)</th><th>Standard Cross-Dataset</th><th>SOTA</th></tr></thead><tbody><tr><td>BB</td><td>0.34</td><td>0.17</td><td>0.00</td><td>-</td></tr><tr><td>GeoVirus</td><td>0.54</td><td>0.07</td><td>0.79</td><td>-</td></tr><tr><td>CoNLL-2003</td><td>0.79</td><td>0.12</td><td>0.24</td><td>0.93 [19]</td></tr><tr><td>MIT restaurants</td><td>0.46</td><td>0.12</td><td>0.09</td><td>0.36[23]</td></tr><tr><td>MIT movies</td><td>0.62</td><td>0.02</td><td>0.12</td><td>0.63[21]</td></tr><tr><td>NCBI diseases</td><td>0.24</td><td>0.15</td><td>0.00</td><td>-</td></tr></tbody></table>

in the BB dataset, the F-measure for "Microorganism" is 0.75, contrasting with 0.15 for "Habitat" and 0.12 for "Phenotype." Our interpretation is as follows: our approach relies on Wikipedia descriptions of entity types, and some entities are inherently more challenging to elucidate. In the BB dataset, the concept of "Microorganism" is more evident than that of "Habitat" or "Phenotype," for which even human annotators can make errors. Moreover, entities labeled as "Microorganism" are commonly denoted in text through specific Latin terms, thereby aiding in their identification. This intuition is further supported by the fact that we also conducted experiments by automatically selecting Wikipedia articles with an entity type as the article title, and the results were similar to a baseline. Thus, the quality of the texts chosen for describing entity types is crucial.

Another notable point concerns the automated acquisition of semantic representations for entity types. This method faces challenges as the type labels may not precisely capture the semantics of the entity type. For example, in the NCBI diseases dataset, if a disease description lacks a representation by a noun (e.g., *VHL* in the *VHL gene*), it is annotated with a "Modifier" entity type. Regrettably, this does not faithfully convey the semantics of the entity type. However, it is worth noting that despite these challenges, obtaining relevant textual descriptions for each entity type remains a significantly less time and resource-consuming task compared to annotating a corpus for those entity types.

A surprising observation arises from the GeoVirus corpus, where the zero-shot mode outperforms the standard mode. We attribute this phenomenon to an extensive overlap of "Location" entities with the Conll corpus, which is larger compared to the GeoVirus corpus.

Additionally, it is worth noting that we conducted comparison between our zero-shot algorithm and the best-performing zero-shot models for each dataset. However, the same algorithms do not necessarily outperform our approach on other datasets. For instance, [21] obtained a score of 0.21 on MIT Movies, which is significantly lower than other results. Similarly, [23] reported an F-measure of 0.49 for MIT Restaurants, which is comparable to our score. Additionally, all SOTA systems were trained on multiple datasets, with a substantial overlap in common labels, particularly evident in the CoNLL-2003 dataset. Consequently, the task becomes simpler for these entities, making a direct comparison with our
approach less meaningful and honest. These findings raise questions about the
robustness of existing systems.

6 Conclusion

In this work we presented an original approach to zero-shot domain adaptation for NER that requires minimal annotated data. We tested our approach on several datasets from general and biological domains. Our results demonstrate the superior performance of our approach compared to existing zero-shot techniques on some datasets. Looking ahead, future research will focus on examining the robustness and practical applications of this model within existing systems. Additionally, we aim to explore the automation of semantic representations for entity types, further advancing the versatility and effectiveness of our proposed approach.

References

1. Bossy, R., Deléger, L., Chaix, E., Ba, M., Nédellec, C.: Bacteria biotope at bionlp open shared tasks 2019. In: Proceedings of the 5th workshop on BioNLP open shared tasks. pp. 121–131 (2019)

2. Chaudhary, A., Xie, J., Sheikh, Z., Neubig, G., Carbonell, J.: A little annotation does a lot of good: A study in bootstrapping low-resource named entity recognizers. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 5164–5174 (2019)

3. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). pp. 4171–4186 (2019)

4. Doğan, R.I., Leaman, R., Lu, Z.: Ncbi disease corpus: A resource for disease name recognition and concept normalization. Journal of Biomedical Informatics **47**, 1–10 (2014)

5. Gritta, M., Pilehvar, M.T., Collier, N.: Which Melbourne? augmenting geocoding with maps. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 1285–1296 (2018)

6. Iovine, A., Fang, A., Fetahu, B., Rokhlenko, O., Malmasi, S.: Cyclener: an unsupervised training approach for named entity recognition. In: Proceedings of the ACM Web Conference 2022. pp. 2916–2924 (2022)

7. Jia, C., Liang, X., Zhang, Y.: Cross-domain ner using cross-domain language modeling. In: Proceedings of the 57th annual meeting of the association for computational linguistics. pp. 2464–2474 (2019)

8. Kocaman, V., Talby, D.: Biomedical named entity recognition at scale. In: Pattern Recognition. ICPR International Workshops and Challenges: Virtual Event. pp. 635–646 (2021)
9. Le Guillarme, N., Thuiller, W.: Taxonomer: deep neural models for the recognition of taxonomic entities in the ecological and evolutionary literature. Methods in Ecology and Evolution **13**(3), 625–641 (2022)

10. Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: Biobert: a pre-trained biomedical language representation model for biomedical text mining. *Bioinformatics* **36**(4), 1234–1240 (2020)

11. Liu, J., Pasupat, P., Cyphers, S., Glass, J.: Asgard: A portable architecture for multilingual dialogue systems. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 8386–8390 (2013)

12. Ma, J., Ballesteros, M., Doss, S., Anubhai, R., Mallya, S., Al-Onaizan, Y., Roth, D.: Label semantics for few shot named entity recognition. In: Findings of the Association for Computational Linguistics: ACL 2022. pp. 1956–1971 (2022)

13. Palm, R.B., Hovy, D., Laws, F., Winther, O.: End-to-end information extraction without token-level supervision. In: Proceedings of the Workshop on Speech-Centric Natural Language Processing. pp. 48–52 (2017)

14. Popovski, G., Kochev, S., Korousic-Seljak, B., Eftimov, T.: Foodie: A rule-based named-entity recognition method for food information extraction. *ICPRAM* **12** (2019)

15. Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 3982–3992 (2019)

16. Sang, E.T.K., De Meulder, F.: Introduction to the conll-2003 shared task: Language-independent named entity recognition. In: Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003. pp. 142–147 (2003)

17. Shen, Y., Song, K., Tan, X., Li, D., Lu, W., Zhuang, Y.: Diffusionner: Boundary diffusion for named entity recognition. arXiv preprint arXiv:2305.13298 (2023)

18. Wang, J., Hu, Y.: Are we there yet? evaluating state-of-the-art neural network based geoparsers using eupeg as a benchmarking platform. In: Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Geospatial Humanities. pp. 1–6 (2019)

19. Wang, S., Li, X., Meng, Y., Zhang, T., Ouyang, R., Li, J., Wang, G.: *k* nn-ner: Named entity recognition with nearest neighbor search. arXiv preprint arXiv:2203.17103 (2022)

20. Wang, S., Sun, X., Li, X., Ouyang, R., Wu, F., Zhang, T., Li, J., Wang, G.: Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428 (2023)

21. Wang, X., Zhou, W., Zu, C., Xia, H., Chen, T., Zhang, Y., Zheng, R., Ye, J., Zhang, Q., Gui, T., et al.: Instructuie: Multi-task instruction tuning for unified information extraction. arXiv e-prints pp. arXiv:2304 (2023)

22. Wang, X., Jiang, Y., Bach, N., Wang, T., Huang, Z., Huang, F., Tu, K.: Automated concatenation of embeddings for structured prediction. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). pp. 2643–2660 (2021)

23. Zhou, W., Zhang, S., Gu, Y., Chen, M., Poon, H.: Universalner: Targeted distillation from large language models for open named entity recognition. arXiv preprint arXiv:2308.03279 (2023)