Minimalist Entity Disambiguation for Mid-Resource Languages
Year: 2023

Abstract:
For many languages and applications, even though enough data is available for training Named Entity Disambiguation (NED) systems, few off-the-shelf models are available for use in practice. This is due to both the large size of state-of-the-art models, and to the computational requirements for recreating them from scratch. However, we observe that in practice, acceptable models can be trained and run with far fewer resources. In this work, we introduce MiniNED, a framework for creating small NED models from medium-sized datasets. The resulting models can be tuned for applicationspecific objectives and trade-offs, depending on practitioners’ requirements concerning model size, frequency bias, and out-of-domain generalization. We evaluate the framework in nine languages, and achieve reasonable performance using models that are a fraction of the size of recent work.

text: Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 299–306 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Motivation and Problem. Named Entity Disambiguation (NED), is the task of linking preidentified entity names to their corresponding entries in a knowledge base, such as Wikipedia. As a crucial component of Entity Linking (EL) applications, it has been extensively studied for almost two decades. Presently, powerful state-of-the-art EL systems are available based on (English or multilingual) Neural Language Models (Botha et al., 2020; Wu et al., 2020; van Hulst et al., 2020; De Cao et al., 2022). Unfortunately, for many languages and applications there are few off-the-shelf models that are easy to distribute, customize, or use in constrained practical settings. This is due to the large size of trained models, as well as the computational requirements for creating them from scratch. As a result, EL systems are often unavailable or too large to run for many applications.
Approach. In this work, we focus on midresource languages (e.g. Persian, Japanese, and
Tamil), which have some linguistic resources and tools available but not many (Ortiz Suárez et al., 2020). We claim that for many of these languages, simpler and smaller models can perform well enough with careful trade-off analyses. Our main observation is that here the range of reasonable NED performance is quite narrow. In other words, the lower bound (i.e. simply predicting the most commonly linked entity for a given name) and the non-zero-shot upper bound (i.e. perfectly disambiguating all names that are seen in training) are very close together. Based on this insight, we demonstrate that small NED models can achieve acceptable performance with limited resources. We examine the trade-offs between model size and performance for different configurations and highlight the importance of language-specific phenomena, such as morphological differences, in determining optimal parameter settings. We argue that the tuning of NED models for mid-resource languages requires careful consideration and can only be done sustainably on small models.
Contribution. We introduce and evaluate MiniNED1, a Python library for creating NED models from Wikipedia data in many languages. We show that much simpler models than state-of-theart systems can achieve acceptable performance in practice. We also show how our framework allows practitioners to control model complexity and adjust for specific use-cases, while maintaining performance.

heading: 2 Observations
text: Examining the Mewsli-9 benchmark (Botha et al., 2020), we can make several observations about the distribution of data that is available in Wikipedia2 for training NED models. 1https://github.com/bennokr/miniNED 2All experiments were performed on Wikipedia dumps from 2022-03-01. In Mewsli-9, we replace English by Dutch due to our focus on mid-resource languages.
299
In Figure 1, we show the distribution of how often entities from the benchmark data are hyperlinked on Wikipedia. While we observe a typical long-tailed distribution, most entities can still be observed between 10-1000 times. For these languages, this provides enough training data to learn to disambiguate entity mentions from their context.
In Figure 2, however, we observe that the baseline performance of simply predicting the top most commonly linked entity for a given ambiguous name (combined with straightforwardly linking unambiguous names) can already achieve relatively high performance. Additionally, many entity-name pairs (which we will refer to as mentions) in the benchmark data cannot be observed in training at all; such unseen cases would require zero-shot generalization. Consequently, the upper and lower bound for simple models are very close together. Thus, we may comclude that the main challenge lies in predicting shadowed entity mentions (Provatorova et al., 2021), which share a surface form with more popular entities. Due to the overwhelming imbalance of training instances for shadowed entities, particular attention should be given to selecting appropriate training data and to the assumptions that underly the model.
Another observation is that language-specific phenomena make a big difference. The distribution of observed ambiguity changes when names are stemmed. Stemming removes word inflections, which increases the ambiguity of names. We can see that due to morphological differences, the effect of stemming is very different per language. Overall, stemming decreases the number of unseen mentions, but also widens the range of ambiguous names.
Finally, Wikipedia hyperlink data is noisy (Gerlach et al., 2021), as it includes links to disambiguation pages and incorrect entity links. This becomes
a larger problem when less data is available (as for low-resource languages and domains). We argue that optimally tuning the training pipeline to overcome this noise (by fixing this data or patching the model) can only be done sustainably when the model itself is simple.

heading: 3 Approach
text: We train multinomial logistic regression classifiers with hashed Bag-of-Word features, which are trained to rank candidate entities using Vowpal Wabbit (vw, Langford et al., 2007). The candidates are created by filtering entity mention counts from Wikipedia dumps using heuristics. These heuristics identify valid surface forms based on their appearance on disambiguation pages, their string similarity to the entity labels, and the entropy of the prior probability distribution of hyperlink targets. Re-
garding the entropy, a cutoff threshold determines when a long, flat prior distribution is discarded (e.g. specific villages for the anchor text “village”). This also leads to anchor texts with flat distributions of very similarly frequent targets to be discarded entirely (e.g. for strings such as “click here”).
The candidate set can be further filtered by only selecting a percentage (i.e. quantile) of most frequently observed ambiguous mentions. In Figure 3, we show the tradeoff between candidate mention filtering and the maximum recall that a model trained to disambiguate between these candidates could achieve. We observe that selecting only the most frequently mentioned entities results in quick gains with only a fraction of the candidate set size. However, we also observe that our pre-processing heuristics discard valid mentions for some languages, so that even on the full set of candidates, perfect recall cannot be attained.
The vw model size is controlled by the number of bits of the feature hash, which also works as regularisation to prevent overfitting. By exchanging single coefficients per mention-feature pair for smaller models with more hash collisions, we are able to find the optimal tradeoff between model size and accuracy using a hyperparameter sweep. Although the features are hashed, the model can still be audited by keeping track of feature hashes for specific analyses. This can be useful for explaining individual predictions (showing which context words have a strong influence), or examining the
coefficients that are used to disambiguate a single surface form.

heading: 4 Evaluation
text: Our analysis compares models of different sizes and candidate filtering thresholds, and the effect of stemming in different languages. We modify the Mewsli-9 benchmark to discard links to disambiguation pages and list pages (statistics in Appendix B), and we generate the Dutch data using the scripts provided by Botha et al. (2020). We train on lowercased mentions that occur more than once, which are filtered by discarding names which both (1) have less than 10% of tokens appear in an entity label on Wikidata and (2) have a high candidate entropy (> 1 nat), except if they are used as
main links on disambiguation pages. For stemming, we use PersianStemmer (Taghi-Zadeh et al., 2015), MeCab for Japanese (Kudo, 2006), and Snowball for other languages (Porter, 2001).
In Figure 4, we report normalized disambiguation micro-F1 scores, where per-language scores are transformed with respect to the baseline (the most frequent target of entity labels) and the upper bound (on observed mentions). Unnormalized results are presented per language for bestperforming models in Table 1 and in Appendix A.
We observe that the effect of stemming and the trade-off between model size and performance is different per language, but clear trends are visible, with diminishing returns of model sizes above a few hundred MB.
Explainability and Denoising. By keeping track of which features hash to which parameters for a set of example instances, we can visualize which context words have a strong influence on model predictions (Figure 5). This is useful for improving models for which the training data may have been noisy, allowing practitioners to modify preprocessing pipelines or employ data re-labeling efforts.

heading: 5 Related Work
text: Mewsli-9 was introduced by Botha et al. (2020) and used for evaluation by De Cao et al. (2022). Our evaluation results are not directly comparable to theirs because we remove links to disambiguation pages and list pages.
Some EL systems for mid-resource languages exist. Most prominently, DBpedia Spotlight (Daiber et al., 2013) publish EL models for some languages, but these are not tunable for size. Tsai and Roth (2016) perform cross-lingual wikification using multilingual embeddings; we plan to replace our BoW features by such embeddings in future work.
Pappu et al. (2017) train lightweight multilingual entity linking models, but not for mid-resource languages. Gerlach et al. (2021) focus on precision, while we focus on F1-scores and model size.
Modern EL models often combine Mention Detection (MD) and NED end-to-end. Hachey et al. (2013) describe the interplay of MD and NED in English EL. Ling et al. (2015) extend this description, and make similar observations to ours about NED baselines. Kolitsas et al. (2018) are the first to train end-to-end neural EL models, improved later by De Cao et al. (2021). These efforts were extended to multilingual models by Botha et al. (2020) and De Cao et al. (2022). Such end-to-end neural models require many GPU-hours to train, making it impossible to tune them sustainably for specific applications. In contrast, we focus on the smallest possible NED models, because small MD models can be achieved with the use of gazetteers and their interplay may be optimized by tuning.

heading: 6 Conclusion and Future Work
text: We introduce and evaluate MiniNED, a Python library for creating NED models from Wikipedia data in many languages. We show that much simpler models than state-of-the-art systems can achieve acceptable performance in practice. We also show how our framework allows practitioners to control model complexity and adjust for specific use-cases, while maintaining performance.
For future research, we expect this approach to also be useful when incorporating more background knowledge about the entities with richer feature representations and using weak supervision (Orr et al., 2021). Also, the tradeoffs that this work analyses are strongly related to the difficulty of determining the appropriate granularity of EL systems (Van Erp and Groth, 2020). For example, the knowledge base that the mentions are linked to may distinguish between municipalities as admin-
istrative entities and the cities within them, while for many applications this distinction is not relevant. In the future we aim to add support for tuning models to the desired levels of granularity.

heading: Acknowledgements
text: This research is partially funded by Huawei Amsterdam Research Center.
I would like to thank Thiviyan Thanapalasingam, Majid Mohammedi and Erman Acar for their early feedback on language-specific models, and the members of the VU Amsterdam Knowledge in AI and Learning & Reasoning groups, Winston Wansleeben, Rens Hassfeld and Mara Spadon for their feedback on drafts of this work.

heading: A Evaluation Results
text: Micro-F1 scores per model size. Line width: Candidate selection filtering quantile. Color: Use of fallback to baseline (most frequent target). 0.6 0.7 F 1 -S co re
Tamil, Unstemmed Tamil, Stemmed
0.7
0.8 0.9 F 1 -S co
re Persian, Unstemmed Persian, Stemmed
0.7
0.8
0.9
F 1 -S
co re
Turkish, Unstemmed Turkish, Stemmed
0.7
0.8
F 1 -S
co re
Dutch, Unstemmed Dutch, Stemmed
0.8
0.9
F 1 -S
co re
Arabic, Unstemmed Arabic, Stemmed
0.6
0.8
F 1 -S
co re
Serbian, Unstemmed Serbian, Stemmed
0.7
0.8
F 1 -S
co re
Spanish, Unstemmed Spanish, Stemmed
0.6
0.8
F 1 -S
co re
Japanese, Unstemmed Japanese, Stemmed
16 1MB
20 16MB
24 256MB
28 4GB
Hash Bits; Model Size
0.7
0.8
0.9
F 1 -S
co re
German, Unstemmed
16 1MB
20 16MB
24 256MB
28 4GB
Hash Bits; Model Size
German, Stemmed
Fallback False True Quantile 100% 50% 25% Pre-processing Unstemmed Stemmed
Baseline Base Base Fallback Base Fallback Stemmed Upper Bound Upper Fallback Upper Fallback Stemmed

heading: B Mewsli-9 Modification

heading: Future Work
text: For future research, we expect this approach to also be useful when incorporating more background knowledge about the entities with richer feature representations and using weak supervision (Orr et al. , 2021). Also, the tradeoffs that this work analyses are strongly related to the difficulty of determining the appropriate granularity of EL systems (Van Erp and Groth, 2020). For example, the knowledge base that the mentions are linked to may distinguish between municipalities as admin-
istrative entities and the cities within them, while for many applications this distinction is not relevant. In the future we aim to add support for tuning models to the desired levels of granularity