SLIMER-IT: Zero-Shot NER on Italian Language
Authors: Andrew Zamai, Leonardo Rigutini, Marco Maggini, Andrea Zugarini (Università di Siena, expert.ai, 2024)
Topic: Language Models

Abstract
Traditional approaches to Named Entity Recognition (NER) frame the task into a BIO sequence labeling problem. Although these systems often excel in the downstream task at hand, they require extensive annotated data and struggle to generalize to out-of-distribution input domains and unseen entity types. On the contrary, Large Language Models (LLMs) have demonstrated strong zero-shot capabilities. While several works address Zero-Shot NER in English, little has been done in other languages. In this paper, we define an evaluation framework for Zero-Shot NER, applying it to the Italian language. Furthermore, we introduce SLIMER-IT, the Italian version of SLIMER, an instruction-tuning approach for zero-shot NER leveraging prompts enriched with definition and guidelines. Comparisons with other state-of-the-art models, demonstrate the superiority of SLIMER-IT on never-seen-before entity tags.

Introduction
Named Entity Recognition (NER) plays a fundamental role in Natural Language Processing (NLP), often being a key component in information extraction pipelines. The task involves identifying and categorizing entities in a given text according to a predefined set of labels. While person, organization, and location are the most common, applications of NER in certain fields may require the identification of domain-specific entities. \centering \includegraphics[width=0.85\linewidth]{SLIMERIT_prompt.png and guidelines steer the model labelling. Manually annotated data has always been critical for the training of NER systems AUTHOR. Traditional methods tackle NER as a token classification problem, where models are specialized on a narrow domain and a pre-defined labels set AUTHOR. While achieving strong performance for the data distribution they were trained on, they require extensive human annotations relative to the downstream task at hand. Additionally, they lack generalization capabilities when it comes to addressing out-of-distribution input domains and/or unseen labels AUTHOR. On the contrary, Large Language Models (LLMs) have recently demonstrated strong zero-shot capabilities. Models like GPT-3 can tackle NER via In-Context Learning AUTHOR, with Instruction-Tuning further improving performance AUTHOR. To this end, several models have been proposed to tackle zero-shot NER AUTHOR. In particular, SLIMER AUTHOR proved to be particularly effective on unseen named entity types, by leveraging definitions and guidelines to steer the model generation. However, little has been done for zero-shot NER in non-English data. More in general, as pointed out in AUTHOR, NER is understudied in languages like Italian, especially outside the traditional news domain and person, location, organization classes. To this end, we propose in this paper an evaluation framework for Zero-Shot NER, and we apply it to the Italian language. In addition, we fine-tune a version of SLIMER for Italian, which we call SLIMER-IT}. In the experiments, we explore different LLM backbones and we assess the impact of Definition and Guidelines (D\&G). When comparing SLIMER-IT with state-of-the-art approaches, either using models pre-trained on English or adapted for Italian, results demonstrate SLIMER-IT superiority in labelling unseen entity tags.

Conclusion
In this paper, we proposed an evaluation framework for Zero-Shot NER that we applied to Italian. Thanks to such a framework, we can better investigate different zero-shot properties depending on the scenario (in-domain, OOD, unseen NEs). On top of that, we compared several state-of-the-art approaches, with particular focus on SLIMER, which, thanks to the usage of definition and guidelines, is well suited to deal with novel entity types. Indeed, SLIMER-IT, our fine-tuned model based on LLaMAntino-3, surpasses other state-of-the-art techniques by large margins. In the future, we plan to further extend the zero-shot NER benchmark, and implement an input caching mechanism for scalability to large label sets. \clearpage