Challenging specialized transformers on zero-shot classification
Authors: Serena Auriemma, Mauro Madeddu, Martina Miliani, Alessandro Bondielli, Lucia C. Passaro, Alessandro Lenci (Università di Pisa, 2023)
Topic: Language Models

Abstract
This paper explores the feasibility of using basic prompting systems with domain-specific language models, focusing on bureaucratic language through experiments with the recently introduced BureauBERTo model. The experiments reveal that, although additional pre-trained models show reduced robustness in terms of general knowledge, they exhibit higher adaptability for domain-specific tasks, even in a zero-shot setting. These findings highlight the potential of simple prompting systems in specialized contexts, offering valuable insights for both research and industry.

Introduction
Pre-trained Language Models (PLMs) have greatly impacted Natural Language Processing (NLP), establishing the pre-train and fine-tune paradigm as the leading approach for a wide range of downstream tasks. However, a major limitation of PLMs is the scarcity of annotated data required to fine-tune classification layers, particularly for domain-specific applications. Recently, prompt-based tuning has emerged as a promising alternative that significantly reduces the need for annotated data, proving especially effective with Large Language Models (LLMs). Unfortunately, LLMs are often unavailable for low-resource languages and exhibit reduced performance in specialized domains. To address these issues, we tested BureauBERTo, a model pre-trained on Italian bureaucratic texts, in a zero-shot setting using prompt-based tuning. Given BureauBERTo’s accuracy in fill-mask tasks, we further examined its domain-specific lexical knowledge for two classification tasks in the public administration (PA) domain: predicting PA text topics and identifying generic and PA-related named entities in administrative sentences. We conducted two prompting experiments per task, first using Italian classification labels as label words and then assigning in-domain terms to each class, also comparing BureauBERTo to the Italian generic PLM UmBERTo. Our findings indicate that in zero-shot classification, where label words are only loosely related to the text or entity, both generic and domain-specialized models perform poorly. However, when classes are represented by semantically aligned multi-word labels, PLMs significantly improve, particularly the domain-adapted BureauBERTo, which outperformed UmBERTo in both tasks. This suggests that the additional domain-specific pre-training phase equips BureauBERTo with valuable lexical knowledge for prompt-based tuning, especially in scenarios where word knowledge plays a critical role.

Conclusion
In this paper, we propose a zero-shot prompt tuning classification approach for solving two tasks related to the Italian PA domain: the classification of documents according to their topic and the recognition of the entity types occurring in administrative sentences. We compared the results obtained in these two tasks by the PA-specialized model BureauBERTo with those of the domain-agnostic model UmBERTo. Our findings show that by enriching with domain terms the set of word labels encoded in the prompt verbalizer both models demonstrated enhanced performances. Moreover, BureauBERTo exhibited an improvement over UmBERTo of +0.06 Weighted Average F1 score in the document classification (0.51 vs. 0.57) and of more than twofold in the entity typing task (0.516 vs. 0.368 for Macro Average F1 score), meaning that the domain adapted model is more proficient in leveraging domain-specific terminology. These results underscore the importance of tailoring language models to specific domains to unlock their full potential and address the nuanced challenges posed by diverse subject matters. However, it is also worth mentioning that we noticed a high variability in the task results according to different prompting and different label words. In particular, when the label words adopted to depict a certain topic class are, within the domain context, semantically related to the label words of another class, the models’ classification output seems to be biased in favor of one of the two classes. In conclusion, our study underscores the critical need for a thorough exploration of prompt engineering, particularly in the context of the entity typing task. This imperative arises not only from the potential to augment the predictive capabilities of models, but also from the need to consolidate the knowledge related to general entity classes. Notably, the Public Administration (PA) domain exhibits distinctive characteristics, both in terms of referencing entity names within documents and employing domain-specific terminology. Notably, the identified patterns within the PA domain deviate from the broader, general-purpose Italian style, indicating the necessity for tailored, domain-specific prompt experimentation. This investigative effort shed the linguistic intricacies that exert an impact on Transformer model performance. Our findings, as revealed in the ablation study on entity linking, emphasize the pivotal importance of delving into the interplay among different entity classes present in datasets. A nuanced analysis of how these classes interact and potentially overlap is indispensable for honing the model’s ability to distinguish between them in a domain-specific context. To conclude, this leads us to surmise as a future direction for our work a further inspection of how domain-adapted PLMs encode in their embedding the semantics of domain-related terms and how this information relates to their performance in prompt-based tasks.