Subcategorization of Italian Verbs with LLMs and T-PAS
Authors: Luca Simonetti, Elisabetta Jezek, Guido Vetere (Università Guglielmo Marconi, Università di Udine, Università di Pavia, 2024)
Topic: Language Models

Abstract
This study explores the application of Large Language Models (LLMs) to verb subcategorization in Italian, focusing on the identification and classification of syntactic patterns in sentences. While LLMs have made lexical analysis more implicit, explicit argument structure identification remains crucial in domain-specific contexts. The research leverages T-PAS, a rich lexical resource for Italian verbs, to fine-tune the open multilingual model Mistral 7B using the Iterative Reasoning Preference Optimization (IRPO) technique. This approach aims to enhance the recognition and extraction of verbal patterns from Italian sentences, addressing challenges in resource quality, coverage, and frame extraction methods. By combining curated lexical-semantic resources with neural language models, this work contributes to improving verb subcategorization tasks, particularly for the Italian language, and demonstrates the potential of LLMs in refining linguistic analysis tools.

Introduction
Verb subcategorization is the task of identifying and classifying the syntactic patterns (or frames) taken by verbs in sentences. These patterns encode the possible combinations of arguments (such as subjects, objects, and complements) that a verb can have, specifying the number and type of arguments as well as their syntactic and semantic roles. Verb subcategorization is often used in Natural Language Understanding (NLU) to provide the main interpretation backbone. Although recent developments brought about by Large Language Models (LLM) make lexical analysis somewhat implicit, there are cases in which the identification of the argument structure of the verb is required, especially those where extensive domain-specific knowledge is required. Semantic lexical resources such as VerbNet AUTHOR, FrameNet AUTHOR and PropBank AUTHOR have been largely employed for several NLP tasks in the past decades, including accomplishing verbal framing for the English language. VerbNet, for example, has been used to improve semantic role labeling, verb sense disambiguation and ontology mapping ( AUTHOR, AUTHOR); its new enhanced semantic representations have also recently been used for entity state tracking AUTHOR. The main problems addressed in these experiences concern the quality and coverage of such resources and the methods used to extract frames from sentences. Neural Language Models can help address both these issues. On the one hand, they may facilitate the construction of curated lexical-semantic resources; on the other hand, they can power robust frame-sentence matching procedures. The present work focuses on the Italian language. It concerns an experiment of using a rich lexical resource for Italian verbs, namely T-PAS AUTHOR to fine-tune an open multilingual model, namely Mistral 7B AUTHOR, to recognize and extract verbal patterns from Italian sentences using a technique called IRPO AUTHOR. The paper is organized as follows: in Section 2 we introduce the T-PAS resource for Italian verbs, which we used in our experiments. Section 3 discusses in detail the methodology we applied and references closely related works, whereas Section 4 illustrates the experimental setup. We complete the paper by discussing our results in Section 5 and by drawing some conclusions as well as making suggestions for future research in Section 6.

Conclusion
In conclusion, we can say that small multilingual baseline models such as Mistral 7B perform poorly on semantic analysis of Italian sentences. We observe that the poor behavior is due to the model's inability to discern the correct answer, either because it lacks the linguistic knowledge, therefore mostly resorting on random guesses, or because it follows an incorrect explanation for the answer is about to give. However, our research also demonstrates that the model can be significantly improved using IRPO techniques without affecting the baseline performance on common sense and reasoning tasks. Notably, we observe the ability to generalize across predicates, likely due to underlying linguistic skills, though further investigation is needed to fully understand this phenomenon. The production of small open language models is rapidly evolving, approaching the level of huge close models which were available on the cloud a couple of years ago. At present, Italian monolingual models have room for improvement in terms of performance levels, } while multilingual models, e.g. the recently released Gemma 2 AUTHOR, show increasing proficiency in our language, probably due to transfer learning effects. Our research shows the potential of leveraging such models in combination with high-quality lexical resources to develop a new class of task-specific models for the Italian language. These models, while small in scale, are expected to exhibit remarkable proficiency in executing complex analytical tasks, such as those related to verbs. With this in mind, our future work is aimed, on the one hand at enriching lexicographic resources and refining the ways to obtain training material from them, and on the other hand at continuously evaluating the improvements brought about by the progress of general-purpose open models. One promising application is the use of a verbal subcategorization and frame extraction system to extract content from specialist documents, such as legal AUTHOR or medical texts AUTHOR. Furthermore, the ability to analyze the complex argument structure of verbs has potential for use in language learning systems AUTHOR, e.g. providing support for immigrants to learn Italian affordably. Finally, we made our fine-tuned model publicly available on huggingface\footnote{ https://huggingface.co/theGhoul21/srl-base-irpo-080524-16bit-v0.3-lighning-ai-6000 along with a visual report on wandb. \appendix