BaBIEs: A Benchmark for the Linguistic Evaluation of Italian Baby Language Models
Authors: Luca Capone, Alice Suozzi, Gianluca E. Lebani, Alessandro Lenci (Università di Pisa, Università Ca' Foscari Venezia, 2024)
Topic: Language Models

Abstract
The possibility of comparing the linguistic competence of Language Models (LMs) to that of children has gained growing attention lately, raising the need for effective tools for evaluating both the former and the latter. To this purpose, we developed a resource for the linguistic evaluation of BabyLMs, which are LMs trained on datasets that comparable to the linguistic stimulus received by children. This resource adapts four standardized tests for the evaluation of linguistic skills of Italian-speaking children (BVL, TROG-2, TCGB-2 and Peabody). To verify the effectiveness of our benchmark, we administered it to Minerva, a LLM pretrained from scratch on Italian. Our results indicate that Minerva struggles to master certain linguistic aspects, achieving an age-equivalent score of 4 years, and that the type of task administered affects the model's performance.

Introduction
This paper presents BaBIEs (Baby Benchmark for Italian linguistic Evaluations), a new resource for the standardized evaluation of Italian BabyLMs, that is, language models (LMs) trained on datasets that are qualitatively and quantitatively comparable to the type of stimulus received by humans during language acquisition. The aim of this resource is twofold: [label=(\roman*)] \item to evaluate the quality of the training data and strategies, in particular curriculum learning techniques, used in the development of BabyLMs and \item to provide a benchmark for comparing the performance of LMs, especially BabyLMs, with that of young human speakers. The paper is structured as follows: Section sota reviews related work and delineates the rationale for this study; Section babies details the characteristics of the BaBIEs benchmark, which results from the adaptation of standardized tests for evaluating the linguistic abilities of Italian-speaking children. In Section test, we report a first test of the dataset with the Minerva Italian LM. The benchmark effectiveness is discussed in the light of the experiments in Section discussion. Finally, in Section conclusion, some conclusions and possible future research directions are outlined.

Conclusion
This paper presents BaBIEs, a novel resource specifically designed to evaluate the linguistic competence of BabyLMs and compare them to those of children. After having detailed the sources and the creation process of this resource, we provided the procedure for testing the Minerva model with the resource itself. Finally, we presented and discussed the results the model’s performance. Based on the presented findings, the resource appears a valuable tool for evaluating not only BabyLMs but LMs in general. The poor performance exhibited by Minerva underscores the gap between child language acquisition and current language model training. This highligths the necessity for modifying model training to better encode human language and, more generally, human linguistic competence. Future work will involve a more systematic linguistic analysis of the model's performance, together with a comprehensive error analysis and a comparison to adult Italian-speakers. Furthermore, it will involve the development of a multimodal version of the test, which will more closely reflect the original tests and allow the evaluation of multimodal BabyLMs. Additionally, a BabyLM trained exclusively with Italian child-directed speech will be developed and evaluated with both the standard and multimodal versions of the test. We acknowledge financial support under the PRIN 2022 Project Title "Computational and linguistic benchmarks for the study of verb argument structure" – CUP I53D23004050006 - Grant Assignment Decree No. 1016 adopted on 07/07/2023 by the Italian Ministry of University and Research (MUR). This research was also partly funded by PNRR—M4C2—Investimento 1.3, Partenariato Esteso PE00000013—“FAIR—Future Artificial Intelligence Research”—Spoke 1 “Human-centered AI,” funded by the European Commission under the NextGeneration EU programme. \newpage \appendix \clearpage \onecolumn