Venue: NAACL 2025

# Italic:

_An Italian Culture-Aware Natural Language Benchmark_

Andrea Seveso\({}^{1,3}\), Daniele Poterti\({}^{2}\), Edoardo Federici, Mario Mezzanzanica\({}^{1,3}\), Fabio Mercorio\({}^{1,3}\)

###### Abstract

We present \(\tt ITAIC^{1}\), a large-scale benchmark dataset of 10,000 multiple-choice questions designed to evaluate the natural language understanding of the Italian language and culture. \(\tt ITAIC\) spans 12 domains, exploiting public tests to score domain experts in real-world scenarios. We detail our data collection process, stratification techniques, and selection strategies. \(\tt ITAIC\) provides a comprehensive assessment suite that captures commonsense reasoning and linguistic proficiency in a morphologically rich language. We establish baseline performances using 17 state-of-the-art LLMs, revealing current limitations in Italian language understanding and highlighting significant linguistic complexity and cultural specificity challenges. \(\tt ITAIC\) serves as a benchmark for evaluating existing models and as a roadmap for future research, encouraging the development of more sophisticated and culturally aware natural language systems.


coverage of common sense and cultural knowledge. To evaluate whether an LLM aligns with Italian culture, we propose **ITALIan**Cultural Benchmark (ITALIC), a novel dataset using a set of cultural questions taken from exams for public competitions in Italy. These tests assess candidates' cultural knowledge, and applying them to LLMs would provide a useful tool to identify models not aligned with Italian culture. Fig. 1 shows illustrative examples from ITALIC.

Standardised aptitude and knowledge assessments are used in public domains such as university admissions, military recruitment, public sector positions, and medical license examinations, among others. The primary aim of these tests is to evaluate applicants' domain knowledge, general knowledge, and linguistic and analytical capabilities, which are deemed essential skills that public institutions and employers prioritise by Italian national standards requirements. These assessments are crucial for a merit-based selection process across diverse fields and positions Ruffini et al. (2023). In addition to evaluating general knowledge and analytical skills, many standardised assessments include a dedicated section focused on language and culturally relevant topics, reflecting Italy's rich cultural heritage. It is an essential criterion for public and professional roles where cultural awareness and communication skills are paramount, and candidates must be aligned with the cultural norms and values embedded within the Italian professional and public spheres.

Since these culturally relevant sections present a unique challenge for language models, our study seeks to create a standard evaluation suite for advanced large language models by utilising the structure of Italian public exams as a comprehensive testing framework. The tests' structured and standardised nature makes them an excellent benchmark for comparing different LLMs with questions that are contextually relevant to Italian culture.

### Contributions

The main contributions of our work are three-fold:

1. We design and publicly release ITALIC, a benchmark dataset designed to evaluate LLMs' Italian cultural and linguistic understanding across diverse domains, including history, geography, literature, and civic knowledge.
2. We comprehensively categorise the dataset into cultural and linguistic domains, enabling a clear assessment of models' performance across specific tasks.
3. We empirically evaluate and discuss 17 different model configurations, showing the current limitations of LLMs in understanding Italian culture and language and highlighting areas where future improvements in culturally aware NLP are needed.

## 2 Related Work

Large Language Models are deep learning models trained on vast amounts of multilingual text data, capable of understanding and generating language in a sophisticated manner Armengol-Estape et al. (2022); Le Scao et al. (2023). While exhibiting emerging capabilities across multiple languages such as German, French, Spanish, and Italian, multilingual models do not perform as well as in the primary training language Touvron et al. (2023); Jiang et al. (2024). Therefore, it is essential to evaluate their performance using language-specific metrics.

Available English Benchmarks.Several benchmarks have been developed to assess language models' general intelligence and reasoning capabilities. Notably, MMLU Massive Multitask Language Understanding Hendrycks et al. (2021) is a comprehensive benchmark that evaluates language models across 57 diverse subjects, including STEM, humanities, and social sciences. AGIEval Zhong et al. (2024), on the other hand, is designed to assess the capabilities of models in artificial general intelligence across a broad range of tasks and skills.

Cultural Benchmarks.Cultural benchmarks have emerged as crucial tools in evaluating the adaptability and fairness of language models across different social and cultural contexts Hershcovich et al. (2022). Several studies have aimed to quantify how well these models perform across diverse demographic groups and cultural settings.

NORMAD Norms and Adaptability of Language Models Rao et al. (2024) focuses on the cultural adaptability of LLMs. By utilizing 2,600 stories from 75 countries, NORMAD assesses how well these models understand and replicate social norms and etiquette as proxies for cultural comprehension. In a complementary effort, the BLEND benchmark Myung et al. (2024) expands the scope by offering a dataset of 52.6k question-answer pairs 

spanning 13 languages and 16 countries/regions. This large-scale study reveals a significant imbalance in how LLMs process cultural knowledge, with models often performing better on questions related to highly represented cultures. BLEND highlights the need for more equitable training approaches that ensure underrepresented cultures are not neglected in model performance. Lastly, CLICK (Cultural and Linguistic Intelligence in Korean) Kim et al. (2024) provides a localised examination by testing models on their understanding of Korean culture and language. Based on 1,995 QA pairs from official exams and textbooks, the benchmark covers 11 categories related to Korean cultural knowledge, thereby addressing the linguistic and cultural nuances specific to the region.

Each benchmark focuses on a different aspect--social biases, adaptability to diverse social norms, cultural representation, or linguistic specificity--yet they all contribute to a broader understanding of the limitations and potentials of LLMs in culturally complex environments. While the above benchmarks have been instrumental in advancing the field, they primarily focus on English or specific cultural contexts. Some may include some tests in Italian but are not focused on Italian cultural commonsense and language capabilities.

Italian benchmarksCompared to the English community, the Italian NLP community lacks the depth of original language evaluation benchmarks Basile et al. (2023) present a Unified Benchmark for Italian Natural Language Understanding, which includes tasks such as textual entailment, event detection and classification, factuality classification, sentiment polarity classification, irony detection, and hate speech detection. Lai et al. (2023) propose a collaborative benchmark covering 13 tasks. Despite their contributions, both benchmarks mainly address classification-based tasks and do not examine LLM capabilities like commonsense reasoning. Landro et al. (2022) provides an Italian benchmark for news text summarization. Mercorio et al. (2024) introduces an adaptation of the INVALSI standardised tests for the automated evaluation of LLMs, aiming to improve the assessment of these models within diverse Italian linguistic and cultural contexts.

Our work aims to build upon these foundations by developing a specialised benchmark tailored for cultural knowledge assessment specific to Italian.

## 3 Dataset Curation

ITALIC contains 10,000 carefully curated questions selected from an initial corpus of 2,110,643 questions. The dataset curation process overview is summarised in Fig. 2. The initial data was sourced from various files in PDF, HTML, DOC, and other formats published by official bodies that announce individual competitive public examinations3. The preprocessing process involved standard crawling techniques, including rate-limited requests to ensure politeness and acquire publicly available documents from official governmental sources and public registries. Document collection was followed by format conversion from various source formats (HTML, PDF) using regex and standard libraries. The corpus comprises questions and tasks from real-world exams, professional assessments, and domain-specific challenges. This ensures the dataset reflects the knowledge and skills required in Italy's various professional and academic contexts, like medical and military. Given that the data originates from institutional sources, it is expected to maintain a high standard of quality and accuracy, as domain experts crafted it for public

Figure 2: Overview of the data collection and curation process for the ITALIC benchmark.

 

evaluations. The collected data, including the questions, answers, relevant metadata, and associated images, were normalised into JSON. Each entry follows a standard multiple-choice format with a single correct option, providing a clear structure for model evaluation.

### Dataset Categorisation

We filtered out any questions that lacked answers or were incorrectly formatted, aiming to preserve only high-quality entries. We also removed any question that required reading and understanding a contextual paragraph and any question containing images. We conducted deduplication for questions that shared the same question and answer options. As a result, some questions differ only in answer options, which we treat as distinct items for evaluation purposes. However, these items will not appear together in the final dataset during subsequent processing steps due to their similarity. We applied simple preprocessing techniques to clean the dataset, such as removing question numbers and artefacts that appeared at the beginning of some questions. These artefacts included formatting characters, extraneous symbols, or other irrelevant text that could interfere with the clarity of the questions.

Using the collected metadata, the data was refined to include only questions from the dedicated cultural sections of the tests that fell into two main categories: _Culture and Commonsense_ and _Language Capability_. The Culture and Commonsense category encompasses history, geography, civic education, current events, literature, art history, and tourism, reflecting Italy's cultural and social context. The Language Capability category includes tasks that assess knowledge of synonyms and antonyms, orthography, vocabulary, morphology, and syntax, which are crucial for evaluating linguistic proficiency in Italian. These subcategories assess a model's ability to understand Italian grammar rules, word formations, spelling conventions, and linguistic richness. Tasks related to Syntax determine sentence structure and grammatical correctness, while Morphology tests the model's ability to process word inflexions and conjugations, which are particularly complex in Italian. Lexicon addresses vocabulary knowledge, while Synonyms and Antonyms test semantic understanding and Orthography ensures models can accurately handle Italian spelling rules. After filtering, the dataset was reduced to 1,193,911 questions (56% of the initial).

**Assessing cultural relevance in the Italian context.** To evaluate the cultural significance of the questions, we employ the LLM-as-a-Judge methodology, using LLMs to assess the quality of responses based on a given prompt. Previous work Yuan et al. (2024) uses this technique to evaluate task difficulty, asking the LLM judge to determine the response using five additive criteria (relevance, coverage, usefulness, clarity and expertise), covering various quality aspects. We adopt this methodology, asking not one but three LLMs: GPT-4o-mini Achiam et al. (2023), Gemini-flash-1.5 Reid et al. (2024) and Llama 3.1 70b Dubey et al. (2024). Our filtering criteria require that each question not be time-sensitive or dependent on events currently happening and must be closely related to aspects of Italian culture, such as language, history, art, and other relevant cultural elements. The prompt to rate the LLMs is shown in Fig. 3.

Based on these criteria, we retain only those items that all three models deem appropriate in every category. This filtering process removes 27.8% of questions.

Meaningfully reducing the sample size.Finally, to further reduce the size of the dataset and ensure the benchmark is not too expensive to run, we apply Farthest Point Sampling (FPS) Eldar et al. (1997) to extract a sample of 10,000 questions. FPS iteratively selects the item farthest from the already selected items in the embedding space. This ensures the selected subset is diverse, removing items with similar questions and options, and captures a wide

Figure 3: Cultural relevance assessment prompt.

 

range of characteristics in the entire dataset. We first create word embeddings of each question in the dataset using the highest-performing model4 on the MTEB leaderboard (Muennighoff et al., 2023). We then apply the FPS implementation proposed in Li et al. (2022).

Footnote 4: dunzhang/stella_en_1 ̇5B ̇v5

### Dataset Description

The dataset comprises 10,000 questions, comprehensively evaluating models' abilities to understand and reason within the Italian context. Tab. 1 provides a detailed breakdown of the number of questions in each category, showcasing the distribution and variety within the dataset. Each question is formatted as a multiple-choice query, with an average question length of 87 characters and a median of 4 answer options. The longest question in ITAIC is 577 characters long. The minimum number of choices per question is 2, while the maximum is 5. The total number of tokens across the input data amounts to 499,963.

### Evaluation Strategy

Our approach is consistent with standard LLM evaluation frameworks like OpenAI's simple-evals5. We generate responses for open-source models and API-based LLMs and then compare these to the labelled correct answers. We use a decoding temperature of 0 to ensure deterministic output in this setting. Each evaluation has been run exactly once. We use zero-shot and five-shot prompts to ensure a fair assessment. We provide the models with no examples or demonstrations in the zero-shot scenario. In contrast, in the five-shot evaluation, in each question, we include in the prompt five high-quality examples that have been excluded from ITAIC during the previous filtering phase. The models are instructed to respond with only the corresponding letter (e.g., A, B, C, D) without providing any explanations or further reasoning. The evaluation is strictly based on identifying the correct answer letter, using a simple pattern-matching technique to find the first occurrence of the predicted letter in the response. The primary metric to measure performance is accuracy, reflecting the proportion of correctly answered questions.

Footnote 5: [https://github.com/openai/simple-evals](https://github.com/openai/simple-evals)

Models employed.Our evaluation includes closed-source and open-source language models and models specifically fine-tuned for the Italian language. We include only instruction-tuned models, ensuring they are optimised for following prompts and generating responses in alignment with specific instructions. Among the closed-source models, we evaluate OpenAI's GPT-4o-mini (Achiam et al., 2023) and GPT-4o, along with Anthropic's Claude 3 Haiku and Claude 3.5 Sonnet. Additionally, we include Google's Gemini series, specifically Gemini Flash and Pro 1.5 Reid et al. (2024). For open-source models, we assess Meta's Llama 3.1 series, including the 8b, 70b and 405b versions Dubey et al. (2024), as well as Mistral's models, Mistral Nemo and Large Jiang et al. (2024). To ensure a focus on Italian-specific capabilities, we also evaluate models fine-tuned for Italian, such as LLaMAntino-3 Polignano et al. (2024), popular models Llama-3.1-8b-Ita6 and maestralechat-v0.47, as well as the foundational models iGeniusAI's Italia 9B8, Minerva 7B Orlando et al. (2024) and Almawave's Velvet 14B9.

Footnote 6: DeepMount00/Llama-3.1–8b-Ita

Footnote 7: mii-llm/maestralehar-ct-v0.4-beta

Footnote 8: GeniusAI/Italia-9B-Instruct-v0.1

Footnote 9: Almawave/Velvet-14B

#### Maintenance

ITALIC provides a snapshot of language and cultural knowledge based on existing datasets and is designed to be robust and fully operational upon release, with no need for routine maintenance. However, as language and cultural norms evolve, periodic updates will be required to ensure the benchmark remains relevant. A new dataset version will be created and made available in such cases.

Previous versions of ITAIC will remain acces

\begin{table}
\begin{tabular}{l l r} \hline \hline \multicolumn{1}{c}{**Category**} & \multicolumn{1}{c}{**Subcategory**} & \multicolumn{1}{c}{**\#**} \\ \hline  & Art History & 980 \\  & Civic Education & 973 \\  & Current Events & 92 \\
**Cluture and Commonsense** & Geography & 979 \\  & History & 978 \\  & Literature & 984 \\  & Tourism & 980 \\ \hline \multirow{3}{*}{**Language Capability**} & Lexicon & 979 \\  & Morphology & 140 \\  & Orthography & 971 \\  & Synonyms and Antonyms & 971 \\  & Syntax & 973 \\ \hline \multicolumn{1}{c}{**Total**} & \multicolumn{1}{r}{10000} \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of questions in ITAIC divided by _Culture and Commonsense_ and _Language Capability_.

 

sible and maintained through Zenodo, which offers a detailed version history to allow users to refer back to earlier iterations as needed. Researchers are encouraged to extend, augment, or contribute to the dataset. Those interested in making contributions can contact us with proposed additions or modifications, and after verification, the new data will be integrated into the existing dataset. A new version will then be released, ensuring all users can access the latest updates.

## 4 Results and Discussion

In this section, we present the performance of LLMs on the ITAIC benchmark, assessing their proficiency in Italian cultural and linguistic aspects. We evaluated the open-source models on an NVIDIA A100 80GB PCIE GPU. For this evaluation, we utilised VLLM with its default OpenAI-compatible server, running models in bf16 format. The entire process, including setup and evaluation, took approximately 5 hours. Closed-source models and models with over 70 billion parameters were accessed from API services, costing around 100$. Tab. 2 shows the accuracy (%) of each model across several categories within the Italian cultural and linguistic context.

Model performanceAmong the models tested, Anthropic's Claude 3.5 Sonnet demonstrates the highest overall accuracy across cultural and language tasks, closely followed by GPT-4o. The open-source models also perform well, with the 405b version of LLaMa standing out.

As expected, larger models tend to achieve higher scores, aligning with scaling laws Kaplan et al. (2020) that indicate performance improves with increased training compute, dataset size and model parameters. When comparing Italian-specific models to multilingual models, we observe that although the former are fine-tuned or trained on a large percentage of Italian data, they still struggle to outperform larger models trained with a greater computational budget and higher-quality multilingual data. Italian fine-tuning of models like Llama does result in a few percentage points of improvement over the multilingual baseline. However, the performance of the foundational Italian-specific models evaluated suggests room for significant enhancements in its training approach and data quality. Notably, Velvet performs better than other Italian foundational models, though it is of larger parameter size, making direct comparisons less straightforward.

This aligns with previous findings Tang et al. (2024) that a small subset of language-specific neurons and layers significantly influences the proficiency of LLMs in comprehending a particular language. Fan et al. (2025) finds that the lower layers of a model are often involved in learning language-specific representations, which are then transformed into universal representations in higher-level layers. Along with higher training budgets and substantial computational resources, these mechanisms enable multilingual models to process a broader range of languages effectively. This contributes to their superior performance over language-specific models, even if they are not as specialised.

Qualitative studyAnalysing the most frequently incorrectly answered questions by every model suggests the lack of specific linguistic knowledge and reasoning on language capabilities.

Most models consistently struggle to answer the above orthography question accurately, including variations of preparoxytone (a word with stress on the third last syllable) shown in Fig. 4. Models lack sufficient knowledge to handle this kind of very specific question effecti

Figure 4: Examples of frequently failed questions in ITAIC.

 

with performing language-based reasoning tasks. They also tend to falter in related tasks, such as identifying direct objects or time complements.

While analysing the most frequent wrong answers to cultural questions, we found that these focus on narrow geographical and cultural aspects. For example, the incorrect answers in the Art section were about specific details about lesser-known Italian art collections or technical elements like the architectural style of Greek temples in Italy or restoration techniques, as in Fig. 5.

Another example of the most frequent wrong answers in the context of geographical questions requires in-depth knowledge of Italian history and human geography. Physical geography and mountain morphology are also explored, along with cultural and productive aspects such as viticulture and traditional craftsmanship. Certain questions focus on Italian geographic areas and institutions, including ports, Apennine mountain ranges, international connections, and UNESCO heritage sites, exemplified in Fig. 6.

\begin{table}
\begin{tabular}{|p{142.3pt}|} \hline  & **Colutes and Commensense** \\ \hline \multicolumn{1}{c|}{_Large Capability_} & **Total** \\ \hline charco-3sut Accuracy & Art (980) & Civic (973) & Eve (92) & Odor (979) & Her (979) & Lite (984) & Tome (100) & Morr (140) & Orn (971) & Syno (971) & Syno (971) & Syno (971) \\ \hline charco-3sutness & 87.24 & **94.76** **973.83** & **95.09** & **90.03** & **98.99** & **96.58** & **77.46** & 86.92 & **96.81** & **87.77** & **92.38** \\ clanco-3suits & 76.63 & 62.53 & **74.8** & 75.0 & 65.9 & 8.03 & 78.67 & 90.41 & 67.16 & 87.44 & 72.25 & 81.44 \\ ggg+3024-0.06e & 87.51 & **89.31** & **97.82** & **94.97** & **94.91** & **89.89** & **96.80** & 80.00 & 84.32 & **96.91** & 85.92 & **91.36** \\ ggg+400-0.06e & 76.73 & 82.53 & 80.30 & 83.46 & 84.46 & 87.22 & 78.90 & 90.00 & 77.18 & 92.72 & **74.20** & 82.22 \\ germi-po-1.5 & 82.65 & 89.41 & 92.93 & 91.62 & 91.00 & 88.41 & 30.66 & 93.97 & 65.00 & 75.39 & 94.03 & 79.45 & 86.66 \\ germi-foah-1.5 & 76.33 & 82.77 & 89.36 & 96.62 & 87.34 & 81.71 & 74.55 & 98.81 & 62.44 & 75.93 & 92.77 & **74.11** & 81.66 \\ lianns-3.1-exp & 86.22 & 80.65 & **76.04** & 89.37 & 89.41 & 89.01 & 86.84 & 94.18 & 62.34 & 78.06 & 95.98 & 78.53 & 88.89 \\ lianns-3.1-prob & 81.12 & 84.55 & **83.48** & 88.56 & 90.90 & 80.08 & 82.54 & 90.93 & 61.43 & 69.22 & 95.28 & 79.14 & 83.61 \\ lianns-3.1-abs & 66.22 & 69.48 & 77.11 & 77.02 & 74.95 & 64.13 & 69.39 & 74.47 & 40.00 & **99.33** & 70.44 & **50.06** & 66.38 \\ national-large & 22.96 & 82.39 & **82.95** & **82.92** & **82.72** & 83.72 & 83.87 & 86.87 & 46.57 & 74.16 & **82.05** & 73.15 & 83.30 \\ rusti-semon & 66.33 & 69.27 & 62.61 & 76.00 & 74.34 & 68.39 & 67.76 & 79.37 & 52.86 & 55.64 & 74.25 & 54.78 & 65.53 \\ \hline lianns-3.1-norm-loss & 79.00 & 71.22 & 78.61 & 79.35 & 76.02 & 74.53 & 67.08 & 70.82 & 76.61 & 80.00 & 52.83 & 76.42 & 54.47 & 63.87 \\ lianns-3.1-log & 79.10 & 71.22 & 78.61 & 79.26 & 77.40 & 67.11 & 77.15 & 81.51 & 52.14 & 53.04 & 80.11 & 81.55 & 67.09 \\ lianns-data-data-0.4 & 67.35 & 70.20 & 80.43 & 76.61 & 72.39 & 70.93 & 60.08 & 78.45 & 42.92 & 54.17 & 70.34 & 55.40 & 63.30 \\ lvviet-lids & 67.86 & 72.56 & 73.00 & 76.81 & 77.10 & 69.21 & 71.43 & 77.32 & 42.68 & 40.08 & 70.00 & 45.69 & 67.04 \\ lianns-data-plus & 55.20 & 52.00 & 52.22 & 62.82 & 75.16 & 54.47 & 59.18 & 59.34 & **72.74** & **32.12** & **31.24** & 43.46 & **31.35** & **50.53** \\ lianns-7br & 48.33 & 44.50 & 52.21 & 49.13 & **47.24** & 40.38 & 51.33 & **45.45** & **20.36** & **20.25** & **38.31** & **22.13** & **43.58** \\ \hline Muckh AFB & 73.16 & 76.78 & 84.46 & 81.78 & 80.43 & 76.18 & 78.52 & 82.99 & **25.57** & 61.83 & 80.53 & 63.12 & 75.83 \\ \hline \multicolumn{1}{c|}{_Five-short Accuracy_} & Art (980) & Civic (973) & Eve (92) & Geo (979) & Heat (979) & Lua (979) & Murr (140) & Ochn (971) & Syu (971) & Syu (973) & Ulua (971) & Syu (973) & Ulua (974) \\ clanco-3suit & 89.18 & **86.20** & **76.74** & 96.02 & 96.61 & 95.93 & 98.94 & 98.06 & 81.43 & 85.87 & 97.22 & 97.20 \\ clanco-3suit & 78.37 & 85.30 & 88.38 & 85.17 & **74.35** & 85.77 & 81.33 & 92.82 & 84.22 & 70.32 & 92.28 & 72.25 & 83.42 \\ grg+3024-08-06e & 85.71 & **93.01** & **97.98** & **95.71** & **92.92** & **94.92** & 89.90 & 88.00 & 87.84 & 97.68 & 97.68 & 92.30 \\ grg+40-simi & 77.45 & 83.14 & 89.13 & 89.98 & 84.57 & 84.55 & 81.10 & 92.74 & 67.56 & 75.08 & 95.16 & 75.31 & 83.64 \\ grg+40-simi & 84.39 & 91.57 & 94.57 & 93.36 & 93.46 & 93.86 & 86.63 & 96.32 & 63.57 & 79.30 & 96.09 & 84.39 & 89.42 \\ grg+40-simi & 77.04 & 84.45 & 85.57 & 85.05 & 85.93 & 84.15 & 79.49 & 93.87 & 64.43 & 76.21 & 94.44 & 74.20 & 83.51 \\ flianns-3.1-exp & 87.14 & 81.47 & **96.94** & **94.97** & **94.97** & **94.97** & **80.96** & **96.94** & **96.50** & **96.76** & **97.11** & **96.00** & **97.55 

Performance gaps within categories.Models consistently perform better in the culture and commonsense section than in language capability, likely due to the linguistic similarity of these tasks with their training data. Like many morphologically rich languages, Italian has intricate rules governing word formation, syntax, and orthographic structures. These complexities are difficult for models to grasp. Even when challenged in cultural questions, difficulties remain in nuanced tasks such as tourism, where even the best-performing models show relative weaknesses, especially in zero-shot evaluations. In particular, poor performance on Morphology and Orthography scores in both few-shot and zero-shot settings indicates that even the best models struggle with the subtleties of the Italian language structure, especially when examples are not provided.

One key observation from the results is the consistent difference in performance between the few-shot and zero-shot settings. The difference in the average score between the two tests is 1.48 percentage points in favour of the few-shot scenario for the culture and commonsense section and 2.08 for the language capability section (1.76 overall). In few-shot learning, models are given several task examples before making predictions. This allows the model to understand the structure, vocabulary, and nuances specific to the task or domain better. Questions in the language capability section are intertwined with language-specific rules, which may not be immediately obvious in a zero-shot setting.

## 5 Conclusion and Future Work

Itatic represents a comprehensive and nuanced benchmark for evaluating Italian language understanding and cultural knowledge in AI models, providing diverse tasks across multiple domains and a robust tool for assessing and driving progress in Italian NLP.

Our evaluation of 17 state-of-the-art LLMs revealed that while models perform reasonably well in commonsense and culturally relevant tasks, they face significant challenges in mastering the linguistic intricacies of Italian, particularly in tasks involving morphology, syntax, and orthography. We observed a consistent improvement in few-shot settings compared to zero-shot, demonstrating that providing task-specific examples greatly enhances model performance.

Itatic highlights several areas for improvement in Italian language models, such as enhanced cultural training data to capture better Italian-specific knowledge and contexts and specialised fine-tuning for professional domains relevant to the Italian market. As the field continues to advance, Itatic will serve as a valuable resource for researchers and practitioners working on Italian language AI, helping to ensure that these technologies can effectively serve the needs of Italian-speaking users and applications.

### Future Work

We envision several directions for expanding and improving Itatic: (i) Periodic updates are needed to reflect evolving language use and cultural references in Italy, ensuring that benchmarks remain relevant for long-term evaluations. (ii) Integrating multimodal data (e.g., images, videos, audio) could challenge multimodal models. For example, combining visual art references or historical monuments with textual descriptions could improve cultural comprehension in models. (iii) Finally, future work could extend the framework used to build Itatic to other underrepresented languages with rich cultural and linguistic diversity. This would help address the broader challenge of multilingual language models' ability to handle culturally specific tasks across different global contexts.

_Resource Availability Statement._Itatic is accessible on Zenodo10 under the MIT license, with no IP-based or other restrictions. Additionally, the code used for the evaluation is available on GitHub11.

Footnote 10: [https://doi.org/10.5281/zenodo.14725822](https://doi.org/10.5281/zenodo.14725822)

Footnote 11: [https://github.com/Crisp-Unimib/ITALIC](https://github.com/Crisp-Unimib/ITALIC)

## 6 Limitations

Scope of evaluation.The focus of Itatic is exclusively on the Italian language and cultural context. While this is useful for evaluating models' performance in a specific language, the results may not generalise to other underrepresented languages or cultural settings.

Large-scale manual evaluation.Our dataset is based on a thoroughly validated source: standardised tests for public national assessments. Moreover, we conducted comprehensive human validation across all contexts to assess the dataset's validity. However, ideally, a human evaluation study would also require a large-scale assessment involving annotators from diverse cultural and back 

ground perspectives, ensuring an unbiased human effort.

## 7 Ethical Considerations

The dataset does not contain confidential information. It consists entirely of publicly available standardised test questions and does not include real data from individuals or non-public communications. Furthermore, all data and supplementary sources used in the collection process do not contain personally identifiable information or sensitive content. The dataset is also free from content that could be considered offensive, insulting, threatening, or distressing. Since it solely comprises data from standardised tests and does not involve human subjects or personal data, an ethical review process was not required. Potential risks of misuse include using the benchmark results to justify or argue against the need to develop native LLMs specifically tailored for the Italian language. This possibility should be considered to avoid misinterpretations or unintended consequences when leveraging the evaluation outcomes.

## References

* Achiam et al. (2013) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Alteschmmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt+4 technical report. _arXiv preprint arXiv:2303.08774_.
* Armengol-Estape et al. (2022) Jordi Armengol-Estape, Ona de Gibert Bonet, and Maite Melero. 2022. On the multilingual capabilities of very large-scale English language models. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 3056-3068, Marseille, France. European Language Resources Association.
* Basile et al. (2023) Valerio Basile, Livio Bioglio, Alessio Bosca, Cristina Bosco, and Viviana Patti. 2023. Uinauli: A unified benchmark for tialian natural language understanding. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_, pages 348-356.
* Chang et al. (2024) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. 2024. A survey on evaluation of large language models. _ACM Trans. Intell. Syst. Technol._, 15(3).
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhiri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Lettman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Eldar et al. (1997) Yuval Eldar, Michael Lindenbaum, Moshe Porat, and Yehoshua Y Zeevi. 1997. The farthest point strategy for progressive image sampling. _IEEE transactions on image processing_, 6(9):1305-1315.
* Fan et al. (2025) Yuchun Fan, Yongyu Mu, Yilin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, and Jingbo Zhu. 2025. SIAM: Towards efficient multilingual reasoning via selective language alignment. In _Proceedings of the 31st International Conference on Computational Linguistics_, pages 9499-9515.
* Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _International Conference on Learning Representations_.
* Hershcovich et al. (2022) Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Ihoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Iias Chalkidis, Ruixiang Cui, et al. 2022. Challenges and strategies in cross-cultural nlp. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 6997-7013.
* Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. _arXiv preprint arXiv:2401.04088_.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _arXiv preprint arXiv:2001.08361_.
* Kim et al. (2024) Eunsu Kim, Juyoung Suk, Philhoon Oh, Haneul Yoo, James Thorne, and Alice Oh. 2024. CLlcK: A benchmark dataset of cultural and linguistic intelligence in Korean. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LRECOLING 2024)_, pages 3335-3346, Torino, Italia. ELRA and ICCL.
* Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulizi-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. 2022. Quality at a glance: An audit of web-crawled multilingual datasets. _Transactions of the Association for Computational Linguistics_, 10:50-72.
* Lai et al. (2023) Mirko Lai, Stefano Menini, Marco Polignano, Valentina Russo, Rachele Sprugnoli, Giulia Venturi, et al. 2023. Evalita 2023: Overview of the 8th evaluation campaign of natural language processing and speech tools for italian. In _Proceedings of the Eighth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2023), CEUR. org, Parma, Italy_ 

Nicola Landro, Ignazio Gallo, Riccardo La Grassa, and Edoardo Federici. 2022. Two new datasets for italian-language abstractive text summarization. Information13(5):228.
* Le Scao et al. [2023] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, Francois Yvon, Matthias Galle, et al. 2023. Bloom: A 176b-parameter open-access multilingual language model. _Preprint_, arXiv:2405.07101.
* Li et al. [2022] Jingtao Li, Jian Zhou, Yan Xiong, Xing Chen, and Chaitali Chakrabarti. 2022. An adjustable farthest point sampling method for approximately-sorted point cloud data. In _2022 IEEE Workshop on Signal Processing Systems (SiPS)_, pages 1-6.
* Liang et al. [2023] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yuan Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2023. Holistic evaluation of language models. _Transactions on Machine Learning Research_. Featured Certification, Expert Centification.
* Mercorio et al. [2024] Fabio Mercorio, Mario Mezzanzancia, Daniele Poterti, Antonio Serino, and Andrea Sevesco. 2024. Disco aut deficere: Evaluating llm's proficiency on the invalsi tialian benchmark. _arXiv preprint arXiv:2406.17535_.
* Muennighoff et al. [2023] Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 2014-2037, Dubrovnik, Croatia. Association for Computational Linguistics.
* Myung et al. [2024] Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Puttri, Dimosthenis Antypas, Hsuvas Borkakovty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. 2024. BLEU: A benchmark for LLMs on everyday knowledge in diverse cultures and languages.
* Orlando et al. [2024] Riccardo Orlando, Luca Moroni, Pere-Lluis Huguet Cabot, Edoardo Barba, Simone Conia, Sergio Prandini, Giuseppe Faimeni, Roberto Navigli, et al. 2024. Minerva llms: The first family of large language models trained from scratch on italian data. In _Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)_.
* Polignano et al. [2024] Marco Polignano, Pierpaolo Basile, and Giovanni Semeraro. 2024. Advanced natural-based interaction for the italian language: LIlamantino-3-annita. _Preprint_, arXiv:2405.07101.
* Rao et al. [2024]Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2024. Normad: A benchmark for measuring the cultural adaptability of large language models. _arXiv preprint arXiv:2404.12464_.
* Reid et al. [2024] Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-banpistste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittweiser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _arXiv preprint arXiv:2403.05530_.
* Ruder et al. [2021] Sebastian Ruder, Noah Constant, Jan Botha, Aditya Sidhadant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, et al. 2021. Xtreme-r: Towards more challenging and nuanced multilingual evaluation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Ruffini et al. [2023] Renato Ruffini, Marta Ingaggiiati, et al. 2023. _Evoluzione dei concorsi pubblici in Italia: la valorieziazione delle competenze_, volume 33. Giuffle Francis Lefebvre.
* Srivastava et al. [2023] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abuwal Md Shoeb, Abbakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adria Garriga-Alonso, et al. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_.
* Talat et al. [2022] Zeerak Talat, Aurelie Neveol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Mariam Masoud, Margaret Mitchell, Dragomir Radev, et al. 2022. You reap what you sow: On the challenges of bias evaluation under multilingual settings. In _Proceedings of BigScience Episode# 5-Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 26-41.
* Tang et al. [2024] Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. _arXiv preprint arXiv:2402.16438_.
* Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. _arXiv preprint arXiv:2307.09288_.
* Yuan et al. [2024] Weizhe Yuan, Richard Yuanze Pang, Kyunghyun Cho, Sainbayar Sukhbbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. _arXiv preprint arXiv:2401.10020_.
* Zhong et al. [2024] Wanjun Zhong, Ruixiang Cui, Yaduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. AGIEval: A human-centric benchmark for evaluating foundation models. In _Findings of the Association for Computational Linguistics: NAACL 2024_, pages 2299-2314, Mexico City, Mexico. Association for Computational Linguistics.