Commonsense Reasoning in Arab Culture
Authors: Abdelrahman Boda Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, Fajri Koto
Accepted at: acl25

Commonsense Reasoning in Arab Culture

Abdelrahman Sadallah1  Junior Cedric Tonga1  Khalid Almubarak2,5
Saeed Almheiri1  Farah Atif1  Chatrine Qwaider1  Karima Kadaoui1
Sara Shatnawi3  Yaser Alesh4  Fajri Koto1
1Mohamed bin Zayed University of Artificial Intelligence
2SDAIA  3Al-Balqa Applied University
4Khalifa University  5HUMAIN, Data and AI Models
{abdelrahman.sadallah,fajri.koto}@mbzuai.ac.ae

Abstract

Despite progress in Arabic large language models, such as Jais and AceGPT, their evaluation on commonsense reasoning has largely relied on machine-translated datasets, which lack cultural depth and may introduce Anglocentric biases. Commonsense reasoning is shaped by geographical and cultural contexts, and existing English datasets fail to capture the diversity of the Arab world. To address this, we introduce ArabCulture, a commonsense reasoning dataset in Modern Standard Arabic (MSA), covering cultures of 13 countries across the Gulf, Levant, North Africa, and the Nile Valley. The dataset was built from scratch by engaging native speakers to write and validate culturally relevant questions for their respective countries. ArabCulture spans 12 daily life domains with 54 fine-grained subtopics, reflecting various aspects of social norms, traditions, and everyday experiences. Zero-shot evaluations show that open-weight language models with up to 32B parameters struggle to comprehend diverse Arab cultures, with performance varying across regions. These findings highlight the need for more culturally aware models and datasets tailored to the Arabic-speaking world.1

1 Introduction

Commonsense reasoning is the ability to make judgments and inferences based on everyday human knowledge and experiences (Sap et al., 2020). It is a fundamental aspect of human cognition and has been extensively studied in the context of large language models (LLMs) (OpenAI et al., 2024b; Grattafiori et al., 2024; Liu et al., 2023). However, commonsense reasoning is not universal—it is shaped by culture, which encompasses the shared knowledge, values, customs, and behaviors that define a society (Macionis, 2012; Giddens and Sutton, 2014).

1ArabCulture can be accessed at https://huggingface.co/datasets/MBZUAI/ArabCulture

<table>
  <tr>
    <th>Location: Algeria, North Africa</th>
    <th>الموقع: الجزائر، شمال أفريقيا</th>
  </tr>
  <tr>
    <td>Karim eats a dish of couscous for lunch<br>A. Karim eats couscous with kesra<br>B. Karim eats couscous with meat sauce<br>C. Karim eats couscous with vegetables</td>
    <td>كاريم يأكل طبق الكسكسي للغداء<br>أ. كريم يأكل الكسكسي مع كسرة<br>ب. كريم يأكل الكسكسي مع عسل اللحم<br>ج. كريم يأكل الكسكسي مع الخضروات</td>
  </tr>
  <tr>
    <th>Location: Egypt, Nile Valley</th>
    <th>الموقع: مصر، وادي النيل</th>
  </tr>
  <tr>
    <td>Asmaa prepares koshari for lunch<br>A. Hot sauce and tomato sauce are added to the koshari<br>B. Taro is added to the koshari<br>C. Peas are added to the koshari</td>
    <td>آسما تجهز الكشري للغداء<br>أ. يضاف الصلصة الحارة والصلصة الطماطم إلى الكشري<br>ب. يضاف التوابل إلى الكشري<br>ج. يضاف الفاصوليا إلى الكشري</td>
  </tr>
  <tr>
    <th>Location: Syria, the Levant</th>
    <th>الموقع: سوريا، الشام</th>
  </tr>
  <tr>
    <td>Mohammad eats fried kibbeh for lunch<br>A. The kibbeh is stuffed with a cheese filling<br>B. The kibbeh is stuffed with a filling of meat and onions<br>C. The kibbeh is stuffed with a variety of vegetables</td>
    <td>محمد يأكل الكبة المقلية للغداء<br>أ. الكبة ممتلئة بملل الجبن<br>ب. الكبة ممتلئة بملل اللحم والبصل<br>ج. الكبة ممتلئة بملل الخضروات المختلفة</td>
  </tr>
  <tr>
    <th>Location: KSA, the Gulf</th>
    <th>الموقع: السعودية، الخليج</th>
  </tr>
  <tr>
    <td>Jareesh is one of the meals served in winter<br>A. Jareesh is served with fried garlic<br>B. Jareesh is served with bread<br>C. Jareesh is served with fried onions</td>
    <td>جاريش هو أحد الأكلات التي تقدم في فصل الشتاء<br>أ. جاريش يقدَّم مع البصل المقلي<br>ب. جاريش يقدَّم مع الخبز<br>ج. جاريش يقدَّم مع البصل المقلي</td>
  </tr>
</table>

Figure 1: ArabCulture covers four regions across the Middle East and North Africa, spanning 13 countries. The highlighted areas on the map represent the regions included in ArabCulture. We present example questions for the Lunch category from Algeria, Egypt, Syria, and KSA, each with three answer choices, with the correct answer marked in bold. English translations are provided for illustration.

The Arab world, home to approximately 456 million people (Diab et al., 2017; Shoufan and Alameri, 2015), is characterized by its linguistic unity through Modern Standard Arabic (MSA) while encompassing diverse traditions, religions, and customs (Mirkin, 2010). This cultural diver-
sity influences not only social interactions but also reasoning patterns, making it crucial to develop evaluation benchmarks that reflect these variations. However, most existing commonsense reasoning datasets are developed with Western-centric assumptions, limiting their applicability to Arabic-speaking societies.

Despite recent advancements in Arabic LLMs (Sengupta et al., 2023; Huang et al., 2024; Team, 2024; Bari et al., 2024b), their evaluation has largely relied on machine-translated datasets originally created in English. Many commonsense reasoning benchmarks (Bisk et al., 2019; Sap et al., 2019a) fail to capture Arab cultural perspectives, as simple translations do not account for region-specific knowledge, potentially introducing bias. Given the significant cultural variations across the Arab world, these datasets do not provide a holistic measure of Arabic LLMs’ ability to reason within culturally specific contexts. This raises an important question: To what extent can existing LLMs accurately reason about commonsense knowledge in diverse cultural settings, particularly in the Arab world?

To address this gap, we introduce ArabCulture, a commonsense reasoning dataset specifically designed to assess the cultural knowledge of Arabic LLMs. The dataset consists of 3,482 questions written in Modern Standard Arabic (MSA), covering 13 countries across the Gulf, Levant, North Africa, and the Nile Valley (see Figure 1). It spans 12 major domains and 54 fine-grained subtopics, reflecting various aspects of social norms, traditions, and daily life in the Arab world. Unlike existing benchmarks, which often rely on translated datasets, ArabCulture was built from scratch by directly engaging native speakers to write culturally relevant questions for their respective countries. We carefully implemented quality control measures throughout the dataset creation process, including rigorous validation steps to maintain accuracy, relevance, and cultural sensitivity.

We evaluate a range of closed-weight and open-weight Arabic and multilingual LLMs in a zero-shot setting to assess their cultural commonsense reasoning capabilities. Inspired by Koto et al. (2024b), we frame the task in two ways: multiple-choice questions (MCQ) and completion tasks. Additionally, we introduce three levels of location-based contextual grounding: (1) no additional location information, (2) specifying the broader region (e.g., Gulf or Levant), and (3) specifying the exact country along with its regional classification. This setup allows us to analyze how effectively LLMs incorporate geographical and cultural cues in their reasoning.

Our results show that even LLMs with up to 32B parameters struggle with cultural commonsense reasoning, with performance varying significantly across regions. We conduct a detailed analysis of the best-performing models, identifying strengths and weaknesses across different cultural contexts. We also conducted a small manual experiment to test the models’ ability to explain their chosen answers. Additionally, we explore whether enriching prompts with cultural facts improves performance in smaller language models, finding that while it helps in some cases, it does not provide a universal solution.

2 Related Work

2.1 Commonsense Reasoning in English

Early research on commonsense reasoning primarily focused on linguistic reasoning, as seen in the Winograd Schema Challenge (Levesque et al., 2012) and Winogrande (Sakaguchi et al., 2021), which evaluate pronoun coreference resolution within social and linguistic contexts. Other works have explored physical commonsense reasoning, assessing model’s understanding of real-world properties and relationships (Bisk et al., 2019), as well as social reasoning, where models are tested on their ability to interpret human emotions, actions, and social norms (Sap et al., 2019b). Research has also expanded into numerical (Lin et al., 2020; Akhtar et al., 2023), temporal (Tan et al., 2023), and causal reasoning (Roemmele et al., 2011; Du et al., 2022), broadening the scope of commonsense evaluation. However, these benchmarks are all developed in English and shaped by Western cultural assumptions, limiting their applicability to the Arab world.

2.2 Arabic Large Language Models and Their Evaluation on Commonsense Reasoning

A limited number of Arabic language models have been developed with more than 7B parameters, all of which are decoder-only architectures. These include JAIS (Sengupta et al., 2023), Fanar (Fanar-Team, 2024), AceGPT (Huang et al., 2024), and ALLAM (Bari et al., 2024b). Their evaluation of commonsense reasoning has been primarily based on machine-translated datasets from English to
<table>
  <tr>
    <th>Dataset</th>
    <th>Size</th>
    <th>Data Construction Method</th>
    <th>Cultural?</th>
    <th>Location?</th>
    <th>#Topic</th>
    <th>#Country</th>
    <th>Reasoning?</th>
  </tr>
  <tr>
    <td>ArabCulture (Ours)</td>
    <td>3,482</td>
    <td>Manually built, validated by native</td>
    <td>✓</td>
    <td>✓</td>
    <td>54</td>
    <td>13</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>AraDiCE-Culture (Mousi et al., 2025)</td>
    <td>180</td>
    <td>Manually built, validated by native</td>
    <td>✓</td>
    <td>✓</td>
    <td>9</td>
    <td>1</td>
    <td>–</td>
  </tr>
  <tr>
    <td>AraDiCE-WinoGrande (Mousi et al., 2025)</td>
    <td>1,267</td>
    <td>Machine-translated, post-edited</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>AraDiCE-PIQA (Mousi et al., 2025)</td>
    <td>1,838</td>
    <td>Machine-translated, post-edited</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>AraDiCE-OpenBookQA (Mousi et al., 2025)</td>
    <td>500</td>
    <td>Machine-translated, post-edited</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>AlGhafa (COPA Ar) (Almazrouei et al., 2023)</td>
    <td>89</td>
    <td>Machine-translated, verified by humans</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>–</td>
    <td>✓</td>
  </tr>
  <tr>
    <td>ACVA (?)</td>
    <td>2,486</td>
    <td>ChatGPT generated, verified by humans</td>
    <td>✓</td>
    <td>–</td>
    <td>50</td>
    <td>–</td>
    <td>–</td>
  </tr>
</table>

Table 1: Comparison of our dataset with other Arabic cultural commonsense reasoning datasets. The metadata includes Size (number of Arabic instances), Cultural? (whether the data considers cultural nuances), Location? (whether the data includes fine-grained location information, such as regions and countries per region), #Topic (number of fine-grained topics covered), #Country (total number of countries across all regions) and Reasoning? (whether the data emphasizes commonsense reasoning or not).

Arabic (e.g., Tawalbeh and Al-Smadi (2020); Al-Bashabsheh et al. (2021)). Although this approach provides a useful reference, it does not offer a comprehensive assessment of how well these models capture culturally grounded commonsense knowledge.

Much of the recent Arabic-centric benchmarks have focused on classic NLP tasks, including syntax, semantics, and question answering, as seen in LaraBench (Abdelali et al., 2024), natural language generation in DOLPHIN (Elmadany et al., 2023a), and natural language understanding in ORCA (Elmadany et al., 2023b). Only a few studies have shifted their focus toward knowledge-intensive tasks and reasoning abilities. Among them, ArabicMMU (Koto et al., 2024a) compiles exam questions from different education levels across Arabic-speaking countries, offering a broad knowledge assessment but placing less emphasis on cultural reasoning.

Table 1 compares ArabCulture with related Arabic datasets. Most existing datasets are derived from machine translation with post-editing, prioritizing linguistic accuracy over cultural relevance. While some assess reasoning, they often lack cultural grounding, location metadata, and fine-grained topic categorization. ACVA (Huang et al., 2024), generated using ChatGPT (Ouyang et al., 2022), is not designed for reasoning evaluation. Similarly, AraDiCE-Culture (Mousi et al., 2025) consists of only 180 samples and focuses on open-ended cultural knowledge rather than structured reasoning tasks. These gaps highlight the need for larger, more diverse benchmarks that better capture Arabic cultural contexts and reasoning abilities.

3 ArabCulture Dataset

ArabCulture is a sentence completion task in MSA, comprising 3,482 unique instances. Each question consists of a one-sentence premise and three answer choices that are both logically and syntactically valid. As illustrated in Figure 1, instances are drawn from various Arab regions.

Solving these questions requires cultural knowledge specific to the country referenced, as the correct answer aligns with culturally relevant context. This makes ArabCulture a valuable benchmark for assessing an LLM’s ability to incorporate cultural understanding and knowledge in Arabic-language tasks.

3.1 Dataset Construction

ArabCulture is built from scratch without relying on web-scraped text, minimizing the risk of training data leakage when evaluating LLMs. It is manually created and validated by native speakers from 13 Arabic-speaking countries. To further ensure quality, the authors conduct rigorous manual checks for lexical accuracy, semantic coherence, and contextual relevance.2

Worker requirements We hired 26 expert workers from 13 Arab countries, with two workers per country, that fit defined eligibility criteria: (1) The worker must be a native Arabic speaker; (2) They must have lived in the country for at least 10 years; (3) They must possess a strong understanding of local culture and traditions; (4) Their parents must also be from the country and reside there; (5) They must have at least a high school diploma, while higher degrees were considered an advantage.

2The authors of this paper represent most of the studied countries, contributing diverse regional perspectives to the dataset.
Among the 26 workers, 14 hold a Bachelor’s degree, including seven with a Master’s degree, two with a PhD, and three with a high school diploma.

All workers were required to attend a one-hour online workshop or watch a recorded session. The workshop introduced the project concept, explained task guidelines, and addressed any potential questions. To ensure a clear understanding of the task, we conducted a pilot study before the main annotation phase.

Each worker was assigned two tasks: (1) Writing instances and (2) Reviewing and verifying the work of their peer from the same country. The payment was determined based on the minimum monthly salary for data entry jobs in each worker’s country, and each worker was compensated for the equivalent of four full-time working days.

Country Selection We selected countries that ensured broad geographic coverage of the Arab world while remaining within budget constraints. Priority was given to countries with larger populations and land areas, resulting in a selection of 13 countries across four regions, representing approximately 82% of the total Arab world population. These include: (1) The Gulf: Saudi Arabia, Yemen, UAE; (2) The Levant: Syria, Jordan, Palestine, Lebanon; (3) North Africa: Morocco, Algeria, Tunisia, Libya; and (4) The Nile Valley: Egypt and Sudan.

Topic taxonomy We define 12 daily life topics with 54 fine-grained subtopics to build ArabCulture. The topic selection is based on Koto et al. (2024b) and adapted to reflect Arab regional culture. Native Arabic speakers from nine countries contributed to determining these topics, ensuring cultural relevance, diversity, and regional representation (e.g., Ramadan traditions). Additionally, we carefully balanced the dataset by assigning an appropriate number of examples to each topic. Table 8 in the Appendix provides an overview of the topics and their subtopics, which include food, weddings, holiday activities, daily activities, habits, traditional games, death, art, parenting, agriculture, family relationships, and idioms.

Instance Writing In the first stage, each worker was tasked with writing short two-sentence stories. For each entry, they were provided with a pre-defined topic and instructed to write a one-sentence premise followed by three candidate completions for the second sentence. The completions had to adhere to the following rules: (1) all had to be valid syntactic continuations of the premise, (2) none could introduce logical contradictions (e.g., ensuring consistency in topic and narrative), and (3) only one of the three sentences should be culturally accurate for the specified country. This design ensures that model predictions are influenced by cultural knowledge rather than grammatical or logical inconsistencies. Each worker was required to write 150 instances with two workers assigned per country.

Two-stage of Quality Control In stage 1, each of the 13 countries had a designated representative involved in dataset development, all of whom are also authors of this project. After workers completed their assigned instances, the respective country representatives manually reviewed their submissions to ensure adherence to the guidelines. Linguistic errors were corrected through manual editing, but if an instance did not meet the guidelines, workers were required to revise and resubmit it.

To further ensure quality, stage 2 involved a peer validation process, where each worker reviewed their colleague’s work. The data was reformatted into multiple-choice questions, with the second sentence of each instance shuffled among three options. Workers were then asked to select the correct culturally appropriate completion and were allowed to consult external sources if unsure. If the worker selected the correct answer, it indicated agreement between annotators on the cultural validity of the instance. However, if the worker selected the wrong answer, the example was discarded, as it suggested ambiguity or cultural disagreement in the instance.

Country-Specific Annotation Beyond quality control, we also tasked the quality control workers with annotating whether the cultural context described in an instance could be relevant to other countries. The goal was to distinguish instances that are truly unique to the designated country from those that are shared across multiple countries. To ensure accuracy, the authors of this paper conducted a second round of annotation. If an instance was marked as culturally relevant to more than one country by at least one annotator, we flagged it as Not Country-Specific (~CS); otherwise, it was labeled as Country-Specific (CS). This categorization was used in our analysis experiments to better understand the distribution of culturally unique and widely shared knowledge.
<table>
  <tr>
    <th>Region</th>
    <th>#data</th>
    <th>CS (%)</th>
    <th>\( \mu \) (words)</th>
    <th>\( \mu \) (chars)</th>
  </tr>
  <tr>
    <td>Gulf</td>
    <td>817</td>
    <td>49.1</td>
    <td>34.6</td>
    <td>188</td>
  </tr>
  <tr>
    <td>KSA</td>
    <td>261</td>
    <td>36.4</td>
    <td>33.2</td>
    <td>185</td>
  </tr>
  <tr>
    <td>UAE</td>
    <td>283</td>
    <td>35.3</td>
    <td>38.1</td>
    <td>205</td>
  </tr>
  <tr>
    <td>Yemen</td>
    <td>273</td>
    <td>75.5</td>
    <td>32.4</td>
    <td>172</td>
  </tr>
  <tr>
    <td>Levant</td>
    <td>1,097</td>
    <td>16.9</td>
    <td>30.2</td>
    <td>170</td>
  </tr>
  <tr>
    <td>Lebanon</td>
    <td>255</td>
    <td>38.8</td>
    <td>29.5</td>
    <td>167</td>
  </tr>
  <tr>
    <td>Syria</td>
    <td>279</td>
    <td>16.5</td>
    <td>27.7</td>
    <td>143</td>
  </tr>
  <tr>
    <td>Palestine</td>
    <td>273</td>
    <td>8.4</td>
    <td>31.7</td>
    <td>177</td>
  </tr>
  <tr>
    <td>Jordan</td>
    <td>290</td>
    <td>5.9</td>
    <td>31.6</td>
    <td>190</td>
  </tr>
  <tr>
    <td>North Africa</td>
    <td>1,047</td>
    <td>32.4</td>
    <td>32.4</td>
    <td>179</td>
  </tr>
  <tr>
    <td>Tunisia</td>
    <td>261</td>
    <td>31.8</td>
    <td>28.1</td>
    <td>154</td>
  </tr>
  <tr>
    <td>Algeria</td>
    <td>271</td>
    <td>27.3</td>
    <td>32.5</td>
    <td>180</td>
  </tr>
  <tr>
    <td>Morocco</td>
    <td>276</td>
    <td>37.3</td>
    <td>28.6</td>
    <td>161</td>
  </tr>
  <tr>
    <td>Libya</td>
    <td>239</td>
    <td>33.1</td>
    <td>41.4</td>
    <td>226</td>
  </tr>
  <tr>
    <td>Nile Valley</td>
    <td>521</td>
    <td>65.5</td>
    <td>34.3</td>
    <td>195</td>
  </tr>
  <tr>
    <td>Egypt</td>
    <td>265</td>
    <td>74.3</td>
    <td>32.4</td>
    <td>178</td>
  </tr>
  <tr>
    <td>Sudan</td>
    <td>256</td>
    <td>56.2</td>
    <td>36.3</td>
    <td>213</td>
  </tr>
  <tr>
    <th>All</th>
    <th>3,482</th>
    <th>46.0</th>
    <th>32.5</th>
    <th>181</th>
  </tr>
</table>

Table 2: Overall statistics of ArabCulture. CS samples represent the percentage of country-specific instances for each location. The last two columns include the average number of words and characters.

3.2 Data Statistics

During the instance writing phase, we initially aimed to collect 3,900 samples (26 workers × 150 samples each). However, the first quality-check round (§3.1) resulted in 3,606 samples. Following the second quality control, we discarded 124 samples, leaving a final dataset of 3,482 instances.

Table 2 shows the distribution of ArabCulture across regions and their respective countries. The overall proportion of country-specific instances is 46%, indicating notable cultural similarities among Arab countries. In terms of word and character count, the dataset shows consistent length across countries, with an average of 32.5 words and 181 characters per instance. However, Libyan examples tend to be longer, averaging 226 characters per instance. Furthermore, Figure 2 illustrates the total number of samples for each topic. Overall, the dataset covers a wide range of topics, with food, daily activities, and holiday activities being the most frequent, while parenting, family relationships, and agriculture are the least frequent.

![Bar chart showing the total number of samples for each topic](page_1012_186_410_312.png)

Figure 2: Total number of samples for each topic.

20 multilingual models of various sizes, such as BLOOMZ (Muennighoff et al., 2023), mTQ-xxl (Muennighoff et al., 2023), Llama–3 (Grattafiori et al., 2024), Aya-Expanse (Dang et al., 2024), Gemma-2 (Riviere et al., 2024), and Qwen2.5 (Qwen et al., 2025); (2) 10 Arabic-centric models of different sizes, including Jais (Sengupta et al., 2023), SILMA (Team, 2024), AceGPT-v2 (Huang et al., 2024), and ALLaM (Bari et al., 2024a); (3) 3 reasoning models that are distilled from the DeepSeek-R1 model (DeepSeek-AI et al., 2025). (4) 1 closed-weight model, GPT-4o (OpenAI et al., 2024a). All experiments are done using zero temperature (greedy sampling) to enforce the models to produce factual outputs.

We conducted experiments using both Arabic and English prompts (Figure 5) and evaluated language models using two strategies: (1) sentence completion and (2) MCQ. In the sentence completion approach, we concatenate the premise with each candidate’s second sentence and select the one with the highest likelihood. For MCQ, we assign alphabetical labels to the answer choices (A, B, C for English, and ١, ٢, ٣ for Arabic), and the selected answer corresponds to the option with the highest probability. We constructed these experiments using the LM-Evaluation-Harness Framework (Gao et al., 2024). Note that for the closed-weight model, we only perform MCQ-style evaluation, instructing the model to generate the answer as a JSON object containing only the answer character.

As discussed in Section 1, cultural knowledge varies across locations, and we hypothesize that providing geographical context can enhance a model’s reasoning ability in cultural contexts. To test this, we complement our experiments with
<table>
  <tr>
    <th rowspan="2">Model (#parameter)</th>
    <th colspan="3">Completion</th>
    <th colspan="3">MCQ</th>
  </tr>
  <tr>
    <th>\( \ell = \text{None} \)</th>
    <th>\( \ell = R \)</th>
    <th>\( \ell = R + C \)</th>
    <th>\( \ell = \text{None} \)</th>
    <th>\( \ell = R \)</th>
    <th>\( \ell = R + C \)</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>--</td>
    <td>--</td>
    <td>100.0</td>
    <td>--</td>
    <td>--</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>BLOOMZ (7B)</td>
    <td>31.6</td>
    <td>31.4</td>
    <td>31.7</td>
    <td>57.9</td>
    <td>57.8</td>
    <td>58.5</td>
  </tr>
  <tr>
    <td>mT0xL (14B)</td>
    <td>27.3</td>
    <td>27.6</td>
    <td>27.4</td>
    <td>65.9</td>
    <td>66.3</td>
    <td>67.0</td>
  </tr>
  <tr>
    <td>Llama-3.1 (8B)</td>
    <td>29.7</td>
    <td>29.7</td>
    <td>29.6</td>
    <td>35.0</td>
    <td>34.7</td>
    <td>34.9</td>
  </tr>
  <tr>
    <td>Llama-3.1 Instruct (8B)</td>
    <td>30.9</td>
    <td>31.0</td>
    <td>31.2</td>
    <td>47.5</td>
    <td>47.0</td>
    <td>49.1</td>
  </tr>
  <tr>
    <td>Llama-3 Instruct (70B)</td>
    <td>39.1</td>
    <td>39.5</td>
    <td>39.4</td>
    <td>34.3</td>
    <td>34.3</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>Llama-3.3 Instruct (70B)</td>
    <td>39.9</td>
    <td><b>40.6</b></td>
    <td><b>41.1</b></td>
    <td>75.4</td>
    <td>74.0</td>
    <td>71.2</td>
  </tr>
  <tr>
    <td>Aya-Expanse (8B)</td>
    <td>33.7</td>
    <td>37.2</td>
    <td>38.2</td>
    <td>39.6</td>
    <td>40.7</td>
    <td>41.8</td>
  </tr>
  <tr>
    <td>Aya-Expanse (32B)</td>
    <td>37.9</td>
    <td>37.9</td>
    <td>39.5</td>
    <td>52.6</td>
    <td>49.5</td>
    <td>49.5</td>
  </tr>
  <tr>
    <td>Gemma-2 (9B)</td>
    <td>31.8</td>
    <td>31.7</td>
    <td>31.8</td>
    <td>35.2</td>
    <td>34.5</td>
    <td>34.5</td>
  </tr>
  <tr>
    <td>Gemma-2 Instruct (9B)</td>
    <td>33.5</td>
    <td>33.8</td>
    <td>33.9</td>
    <td>58.7</td>
    <td>55.3</td>
    <td>57.0</td>
  </tr>
  <tr>
    <td>Gemma-2 (27B)</td>
    <td>32.6</td>
    <td>33.0</td>
    <td>33.2</td>
    <td>34.3</td>
    <td>34.3</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>Gemma-2 Instruct (27B)</td>
    <td>38.0</td>
    <td>38.9</td>
    <td>39.8</td>
    <td>61.6</td>
    <td>64.7</td>
    <td>64.2</td>
  </tr>
  <tr>
    <td>Qwen2.5 (7B)</td>
    <td>29.0</td>
    <td>31.5</td>
    <td>31.8</td>
    <td>52.1</td>
    <td>48.1</td>
    <td>49.0</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (7B)</td>
    <td>33.2</td>
    <td>33.5</td>
    <td>33.6</td>
    <td>53.1</td>
    <td>47.9</td>
    <td>48.8</td>
  </tr>
  <tr>
    <td>Qwen2.5 (14B)</td>
    <td>33.6</td>
    <td>34.5</td>
    <td>35.4</td>
    <td>55.2</td>
    <td>62.5</td>
    <td>61.6</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (14B)</td>
    <td>36.5</td>
    <td>35.9</td>
    <td>37.0</td>
    <td>67.7</td>
    <td>67.9</td>
    <td>69.3</td>
  </tr>
  <tr>
    <td>Qwen2.5 (32B)</td>
    <td>34.9</td>
    <td>35.6</td>
    <td>35.9</td>
    <td>51.6</td>
    <td>56.6</td>
    <td>53.3</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (32B)</td>
    <td>37.6</td>
    <td>37.3</td>
    <td>38.6</td>
    <td>75.2</td>
    <td>75.8</td>
    <td>76.5</td>
  </tr>
  <tr>
    <td>Qwen2.5 (72B)</td>
    <td>35.5</td>
    <td>36.7</td>
    <td>37.4</td>
    <td>56.1</td>
    <td>51.6</td>
    <td>51.8</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (72B)</td>
    <td><b>40.1</b></td>
    <td>40.2</td>
    <td>40.3</td>
    <td><b>80.1</b></td>
    <td><b>79.8</b></td>
    <td><b>80.0</b></td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill-Llama (70B)</td>
    <td>37.4</td>
    <td>37.6</td>
    <td>38.4</td>
    <td>34.3</td>
    <td>35.2</td>
    <td>34.5</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill-Qwen (32B)</td>
    <td>34.8</td>
    <td>34.7</td>
    <td>35.0</td>
    <td>34.3</td>
    <td>34.3</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>QwQ (32B)</td>
    <td>34.4</td>
    <td>35.7</td>
    <td>36.4</td>
    <td>36.7</td>
    <td>35.1</td>
    <td>35.6</td>
  </tr>
  <tr>
    <td>Jais (13B)</td>
    <td>39.3</td>
    <td>39.0</td>
    <td>39.3</td>
    <td>34.1</td>
    <td>34.9</td>
    <td>34.8</td>
  </tr>
  <tr>
    <td>Jais Chat (13B)</td>
    <td>40.8</td>
    <td>40.8</td>
    <td>41.9</td>
    <td>58.3</td>
    <td>54.1</td>
    <td>54.4</td>
  </tr>
  <tr>
    <td>Jais-v3 (30B)</td>
    <td>39.4</td>
    <td>38.4</td>
    <td>39.1</td>
    <td>34.3</td>
    <td>34.3</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>Jais-v3 Chat (30B)</td>
    <td>33.3</td>
    <td>33.6</td>
    <td>33.4</td>
    <td>60.1</td>
    <td>56.2</td>
    <td>54.0</td>
  </tr>
  <tr>
    <td>SILMA Instruct (9B)</td>
    <td>32.7</td>
    <td>33.0</td>
    <td>33.2</td>
    <td>71.5</td>
    <td>71.0</td>
    <td>72.0</td>
  </tr>
  <tr>
    <td>AceGPT-v2 (8B)</td>
    <td>32.4</td>
    <td>34.0</td>
    <td>34.6</td>
    <td>35.1</td>
    <td>35.0</td>
    <td>35.1</td>
  </tr>
  <tr>
    <td>AceGPT-v2 Chat (8B)</td>
    <td>36.0</td>
    <td>36.4</td>
    <td>37.3</td>
    <td>43.1</td>
    <td>39.7</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>AceGPT-v2 Chat (32B)</td>
    <td>38.5</td>
    <td>39.2</td>
    <td>40.0</td>
    <td><b>79.7</b></td>
    <td><b>79.1</b></td>
    <td><b>79.6</b></td>
  </tr>
  <tr>
    <td>AceGPT-v2 Chat (70B)</td>
    <td><b>43.2</b></td>
    <td><b>44.3</b></td>
    <td><b>44.5</b></td>
    <td>61.9</td>
    <td>61.7</td>
    <td>62.4</td>
  </tr>
  <tr>
    <td>LLaM-Instruct-preview (7B)</td>
    <td>37.7</td>
    <td>38.4</td>
    <td>39.2</td>
    <td>67.4</td>
    <td>72.0</td>
    <td>72.6</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>--</td>
    <td>--</td>
    <td>--</td>
    <td><b>88.5</b></td>
    <td><b>89.6</b></td>
    <td><b>90.0</b></td>
  </tr>
</table>

Table 3: Zero-shot accuracy results for the English prompt across various models and settings. "MCQ" refers to the multiple-choice question evaluation method, and \( \ell \) represents the inclusion of location context ("R" indicates the region, and "C" denotes the corresponding country). Bolded numbers highlight the highest score within each model group

three different levels of location context \( \ell \in \{\text{none}, \text{region}, \text{region + country}\} \).

4.2 Zero-shot Experiments

Our observations show that evaluation with English prompts outperforms Arabic prompts, consistent with findings from Koto et al. (2024a); Kmainasi et al. (2024). We speculate that this is due to the dominance of English instruction-tuning datasets in LLM development. Therefore, we present the results using English prompts in the main text and include the Arabic results in Appendix D.

Overall Observation The overall results presented in Table 3 reveal notable performance differences between open-weight and closed-weight models in understanding Arabic culture and norms. While some large-scale open-weight models achieve relatively high accuracy—such as Qwen-2.5-Instruct (72B) with 80%, LLaMA-3.3-Instruct (70B) with 75.4%, and AceGPT-v2-Chat (32B) with 79.7%—, closed-weight models represented by GPT-4o demonstrate significantly stronger performance. Arabic-centric models do not consistently outperform multilingual models. Some, such as Jais, struggle despite being tailored for Arabic, whereas certain multilingual models like Qwen2.5 and Llama-3.3 Instruct surpass them in accuracy. Within the same model family, performance generally improves with larger
model sizes—except for Jais and AceGPT. However, across different model families, scaling does not guarantee better results. This indicates that factors beyond model size, such as pretraining data, architecture, and training recipes, significantly impact cultural comprehension. When comparing base models to instruction-tuned variants, we observe modest improvements in completion tasks but substantial gains in MCQ tasks.

While reasoning models have demonstrated impressive capabilities in domains such as mathematics and programming, they perform poorly on our cultural reasoning tasks. This discrepancy suggests that cultural reasoning involves fundamentally different challenges that remain underexplored and require dedicated attention.

Overall, these findings highlight the need for improvements in Arabic and multilingual models to enhance their comprehension of Arabic cultural contexts. Addressing this gap can also help mitigate cultural biases, which have been identified in recent studies, such as the work by Naous et al. (2024).

Multiple-Choice (MCQ) Outperforms Completion In Table 3, we observe that sentence completion is not as reliable as MCQ, despite being a more natural approach that aligns with the sentence completion framework of ArabCulture. Qwen-2.5 Instruct (32B), for example, achieves 75.2% accuracy in MCQ but drops significantly to 37.6% in sentence completion. Similar disparities are also evident in smaller models; for instance, BLOOMZ (7B) achieves 58.5% in MCQ but performs at random (31.7%) in sentence completion. Some models, such as Aya-Expanse (8B), show only a small gap between MCQ and sentence completion, likely due to pretraining or fine-tuning gaps that hinder their ability to leverage structured prompts effectively. Interestingly, the older version of Llama-3 Instruct (70B) does not benefit from the MCQ strategy, whereas the latest version (Llama-3.3 Instruct) shows a dramatic improvement, increasing from 41% to 71%. Given that MCQ yields the best results, we use it for further analysis.

Impact of Location Granularity Adding finer-grained location information in the prompt does not produce a consistently positive or negative effect on zero-shot performance, yielding mixed results. For instance, when region and country context are included, Jais-v3 Chat (30B) experiences a 6-point accuracy drop compared to the vanilla prompt (\( \ell = \text{None} \)). In contrast, Qwen-2.5 (14B) shows a substantial improvement, increasing from 55.2% (\( \ell = \text{None} \)) to 61.6% when provided with country-level context. These fluctuations suggest that in some models, location specificity does not necessarily enhance cultural understanding.

<table>
  <tr>
    <th rowspan="2">Location</th>
    <th colspan="2">GPT-4o</th>
    <th colspan="2">Qwen-2.5</th>
    <th colspan="2">AceGPT-v2</th>
  </tr>
  <tr>
    <th>CS</th>
    <th>−CS</th>
    <th>CS</th>
    <th>−CS</th>
    <th>CS</th>
    <th>−CS</th>
  </tr>
  <tr>
    <td>Gulf</td>
    <td>87.8</td>
    <td>91.6</td>
    <td>72.3</td>
    <td>80.0</td>
    <td>73.1</td>
    <td>82.0</td>
  </tr>
  <tr>
    <td>KSA</td>
    <td>88.4</td>
    <td>91.0</td>
    <td>82.1</td>
    <td>78.9</td>
    <td>77.9</td>
    <td>81.3</td>
  </tr>
  <tr>
    <td>UAE</td>
    <td>93.0</td>
    <td>93.4</td>
    <td>75.0</td>
    <td>83.6</td>
    <td>76.0</td>
    <td>84.2</td>
  </tr>
  <tr>
    <td>Yemen</td>
    <td>85.0</td>
    <td>88.1</td>
    <td>66.5</td>
    <td>73.1</td>
    <td>69.4</td>
    <td>77.6</td>
  </tr>
  <tr>
    <td>Levant</td>
    <td>83.8</td>
    <td>93.2</td>
    <td>69.9</td>
    <td>86.7</td>
    <td>69.4</td>
    <td>85.6</td>
  </tr>
  <tr>
    <td>Lebanon</td>
    <td>78.8</td>
    <td>82.1</td>
    <td>61.6</td>
    <td>68.6</td>
    <td>63.6</td>
    <td>66.7</td>
  </tr>
  <tr>
    <td>Syria</td>
    <td>85.0</td>
    <td>93.7</td>
    <td>75.0</td>
    <td>85.4</td>
    <td>67.5</td>
    <td>83.3</td>
  </tr>
  <tr>
    <td>Palestine</td>
    <td>95.7</td>
    <td>94.4</td>
    <td>87.0</td>
    <td>89.2</td>
    <td>87.0</td>
    <td>90.0</td>
  </tr>
  <tr>
    <td>Jordan</td>
    <td>100.0</td>
    <td>97.8</td>
    <td>90.9</td>
    <td>95.7</td>
    <td>90.9</td>
    <td>94.3</td>
  </tr>
  <tr>
    <td>Nile Valley</td>
    <td>87.1</td>
    <td>97.8</td>
    <td>76.8</td>
    <td>93.3</td>
    <td>74.8</td>
    <td>88.9</td>
  </tr>
  <tr>
    <td>Egypt</td>
    <td>87.3</td>
    <td>97.1</td>
    <td>78.2</td>
    <td>91.2</td>
    <td>73.1</td>
    <td>83.8</td>
  </tr>
  <tr>
    <td>Sudan</td>
    <td>86.8</td>
    <td>98.2</td>
    <td>75.0</td>
    <td>94.6</td>
    <td>77.1</td>
    <td>92.0</td>
  </tr>
  <tr>
    <td>North Africa</td>
    <td>86.1</td>
    <td>86.9</td>
    <td>70.8</td>
    <td>81.1</td>
    <td>73.7</td>
    <td>79.9</td>
  </tr>
  <tr>
    <td>Tunisia</td>
    <td>75.9</td>
    <td>80.9</td>
    <td>61.4</td>
    <td>69.7</td>
    <td>62.7</td>
    <td>72.5</td>
  </tr>
  <tr>
    <td>Algeria</td>
    <td>85.1</td>
    <td>83.8</td>
    <td>63.5</td>
    <td>77.2</td>
    <td>68.9</td>
    <td>72.6</td>
  </tr>
  <tr>
    <td>Morocco</td>
    <td>91.3</td>
    <td>94.2</td>
    <td>75.7</td>
    <td>93.1</td>
    <td>80.6</td>
    <td>93.6</td>
  </tr>
  <tr>
    <td>Libya</td>
    <td>91.1</td>
    <td>89.4</td>
    <td>81.0</td>
    <td>85.6</td>
    <td>81.0</td>
    <td>82.5</td>
  </tr>
</table>

Table 4: Performance of GPT-4o, Qwen-2.5–72B Instruct, and AceGPT-v2–32B Chat across countries and regions. CS denotes country-specific examples, while −CS otherwise. Green cells indicate the top three scores, while red cells highlight the bottom three.

<table>
  <tr>
    <th rowspan="2">Topic</th>
    <th colspan="2">GPT-4o</th>
    <th colspan="2">Qwen-2.5</th>
    <th colspan="2">AceGPT-v2</th>
  </tr>
  <tr>
    <th>CS</th>
    <th>−CS</th>
    <th>CS</th>
    <th>−CS</th>
    <th>CS</th>
    <th>−CS</th>
  </tr>
  <tr>
    <td>Agriculture</td>
    <td>91.5</td>
    <td>91.7</td>
    <td>83.0</td>
    <td>90.0</td>
    <td>78.7</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td>Art</td>
    <td>87.7</td>
    <td>92.4</td>
    <td>74.8</td>
    <td>84.8</td>
    <td>73.5</td>
    <td>84.8</td>
  </tr>
  <tr>
    <td>Daily Act.</td>
    <td>79.4</td>
    <td>90.2</td>
    <td>65.4</td>
    <td>83.2</td>
    <td>68.4</td>
    <td>86.6</td>
  </tr>
  <tr>
    <td>Death</td>
    <td>84.4</td>
    <td>91.3</td>
    <td>78.1</td>
    <td>83.1</td>
    <td>84.4</td>
    <td>81.2</td>
  </tr>
  <tr>
    <td>Family Rel.</td>
    <td>90.0</td>
    <td>91.1</td>
    <td>80.0</td>
    <td>88.9</td>
    <td>75.0</td>
    <td>83.3</td>
  </tr>
  <tr>
    <td>Food</td>
    <td>87.0</td>
    <td>90.4</td>
    <td>71.5</td>
    <td>79.4</td>
    <td>70.9</td>
    <td>79.4</td>
  </tr>
  <tr>
    <td>Habits</td>
    <td>81.8</td>
    <td>91.0</td>
    <td>68.8</td>
    <td>86.2</td>
    <td>71.4</td>
    <td>82.1</td>
  </tr>
  <tr>
    <td>Holiday Act.</td>
    <td>89.2</td>
    <td>94.4</td>
    <td>70.7</td>
    <td>89.1</td>
    <td>66.9</td>
    <td>90.1</td>
  </tr>
  <tr>
    <td>Idioms</td>
    <td>88.9</td>
    <td>91.9</td>
    <td>70.4</td>
    <td>83.8</td>
    <td>86.4</td>
    <td>81.1</td>
  </tr>
  <tr>
    <td>Parenting</td>
    <td>76.2</td>
    <td>93.8</td>
    <td>66.7</td>
    <td>87.5</td>
    <td>66.7</td>
    <td>84.4</td>
  </tr>
  <tr>
    <td>Trd. Games</td>
    <td>87.5</td>
    <td>89.6</td>
    <td>80.0</td>
    <td>84.7</td>
    <td>70.0</td>
    <td>80.6</td>
  </tr>
  <tr>
    <td>Wedding</td>
    <td>89.4</td>
    <td>89.1</td>
    <td>78.0</td>
    <td>79.6</td>
    <td>81.8</td>
    <td>78.9</td>
  </tr>
</table>

Table 5: Performance of GPT-4o, Qwen-2.5–72B Instruct, and AceGPT-v2–32B Chat across topics. CS denotes country-specific examples, while −CS otherwise. Green cells indicate the top three scores, while red cells highlight the bottom three.
5 Analysis

5.1 Result by Categories

In this section, we expand on our findings based on the top three models from Table 3: (1) GPT-4o, (2) Qwen-2.5-72B-Instruct, and (3) AceGPT-v2-32B-Chat. We provide a detailed analysis across country and topic, with each sample categorized as either country-specific (CS) or non-country-specific (~CS).

Country  In Table 4, we observe significant variation in LLM performance across countries, emphasizing the need for country-specific adaptation when deploying models. Questions from Jordan are consistently predicted with high accuracy, exceeding 90% across all models. However, performance drops significantly for Lebanon and Tunisia, where models struggle to provide correct answers. Even the Arabic-centric AceGPT-v2 achieves only 63.6% accuracy for Lebanon and 62.7% for Tunisia. Across the four regions, we find that the Levant is the most challenging, underscoring the difficulty of achieving reliable performance across different cultural and linguistic contexts.

More interestingly, country-specific questions prove to be more challenging than non-country-specific ones across nearly all countries and models. For instance, in the Nile Valley region, GPT-4o experiences an accuracy drop of nearly 10 points when handling country-specific questions, while in the Levant region, Qwen-2.5 sees a 17-point decline. This suggests that when cultural knowledge is not shared across multiple countries or regions, it becomes more distinct and difficult for LLMs to capture accurately.

Topic  Table 5 shows that LLMs encode cultural knowledge differently across various aspects of Arab culture. For example, GPT-4o performs best in agriculture and family relationships, while AceGPT excels in topics related to death and idioms. Meanwhile, Qwen achieves its highest accuracy in agriculture and traditional games. The accuracy gap between the highest- and lowest-performing topics across models ranges from 10 to 20 points, highlighting the difficulty of adapting cultural knowledge in LLMs. Additionally, we observe a consistent trend with Table 4, where country-specific samples are more challenging than non-country-specific ones.

![Performance comparison between jais-30b-chat-v3, AceGPT-v2-32B-Chat, and SILMA-9B-Instruct-v1.0 based on text generation output.](page_820_120_570_370.png)

Figure 3: Performance comparison between jais-30b-chat-v3, AceGPT-v2-32B-Chat, and SILMA-9B-Instruct-v1.0 based on text generation output. “Answer (T)” indicates that the generated answer is true, while “Exp (F)” denotes that the answer explanation is false.

5.2 Can the Model Provide a Reasonable Explanation to Support the Answer?

We focus on Arabic-centric models—Jais, AceGPT, and Silma—to evaluate their actual generation capabilities. For 200 randomly selected samples, we generate responses by appending مع ذكر السبب ("with mentioning the reason") to the Arabic prompt (§C.1) to instruct the model to provide a brief explanation for its choice. We then manually assess the outputs to verify both the correctness of the answer and the validity of the explanation.

Figure 3 presents the results for each model, comparing their generation accuracy with their MCQ performance from Table 3. Jais demonstrated a significant improvement, increasing from 40% in MCQ to 72% in the manual evaluation. In contrast, Silma’s performance dropped from 73% in MCQ to 67% in the generation task. Notably, Silma often failed to generate explanations, instead providing only the answer key or, at times, just the answer text. Meanwhile, AceGPT maintained a consistent performance across both MCQ and generation tests, showing no significant change in accuracy.

5.3 Improving Small Language Model with Additional Context from GPT-4o

We evaluate six base models with ≤3B parameters to assess the impact of cultural context augmentation. Using GPT-4o, we generate five factual Arabic sentences conditioned on the premise, subtopic, and country, following the Arabic prompt in Figure 6. These sentences are then incorporated into
Figure 4: Accuracies per models before and after context augmentation. This experiment uses the Arabic prompt template for MCQ and location \( \ell \in \{ \text{region} + \text{country} \} \).

the Arabic multiple-choice question prompt (§C.2) with location context \( \ell \in \{ \text{region} + \text{country} \} \). Results (Figure 4) show accuracy gains for Qwen and Llama models, except for Qwen2.5-0.5B (decline) and Gemma-2-2B (unchanged). Qwen2.5-3B achieves the highest improvement across all countries. These variations likely stem from differences in training data: Qwen and Llama were trained on multilingual datasets, whereas Gemma-2, primarily trained in English, has limited multilingual support.

6 Conclusion

We introduced ArabCulture, a benchmark for evaluating cultural commonsense reasoning in the Arab world. The dataset comprises 3,482 questions across 13 countries, covering 12 daily life domains with 54 fine-grained subtopics, all authored and validated by native speakers. Evaluations on 31 LLMs show significant performance gaps, with open-weight models up to 32B struggling to capture Arab cultural contexts. Variability across countries, regions, and topics highlights the need for more culturally aware models and datasets tailored to the Arabic-speaking world.

Limitations

Culture is only one side of reasoning While cultural knowledge plays a crucial role in shaping commonsense reasoning, it is only one of several dimensions that contribute to a model’s overall reasoning capabilities (Plaat et al., 2024). Reasoning in LLMs encompasses a broad range of cognitive skills, including logical inference, numerical reasoning, and causal understanding, among others.

The Influence of Dialects on Cultural Reasoning
The Arab world is characterized by rich dialects that vary not only across countries but also within different regions of the same country. These dialects significantly shape cultural expression, influencing language use in areas such as proverbs, humor, and everyday communication. This is particularly evident in topics like "idioms," where meaning and usage are deeply tied to specific dialects and local linguistic conventions.

However, to ensure that our evaluation isolates cultural commonsense reasoning rather than a model’s proficiency in specific dialects, we constructed ArabCulture in Modern Standard Arabic (MSA). MSA serves as a unifying linguistic medium across Arabic-speaking countries, allowing us to control for dialectal variation while still capturing essential cultural knowledge. While this approach enhances comparability across regions, it also introduces a limitation: certain cultural concepts that are best expressed through dialect-specific phrasing or context may not be fully represented in our dataset.

Location Leakage Despite our efforts to systematically control the granularity of location information, some questions or corresponding multiple-choice completions inadvertently reveal the location through the inclusion of location cues (landmarks, national events, etc.) within prompts. Thus, resulting in unintended location leakage, where the model gains access to country-specific cues directly from text rather than controlled contexts, making it difficult to isolate the effect of the location granularity control.

Coverage of All Arab Countries While our study covers a significant portion of the Arab world, representing 82% of the total population, certain unique cultures remain underrepresented. Notably, countries such as Mauritania, Somalia, and Comoros were not included, despite their distinct cultural and linguistic characteristics. These nations, located in North Africa, the Horn of Africa, and the Indian Ocean, respectively, contribute to the broader diversity of the Arab world. Their exclusion was primarily due to the difficulty in sourcing human annotators from these regions.

Acknowledgments

We are grateful to the supercomputing center at MBZUAI for providing resources for this research.
We also gratefully acknowledge the support of the university for providing funding to hire the human annotators.

References

Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdallahli, Yassine El Kheir, Daniel Izham, Fahim Dalvi, and 1 others. 2024. Larabench: Benchmarking arabic ai with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487–520.

Mubashara Akhtar, Abhilash Shankarampeta, Vivek Gupta, Arpit Patil, Oana Cocarascu, and Elena Simeperl. 2023. Exploring the numerical reasoning capabilities of language models: A comprehensive analysis on tabular data. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15391–15405, Singapore. Association for Computational Linguistics.

Emran Al-Bashabsheh, Huthaifa Al-Khazaleh, Omar Elayan, and Rehab Duwairi. 2021. Commonsense validation for arabic sentences using deep learning. In 2021 22nd International Arab Conference on Information Technology (ACIT), pages 1–7. IEEE.

Ebtesam Almazrouei, Ruxandra Cojocaru, Michele Baldo, Quentin Malartic, Hamza Alobeidli, Daniele Mazzotta, Guilherme Penedo, Giulia Campesan, Mughariya Farooq, Maitha Alhammadi, Julien Launay, and Badreddine Noune. 2023. AlGhaf evaluation benchmark for Arabic language models. In Proceedings of ArabicNLP 2023, pages 244–275, Singapore (Hybrid). Association for Computational Linguistics.

M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Aloitaib, Hisham A. Alyahya, Sultan Al-Rashed, Faisal A. Mirza, Shaykhah Z. Alsuaiba, Hassan A. Alahmed, Ghadah Abduljabbar, Raghad Alkhathran, Yousef Almushtaqhy, Raneem Alnajim, Salman Alsubahie, Maryam Al Mansour, Majed Alrubaiyan, Ali Alammar, Zakl Alawami, Abdulmohsen Al-Thubaity, and 6 others. 2024a. Allam: Large language models for arabic and english. Preprint, arXiv:2407.15390.

M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Aloitaib, Hisham A Alyahya, Sultan Al-Rashed, Faisal A Mirza, Shaykhah Z Alsuaiba, Hassan A Alahmed, Ghadah Abduljabbar, and 1 others. 2024b. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence.

John Dang, Shivalka Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, and 26 others. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.

DeepSeek-Al. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shiron Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Mona Diab, Nizar Habash, and Imed Zitouni. 2017. NLP for Arabic and related languages. Traitement Automatique des Langues, 58(3):9–13.

Li Du, Xiao Ding, Kai Xiong, Ting Liu, and Bing Qin. 2022. e-CARE: a new dataset for exploring explainable causal reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 432–446, Dublin, Ireland. Association for Computational Linguistics.

Abdelrahim Elmaldany, Ahmed El-Shangiti, Muhammad Abdul-Mageed, and 1 others. 2023a. Dolphin: A challenging and diverse benchmark for arabic nlg. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1404–1422.

AbdelRahim Elmaldany, ElMoatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023b. ORCA: A challenging benchmark for Arabic language understanding. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9559–9586, Toronto, Canada. Association for Computational Linguistics.

Fanar-Team. 2024. Fanar: An arabic-centric multimodal generative ai platform. https://fanar.qa/en. arXiv preprint arXiv:2409.11404.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noach’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others. 2024. A framework for few-shot language model evaluation.

A. Giddens and P.W. Sutton. 2014. Essential Concepts in Sociology. Polity Press.

Aaron Grattafiore, Abhimanyu Dubey, Abhinav Jauhari, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, and 542 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Huang Huang, Fei Yu, Jianqing Zhu, Xuening Sun, Hao Cheng, Song Dingjie, Zhihong Chen, Mosen Alharthi, Bang An, Juncai He, Ziche Liu, Junying Chen, Jianquan Li, Benyou Wang, Lian Zhang, Ruoyu Sun, Xiang Wan, Haizhou Li, and Jinchao Xu.
2024. AceGPT, localizing large language models in Arabic. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8139–8163, Mexico City, Mexico. Association for Computational Linguistics.

Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, and Firoj Alam. 2024. Native vs non-native language prompting: A comparative analysis. Preprint, arXiv:2409.07054.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shenhata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024a. ArabicMMLU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics.

Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024b. IndoCulture: Exploring geographically influenced cultural commonsense reasoning across eleven Indonesian provinces. Transactions of the Association for Computational Linguistics, 12:1703–1719.

Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning.

Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868, Online. Association for Computational Linguistics.

Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhao Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, and 9 others. 2023. Llm360: Towards fully transparent open-source llms. ArXiv, abs/2312.06550.

J.J. Macionis. 2012. Sociology: Fourteenth Edition. Pearson.

Barry Mirkin. 2010. Population levels, trends and policies in the arab region: Challenges and opportunities.

Basel Mousi, Nadir Durrani, Fatema Ahmad, Md. Arid Hasan, Maram Hasanain, Tameem Kabani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics.

Niklas Muenninghoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Haley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.

Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.

OpenAI, ; Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Wellinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024a. Gpt-4o system card. Preprint, arXiv:2410.21276.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shayman Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others. 2024b. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. 2024. Reasoning with large language models, a survey. Preprint, arXiv:2407.11511.

Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L’eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram’e, Johan Ferret, Peter Liu, Pouya Dehghani Tafui, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, and 176 others. 2024. Gemma 2: Improving open language models at a practical size. ArXiv, abs/2408.00118.

Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019a. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019b. Social IQa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.

Maarten Sap, Vered Shwartz, Antoine Bosselut, Yejin Choi, and Dan Roth. 2020. Commonsense reasoning for natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, pages 27–33, Online. Association for Computational Linguistics.

Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, and I others. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. arXiv preprint arXiv:2308.16149.

Abdulhadi Shoufan and Sumaya Alameri. 2015. Natural language processing for dialectical Arabic: A survey. In Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 36–48, Beijing, China. Association for Computational Linguistics.

Qingyu Tan, Hwee Tou Ng, and Lidong Bing. 2023. Towards benchmarking and improving the temporal reasoning capability of large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14820–14835, Toronto, Canada. Association for Computational Linguistics.

Saja Khaled Tawalbeh and Mohammad Al-Smadi. 2020. Is this sentence valid? an arabic dataset for commonsense validation. ArXiv, abs/2008.10873.

Silma Team. 2024. Silma.

A Dataset Statement for ArabCulture

A.1 General Information

Dataset title ArabCulture
Dataset version 1.0 (Feb 2025)

A.2 Executive Summary

ArabCulture is a cultural Arabic commonsense reasoning dataset covering 13 Arabic countries in the Gulf, Levant, North Africa, and the Nile Valley. The dataset spans 12 daily life domains and 54 fine-grained subtopics. It was created from scratch by native speakers who validated culturally relevant questions.

A.3 Curation Rationale

ArabCulture serves as a cultural benchmark to assess large language models’ ability to reason within culturally specific contexts. Built from scratch by native speakers, it avoids web-scraped text and undergoes rigorous quality control to ensure lexical accuracy, semantic coherence, and cultural sensitivity.

The dataset creation process involves:

1. Coverage Determination: Selecting relevant countries and topics.
2. Annotator Selection: Hiring qualified native speakers with deep cultural knowledge.
3. Example Generation: Each annotator produces 150 examples, with two annotators per country.
4. Cross-Review: Annotators validate each other’s work by confirming correct answers.
5. Final Review: Unclear samples are revised or discarded based on comprehensive quality checks.

A.4 Documentation for Source Datasets

ArabCulture is built entirely from scratch, without relying on web-scraped text. All data is manually created and validated by native speakers from 13 Arabic-speaking countries.

A.5 Country and Regional Diversity

ArabCulture covers 13 Arab countries, ensuring rich cultural perspectives.

Our country selection was guided by the goal of broad geographic representation across the Arab world. These 13 countries span four key regions:

• The Gulf: Saudi Arabia, Yemen, UAE.
• The Levant: Syria, Jordan, Palestine, Lebanon.
• North Africa: Morocco, Algeria, Tunisia, Libya.
• The Nile Valley: Egypt, Sudan.
A.6 Annotator Demographics
We recruited 26 expert annotators from 13 Arab countries, with two annotators representing each country. To ensure cultural authenticity and linguistic proficiency, we enforced the following eligibility criteria:
• Native Arabic speakers.
• Residency in the country for at least 10 years.
• Deep understanding of local culture and traditions.
• Both parents are native to and reside in the country.
• Minimum educational requirement of a high school diploma (higher degrees are preferred).
Among the 26 annotators, 14 hold a Bachelor’s degree, seven have a Master’s degree, two have a PhD, and three have a high school diploma. All annotators participated (attended or watched a record) in an initial online workshop to ensure a clear understanding of the project guidelines.

A.7 Topic Diversity
ArabCulture features a carefully curated taxonomy of daily life topics. It includes 12 main topics with 54 fine-grained subtopics—covering areas such as food, weddings, holiday activities, daily routines, habits, traditional games, death, art, parenting, agriculture, family relationships, and idioms. This extensive range ensures that the dataset captures both common and unique cultural experiences across the Arab world.

B Chosen Topics Distribution
Table 8 shows the distribution of topics and their corresponding subtopics.

C Prompts
C.1 Zero-shot Experiment prompts
Figure 5 shows the prompts in Arabic and English that we used for the zero-shot experiments. The Arabic prompt is also used to generate responses for 5.2.

C.2 Cultural Context prompts
Figure 6 displays the Arabic prompt used to generate five culturally grounded sentences for enhancing the small language model with additional context from GPT-4o. An English translation is provided to assist readers unfamiliar with Arabic in understanding the prompt.

ARABIC PROMPT
يمكنك اختيار النموذج المناسب بناءً على السياق المقدم أدناه.
الموقع: [country]
الملخص: [PREMISE]
يرجى إنشاء النصوص الثقافية المتنوعة والنموذج الإيجابي الأكثر ملاءمة من الخيارات التالية:
[OPTIONS]

ENGLISH PROMPT
You are tasked with selecting the most culturally appropriate option based on the context provided below.
Location: [country]
Statement: [PREMISE]
Consider the cultural nuances of the specified location and choose the most suitable response from the options provided.
Options: [OPTIONS]

Figure 5: Templates for multiple-choice question prompts. Sentence completion prompt template is the same but without the options section. Location \( \ell \in \{ \text{none, region, region + country} \} \).

ARABIC PROMPT
قم بإنشاء 5 جمل عربية وواقعية حول الفرضية والموضوع التاليين:
الفرضية: [premise]
الموضوع: [topic]
يجب أن تكون هذه الجمل خاصة ببلدك أو بلدك، مع الاحتفاظ بالاستثناءات التي تذكرها. تأكد من أن المعلومات ذات صلة، وواقعية، وملائمة لثقافتك، مع الاحتفاظ بالاستثناءات غير مرتبطة ببلدك.
نطلب وضع اقتراحات غير مرتبطة ببلدك.

ENGLISH PROMPT
Generate 5 factual Arabic sentences about the following premise and topic:
Premise: [premise]
Topic: [topic].
These sentences should be specific to the context of the following country: [country]. Ensure that the information is relevant, factual, and culturally appropriate while remaining concise. Avoid making assumptions that are not supported by established facts.

Figure 6: Prompts to generate culturally relevant sentences for context augmentation. The Arabic prompt was used for the generation.

D Results of the Arabic prompt in the zero-shot experiments
Table 9 presents the zero-shot experiment results using the Arabic prompt, which is illustrated in Figure 5.
D.1 Results by Geographic Location - Arabic Prompt

<table>
  <tr>
    <th rowspan="2">Topic</th>
    <th colspan="2">GPT-4o</th>
    <th colspan="2">AceGPT-v2</th>
    <th colspan="2">Llama-3.3</th>
  </tr>
  <tr>
    <th>CS</th>
    <th>¬CS</th>
    <th>CS</th>
    <th>¬CS</th>
    <th>CS</th>
    <th>¬CS</th>
  </tr>
  <tr>
    <td><b>Gulf</b></td>
    <td>81.0</td>
    <td>92.5</td>
    <td>76.3</td>
    <td>85.3</td>
    <td>69.1</td>
    <td>81.5</td>
  </tr>
  <tr>
    <td>KSA</td>
    <td>89.5</td>
    <td>93.4</td>
    <td>77.9</td>
    <td>84.3</td>
    <td>75.8</td>
    <td>80.1</td>
  </tr>
  <tr>
    <td>UAE</td>
    <td>90.0</td>
    <td>93.4</td>
    <td>75.0</td>
    <td>90.2</td>
    <td>66.0</td>
    <td>87.4</td>
  </tr>
  <tr>
    <td>Yemen</td>
    <td>72.8</td>
    <td>88.1</td>
    <td>76.2</td>
    <td>74.6</td>
    <td>67.5</td>
    <td>68.7</td>
  </tr>
  <tr>
    <td><b>Levant</b></td>
    <td>77.5</td>
    <td>92.3</td>
    <td>67.6</td>
    <td>86</td>
    <td>66.5</td>
    <td>84.4</td>
  </tr>
  <tr>
    <td>Lebanon</td>
    <td>70.7</td>
    <td>78.8</td>
    <td>60.6</td>
    <td>67.9</td>
    <td>62.6</td>
    <td>66.0</td>
  </tr>
  <tr>
    <td>Syria</td>
    <td>82.5</td>
    <td>94.6</td>
    <td>72.5</td>
    <td>87.0</td>
    <td>70.0</td>
    <td>80.8</td>
  </tr>
  <tr>
    <td>Palestine</td>
    <td>87.0</td>
    <td>94.4</td>
    <td>78.3</td>
    <td>87.2</td>
    <td>65.2</td>
    <td>88.8</td>
  </tr>
  <tr>
    <td>Jordan</td>
    <td>100.0</td>
    <td>96.1</td>
    <td>90.9</td>
    <td>94.3</td>
    <td>90.9</td>
    <td>93.9</td>
  </tr>
  <tr>
    <td><b>Nile Valley</b></td>
    <td>86.8</td>
    <td>96.1</td>
    <td>70.7</td>
    <td>86.1</td>
    <td>76.0</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td>Egypt</td>
    <td>88.8</td>
    <td>98.5</td>
    <td>72.1</td>
    <td>79.4</td>
    <td>76.1</td>
    <td>72.1</td>
  </tr>
  <tr>
    <td>Sudan</td>
    <td>84.0</td>
    <td>94.6</td>
    <td>68.8</td>
    <td>90.2</td>
    <td>75.7</td>
    <td>92.9</td>
  </tr>
  <tr>
    <td><b>North Africa</b></td>
    <td>82.0</td>
    <td>86.0</td>
    <td>73.7</td>
    <td>78.8</td>
    <td>73.5</td>
    <td>80.9</td>
  </tr>
  <tr>
    <td>Tunisia</td>
    <td>67.5</td>
    <td>79.8</td>
    <td>62.7</td>
    <td>72.5</td>
    <td>68.7</td>
    <td>69.7</td>
  </tr>
  <tr>
    <td>Algeria</td>
    <td>83.8</td>
    <td>84.3</td>
    <td>68.9</td>
    <td>73.1</td>
    <td>63.5</td>
    <td>80.2</td>
  </tr>
  <tr>
    <td>Morocco</td>
    <td>90.3</td>
    <td>94.2</td>
    <td>81.6</td>
    <td>93.1</td>
    <td>83.5</td>
    <td>93.1</td>
  </tr>
  <tr>
    <td>Libya</td>
    <td>84.8</td>
    <td>86.2</td>
    <td>79.7</td>
    <td>77.5</td>
    <td>74.7</td>
    <td>81.2</td>
  </tr>
</table>

Table 6: Performance of the best three models using the Arabic prompt. The results for GPT-4o are using the Region prompt, while the results for AceGPT-v2-32B-Chat and Llama-3.3-70B-Instruct are using the Country_Region prompt. The first column includes the different locations (countries and regions) with the regions in bold. CS refers to the Country Specific examples, while ¬CS refers to the rest of the examples. The green and red cells indicate the top three and bottom three scores, respectively.

Table 6 presents the country-level breakdown analysis based on the Arabic prompt.

D.2 Results by Topic - Arabic Prompt

Table 7 presents the breakdown analysis by topic based on the Arabic prompt.

<table>
  <tr>
    <th rowspan="2">Topic</th>
    <th colspan="2">GPT-4o</th>
    <th colspan="2">AceGPT-v2</th>
    <th colspan="2">Llama-3.3</th>
  </tr>
  <tr>
    <th>CS</th>
    <th>¬CS</th>
    <th>CS</th>
    <th>¬CS</th>
    <th>CS</th>
    <th>¬CS</th>
  </tr>
  <tr>
    <td>Agriculture</td>
    <td>87.2</td>
    <td>95.0</td>
    <td>76.6</td>
    <td>85.0</td>
    <td>78.7</td>
    <td>83.3</td>
  </tr>
  <tr>
    <td>Art</td>
    <td>85.2</td>
    <td>89.9</td>
    <td>74.2</td>
    <td>87.3</td>
    <td>74.8</td>
    <td>86.1</td>
  </tr>
  <tr>
    <td>Daily Activities</td>
    <td>76.5</td>
    <td>89.9</td>
    <td>67.6</td>
    <td>83.2</td>
    <td>69.9</td>
    <td>84.3</td>
  </tr>
  <tr>
    <td>Death</td>
    <td>78.1</td>
    <td>90.3</td>
    <td>87.5</td>
    <td>84.1</td>
    <td>78.1</td>
    <td>82.6</td>
  </tr>
  <tr>
    <td>Family Relationships</td>
    <td>85.0</td>
    <td>91.1</td>
    <td>80.0</td>
    <td>85.6</td>
    <td>60.0</td>
    <td>85.6</td>
  </tr>
  <tr>
    <td>Food</td>
    <td>82.0</td>
    <td>91.2</td>
    <td>69.3</td>
    <td>79.9</td>
    <td>68.7</td>
    <td>76.5</td>
  </tr>
  <tr>
    <td>Habits</td>
    <td>80.5</td>
    <td>88.4</td>
    <td>68.8</td>
    <td>84.0</td>
    <td>72.7</td>
    <td>85.8</td>
  </tr>
  <tr>
    <td>Holiday Activities</td>
    <td>82.8</td>
    <td>92.1</td>
    <td>74.5</td>
    <td>88.8</td>
    <td>70.7</td>
    <td>87.2</td>
  </tr>
  <tr>
    <td>Idioms</td>
    <td>84.0</td>
    <td>97.3</td>
    <td>80.2</td>
    <td>78.4</td>
    <td>70.4</td>
    <td>75.7</td>
  </tr>
  <tr>
    <td>Parenting</td>
    <td>71.4</td>
    <td>91.7</td>
    <td>71.4</td>
    <td>84.4</td>
    <td>76.2</td>
    <td>88.5</td>
  </tr>
  <tr>
    <td>Traditional Games</td>
    <td>86.2</td>
    <td>89.6</td>
    <td>65.0</td>
    <td>77.8</td>
    <td>72.5</td>
    <td>77.8</td>
  </tr>
  <tr>
    <td>Wedding</td>
    <td>84.1</td>
    <td>89.8</td>
    <td>80.3</td>
    <td>85.7</td>
    <td>75.8</td>
    <td>81.6</td>
  </tr>
</table>

Table 7: Performance of the best three models using the Arabic prompt across the different Topics. The results for GPT-4o are using the Region prompt, while the results for AceGPT-v2-32B-Chat and Llama-3.3-70B-Instruct are using the Country_Region prompt. CS refers to the Country Specific examples, while ¬CS refers to the rest of the examples. The green and red cells indicate the top three and bottom three scores, respectively.
<table>
  <tr>
    <th>Topics</th>
    <th>Sub-topics</th>
    <th>#Samples</th>
  </tr>
  <tr>
    <td>Food</td>
    <td>Breakfast (5), Lunch (5), Dinner (2), Sahoor (Ramadan) (5), Iftar (Ramadan) (5), Dessert (3), Fruits (3), Snacks (2)</td>
    <td>30</td>
  </tr>
  <tr>
    <td>Wedding</td>
    <td>Wedding location (1), Wedding food (1), Wedding dowry (1), Wedding other logistics (2), Men ceremony vs. women ceremony (2), Songs and activities during the wedding (5)</td>
    <td>12</td>
  </tr>
  <tr>
    <td>Holiday Activities</td>
    <td>Traditions before religious holidays (5), Traditions during religious holidays (10), Activities for non-religious holidays (5)</td>
    <td>20</td>
  </tr>
  <tr>
    <td>Daily Activities</td>
    <td>Before going to work/college (4), While on the way to college/work (3), Things you do with colleagues/friends while at work/college (2), Things you do after coming back from work/uni (men) (2), Things you do after coming back from work/uni (women) (2), Household activities (groceries, fixing things, cleaning, etc.) (6), Things you do in your free time (indoors or outdoors) (5)</td>
    <td>24</td>
  </tr>
  <tr>
    <td>Habits</td>
    <td>Eating habits (3), Stereotypes (5), Communication habits (3), Financial habits (1), Gift-giving practices (1), Cleanliness habits (2)</td>
    <td>14</td>
  </tr>
  <tr>
    <td>Traditional Games</td>
    <td>Childhood games indoors (5), Childhood games outdoors (5)</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Death</td>
    <td>Before burying (3), Burying rituals (2), After burying ceremonies (4), Inheritance (1)</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Art</td>
    <td>Musical instruments (3), Local songs (3), Local dances (4)</td>
    <td>10</td>
  </tr>
  <tr>
    <td>Parenting</td>
    <td>Parents-child actions (3), Grandparents-child actions (2)</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Agriculture</td>
    <td>What to plant (3), While planting (1), Harvest (1)</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Idioms</td>
    <td>Idioms in context (5)</td>
    <td>5</td>
  </tr>
  <tr>
    <td>Family Relationships</td>
    <td>Between siblings (2), With cousins (1), Relationship of parents and child (2)</td>
    <td>5</td>
  </tr>
</table>

Table 8: Overview of topics, sub-topics, and the sample counts for each topic. The number in parenthesis beside each subtopic represents the number of samples for each subtopic.
<table>
  <tr>
    <th rowspan="2">Model (#parameter)</th>
    <th colspan="3">Completion</th>
    <th colspan="3">MCQ</th>
  </tr>
  <tr>
    <th>\( \ell = \text{None} \)</th>
    <th>\( \ell = R \)</th>
    <th>\( \ell = R + C \)</th>
    <th>\( \ell = \text{None} \)</th>
    <th>\( \ell = R \)</th>
    <th>\( \ell = R + C \)</th>
  </tr>
  <tr>
    <td>Human</td>
    <td>--</td>
    <td>--</td>
    <td>100.0</td>
    <td>--</td>
    <td>--</td>
    <td>100.0</td>
  </tr>
  <tr>
    <td>Random</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>BLOOMZ (7B)</td>
    <td>30.1</td>
    <td>30.7</td>
    <td>30.9</td>
    <td>50.6</td>
    <td>52.2</td>
    <td>52.7</td>
  </tr>
  <tr>
    <td>mT0.27 (14B)</td>
    <td>26.6</td>
    <td>26.3</td>
    <td>26.9</td>
    <td>65.5</td>
    <td>66.3</td>
    <td>66.4</td>
  </tr>
  <tr>
    <td>Llama-3.1 (8B)</td>
    <td>27.7</td>
    <td>27.5</td>
    <td>27.6</td>
    <td>34.3</td>
    <td>34.1</td>
    <td>34.2</td>
  </tr>
  <tr>
    <td>Llama-3.1 Instruct (8B)</td>
    <td>32.2</td>
    <td>31.0</td>
    <td>31.3</td>
    <td>37.8</td>
    <td>36.8</td>
    <td>37.8</td>
  </tr>
  <tr>
    <td>Llama-3 Instruct (70B)</td>
    <td>36.6</td>
    <td>37.5</td>
    <td>38.5</td>
    <td>47.1</td>
    <td>37.0</td>
    <td>38.7</td>
  </tr>
  <tr>
    <td>Llama-3.3 Instruct (70B)</td>
    <td>39.0</td>
    <td>39.6</td>
    <td>39.9</td>
    <td><b>78.4</b></td>
    <td><b>77.8</b></td>
    <td><b>78.8</b></td>
  </tr>
  <tr>
    <td>Aya-Expanse (8B)</td>
    <td>34.7</td>
    <td>35.8</td>
    <td>36.9</td>
    <td>36.3</td>
    <td>38.1</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>Aya-Expanse (32B)</td>
    <td>37.9</td>
    <td>39.3</td>
    <td>39.9</td>
    <td>38.4</td>
    <td>43.7</td>
    <td>44.6</td>
  </tr>
  <tr>
    <td>Gemma-2 (9B)</td>
    <td>32.0</td>
    <td>32.2</td>
    <td>32.7</td>
    <td>34.5</td>
    <td>35.8</td>
    <td>35.4</td>
  </tr>
  <tr>
    <td>Gemma-2 Instruct (9B)</td>
    <td>32.5</td>
    <td>33.5</td>
    <td>33.5</td>
    <td>34.4</td>
    <td>34.3</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>Gemma-2 (27B)</td>
    <td>33.8</td>
    <td>34.4</td>
    <td>35.0</td>
    <td>34.3</td>
    <td>34.3</td>
    <td>34.4</td>
  </tr>
  <tr>
    <td>Gemma-2 Instruct (27B)</td>
    <td>35.9</td>
    <td>36.4</td>
    <td>37.1</td>
    <td>34.4</td>
    <td>34.9</td>
    <td>34.9</td>
  </tr>
  <tr>
    <td>Qwen2.5 (7B)</td>
    <td>30.0</td>
    <td>30.4</td>
    <td>30.4</td>
    <td>47.9</td>
    <td>47.0</td>
    <td>47.7</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (7B)</td>
    <td>32.5</td>
    <td>33.2</td>
    <td>33.8</td>
    <td>51.6</td>
    <td>37.7</td>
    <td>39.3</td>
  </tr>
  <tr>
    <td>Qwen2.5 (14B)</td>
    <td>32.4</td>
    <td>33.1</td>
    <td>33.5</td>
    <td>46.5</td>
    <td>57.8</td>
    <td>57.4</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (14B)</td>
    <td>36.5</td>
    <td>37.7</td>
    <td>37.2</td>
    <td>52.2</td>
    <td>58.4</td>
    <td>59.5</td>
  </tr>
  <tr>
    <td>Qwen2.5 (32B)</td>
    <td>33.4</td>
    <td>34.2</td>
    <td>34.5</td>
    <td>42.0</td>
    <td>43.7</td>
    <td>42.7</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (32B)</td>
    <td>36.6</td>
    <td>37.7</td>
    <td>37.8</td>
    <td>70.7</td>
    <td>74.6</td>
    <td>76.3</td>
  </tr>
  <tr>
    <td>Qwen2.5 (72B)</td>
    <td>35.4</td>
    <td>36.2</td>
    <td>36.4</td>
    <td>48.0</td>
    <td>52.2</td>
    <td>57.3</td>
  </tr>
  <tr>
    <td>Qwen2.5 Instruct (72B)</td>
    <td><b>39.6</b></td>
    <td><b>40.1</b></td>
    <td><b>41.0</b></td>
    <td>61.5</td>
    <td>64.5</td>
    <td>65.4</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill-Llama (70B)</td>
    <td>36.7</td>
    <td>37.1</td>
    <td>37.6</td>
    <td>34.7</td>
    <td>34.5</td>
    <td>35.0</td>
  </tr>
  <tr>
    <td>DeepSeek-R1-Distill-Qwen (32B)</td>
    <td>33.6</td>
    <td>34.4</td>
    <td>35.0</td>
    <td>34.3</td>
    <td>34.3</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>QwQ (32B)</td>
    <td>22.83</td>
    <td>22.29</td>
    <td>22.77</td>
    <td>32.88</td>
    <td>32.62</td>
    <td>32.28</td>
  </tr>
  <tr>
    <td>Jais (13B)</td>
    <td>37.9</td>
    <td>37.8</td>
    <td>38.3</td>
    <td>33.9</td>
    <td>33.6</td>
    <td>33.8</td>
  </tr>
  <tr>
    <td>Jais chat (13B)</td>
    <td>39.8</td>
    <td>39.9</td>
    <td>40.6</td>
    <td>40.7</td>
    <td>39.6</td>
    <td>38.4</td>
  </tr>
  <tr>
    <td>Jais-v3 (30B)</td>
    <td>39.9</td>
    <td>40.4</td>
    <td>40.6</td>
    <td>34.8</td>
    <td>35.8</td>
    <td>35.6</td>
  </tr>
  <tr>
    <td>Jais-v3 Chat (30B)</td>
    <td>34.0</td>
    <td>34.0</td>
    <td>34.5</td>
    <td>35.2</td>
    <td>36.2</td>
    <td>39.7</td>
  </tr>
  <tr>
    <td>SILMA Instruct (9B)</td>
    <td>32.5</td>
    <td>33.3</td>
    <td>33.4</td>
    <td>70.2</td>
    <td>70.7</td>
    <td>70.7</td>
  </tr>
  <tr>
    <td>AceGPT-v2 (8B)</td>
    <td>29.9</td>
    <td>31.2</td>
    <td>31.6</td>
    <td>34.3</td>
    <td>34.2</td>
    <td>34.2</td>
  </tr>
  <tr>
    <td>AceGPT-v2 Chat (8B)</td>
    <td>34.6</td>
    <td>35.1</td>
    <td>35.9</td>
    <td>44.8</td>
    <td>45.0</td>
    <td>45.4</td>
  </tr>
  <tr>
    <td>AceGPT-v2 Chat (32B)</td>
    <td>37.7</td>
    <td>38.9</td>
    <td>39.0</td>
    <td><b>78.8</b></td>
    <td><b>78.2</b></td>
    <td><b>79.8</b></td>
  </tr>
  <tr>
    <td>AceGPT-v2 Chat (70B)</td>
    <td><b>42.9</b></td>
    <td><b>44.5</b></td>
    <td><b>45.1</b></td>
    <td>73.6</td>
    <td>73.0</td>
    <td>74.4</td>
  </tr>
  <tr>
    <td>ALLaM-Instruct-preview (7B)</td>
    <td>36.5</td>
    <td>37.2</td>
    <td>37.9</td>
    <td>70.0</td>
    <td>71.9</td>
    <td>74.4</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td colspan="3"></td>
    <td><b>88.5</b></td>
    <td><b>91.9</b></td>
    <td><b>91.1</b></td>
  </tr>
</table>

Table 9: Zero-shot accuracy results for the Arabic prompt across various models and settings. "MCQ" refers to the multiple-choice question evaluation method, and \( \ell \) represents the inclusion of location context ("R" indicates the region, and "C" denotes the corresponding country). Bolded numbers highlight the highest score within each model group.

Reviewer paper summaries: This paper presents a novel benchmark to evaluate commonsense reasoning abilities of LLMs in the Arabic cultural context. The dataset spans across 13 Arabic countries, 12 domains and 54 subtopics. Authors benchmark several multi-lingual and Arabic LLMs and show that models under 32B struggle with cultural nuances of Arabic culture.

This paper focuses on the evaluation and development of LLMs in commonsense reasoning for the Arabic language. Unlike previous datasets based on machine translation, the paper proposes **ArabCulture**, a commonsense reasoning dataset in Modern Standard Arabic (MSA) that covers the cultures of 13 countries across the Gulf, Levant, North Africa, and the Nile Valley. The dataset was built from scratch by native speakers who created and validated culturally relevant questions for their respective countries. This work contributes to the advancement of LLMs in the Arabic language.
Summary of strengths: 1. Paper is well-written and appropriately structured
2. The benchmark is crowdsourced from scratch using Arabic native speakers instead of translating existing Western-centric datasets and hence, can better capture cultural nuances of Arabic countries.
3. The benchmark is reasonably large (3.4K) and diverse spanning across majority of Arabic countries, several daily life domains and the data collection has been done rigorously through multiple human verification steps.
4. Authors conduct a comprehensive benchmarking with several multi-lingual and Arabic-specific LLMs and analyze their results across domains and locations.

1. The paper presents a dataset contributed by native Arabic speakers, covering various domains and subtopics, which is beneficial for the development of LLM reasoning in the Arabic language.
2. The paper is well-written.
3. The experimental results show that the dataset has a certain level of discriminative power.
Summary of weaknesses: 1. Paper lacks details (how many people, what backgrounds, how many samples etc.) for the human performance on the benchmark. Additionally, authors report human performance for only the case where the region and the country is specified in the context. It is not clear why others are not reported which would allow for a fair comparison. 
2. While models under 32B don't perform well, bigger models already show quite good performance (over 80%) making the benchmark less challenging.
3. Some error analysis of the reasoning chains would be nice to gain further insight into the LLMs' knowledge of Arabic culture.

1. The model "4o" has already achieved a score close to 90 (MCQ) on this dataset. I’m afraid that newer and better models will quickly dominate this leaderboard.
2. How did the authors ensure the diversity of queries within each subtopic? Allowing a single annotator to write questions might introduce certain prior biases.
Comments, suggestions and typos: 1. Highest score is not bolded for all model groups (e.g. Gemma)
2. On line 542, I think the argument is weak because the Llama improvements seem negligible and Qwen-3B improvement is significantly high.

1. The improvement of instruction-tuned models in the MCQ setting is remarkably significant, especially for larger-scale models (e.g., Qwen2.5 72B vs. Qwen2.5 7B). How do the authors interpret this phenomenon?
2. How would reasoning-focused models perform on this benchmark?
3. The completion setting seems a bit unusual (almost all models score around 40). Do the authors have deeper insights into this task?
4. In lines 452-454, this conclusion seems somewhat counterintuitive. Intuitively, local information should help the model better understand the context. Do the authors have further insights into this?
Overall assessment scores: 4

3.5