Unveiling Cultural Blind Spots: Analyzing the Limitations of mLLMs in Procedural Text Comprehension
Authors: Amir Hossein Yari, Fajri Koto
Accepted at: acl25

Unveiling Cultural Blind Spots: Analyzing the Limitations of mLLMs in Procedural Text Comprehension

Amir Hossein Yari1    Fajri Koto2

1Sharif University of Technology
2Mohamed bin Zayed University of Artificial Intelligence
ahyari2002@gmail.com, fajri.koto@mbzui.ac.ae

Abstract

Despite the impressive performance of multilingual large language models (mLLMs) in various natural language processing tasks, their ability to understand procedural texts, particularly those with culture-specific content, remains largely unexplored. Texts describing cultural procedures, including rituals, traditional craftsmanship, and social etiquette, require an inherent understanding of cultural context, presenting a significant challenge for mLLMs. In this work, we introduce CAPTEx, a benchmark designed to evaluate mLLMs’ ability to process and reason over culturally diverse procedural texts in multiple languages. Using a range of evaluation methods, we find that (1) mLLMs struggle with culturally contextualized procedural content, particularly in low-resource languages; (2) performance varies across cultural domains, with some proving more difficult than others; and (3) models perform better on multiple-choice tasks presented in conversational formats than on direct questions. These results highlight the current limitations of mLLMs and emphasize the need for culturally informed benchmarks like CAPTEx to support more accurate and inclusive language understanding.1

1 Introduction

Procedural texts encompass a genre of writing that provides systematic instructions or guidance to navigate a sequence of actions or steps, aiming to achieve a specific outcome. These texts are common in various fields, including technical documentation, user manuals, and cookbooks. The core characteristic of procedural texts is their sequential and organized structure, with each instruction building on the previous one to ensure readers can successfully reach the intended outcome. Unlike other writing styles, such as narrative or descriptive, procedural texts emphasize clarity, accuracy, and a straightforward progression of actions to enable effective task completion.

Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing (NLP) tasks, such as text summarization (Jin et al., 2024), multi-modal machine translation (Shen et al., 2024a), solving complex tasks modeled as state machines (Wu et al.), and code generation and understanding (Wong et al., 2023). Unlike traditional models that rely on task-specific training, LLMs can be adapted to a wide range of applications through effective prompting strategies, making them suitable for diverse and dynamic contexts (Ouyang et al., 2022; Dai et al., 2023).

One particularly significant application area for LLMs is their ability to accurately interpret procedural texts. This capability is becoming increasingly vital as these models are employed in tasks like generating automated instructions and facilitating human-computer interactions (Kosch and Feger, 2024). In such scenarios, the demand for clear, contextually appropriate, and executable steps is critical. However, inaccuracies or ambiguities in interpreting procedural instructions can result in miscommunication, errors, and inefficiencies. These issues are particularly concerning in real-world domains such as healthcare, education, and technical support, where precision and clarity are paramount. Consequently, ensuring that LLMs can reliably process procedural texts is essential for their effective and responsible integration into various systems.

Culture plays a fundamental role in both the creation and comprehension of procedures. A step-by-step instruction in one culture may rely on shared knowledge or tacit understanding that is absent in others (Steffensen et al., 1979). For instance, instructions on performing a ritual practice may have

1This dataset is publicly released under a Creative Commons license at https://huggingface.co/datasets/AmirHossein2002/CAPTEx
<table>
  <tr>
    <th>Step</th>
    <th>Iran</th>
    <th>Indonesia</th>
  </tr>
  <tr>
    <td>1</td>
    <td>The body is taken to the cemetery for burial.</td>
    <td>The traditional Toraja house is prepared in silence, and tools are gathered.</td>
  </tr>
  <tr>
    <td>2</td>
    <td>The deceased’s body is washed according to Islamic rituals.</td>
    <td>The body is wrapped, and the coffin is decorated in a ritual performance.</td>
  </tr>
  <tr>
    <td>3</td>
    <td>The body is wrapped in a plain white shroud.</td>
    <td>A cultural parade is held, transporting the body from the house to the burial site.</td>
  </tr>
  <tr>
    <td>4</td>
    <td>A special prayer is performed for the deceased.</td>
    <td>A traditional Toraja dance is performed as part of the ceremony.</td>
  </tr>
  <tr>
    <td>5</td>
    <td>The body is buried facing Mecca, with a layer of dirt and stones over the grave.</td>
    <td>Animal sacrifices, typically buffalo and pigs, are offered as part of the final rites.</td>
  </tr>
</table>

Table 1: Comparison of funeral practices in Iran and Indonesia. For Indonesia, it’s a tradition from North Sumatra.

meanings, symbols, and steps deeply embedded in the traditions of a particular culture, as demonstrated in Table 1. If language models are unable to recognize and navigate these cultural dimensions, they risk misrepresenting the intent or structure of the procedure, leading to errors or misinterpretations.

Studies have shown that current large language models (LLMs) tend to exhibit biases favoring Western perspectives, mirroring the cultural norms and values of Western, educated, industrialized, rich, and democratic (WEIRD) societies (Durmus et al., 2024; Naous et al., 2024), while often inadequately representing other cultural contexts (Prabhakaran et al., 2022). These biases primarily stem from the nature of training data (Arora et al., 2022; Ganguli et al., 2022; Nadeem et al., 2021) and design decisions, including model architecture, tokenization approaches, evaluation methods, and instruction-tuning techniques. The significance of LLMs accurately comprehending these texts extends beyond the technical understanding of instructions; it also involves ensuring fairness, accessibility, and cultural sensitivity. Evaluating how well LLMs can decode procedural texts, particularly when cultural context plays a pivotal role, is critical for advancing their capability to serve a diverse range of users across different linguistic and cultural backgrounds.

Cultural procedural texts, which are deeply intertwined with societal norms, values, and traditions, pose unique challenges for LLMs. This complexity raises several critical questions about the ability of LLMs to effectively navigate and reason within culturally specific contexts: (1) How do LLMs perform in understanding procedural texts in low-resource languages compared to high-resource languages? (2) How effectively can LLMs recognize, interpret, and preserve the cultural nuances embedded in procedural texts? (3) Do LLMs demonstrate consistent performance across different cultural domains, such as food preparation, religious rituals, and celebration setups? (4) Are there noticeable strengths or weaknesses in LLMs’ understanding depending on the cultural context of the procedural text?

To address these questions, we introduce CAPTex (Culturally-Aware Procedural Texts), an innovative dataset crafted to evaluate multilingual LLMs’ (mLLMs) ability to reason culturally through the lens of procedural text understanding across diverse tasks, including reordering tasks, multiple-choice questions, and conversational frameworks, each of which is elaborated upon in detail in Section 3.2. CAPTex is carefully developed with contributions from native speakers representing seven culturally distinct regions—China, India, Indonesia, Iran, Japan, Nigeria, and Pakistan—ensuring authentic and nuanced cultural representation.

2 Related Work

Procedural text analysis has been a focal point of research, addressing a wide array of tasks within this domain. For example, Cao et al. (2024) tackles the cultural adaptation of recipes between Chinese and English-speaking cuisines. Their work aims to automate the translation and cultural adaptation of recipes, ensuring that cultural nuances—including ingredients, cooking techniques, and unit conversions—are appropriately represented. In contrast, our work extends beyond the food domain, encompassing multiple cultural contexts across seven countries, thereby offering a broader perspective on cross-cultural procedural knowledge.

Several studies have also focused on advancing entity tracking methodologies. NCET (Gupta and Durrett, 2019) introduces a mechanism for continuous-space entity tracking, employing a conditional random field (CRF) to ensure sequential consistency in predictions. Similarly, Huang et al.
(2021) utilizes a graph neural network to model semantic relationships among entities, actions, and locations, enhancing the understanding of procedural text.

Incorporating temporal aspects into procedural comprehension, Rajaby Faghihi and Kordjamshidi (2021) propose the Time-Stamped Language Model (TSLM), which augments pre-trained language models with timestamp embeddings. This approach has significantly improved performance on datasets such as Propara (Dalvi et al., 2018) and NPN-Cooking. Additionally, Tang et al. (2020) introduces the Interactive Entity Network (IEN), a recurrent network with memory designed to capture diverse entity interactions for state tracking. Meanwhile, Amini et al. (2020) develops an algorithm for procedural reading comprehension, translating texts into a formalism that represents processes as sequences of transitions over entity attributes.

Efforts to integrate multimodal data have also advanced procedural text analysis. For instance, Liu et al. (2020) introduces a transformer-based model that combines textual and visual information for processing multimodal recipe datasets effectively. Building on this, Wu et al. (2022) conducts benchmarking on reasoning and sequencing unordered multimodal instructions, highlighting that state-of-the-art models still fall short of human-level performance. While their work primarily focuses on step reordering, our evaluation framework is more comprehensive, introducing three additional tasks to assess LLMs’ capabilities. Furthermore, rather than being restricted to English, our research incorporates the native languages of the targeted countries, ensuring that the procedures analyzed are culturally unique rather than globally common.

Despite these advancements, a holistic benchmark for procedural text comprehension remains elusive. Our work sets a new standard by extending beyond food-related tasks to encompass multiple domains, incorporating a diverse range of languages beyond English, and evaluating the capabilities and limitations of mLLMs through a multifaceted assessment framework. In the following section, we will elaborate on these methodologies in detail, highlighting how our benchmark surpasses prior efforts.

3 CAPTex

To address our research objectives, we introduce CAPTex, a dataset that incorporates cultural procedural texts across English and seven additional languages from diverse linguistic and geographical backgrounds. CAPTex is built upon three foundational components: (1) a curated collection of procedures spanning ten unique categories, (2) a series of thoughtfully crafted multiple-choice questions designed to evaluate the comprehension of each procedure, and (3) a rich corpus of conversational exchanges offering clarifications on the corresponding procedures.

<table>
  <tr>
    <th>Language</th>
    <th>Class</th>
  </tr>
  <tr>
    <td>Chinese (Mandarin)</td>
    <td>5 - The Winners</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>5 - The Winners</td>
  </tr>
  <tr>
    <td>Persian</td>
    <td>4 - The Underdogs</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>4 - The Underdogs</td>
  </tr>
  <tr>
    <td>Indonesian</td>
    <td>3 - The Rising Stars</td>
  </tr>
  <tr>
    <td>Urdu</td>
    <td>3 - The Rising Stars</td>
  </tr>
  <tr>
    <td>Hausa</td>
    <td>2 - The Hopefuls</td>
  </tr>
</table>

Table 2: Resource availability of languages in CAPTex

3.1 Data Construction

Language Coverage Contemporary large language models demonstrate impressive performance in languages with abundant training data; however, their capabilities diminish when applied to low-resource languages and intricate cultural contexts, thereby constraining their global applicability (Rasheed et al., 2025). To promote linguistic diversity, we selected languages—Chinese (Mandarin), Japanese, Persian, Hindi, Indonesian, Urdu, and Hausa—representing a spectrum of resource availability, as measured by the criteria established by Joshi et al. (2020). Table 2 presents a detailed summary of the resource availability for each language featured in CAPTex.

Topic Taxonomies The procedures in CAPTex span ten culturally significant domains: (1) food and cuisine, (2) celebrations and festivals, (3) social etiquette and hospitality, (4) craftsmanship and artisan skills, (5) traditional attire and dress, (6) agricultural and seasonal practices, (7) religious and spiritual practices, (8) life milestones and family rites, (9) sports, games, and competitions, and (10) environmental and nature-based practices. These categories were selected by adapting and expanding upon the taxonomy from IndoCulture (Koto et al., 2024b) to ensure comprehensive coverage of cultural traditions and everyday practices. Please refer
Figure 1: End-to-End process of dataset creation

to Appendix for more detailed topic taxonomies.

Writing Procedural Text  As shown in Figure 1, for each language, we employed two native speakers from each target nation to manually write procedural texts in their native language along with their English translations. These workers have deep cultural ties, having spent their entire lives in their native lands, ensuring strong familiarity with local traditions. The only exceptions are two individuals who lived in their home countries for the first 25 years before moving abroad, but have been residing outside their homeland for fewer than five years.2

Given a specific topic or category, each worker manually wrote procedural texts in their native language and translated them into English. They were strictly prohibited from using AI-based text generation tools but were allowed to reference reliable literature to verify cultural details and improve the accuracy of their writing. In total, we produce 1,400 human-written procedural texts (100 texts per language × 7 native languages × 2, including English translations).

Quality Control  We ensure the high quality of CAPTex through two quality checks. First, we conduct an automated check to verify that each procedure consists of 5–10 steps and that the step count aligns between the native language and its English translation. Second, we perform a manual review by having workers cross-check procedural texts written by their peers. This evaluation follows a detailed checklist assessing conceptual accuracy, cultural relevance, logical progression, step order, grammatical correctness, and the accuracy and consistency of English translations. Any issues identified during the manual review are addressed by having the original worker revise their text accordingly. A common challenge arises from the lack of logical progression, which requires consolidating certain steps or eliminating those that do not impact the sequence. This restructuring makes the order of the remaining steps critical, as they become inherently non-interchangeable. For a more detailed description of the data collection process and quality control measures, refer to Appendix A.2.

3.2 Task Formulation

Using CAPTex, we developed four tasks, each designed with specific objectives. The following sections provide a detailed explanation of these tasks and their intended goals.

Task 1: Step Reordering  In this task, procedural steps are initially shuffled and labeled with sequential letters (such as A, B, C, D). The model is then tasked with reconstructing the correct sequence, outputting a comma-separated list of these letters without any additional explanation.

To gauge performance, we utilize three established metrics. The first, Spearman’s rank correlation (Spearman, 1904), examines the monotonic relationship between the predicted and actual rankings. The second, Levenshtein distance (Levenshtein, 1965), measures the minimum number of edit operations needed to transform the predicted sequence into the correct one. Lastly, Kendall’s Tau rank correlation (Kendall, 1938) assesses the ordinal agreement between the predicted and true sequence by counting pairwise swaps. This assessment is performed in both English and a native language, ensuring a thorough evaluation of the model’s ability to generalize across different linguistic environments.

Task 2: Procedure-Based Multiple-Choice Questions (PB-MCQ)  We design a comprehensive multiple-choice question (MCQ) framework for each procedure to evaluate mLLMs’ comprehension and reasoning abilities in identifying both sub-

2Each worker is compensated fairly based on a five-day workload, with payments aligned with the minimum wage in their respective country.
sequent and preceding steps. We created affirmative and negative versions for each question type, constructing them in the original language as well as in English. This approach ensured consistency across all question types (Subsequent Affirmative, Subsequent Negative, Antecedent Affirmative, Antecedent Negative) in both languages. To maintain linguistic parity, we construct all questions in both the original language and English. Each question consists of four answer choices, with one correct option. As a result, we generate eight MCQs per procedure, leading to a total of 5,600 questions (4 question types × 100 procedures × 14 languages, including native languages and their English translations).

In our question formulation, we ensured that for queries about upcoming steps, the correct answer was the next step, while three incorrect options were randomly chosen from earlier steps. Likewise, for questions regarding previous steps, the correct response was the preceding step, with three incorrect choices randomly selected from later steps. We confirmed that each question had one correct answer and three incorrect options. An illustrative example of MCQs is provided in Table 7. For additional information about MCQs, please refer to appendix A.3

The language model is prompted to generate only the correct choice option (A, B, C, or D) as its output. The primary evaluation metric for assessing model performance in this task is accuracy, measured by the proportion of correctly selected answers. Beyond evaluating the cultural understanding and procedural reasoning of language models, this task also enables an analysis of how well mLLMs comprehend affirmative and negative questions. Additionally, it assesses the model’s ability to predict both the subsequent and the antecedent procedural steps, further refining our understanding of its reasoning and contextual awareness.

Task 3: Conversation-Based Multiple-Choice Questions (CB-MCQ) To evaluate the reasoning capabilities of large language models (LLMs) using procedural text, we created procedurally grounded conversations in English. We utilized GPT-4o along with a specially designed prompt (detailed in Appendix A.4) to generate a series of four-utterance dialogues, conditioned on CAPTex. These dialogues simulate a natural conversation between two individuals, referred to as Person A and Person B. In the conversation, Person A starts by asking about a particular procedure. Person B, acting as a knowledgeable respondent, introduces the procedure and describes the initial steps involved. Person A then asks a follow-up question to clarify the next step in the process, and Person B provides a detailed explanation in response. Table 8 presents an example of the generated conversations.

Based on the conversation structure, the third utterance presents a question regarding the next step of the process. However, the fourth utterance, which contains the explanation of that step, is intentionally omitted. The language model must then select the correct answer from four options corresponding to the missing explanation, using the same answer choices as PB-MCQ. Comparing PB-MCQ and CB-MCQ allows us to assess whether models perform better in structured question-answering or conversational reasoning. PB-MCQ evaluates direct procedural knowledge, while CB-MCQ tests inference within dialogue, providing insights into model adaptability across different task formats.

For quality control, we employ a native speaker to review each dialogue assessing conceptual accuracy, grammatical correctness, and the consistency of Person B’s explanations with the actual steps outlined in the procedure. If any errors are identified, the worker manually corrects them to maintain accuracy and coherence. Similar to PB-MCQ, we use accuracy as the evaluation metric for CB-MCQ.

Task 4: Conversation-Based Question Answering (CB-QA) This task is similar to CB-MCQ but differs in that it requires the model to generate a response as if it were the conversational participant, rather than selecting from predefined choices. This task is chosen over CB-MCQ to evaluate the model’s ability to produce natural, contextually appropriate responses, reflecting a deeper understanding of procedural knowledge in dialogue. The performance of mLLMs is evaluated using three metrics: ROUGE-L score (Lin, 2004), BERTScore (Zhang et al., 2020), and additional semantic similarity score (Corley and Mihalcea, 2005), which quantifies meaning alignment between generated and reference responses.

3.3 Data Statistics

For each country and category, CAPTex incorporates 10 procedures, totaling 100 distinct procedures per country. Figure 2 represents the distribution of procedural steps across countries, with num-
bers above each bar indicating the average number of steps.

The methodology guarantees an equitable distribution of multiple-choice questions (MCQs) across various countries, categories, and formats. For each designated format3 a single question is crafted. This approach results in the creation of 2,800 unique questions in the original language and an additional 2,800 in English, culminating in a total of 5,600 MCQs. Tables 3 and 6 present the mean lexical density of CAPTex, analyzed across countries and categories, respectively.

The conversations dataset is thoroughly balanced, with one conversation constructed for each procedure. This approach yielded a total of 700 English conversations.

4 Experiments

4.1 Setup

We assessed 31 multilingual language models of different sizes, including DeepSeek (DeepSeek-AI et al., 2025), Gemma-2 (Team et al., 2024), Llama-3 (Grattafiori et al., 2024), Mamba (Gu and Dao, 2024), Mistral (Jiang et al., 2023), Qwen2.5 (Qwen et al., 2025), BLOOMZ (Muenninghoff et al., 2023), Aya-Expanse (Dang et al., 2024), mT0 (Muenninghoff et al., 2023), GPT-4 (OpenAI et al., 2024), and O3-mini (OpenAI, 2025).

We conducted zero-shot evaluations using prompt templates exclusively in English. Prior research has shown that prompting in different languages can lead to variations in responses to similar queries (Lin et al., 2022; Shen et al., 2024b). Moreover, studies on multilingual LLMs have consistently found that these models tend to perform better when prompted in English rather than in other languages (Muenninghoff et al., 2023; Ozsoy, 2024; Koto et al., 2024a).

3Subsequent Affirmative, Subsequent Negative, Antecedent Affirmative, and Antecedent Negative

Figure 2: Procedures step counts by country

4.2 Results and Analysis

Table 4 provides a comprehensive overview of model performance across all four tasks, consistently highlighting GPT-4o as the top-performing model. Among open-weight models, Gemma-2-9b-it outperforms others in the reordering task, while Qwen2.5-14B-Instruct achieves the highest accuracy in MCQ tasks, and Qwen2.5-7B demonstrates the strongest performance in the CB-QA task.

Among models of comparable size, Qwen2.5 demonstrates superior performance relative to its counterparts. Notably, Mamba exhibits significantly weaker performance in procedural text comprehension compared to transformer-based models. Our findings indicate that increasing the number of parameters within the same model family generally enhances performance. Additionally, for language models with an available Instruct variant, the Instruct versions consistently achieve higher performance—except in the cases of Gemma-2-2B and Qwen2.5-1.5B.

We calculate Kendall rank correlation scores across four tasks to assess task sensitivity when comparing LLMs. Our analysis shows strong correlations (0.8–0.9) between reordering, PB-MCQ, and CB-MCQ, indicating that these tasks rank models similarly. However, CB-QA (the generation task) has a lower correlation (0.4–0.5) with the others, suggesting that text generation captures different aspects of procedural reasoning and is a valuable addition to the evaluation.

Analysis of PB-MCQ Subtypes Table 5 showcases the performance of the top models across PB-MCQ task for four distinct question types: The results reveal that antecedent affirmative (AA) questions are the easiest for language models, while subsequent affirmative (SA) questions are the most challenging. This suggests that models find it easier to reason about preceding steps than following ones. Interestingly, while prior studies (Truong et al., 2023; She et al., 2023; Kassner and Schütze, 2020) indicate that negation generally weakens model performance in NLP tasks, this pattern does not hold in the procedural context. Subsequent negative (SN) leads to better performance, whereas affirmative negative (AN) questions result in a notable decline in accuracy.

Language Effects on Performance We examine the influence of language on the performance of Qwen2.5-14B-Instruct, the top-performing open-
<table>
  <tr>
    <th rowspan="2">Country</th>
    <th colspan="3">Procedures</th>
    <th colspan="2">MCQ (English/Native)</th>
    <th colspan="4">Conversations (Utterances)</th>
  </tr>
  <tr>
    <th>(English/Native)</th>
    <th>Question</th>
    <th>Correct Ans.</th>
    <th>Incorrect Ans.</th>
    <th>First</th>
    <th>Second</th>
    <th>Third</th>
    <th>Fourth</th>
  </tr>
  <tr>
    <td>China</td>
    <td>157.8 / 138.8</td>
    <td>38.0 / 39.8</td>
    <td>24.5 / 21.2</td>
    <td>26.0 / 22.8</td>
    <td>9.4</td>
    <td>46.2</td>
    <td>13.1</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>India</td>
    <td>37.5 / 50.4</td>
    <td>16.5 / 21.8</td>
    <td>4.8 / 6.3</td>
    <td>4.7 / 6.3</td>
    <td>9.3</td>
    <td>37.2</td>
    <td>12.7</td>
    <td>23.6</td>
  </tr>
  <tr>
    <td>Indonesia</td>
    <td>121.9 / 98.9</td>
    <td>39.1 / 31.9</td>
    <td>19.5 / 15.6</td>
    <td>20.1 / 16.4</td>
    <td>14.8</td>
    <td>41.1</td>
    <td>12.4</td>
    <td>30.1</td>
  </tr>
  <tr>
    <td>Iran</td>
    <td>144.5 / 166.4</td>
    <td>34.3 / 40.3</td>
    <td>21.6 / 24.7</td>
    <td>22.4 / 25.8</td>
    <td>10.0</td>
    <td>44.7</td>
    <td>13.2</td>
    <td>30.2</td>
  </tr>
  <tr>
    <td>Japan</td>
    <td>120.6 / 147.1</td>
    <td>30.6 / 45.5</td>
    <td>16.2 / 19.6</td>
    <td>15.8 / 19.2</td>
    <td>12.3</td>
    <td>43.8</td>
    <td>13.1</td>
    <td>29.5</td>
  </tr>
  <tr>
    <td>Nigeria</td>
    <td>162.3 / 173.3</td>
    <td>35.8 / 38.1</td>
    <td>21.1 / 22.8</td>
    <td>21.8 / 23.5</td>
    <td>11.0</td>
    <td>45.7</td>
    <td>13.2</td>
    <td>29.6</td>
  </tr>
  <tr>
    <td>Pakistan</td>
    <td>76.6 / 99.7</td>
    <td>23.9 / 36.4</td>
    <td>9.2 / 12.0</td>
    <td>9.4 / 12.2</td>
    <td>12.0</td>
    <td>42.0</td>
    <td>12.9</td>
    <td>27.0</td>
  </tr>
</table>

Table 3: Average word counts for CAPTex components (Procedures, MCQs, and Conversations) by country.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">Reordering</th>
    <th colspan="2">PB-MCQ</th>
    <th colspan="2">CB-MCQ</th>
    <th colspan="3">CB-QA</th>
  </tr>
  <tr>
    <th>\( \rho \uparrow \)</th>
    <th>LD\(\downarrow\)</th>
    <th>\( \tau \uparrow \)</th>
    <th>R-L</th>
    <th>BS</th>
    <th>SS</th>
  </tr>
  <tr>
    <td>Random</td>
    <td>0.00</td>
    <td>5.56</td>
    <td>0.00</td>
    <td>0.25</td>
    <td>0.25</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.00</td>
  </tr>
  <tr>
    <td>DeepSeek-R1(Distill-Llama-8B)</td>
    <td>0.30</td>
    <td>5.48</td>
    <td>0.14</td>
    <td>0.27</td>
    <td>0.38</td>
    <td>0.20</td>
    <td>0.53</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>DeepSeek-R1(Distill-Qwen-14B)</td>
    <td>0.43</td>
    <td>4.47</td>
    <td>0.33</td>
    <td>0.42</td>
    <td>0.54</td>
    <td>0.22</td>
    <td>0.55</td>
    <td>0.50</td>
  </tr>
  <tr>
    <td>Gemma-2-2b</td>
    <td>0.30</td>
    <td>5.49</td>
    <td>0.14</td>
    <td>0.27</td>
    <td>0.30</td>
    <td>0.23</td>
    <td>0.58</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Gemma-2-2b-it</td>
    <td>0.21</td>
    <td>5.48</td>
    <td>0.10</td>
    <td>0.08</td>
    <td>0.26</td>
    <td>0.25</td>
    <td>0.55</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Gemma-2-9b</td>
    <td>0.48</td>
    <td>4.94</td>
    <td>0.32</td>
    <td>0.37</td>
    <td>0.57</td>
    <td>0.21</td>
    <td>0.53</td>
    <td>0.47</td>
  </tr>
  <tr>
    <td>Gemma-2-9b-it</td>
    <td><b>0.75</b></td>
    <td><b>3.09</b></td>
    <td><b>0.66</b></td>
    <td><b>0.43</b></td>
    <td>0.46</td>
    <td>0.15</td>
    <td>0.50</td>
    <td>0.42</td>
  </tr>
  <tr>
    <td>Llama-3-1.8B</td>
    <td>0.30</td>
    <td>5.49</td>
    <td>0.14</td>
    <td>0.28</td>
    <td>0.32</td>
    <td>0.24</td>
    <td>0.59</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Llama-3-1.8B-Instruct</td>
    <td>0.43</td>
    <td>4.69</td>
    <td>0.33</td>
    <td>0.37</td>
    <td>0.48</td>
    <td>0.24</td>
    <td>0.59</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Llama-3-2.1B</td>
    <td>0.29</td>
    <td>5.56</td>
    <td>0.12</td>
    <td>0.25</td>
    <td>0.27</td>
    <td>0.22</td>
    <td>0.57</td>
    <td>0.52</td>
  </tr>
  <tr>
    <td>Llama-3-2.1B-Instruct</td>
    <td>0.31</td>
    <td>5.44</td>
    <td>0.15</td>
    <td>0.26</td>
    <td>0.34</td>
    <td>0.23</td>
    <td>0.58</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Llama-3-2.3B</td>
    <td>0.29</td>
    <td>5.56</td>
    <td>0.12</td>
    <td>0.26</td>
    <td>0.28</td>
    <td>0.23</td>
    <td>0.58</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Llama-3-2.3B-Instruct</td>
    <td>0.25</td>
    <td>5.35</td>
    <td>0.14</td>
    <td>0.33</td>
    <td>0.36</td>
    <td>0.24</td>
    <td>0.59</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Mamba-1.4B-hf</td>
    <td>0.30</td>
    <td>5.59</td>
    <td>-0.14</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.13</td>
    <td>0.59</td>
    <td>0.25</td>
  </tr>
  <tr>
    <td>Mamba-2.8B-hf</td>
    <td>0.25</td>
    <td>5.51</td>
    <td>0.14</td>
    <td>0.00</td>
    <td>0.00</td>
    <td>0.12</td>
    <td>0.39</td>
    <td>0.26</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.2</td>
    <td>0.51</td>
    <td>4.60</td>
    <td>0.40</td>
    <td>0.34</td>
    <td>0.48</td>
    <td>0.23</td>
    <td>0.58</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Mistral-7B-v0.3</td>
    <td>0.19</td>
    <td>5.45</td>
    <td>0.11</td>
    <td>0.32</td>
    <td>0.31</td>
    <td>0.24</td>
    <td>0.58</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Mistral-7B-Instruct-v0.3</td>
    <td>0.38</td>
    <td>4.75</td>
    <td>0.30</td>
    <td>0.37</td>
    <td>0.41</td>
    <td>0.24</td>
    <td>0.59</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-Base-2407</td>
    <td>0.33</td>
    <td>5.34</td>
    <td>0.18</td>
    <td>0.34</td>
    <td>0.50</td>
    <td>0.20</td>
    <td>0.53</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-Instruct-2407</td>
    <td>0.43</td>
    <td>4.59</td>
    <td>0.34</td>
    <td>0.39</td>
    <td>0.57</td>
    <td>0.21</td>
    <td>0.53</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B</td>
    <td>0.38</td>
    <td>5.15</td>
    <td>0.26</td>
    <td>0.31</td>
    <td>0.34</td>
    <td>0.23</td>
    <td>0.58</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-1.5B-Instruct</td>
    <td>0.42</td>
    <td>4.82</td>
    <td>0.32</td>
    <td>0.33</td>
    <td>0.21</td>
    <td>0.23</td>
    <td>0.58</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>0.63</td>
    <td>3.94</td>
    <td>0.52</td>
    <td>0.45</td>
    <td>0.54</td>
    <td>0.25</td>
    <td>0.59</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B-Instruct</td>
    <td>0.69</td>
    <td>3.65</td>
    <td>0.60</td>
    <td>0.50</td>
    <td>0.60</td>
    <td>0.24</td>
    <td>0.59</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>0.70</td>
    <td>3.33</td>
    <td>0.62</td>
    <td>0.48</td>
    <td>0.64</td>
    <td>0.26</td>
    <td>0.59</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Instruct</td>
    <td>0.72</td>
    <td>3.21</td>
    <td>0.64</td>
    <td><b>0.56</b></td>
    <td><b>0.70</b></td>
    <td>0.24</td>
    <td>0.59</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>Aya-Expanse-8b</td>
    <td>0.47</td>
    <td>4.71</td>
    <td>-0.37</td>
    <td>0.39</td>
    <td>0.43</td>
    <td>0.24</td>
    <td>0.58</td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>Bloomz-560m</td>
    <td>0.37</td>
    <td>5.39</td>
    <td>0.22</td>
    <td>0.15</td>
    <td>0.00</td>
    <td>0.14</td>
    <td>0.45</td>
    <td>0.32</td>
  </tr>
  <tr>
    <td>Bloomz-7b1</td>
    <td>0.29</td>
    <td>5.48</td>
    <td>0.13</td>
    <td>0.09</td>
    <td>0.20</td>
    <td>0.17</td>
    <td>0.50</td>
    <td>0.38</td>
  </tr>
  <tr>
    <td>mT0-xxl</td>
    <td>0.29</td>
    <td>5.54</td>
    <td>0.13</td>
    <td>0.35</td>
    <td>0.27</td>
    <td>0.14</td>
    <td>0.50</td>
    <td>0.43</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td><b>0.81</b></td>
    <td><b>2.38</b></td>
    <td><b>0.75</b></td>
    <td>0.58</td>
    <td><b>0.74</b></td>
    <td><b>0.29</b></td>
    <td><b>0.62</b></td>
    <td><b>0.60</b></td>
  </tr>
  <tr>
    <td>O3-mini</td>
    <td>0.78</td>
    <td>2.54</td>
    <td>0.72</td>
    <td><b>0.66</b></td>
    <td>0.65</td>
    <td>0.27</td>
    <td>0.61</td>
    <td>0.59</td>
  </tr>
</table>

Table 4: Models’ performance across tasks. Metrics include Spearman’s Rank Correlation (\( \rho \)) [-1,1], Levenshtein Distance (LD) [0,\infty), and Kendall’s Tau Rank Correlation (\( \tau \)) [-1,1] for Reordering; accuracy [0,1] for PB-MCQ and CB-MCQ; and ROUGE-F1 [0,1], BERT-F1 [0,1], and Semantic Similarity (SS) [0,1] for CB-QA. Higher values indicate better performance for all metrics except LD, where lower is better.

weight model, in the reordering and PB-MCQ tasks. To quantify this effect, we first normalize the evaluation metrics for these tasks and then compute an aggregated score using a weighted sum approach.\footnote{The weighted sum assigns 60% of the total score to the reordering task (20% for each metric), with Levenshtein Distance (LD) normalized as \( 1 - (L / L_{max}) \) to ensure higher values indicate better performance. The remaining 40% is assigned to PB-MCQ accuracy.} As depicted in Figure 3, with the exception of China, Qwen2.5-14B-Instruct generally
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">PB-MCQ</th>
  </tr>
  <tr>
    <th>SA</th>
    <th>SN</th>
    <th>AA</th>
    <th>AN</th>
  </tr>
  <tr>
    <td>Gemma-2-9b-it</td>
    <td>0.31</td>
    <td>0.51</td>
    <td>0.56</td>
    <td>0.36</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>0.24</td>
    <td>0.38</td>
    <td>0.55</td>
    <td>0.33</td>
  </tr>
  <tr>
    <td>Mistral-Nemo-Instruct-2407</td>
    <td>0.29</td>
    <td>0.34</td>
    <td>0.60</td>
    <td>0.34</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B-Instruct</td>
    <td>0.45</td>
    <td>0.53</td>
    <td>0.69</td>
    <td>0.55</td>
  </tr>
  <tr>
    <td>Aya-Expanse-8b</td>
    <td>0.33</td>
    <td>0.38</td>
    <td>0.61</td>
    <td>0.24</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>0.46</td>
    <td>0.57</td>
    <td><b>0.73</b></td>
    <td>0.54</td>
  </tr>
  <tr>
    <td>O3-mini</td>
    <td><b>0.51</b></td>
    <td><b>0.70</b></td>
    <td>0.72</td>
    <td><b>0.72</b></td>
  </tr>
</table>

Table 5: Model performance on PB-MCQ across question types. “SA”, “SN”, “AA”, and “AN” denote Subsequent Affirmative, Subsequent Negative, Antecedent Affirmative, and Antecedent Negative, respectively.

![Bar chart showing language impact on Qwen2.5-14B-Instruct performance](page_184_682_670_314.png)

Figure 3: Language impact on Qwen2.5-14B-Instruct performance

outperforms in English across other countries, especially in low-resource languages like Hausa and Urdu. This divergence may stem from the model’s extensive proficiency in English, whereas the linguistic nuances, idiomatic expressions, and procedural reasoning structures inherent to Chinese contexts might be underrepresented in the training data.

Performance Across Cultural Dimensions  Figure 4 presents the aggregated performance scores of the top-performing open-weight model, Qwen2.5-14B-Instruct, across all evaluation tasks. Scores are normalized and combined using a weighted sum approach, with task-specific weights designed to better reflect the varying cognitive demands and real-world relevance of each evaluation. Specifically, reordering is given a higher weight (30%) due to its requirement for holistic sequence reconstruction, encompassing temporal logic, global coherence, and dependency tracking across steps. In contrast, PB-MCQ and CB-MCQ (each weighted at 20%) test narrower, one-step reasoning skills and are thus weighted slightly lower. CB-QA is assigned a higher weight (30%) as it evaluates open-ended generative capabilities, probing the model’s contextual understanding, fluency, and ability to engage in realistic procedural dialogue. Notably, its lower correlation with the other tasks highlights its role in capturing distinct aspects of procedural reasoning. This weighted aggregation offers a more nuanced and representative measure of procedural competence than a simple average.

Our analysis reveals that Qwen2.5-14B-Instruct’s knowledge of procedural texts varies across cultural domains. For instance, it demonstrates strong familiarity with Indian agriculture but performs less effectively on Chinese agricultural topics. Conversely, for craftsmanship and artisan skills, Qwen2.5-14B-Instruct encodes Chinese cultural knowledge better than other countries. In the food category, Indian cuisine is better represented, while Iranian religious practices appear more prominently. Additionally, Indonesian social etiquette are well captured by Qwen2.5-14B-Instruct, suggesting variation in how different cultural aspects are reflected in the model.

![Heatmap showing cultural dimension performance by country](page_1012_682_370_180.png)

Figure 4: Cultural dimension performance by country

![Line chart showing impact of step count on GPT-4o reordering](page_1012_882_370_314.png)

Figure 5: Impact of step count on GPT-4o reordering

Impact of Procedure Length on Ordering  Figure 5 shows that as procedure length increases, Levenshtein distance rises, indicating greater re-
ordering difficulty. However, Spearman’s rank and Kendall’s Tau correlations remain high, suggesting that GPT-4o generally preserves step order despite complexity. Shorter procedures introduce more ambiguity, making errors more impactful, while longer sequences benefit from stronger local dependencies, aiding order retention. Notably, procedures with ten steps achieve the highest rank correlations, indicating that structural cues in longer sequences enhance model performance. Most errors involve minor swaps rather than complete mis-ordering.

5 Conclusion

We introduce CAPTex, a benchmark for evaluating mLLMs’ ability to process culturally diverse procedural texts across seven languages. Our findings show that model performance varies across cultural domains, with greater challenges in tasks requiring implicit cultural knowledge, such as environmental practices, while structured domains like craftsmanship are better handled. Multiple-choice tasks in conversational contexts improve reasoning, while generation evaluation highlights gaps in procedural text comprehension. As an important benchmarking tool, CAPTex helps identify specific areas where mLLMs need improvement in handling culturally diverse procedural text. It thereby establishes a crucial foundation for guiding future advancements towards developing more robust, culturally aware, and effective mLLMs capable of complex procedural reasoning across a wide range of languages and cultural contexts.

Limitations

Our study provides valuable insights into the performance of mLLMs on procedural texts, but there are a few limitations to consider. Firstly, our research is limited to textual data and does not include multi-modal inputs, such as procedural texts with images. Incorporating images would enhance model understanding, but due to the added complexity, this is reserved for future work.

Additionally, our dataset focuses on seven countries, primarily due to budget constraints. While this may seem limited, the selected countries offer diverse language categories and varying resource levels, ensuring a meaningful analysis of cultural gaps in mLLMs. These findings can be generalized to a broader context, given the representativeness of the samples.

Finally, the conversation dataset consists of exchanges with exactly four utterances. While real-world dialogues are typically longer and more dynamic, this limitation was made for practical reasons. Despite the brevity, we found that mLLMs still struggle with maintaining coherence and understanding conversational flow, underscoring the challenges these models face, even in simpler dialogue settings.

Acknowledgments

We would like to thank the workers and reviewers whose contributions were essential in creating CAPTex. Their dedication in annotating data and providing valuable feedback greatly enhanced the quality of this work.

References

Aida Amini, Antoine Bosselut, Bhavana Dalvi Mishra, Yejin Choi, and Hannaneh Hajishirzi. 2020. Procedural reading comprehension with attribute-aware context flow. Preprint, arXiv:2003.13878.

Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. 2022. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 700–710, Dublin, Ireland. Association for Computational Linguistics.

Yong Cao, Yova Kementchedjhieva, Ruixiang Cui, Antonia Karamolegkou, Li Zhou, Megan Dare, Lucia Donatelli, and Daniel Hershcovich. 2024. Cultural adaptation of recipes. Transactions of the Association for Computational Linguistics, 12:80–99.

Courtney Corley and Rada Mihalcea. 2005. Measuring the semantic similarity of texts. In Proceedings of the ACL Workshop on Empirical Modeling of Semantic Equivalence and Entailment, pages 13–18, Ann Arbor, Michigan. Association for Computational Linguistics.

Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gašević, and Guanliang Chen. 2023. Can large language models provide feedback to students? a case study on chatgpt. In 2023 IEEE International Conference on Advanced Learning Technologies (ICALT), pages 323–325. IEEE.

Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1595–1604, New Orleans, Louisiana. Association for Computational Linguistics.
John Dang, Shivalka Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Virata Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Aycr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Venkitesh, David Cairuz, Bowen Yang, Tim Chung, Wei-Yin Ko, Sylvie Shang Shi, Amir Shukayev, Sammie Bae, Aleksandra Piktus, Roman Castagné, Felipe Cruz-Salinas, Eddie Kim, Lucas Crawhall-Stein, Adrien Morisot, Sudip Roy, Phil Blunsom, Ivan Zhang, Aidan Gomez, Nick Frosst, Marzieh Fadaee, Beyza Ermis, Ahmet Üstün, and Sara Hooker. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. Preprint, arXiv:2412.04261.

DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kejin Huang, Kuai Yu, Leand Wang, Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leiye Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingqing Li, Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuan Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyin Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyu Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yu Yu, Yichao Zhang, Yifan Yin, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yu-jia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyiu Zhang, and Zhen Zhang. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. Preprint, arXiv:2501.12948.

Esin Durmus, Karina Nguyen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, et al. 2024. Towards measuring the representation of subjective global opinions in language models. In First Conference on Language Modeling.

Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, and Jack Clark. 2022. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 1747–1764, New York, NY, USA. Association for Computing Machinery.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schellen, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aoob Yang, Archi Mitra, Archie Sravanakumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alionsius, Daniel Song, Danielle Pintz, Danny Livshits, Danny Wyatt, David Esioubi, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Francisco Guzmán, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocc, Joshua Johnstun, Joshua Saxe, Jun-teng Jia, Kalyan Vasudevan Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth
Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam-badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Ning Zhang, Olivier Duchenne, Onur Celebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vace, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raleanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ryan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthi, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Van denhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Taker Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vitor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyu Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yunjing Mao, Zacharie Delpierre Couder, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambhe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Chingsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Cynthia Gao, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcuate, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hakan Inan, Hamid Shojanaz-eri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leonardits, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, Jean-Baptiste Gay, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihiir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Melhta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavav Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Prithish Yuvaraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Ranaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuvigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Koehler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun
Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihăilescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaoqian Wu, Xiaolan Wang, Xilun Wu, Xinbo Gao, Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Albert Gu and Tri Dao. 2024. Mamba: Linear-time sequence modeling with selective state spaces. Preprint, arXiv:2312.00752.

Aditya Gupta and Greg Durrett. 2019. Tracking discrete and continuous entity state for process understanding. In Proceedings of the Third Workshop on Structured Prediction for NLP, pages 7–12, Minneapolis, Minnesota. Association for Computational Linguistics.

Hao Huang, Xiubo Geng, Jian Pei, Guodong Long, and Daxin Jiang. 2021. Reasoning over entity-action-location graph for procedural text understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5100–5109, Online. Association for Computational Linguistics.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. 2024. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901.

Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Computational Linguistics.

Nora Kassner and Hinrich Schütze. 2020. Negated and misprised probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics.

M. G. Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93.

Thomas Kosch and Sebastian Feger. 2024. Risk or chance? large language models and reproducibility in hci research. Interactions, 31(6):44–49.

Fajri Koto, Haonan Li, Sara Shatnawi, Jad Doughman, Abdelrahman Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin. 2024a. ArabicMMU: Assessing massive multitask language understanding in Arabic. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5622–5640, Bangkok, Thailand. Association for Computational Linguistics.

Fajri Koto, Rahmad Mahendra, Nurul Aisyah, and Timothy Baldwin. 2024b. IndoCulture: Exploring geographically influenced cultural commonsense reasoning across eleven Indonesian provinces. Transactions of the Association for Computational Linguistics, 12:1703–1719.

Vladimir I. Levenshtein. 1965. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics. Doklady, 10:707–710.

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuhui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019–9052, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ao Liu, Shuai Yuan, Chenbin Zhang, Congjian Luo, Yaqing Liao, Kun Bai, and Zenglin Xu. 2020. Multilevel multimodal transformer network for multimodal recipe comprehension. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 1781–1784, New York, NY, USA. Association for Computing Machinery.

Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Haley Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5356–5371, Online. Association for Computational Linguistics.

Tarek Naous, Michael J Ryan, Alan Ritter, and Wei Xu. 2024. Having beer after prayer? measuring cultural bias in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16366–16393, Bangkok, Thailand. Association for Computational Linguistics.

OpenAI. 2025. Openai o3-mini system card.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Allemann, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgium, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madeleine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Justin Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Hee-woo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondratiuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emry Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroksy Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillett, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jeremy Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayaragha, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weimann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Junting Zhuang, William Zhuk, and Barrett Zoph. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.

Makbulé Gulcin Ozsoy. 2024. Multilingual prompts in llm-based recommenders: Performance across languages. Preprint, arXiv:2409.07604.

Vinodkumar Prabhakaran, Rida Qadri, and Ben Hutchinson. 2022. Cultural incongruencies in artificial intelligence. Preprint, arXiv:2211.13069.

Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Kepin Bao, Kexin Yang, Le Yu, Mei Li,
Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Hossein Rajab Faghihi and Parisa Kordjamshidi. 2021. Time-stamped language model: Teaching language models to understand the flow of events. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4560–4570, Online. Association for Computational Linguistics.

Hanoona Rasheed, Muhammad Maaz, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M Anwer, Tim Baldwin, Michael Felsberg, and Fahad S Khan. 2025. Palo: A polyglot large multimodal model for 5b people. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1745–1754. IEEE.

Jingyuan S. She, Christopher Potts, Samuel R. Bowman, and Atticus Geiger. 2023. ScoNe: Benchmarking negation reasoning in language models with fine-tuning and in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1803–1821, Toronto, Canada. Association for Computational Linguistics.

Huangjun Shen, Liangying Shao, Wenbo Li, Zhibin Lan, Zhanyu Liu, and Jinsong Su. 2024a. A survey on multi-modal machine translation: Tasks, methods and challenges. Preprint, arXiv:2405.12669.

Lingfeng Shen, Weiting Tan, Siho Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024b. The language barrier: Dissecting safety challenges of LLMs in multilingual contexts. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2668–2680, Bangkok, Thailand. Association for Computational Linguistics.

C. Spearman. 1904. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72–101.

Margaret S. Steffensen, Chitra Joag-Dev, and Richard C. Anderson. 1979. A cross-cultural perspective on reading comprehension. Reading Research Quarterly, 15(1):10–29.

Jizhi Tang, Yansong Feng, and Dongyan Zhao. 2020. Understanding procedural text using interactive entity networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7281–7290, Online. Association for Computational Linguistics.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Taffi, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbisson, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Letícia Lago, Lilly McNealus, Livio Baldino Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidson, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwook Park, Mofi Rahman, Mohit Khatwani, Natalie Diao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardesher Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotrutz, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kennedy, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.
Thinh Hung Truong, Timothy Baldwin, Karin Verspoor, and Trevor Cohn. 2023. Language models are not naysayers: an analysis of language models on negation benchmarks. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 101–114, Toronto, Canada. Association for Computational Linguistics.

Man-Fai Wong, Shangxin Guo, Ching-Nam Hang, Siu-Wai Ho, and Chee-Wei Tan. 2023. Natural language generation and understanding of big code for ai-assisted programming: A review. Entropy, 25(6).

Te-Lin Wu, Alex Spangher, Pegah Alipoormolabashi, Marjorie Freedman, Ralph Weischel, and Nanyun Peng. 2022. Understanding multimodal procedural knowledge by sequencing multimodal instructional manuals. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4525–4542, Dublin, Ireland. Association for Computational Linguistics.

Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. Stateflow: Enhancing llm task-solving through state-driven workflows. In NeurIPS 2024 Workshop on Open-World Agents.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.

A Dataset

A.1 Detailed Categories

1. Food and Cuisine: Captures cultural practices related to the preparation and consumption of food.

2. Celebrations and Festivals: Represents rituals and activities associated with cultural events and festivities.

3. Social Etiquette and Hospitality: Reflects societal norms and traditions in interpersonal interactions.

4. Craftsmanship and Artisan Skills: Showcases traditional methods of creating culturally significant artifacts.

5. Traditional Attire and Dress: Highlights practices involving culturally significant clothing and adornments.

6. Agricultural and Seasonal Practices: Documents cultural procedures tied to agricultural cycles and seasonal changes.

7. Religious and Spiritual Practices: Encapsulates practices integral to religious and spiritual traditions.

8. Life Milestones and Family Rites: Represents cultural customs marking significant life events.

9. Sports, Games, and Competitions: Covers traditional recreational and competitive activities.

10. Environmental and Nature-Based Practices: Focuses on cultural interactions with and stewardship of the natural environment.

A.2 Procedures Collection

In the initial phase, we conducted a pilot study using Prolific 5 to recruit workers and gather some procedures. However, upon evaluating the quality of the collected procedures, we determined that Prolific was unsuitable for this specific task. Consequently, we opted to directly engage reliable workers who met our stringent requirements. Native speakers of the target languages were responsible for crafting each procedure in their native language. These were then precisely translated into English, ensuring both versions accurately conveyed the same content. Each procedure consisted of five to ten sequential steps, where the order was crucial for the proper understanding and execution of the tasks. To preserve the dataset’s authenticity, workers were strictly prohibited from using AI-based text generation tools, which could introduce inaccuracies or fabrications, thereby compromising the reliability of the procedures. We implemented a two-step quality control process, comprising automated checks and peer evaluations through cross-verification. An all-inclusive checklist was employed to assess conceptual precision, cultural authenticity, logical flow, the necessity of maintaining step sequence, grammatical accuracy, and adherence to the required number of steps. This rigorous process ensured that both the native language and English versions met our high standards.

A.3 MCQ Design and Structure

As previously outlined, four distinct categories of questions are systematically generated using Python code. These questions adhere to the following templates for inquiries in English:

• Subsequent Affirmative: In the procedure “Procedure Name”, what is the next step after: “Reference Step”?

5http://prolific.com/
• Subsequent Negative: In the procedure “Procedure Name”, which one is not the step before: “Reference Step”?

• Antecedent Affirmative: In the procedure “Procedure Name”, which step must be completed before: “Reference Step”?

• Antecedent Negative: In the procedure “Procedure Name”, which step does not come after: “Reference Step”?

These templates ensure clarity and consistency in question generation, aligning with the procedural framework.

A.4 Conversation Generation

The following prompt is used to generate conversations using GPT-4o: Create a short conversation between two people, Person A and Person B, based on the following procedure. The conversation should begin with (Have you heard about the “Procedure Name” from “Country”? ) and progress naturally, with each message reflecting a clear and logical flow of ideas related to the steps of the procedure. The conversation should consist of four messages. In the third message, Person A asks a question about the next step in the procedure. In the last message, Person B should respond according to the procedure’s step, providing a clear answer or action related to the next step, and explicitly mention the step number.

Example Structure:
Person A: Have you heard about the “Procedure Name” from “Country”?
Person B: (Response introducing the procedure and discussing some of the first steps)
Person A: (Asks a follow-up question to clarify the next step)
Person B: (Explains the first next step in the procedure according to the procedure’s step)
Next step: (number of next step)

IMPORTANT: Ensure the generated conversation adheres to the above structure. The last message should always be "Next step: (number)", where (number) is the next step in the procedure.

The procedure is as follows:
“Procedure Steps”

Given the suboptimal performance of GPT-4o in processing lower-resource languages, this section is exclusively dedicated to generating conversations in English.

The generated conversations are subjected to a two-phase verification process. The initial phase involves automated verification using a Python script, ensuring that each conversation comprises exactly four utterances. The subsequent phase entails thorough evaluation by qualified human annotators. These annotators assess each dialogue against a detailed checklist, which evaluates conceptual accuracy, grammatical precision, and alignment of Person B’s responses with the procedural steps. During the review process, two types of errors were identified: 76 conversations had an incorrect next step number, and 45 conversations included explanations of multiple steps in the final utterance.

B Additional Results

Figure 8 presents the aggregated performance6 of the models across all four tasks.

Language Effects on Performance As shown in Figure 6, the results for GPT-4o, the top-performing model, are presented in a manner similar to those for Qwen2.5-14B-Instruct in Figure 3. GPT-4o generally performs better in English across most countries, with the exceptions of China and Indonesia. A comparison of the two figures reveals that GPT-4o maintains a more balanced performance between English and native languages in most countries. While GPT-4o tends to perform better in English, reflecting its extensive training on high-resource languages, it displays a smaller performance gap in low-resource languages such as Hausa (Nigeria) and Urdu (Pakistan). In contrast, Qwen2.5-14B-Instruct shows a more pronounced decline in native languages, indicating that GPT-4o may possess superior cross-lingual capabilities and stronger support for languages with limited resources. This gives GPT-4o a distinct advantage in multilingual contexts, while Qwen2.5-14B-Instruct shows a stronger preference for English, particularly in regions with fewer linguistic resources.

Performance Across Cultural Dimensions The aggregated performance scores of GPT-4o, the top-performing model, are displayed in Figure 7 for all evaluation tasks. Our analysis reveals that GPT-4o’s understanding of procedural texts differs

6Scores are normalized and computed using a weighted sum approach, with the following weight distribution: 0.3 for reordering, 0.1 for each metric, 0.2 for PB-MCQ, 0.2 for CB-MCQ, and 0.3 for CB-QA.
Figure 6: Language impact on GPT-4o performance

<table>
  <tr>
    <th></th>
    <th>China</th>
    <th>India</th>
    <th>Indonesia</th>
    <th>Iran</th>
    <th>Japan</th>
    <th>Nigeria</th>
    <th>Pakistan</th>
  </tr>
  <tr>
    <td>Agricultural</td>
    <td>0.67</td>
    <td>0.63</td>
    <td>0.71</td>
    <td>0.51</td>
    <td>0.67</td>
    <td>0.61</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.65</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>Celebrations</td>
    <td>0.78</td>
    <td>0.63</td>
    <td>0.71</td>
    <td>0.51</td>
    <td>0.67</td>
    <td>0.61</td>
    <td>0.68</td>
    <td>0.63</td>
    <td>0.70</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>Craftsmanship</td>
    <td>0.64</td>
    <td>0.69</td>
    <td>0.77</td>
    <td>0.58</td>
    <td>0.70</td>
    <td>0.71</td>
    <td>0.70</td>
    <td>0.72</td>
    <td>0.76</td>
    <td>0.61</td>
  </tr>
  <tr>
    <td>Environmental</td>
    <td>0.73</td>
    <td>0.68</td>
    <td>0.76</td>
    <td>0.78</td>
    <td>0.73</td>
    <td>0.76</td>
    <td>0.78</td>
    <td>0.69</td>
    <td>0.73</td>
    <td>0.65</td>
  </tr>
  <tr>
    <td>Food</td>
    <td>0.58</td>
    <td>0.71</td>
    <td>0.73</td>
    <td>0.56</td>
    <td>0.60</td>
    <td>0.72</td>
    <td>0.69</td>
    <td>0.71</td>
    <td>0.67</td>
    <td>0.71</td>
  </tr>
  <tr>
    <td>Life Milestones</td>
    <td>0.73</td>
    <td>0.59</td>
    <td>0.68</td>
    <td>0.64</td>
    <td>0.67</td>
    <td>0.61</td>
    <td>0.65</td>
    <td>0.60</td>
    <td>0.71</td>
    <td>0.73</td>
  </tr>
  <tr>
    <td>Religious</td>
    <td>0.71</td>
    <td>0.45</td>
    <td>0.73</td>
    <td>0.69</td>
    <td>0.75</td>
    <td>0.61</td>
    <td>0.69</td>
    <td>0.54</td>
    <td>0.71</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Social Etiquette</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Sports and Game</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Attire and Dress</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Figure 7: Cultural dimension performance by country

across cultural contexts. Specifically, the model demonstrates robust familiarity with Indian agricultural practices, while its performance on Japanese agricultural topics is comparatively weaker. In contrast, GPT-4o excels in encoding Japanese cultural knowledge related to celebrations and festivals, surpassing its representations of other cultures. In the culinary domain, Pakistani cuisine is more accurately captured, whereas Iranian religious practices are more prominently reflected. Furthermore, the model effectively encapsulates Indonesian social etiquette, highlighting the diversity in how various cultural elements are represented within the model.
Figure 8: Performance of LLMs by model size

<table>
  <tr>
    <th rowspan="2">Category</th>
    <th colspan="2">Procedures (English/Native)</th>
    <th colspan="3">MCQs (English/Native)</th>
    <th colspan="4">Conversations (Utterances)</th>
  </tr>
  <tr>
    <th>Question</th>
    <th>Correct Answer</th>
    <th>Wrong Answer</th>
    <th>First</th>
    <th>Second</th>
    <th>Third</th>
    <th>Fourth</th>
  </tr>
  <tr>
    <td>Agricultural and Seasonal Procedures</td>
    <td>94.6 / 104.1</td>
    <td>26.8 / 32.7</td>
    <td>13.6 / 14.6</td>
    <td>13.4 / 14.3</td>
    <td>10.4</td>
    <td>41.1</td>
    <td>12.7</td>
    <td>28.5</td>
  </tr>
  <tr>
    <td>Celebrations and Festivals</td>
    <td>118.2 / 124.2</td>
    <td>31.7 / 35.8</td>
    <td>17.4 / 17.9</td>
    <td>17.8 / 18.2</td>
    <td>11.0</td>
    <td>43.3</td>
    <td>12.5</td>
    <td>29.0</td>
  </tr>
  <tr>
    <td>Craftsmanship and Artisan Skills</td>
    <td>132.4 / 146.5</td>
    <td>32.6 / 39.0</td>
    <td>18.6 / 20.5</td>
    <td>18.7 / 20.3</td>
    <td>10.3</td>
    <td>45.0</td>
    <td>13.2</td>
    <td>30.6</td>
  </tr>
  <tr>
    <td>Environmental and Nature-Based Procedures</td>
    <td>93.3 / 108.8</td>
    <td>29.1 / 36.0</td>
    <td>14.2 / 16.3</td>
    <td>15.3 / 17.4</td>
    <td>11.3</td>
    <td>42.5</td>
    <td>12.9</td>
    <td>28.8</td>
  </tr>
  <tr>
    <td>Food and Cuisine</td>
    <td>137.8 / 139.9</td>
    <td>32.2 / 36.3</td>
    <td>18.2 / 18.6</td>
    <td>18.5 / 18.5</td>
    <td>10.7</td>
    <td>43.2</td>
    <td>13.1</td>
    <td>28.6</td>
  </tr>
  <tr>
    <td>Life Milestones and Family Rites</td>
    <td>132.3 / 137.0</td>
    <td>32.8 / 38.7</td>
    <td>18.0 / 18.0</td>
    <td>18.7 / 19.3</td>
    <td>11.7</td>
    <td>43.3</td>
    <td>12.7</td>
    <td>29.5</td>
  </tr>
  <tr>
    <td>Religious and Spiritual Practices</td>
    <td>105.1 / 109.7</td>
    <td>30.4 / 34.4</td>
    <td>15.2 / 15.3</td>
    <td>15.7 / 16.0</td>
    <td>12.1</td>
    <td>41.4</td>
    <td>12.5</td>
    <td>28.0</td>
  </tr>
  <tr>
    <td>Social Etiquette and Hospitality</td>
    <td>115.0 / 125.2</td>
    <td>32.8 / 38.7</td>
    <td>17.2 / 18.6</td>
    <td>18.5 / 19.7</td>
    <td>13.0</td>
    <td>43.5</td>
    <td>13.3</td>
    <td>29.6</td>
  </tr>
  <tr>
    <td>Sports, Games, and Competitions</td>
    <td>112.2 / 122.1</td>
    <td>30.5 / 36.0</td>
    <td>16.1 / 16.9</td>
    <td>16.9 / 18.3</td>
    <td>10.5</td>
    <td>42.6</td>
    <td>13.6</td>
    <td>28.2</td>
  </tr>
  <tr>
    <td>Traditional Attire and Dress</td>
    <td>132.5 / 131.8</td>
    <td>32.7 / 36.2</td>
    <td>18.0 / 17.9</td>
    <td>18.4 / 18.1</td>
    <td>11.6</td>
    <td>43.6</td>
    <td>12.9</td>
    <td>31.1</td>
  </tr>
</table>

Table 6: Average word counts for CAPTex components (Procedures, MCQs, and Conversations) by category
<table>
  <tr>
    <th>Country</th>
    <th>Category</th>
    <th>Language</th>
    <th>Type</th>
    <th>Question</th>
    <th>Choices</th>
  </tr>
  <tr>
    <td>Japan</td>
    <td>Food and Cuisine</td>
    <td>Japanese</td>
    <td>SA</td>
    <td>手順 '昆布巻きの作り方（4人前）'では、「鍋に6と5の戻し汁を入れ、中火で沸騰させる。」の後の次のステップは何ですか？</td>
    <td>A. 1を昆布と戻し汁に分ける。昆布を端からしっかり巻き付けて爪楊枝で留める。<br>B. 4を巻き付け、きつくるんしたら爪楊枝を抜き、余分なかんぴょうを切る。<br>C. 醤油大さじ1.5、顆粒和風だし大さじ1、料理酒大さじ1、みりん大さじ2、砂糖大さじ2、塩小さじ1/2を加える。<br>D. 塩を洗い流し、水気を切ったら20cm幅に切り、12本にする。</td>
  </tr>
  <tr>
    <td>Japan</td>
    <td>Food and Cuisine</td>
    <td>English</td>
    <td>SA</td>
    <td>In the procedure 'How to Make Kelp Rolls (for 4 Persons)', what is the next step after: 'Place the rolls into a saucepan and add the stock. Bring to a boil over medium heat.'?</td>
    <td>A. Separate the kelp and stock.<br>Wrap the kelp tightly around the filling and secure it with a toothpick.<br>B. Wrap a piece of kanpyo around the roll and tie it tightly. Remove the toothpick and trim any excess kanpyo.<br>C. Add 1.5 tbsp soy sauce, 1 tbsp granulated Japanese soup stock, 1 tbsp cooking sake, 2 tbsp mirin, 2 tbsp sugar, and 1/2 tsp salt.<br>D. Rinse off the salt, drain, and cut the kanpyo into 20 cm lengths, then divide into 12 pieces.</td>
  </tr>
</table>

Table 7: Example of MCQs
<table>
  <tr>
    <th>Country</th>
    <th>Category</th>
    <th>Conversation</th>
  </tr>
  <tr>
    <td>Iran</td>
    <td>Sports, Games, and Competitions</td>
    <td>
      Person A: Have you heard about the Alak-Dolak Game from Iran?<br>
      Person B: Yes, it’s quite an interesting traditional game! It starts with forming two teams, and they draw lots to see which team gets to go first. Each team also has a leader who plays a crucial role at the beginning. For instance, the leader places two stones and then positions the "Alak" stick on top of them while holding the "Dolak" stick in their hand.<br>
      Person A: That’s fascinating! What does the leader do next after placing the Alak stick on the stones?<br>
      Person B: After placing the Alak stick, the leader uses the Dolak stick by positioning its tip under the Alak stick. Then they move the Dolak stick left and right to launch the Alak stick into the air. Next step: 3.
    </td>
  </tr>
  <tr>
    <td>Nigeria</td>
    <td>Celebrations and Festivals</td>
    <td>
      Person A: Have you heard about the Osun-Osogbo Festival from Nigeria?<br>
      Person B: Yes, I have! It’s a fascinating cultural event that spans two weeks, starting with cleansing rituals called Iwopopo. The festival begins by preparing the Arugba, a virgin maiden who carries a sacred calabash. She plays a vital role as she represents purity and connects the people with the goddess Osun. It’s all part of cleansing the community of evil spirits.<br>
      Person A: That sounds amazing. What happens after the cleansing rituals?<br>
      Person B: After the cleansing, traditional Yoruba music, drumming, and dances are performed daily, highlighting the community’s rich cultural heritage. Craftsmen and vendors set up in the marketplace near the sacred Osun Grove to display arts, crafts, and local foods.<br>
      Next step: 2
    </td>
  </tr>
</table>

Table 8: Examples of a procedurally grounded conversation

Reviewer paper summaries: This paper introduces CAPTex, a benchmark for evaluating reasoning over procedural texts form different cultures (operationalized as countries) in multiple languages. They ask native speakers from the respective countries to write procedural text (list of steps) describing procedures for 10 categories of tasks (eg recipes, celebration of festivals, etc). They then evaluate a large number of models for 4 different settings of reasoning over the procedural texts using multiple evaluation metrics.

The core purpose of this paper is to benchmark LLMs' capability to correctly understand the ordering of procedural instructions in multilingual instructional documents through both multiple-choice questions and conversational contexts.

To assess performance on these tasks, the authors propose a novel benchmark named CAPTex (Culturally-Aware Procedural Texts), covering culturally specific procedural texts from diverse regions, including China, India, Indonesia, Iran, Japan, Nigeria, and Pakistan. For each language, procedural texts, multiple-choice questions, and conversational exchanges are provided in both native languages and their English translations.

Using the CAPTex benchmark, this research conducted experiments on four distinct tasks—Step Reordering, PB-MCQ, CB-MCQ, and CB-QA—across 31 models, including GPT-4, Qwen2.5, Llama-3, and Mistral. The results show GPT-4 performing best overall across procedural reordering, multiple-choice questions, and conversational generation tasks. Performance generally improved with English prompts, while lower-resource languages showed reduced accuracy. Models fine-tuned with Instruct methods consistently outperformed their base versions. Among various open-source mLLMs, Qwen2.5-14B-Instruct showed superiority in multiple-choice tasks, whereas Gemma-2-9b excelled at reordering tasks, revealing model-specific strengths. Detailed results showed significant variability depending on region and cultural domain, highlighting the necessity for future research expansion.
Summary of strengths: - Tackles a unique task, understanding and reasoning over procedural texts that are culturally contextual. This task is novel, and to the best of my knowledge does not have existing data resources for evaluation. At the same time, the motivation behind culturally contextual reasoning over procedural  text makes sense because processes across cultures (even for similar tasks) might be different. Overall, this an interesting and novel task for evaluation.
    
- Contribute a dataset of procedural texts from 7 cultures written in English and in one language of the culture. The dataset collection process is well thought out and has rigorous quality control.
    
- Interesting (and sound) setting of evaluation of reasoning over the collected procedural texts. Multiple settings are used including reordering, MCQ, and conversational setting. The metrics used for the settings are  sound.
    
- Evaluation for a large number of open-weight model and variants from within a model family.

Unlike benchmarks primarily focusing on general question-answering or common knowledge, this paper addresses culturally contextualized, step-by-step procedural tasks deeply rooted in local traditions.

The dataset's completeness and quality are strong points, including texts authored by native speakers, mutual cross-checking for accuracy, and diverse approaches to constructing multiple-choice questions.

The authors emphasize diversity throughout the research, including data type selection, task design, and model choices. They also provide detailed experimental analyses, clearly demonstrating areas of model failure by comparing performance across different question types.
Additionally, the paper offers specific future research directions, such as multimodal extensions, constructing longer conversational data, and incorporating a wider variety of cultures.
Summary of weaknesses: Limitations: 

- It is unclear how much of the model’s failure can be explained by failure to understand cultural context, limitation because of the model's performance in the language (esp for low-resource languages), or simple reasoning failure for (any) procedural tasks. It will be useful add ablations/control experiments to delineate this. Alternatively, connect the model performances obtained here to results in literature about these model’s performances on other procedural reasoning datasets and/or other tasks that require cultural context.
    
- Some discrepancies/lack of clarity in aggregation of reported results -- see questions below.
    
- No discussion section about the implications of the results. No discussion/ablations to understand why the results are a certain way (for eg control experiments as described in 1)
    

Questions about results and aggregation:

- What does lines 455-456 mean? The procedures are written in English and the respective language, lines 455-456 suggest that the zero-shot evaluation is only english prompts?  But figure 3 seems to suggest that both the native language data and English data were used in evaluation?
    
- What does lines 517-519 mean? Why is there a reweighting of task-specific metric scores when computing a aggregated score?
    
- Why is the weighting scheme of footnote 4 (aggregation for fig 3) and footnote 5 (aggregation for fig 4) different? How was this weighting scheme decided?
    
- Why are the results in figure 3 limited to aggregating results of only two tasks instead of all tasks like fig 4?
    
- Is table 4 result averaged across categories, cultures, and procedures? Does this use simple averaging or some reweighting aggregation (like done for fig 3, 4)?
    
- How are the results for figure 5 aggregated? Is it simply the average <metric> for procedures of all cultures and all topics and both english and native language with that specific step count?
    

On the significance of results:

- Some of the results in table 3 seem  close to random baseline -- adding significance testing to understanding whether the model performances are indeed different from random, or whether the best performing model is indeed better is important.
    
- In MCQ setting, it would be important to ensure that there are no artifacts in the data (eg correct ans always being at option 3) or in model responses (such as model always selecting the first option out of the four) -- this can be done in many ways, for instance by running the evaluation multiple times with different option ordering or by ensuring that option ordering is random.
    

Reproducibility:

- Full evaluation prompts for all the types of tasks should be reported in appendix. This should include details about how the model was instructed to output the answer for a specific task, how the procedural texts were passed (how was the text being larger than model context size handled, if it happened), system prompts and instructions, etc?

- How were the responses from zero-shot evaluation parsed and was there any human-verification to ensure that the parsing did not introduce any artifacts in the results (for eg, model outputting option number 1/2/3/4 while the parsing of expects an option letter a/b/c/d leading to bad performance metric; I would expect that non-instruct models are more prone to this type of issues because they are not instruction tuned and hence struggle with following the instructions of performing the task or output formatting -- for example, instead of outputting an option they will continue generating more options).
    
- How many prompt templates were used for the zero-short evaluation? How many responses were generated per prompt (one or more, if more than one, how were they aggregated?) What were the hyperparameter settings of the generation (for eg temperature)?
    
- Main text has results with a lot of aggregation (for example, across cultures, languages, domains, etc), which is understandable, but the disaggregated results should be reported in the appendix.

The paper is well-structured, with no significant flaws, from research questions to methods and performance analyses. However, there are certain areas in the interpretation of results and the explanation of contributions that require further refinement:

While four research questions were outlined in the introduction, the first three were adequately addressed, but deeper discussion was lacking for the last question, "Are there noticeable strengths or weaknesses in LLMs' understanding depending on the cultural context of the procedural text?"

Although the paper aims to uncover "Cultural Blind Spots," it does not clearly articulate precisely what weaknesses multilingual LLMs have when processing multilingual procedural texts.

When analyzing performance results across LLMs, the paper notes that Qwen2.5-14B performs best. However, considering this model's relatively large size and extensive training primarily in Chinese and English texts, its superior performance is somewhat predictable. A more detailed analysis considering model size and training methodology would be necessary.

On page 8, a formula is presented in the footnote assigning different weights to the five tasks to calculate a composite score. A rationale or explanation for choosing these specific weights is needed.

When analyzing factors influencing performance evaluation, additional factors such as differences in step count per language (shown in Figure 2) and differences in LLM parameter sizes should be examined, in addition to the already mentioned factors like language differences, text length, and domain variations.
Comments, suggestions and typos: - Please consider including a datasheet in the appendix that details the data collection procedure more descriptively. (ref: https://dl.acm.org/doi/10.1145/3458723; see https://github.com/AudreyBeard/Datasheets-for-Datasets-Template for overleaf friendly template)
    
- The language categorization (high/low-resource) from Joshi et al (from 2020) might no longer be reflective of the resource availability for these languages overall or in the training datasets of the models tested in 2025. Considering reporting (as available) resource availability statistics of these languages from more recent training corpora or from the technical reports of the models tested.
    
- It is unclear the level of granularity of the topic/category provided to the workers when writing the procedures. For example, for the category of recipes, was the worker given a specific dish that they had to write a procedure for or could they pick any recipe that they thought was appropriate for the culture?
    
- Missing arrow (expectation) next metrics PB-MCQ, CB-MCQ, and CB-QA in Table 4. NIT: this table is a wall of numbers that is not very informative in terms of interesting insights other than reflecting that a large number of models and model variants were tested. Consider using visualization that is more clear and easier to read in terms of what we are looking at.

- Consider adding more insights (in appendix) on how the difference of the task setup affects model performance.
    
- There could be spectrum in the aspects of adapting procedural texts for cultures. On one end of the spectrum, there are procedures where the outcome of the procedure is the same. For example, in a procedure on cooking plain rice, someone from India might use a pressure cooker while someone in the US might use an instapot. The procedure to cook rice (a common goal) can then be adjusted for cultural context (eg use of pressure cooker vs instapot). On the other the hand, there might be procedures where the outcome itself are very different. For example, the funeral in some cultures might involve cremation while those in another might involve a burial. Despite a common higher level goal, i.e “funeral”, the overall procedures of these two cultural contexts are very different. It appears that CAPTex probably covers a range on this spectrum, because there is everything from recipes to festivals in the taxonomy of the data collected. It would be good to see a discussion on teasing apart the level of differences in the procedures written for different cultures. A simplest metric could be -- pair-wise set overlap of procedure steps for every pair of cultures. Then, in the example above -- the rice-cooking procedures for India and US will overlap more than the funeral procedures. It would be interesting to see disaggregated results of model performance depending on whether certain procedures are more or less similar across cultures.

It is necessary to enrich the existing benchmark data by incorporating more open-ended descriptive QA data. This will provide a more precise measurement of LLMs' capability to handle cultural contexts. Additionally, a deeper error analysis is recommended to enhance the discussion of cultural context processing abilities. By specifically presenting examples of incorrect step reorderings or erroneous QA responses, future research can identify what cultural aspects LLMs misunderstand or omit. Diagnosing the causes behind task failures and proposing solutions based on this analysis would significantly increase the paper's completeness.

The completeness of sentences throughout the document was high, so I couldn't find any special edits.
Overall assessment scores: 3.5

4