M-IFEval: On Multilingual Instruction-Following Capability of Large Language Models
Authors: Bo Zeng, Chenyang Lyu, Sinuo Liu, Mingyan Zeng, Minghao Wu, Xuanfan Ni, Tianqi Shi, Yu Zhao, YEFENG LIU, Chenyu Zhu, Ruizhe Li, Jiahui Geng, Qing Li, Yu Tong, Longyue Wang, Weihua Luo, Kaifu Zhang
Accepted at: acl25

Marco-Bench-MIF: On Multilingual Instruction-Following Capability of Large Language Models

Bo Zeng1, Chenyang Lyu1*, Sinuo Liu1, Mingyan Zeng, Minghao Wu1, Xuanfan Ni1, Tianqi Shi1, Yu Zhao1, Yefeng Liu1, Chenyu Zhu1, Ruizhe Li2, Jahui Geng3, Qing Li3, Yu Tong, Longyue Wang1, Weihua Luo1, Kaifu Zhang1

1Alibaba International Digital Commerce, 2University of Aberdeen, 3MBZUAI

Abstract
Instruction-following capability has become a major ability to be evaluated for Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; Bai et al., 2023). However, existing datasets, such as IFEval (Zhou et al., 2023; Zeng et al., 2024), are either predominantly monolingual and centered on English or simply machine translated to other languages, limiting their applicability in multilingual contexts. In this paper, we present an carefully-curated extension of IFEval to a localized multilingual version named Marco–Bench–MIF, covering 30 languages with varying levels of localization. Our benchmark addresses linguistic constraints (e.g., modifying capitalization requirements for Chinese) and cultural references (e.g., substituting region-specific company names in prompts) via a hybrid pipeline combining translation with verification. Through comprehensive evaluation of 20+ LLMs on our Marco–Bench–MIF, we found that: (1) 25-35% accuracy gap between high/low-resource languages, (2) model scales largely impact performance by 45-60% yet persists script-specific challenges, and (3) machine-translated data underestimates accuracy by 7-22% versus localized data. Our analysis identifies challenges in multilingual instruction following, including keyword consistency preservation and compositional constraint adherence across languages. Our Marco–Bench–MIF will be made publicly available to the community.

1 Introduction
Instruction-following datasets play a critical role in evaluating and fine-tuning Large Language Models (LLMs) (Team, 2023; Hurst et al., 2024), enabling them to better align with user intents. Among these, IFEval (Zeng et al., 2024) has emerged as a widely used benchmark for assessing instruction-following capabilities. However, its monolingual nature restricts its utility in multilingual and cross-lingual applications, where language-specific nuances and cultural contexts must be considered. Existing datasets, including IFEval and Multi-IF (Zhou et al., 2023; Zeng et al., 2024), are either predominantly monolingual and centered on English or simply machine-translated to other languages, limiting their applicability in multilingual contexts. Machine-Translated (MT) data often fails to capture the linguistic and cultural subtle differences required for accurate evaluation, leading to underestimation of model performance in multilingual settings.

The construction of a multilingual instruction-following dataset should focus on adaptation of instructions, prompts, and constraints to diverse linguistic and cultural contexts. For example, tasks involving capitalization, such as “change all letters to uppercase” should be modified for non-Latin scripts like Chinese and Greek, where such constraints are not applicable. Also tasks involving specific grammatical structures, such as “use the passive voice in your response”, should be carefully adapted for languages like Japanese and Turkish, where passive constructions differ significantly from English. Similarly, prompts containing culturally specific references should be localized to align better with the target language and culture. For instance, the English prompt “Write a short story about a child celebrating Thanksgiving with their family” can be adapted for the Indian context as “Write a short story about a child celebrating Diwali with their family, including the rituals of lighting diyas and sharing sweets.” This adaptation not only preserves the instruction’s structure but also ensures cultural relevance and linguistic appropriateness.

To address this gap, we extend IFEval to a multilingual version, encompassing 30 languages with varying levels of localization. Our benchmark, named Marco–Bench–MIF, is carefully curated to
address linguistic constraints (e.g., modifying capitalization requirements for Chinese) and cultural references (e.g., substituting region-specific company names in prompts). This is achieved through a hybrid pipeline that combines automated translation with two-round human verification to ensure quality and cultural relevance. Our localization process involved a combination of automated and manual methods. Initial translations and adaptations focus on content-specific localization, such as place names, events, and keywords. These outputs were then subjected to rigorous human evaluation and iterative refinement, including two rounds of manual review to ensure quality. Despite these efforts, challenges persisted, particularly in ensuring prompt adherence and accurately translating keywords. For example, in tasks requiring keyword frequency constraints, such as “ensure the word ‘sneaker’ appears at least 10 times”, translated keywords often required manual adjustment to maintain semantic consistency.

Our comprehensive evaluation of 20+ LLMs across 30 languages reveals critical insights into multilingual instruction-following capabilities. We observe that instruction-level accuracy consistently surpasses prompt-level metrics by 10-20% across all models, highlighting persistent challenges in compositional instruction adherence. While model scale strongly correlates with performance (70B+ models achieve 45-60% higher accuracy than 8B counterparts), even smaller models like Qwen2.5-7B attain 64.42% strict instruction-level accuracy. Proprietary models (GPT-4o (Hurst et al., 2024), Claude3.5-Sonnet (Anthropic, 2024)) significantly outperform open LLMs, with a 25-35% accuracy gap in low-resource languages. The evaluation exposes substantial cross-lingual disparities: performance on high-resource languages (de, zh) is 75-85% accuracy versus 50-60% for low-resource languages (yo, ne), with script-specific challenges for some languages like Arabic (ar). Our analysis of localized versus machine-translated data demonstrates 7–22% performance underestimation using translated data, showing the necessity for culturally-grounded evaluations. Our contributions in this paper are summarised as follows:

1. We extend IFEval to a multilingual benchmark, Marco-Bench-MIF, covering 30 languages with fine-grained localization to account for linguistic and cultural diversity.

2. We propose a systematic framework for adapting prompts, instructions, and constraints to diverse linguistic contexts, combining automated and manual localization methods, and analyze the challenges encountered during this process.

3. We conduct a comprehensive evaluation of 20+ LLMs on Marco-Bench-MIF, providing insights into multilingual instruction-following capabilities, including the impact of model scale, resource availability, and the limitations of machine-translated data.

We believe this work represents a significant step forward in enabling robust and culturally-aware evaluation of LLMs instruction following ability across languages.

2 Related Work

Instruction-following datasets have become essential for evaluating LLMs. Early works such as GPT-3 (Brown et al., 2020) and GPT-4 (OpenAI, 2023) demonstrated the importance of this ability. However, existing benchmarks, such as IFEval (Zeng et al., 2024) and Multi-IF (He et al., 2024), are either predominantly monolingual (English) or just machine-translated data, limiting their applicability to multilingual and culturally diverse contexts. Localization and cultural adaptation are key challenges as instructions must align with linguistic and cultural norms. For example, tasks involving capitalization are not relevant for non-Latin scripts like Chinese or Arabic. Cultural and linguistic appropriateness in LLM evaluation has also been explored, studies (Liu et al., 2023; Romero et al., 2024; Myung et al., 2024; Chiu et al., 2024) highlighted the importance of cultural alignment to ensure inclusivity and sensitivity in multilingual and multi-cultural settings. However, few works address the fine-grained challenges of adapting instructions for diverse languages and scripts. Manual intervention and human evaluation remain critical for ensuring quality and accuracy in multilingual datasets. Our work builds on these efforts by extending IFEval to a multilingual benchmark covering 30 languages. Unlike previous datasets, our extension emphasizes fine-grained localization to account for language-specific features and cultural differences. This includes adapting tasks that are not applicable to certain scripts and ensuring instructions are contextually relevant.
3 Marco-Bench-MIF Dataset Construction

We extend the IFEval benchmark into multilingual version through a systematic cross-lingual adaptation and a comprehensive pipeline. The framework consists of three main stages: preprocessing, translation localization, and post-processing. Our approach preserves instruction constraints while accommodating linguistic diversity across 30 languages from 6 major language families (Indo-European, Sino-Tibetan, Afro-Asiatic, etc.). Each language includes 541 instruction-response pairs, carefully constructed to ensure consistency and fidelity.

3.1 Preprocessing

The preprocessing stage involves categorizing and filtering the original IFEval dataset to prepare it for multilingual adaptation.

3.1.1 Constraint Categorization

We categorize the 541 English instruction-response pairs based on two major dimensions:

• Cardinality: 1) Single-Constraint (SC): Instructions with a single requirement. 2) Multi-Constraint (MC): Instructions with two or more interdependent requirements.

• Type: 1) Expressive Constraints (EC): Dictate response structure or format. 2) Content Constraints (CC): Require specific information inclusion.

This categorization ensures a systematic adaptation process, starting with simpler constraints (SC+EC) before handling more complex ones (MC+CC). Table 1 shows the distribution of constraint types in the English benchmark. This graduated approach reduces error propagation risks, with simpler constraints (SC+EC) serving as anchors for more complex cases (MC+CC).

<table>
  <tr>
    <th>Cardinality</th>
    <th>Const. Comp.</th>
    <th>Count</th>
    <th>Prop.</th>
    <th>EC Freq.</th>
    <th>CC Freq.</th>
  </tr>
  <tr>
    <td rowspan="2">SC</td>
    <td>Pure EC</td>
    <td>109</td>
    <td>20.1%</td>
    <td>109</td>
    <td>0</td>
  </tr>
  <tr>
    <td>Pure CC</td>
    <td>161</td>
    <td>29.8%</td>
    <td>0</td>
    <td>161</td>
  </tr>
  <tr>
    <td></td>
    <td>SC Subtotal</td>
    <td>270</td>
    <td>49.9%</td>
    <td>109</td>
    <td>161</td>
  </tr>
  <tr>
    <td rowspan="3">MC</td>
    <td>EC-only</td>
    <td>128</td>
    <td>23.7%</td>
    <td>128</td>
    <td>0</td>
  </tr>
  <tr>
    <td>CC-only</td>
    <td>18</td>
    <td>3.3%</td>
    <td>0</td>
    <td>18</td>
  </tr>
  <tr>
    <td>EC+CC Hybrid</td>
    <td>125</td>
    <td>23.1%</td>
    <td>125</td>
    <td>125</td>
  </tr>
  <tr>
    <td></td>
    <td>MC Subtotal</td>
    <td>271</td>
    <td>50.1%</td>
    <td>253</td>
    <td>143</td>
  </tr>
  <tr>
    <td colspan="2">Total</td>
    <td>541</td>
    <td>100.0%</td>
    <td>362</td>
    <td>304</td>
  </tr>
</table>

Table 1: Constraint Composition (Const. Comp.) analysis of English benchmark (N=541)

3.2 Translation and Localization

After the pre-processing of the original instruction-following data, we move forward to expand it to multilingual via careful localization beyond simple translation. The translation localization stage adapts the dataset to 30 languages while preserving instruction constraints. Figure 1 show the amount of examples and average length for the data in each language.

3.2.1 Translation

The translation process involves: 1) Initial translation using Google Translate. 2) Bilingual checks and manual corrections by professional translators. 3) Validation using LLMs for error correction. For five typologically diverse languages (Arabic, Spanish, Malay, Yoruba, Chinese), we create parallel corpus variants: machine-translated (MT) baselines and culturally localized versions. For the remaining 24 languages, we apply full cultural localization to SC+EC instances and strategic sampling for SC+CC and MC+EC cases.

3.2.2 Localization

Regarding further transforming the translated data to culturally localized version, we employ a three-step localization method:

• Lexical Substitution: Replace culture-specific terms (e.g., names, locations) while maintaining constraint positions.

• Topical Transposition: Adapt scenario contexts to culturally familiar domains without altering constraint structures.

• Pragmatic Restructuring: Reformulate instructions using target-language rhetorical conventions under fixed constraints.

Cultural localization is achieved through a process guided by ten main sociolinguistic dimensions:
Figure 1: The number of examples and average prompt length for each language in our Marco-Bench–MIF dataset.

historical contexts, social customs, lifestyle patterns, regional characteristics, geographical references, natural landscapes, traditional crafts, culinary traditions, entertainment forms, and national identity markers. We employ advanced LLMs perform constrained content regeneration across randomly selected thematic categories, preserving original task requirements while adapting cultural references. Then we use consensus for three LLMs to evaluate the localized outputs, any content receiving majority disapproval from three LLMs would receive mandatory human re-evaluation.

3.3 Post-processing

The dataset refinement process employs a multi-layered approach to ensure cross-lingual reliability through three interlocking mechanisms. For error reduction, we combine automated pattern detection with human check. The system specifically targets six prevalent translation failure points: keywords, exit statement, echo content, postscript consistency maintenance, case sensitivity adherence, and control of Latin character frequency in non-Latin script languages. We employ an LLM that generates initial outputs, which then undergo evaluation, with another LLM subsequently analyzing failure cases to distinguish between model capability limitations, instruction set defects, and evaluation logic flaws. Human reviewers then prioritize model-flagged cases for in-depth inspection of instruction adherence patterns and assessment code validity.

The evaluation framework undergoes systematic localization across 30 languages through four adaptation types: punctuation convention alignment, response language appropriateness verification, multi-section coherence validation, and constrained output requirement checking. Language selection balances regional prevalence, resource availability, and orthographic diversity, ensuring comprehensive coverage of global languages. This refinement process establishes Marco-Bench-MIF as a culturally-grounded benchmark for multilingual instruction following.

4 Evaluation on Marco-Bench-MIF

4.1 Evaluation Setup

Our evaluation follows IFEval’s protocols (Zhou et al., 2023) with additional adjustments. Basically, we prompt the LLMs with specific instructions \( i \) (can be multiple instructions) to get a response \( r \), we then analyze the response \( r \) to verify whether the LLMs adhere the instruction \( i \) we gave to it. Specifically, we employ rule-based verification and implement an additional one with text normalization. The strict metric is defined as following:

\[
E_{\text{strict}}(r, i) = \begin{cases}
1 & \text{if } r \text{ satisfies } i, \\
0 & \text{otherwise}
\end{cases}
\]

and loose metric with text normalization is:

\[
E_{\text{loose}}(r, i) = \bigwedge_{t \in \mathcal{T}} E(\tau_t(r), i)
\]

where \( \mathcal{T} \) indicating transformation operations (e.g., markdown removal, boundary adjustment) to reduce measurement variance while preserving instruction fidelity. If there is one text normalization
can let \( \tau_i(\tau) \) meet the instruction \( i \) then we think is response is correct.

Moreover, within our evaluation methodology we have both the prompt and instruction level accuracy, which means:

Prompt-level Accuracy: This metric evaluates the model’s ability to completely adhere to all verifiable instructions within a single prompt. For a response to be considered correct, the model must follow every instruction in its entirety, which is a metric that considers the models’s ability for following multiple instructions together.

Instruction-level Accuracy: This measure assesses the model’s compliance with each individual instruction within a prompt. A response is considered accurate on this level for the specific instruction if the model successfully follows the corresponding instructions provided, which is a more fine-grained metric for instruction-level performance.

4.2 Evaluated LLMs

We evaluate a diverse set of LLMs, encompassing both proprietary and open-source systems. The proprietary models include state-of-the-art systems including GPT-4o, Gemini, etc., while the open models include advanced LLMs with varying scales and architectures:

Proprietary Models. The proprietary models evaluated in this study include GPT (GPT-4o, GPT-4o-Mini) (Hurst et al., 2024), Claude (Claude3.5-Sonnet) (Anthropic, 2024), Gemini (Gemini1.5-Flash) (Team, 2023), and Qwen-Proprietary (Qwen-Plus, Qwen-Max) (Bai et al., 2023).

Open Models. The open-source models include Aya (Aya-23: 8B, 35B; Aya-Expansive: 8B, 32B) (Üstün et al., 2024), Gemma (Gemma2: 9B, 27B) (Team et al., 2024), LLaMA (LLaMA3.1-8B, LLaMA3.3-70B) (Dubey et al., 2024), Qwen (7B, 14B, 72B) (Qwen et al., 2025), and Mistral (Mistral-Large, Minstral-8B) (Jiang et al., 2023). Additionally, we include EuroLLM-9B 1.

4.3 Experimental Results

4.3.1 Overall Results

The overall results of various LLMs on our Marco-Bench-MIF dataset are shown in Table 2. The evaluation results across 30 languages show several interesting insights into the multilingual instruction-following capabilities of current LLMs. First, a consistent trend across all models is that the instruction-level accuracy is substantially higher than the prompt-level accuracy (10-20%). This indicates that although models may struggle to satisfy all instructions simultaneously when evaluated as a whole, they are often capable of executing individual instructions correctly. This effect is more obvious in smaller models like Minstral-8B (21.74 vs. 46.45 points difference), suggesting that compositional reasoning remains a key differentiator between model scales. Furthermore, the difference between strict and loose metrics underscores the influence of text normalization. The loose metric, which mitigates minor formatting or boundary discrepancies, typically produces higher accuracy scores than the strict metric (+3-5%). This suggests that many models are close to correctly following instructions but occasionally falter due to superficial deviations in formatting, which do not fundamentally undermine the semantic intent of their responses.

Another observation is the trend in performance as model capacity increases, we observe a strong correlation between model scale and performance, with larger models (70B+ parameters) achieving 45-60% higher absolute accuracy than 8B LLMs. However, this scaling effect is not as obvious as it is in prompt-level for instruction-level metrics, where even smaller models like Qwen2.5-7B achieve 64.42% strict accuracy, suggesting that basic in-

<table>
  <tr>
    <th>Model</th>
    <th>Prompt (S)</th>
    <th>Prompt (L)</th>
    <th>Inst. (S)</th>
    <th>Inst. (L)</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td>Minstral-8B</td>
    <td>21.74</td>
    <td>24.49</td>
    <td>46.45</td>
    <td>49.72</td>
    <td>35.60</td>
  </tr>
  <tr>
    <td>Aya-23-8B</td>
    <td>23.46</td>
    <td>26.03</td>
    <td>46.89</td>
    <td>49.23</td>
    <td>36.40</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B</td>
    <td>25.48</td>
    <td>28.36</td>
    <td>48.98</td>
    <td>51.66</td>
    <td>38.62</td>
  </tr>
  <tr>
    <td>EuroLLM-9B</td>
    <td>32.02</td>
    <td>35.33</td>
    <td>56.42</td>
    <td>59.32</td>
    <td>45.77</td>
  </tr>
  <tr>
    <td>Aya-23-35B</td>
    <td>34.95</td>
    <td>38.79</td>
    <td>57.24</td>
    <td>60.43</td>
    <td>47.85</td>
  </tr>
  <tr>
    <td>Aya-expansive-8B</td>
    <td>35.44</td>
    <td>40.49</td>
    <td>57.82</td>
    <td>61.63</td>
    <td>48.84</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>42.99</td>
    <td>47.43</td>
    <td>64.42</td>
    <td>68.02</td>
    <td>55.72</td>
  </tr>
  <tr>
    <td>Aya-expansive-32B</td>
    <td>49.73</td>
    <td>55.56</td>
    <td>71.33</td>
    <td>75.20</td>
    <td>62.96</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>51.97</td>
    <td>54.53</td>
    <td>72.16</td>
    <td>73.89</td>
    <td>63.14</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>53.86</td>
    <td>58.41</td>
    <td>71.20</td>
    <td>74.39</td>
    <td>64.46</td>
  </tr>
  <tr>
    <td>Gemma3-7B</td>
    <td>58.86</td>
    <td>61.71</td>
    <td>77.71</td>
    <td>78.78</td>
    <td>70.95</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>62.66</td>
    <td>67.16</td>
    <td>79.02</td>
    <td>81.88</td>
    <td>72.70</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>64.84</td>
    <td>69.80</td>
    <td>80.80</td>
    <td>83.96</td>
    <td>74.85</td>
  </tr>
  <tr>
    <td>LLaMA3.3-70B</td>
    <td style="background-color:#b3daff">67.42</td>
    <td style="background-color:#b3daff">70.32</td>
    <td style="background-color:#b3daff">80.43</td>
    <td style="background-color:#b3daff">82.25</td>
    <td style="background-color:#b3daff">75.11</td>
  </tr>
  <tr>
    <td>Qwen-Plus</td>
    <td>68.98</td>
    <td>73.02</td>
    <td>82.03</td>
    <td>84.64</td>
    <td>77.17</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>69.29</td>
    <td>72.67</td>
    <td>82.31</td>
    <td>84.53</td>
    <td>77.20</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>68.79</td>
    <td>73.02</td>
    <td>82.42</td>
    <td>85.07</td>
    <td>77.33</td>
  </tr>
  <tr>
    <td>Gemini1.5-flash</td>
    <td>71.40</td>
    <td>75.88</td>
    <td>84.17</td>
    <td>86.95</td>
    <td>79.60</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>71.43</td>
    <td>75.89</td>
    <td>84.49</td>
    <td>87.13</td>
    <td>79.73</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet</td>
    <td style="background-color:#e6b9b9">73.61</td>
    <td style="background-color:#e6b9b9">76.77</td>
    <td style="background-color:#e6b9b9">85.62</td>
    <td style="background-color:#e6b9b9">87.71</td>
    <td style="background-color:#e6b9b9">80.93</td>
  </tr>
</table>

Table 2: Overall results of various LLMs on Marco-Bench-MIF with average accuracy, where the best performance of open models and proprietary model are marked in Red and Blue respectively.

1 https://huggingface.co/utter-project/EuroLLM-9B-Instruct
<table>
  <tr>
    <th>Model</th>
    <th>vi</th>
    <th>hi</th>
    <th>ne</th>
    <th>es</th>
    <th>ru</th>
    <th>id</th>
    <th>ja</th>
    <th>sw</th>
    <th>en</th>
    <th>zh</th>
    <th>fr</th>
    <th>pl</th>
    <th>te</th>
    <th>tk</th>
    <th>it</th>
    <th>ur</th>
    <th>pt</th>
    <th>no</th>
    <th>sv</th>
    <th>xh</th>
    <th>th</th>
    <th>ar</th>
    <th>hi</th>
    <th>ko</th>
    <th>tl</th>
    <th>ur</th>
    <th>ko</th>
    <th>ru</th>
    <th>xh</th>
    <th>in</th>
    <th>ugg</th>
  </tr>
  <tr><td>LM+23-BB</td><td>84.76</td><td>11.84</td><td>53.14</td><td>39.74</td><td>151.34</td><td>244.40</td><td>79.79</td><td>59.13</td><td>82.23</td><td>34.26</td><td>41.45</td><td>49.48</td><td>30.41</td><td>350.42</td><td>90.76</td><td>27.53</td><td>48.17</td><td>90.47</td><td>27.56</td><td>47.24</td><td>43.22</td><td>67.82</td><td>35.62</td><td>307.00</td><td>40.51</td><td>276.00</td><td>37.65</td><td>49.53</td><td>179.57</td></tr>
  <tr><td>Aya-expanse-7BB</td><td>111.63</td><td>14.89</td><td>57.53</td><td>47.67</td><td>145.81</td><td>150.67</td><td>65.94</td><td>57.10</td><td>94.59</td><td>46.00</td><td>58.95</td><td>61.19</td><td>49.38</td><td>378.30</td><td>79.68</td><td>40.06</td><td>69.39</td><td>80.36</td><td>32.65</td><td>61.46</td><td>67.67</td><td>72.86</td><td>44.88</td><td>63.20</td><td>41.93</td><td>279.95</td><td>57.23</td><td>280.42</td><td>42.62</td><td>42.74</td><td>96.17</td></tr>
  <tr><td>LlamaMA-1.3B</td><td>104.21</td><td>145.89</td><td>54.31</td><td>43.77</td><td>63.39</td><td>43.92</td><td>41.27</td><td>249.76</td><td>51.35</td><td>36.30</td><td>48.14</td><td>39.52</td><td>399.93</td><td>78.34</td><td>36.48</td><td>45.35</td><td>75.68</td><td>76.14</td><td>44.06</td><td>49.88</td><td>74.77</td><td>278.76</td><td>74.67</td><td>292.60</td><td>66.90</td><td>44.44</td><td>107.41</td><td>94.26</td><td>43.71</td><td>44.47</td><td>112.84</td></tr>
  <tr><td>LlamaMA.3-7B</td><td>114.31</td><td>97.02</td><td>46.10</td><td>51.71</td><td>55.84</td><td>59.58</td><td>52.93</td><td>84.49</td><td>56.65</td><td>70.05</td><td>64.51</td><td>46.14</td><td>496.85</td><td>80.83</td><td>35.54</td><td>37.32</td><td>75.93</td><td>120.72</td><td>57.41</td><td>96.03</td><td>88.63</td><td>166.22</td><td>90.09</td><td>268.00</td><td>71.95</td><td>56.37</td><td>83.41</td><td>106.13</td><td>52.74</td><td>87.99</td></tr>
  <tr><td>Qwen-2</td><td>259.12</td><td>141.45</td><td>31.74</td><td>50.21</td><td>31.76</td><td>55.52</td><td>58.16</td><td>127.80</td><td>14.93</td><td>123.14</td><td>143.14</td><td>90.21</td><td>632.02</td><td>88.54</td><td>58.44</td><td>-</td><td>75.68</td><td>157.17</td><td>86.93</td><td>87.45</td><td>98.07</td><td>70.71</td><td>358.00</td><td>85.93</td><td>170.68</td><td>106.83</td><td>86.34</td><td>151.47</td><td>75.17</td><td>91.38</td><td>147.99</td><td>-</td><td>122.91</td></tr>
  <tr><td>Claude-3-Beta</td><td>82.89</td><td>64.60</td><td>50.79</td><td>90.68</td><td>71.73</td><td>85.39</td><td>57.78</td><td>70.65</td><td>47.95</td><td>56.65</td><td>80.59</td><td>30.03</td><td>1090.14</td><td>98.14</td><td>76.60</td><td>60.47</td><td></td><td>128.25</td><td>21.98</td><td>89.57</td><td>83.11</td><td>85.59</td><td>70.56</td><td>59.47</td><td>73.26</td><td>21.85</td><td>87.56</td><td>34.53</td><td>80.46</td><td>62.67</td></tr>
  <tr><td>Avg.</td><td>102.21</td><td>80.94</td><td>52.75</td><td>42.49</td><td>37.84</td><td>36.86</td><td>43.49</td><td>59.94</td><td>46.37</td><td>38.07</td><td>23.71</td><td>-</td><td>369.62</td><td>78.14</td><td>55.45</td><td>62.17</td><td>78.05</td><td>74.66</td><td>44.57</td><td>57.27</td><td>43.40</td><td>38.03</td><td>78.20</td><td>54.97</td><td>82.34</td><td>68.74</td><td>77.83</td><td>41.84</td>-<td>56.39</td><td>92.26</td></tr>
</table>

Table 3: Overall results for our evaluated LLMs divided by language, where the best performer of open models and proprietary model are marked in Red and Blue respectively.

![Results of LLMs on Arabic (Ar) and Chinese (Zh) data divided by categories.](page_1012_1372_1153_295.png)

struction understanding can be achieved at that scales. Moreover, proprietary LLMs all exhibit strong performance against open LLMs, which demonstrates the superiority of these models and that there is still large room for improvement for open models.

4.3.2 Results per Language
We split the evaluation results by the language in our Marco-Bench-MIF dataset, the results are shown in Table 3. The results show three critical patterns in multilingual instruction following capability of LLMs. First, the performance of LLMs is largely depended on the availability of the language resources: High-resource European (de, fr) and East Asian languages (zh, ja) achieve 75-85% accuracy across top models, while low-resource languages (yo, ne, kk) performance is at 50-60% even for Claude3.5-sonnet. This 25-35 point gap persists across difference LLM scales, suggesting that current state-of-the-art LLMs still struggle for low-resource languages at instruction-following.

Second, script-specific capabilities significantly impact performance. Most LLMs exhibit 15-20 point disparities between languages using the same script family (e.g., Romance languages es/pt vs. Romanian ro), indicating character-level encoding challenges. Right-to-left scripts (ar, he) show particular sensitivity, with LLaMA.3-70B scoring 78.5 in Hebrew versus 54.7 in Urdu (ur) despite similar pretraining data sizes. We see an interesting point from the results that while multilingual-specialized models like Aya-expanse-32B achieve best-in-class performance for mid-resource languages (hi, sw: 39.8), general-purpose models like Qwen2.5-72B outperform them in 22/30 languages through scale advantages. We identify three language response types: 1). Scale-sensitive languages (ru, tr) where 70B models achieve almost 20 point gains over 8B LLMs 2). Model-dependent languages (he, ja) showing 15-30 point variations between equally-sized models 3). Extremely-hard languages (yo, kk) where even GPT-40 scores less than 60% accuracy. Moreover, cross-lingual interference patterns appear in related language pairs. For Turkish languages, Turkish (tr) performance strongly predicts Kazakh (kk) accuracy (r=0.89), suggesting models transfer capabilities within language families. However, this correlation does not hold for Slavic languages, where Ukrainian (uk) shows unexpectedly low scores relative to Russian (ru) across all models.

We further plot the categorized results on two languages with different scripts (Latin vs non-Latin) of our Marco-Bench-MIF, results shown in Figure 2. There are some interesting findings and patterns. Closed-source models like Qwen-
Figure 3: Comparison between the performance (average accuracy of Strict and Loose mode for both instruction and prompt-level) of various LLMs on the localized data vs machine-translated data.

Max and GPT-4o maintain robust performance across both languages (less than 5% relative difference in detectable_content), we observe significant language-specific degradation in smaller open-source models - particularly for structural constraints like startend (\( \Delta = 28\% \) for Qwen2-7B between zh/ar). Chinese processing shows stronger format preservation (avg. 89% vs. 82% in detectable_format) while Arabic models better handle language identification (94% vs. 91% avg.), suggesting script-specific pattern learning biases. The non-Latin script challenge emerges clearly in change_case tasks, where all models show 12-15% performance drops for Arabic compared to Chinese, revealing limitations in script-invariant processing. We identify a critical cross-lingual transfer gap: Mistral-Large achieves 87%/94% detectable_content for ar/zh respectively, while LLaMA3-70B shows dramatic variance (35% vs. 95%), indicating inconsistent multilingual generalization. Punctuation handling proves most language-agnostic (\( \Delta < 3\% \) for top models), suggesting structural linguistic features rather than script properties dominate this capability. Our findings emphasize that effective multilingual instruction following requires separate optimization beyond simple vocabulary extension, particularly for combinatorial constraints in right-to-left scripts (\( \Delta = 19\% \) for combination tasks ar vs. zh).

4.3.3 Responding in Specific Language

We study an interesting aspect of multilingual instruction following capability - let LLMs respond in specified language. This sub-task directly reflects the text generation capability of LLMs in various language. The results are shown in Table 4. Results show that multilingual-enhanced LLMs: Aya-expanse outperform proprietary models regardless of size, with the Aya-expanse-32B variant (95.69) surpassing most proprietary systems. This demonstrates the effectiveness of targeted multilingual enhancement. Gemma models (both 9B and 27B) also achieve comparable performance (91+) to top proprietary models. We can see that even top LLMs like GPT-4o and Claude still can not fully follow the instruction that respond in specified language (5% wrong outputs).

<table>
  <tr>
    <th>Model</th>
    <th>Response Language</th>
  </tr>
  <tr>
    <th colspan="2">Open Models (< 10B)</th>
  </tr>
  <tr>
    <td>LLaMA3.1-8B</td>
    <td>58.34</td>
  </tr>
  <tr>
    <td>Aya23-8B</td>
    <td>65.57</td>
  </tr>
  <tr>
    <td>Mistral-8B</td>
    <td>66.03</td>
  </tr>
  <tr>
    <td>Aya-expanse-8B</td>
    <td>75.56</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>78.95</td>
  </tr>
  <tr>
    <td>EuroLLM-9B</td>
    <td>79.19</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>91.82</td>
  </tr>
  <tr>
    <th colspan="2">Open Models (> 10B)</th>
  </tr>
  <tr>
    <td>Aya23-35B</td>
    <td>68.88</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>79.69</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>91.88</td>
  </tr>
  <tr>
    <td>Aya-expanse-32B</td>
    <td>95.69</td>
  </tr>
  <tr>
    <td>LLaMA3.3-70B</td>
    <td>82.38</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>91.02</td>
  </tr>
  <tr>
    <th colspan="2">Proprietary Models</th>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>90.67</td>
  </tr>
  <tr>
    <td>Qwen-Plus</td>
    <td>92.22</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>93.20</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet</td>
    <td>94.03</td>
  </tr>
  <tr>
    <td>Gemini1.5-flash</td>
    <td>94.08</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>95.09</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>95.43</td>
  </tr>
</table>

Table 4: Results of the Response Language instruction (average accuracy of Strict and Loose mode).

4.3.4 Effect of Translated and Localised Data

We also conduct experiments investigating the effects of localized data versus machine-translated data, the results are shown in Figure 3. The results clearly demonstrate that evaluations based exclusively on MT data can be misleading. In high-resource languages such as Arabic and Chinese, the differences appear minimal, but for low-resource languages like Yoruba the gap is more larger (prompt-level: avg 22.13% localized vs. 19.77% MT). For example, the performance of Claude3.5-sonnet in Yoruba is largely underestimated by 7.1% when evaluated using MT data. These discrepancies indicate that MT-data can not precisely capture the full complexity of instruction following—especially when it comes to culturally and linguistically fine-grained inputs. As task com-
<table>
  <tr>
    <th>Input Instruction</th>
    <th>Open Models</th>
    <th>Proprietary Models</th>
  </tr>
  <tr>
    <td><b>Chinese:</b><br>讲一个包含单词“星期四”和“合并”的笑话，但只能使用斯瓦希里语，不允许使用其他语言。（Tell a joke containing the words “Thursday” and “merge” in Swahili only, no other languages allowed.)</td>
    <td>Aya-expanse-32B:<br>Sijambo, ni shehemu ambako tunapokaa tuck kulaungiana mikosi ya juma la saba na juma la mwe... (Hello, it’s a place where we stay to merge the misfortunes of Saturday and Thursday...)<br><span style="color:green;">correct</span></td>
    <td>GPT-4o:<br>为什么星期四和星期五不能合并？因为星期四还在努力工作，而星期五已经开始庆祝周末了！（Why can’t Thursday and Friday merge? Because on Thursday you’re still working hard, but on Friday you’ve already started celebrating the weekend!）<br><span style="color:red;">wrong</span></td>
  </tr>
  <tr>
    <td><b>Spanish:</b><br>Reescribe la frase: 'Lancé mi copla al viento de Levante'. Escribe toda tu respuesta entre comillas dobles. (Rewrite the phrase: 'I threw my verse to the Levante wind'. Write your entire answer in double quotes.)</td>
    <td>LLaMA3.3-70B:<br>"Dejo que mi canción se escuche llevada por el viento de Levante" (I let my song be heard carried by the Levante wind)<br><span style="color:red;">wrong</span></td>
    <td>Claude-3.5:<br>"Al viento de Levante arrojé mi copla" (To the Levante wind I threw my verse)<br><span style="color:red;">wrong</span></td>
  </tr>
  <tr>
    <td><b>Malay:</b><br>Boleh buat jadual lawatan 5 hari ke Istana Maimoon yang mengandungi tepat 3 titik peluru dalam format senarai menurun dengan huruf kecil dan nota di akhir bermula dengan P.S.? (Can you make a 5-day visit schedule to Maimoon Palace containing exactly 3 bullet points in a descending list format with lowercase letters and a note at the end starting with P.S.? Please answer in English.)</td>
    <td>Qwen2.5-72B:<br>Pulau Riau emas terbit<br>Laut lebar burung terbang<br>Rindang kelapa berjajar<br>Harapan bersama di Nusantara (Riau Island golden sunrise / Wide sea birds flying / Palm trees in rows / Hope together in Nusantara)<br><span style="color:red;">wrong</span></td>
    <td>Gemini1.5-flash:<br>Pulau Bintan indah sekali<br>Lautnya biru pasir putih<br>Harta budaya Melayu asli<br>Terserab luas di Nusantara (Bintan Island is beautiful / Blue sea white sand / Original Malay cultural heritage / Spread widely in Nusantara)<br><span style="color:red;">wrong</span></td>
  </tr>
</table>

Table 5: Examples of multilingual instruction-following capabilities across different models. English translations are provided in italics below each non-English text.

plexity increases, the inadequacy of MT data becomes even more evident, suggesting that localized data is crucial for a precise evaluation of LLMs’ instruction-following abilities.

4.4 Case Study

We demonstrate some test cases and model outputs from our Marco-Bench–MIF dataset in Table 5. The three representative cases provide some concrete observation of LLMs behavior when facing localized multilingual instructions. The Chinese-Swahili joke generation exposes fundamental language switching limitations, where GPT-4o defaults to Chinese despite explicit instructions, while Aya-expanse-32B successfully maintains Swahili output through specialized multilingual embeddings. Both LLaMA3.3-70B and Claude-3.5 fail the Spanish rewriting task’s structural requirement (strict quote enclosure), demonstrating that formatting constraints remain under-addressed in current multilingual training paradigms. The Malay schedule generation reveals deeper instruction comprehension failures - proprietary models generate irrelevant poetry instead of bullet-point schedules, suggesting overfitting to frequent Malay literary patterns rather than practical task requirements. These failures persist across model scales and architectures, indicating systemic gaps in non-English instruction following that correlate with training data priorities rather than model capacity. The cases highlight three underdeveloped capabilities of LLMs: 1) cross-lingual constraint propagation, 2) non-English structural formatting, and 3) culture-specific task understanding.

5 Conclusion

Our work presents Marco–Bench–MIF, a comprehensive multilingual instruction-following benchmark covering 30 languages with localized adaptations. Through extensive evaluation, we identify key challenges in multilingual instruction following, including formatting fragility, compositional reasoning limits, and structural-task disparity. Our results demonstrate that while large-scale models achieve strong performance in high-resource languages, significant gaps remain for low-resource languages and complex structural constraints. These findings highlight the need for more robust evaluation frameworks that account for linguistic diversity and task complexity. Specialized models like Aya-expanse-32B in certain multilingual instructions suggests that targeted architectural improvements can complement scaling strategies. However, the persistent performance gaps in low-resource languages and fine-grained cultural-related tasks indicate that data quality and diversity remain critical bottlenecks. Future research will focus on expanding Marco–Bench–MIF to include more languages particularly those with unique ty-
pological features, will provide a more comprehensive understanding of multilingual instruction-following capabilities. We also plan to include more instruction types to better reflect real-world scenarios.

Limitations

Our work has several limitations that highlight directions for improvement. First, while our Marco-Bench-MIF covers 30 languages, it under-represents languages with non-Latin scripts (e.g., Ethiopic, Cherokee) and dialectal variations (e.g., Arabic dialects), which limits investigation into instruction with more diverse scripts. Second, our cultural localization focuses on surface-level adaptations (e.g., date formats) with a few fine-grained cultural pragmatics (e.g., politeness strategies in Japanese), potentially missing more deeper instruction-following failures. Moreover, while we mitigated translation artifacts through human evaluation, residual biases from automated localization (e.g., GPT-4’s tendency to formal register) may still exist in certain languages. Finally, our evaluation focuses on static prompts rather than interactive instruction refinement, limiting insights into models’ ability to recover from errors in dynamic conversations.

Ethics and Broader Impact

In developing Marco–Bench–MIF, a multilingual instruction-following benchmark covering 30 languages, we prioritize ethical considerations in dataset creation, model evaluation, and broader societal impacts. Our localization process combines automated translation with human verification to ensure cultural sensitivity and linguistic appropriateness, avoiding stereotypes or misrepresentations. The dataset will be made publicly available with detailed documentation to ensure transparency and reproducibility, while clear usage guidelines will mitigate potential misuse, such as deploying LLMs in harmful applications. We also recognize the environmental impact of computational resources used in this work and advocate for energy-efficient methods in NLP research. We are committed to responsible AI development and have taken steps to ensure fairness, inclusivity, and respect for human contributors. Human annotators and translators were compensated fairly and provided informed consent, with their privacy and rights upheld throughout the project.

References

Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku.

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vi: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc.

Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et al. 2024. Culturalbench: a robust, diverse and challenging benchmark on measuring the (lack of) cultural knowledge of llms. arXiv preprint arXiv:2410.02677.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravan Kumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Alonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esibov, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Gregoire Milon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vrane, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasudevan Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone,
Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kamadbur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qinxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidorio, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaoy, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gouget, Virginie Do, Vish Vogeti, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenying Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiore, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kaliet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsian Chu, Chris Cai, Chris Tindal, Christoph Hockenhofer, Damon Civin, Dana Beaty, Daniel Kreymar, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Tessugine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badee, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojaianzeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegen, Hunter Goldman, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gay, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelenea, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martyna Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatkovskiy, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpeter, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavani Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanachandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswaru, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihaieluscu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiao-
jian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhwei Zhao. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.

Yun He, Di Jin, Chaoqi Wang, Chloe Bi, Karishma Mandyam, Hejia Zhang, Chen Zhu, Ning Li, Tengyu Xu, Hongjiang Lv, Shruti Bhosale, Chenguang Zhu, Karthik Abhinav Sankararaman, Eryk Helenowski, Melanie Kambadur, Aditya Tayade, Hao Ma, Han Fang, and Sinong Wang. 2024. Multi-if: Benchmarking llms on multi-turn and multilingual instructions following. Preprint, arXiv:2410.15553.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os-trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. Preprint, arXiv:2310.06825.

Bingshuai Liu, Longyue Wang, Chenyang Lyu, Yong Zhang, Jinsong Su, Shuming Shi, and Zhaopeng Tu. 2023. On the cultural gap in text-to-image generation. arXiv preprint arXiv:2307.02971.

Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvaa Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinev Ali Ayele, et al. 2024. Blend: A benchmark for llms on everyday knowledge in diverse cultures and languages. arXiv preprint arXiv:2406.09948.

OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774.

Qwen, ;, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, Atnafu Lambebo Tonja, et al. 2024. Cvqa: Culturally-diverse multilingual visual question answering benchmark. arXiv preprint arXiv:2406.05967.

Gemini Team. 2023. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nina Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbisron, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimeczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Leticia Lago, Lilly McNealus, Livio Baldino Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidson, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardesher Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin,
<table>
  <tr>
    <th>Model</th>
    <th>no_comma</th>
    <th>end_checker</th>
    <th>quotation</th>
  </tr>
  <tr>
    <td>Aya23-8B</td>
    <td>28.08</td>
    <td>26.80</td>
    <td>25.61</td>
  </tr>
  <tr>
    <td>Ministral-8B</td>
    <td>41.15</td>
    <td>27.91</td>
    <td>21.66</td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B</td>
    <td>44.73</td>
    <td>32.77</td>
    <td>28.88</td>
  </tr>
  <tr>
    <td>EuroLLM-9B</td>
    <td>41.64</td>
    <td>38.30</td>
    <td>37.75</td>
  </tr>
  <tr>
    <td>Aya-expanse-8B</td>
    <td>24.57</td>
    <td>44.06</td>
    <td>54.68</td>
  </tr>
  <tr>
    <td>Aya23-35B</td>
    <td>46.21</td>
    <td>40.92</td>
    <td>56.38</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>67.33</td>
    <td>46.22</td>
    <td>56.09</td>
  </tr>
  <tr>
    <td>Aya-expanse-32B</td>
    <td>46.75</td>
    <td>52.54</td>
    <td>74.00</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>68.67</td>
    <td>63.92</td>
    <td>64.34</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>67.03</td>
    <td>69.16</td>
    <td>68.05</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>86.48</td>
    <td>70.62</td>
    <td>64.68</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>87.84</td>
    <td>66.92</td>
    <td>78.92</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>87.44</td>
    <td>72.00</td>
    <td>85.26</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>87.80</td>
    <td>73.39</td>
    <td>83.86</td>
  </tr>
  <tr>
    <td>Qwen-Plus</td>
    <td>90.36</td>
    <td>72.44</td>
    <td>83.09</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>91.84</td>
    <td>73.50</td>
    <td>81.75</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>86.50</td>
    <td>75.23</td>
    <td>89.42</td>
  </tr>
  <tr>
    <td>LLaMA3.3-70B</td>
    <td>92.07</td>
    <td>82.16</td>
    <td>83.27</td>
  </tr>
  <tr>
    <td>Gemini1.5-flash</td>
    <td>93.23</td>
    <td>77.90</td>
    <td>89.73</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet</td>
    <td>95.56</td>
    <td>77.89</td>
    <td>94.12</td>
  </tr>
</table>

Table 6: Format style Instruction Following Results

Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.

Zhiyuan Zeng, Jiataong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations.

Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911.

Ahmet Üstün, Virat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargas, Phil Blunsom, Shayne Longpre, Niklas Muenninghoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction finetuned open-access multilingual language model. Preprint, arXiv:2402.07827.

A Appendix

A.1 Results per Category

Format Style: Models exhibit formatting fragility, with comma/quote adherence showing 40-60 point gaps between small and large models. Specialized architectures (Aya-expanse-32B:74% quotation) outperform general models of similar size (Gemma2-9B:64.34%), suggesting targeted training improves stylistic precision. Claude3.5’s 94.12% quotation accuracy demonstrates proprietary models’ edge in text surface constraints.

Content Constraints: Complex multi-instruction tasks (repeat_prompt:56.66-95.56%) reveal compositional reasoning limits. While all models struggle with numerical placeholders (<95.36%), open-source models show particular weakness in frequency counting (Gemma2-27B:67.8% vs Claude3.5:82.8%). The 35-point spread in forbidden_words detection highlights fundamental differences in constraint understanding architectures.

Structure: JSON formatting proves most discriminatory (Claude3.5:95.1% vs Qwen2.5-7B:63.3%), exposing syntax-semantics decoupling. While all models achieve >90% on constrained_response, multi-section generation (81.04% max) remains challenging, suggesting current architectures prioritize content over structural fidelity. Length: Positional counting (nth_paragraph_first_word) shows severe scale sensitivity—SB models average 17.9% vs 70B+ at 76.3%. Word/paragraph limits reveal inverted patterns: Qwen2.5-72B outperforms GPT-4o in word counts (73.27% vs 67.62%), indicating counting mechanisms vary significantly across architectures. Language: Proprietary models lead response_language accuracy (95.43% max), but specialized open models (Aya-expanse-32B:95.69%) close the gap through linguistic focusing. The 26-point spread between similarly-sized models (Gemma2-9B:91.82% vs EuroLLM-9B:79.19%) underscores the impact of multilingual pretraining strategies.

Our analysis reveals three main limitations in multilingual instruction following: (1) Structural-task disparity where models handle content generation better than formatting/structural constraints, (2) Scale-task decoupling as certain capabilities (positional awareness) require extreme scaling while others (basic translation) plateau early, and (3) Architecture-task alignment where model specialization (multilingual vs general-purpose) creates 15-30 point performance variations independent of scale. The results suggest current benchmarks underestimate cross-task capability variance, necessitating more granular evaluation frameworks.
<table>
  <tr>
    <th>Model</th>
    <th>repeat_prompt</th>
    <th>two_responses</th>
    <th>number_placeholders</th>
    <th>postscript</th>
    <th>existence</th>
    <th>forbidden_words</th>
    <th>frequency</th>
    <th>letter_frequency</th>
  </tr>
  <tr><td>Aya23-8B</td><td>6.86</td><td>10.02</td><td>44.66</td><td>61.18</td><td>44.86</td><td>68.70</td><td>41.00</td><td>47.19</td></tr>
  <tr><td>EuroLLM-9B</td><td>7.86</td><td>32.25</td><td>41.33</td><td>75.39</td><td>54.95</td><td>66.73</td><td>46.53</td><td>48.60</td></tr>
  <tr><td>Ministral-8B</td><td>14.15</td><td>19.32</td><td>51.42</td><td>71.69</td><td>42.50</td><td>85.39</td><td>43.96</td><td>46.60</td></tr>
  <tr><td>LLaMA3.1-8B</td><td>14.73</td><td>33.59</td><td>46.51</td><td>56.31</td><td>48.15</td><td>82.90</td><td>47.15</td><td>58.84</td></tr>
  <tr><td>Aya23-35B</td><td>33.99</td><td>27.75</td><td>54.39</td><td>78.48</td><td>57.90</td><td>69.10</td><td>51.56</td><td>46.97</td></tr>
  <tr><td>Aya-expanse-8B</td><td>20.77</td><td>16.42</td><td>60.90</td><td>88.63</td><td>68.45</td><td>68.73</td><td>58.85</td><td>50.79</td></tr>
  <tr><td>Qwen2.5-7B</td><td>34.59</td><td>59.75</td><td>63.13</td><td>84.48</td><td>68.45</td><td>76.74</td><td>53.91</td><td>50.37</td></tr>
  <tr><td>Aya-expanse-32B</td><td>28.44</td><td>45.16</td><td>70.68</td><td>93.55</td><td>77.30</td><td>73.84</td><td>68.95</td><td>52.30</td></tr>
  <tr><td>Gemma2-9B</td><td>47.89</td><td>60.76</td><td>73.50</td><td>95.40</td><td>67.70</td><td>87.23</td><td>58.86</td><td>51.28</td></tr>
  <tr><td>Qwen2.5-14B</td><td>43.07</td><td>77.59</td><td>77.65</td><td>90.79</td><td>73.20</td><td>82.21</td><td>66.81</td><td>57.95</td></tr>
  <tr><td>Gemma2-27B</td><td>55.08</td><td>87.33</td><td>83.13</td><td>67.77</td><td>70.15</td><td>87.68</td><td>67.80</td><td>53.75</td></tr>
  <tr><td>Qwen2.5-72B</td><td>54.98</td><td>81.67</td><td>89.64</td><td>94.94</td><td>81.50</td><td>87.47</td><td>74.66</td><td>54.49</td></tr>
  <tr><td>Mistral-Large</td><td>56.15</td><td>82.92</td><td>84.75</td><td>95.71</td><td>81.50</td><td>89.69</td><td>78.44</td><td>55.82</td></tr>
  <tr><td>Qwen-Plus</td><td>55.89</td><td>83.16</td><td>87.42</td><td>93.87</td><td>85.79</td><td>85.86</td><td>79.87</td><td>55.14</td></tr>
  <tr><td>Qwen-Max</td><td>55.93</td><td>85.30</td><td>88.81</td><td>90.01</td><td>84.88</td><td>89.25</td><td>77.36</td><td>55.84</td></tr>
  <tr><td>GPT-4o-mini</td><td>68.32</td><td>88.33</td><td>78.57</td><td>98.30</td><td>80.80</td><td>89.16</td><td>75.96</td><td>48.54</td></tr>
  <tr><td>LLaMA3.3-70B</td><td>61.47</td><td>88.17</td><td>95.26</td><td>92.64</td><td>81.10</td><td>92.38</td><td>81.90</td><td>61.52</td></tr>
  <tr><td>Gemini1.5-flash</td><td>70.80</td><td>91.78</td><td>95.36</td><td>98.25</td><td>76.65</td><td>87.76</td><td>79.33</td><td>35.11</td></tr>
  <tr><td>GPT-4o</td><td>74.84</td><td>90.47</td><td>92.29</td><td>98.91</td><td>82.80</td><td>91.08</td><td>82.30</td><td>51.99</td></tr>
  <tr><td>Claude3.5-sonnet</td><td>56.66</td><td>93.62</td><td>94.61</td><td>96.07</td><td>85.08</td><td>95.80</td><td>82.80</td><td>64.28</td></tr>
</table>

Table 7: Content constraints Instruction Following Results

<table>
  <tr>
    <th>Model</th>
    <th>constrained_response</th>
    <th>json_format</th>
    <th>multiple_sections</th>
    <th>number_bullet_lists</th>
    <th>number_highlighted_sections</th>
    <th>title</th>
  </tr>
  <tr><td>Ministral-8B</td><td>61.20</td><td>36.23</td><td>36.29</td><td>24.07</td><td>59.92</td><td>46.49</td></tr>
  <tr><td>Aya23-8B</td><td>67.16</td><td>52.89</td><td>30.19</td><td>27.96</td><td>57.31</td><td>32.63</td></tr>
  <tr><td>LLaMA3.1-8B</td><td>68.80</td><td>17.05</td><td>46.85</td><td>34.40</td><td>68.91</td><td>70.94</td></tr>
  <tr><td>EuroLLM-9B</td><td>72.80</td><td>43.88</td><td>51.72</td><td>36.52</td><td>66.34</td><td>72.86</td></tr>
  <tr><td>Aya23-35B</td><td>78.80</td><td>60.13</td><td>36.86</td><td>46.06</td><td>68.17</td><td>72.33</td></tr>
  <tr><td>Aya-expanse-8B</td><td>74.80</td><td>62.35</td><td>44.27</td><td>45.09</td><td>92.26</td><td>56.12</td></tr>
  <tr><td>Qwen2.5-7B</td><td>90.40</td><td>63.30</td><td>45.43</td><td>52.90</td><td>77.00</td><td>88.12</td></tr>
  <tr><td>Gemma2-9B</td><td>84.20</td><td>50.59</td><td>54.28</td><td>66.65</td><td>87.00</td><td>84.55</td></tr>
  <tr><td>Aya-expanse-32B</td><td>73.80</td><td>81.17</td><td>60.57</td><td>55.03</td><td>92.60</td><td>74.50</td></tr>
  <tr><td>Qwen2.5-14B</td><td>89.20</td><td>71.77</td><td>71.70</td><td>58.70</td><td>78.58</td><td>92.87</td></tr>
  <tr><td>Gemma2-27B</td><td>83.80</td><td>68.48</td><td>66.86</td><td>69.02</td><td>89.92</td><td>85.20</td></tr>
  <tr><td>Mistral-Large</td><td>83.20</td><td>85.77</td><td>72.86</td><td>62.44</td><td>94.00</td><td>89.95</td></tr>
  <tr><td>Qwen2.5-72B</td><td>91.60</td><td>71.41</td><td>75.15</td><td>69.74</td><td>92.93</td><td>95.25</td></tr>
  <tr><td>Qwen-Plus</td><td>90.86</td><td>81.68</td><td>79.40</td><td>70.87</td><td>88.47</td><td>92.51</td></tr>
  <tr><td>LLaMA3.3-70B</td><td>90.00</td><td>73.65</td><td>77.72</td><td>81.48</td><td>93.93</td><td>88.33</td></tr>
  <tr><td>Gemini1.5-flash</td><td>89.43</td><td>86.39</td><td>69.80</td><td>78.48</td><td>93.28</td><td>95.84</td></tr>
  <tr><td>Qwen-Max</td><td>93.28</td><td>83.27</td><td>81.03</td><td>71.43</td><td>92.14</td><td>94.61</td></tr>
  <tr><td>GPT-4o-mini</td><td>92.29</td><td>88.22</td><td>83.27</td><td>73.35</td><td>92.03</td><td>93.07</td></tr>
  <tr><td>GPT-4o</td><td>91.14</td><td>87.46</td><td>84.09</td><td>73.50</td><td>95.42</td><td>96.23</td></tr>
  <tr><td>Claude3.5-sonnet</td><td>92.86</td><td>95.10</td><td>81.04</td><td>76.77</td><td>91.80</td><td>97.31</td></tr>
</table>

Table 8: Structure Instruction Following Results
<table>
  <tr>
    <th>Model</th>
    <th>nth_paragraph_first_word</th>
    <th>number_paragraphs</th>
    <th>number_sentences</th>
    <th>number_words</th>
  </tr>
  <tr><td>Ministral-8B</td><td>10.50</td><td>12.36</td><td>59.53</td><td>52.97</td></tr>
  <tr><td>Aya23-8B</td><td>13.76</td><td>19.56</td><td>59.36</td><td>51.91</td></tr>
  <tr><td>EuroLLM-9B</td><td>20.83</td><td>24.87</td><td>36.54</td><td>54.57</td></tr>
  <tr><td>LLaMA3.1-8B</td><td>20.65</td><td>29.91</td><td>57.65</td><td>68.77</td></tr>
  <tr><td>Aya-expanse-8B</td><td>26.66</td><td>37.40</td><td>54.03</td><td>62.37</td></tr>
  <tr><td>Qwen2.5-7B</td><td>19.50</td><td>46.15</td><td>60.23</td><td>59.59</td></tr>
  <tr><td>Aya23-35B</td><td>34.16</td><td>36.66</td><td>57.45</td><td>61.15</td></tr>
  <tr><td>Aya-expanse-32B</td><td>51.17</td><td>58.23</td><td>56.38</td><td>60.25</td></tr>
  <tr><td>Gemma2-9B</td><td>52.83</td><td>60.92</td><td>60.38</td><td>52.50</td></tr>
  <tr><td>Gemma2-27B</td><td>52.34</td><td>63.73</td><td>61.46</td><td>55.12</td></tr>
  <tr><td>Qwen2.5-14B</td><td>40.16</td><td>68.02</td><td>64.03</td><td>67.34</td></tr>
  <tr><td>Mistral-Large</td><td>60.66</td><td>69.42</td><td>64.77</td><td>61.25</td></tr>
  <tr><td>Gemini1.5-flash</td><td>56.90</td><td>87.52</td><td>65.75</td><td>70.14</td></tr>
  <tr><td>Qwen-Max</td><td>72.03</td><td>81.92</td><td>67.45</td><td>64.02</td></tr>
  <tr><td>GPT-4o-mini</td><td>81.66</td><td>78.95</td><td>61.54</td><td>63.61</td></tr>
  <tr><td>Qwen-Plus</td><td>69.41</td><td>83.34</td><td>69.04</td><td>66.05</td></tr>
  <tr><td>Qwen2.5-72B</td><td>72.01</td><td>75.50</td><td>67.38</td><td>73.27</td></tr>
  <tr><td>GPT-4o</td><td>81.78</td><td>86.00</td><td>62.80</td><td>67.62</td></tr>
  <tr><td>Claude3.5-sonnet</td><td>82.38</td><td>89.27</td><td>72.52</td><td>59.65</td></tr>
  <tr><td>LLaMA3.3-70B</td><td>69.49</td><td>89.05</td><td>73.73</td><td>75.22</td></tr>
</table>

Table 9: Length Instruction Following Results

<table>
  <tr>
    <th>Model</th>
    <th>ar_local</th>
    <th>ar_mt</th>
    <th>es_local</th>
    <th>es_mt</th>
    <th>ms_local</th>
    <th>ms_mt</th>
    <th>yo_local</th>
    <th>yo_mt</th>
    <th>zh_local</th>
    <th>zh_mt</th>
  </tr>
  <tr><td>Ministral-8B</td><td>7.85</td><td>7.50</td><td>39.30</td><td>36.70</td><td>25.40</td><td>26.35</td><td>12.10</td><td>12.60</td><td>39.25</td><td>34.65</td></tr>
  <tr><td>LLaMA3.1-8B</td><td>10.45</td><td>11.10</td><td>42.70</td><td>47.30</td><td>35.60</td><td>39.65</td><td>9.35</td><td>11.35</td><td>19.60</td><td>20.15</td></tr>
  <tr><td>Aya23-8B</td><td>37.15</td><td>34.10</td><td>28.95</td><td>31.95</td><td>25.80</td><td>25.00</td><td>10.35</td><td>8.95</td><td>26.15</td><td>20.50</td></tr>
  <tr><td>LLaMA3-8B</td><td>13.25</td><td>11.85</td><td>50.55</td><td>56.45</td><td>35.60</td><td>39.55</td><td>9.45</td><td>11.35</td><td>12.50</td><td>13.85</td></tr>
  <tr><td>Qwen2-7B</td><td>34.55</td><td>34.85</td><td>41.40</td><td>45.05</td><td>32.65</td><td>34.55</td><td>10.80</td><td>10.20</td><td>46.60</td><td>44.60</td></tr>
  <tr><td>EuroLLM-9B</td><td>45.45</td><td>40.65</td><td>50.50</td><td>48.85</td><td>29.65</td><td>27.45</td><td>14.05</td><td>11.85</td><td>46.85</td><td>41.55</td></tr>
  <tr><td>Aya23-35B</td><td>47.15</td><td>44.55</td><td>44.00</td><td>46.20</td><td>38.85</td><td>37.80</td><td>11.35</td><td>10.45</td><td>38.80</td><td>45.70</td></tr>
  <tr><td>LLaMA3.1-70B</td><td>19.70</td><td>20.15</td><td>59.65</td><td>59.60</td><td>52.65</td><td>55.05</td><td>21.05</td><td>13.50</td><td>31.15</td><td>32.80</td></tr>
  <tr><td>LLaMA3-70B</td><td>25.30</td><td>21.70</td><td>67.85</td><td>70.95</td><td>52.30</td><td>55.05</td><td>21.05</td><td>13.50</td><td>22.60</td><td>18.05</td></tr>
  <tr><td>Aya-expanse-8B</td><td>50.35</td><td>48.05</td><td>53.60</td><td>49.70</td><td>37.80</td><td>38.40</td><td>14.90</td><td>12.55</td><td>52.40</td><td>51.60</td></tr>
  <tr><td>Qwen2.5-7B</td><td>42.05</td><td>37.70</td><td>57.95</td><td>63.10</td><td>54.25</td><td>50.95</td><td>13.05</td><td>10.70</td><td>59.85</td><td>58.70</td></tr>
  <tr><td>Gemma2-9B</td><td>55.95</td><td>53.80</td><td>62.85</td><td>65.30</td><td>59.70</td><td>63.40</td><td>21.55</td><td>19.05</td><td>57.85</td><td>55.35</td></tr>
  <tr><td>Aya-expanse-32B</td><td>65.05</td><td>61.75</td><td>64.25</td><td>63.60</td><td>58.15</td><td>55.90</td><td>14.05</td><td>14.35</td><td>65.70</td><td>65.20</td></tr>
  <tr><td>Qwen2.5-14B</td><td>61.10</td><td>56.85</td><td>67.15</td><td>70.90</td><td>60.90</td><td>60.55</td><td>14.70</td><td>11.35</td><td>65.60</td><td>66.35</td></tr>
  <tr><td>Gemma2-27B</td><td>60.20</td><td>57.40</td><td>68.10</td><td>69.40</td><td>61.45</td><td>67.90</td><td>27.90</td><td>22.65</td><td>63.40</td><td>59.05</td></tr>
  <tr><td>Qwen2-72B</td><td>64.80</td><td>60.10</td><td>71.70</td><td>70.95</td><td>65.90</td><td>70.05</td><td>12.10</td><td>10.60</td><td>69.60</td><td>67.20</td></tr>
  <tr><td>Mistral-Large</td><td>66.55</td><td>63.95</td><td>75.70</td><td>76.25</td><td>68.00</td><td>71.40</td><td>11.35</td><td>10.70</td><td>69.85</td><td>66.90</td></tr>
  <tr><td>LLaMA3.3-70B</td><td>50.75</td><td>48.05</td><td>79.50</td><td>78.85</td><td>76.10</td><td>71.60</td><td>29.95</td><td>20.80</td><td>66.55</td><td>62.75</td></tr>
  <tr><td>Qwen2.5-72B</td><td>69.80</td><td>66.55</td><td>79.20</td><td>80.95</td><td>74.10</td><td>76.15</td><td>19.60</td><td>17.00</td><td>74.65</td><td>73.10</td></tr>
  <tr><td>Qwen-Plus</td><td>78.25</td><td>75.75</td><td>83.00</td><td>82.70</td><td>78.50</td><td>75.25</td><td>17.45</td><td>12.45</td><td>77.30</td><td>77.20</td></tr>
  <tr><td>Qwen-Max</td><td>75.40</td><td>71.80</td><td>78.20</td><td>81.25</td><td>77.65</td><td>75.40</td><td>31.65</td><td>21.25</td><td>73.15</td><td>74.70</td></tr>
  <tr><td>GPT-4o-mini</td><td>71.90</td><td>71.25</td><td>77.25</td><td>78.30</td><td>72.95</td><td>75.75</td><td>44.85</td><td>48.45</td><td>74.50</td><td>72.35</td></tr>
  <tr><td>GPT-4o</td><td>74.20</td><td>72.90</td><td>78.35</td><td>79.40</td><td>74.95</td><td>80.10</td><td>51.75</td><td>56.85</td><td>77.15</td><td>73.40</td></tr>
  <tr><td>Gemini1.5-flash</td><td>79.10</td><td>71.55</td><td>78.30</td><td>79.20</td><td>80.30</td><td>79.85</td><td>53.30</td><td>53.40</td><td>74.60</td><td>72.05</td></tr>
  <tr><td>Claude3.5-sonnet</td><td>75.90</td><td>75.40</td><td>81.85</td><td>81.50</td><td>78.65</td><td>80.90</td><td>55.45</td><td>48.35</td><td>75.40</td><td>72.30</td></tr>
  <tr><td>Avg</td><td>51.29</td><td>48.77</td><td>63.27</td><td>64.58</td><td>56.31</td><td>57.36</td><td>22.13</td><td>19.77</td><td>55.24</td><td>53.60</td></tr>
</table>

Table 10: Prompt Accuracy results on Marco-Bench-MIF for specific languages.
<table>
  <tr>
    <th>Model</th>
    <th>ar_local</th>
    <th>ar_mt</th>
    <th>es_local</th>
    <th>es_mt</th>
    <th>ms_local</th>
    <th>ms_mt</th>
    <th>yo_local</th>
    <th>yo_mt</th>
    <th>zh_local</th>
    <th>zh_mt</th>
  </tr>
  <tr>
    <td>Aya23-8B</td>
    <td>63.45</td>
    <td>62.55</td>
    <td>58.40</td>
    <td>42.75</td>
    <td>39.20</td>
    <td>38.20</td>
    <td>21.40</td>
    <td>18.40</td>
    <td>53.10</td>
    <td>48.35</td>
  </tr>
  <tr>
    <td>LLaMA3-1.8B</td>
    <td>37.95</td>
    <td>37.50</td>
    <td>68.15</td>
    <td>57.85</td>
    <td>47.70</td>
    <td>52.00</td>
    <td>23.35</td>
    <td>26.25</td>
    <td>47.30</td>
    <td>49.35</td>
  </tr>
  <tr>
    <td>Ministral-SB</td>
    <td>35.80</td>
    <td>35.10</td>
    <td>67.50</td>
    <td>50.45</td>
    <td>41.10</td>
    <td>43.85</td>
    <td>29.15</td>
    <td>28.80</td>
    <td>66.70</td>
    <td>64.70</td>
  </tr>
  <tr>
    <td>LLaMA3-8B</td>
    <td>44.25</td>
    <td>42.95</td>
    <td>75.65</td>
    <td>69.30</td>
    <td>47.80</td>
    <td>51.95</td>
    <td>23.40</td>
    <td>26.05</td>
    <td>47.05</td>
    <td>46.00</td>
  </tr>
  <tr>
    <td>Aya23-35B</td>
    <td>70.80</td>
    <td>70.00</td>
    <td>69.95</td>
    <td>57.15</td>
    <td>51.00</td>
    <td>50.40</td>
    <td>20.85</td>
    <td>19.60</td>
    <td>65.80</td>
    <td>68.80</td>
  </tr>
  <tr>
    <td>LLaMA3-1.70B</td>
    <td>45.05</td>
    <td>46.80</td>
    <td>78.40</td>
    <td>67.30</td>
    <td>64.30</td>
    <td>65.20</td>
    <td>36.80</td>
    <td>31.30</td>
    <td>58.45</td>
    <td>59.60</td>
  </tr>
  <tr>
    <td>Qwen2-7B</td>
    <td>66.60</td>
    <td>66.20</td>
    <td>70.20</td>
    <td>57.75</td>
    <td>48.70</td>
    <td>52.00</td>
    <td>27.25</td>
    <td>21.50</td>
    <td>72.50</td>
    <td>71.35</td>
  </tr>
  <tr>
    <td>Eurol.LM-9B</td>
    <td>72.90</td>
    <td>68.40</td>
    <td>75.60</td>
    <td>61.20</td>
    <td>44.95</td>
    <td>44.45</td>
    <td>29.40</td>
    <td>26.50</td>
    <td>73.05</td>
    <td>69.75</td>
  </tr>
  <tr>
    <td>Aya-expans-8B</td>
    <td>73.10</td>
    <td>71.70</td>
    <td>75.85</td>
    <td>63.25</td>
    <td>49.25</td>
    <td>51.10</td>
    <td>27.00</td>
    <td>24.50</td>
    <td>74.60</td>
    <td>74.30</td>
  </tr>
  <tr>
    <td>LLaMA3-70B</td>
    <td>57.85</td>
    <td>54.75</td>
    <td>85.10</td>
    <td>81.20</td>
    <td>63.95</td>
    <td>65.25</td>
    <td>36.75</td>
    <td>31.35</td>
    <td>58.45</td>
    <td>53.55</td>
  </tr>
  <tr>
    <td>Qwen2.5-7B</td>
    <td>68.65</td>
    <td>65.45</td>
    <td>77.70</td>
    <td>71.35</td>
    <td>64.65</td>
    <td>64.95</td>
    <td>26.60</td>
    <td>25.55</td>
    <td>81.25</td>
    <td>79.10</td>
  </tr>
  <tr>
    <td>Qwen2.5-14B</td>
    <td>79.35</td>
    <td>76.65</td>
    <td>82.20</td>
    <td>76.55</td>
    <td>68.40</td>
    <td>66.60</td>
    <td>27.85</td>
    <td>24.20</td>
    <td>83.95</td>
    <td>83.65</td>
  </tr>
  <tr>
    <td>Aya-expans-32B</td>
    <td>84.20</td>
    <td>81.35</td>
    <td>83.55</td>
    <td>75.50</td>
    <td>71.50</td>
    <td>69.25</td>
    <td>24.25</td>
    <td>23.40</td>
    <td>85.05</td>
    <td>84.00</td>
  </tr>
  <tr>
    <td>Gemma2-9B</td>
    <td>79.30</td>
    <td>76.50</td>
    <td>82.40</td>
    <td>76.30</td>
    <td>72.25</td>
    <td>75.05</td>
    <td>38.75</td>
    <td>34.75</td>
    <td>79.80</td>
    <td>77.75</td>
  </tr>
  <tr>
    <td>Qwen2-72B</td>
    <td>83.25</td>
    <td>80.35</td>
    <td>87.15</td>
    <td>79.05</td>
    <td>77.40</td>
    <td>79.00</td>
    <td>26.85</td>
    <td>24.75</td>
    <td>86.35</td>
    <td>84.25</td>
  </tr>
  <tr>
    <td>Gemma2-27B</td>
    <td>81.55</td>
    <td>78.75</td>
    <td>85.50</td>
    <td>79.60</td>
    <td>74.30</td>
    <td>77.70</td>
    <td>40.70</td>
    <td>36.15</td>
    <td>82.90</td>
    <td>79.20</td>
  </tr>
  <tr>
    <td>Mistral-Large</td>
    <td>83.75</td>
    <td>82.75</td>
    <td>88.25</td>
    <td>83.30</td>
    <td>78.05</td>
    <td>80.85</td>
    <td>24.25</td>
    <td>24.05</td>
    <td>86.70</td>
    <td>84.45</td>
  </tr>
  <tr>
    <td>LLaMA3-3.70B</td>
    <td>73.15</td>
    <td>71.05</td>
    <td>89.65</td>
    <td>83.95</td>
    <td>81.60</td>
    <td>79.00</td>
    <td>45.15</td>
    <td>36.45</td>
    <td>81.85</td>
    <td>78.85</td>
  </tr>
  <tr>
    <td>Qwen2.5-72B</td>
    <td>85.80</td>
    <td>83.70</td>
    <td>90.25</td>
    <td>87.05</td>
    <td>82.50</td>
    <td>83.55</td>
    <td>35.60</td>
    <td>35.15</td>
    <td>88.85</td>
    <td>87.60</td>
  </tr>
  <tr>
    <td>Qwen-Plus</td>
    <td>90.15</td>
    <td>88.30</td>
    <td>92.40</td>
    <td>88.65</td>
    <td>85.90</td>
    <td>83.70</td>
    <td>32.10</td>
    <td>29.00</td>
    <td>90.15</td>
    <td>89.50</td>
  </tr>
  <tr>
    <td>Qwen-Max</td>
    <td>88.25</td>
    <td>86.00</td>
    <td>90.00</td>
    <td>87.05</td>
    <td>84.15</td>
    <td>83.60</td>
    <td>47.75</td>
    <td>39.70</td>
    <td>88.10</td>
    <td>88.40</td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td>87.50</td>
    <td>86.15</td>
    <td>89.20</td>
    <td>84.70</td>
    <td>81.70</td>
    <td>83.30</td>
    <td>58.05</td>
    <td>65.20</td>
    <td>88.45</td>
    <td>87.00</td>
  </tr>
  <tr>
    <td>Gemini1.5-flash</td>
    <td>90.75</td>
    <td>86.55</td>
    <td>90.10</td>
    <td>86.30</td>
    <td>87.20</td>
    <td>86.50</td>
    <td>67.75</td>
    <td>67.65</td>
    <td>88.65</td>
    <td>86.50</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>88.15</td>
    <td>86.70</td>
    <td>90.15</td>
    <td>86.60</td>
    <td>83.80</td>
    <td>87.00</td>
    <td>66.50</td>
    <td>71.20</td>
    <td>90.00</td>
    <td>88.00</td>
  </tr>
  <tr>
    <td>Claude3.5-sonnet</td>
    <td>88.95</td>
    <td>88.10</td>
    <td>92.20</td>
    <td>87.35</td>
    <td>86.20</td>
    <td>87.85</td>
    <td>69.15</td>
    <td>63.25</td>
    <td>89.45</td>
    <td>87.60</td>
  </tr>
  <tr>
    <th>Avg</th>
    <th>72.82</th>
    <th>70.97</th>
    <th>81.42</th>
    <th>73.66</th>
    <th>67.10</th>
    <th>68.09</th>
    <th>36.27</th>
    <th>34.19</th>
    <th>76.34</th>
    <th>74.86</th>
  </tr>
</table>

Table 11: Instruction Accuracy results on Marco–Bench–MIF for specific languages.

A.2 Detailed Results of Performance on Localised Data vs MT Data

From the results in Table 10 and Table 11, we can see that the performance of most LLMs on localized data consistently outperforms MT data across all languages and model scales, but the gap varies significantly by language family. For high-resource languages like Spanish (es), the difference is minimal (prompt-level: 63.27% vs 64.58%), while low-resource languages like Yoruba (yo) show substantial disparities (prompt-level: 22.13% vs 19.77%).

Furthermore, our evaluation results show critical limitations in using machine-translated data for evaluating multilingual instruction-following capabilities. Localized data exposes significant gaps in LLMs’ ability to handle culturally and linguistically fine-grained instructions, which are often overlooked in MT data as translated could contain unexpected errors making it hard for LLMs to understand the instruction. For example, in Yoruba (yo), localized data shows a 7.1% higher prompt-level accuracy than MT data for Claude3.5-sonnet, highlighting that MT evaluations underestimate model performance especially in low-resource languages. The divergence between localized and MT results increases with task complexity. While MT data performs comparably for simple constraints (e.g., Spanish es instruction-level: 73.66% MT vs 81.42% localized), it fails to capture the challenges of culturally adapted multi-instruction prompts (e.g., Arabic ar prompt-level: 51.29% localized vs 48.77% MT). This suggests that MT benchmarks overestimate LLMs’ ability to follow instructions in more complex multilingual scenarios. We found that model scale also affects the performance of LLMs: smaller models (e.g., Aya23-8B) show inconsistent localization benefits (ar: +0.9% prompt-level, -20.65% instruction-level), while larger models (e.g., Claude3.5-sonnet) demonstrate more stable gains (yo: +7.1% prompt-level, +5.9% instruction-level). This suggests that scale amplifies localization advantages, particularly for low-resource languages. The evaluation results underscores the need for localized instruction-following data to precisely assess LLMs’ instruction-following capabilities

Reviewer paper summaries: The M-IFEval benchmark expands instruction-following assessment to over 30 languages (from 6 language families), developing a framework for adapting prompts, instructions, and constraints to diverse linguistic contexts, combining automated and manual localization methods. They conduct evaluation of 20+ LLMs on the benchmark. The framework consists of three main stages towards preserving instruction constraints: preprocessing (constraint categorization and data filtering), translation and cultural localization, and post-processing (error reduction and human reviews). Evaluation consists of checking for prompt and instruction level accuracy with a strict and loose metric. The paper is written well and lends an interesting study to the multilingual capabilities of many open and closed models.
Summary of strengths: - The paper presents the M-IFEval benchmark for instruction following with constraints in multiple languages, curated with a relevant methodology involving human review as well
- Multiple models are evaluated, giving a comprehensive overview of 20+ LLMs both open and closed source
- The evaluation methods look at constraint following in keeping with the predecessor benchmark IFEval
- The study has interesting findings which are useful to a broader audience
Summary of weaknesses: - Details of the human review for benchmark construction are possibly missing - it would strengthen the paper to discuss how the review was conducted (How many annotators for each data point? What were the instructions given to the reviewers? What was the inter-annotator agreement?)
- Figure 2 presents a line chart for language wise results which are not continuous variables, and is difficult to read. It presents some interesting results which could be presented better, possibly with size and open vs closed based distributions for the models evaluated.
Comments, suggestions and typos: Line 282 - Requires grammatical correction “We employ a LLM generates initial outputs which undergo evaluation”

Line 451 - “resutls”

Line 453 - “Closed-sourfce”
Overall assessment scores: 4