EXECUTE: A Multilingual Benchmark for LLM Token Understanding
Authors: Lukas Edman, Helmut Schmid, Alexander Fraser
Accepted at: acl25

EXECUTE: A Multilingual Benchmark for LLM Token Understanding

Lukas Edman1,3    Helmut Schmid2    Alexander Fraser1,3,4
1School of Computation, Information and Technology, TU Munich
2Center for Information and Language Processing, LMU Munich
3Munich Center for Machine Learning
4Munich Data Science Institute
lukas.edman@tum.de, schmid@cis.lmu.de

Abstract
The CUTE benchmark showed that LLMs struggle with character understanding in English. We extend it to more languages with diverse scripts and writing systems, introducing EXECUTE. Our simplified framework allows easy expansion to any language. Tests across multiple LLMs reveal that challenges in other languages are not always on the character level as in English. Some languages show word-level processing issues, some show no issues at all. We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs’ understanding of character components.

1 Introduction
LLMs perform well on many tasks but struggle when they are asked to manipulate character sequences, as shown by the CUTE benchmark (Edman et al., 2024). While CUTE tested Russian, showing this issue is not language-specific, it failed to consider other linguistic differences that may affect results. Language variation extends beyond script differences to writing system differences. English and Russian use alphabets. Other languages use Abugidas, where letters are not strictly ordered within syllables, or Abjads, which mark vowels with diacritics or not at all. Chinese uses a logographic script, where most words are just 1-2 characters long. Multilingual LLMs allocate tokens unevenly across languages: high-resource languages are well represented, but some low-resource languages are mainly processed at the byte level.

We explore these languages in our benchmark EXECUTE: the Expandable X(Cross)-Lingual Extension of CUTE.1 We mainly look at 8 languages, shown in Table 1, which vary in script, writing system, tokenization, and resourcedness. We also provide a framework for adding languages to make this benchmark easily expandable. In our results and analysis, we find that:

<table>
  <tr>
    <th>Language</th>
    <th>Script</th>
    <th>Writing System</th>
    <th>c/w</th>
    <th>t/w</th>
    <th>c/t</th>
  </tr>
  <tr>
    <td>Amharic</td>
    <td>Ge’ez</td>
    <td>Abugida</td>
    <td>3.71</td>
    <td>7.69</td>
    <td>0.48</td>
  </tr>
  <tr>
    <td>Arabic</td>
    <td>Arabic</td>
    <td>Abjad</td>
    <td>4.63</td>
    <td>2.43</td>
    <td>1.90</td>
  </tr>
  <tr>
    <td>Chinese</td>
    <td>Simpl. Han</td>
    <td>Logographic</td>
    <td>1.51</td>
    <td>1.25</td>
    <td>2.00</td>
  </tr>
  <tr>
    <td>English</td>
    <td>Latin</td>
    <td>Alphabet</td>
    <td>4.04</td>
    <td>1.32</td>
    <td>3.05</td>
  </tr>
  <tr>
    <td>Hindi</td>
    <td>Devanagari</td>
    <td>Abugida</td>
    <td>3.66</td>
    <td>2.80</td>
    <td>1.31</td>
  </tr>
  <tr>
    <td>Japanese</td>
    <td>Japanese</td>
    <td>Mixed</td>
    <td>1.54</td>
    <td>1.27</td>
    <td>1.22</td>
  </tr>
  <tr>
    <td>Korean</td>
    <td>Hangul</td>
    <td>Featural</td>
    <td>3.38</td>
    <td>2.71</td>
    <td>1.25</td>
  </tr>
  <tr>
    <td>Russian</td>
    <td>Cyrillic</td>
    <td>Alphabet</td>
    <td>5.06</td>
    <td>2.36</td>
    <td>2.14</td>
  </tr>
</table>

Table 1: CWT statistics of EXECUTE’s languages. c, w, and t denote characters, words, and tokens. c/w refers to the average characters per word. t is the average token count across the 5 tokenizers used by the models.

1. Benchmark results for non-English languages often differ from the English results.
2. The results correlate with the languages’ CWT (character-word-token) statistics (see Table 1).
3. Surprisingly, the less an LLM knows a language, the better it performs on EXECUTE.
4. LLMs struggle with understanding sub-character components (see Figure 1).

Our results provide more insight into how LLMs process tokens on different granularities.

2 Related Works
Our work builds upon the CUTE benchmark (Edman et al., 2024), which showed that LLMs struggle with character manipulation tasks. CUTE was mainly created for English but also included Russian tasks, showing similar results. Similar studies probe models to spell or modify text on the character level, but either first train the model (Izihak and Levy, 2022; Kaushal and Mahowald, 2022), or focus on other topics than orthography (Huang et al., 2023; Efrat et al., 2023).

Research on error correction, including spelling correction, has been done for many languages. Maxutov et al. (2024) found spelling correction to be “hard” for LLMs in Kazakh. Li et al. (2023) reported that LLMs perform worse than fine-tuned models for Chinese spelling correction. Similarly,
Figure 1: EXECUTE benchmark. Prompts shortened for brevity. Example of full prompt in Appendix D.

Kwon et al. (2023) showed that fine-tuned models outperform prompted LLMs for Arabic. Spelling correction requires both character-level and semantic knowledge to determine the correct replacement. EXECUTE, like CUTE, aims to remove contextual semantic understanding from the benchmark.

Our sub-character experiments build on work by Wu et al. (2025) who released a detailed analysis of the information in Chinese characters. Our character-to/from-radical tasks resemble theirs, but they focus on simplified Chinese, while we also examine traditional characters via Japanese Kanji.

Character-level LLMs have been proposed as a solution to CUTE and have been shown to outperform subword LLMs in Pagnoni et al. (2024).

3 Benchmark

Figure 1 exemplifies our EXECUTE benchmark. We use the same composition and manipulation tasks as CUTE but drop the similarity tasks which require static embeddings (such as word2vec) and fluent speakers to define similarity thresholds, which vary by language and lack clear criteria. Their removal makes EXECUTE easier to expand. Adding a new language \( X \) now only requires an English\( \rightarrow X \) translation system. As cross-language alignment is not crucial, translations do not need to be perfect: grammaticality is preferable but not necessary. We modify prompt examples and the dataset used, so English and Russian results differ from CUTE’s.\footnote{As our changes are minor, users of the English and Russian datasets should cite Edman et al. (2024).} Although perfect translations are not required, we have fluent speakers verify that most translations preserve meaning and grammar. Table 1 lists these languages, covering eight major scripts and all known writing systems. While some widely used languages (e.g. Spanish) are missing, their script is represented, and Appendix B shows the performance of languages using the same script is highly correlated.

We keep the prompt texts in English but use language-specific examples, since fully Russian prompts did not improve performance for Russian (Edman et al., 2024). It also ensures that the LLMs understand the task consistently across languages.

3.1 Data Preparation

We now describe the exact differences in preprocessing steps between our benchmark and the CUTE benchmark. Although there are several changes, we find that the scores from CUTE and EXECUTE are still largely comparable, as shown in Appendix A.

To start, we use an updated subset of 5000 stories from the TinyStories dataset (Eldan and Li, 2023), which used GPT-4 to produce outputs rather than the GPT-3.5 outputs, used in CUTE. We find this dataset to be cleaner (with no random foreign characters), and it is purported by the dataset authors to also be of higher quality. For non-English languages, we translate all the stories using Google Translate. At this point, for Chinese and Japanese, it is necessary to apply word segmentation. For Chinese, we use jieba\footnote{https://github.com/fxsjy/jieba}, and for Japanese we use nagisa.\footnote{https://github.com/taishi-i/nagisa}

We then generate a character set and vocabulary from the translated stories to use for our tasks. This is unlike what is used in CUTE, which predefined alphabets and vocabularies taken from the Trillion Word Corpus and Wikipedia. This change is necessary as it is more difficult to define a strict character set for some languages, and also more difficult to find a vocabulary.

In the CUTE benchmark, the vocabulary also removed words less than 3 characters to maintain a level of difficulty for the tasks. We remove this cutoff for Chinese, Japanese, and Korean, as it is too restrictive.

As the prompts are few-shot, we require language-specific examples in each prompt. For CUTE, these examples were created manually. Instead, we generate 4 additional examples in the same manner as our test set, with a few additional stipulations:

• At least 2 examples must use a word that contains duplicate letters.
• At least 1 example must operate on the duplicate letters when applicable.
• For the contains tasks, 2 examples must have the label “yes” and 2 “no”.

We specify the duplicate letter restrictions so that the LLM understands that it must modify all of the targeted characters. The first two restrictions were not applied for Chinese however, as it is exceedingly rare for a Chinese word to contain duplicate characters. The last restriction is intended to ensure the model is not biased to answering either “yes” or “no” due to its frequency in the examples, a phenomenon which has been shown to be problematic in Zhao et al. (2021).

Diacritics Abugidas such as Hindi have diacritics to mark vowel sounds, aspirations, and nasalizations. Due to the complex rules surrounding valid diacritics, which also vary between languages, we opt to consider each “character” as the letter plus any diacritics attached, also known as the grapheme. This is already the case for Amharic, as the diacritics have become fused with consonants in the Ge’ez script itself.

3.2 Sub-Character Experiments

Chinese, Japanese, and to a lesser extent Korean, have few characters per word, so we add language-specific tasks to assess their understanding of character components.

In Chinese, each character can be broken down into parts known as Kangxi radicals. An example of a decomposition is: 晚 → 晒日免, where 晒 indicates that 日 should be placed to the left of 免. The radicals often have a related meaning to the composite: 晚 means evening, 日 means sun and 免 means avoid. Japanese Kanji characters originate from traditional Chinese characters and can also be decomposed into radicals. Korean Hangul characters denote syllables and can be split into Jamo, which correspond to phonemes. For example, 들 (dul) becomes ㄷ (d), ㅡ (u), and ㄹ (l).

We test the LLMs’ ability to compose and decompose CJK characters into their components. For Chinese and Japanese, we ask the model to split characters into Kangxi radicals, and vice versa.5 Similarly, we decompose Hangul characters to Jamo and vice versa. These tasks are analogous to the spelling and inverse_spelling tasks. We further add a task (similar to contains) which asks if a character contains a Kangxi radical or Jamo.

Japanese can either be written with Kanji characters or with phonetic Hiragana characters. We test LLMs’ ability to convert Kanji in Appendix C.

3.3 Models

We test 5 popular open-source multilingual LLMs: Aya Expanse, Gemma 2, Llama 3.1 and 3.3, Qwen 2.5, and Mistral (Dang et al., 2024; Gemma Team et al., 2024; Dubey et al., 2024; Owen et al., 2025; Jiang et al., 2023). Their sizes range from 7B to 70B parameters, and their vocabularies contain between 128k and 256k tokens.

4 Results

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">Eng</th>
    <th colspan="3"> </th>
  </tr>
  <tr>
    <th>Amh</th>
    <th>Tzm</th>
    <th>Sat</th>
    <th>Cipher</th>
    <th>Byte</th>
    <th>Reg</th>
  </tr>
  <tr>
    <td>Spell</td>
    <td>96.3</td>
    <td>100.0</td>
    <td>97.6</td>
    <td>100.0</td>
    <td>85.0</td>
    <td>99.5</td>
  </tr>
  <tr>
    <td>Inv Spell</td>
    <td>99.8</td>
    <td>100.0</td>
    <td>99.3</td>
    <td>100.0</td>
    <td>0.0</td>
    <td>99.6</td>
  </tr>
  <tr>
    <td>Cont Char</td>
    <td>91.8</td>
    <td>100.0</td>
    <td>98.0</td>
    <td>98.6</td>
    <td>82.8</td>
    <td>75.7</td>
  </tr>
  <tr>
    <td>Cont Word</td>
    <td>99.6</td>
    <td>99.2</td>
    <td>98.9</td>
    <td>99.0</td>
    <td>96.7</td>
    <td>99.9</td>
  </tr>
  <tr>
    <td>Ins Char</td>
    <td>97.8</td>
    <td>97.8</td>
    <td>98.2</td>
    <td>98.6</td>
    <td>20.9</td>
    <td>13.5</td>
  </tr>
  <tr>
    <td>Ins Word</td>
    <td>92.8</td>
    <td>94.3</td>
    <td>91.9</td>
    <td>97.1</td>
    <td>1.4</td>
    <td>96.6</td>
  </tr>
  <tr>
    <td>Del Char</td>
    <td>97.6</td>
    <td>99.7</td>
    <td>98.7</td>
    <td>98.9</td>
    <td>78.8</td>
    <td>67.5</td>
  </tr>
  <tr>
    <td>Del Word</td>
    <td>97.6</td>
    <td>76.2</td>
    <td>88.8</td>
    <td>95.6</td>
    <td>3.7</td>
    <td>96.5</td>
  </tr>
  <tr>
    <td>Sub Char</td>
    <td>96.6</td>
    <td>98.4</td>
    <td>98.3</td>
    <td>95.5</td>
    <td>61.5</td>
    <td>51.4</td>
  </tr>
  <tr>
    <td>Sub Word</td>
    <td>96.6</td>
    <td>98.4</td>
    <td>98.0</td>
    <td>98.4</td>
    <td>59.1</td>
    <td>59.5</td>
  </tr>
  <tr>
    <td>Swap Char</td>
    <td>93.7</td>
    <td>97.6</td>
    <td>92.8</td>
    <td>98.3</td>
    <td>29.0</td>
    <td>12.7</td>
  </tr>
  <tr>
    <td>Swap Word</td>
    <td>97.3</td>
    <td>87.9</td>
    <td>90.0</td>
    <td>95.9</td>
    <td>6.6</td>
    <td>90.9</td>
  </tr>
  <tr>
    <td>Avg</td>
    <td>96.4</td>
    <td>95.6</td>
    <td>95.2</td>
    <td>98.0</td>
    <td>39.4</td>
    <td>75.2</td>
  </tr>
</table>

Table 2: Llama 3.3 on low-resource languages.

We first examine results by language, showing the best model performance for each in Figure 2.6 Russian and Arabic results resemble English results. Hindi and Korean perform better at the word level than the character level, though the gap is smaller than for English, with stronger results in character-level insertion and swapping. Japanese and Chinese perform better on the character level, which is expected since each character is a word or almost a word. However, word-level tasks may simply be harder in these languages, as they require modifying multiple tokens instead of just one.

4.1 Amharic and Low-Resourcedness

Amharic stands out from the rest of the results in that the performance is nearly perfect in the best-case scenario. This is particularly surprising as Amharic is the lowest-resource language of the 8, and most characters are split into bytes

5One can further split Kangxi radicals down to strokes, but this showed very poor performance in initial tests.
6We show the results per task, as well as results for Aya and Mistral, in Appendix E.
by the tokenizers, meaning each character is represented by 3 tokens. We suspect that the good performance might actually be because of this low-resourcedness. As seen in Edman et al. (2024), and also observed in this work, LLMs are biased to generating real words and grammatical sentences. Their lack of understanding of Amharic might weaken this bias.

<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="2">Gemma 2</th>
    <th colspan="2">Llama 3.1</th>
    <th colspan="2">Llama 3.3</th>
    <th colspan="2">Qwen 2.5</th>
  </tr>
  <tr>
    <th>9B</th>
    <th>27B</th>
    <th>8B</th>
    <th>70B</th>
    <th>70B</th>
    <th>7B</th>
    <th>32B</th>
  </tr>
  <tr>
    <td>Amh</td>
    <td>80.5</td>
    <td>85.3</td>
    <td>75.7</td>
    <td><b>95.9</b></td>
    <td>96.4</td>
    <td>41.9</td>
    <td>74.4</td>
  </tr>
  <tr>
    <td>Ara</td>
    <td>51.6</td>
    <td>62.3</td>
    <td>52.1</td>
    <td>68.1</td>
    <td>67.8</td>
    <td>47.2</td>
    <td><b>68.6</b></td>
  </tr>
  <tr>
    <td>Zho</td>
    <td>70.2</td>
    <td>74.4</td>
    <td>71.3</td>
    <td>81.1</td>
    <td>79.7</td>
    <td>70.4</td>
    <td><b>85.6</b></td>
  </tr>
  <tr>
    <td>Eng</td>
    <td>66.8</td>
    <td>68.1</td>
    <td>61.9</td>
    <td>75.7</td>
    <td>75.2</td>
    <td>62.1</td>
    <td>77.3</td>
  </tr>
  <tr>
    <td>Hin</td>
    <td>67.9</td>
    <td>47.1</td>
    <td>43.8</td>
    <td>54.0</td>
    <td>56.4</td>
    <td>43.5</td>
    <td><b>86.2</b></td>
  </tr>
  <tr>
    <td>Jpn</td>
    <td>60.1</td>
    <td>65.2</td>
    <td>58.8</td>
    <td>73.1</td>
    <td>74.6</td>
    <td>62.1</td>
    <td><b>77.9</b></td>
  </tr>
  <tr>
    <td>Kor</td>
    <td>73.6</td>
    <td><b>80.8</b></td>
    <td>62.1</td>
    <td>76.9</td>
    <td>76.1</td>
    <td>60.2</td>
    <td><b>80.8</b></td>
  </tr>
  <tr>
    <td>Rus</td>
    <td>53.6</td>
    <td>62.6</td>
    <td>51.0</td>
    <td>67.8</td>
    <td>67.8</td>
    <td>52.1</td>
    <td><b>71.2</b></td>
  </tr>
  <tr>
    <td>Avg</td>
    <td>62.8</td>
    <td>68.7</td>
    <td>59.6</td>
    <td>74.1</td>
    <td>74.3</td>
    <td>54.9</td>
    <td><b>77.5</b></td>
  </tr>
</table>

Table 3: Average score per language. Best in bold.

We provide further evidence that language knowledge inversely correlates with EXECUTE performance by adding two low-resource languages, Tamazight and Santali. Their unique scripts (Tifinagh and Ol Chiki) are not used by any higher resource languages, forcing LLM tokenizers to operate at the byte level. These languages were likely seen rarely, if ever, during training. We compare results on Amharic’s best-performing model, Llama 3.3. Additionally, we test two variations of English: one encodes text using a cipher that maps Latin to Amharic characters, and the other forces the inputs to be byte-level (retaining the Latin alphabet). These experiments assess whether byte-level operation alone improves performance or if eliminating English recognition via ciphering is also necessary. We expect ciphered English to perform similarly to Amharic. Table 2 shows that Llama achieves near-perfect results in the low-resource languages, as well as the ciphered English. Byte-level English improves character tasks but fails on word tasks, partly due to bias. The model rarely sees English at the byte level except in social media, leading to random casing and antspeak (extra spacing) in the output. Degenerate output (e.g. “1 1 1 ...”) also occurs. Some fine-tuning with this byte-level approach would likely increase performance considerably.

Does Amharic’s near-perfect score mean character- and word-level processing is solved? No, it shows LLMs can perform arbitrary manipulations but are hampered by their language understanding. As training data increases, Amharic performance will likely decline. So, this benchmark should complement standard NLU benchmarks for a complete assessment.

4.2 Language Clusters

Table 1 groups languages with similar CWT statistics into five categories: 1) Arabic & Russian, 2) Hindi & Korean, 3) Japanese & Chinese, 4) Amharic, and 5) English. Their similar benchmark performance suggests that segmentation, whether natural or from tokenization, impacts results. As expected, the statistics of Tamazight (4.37, 8.83, 0.49) and Santali (3.54, 8.38, 0.42) closely align with Amharic.

4.3 Model Performance

Table 3 shows model performance. Larger models generally perform better. However, this trend does not hold across model families, as Qwen 2.5 (32B) outperforms the larger 70B Llama 3 models. Llama 3.3, despite its stronger performance than Llama 3.1 on standard benchmarks, performs similarly here.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="4">Aya Expans</th>
    <th colspan="4">Gemma 2</th>
    <th colspan="4">Llama 3.1</th>
    <th colspan="4">Llama 3.3</th>
    <th colspan="4">Qwen 2.5</th>
    <th colspan="4">Mistral</th>
  </tr>
  <tr>
    <th>8B</th><th>32B</th><th>9B</th><th>27B</th>
    <th>8B</th><th>70B</th><th>70B</th><th>70B</th>
    <th>7B</th><th>32B</th><th>8B</th><th>24B</th>
    <th>7B</th><th>32B</th><th>8B</th><th>24B</th>
    <th>7B</th><th>32B</th><th>8B</th><th>24B</th>
  </tr>
  <tr>
    <th rowspan="3">Zho</th>
    <td>Char to Rad</td><td>0.0</td><td>2.0</td><td>0.0</td><td>0.8</td><td>0.0</td><td>1.4</td><td>1.8</td><td>1.4</td><td>16.4</td><td>0.8</td><td>3.5</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Rad to Char</td><td>1.4</td><td>8.2</td><td>2.5</td><td>0.8</td><td>2.5</td><td>10.7</td><td>11.9</td><td>7.6</td><td>22.8</td><td>2.5</td><td>8.2</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Contains Rad</td><td>55.4</td><td>65.5</td><td>81.1</td><td>79.3</td><td>69.4</td><td>73.5</td><td>72.9</td><td>68.0</td><td>78.8</td><td>62.2</td><td>74.5</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <th rowspan="3">Jpn</th>
    <td>Char to Rad</td><td>0.0</td><td>0.7</td><td>0.7</td><td>0.0</td><td>0.0</td><td>0.0</td><td>2.2</td><td>2.2</td><td>9.2</td><td>0.4</td><td>3.7</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Rad to Char</td><td>2.6</td><td>8.5</td><td>2.2</td><td>0.4</td><td>1.9</td><td>13.7</td><td>13.3</td><td>5.2</td><td>20.3</td><td>1.5</td><td>7.4</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Contains Rad</td><td>57.9</td><td>61.6</td><td>86.4</td><td>72.7</td><td>69.7</td><td>73.1</td><td>76.0</td><td>73.4</td><td>76.0</td><td>65.7</td><td>83.4</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <th rowspan="3">Kor</th>
    <td>Hangul to Jamo</td><td>7.5</td><td>48.1</td><td>54.6</td><td>65.3</td><td>24.5</td><td>48.8</td><td>45.6</td><td>24.7</td><td>57.4</td><td>35.6</td><td>66.4</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Jamo to Hangul</td><td>24.7</td><td>49.0</td><td>47.2</td><td>63.9</td><td>41.7</td><td>24.5</td><td>24.3</td><td>25.9</td><td>42.2</td><td>28.1</td><td>51.5</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
  <tr>
    <td>Contains Jamo</td><td>63.7</td><td>76.2</td><td>93.2</td><td>96.6</td><td>92.1</td><td>87.5</td><td>90.3</td><td>75.3</td><td>93.4</td><td>73.2</td><td>88.7</td>
    <td></td><td></td><td></td><td></td><td></td><td></td><td></td><td></td>
  </tr>
</table>

Table 4: Sub-character-level results on CJK languages. Best in bold.

Edman et al. (2024) found that more training data improved results on CUTE, but we find no such trend. Among 7-9B models, Gemma was trained on 8T tokens, Llama on 15T, and Qwen on 18T (Gemma Team et al., 2024; Qwen et al., 2025; Dubey et al., 2024), yet their performance is inversely correlated. While this may be coincidental, results on Amharic, Tamazight, and Santali raise doubts about whether more training data improves performance on this benchmark.

4.4 Sub-Character Performance

Table 4 shows sub-character results. For Japanese and Chinese, models struggle to translate characters to and from their radical components but perform better on the Contains task, as it only requires identifying one radical. While some characters, like 晚 (evening), have components that clearly contribute to meaning, others are more ambiguous. For example, 木 (tree) is likely easier for models to identify in 樟 (camphor tree) compared to 章 (chapter, seal).

LLMs are notably better at converting between Hangul and Jamo, likely due to Hangul’s simpler structure or its more frequent decomposition in training data. However, the conversion still falls short of the near-perfect scores seen in the main Spelling and Inverse Spelling tasks.

5 Conclusion

We present a multilingual, multi-script extension of the CUTE benchmark to test token understanding in a variety of languages. The benchmark is designed to be easily expanded to new languages, allowing the token understanding of LLMs to be tested in any language. Our findings show that manipulation on the character level is challenging in some non-English languages, but word-level manipulation is challenging for some languages too. Understanding the components of characters in Chinese, Japanese, and Korean is also lacking. The performance of a language can be somewhat predicted by its character-word-token ratios. Surprisingly, LLMs perform better on lower-resourced languages, due to their knowledge of high-resourced languages acting as a bias against the benchmark’s tasks. While Edman et al. (2024) hypothesized that character-level models would be promising for solving the CUTE benchmark, EXECUTE demonstrates an additional need for debiasing models so they can temporarily forget what they know about a language.

6 Limitations

We limit ourselves to 8 languages for the majority of this work. While we argue that the languages in the same scripts as the ones tested will likely have similar results, pointing to the correlation in results between English, Spanish, German, and Xhosa in Appendix B, we cannot know for sure without testing them all. Several other scripts are not covered which may have differing performances.

We also do not test very large language models above 70B parameters due to compute constraints. The CUTE benchmark added scores for the 405B parameter Llama 3.1 and found it made improvements across the board, but was still lacking on character-level insertion and swapping. We would expect similar improvements for our English results, but it is unclear how it would perform for other languages.

7 Acknowledgments

The work was supported by the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (grant agreement No. 101113091) and by the German Research Foundation (DFG; grant FR 2829/7-1).
References

John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, et al. 2024. Aya expanse: Combining research breakthroughs for a new multilingual frontier. arXiv preprint arXiv:2412.04261.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Lukas Edman, Helmut Schmid, and Alexander Fraser. 2024. CUTE: Measuring LLMs’ understanding of their tokens. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3017–3026, Miami, Florida, USA. Association for Computational Linguistics.

Avia Efrat, Or Honovich, and Omer Levy. 2023. LMentry: A language model benchmark of elementary language tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10476–10501, Toronto, Canada. Association for Computational Linguistics.

Ronen Eldan and Yuanzhi Li. 2023. Tinystories: How small can language models be and still speak coherent english? Preprint, arXiv:2305.07759.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussent, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Taffi, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantanu Thakoor, Jean-Bastien Grill, Behnam Neyshabur, Olivier Bachem, Alanna Walton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coenen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Danila Sinopalnikov, David Weinberger, Dimple Vijaykumar, Dominika Rogozinska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Erica Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Raskin, Gary Wei, Glenn Cameron, Gus Martins, Hadi Hashemi, Hanna Klimeczak-Plucińska, Harleen Batra, Harsh Dhand, Ivan Nardini, Jacinda Mein, Jack Zhou, James Svensson, Jeff Stanway, Jetha Chan, Jin Peng Zhou, Joana Carrasqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersfoort, Josh Gordon, Josh Lipschultz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikeya Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kiranbir Sodhia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Sifre, Lena Heuermann, Letícia Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manvinder Singh, Mark Iverson, Martin Görner, Mat Velloso, Mateo Wirth, Matt Davidson, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynihan, Ming Zhang, Minsuk Kahng, Minwoo Park, Mof Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahltinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshir Rokni, Rishabh Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Cogan, Sarah Perrin, Sébastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulsee Doshi, Vihan Jain, Vikas Yadav, Vilobh Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, WooHyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotrutz, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warkentin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, Jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kennedy, Robert Dadashi, and Alek Andreev. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118.

Jing Huang, Zhengxuan Wu, Kyle Mahowald, and Christopher Potts. 2023. Inducing character-level structure in subword-based language models with type-level interchange intervention training. In Findings of the Association for Computational Linguistics: ACL 2023, pages 12163–12180, Toronto, Canada. Association for Computational Linguistics.

Itay Itzhak and Omer Levy. 2022. Models in a spelling bee: Language models implicitly learn the character composition of tokens. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5061–5068, Seattle, United States. Association for Computational Linguistics.

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825.

Ayush Kaushal and Kyle Mahowald. 2022. What do tokens know about their characters and how do they know it? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2487–2507, Seattle, United States. Association for Computational Linguistics.
Sang Kwon, Gagan Bhatia, El Moatez Billah Nagoudi, and Muhammad Abdul-Mageed. 2023. Beyond English: Evaluating LLMs for Arabic grammatical error correction. In Proceedings of ArabicNLP 2023, pages 101–119, Singapore (Hybrid). Association for Computational Linguistics.

Yinghui Li, Haojing Huang, Shirong Ma, Yong Jiang, Yangning Li, Feng Zhou, Hai-Tao Zheng, and Qingyu Zhou. 2023. On the (in)effectiveness of large language models for chinese text correction. Preprint, arXiv:2307.09007.

Akylbek Maxutov, Ayan Myrzakhmet, and Pavel Braslavski. 2024. Do LLMs speak Kazakh? a pilot evaluation of seven models. In Proceedings of the First Workshop on Natural Language Processing for Turkic Languages (SIGTURK 2024), pages 81–91, Bangkok, Thailand and Online. Association for Computational Linguistics.

Artidoro Pagnoni, Ram Pasunuru, Pedro Rodriguez, John Nguyen, Benjamin Muller, Margaret Li, Chunting Zhou, Lili Yu, Jason Weston, Luke Zettlemoyer, Gargi Ghosh, Mike Lewis, Ari Holtzman, and Srinivasan Iyer. 2024. Byte latent transformer: Patches scale better than tokens. Preprint, arXiv:2412.09871.

Qwen, .; An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dong, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115.

Xiaofeng Wu, Karl Stratos, and Wei Xu. 2025. The impact of visual information in chinese characters: Evaluating large models’ ability to recognize and utilize radicals. Preprint, arXiv:2410.09013.

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. In International conference on machine learning, pages 12697–12706. PMLR.

A Comparison to CUTE

In Table 5, we run Llama 3.1 8B on CUTE and compare the results to English and Russian EXECUTE. The results are very similar, with Insert Word appearing slightly easier in CUTE. This confirms that our changes did not dramatically alter any results.

<table>
  <tr>
    <th></th>
    <th colspan="2">EXECUTE</th>
    <th colspan="2">CUTE</th>
  </tr>
  <tr>
    <th></th>
    <th>Eng</th>
    <th>Rus</th>
    <th>Eng</th>
    <th>Rus</th>
  </tr>
  <tr>
    <td>Spell</td>
    <td>98.7</td>
    <td><b>72.1</b></td>
    <td><b>99.8</b></td>
    <td>64.5</td>
  </tr>
  <tr>
    <td>Inv Spell</td>
    <td>96.2</td>
    <td>37.9</td>
    <td><b>98.4</b></td>
    <td><b>74.1</b></td>
  </tr>
  <tr>
    <td>Cont Char</td>
    <td>65.1</td>
    <td>57.1</td>
    <td>67.1</td>
    <td><b>68.4</b></td>
  </tr>
  <tr>
    <td>Cont Word</td>
    <td><b>97.3</b></td>
    <td><b>97.3</b></td>
    <td>86.8</td>
    <td>97.4</td>
  </tr>
  <tr>
    <td>Ins Char</td>
    <td>4.4</td>
    <td>6.7</td>
    <td>4.2</td>
    <td>7.6</td>
  </tr>
  <tr>
    <td>Ins Word</td>
    <td>48.2</td>
    <td>48.5</td>
    <td><b>62.0</b></td>
    <td>59.2</td>
  </tr>
  <tr>
    <td>Del Char</td>
    <td>56.1</td>
    <td>33.1</td>
    <td><b>56.6</b></td>
    <td>43.7</td>
  </tr>
  <tr>
    <td>Del Word</td>
    <td>76.2</td>
    <td><b>91.8</b></td>
    <td>83.7</td>
    <td>82.3</td>
  </tr>
  <tr>
    <td>Sub Char</td>
    <td><b>39.3</b></td>
    <td>29.0</td>
    <td>34.4</td>
    <td><b>33.8</b></td>
  </tr>
  <tr>
    <td>Sub Word</td>
    <td><b>94.1</b></td>
    <td><b>87.0</b></td>
    <td>90.4</td>
    <td>76.7</td>
  </tr>
  <tr>
    <td>Swap Char</td>
    <td><b>6.6</b></td>
    <td>4.8</td>
    <td>6.1</td>
    <td><b>5.2</b></td>
  </tr>
  <tr>
    <td>Swap Word</td>
    <td>60.4</td>
    <td><b>46.5</b></td>
    <td><b>63.7</b></td>
    <td>33.3</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>61.9</td>
    <td>51.0</td>
    <td><b>62.8</b></td>
    <td><b>53.9</b></td>
  </tr>
</table>

Table 5: EXECUTE versus CUTE with Llama 3.1 8B.

B Language Similarity

We conduct similarity tests to see how similar the trends are across languages. We conduct a Pearson correlation between two languages for each task for a given model and average the models’ correlations together. We show the similarity of the languages as determined by the average correlation of the results from the 5 LLMs of size 7-9B in Table 6. The languages are not particularly similar to one another, apart from Japanese and Chinese (which share some characters) and Arabic and Russian. The similarity between Arabic and Russian is not entirely clear, though it could be that their ratios of characters-per-word and characters-per-token are quite similar (such is also the case for Japanese and Chinese).

<table>
  <tr>
    <th></th>
    <th>Ara</th>
    <th>Zho</th>
    <th>Eng</th>
    <th>Hin</th>
    <th>Jpn</th>
    <th>Kor</th>
    <th>Rus</th>
  </tr>
  <tr>
    <td>Amh</td>
    <td>0.64</td>
    <td>0.01</td>
    <td>0.66</td>
    <td>0.33</td>
    <td>0.23</td>
    <td>0.62</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>Ara</td>
    <td>-0.11</td>
    <td>0.76</td>
    <td>0.44</td>
    <td>0.13</td>
    <td>0.85</td>
    <td>0.92</td>
    <td></td>
  </tr>
  <tr>
    <td>Zho</td>
    <td>0.17</td>
    <td>0.65</td>
    <td>0.32</td>
    <td>0.26</td>
    <td>-0.06</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Eng</td>
    <td>0.45</td>
    <td>0.36</td>
    <td>0.77</td>
    <td>0.86</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Hin</td>
    <td></td>
    <td>0.75</td>
    <td>0.71</td>
    <td>0.38</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Jpn</td>
    <td></td>
    <td></td>
    <td>0.49</td>
    <td>0.17</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Kor</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td>0.84</td>
  </tr>
</table>

Table 6: Average correlations between the results for each language pair.

We also correlate the results from English to other Latin-scripted languages, German, Spanish, and Xhosa, in Table 7. Here we see the average correlation is at least 95% between English, German, and Spanish, and at least 85% to Xhosa. This suggests that the results for other Latin-scripted languages will likely not deviate too much, even if the languages are distant in relation and differing in resourcedness.
<table>
  <tr>
    <th></th>
    <th>Deu</th>
    <th>Spa</th>
    <th>Xho</th>
  </tr>
  <tr>
    <td>Eng</td>
    <td>0.96</td>
    <td>0.95</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>Deu</td>
    <td>0.99</td>
    <td>0.90</td>
    <td></td>
  </tr>
  <tr>
    <td>Spa</td>
    <td></td>
    <td>0.90</td>
    <td></td>
  </tr>
</table>

Table 7: Average correlations between the results for Latin-scripted languages.

![Bar chart showing performance on Kanji to Hiragana conversion for various datasets](page_312_420_495_312.png)

Figure 3: Performance on Kanji to Hiragana conversion.

C Japanese Furigana

Aside from Kanji, Japanese has two other writing forms: Hiragana and Katakana. Typical Japanese text will use all three forms, with several words being a combination of Kanji and Hiragana, and even in rare cases, all three. While Kanji is logographic like Chinese, Hiragana and Katakana are syllabaries. Kanji and Hiragana are the most used, while Katakana is typically only used for foreign words or onomatopoeiae. As such, we focus on Kanji and Hiragana. All Kanji characters can be written as Hiragana, and Kanji is sometimes annotated with its corresponding Hiragana as a method of learning the pronunciation of Kanji characters. This practice is known as Furigana. So we use this Furigana method as a test in our benchmark, prompting the model to translate Kanji to Hiragana.7 With this, we are essentially testing if the LLMs have a phonetic understanding of the Kanji.

In Figure 3, we see the models’ results on the Furigana task. Similar to the Korean Hangul to Jamo, the Kanji to Hiragana tasks show that the LLMs generally understand the task, but have not perfected it. Unlike the other sub-character tasks, converting from Kanji to Hiragana cannot be done purely visually. This requires knowledge of how a Kanji sounds, and which Hiragana denote which sounds. From this, we can see a partial understanding.


 Spell out the word, putting spaces between each letter, based on the following examples:

1. Spell out the word “かわいい”. Answer: “かわいい”
2. Spell out the word “出し”. Answer: “出し”
3. Spell out the word “応援”. Answer: “応援”
4. Spell out the word “親友”. Answer: “親友”

Question: Spell out the word “実行する”. 
 Answer: “実行する”

Figure 4: Example of full prompt for Japanese spelling, with intended output in red. 
 and 
 denote any tokens added to enable normal behavior from each LLM.

D Full Prompt Example

We show an example of a full prompt in Figure 4.

E Full Results

The complete results on EXECUTE for all the models tested are shown in Tables 8 and 9.

7We do not do the reverse as multiple Kanji can have the same phoneme, e.g. 考 and 好 both denote ko.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">Aya Expanse</th>
    <th colspan="4">Gemma 2</th>
    <th colspan="4">Llama 3.1</th>
    <th colspan="4">Llama 3.3</th>
    <th colspan="4">Qwen 2.5</th>
    <th colspan="4">Mistral</th>
  </tr>
  <tr>
    <th>8B</th><th>32B</th><th>9B</th><th>27B</th>
    <th>8B</th><th>70B</th><th>70B</th><th>7B</th>
    <th>32B</th><th>8B</th><th>24B</th>
  </tr>
  <tr>
    <td rowspan="10">Amharic</td>
    <td>Spell</td><td>25.6</td><td>72.4</td><td>99.5</td><td>91.2</td><td>98.6</td><td>98.4</td><td>96.3</td><td>0.9</td><td>14.3</td><td>97.0</td><td>99.8</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>77.6</td><td>71.5</td><td>99.1</td><td>91.4</td><td>98.8</td><td>99.8</td><td>99.8</td><td>8.2</td><td>52.4</td><td>99.1</td><td>100.0</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>58.5</td><td>85.8</td><td>73.0</td><td>81.9</td><td>90.4</td><td>91.8</td><td>91.8</td><td>63.2</td><td>93.5</td><td>94.7</td><td>95.8</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>55.9</td><td>69.5</td><td>97.0</td><td>98.3</td><td>78.5</td><td>99.5</td><td>99.6</td><td>71.7</td><td>99.9</td><td>95.9</td><td>98.8</td>
  </tr>
  <tr>
    <td>Ins Char</td><td>35.2</td><td>58.2</td><td>57.1</td><td>65.5</td><td>26.1</td><td>92.3</td><td>97.8</td><td>10.3</td><td>57.9</td><td>60.6</td><td>92.2</td>
  </tr>
  <tr>
    <td>Ins Word</td><td>70.7</td><td>66.7</td><td>95.6</td><td>91.7</td><td>28.7</td><td>93.3</td><td>92.8</td><td>58.4</td><td>92.6</td><td>70.5</td><td>87.9</td>
  </tr>
  <tr>
    <td>Del Char</td><td>54.5</td><td>85.0</td><td>84.5</td><td>89.4</td><td>89.6</td><td>96.8</td><td>97.6</td><td>43.5</td><td>78.6</td><td>85.3</td><td>97.7</td>
  </tr>
  <tr>
    <td>Del Word</td><td>83.8</td><td>91.2</td><td>63.1</td><td>79.2</td><td>93.2</td><td>99.1</td><td>97.6</td><td>70.0</td><td>91.1</td><td>91.9</td><td>89.4</td>
  </tr>
  <tr>
    <td>Sub Char</td><td>56.6</td><td>79.1</td><td>79.1</td><td>85.2</td><td>79.1</td><td>94.9</td><td>96.6</td><td>29.6</td><td>57.1</td><td>69.1</td><td>93.1</td>
  </tr>
  <tr>
    <td>Sub Word</td><td>91.9</td><td>98.1</td><td>98.1</td><td>93.2</td><td>91.6</td><td>96.7</td><td>96.2</td><td>79.2</td><td>90.9</td><td>90.3</td><td>96.5</td>
  </tr>
  <tr>
    <td>Swap Char</td><td>20.2</td><td>53.8</td><td>53.0</td><td>70.7</td><td>52.8</td><td>91.3</td><td>93.7</td><td>12.8</td><td>65.8</td><td>48.8</td><td>86.4</td>
  </tr>
  <tr>
    <td>Swap Word</td><td>60.7</td><td>84.5</td><td>75.7</td><td>86.0</td><td>73.0</td><td>96.5</td><td>97.3</td><td>55.0</td><td>88.1</td><td>66.0</td><td>94.9</td>
  </tr>
  <tr>
    <td rowspan="10">Arabic</td>
    <td>Spell</td><td>48.7</td><td>74.1</td><td>36.0</td><td>69.9</td><td>50.8</td><td>84.7</td><td>81.0</td><td>20.7</td><td>52.8</td><td>19.3</td><td>40.0</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>48.4</td><td>64.7</td><td>48.3</td><td>63.6</td><td>44.9</td><td>69.7</td><td>63.0</td><td>39.2</td><td>76.4</td><td>27.6</td><td>60.9</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>63.9</td><td>74.5</td><td>70.3</td><td>70.0</td><td>70.6</td><td>77.4</td><td>76.4</td><td>74.0</td><td>90.7</td><td>72.0</td><td>78.6</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>88.1</td><td>97.9</td><td>99.0</td><td>98.7</td><td>96.9</td><td>99.1</td><td>99.1</td><td>95.5</td><td>99.4</td><td>88.7</td><td>99.4</td>
  </tr>
  <tr>
    <td>Ins Char</td><td>13.7</td><td>8.3</td><td>7.6</td><td>16.1</td><td>2.9</td><td>15.7</td><td>17.8</td><td>11.7</td><td>31.4</td><td>4.7</td><td>12.5</td>
  </tr>
  <tr>
    <td>Ins Word</td><td>35.8</td><td>61.1</td><td>90.8</td><td>97.5</td><td>51.2</td><td>89.1</td><td>96.4</td><td>61.3</td><td>96.3</td><td>45.1</td><td>86.4</td>
  </tr>
  <tr>
    <td>Del Char</td><td>36.0</td><td>56.4</td><td>36.3</td><td>45.5</td><td>45.0</td><td>55.8</td><td>59.4</td><td>40.7</td><td>53.1</td><td>20.8</td><td>29.9</td>
  </tr>
  <tr>
    <td>Del Word</td><td>74.0</td><td>90.4</td><td>64.6</td><td>83.4</td><td>92.5</td><td>95.0</td><td>88.6</td><td>82.0</td><td>91.6</td><td>63.7</td><td>88.8</td>
  </tr>
  <tr>
    <td>Sub Char</td><td>17.6</td><td>28.6</td><td>26.0</td><td>33.5</td><td>24.7</td><td>34.7</td><td>36.0</td><td>23.0</td><td>46.2</td><td>20.7</td><td>20.7</td>
  </tr>
  <tr>
    <td>Sub Word</td><td>72.7</td><td>90.6</td><td>95.0</td><td>97.3</td><td>90.8</td><td>97.2</td><td>98.0</td><td>79.3</td><td>95.4</td><td>77.1</td><td>92.2</td>
  </tr>
  <tr>
    <td>Swap Char</td><td>5.9</td><td>9.1</td><td>4.5</td><td>8.0</td><td>8.1</td><td>17.0</td><td>16.7</td><td>4.0</td><td>14.3</td><td>2.5</td><td>7.5</td>
  </tr>
  <tr>
    <td>Swap Word</td><td>26.3</td><td>55.1</td><td>46.3</td><td>63.1</td><td>47.6</td><td>78.0</td><td>81.3</td><td>34.8</td><td>79.8</td><td>29.0</td><td>62.0</td>
  </tr>
  <tr>
    <td rowspan="10">Chinese</td>
    <td>Spell</td><td>83.2</td><td>93.0</td><td>84.3</td><td>91.3</td><td>93.6</td><td>98.4</td><td>98.0</td><td>96.3</td><td>98.2</td><td>93.4</td><td>90.9</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>88.5</td><td>97.2</td><td>95.1</td><td>96.2</td><td>98.6</td><td>98.4</td><td>98.7</td><td>88.0</td><td>99.8</td><td>98.8</td><td>99.6</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>84.0</td><td>97.2</td><td>96.4</td><td>95.4</td><td>92.5</td><td>97.3</td><td>91.1</td><td>98.7</td><td>98.9</td><td>96.0</td><td>98.7</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>84.1</td><td>99.0</td><td>99.4</td><td>98.8</td><td>91.7</td><td>95.1</td><td>86.7</td><td>94.3</td><td>98.1</td><td>94.2</td><td>94.8</td>
  </tr>
  <tr>
    <td>Ins Char</td><td>70.0</td><td>57.2</td><td>78.6</td><td>81.4</td><td>67.3</td><td>90.6</td><td>92.6</td><td>73.6</td><td>95.4</td><td>53.2</td><td>89.3</td>
  </tr>
  <tr>
    <td>Ins Word</td><td>28.4</td><td>33.2</td><td>41.9</td><td>53.0</td><td>20.5</td><td>46.5</td><td>47.5</td><td>43.7</td><td>53.4</td><td>34.1</td><td>50.5</td>
  </tr>
  <tr>
    <td>Del Char</td><td>79.6</td><td>90.0</td><td>85.4</td><td>88.1</td><td>86.8</td><td>97.0</td><td>97.6</td><td>89.2</td><td>97.3</td><td>88.3</td><td>94.8</td>
  </tr>
  <tr>
    <td>Del Word</td><td>38.4</td><td>57.3</td><td>46.8</td><td>56.9</td><td>59.4</td><td>71.1</td><td>70.1</td><td>54.0</td><td>65.1</td><td>53.2</td><td>67.4</td>
  </tr>
  <tr>
    <td>Sub Char</td><td>60.6</td><td>68.4</td><td>69.1</td><td>75.1</td><td>84.0</td><td>94.2</td><td>94.8</td><td>80.0</td><td>94.1</td><td>75.0</td><td>91.3</td>
  </tr>
  <tr>
    <td>Sub Word</td><td>63.2</td><td>55.3</td><td>55.3</td><td>64.7</td><td>57.5</td><td>56.3</td><td>53.4</td><td>43.1</td><td>66.8</td><td>46.5</td><td>66.5</td>
  </tr>
  <tr>
    <td>Swap Char</td><td>63.9</td><td>71.9</td><td>73.3</td><td>69.8</td><td>90.6</td><td>92.0</td><td>92.1</td><td>62.6</td><td>92.0</td><td>75.4</td><td>90.5</td>
  </tr>
  <tr>
    <td>Swap Word</td><td>14.2</td><td>26.6</td><td>15.6</td><td>21.9</td><td>22.8</td><td>36.1</td><td>33.3</td><td>10.7</td><td>43.9</td><td>15.1</td><td>35.0</td>
  </tr>
  <tr>
    <td rowspan="10">English</td>
    <td>Spell</td><td>96.7</td><td>98.5</td><td>99.3</td><td>99.5</td><td>98.7</td><td>99.5</td><td>99.5</td><td>94.7</td><td>98.6</td><td>97.3</td><td>99.0</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>95.4</td><td>98.5</td><td>99.3</td><td>99.6</td><td>96.2</td><td>99.8</td><td>99.6</td><td>98.3</td><td>99.2</td><td>91.0</td><td>98.7</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>62.6</td><td>73.1</td><td>68.0</td><td>69.5</td><td>65.1</td><td>80.3</td><td>75.7</td><td>81.9</td><td>94.8</td><td>66.7</td><td>83.5</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>94.3</td><td>99.4</td><td>99.7</td><td>99.7</td><td>97.3</td><td>100.0</td><td>99.9</td><td>98.1</td><td>100.0</td><td>93.3</td><td>99.9</td>
  </tr>
  <tr>
    <td>Ins Char</td><td>11.8</td><td>6.0</td><td>9.2</td><td>7.8</td><td>4.4</td><td>10.9</td><td>13.5</td><td>7.1</td><td>15.9</td><td>7.4</td><td>4.4</td>
  </tr>
  <tr>
    <td>Ins Word</td><td>39.9</td><td>60.6</td><td>86.7</td><td>96.8</td><td>48.2</td><td>94.9</td><td>96.6</td><td>70.2</td><td>97.6</td><td>51.9</td><td>72.9</td>
  </tr>
  <tr>
    <td>Del Char</td><td>35.0</td><td>56.3</td><td>58.5</td><td>80.4</td><td>56.1</td><td>68.3</td><td>67.5</td><td>56.8</td><td>70.5</td><td>33.6</td><td>72.4</td>
  </tr>
  <tr>
    <td>Del Word</td><td>60.3</td><td>77.5</td><td>53.7</td><td>77.7</td><td>76.2</td><td>97.1</td><td>96.5</td><td>78.5</td><td>95.7</td><td>74.3</td><td>69.8</td>
  </tr>
  <tr>
    <td>Sub Char</td><td>27.7</td><td>42.2</td><td>35.4</td><td>60.5</td><td>39.3</td><td>53.5</td><td>51.4</td><td>29.0</td><td>52.1</td><td>27.7</td><td>51.7</td>
  </tr>
  <tr>
    <td>Sub Word</td><td>82.4</td><td>96.1</td><td>94.7</td><td>97.3</td><td>94.1</td><td>98.1</td><td>98.0</td><td>92.9</td><td>99.0</td><td>92.9</td><td>97.0</td>
  </tr>
  <tr>
    <td>Swap Char</td><td>14.0</td><td>1.8</td><td>6.9</td><td>8.9</td><td>6.6</td><td>14.9</td><td>12.7</td><td>3.2</td><td>11.5</td><td>6.8</td><td>11.0</td>
  </tr>
  <tr>
    <td>Swap Word</td><td>22.6</td><td>64.5</td><td>66.2</td><td>60.8</td><td>60.4</td><td>91.0</td><td>90.9</td><td>34.7</td><td>92.4</td><td>42.2</td><td>85.9</td>
  </tr>
</table>

Table 8: Results for Amharic, Arabic, Chinese, and English.
<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="4">Aya Expanse</th>
    <th colspan="4">Gemma 2</th>
    <th colspan="4">Llama 3.1</th>
    <th colspan="4">Llama 3.3</th>
    <th colspan="4">Qwen 2.5</th>
    <th colspan="4">Mistral</th>
  </tr>
  <tr>
    <th>8B</th><th>32B</th><th>9B</th><th>27B</th>
    <th>8B</th><th>70B</th><th>70B</th><th>7B</th>
    <th>32B</th><th>8B</th><th>24B</th>
  </tr>
  <tr>
    <td>Spell</td><td>48.4</td><td>69.4</td><td>12.6</td><td>16.0</td><td>46.2</td><td>72.5</td><td>71.3</td><td>20.5</td><td>57.7</td><td>23.4</td><td>58.7</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>76.8</td><td>83.2</td><td>71.9</td><td>83.4</td><td>76.0</td><td>92.7</td><td>92.9</td><td>71.2</td><td>94.6</td><td>61.9</td><td>87.2</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>68.4</td><td>85.6</td><td>75.1</td><td>75.7</td><td>72.6</td><td>87.3</td><td>83.2</td><td>90.9</td><td>98.3</td><td>82.4</td><td>88.9</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>89.4</td><td>98.7</td><td>95.7</td><td>98.5</td><td>93.3</td><td>98.9</td><td>93.0</td><td>94.7</td><td>99.3</td><td>81.1</td><td>94.8</td>
  </tr>
  <tr>
    <td>Ins Char</td><td>41.0</td><td>10.8</td><td>25.8</td><td>29.4</td><td>15.8</td><td>29.0</td><td>32.9</td><td>45.9</td><td>89.0</td><td>27.5</td><td>36.5</td>
  </tr>
  <tr>
    <td rowspan="7">Hindi</td><td>Ins Word</td><td>6.3</td><td>11.7</td><td>77.3</td><td>41.8</td><td>13.6</td><td>25.5</td><td>47.6</td><td>31.8</td><td>95.5</td><td>9.1</td><td>16.4</td>
  </tr>
  <tr><td>Del Char</td><td>58.8</td><td>66.5</td><td>50.9</td><td>67.6</td><td>66.5</td><td>76.9</td><td>70.9</td><td>50.9</td><td>80.7</td><td>44.3</td><td>79.5</td></tr>
  <tr><td>Del Word</td><td>6.6</td><td>26.6</td><td>26.6</td><td>24.4</td><td>40.6</td><td>25.7</td><td>31.7</td><td>29.4</td><td>90.8</td><td>13.9</td><td>19.7</td></tr>
  <tr><td>Sub Char</td><td>4.5</td><td>16.6</td><td>21.2</td><td>14.2</td><td>14.4</td><td>6.6</td><td>50.8</td><td>32.2</td><td>65.5</td><td>4.5</td><td>6.5</td></tr>
  <tr><td>Sub Word</td><td>3.7</td><td>14.9</td><td>62.3</td><td>15.9</td><td>19.1</td><td>23.8</td><td>25.4</td><td>15.3</td><td>93.8</td><td>4.9</td><td>17.0</td></tr>
  <tr><td>Swap Char</td><td>14.5</td><td>24.5</td><td>8.4</td><td>29.4</td><td>19.5</td><td>15.1</td><td>22.2</td><td>9.9</td><td>62.8</td><td>21.6</td><td>33.6</td></tr>
  <tr><td>Swap Word</td><td>2.1</td><td>16.9</td><td>30.6</td><td>27.5</td><td>20.1</td><td>39.4</td><td>35.7</td><td>10.3</td><td>82.2</td><td>3.0</td><td>11.5</td></tr>
  <tr>
    <td>Spell</td><td>52.9</td><td>83.2</td><td>68.5</td><td>71.0</td><td>73.2</td><td>93.3</td><td>92.2</td><td>72.3</td><td>86.5</td><td>77.2</td><td>84.5</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>92.4</td><td>88.4</td><td>87.8</td><td>90.5</td><td>78.1</td><td>96.9</td><td>95.1</td><td>90.4</td><td>95.9</td><td>88.5</td><td>96.7</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>67.9</td><td>91.6</td><td>90.9</td><td>88.8</td><td>87.0</td><td>95.2</td><td>93.4</td><td>93.7</td><td>97.8</td><td>86.7</td><td>93.0</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>82.4</td><td>92.7</td><td>98.8</td><td>97.3</td><td>90.0</td><td>89.4</td><td>85.5</td><td>89.9</td><td>94.8</td><td>84.6</td><td>93.9</td>
  </tr>
  <tr>
    <td rowspan="7">Japanese</td><td>Ins Char</td><td>48.3</td><td>31.5</td><td>58.3</td><td>68.8</td><td>18.6</td><td>72.1</td><td>77.6</td><td>69.7</td><td>86.8</td><td>29.3</td><td>78.5</td>
  </tr>
  <tr><td>Ins Word</td><td>21.4</td><td>34.2</td><td>41.7</td><td>61.0</td><td>10.6</td><td>42.1</td><td>47.2</td><td>44.0</td><td>65.9</td><td>30.4</td><td>46.2</td></tr>
  <tr><td>Del Char</td><td>62.0</td><td>78.3</td><td>60.4</td><td>64.4</td><td>79.8</td><td>88.2</td><td>90.0</td><td>70.0</td><td>84.3</td><td>67.1</td><td>86.5</td></tr>
  <tr><td>Del Word</td><td>31.9</td><td>60.4</td><td>49.5</td><td>52.5</td><td>62.4</td><td>68.5</td><td>74.2</td><td>52.3</td><td>61.4</td><td>38.0</td><td>63.5</td></tr>
  <tr><td>Sub Char</td><td>57.4</td><td>68.1</td><td>58.5</td><td>73.5</td><td>62.2</td><td>83.7</td><td>64.7</td><td>68.7</td><td>83.1</td><td>41.7</td><td>81.7</td></tr>
  <tr><td>Sub Word</td><td>32.4</td><td>55.1</td><td>49.8</td><td>58.8</td><td>50.9</td><td>62.8</td><td>65.1</td><td>50.9</td><td>69.3</td><td>38.5</td><td>60.6</td></tr>
  <tr><td>Swap Char</td><td>40.8</td><td>52.0</td><td>50.2</td><td>49.2</td><td>66.6</td><td>63.9</td><td>66.7</td><td>40.4</td><td>76.6</td><td>45.7</td><td>70.1</td></tr>
  <tr><td>Swap Word</td><td>5.9</td><td>13.2</td><td>8.1</td><td>13.3</td><td>12.8</td><td>21.9</td><td>25.0</td><td>7.5</td><td>29.4</td><td>9.3</td><td>20.9</td></tr>
  <tr>
    <td>Spell</td><td>43.6</td><td>71.5</td><td>67.5</td><td>85.4</td><td>51.5</td><td>78.0</td><td>73.4</td><td>42.5</td><td>66.6</td><td>42.3</td><td>56.1</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>85.0</td><td>93.8</td><td>82.6</td><td>86.8</td><td>71.7</td><td>88.9</td><td>88.8</td><td>84.2</td><td>94.8</td><td>64.6</td><td>92.2</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>71.1</td><td>84.5</td><td>81.1</td><td>86.2</td><td>81.6</td><td>85.3</td><td>74.4</td><td>90.7</td><td>95.5</td><td>76.4</td><td>89.0</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>92.6</td><td>97.2</td><td>98.8</td><td>99.0</td><td>95.6</td><td>99.4</td><td>99.1</td><td>96.7</td><td>99.6</td><td>84.9</td><td>98.7</td>
  </tr>
  <tr>
    <td rowspan="7">Korean</td><td>Ins Char</td><td>33.6</td><td>20.6</td><td>47.4</td><td>54.6</td><td>21.0</td><td>52.5</td><td>60.8</td><td>39.6</td><td>69.7</td><td>20.2</td><td>41.7</td>
  </tr>
  <tr><td>Ins Word</td><td>36.4</td><td>72.0</td><td>93.3</td><td>98.6</td><td>50.8</td><td>91.8</td><td>94.1</td><td>66.8</td><td>92.5</td><td>45.0</td><td>87.7</td></tr>
  <tr><td>Del Char</td><td>44.7</td><td>64.6</td><td>69.1</td><td>75.6</td><td>62.5</td><td>62.2</td><td>63.2</td><td>44.6</td><td>64.1</td><td>43.2</td><td>56.6</td></tr>
  <tr><td>Del Word</td><td>56.5</td><td>81.8</td><td>77.3</td><td>88.6</td><td>91.9</td><td>94.3</td><td>86.3</td><td>76.1</td><td>91.0</td><td>75.4</td><td>90.7</td></tr>
  <tr><td>Sub Char</td><td>39.1</td><td>62.0</td><td>71.4</td><td>77.4</td><td>56.0</td><td>65.3</td><td>63.9</td><td>50.4</td><td>60.6</td><td>37.5</td><td>53.0</td></tr>
  <tr><td>Sub Word</td><td>48.9</td><td>78.1</td><td>90.1</td><td>97.0</td><td>73.2</td><td>92.2</td><td>92.1</td><td>68.8</td><td>90.5</td><td>51.1</td><td>89.6</td></tr>
  <tr><td>Swap Char</td><td>27.3</td><td>33.3</td><td>48.9</td><td>47.5</td><td>37.3</td><td>42.0</td><td>44.0</td><td>31.4</td><td>58.1</td><td>20.0</td><td>33.0</td></tr>
  <tr><td>Swap Word</td><td>22.6</td><td>48.3</td><td>55.8</td><td>73.6</td><td>52.5</td><td>71.1</td><td>72.8</td><td>29.9</td><td>79.3</td><td>27.1</td><td>61.1</td></tr>
  <tr>
    <td>Spell</td><td>40.9</td><td>80.4</td><td>54.5</td><td>88.3</td><td>72.1</td><td>97.6</td><td>96.7</td><td>54.0</td><td>88.9</td><td>70.8</td><td>94.8</td>
  </tr>
  <tr>
    <td>Inv Spell</td><td>54.1</td><td>78.7</td><td>71.4</td><td>90.3</td><td>37.9</td><td>86.6</td><td>86.0</td><td>81.9</td><td>94.2</td><td>39.5</td><td>88.0</td>
  </tr>
  <tr>
    <td>Cont Char</td><td>55.0</td><td>68.8</td><td>59.7</td><td>58.7</td><td>57.1</td><td>60.2</td><td>59.1</td><td>75.3</td><td>84.9</td><td>53.5</td><td>77.6</td>
  </tr>
  <tr>
    <td>Cont Word</td><td>96.9</td><td>98.1</td><td>99.6</td><td>99.8</td><td>97.8</td><td>99.8</td><td>99.9</td><td>95.8</td><td>99.5</td><td>94.7</td><td>99.5</td>
  </tr>
  <tr>
    <td rowspan="7">Russian</td><td>Ins Char</td><td>10.9</td><td>3.8</td><td>7.0</td><td>9.4</td><td>6.7</td><td>11.0</td><td>12.0</td><td>7.2</td><td>21.1</td><td>5.7</td><td>11.3</td>
  </tr>
  <tr><td>Ins Word</td><td>29.3</td><td>61.9</td><td>90.6</td><td>95.4</td><td>48.5</td><td>90.7</td><td>93.1</td><td>77.8</td><td>95.1</td><td>62.9</td><td>84.2</td></tr>
  <tr><td>Del Char</td><td>12.7</td><td>34.7</td><td>20.9</td><td>38.1</td><td>33.1</td><td>49.9</td><td>50.9</td><td>24.8</td><td>44.7</td><td>18.9</td><td>45.1</td></tr>
  <tr><td>Del Word</td><td>68.6</td><td>85.0</td><td>75.0</td><td>81.0</td><td>91.8</td><td>97.3</td><td>96.7</td><td>83.1</td><td>90.7</td><td>80.1</td><td>85.7</td></tr>
  <tr><td>Sub Char</td><td>15.5</td><td>26.0</td><td>17.0</td><td>35.3</td><td>29.0</td><td>38.7</td><td>40.2</td><td>22.6</td><td>44.3</td><td>16.2</td><td>39.7</td></tr>
  <tr><td>Sub Word</td><td>63.8</td><td>86.4</td><td>94.8</td><td>92.7</td><td>87.0</td><td>95.6</td><td>93.8</td><td>80.9</td><td>95.2</td><td>86.5</td><td>95.6</td></tr>
  <tr><td>Swap Char</td><td>1.1</td><td>4.8</td><td>2.6</td><td>4.0</td><td>4.2</td><td>15.5</td><td>13.6</td><td>1.4</td><td>10.0</td><td>4.1</td><td>12.0</td></tr>
  <tr><td>Swap Word</td><td>26.1</td><td>48.1</td><td>50.7</td><td>55.8</td><td>46.5</td><td>74.3</td><td>70.0</td><td>20.7</td><td>79.9</td><td>52.6</td><td>79.5</td></tr>
</table>

Table 9: Results for Hindi, Korean, Japanese, and Russian.

Reviewer paper summaries: This paper introduces EXECUTE, a multilingual and multi-script benchmark designed to evaluate LLMs' understanding of tokens at character, sub-character, and word levels across diverse languages and scripts. Building upon the previous CUTE benchmark, EXECUTE extends evaluations beyond English to languages using various writing systems, including alphabets, abugidas, abjads, logographic, and featural scripts.

Empirical analysis involving multiple open-source multilingual LLMs reveals significant variation in performance across languages and tasks. Surprisingly, LLMs demonstrate better performance on lower-resourced languages due to reduced language-specific biases. The benchmark further highlights substantial challenges in understanding sub-character components, particularly radicals and phonetic elements in East Asian scripts.
Summary of strengths: 1. The EXECUTE framework supports various languages and thus could provide valuable resources for the research of multiple languages besides English. The framework currently considers languages with different writing systems and different scripts, further enhancing the contribution.


2. The experiment results of the languages are very insightful. It not only validates that different writing systems have different performances at word and character levels, but also brings up surprising results that the LLMs perform better on the char level tasking when language is low-resource.


3. It provides baselines for current commonly used LLMs.
Summary of weaknesses: 1. The contribution of EXECUTE is unclear. The paper mentioned that EXECUTE is built according to CUTE, but it doesn't mention the difference between the EXECUTE framework and the CUTE benchmark. Adding the distinguishment content will enhance the contribution.


2. Adding a new language method is a little confusing since it mentioned "Adding a new language X now only requires an English→X translation system. As cross-language alignment is not crucial, translations do not need to be perfect: grammaticality is preferable but not
necessary." It does not verify why " grammaticality is preferable but not necessary". If a language is not natively written, how to validate the effectiveness of the score?
Comments, suggestions and typos: The writing overall omits some logic connectors (such as the difference between CUTE and EXECUTE, and detailed motivations) due to the paper length limit. Please consider move some result in appendix and add more such contents.
Overall assessment scores: 3.5