One ruler to measure them all: Benchmarking multilingual long-context language models
Authors: Yekyung Kim
Jenna Russell
Marzena Karpinska
Mohit Iyyer
Venue: COLM 2025 (2025)
Keywords: Multilingual
Benchmark
Long-context
Synthetic dataset
TL;DR: ONERULER is a multilingual benchmark for evaluating long-context LLMs across 26 languages, extending RULER beyond English. It aims to assess model performance in diverse linguistic settings using seven tasks, including detecting absent information.

Abstract: We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al., 2024) by including seven synthetic tasks that test both retrieval and aggregation, including new variations of the "needle-in-a-haystack" task that allow for the possibility of a nonexistent needle. We create ONERULER through a two-step process, first writing English instructions for each task and then collaborating with native speakers to translate them into 25 additional languages. Experiments with both open-weight and closed LLMs reveal a widening performance gap between low- and high-resource languages as context length increases from 8K to 128K tokens. Surprisingly, English is not the top-performing language on long-context tasks (ranked 6th out of 26), with Polish emerging as the top language. Our experiments also show that many LLMs (particularly OpenAI's o3-mini-high) incorrectly predict the absence of an answer, even in high-resource languages. Finally, in cross-lingual scenarios where instructions and context appear in different languages, performance can fluctuate by up to 20% depending on the instruction language. We hope the release of ONERULER will facilitate future research into improving multilingual and cross-lingual long-context training pipelines.

Published as a conference paper at COLM 2025
One ruler to measure them all:
Benchmarking multilingual long-context language models
Yekyung Kim Jenna Russell Marzena Karpinska Mohit lyyer All
University of Maryland, College Park Microsoft UMass Amherst
[yekyung, jennarus, miyyer Yeumd. edu, ikarpinska@mii www.a/eiiiccooooocooooooooo
Abstract
We present ONERULER, a multilingual benchmark designed to evaluate long-contexs language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al., 2024) by including seven synthetic tasks that test both retrieval and aggregation, including new vari- ations of the saredle-in---haystadee nonevistent needle. We create ONERULER through two-step process, first writing English instructions for each task and then collaborating with na- tive speakers to translate them into 25 additional languages. Experiments with both open-weight and closed language models reveal a widening performance gap between low- and high-resource languages as context length increases from 8K to 128K tokens. Surprisingly, English is not the top-performing language on long-context tasks (ranked 6th out of 26), with Polish emerging as the top language. Our experiments also show that many LLMs (particularly OpenAl's 03-mini-high incorrectly predict the absence of an answer, even in high-resource languages. Finally, in cross-lingual scenarios where instructions and context appear in different languages, performance can fluctuate by up to 20% depending the instruction lan- guage. We hope the release of ONERULER will facilitate future research into improving multilingual and cross-lingual loog-conteen training pipelines.
task that allow for the possibility of a
/ http:s//gggtibbcco/////gnngn/nue www./m/uggg/nnenuleee.e.
1 Introduction
Long-context language understanding is essential for real-world applications of large lan- guage models (LLMs) such as summarization and question answering. However, it is difficult and expensive to conduct realistic evaluations for these tasks (Kim et al., 2024; Karpinska et al., 2024), which motivates the use of synthetic benchmarks as proxy diag- nostics. One popular example is the "needle-in-a-hayslle in which a codeword is inserted into a long document and subsequently queried for. The RULER benchmark (Hsieh et al., 2024) contains several variants of NIÀH (e.g., multiple needles and queries) as well as other synthetic tasks to test aggregation and variable tracing. Unfortunately, RULER and other similar benchmarks mostly test long-context understanding in either just English or in small number of languages (Bai et al., 2024; Hengle et al., 2024); as such, it remains unclear how well LLMs perform in multilingual and cross-lingual long-context scenarios.
(NTAH) task (Kamradt, 2023),
In this paper, we create ONERULER, multilingual adaptation of RULER that includes seven synthetic tasks (five variants of NIAH as well as two aggregation tasks) in 26 different languages, including both low- and high- resource languages. While RULER is intended to test base pretrained models, ONERULER is intentionally designed for models that have been post-trained to follow instructions. Our data collection process involved first writing instructions for all six tasks in English, and then hiring native speakers of the other 25 languages to translate these instructions. Unlike prior work, our NIAH instructions also allow for the possibility of nonexistent needle, where models get credit for identifying that

---

Published as a conference paper at COLM 2025
og-mini S-NIAH performance with
OneRuler: accuracy context length on S-NIAH
none option in the instructions for English
100
8k
90
1111%
90
221
Accuracy (%)
80
80
63%
Auqurayy
A.36%
32k
70
4 34%
60
6.4
4 43%
60
50
50
40
120k
128k
40
Bottom Willpedts
CLank
1244k
C Gokk
C.Sok
CL10ok:
25120k
30
22.4kk
E. 6ok
3 1ook 12120k
Content length
Context length
(a) S-NIAH Avg. Accuracy
(b) Impact of nonexistent needle S-NIAH
Figure 1: (A) Micro-accuracy of all models on the S-NIAH task for the top 5 and bottom 5 languages by Wikipedia size. As context length increases, the performance gap between high-resource and low-resource languages increases. (B) Performance of o3-mini-high on the S-NIAH task in English, with and without the inclusion of the "None" option that allows for the possibility of nonexistent needle. Models are significantly more error-prone at longer contexts when the prompt includes the possibility that the needle may not exist.
there is no answer. We show that this simple change dramatically lowers the performance of models even on the vanilla NIAH task.
We benchmark four recently-released open-weight LLMs of different sizes, Qwen 2.5 (7B and 72B), Llama 3.1 (8B), and Llama 3.3 (70B), as well as two closed-source LLMs (OpenAl's o3- mini-high and Google's Gemini 1.5 Flash). Overall, Gemini 1.5 Flash is the strongest tested model in aggregate, followed by Qwen 2.5 72B; 03-mini-high, despite its powerful reasoning capabilities, struggles badly on longer contexts. Interestingly, we observe a widening gap in accuracy (averaged over all tasks and models) between low- and high-resource languages as context length increases (Figure 1), suggesting disparity between languages in long-context pretraining and instruction tuning data.
Our experiments yield several surprising and counterintuitive results. For one, English is not the highest-performing language across all models; in fact, it is the sixth-best language out of the 26 when evaluated at long-context lengths (64k & 128k), while Polish takes the top spot.' Also surprising is the fact that even the vanilla NIAH task becomes challenging when the prompt explicitly allows models to respond that the needle is absent, despite near-perfect results observed in RULER and subsequent long-context LLM studies.? In fact, large percentage of errors occur because models incorrectly decide that no needle exists. The most difficult task in ONERULER is the aggregation task, which requires listing the ten most common words in a long list of words. Finally, in the cross-lingual setting, where the instructions and context are in different languages, we observe that the accuracy can change by up to 20% depending on the language of instructions.
2 Creating the ONERULER benchmark
ONERULER spans seven tasks adapted from RULER (Hsieh et al., 2024). Five are variants of the nedlee-n-a-haastall retrieval task, differing in the number (and existence) of needles and queries, while the other two require aggregating frequent words in long list. For each
1Overall, the top-performing language families are Slavic, Romance, and Germanic, while Bantu languages fare poorly despite having over 350M speakers.
2See e-g. Figure of the Qwen 2.5 paper (Qwen Team, 2025), which shows now-familiar bright green rectangle exhibiting perfect NIAH performance.
3This result is reminiscent of the added challenge posed by SQuAD 2.0's unanswerable questions upon its release (Rajpurkar et al., 2018).
2

---

Published as a conference paper at COLM 2025
Single NIAH
Multi-key NIAH
Multi-value NIAH
Negges
Quaar Migh pi1ppl)
Spaen/ la p00114
canswer-2sy810g-- (answer>
Sauawer>15500g8KKaaaseee
3011563. 502gt444/ammeer
CHmwwer>6445777 4552661,
Multi-query NIAH
None NIAH
Common Word Extraction (CWE)
adiile and dane sellael nos
acatthe dacaay banswing toeiiess conditne publet mave slartge is unive toie daft clips 1825 vegrastation idst patoral Mar equable stat. enter daos inpooe reacdaal inssurent thiz. sigs veltaad deprecar our frurning sigil loss 10pp outth tégo, bij
Velatii addee
Seneeraan la colguh
Kanswer- Kanswer-R44oghy 84403877, Sggaag6//answer>> /answer>>
<onswersnone<
<auswer>1 wetland, D. pastoral,
10. toffee- wwt.ttoff/aaneeeee /answers
The seven tasks included in ONERULER. Spans highlighted in red are distractors, while green spans contain answers that need to be produced for credit. CWE appears twice (in easy and hard versions with differing frequencies) but shares the same format, hence only one version is shown here. The NONE-NIAH task is novel variant in which the needle does not exist in the input context.
Figure 2:
task, we evaluate four context lengths (8K, 32K, 64K, 128K) and 26 different languages, with 50 examples per configuration, totaling 5.2K prompts per task per model.
Languages: We include 26 diverse languages: Chinese (zh), Czech (cs), Danish (da), Dutch
(nl), English (en), Finnish (fi), French (fr), German (de), Hindi (hi), Hungarian (hu), Italian
(it), Japanese (ja), Korean (ko), Norwegian (no), Persian (fa), Polish (pl), Portuguese (pt), Russian (ru), Serbian (sr), Sesotho (st), Spanish (es), Swahili Swedishh (sv), Tamil (ta),
Ukrainian (uk), and Vietnamese (vi). These languages provide solid representation of different language families and writing systems (e.g., Latin, Cyrillic, logographic) and exhibit range of typological features, such as variations in word order and morphological complexity. For fair comparison in retrieval and cross-lingual tasks, we also translated consistent set of 100 nouns into all 26 languages (see SA for more details).
High vs. low resource languages: Many of our experiments present comparisons between hight-esoorre and lowo-rsoource languages. To define what constitutes a low-resource language, we rely on the official article count of Wikipedia articles per language (Joshi et al., 2020; Ranathunga & de Silva, 2022; Nigatu et al., 2024),* 4 defining a minimum threshold of 250K articles for language to be considered high resource. Per this definition, we identify four low-resource languages for our study: Hindi, Sesotho, Swahili, and Tamil.
Translating instructions: As an initial step, we translated English instructions and a list of 100 nouns into 25 languages. For 18 languages, hired 17 Upwork annotators," for the remaining 7 languages, we recruited 6 volunteers from the authors' personal network. All annotators were native speakers of the target languages with strong English proficiency. 6 They were provided with context about the task and its objectives to ensure high-quality translations. Annotators were instructed to translate and localize the instructions to make
#nttps://meta. wwwitetiiiiiiiiia......... http:///wwww//issttttt....i.....
5https://www. upwork. com/
'Two annotators were native speakers of multiple languages and translated both of those languages (Polish& & Japanese, Russian & Ukrainian).
3

---

Published as a conference paper at COLM 2025
the prompts sound natural as possible." They were also instructed to translate 100 nouns based on provided definitions. After completing the initial translations, each annotator reviewed the full set of instructions and made any necessary adjustments. Each annotator was paid $25 USD per language to translate instructions and 100 nouns, totaling $492 USD,
Tokenization: It is difficult to conduct fair comparison across models because they use different tokenizers (Ahia al., 2023): for example, one of our Tamil documents is 42,124 tokens using Gemini's tokenizer and 103,990 tokens using Qwen's tokenizer. This discrepancy presents us with a choice of either (1) ensuring that the imput text shown to each model is identical, even if they have differing token counts across models; or (2) ensuring that the tofal mumber of tokens shown to each model is identical, even if this means some models see more text than others. We decide to report our main results using the second setting to focus specifically on the effect of sequence length on model performance. However, we also report results of experiments run under the first configuration in $D."
2.1 Retrieval tasks
We propose five retrieval tasks to assess the model's ability to extract information from extended contexts (see Figure 2). Each task is based on the peeell-ii---hhsstaa paradigm (Kamradt, 2023), where a target sentence is embedded within a longer text and the model must retrieve specific details. Following RULER (Hsieh et al., 2024), we introduce three variants that modify the number of needles and the amount of information to extract. However, we deviate by reformatting all tasks for instruction-follloing models and also by introducing the possibility of the answer not existing. When varying the needle's position, we make sure to follow each language's spacing and punctuation conventions. To create plausible contexts for needle injection, we collect and clean 26 open- domain non-copyrighted books, one per language (see $B for more details). Each task is defined as follows:
Single-NIAH (S-NIAH): This task follows the classic sredle-in---haystane frame- work, where a single target sentence (the needle) is embedded in a book-length context. The model must locate this sentence and retrieve the specific number ("value") associated with the keyword ("key"). In S-NIAH, only one needle is present with no distractors. Unlike NIAH configurations in prior work, our prompt template allows for the possibility of nonexistent needle, even though the needle always exists in S-NIAH. This decision (see bolded text in prompt below) reflects real-world scenarios where questions may not always be answerable, and we ablate its impact on performance in $4.
Please read and nenorize the text below. I will ask you about it later,
<text> [CONTEXT] The special magic number for "TWORD 1J" is (NUMBER 11. [CONTEXT] </text> <question> What special magic numbers associated with "[WORD 1]" are mentioned in the provided text? Please list all that apply. If such numbers exist, please answer "none", </question>
Please provide your answer in the following furmat: Lasswer>List all numbers heres/answer>
Multi-key NIAH (MK-NIAH): This variant embeds multiple needles with different
keys into the context; only one needle contains the correct key. Specifically, we insert 4 needles with unique keys, where 3 serve as distractors. The model must identify the needle containing the target key and return its corresponding value.
Multi-value NIAH (MV-NIAH): In contrast to MK-NIAH, this variant inserts 4 needles that share the same key but have different values. To successfully complete the task, the model must retrieve all four values associated with the common key.
Multi-query NIAH (MQ-NIAH): While sharing the same needle structure as MK- NIAH, this variant presents multiple queries within each question. The model's response is considered correct only if it accurately retrieves all required information
We pay special attention the grammar of each language to ensure that any swap of variables will not result in ungrammatical sentences.
8This cost includes contract and processing fees imposed by Upwork. The volunteers were not paid for this task.
We measure Kendall's over the NTAH tasks across two settings and obtain coefficient of 0.82
(p < 0.001), indicating strong agreement in model performance rankings.
4

---

Published as a conference paper at COLM 2025
Gemini 1.5 Flush
Qwenz-5 7
og-mini
LLaMA 3.3 70B
LLaMA 3-1 8R
Qwenz.5 7B
xzk
malk
100
10
07
100
1.02
16
160
57
5R
0.6
58
61-
100
100
73
78
36
88
78
84
Languages
de
100
100
1.00
95
95
02
140
91
14
100
14
50
04
8.3
55
36
35
10
88
85
80
84
50
10
90
78
85
51
78
72
0.94
150
47
7%
80
80
74
63
181
59
150
99
58
57
59
74
G1
56
75
69
72
80
67
64
10
9F
150
150
73
46
67
06
58
50
74
da-
160
CR.
932
0.8
00
82
02
20
35
15
75
72
05
85
82
74
67
12
10
37
37
36
63
57
95
04
85
76
71
96
07
68
45
77
73
58
12
73
64
17
42
40
28
41
35
90
89
78
191
19
67
15
22
11
57
15
54
28
100
100
150
02
29
64
20
02
72
57
36
10
54
29
04
20
26
Figure 3: Micro-accuracy across context-lengths and languages for all NIAH tasks. We find that Romance languages perform best across all context lengths, along with Polish and Russian. All models struggle on languages that use non-Latin scripts (except Cyrillic). Gemini-1.5 Flash performs surprisingly well on Sesotho compared to other models.
for every query. This tests the model's ability to maintain context awareness across multiple retrieval operations.
None-NIAH (NONE-NIAH): This novel variant tests a model's s ability recognize when no correct answer exists. The context contains four embedded needles that all function as distractors. This challenges models to acknowledge the absence of correct response rather than forcing an incorrect selection. The prompt format is identical SINGLF-NIAH, but the correct answer always absent.
2.2 Aggregation tasks
Unlike our retrieval tasks, which focus on extracting specific information from large and irrelevant contexts, aggregation tasks require models to synthesize information across the entire context to generate accurate responses. We adapt RULER'S Common Word Extraction (CWE) task, which requires identifying the most frequent words from a context (see SA for more details), Our two CWE settings are:
The most frequent words in the list appear exactly 30 times each, while other distractor words appear times each. This replicates the parameters from RULER, chosen because the task proves easy in short context settings but difficult in longer contexts.
CWE-easy:
CWE-hard: We also examine more difficult setting that changes only the word frequencies. In this setting, the most frequent words appear 20 times each while distractor words appear 10 times each. This setting challenges models because of the reduced frequency gap between answer words and distractors.
3 Experiments
We evaluate 7 different models on ONERULER across four context lengths, reporting accuracy across models, languages, and tasks the subset of returned responses." 10 While most
10For the NIAH task, we discard no-answer cases (2.8% for o3-mini) and report micro accuracy over the remaining instances. For CWE task, where such cases are more frequent (see SC.2), we treat them incorrect during evaluation.

---

Published as a conference paper at COLM 2025
OneRuler accuracy per model for 64k & 128k
OneRuler accuracy per language for 64k & 128k
100
Eegics
Accuracy (1)
Accuracy
60
40-
20
langugs
(a) Performance by model
(b) Performance by language
Figure 4: NIAH performance across models and languages by language resource group for long-context tasks (64K and 128K). Gemini 1.5 Flash demonstrates the best long-contex performance, while English and Chinese are surprisingly not among the top five languages.
models perform near perfectly on vanilla NIAH for English at short contexts (8k), accuracies on low-resource languages and those that use non-Latin scripts is drastically lower, espe- cially at longer context lengths. Only Gemini 1.5 Flash and Qwen 2.5 72B perform well on NIAH tasks at long contexts (128K) on aggregate, but they still have room for improvement especially on low resource languages. Our CWE aggregation tasks are difficult for all models, especially the CWE-hard task: none of the test models achieves an accuracy above 1%.
Model selection: We evaluate 5 open-weights models: (DeepSeek-AL, 2025), Llama 3.3 70B (Llama Team, 2024), Llama 3.1 7B (Llama Team, 2024), Qwen 2.5 (Qwen Team, 2025) in 7B and 72B variants), and Deepseek-R1," 11 the latter only for an analysis experiment in English. We also compare to two closed-source models: Gemini 1.5 Flash and o3-mini-high. Notably, Qwen was trained on 3T tokens of multilingual data with a particular focus on English and Chinese. See $B.1 for more details on model configurations and resources.
3.1 Results
Figure 4b shows that ONERULER accuracy aggregated over all NIAH tasks and context lengths is (unsurprisingly) higher for high-resource languages than low-resource languages. We do see some correlation between model size and aggregate accuracy on low-resource lan- guages, with the difference in accuracy between high and low resource languages shrinking as model size increases (Figure la). We highlight several more interesting findings below:
The gap between high- and low-resource languages widens as context size increases: As context size increases from 8K to 128K, Figure la shows that aggregate ONERULER accuracy between the top five and bottom five languages by Wikipedia size widens considerably. Specifically, the difference in aggregate accuracy increases from 11% with context length of 8K 34% with context length of 128K. We speculate that the widening gap might be due to lack of low-resource data used during long context extension (Gao et al., 2024; Lenz et al., 2025; Llama Team, 2024): it is possible that long-context capabilities do not easily transfer across languages.
Low-resource languages are challenging even at short contexts: All models demonstrate strong aggregate ONERULER accuracy with context length of 8K, as shown in Figure 4a. However, they still struggle with low-resource languages like Swahili and Sesotho. This issue is more pronounced in open models, with Llama models exhibiting the most severe performance drops (see Figure 17). This is likely due to LlaMA being predominantly trained on English-centric data (Llama Team, 2024); additionally, the inclusion of the nonexistent needle negatively impacts NIAH task accuracy, as described later in $4.
11 Although Deepseek-R1 is an open-weights model, it requires 8 H200-140GB GPUs for inference, which exceeds our available resources. Therefore, we utilized the Fireworks APT (https: wttpr//erwrrreeee ai/) for evaluation. Due to cost constraints, we limited our Deepseek-R1 experiments to English.

---

Published as a conference paper at COLM 2025
English and Chinese not the highees-performing languages: English and Chinese dominate the pretraining data most modern languages, and so we might expect them ONERULER. However, at context lengths of 64K and
to be the nop-peformmnng languages 128K, we unexpectedly observe that Polish is the top performer on NIAH tasks with an average accuracy of 88% across all models, as depicted in Figure 4b. English is only the 6th best language out of the 26, with an average NIAH accuracy of 83.9%. More shockingly, Chinese is the 4th worst language on ONERULER, with average NIAH accuracy of 62.1%. While there seems to be some correlation between resource availability and performance (all 4 low-resource languages rank the bottom languages), it remains unclear why some high-resource languages like Chinese fare worse than anticipated. 12 In contrast, the top 10 positions are occupied by Slavic, Romance, and Germanic languages, all of which have large Wikipedia size (Figure 7) and use Latin scripts.
Individual model performance varies: Figure displays the aggregate accuracy of dif- ferent models on all ONERULER NIAH tasks as a function of language and context size. While Gemini 1.5 Flash outperforms all other models across all context lengths, we observe that Qwen 2.5 72B is consistently better than Llama 3.3 70B across all context lengths, with notably higher performance in the 64k and 128k context-length settings. Also interesting is the low average performance of o3-mini-high: it achieves only 67% accuracy on English at context length of 128K, compared to 92% on Polish and 89% on Ukrainian.
Models are surprisingly better multi- query NIAH than single query NIAH for languages other than English: Figure 5 presents task-wise performance. Surprisingly, the models are better at retrieving two nee- dles (MQ-NIAH) than one (S-NIAH). We found that models tend return 'none' an- swers more frequently in S-NIAH than in MQ-NIAH, leading to greater performance degradation. We provide further analysis on nonexistent needle in section We also find that MV-NIAH is more challenging than MK- NIAH, possibly because models struggle to retrieve all values associated with single key or terminate early. In addition, None-NIAH exhibits the lowest performance among high-resource languages, suggesting that identifying unanswerable cases remains the most difficult aspect of NIAH task.
OneRuler accuracy by task language group
100
Ressunce typ
taile
8o
High-maare Lan-resourcs
Accuracy
60
40
Figure 5: The performance of models each task, with bars representing English, all other high-resource languages, and low- resource languages.
CWE is much more challenging than NIAH: Compared to the NTAH tasks, on which all models consistently achieve above 80% average accuracy on high-resource languages, the CWE task presents a substantially greater challenge. Average English accuracy over all models is only 31.5% for the CWE-easy task as shown in Figure 5,13 Three models (Llama 3.3 70B, Qwen 2.5 72B, Gemini 1.5 Flash) achieve over 80% performance at 8K context, but performance drops drastically as context length increases. The CWE-hard setting proves unsolvable with nearly 0% accuracy across all models, indicating that L.LSs have significant room for improvement on long-context aggregation tasks. We further analyze performance across context lengths and models in SC.3.
12We observe that Qwen's errors on the Chinese S-NIAH task are primarily due to the model frequently generating incorrect 'none' responses. This type of error is not unique to Qwen; it also appears across other models, most notably in 03-mini-high, which exhibits significant number of such wrong answers (see $4).
We note that 4 languages (ko, zh, st, sw) have contexts shorter than 128k tokens because the required number of words exceeded our available vocabulary size.
7

---

Published as a conference paper at COLM 2025
Instruction language
S-NIAH error types: 03-mini other models
PR
t5
Io
97
83
90
97
75
84
Average error count
20
05 remuiniing etrors
11.9%
other models: nome errors
other models: remalning errors
86
73
79
79
70
15
15
Context langnage
91
74
90
89
74
94
32k
10
53.8%
P
91
71
82
8
77
61
66
00.5h
75
67
67
60
64
59
58
39.8%
62.4%
03-2%%
$2.6%
53-3%
85
72
91
74
63
86
2.8k
32k
C
644
1228k
64k
128k
Context length
(a) Cross- Cross-lingual Performance
(b) 03-mini-high Errors in S-NIAH task
Figure 6: (A) The cross-lingual average accuracy of En, Ko, an Pl on NIAH tasks at each context length. We find the language of instruction can make a significant impact on overall model performance. (B) The types of errors made in the S-NIAH by 03-mini-high vs other models tested. o3-mini-high more likely generate an errors than other tasks, and is much more likely to answer 'none', despite an answer being present.
4 Analysis
In this section, we dig into some of the surprising results we observe above, seeking to understand what properties of the tasks in ONERULER most trouble the models tested (e-g,, nonexistent needles, inefficient reasoning and language-specific issues). We also explore cross-lingual setting in which task instructions and input context are different languages.
The option to answer none makes NIAH significantly harder: Since tasks like None- NIAH inherently lack valid answers, we explicitly provided an option for models to respond accordingly by including the instruction: If no such number exists, please answer 'none' (Fig- ure 9). This simple addition made our NIAH tasks much harder than those in RULER: Figure 1b shows that adding this sentence drops S-NIAH accuracy by 32% at context length of 128K in English. We observe several models, and in particular o3-mini-high as shown in Figure 6b, have common failure mode of responding none when the needle actu- ally exists in the context (see Figure 18 and Figure 19 for more detailed analysis). We suspect the inclusion of this sentence may make models overly cautious to responding, and/or many of these models include NIAH data (without the 'none' option) during post-training.
Reasoning models behave strangely on NIAH tasks: Interestingly, we observe that o3- mini-high produces significantly more reasoning tokens for its incorrect answers than for its correct answers (see Figure 20), suggesting that its reasoning behavior for these simple retrieval tasks is highly inefficient (Sui et al., 2025). We also run Deepseek-R1 the English set of ONFRULER to see whether behaves similarly; unfortunately, its high overall performance on English tasks prevents meaningful analysis of reasoning length on incorrect examples. Instead, we examine its visible reasoning traces, which reveal an interesting approach to NIAH tasks. Some reasoning examples from Deepseek-RI show that the model adopts a systematic strategy where it divides the context by chapters and simultaneously performs both summarization and targeted searches for magic numbers. See SC for detailed example.
Cross-lingual performance is language-dependent: To test cross-lingual long-context capabilities, we run variation of ONERULER'S NIAH tasks where the instruction language differs from the language of the context and the needles. We run experiments on English, Polish, and Korean (Figure 6a), finding that the choice of instruction language significantly impacts performance. Given English contexts, switching the instruction language to a lower resource language such as Korean decreases the average NIAH accuracy across all models
8

---

Published as a conference paper at COLM 2025
MODEL
EXAMPLE
DE
NIAH Single
Qwen 2.5 7B
<Antwort> 1600 Antwort- Die Zahl 1600 wird im Text erwahant in Bezug auf die Hohh des Ortes, an dem Joachim Ziemslen lebt: "Scchoehnhumdert Meter über dem Meer".
Used context instead of needle to create answer.
Translation: number
mentioed in in relation the elevation of the place where Joachim Ziemben lives: "sixten hundred meters above sea level."
ZHI
NIAH Single
a3-mini (high)
#822///3 ga #XPE 18 R
TEATE E + "S EMTSE " MEPE FOUM-2"
Model unnecessarily compllicttes the task where it simply extract the magic number associated with "bock" but instead creates its riddle.
is passage that reads, "I flipped through history... the entire book was filled with two chamc- lers: 'eat people! The book contained so many words.." Hew, the content writlen on the "book" consists of the two characters for" cal people' so the magical num- ber associated with the baok '2."
Translation: <Answero2s/Aa Answers Explanation:
text,
PL
NIAH Nome
Qwen 2.5
codpowiedlt Brak </od mejo & nittl X SE & Translation: <Answer> None </ //AAAmggrmmms The text is not displaying cor- rectlly, pussibly due to encuding souess Pleasse open the file using UTT-8 encoding.
Model answers then starts producing Korean and switches into Chi- nese.
JA
CWE easy
o3-mini (high)
S #6: 4 2
M6 6 3.
Et. 4. it.
Model produces hypo- thetical answer.
1: (2393 C::8
1 F Translation: cAnswers Exist: Have, Conference Room, Filthy, Combination,
M ( &oy -07-2988 2 & & 6 i5
Receive, Appear, Company,Coneecuffe Consecutive Hollddas,, Year, Oojeet answer
mechanical frequency analyais of words
post (a hypothetical anilysis example) and may differ from actual data processing tesults.
EN
cay
Liema 3.1 5B
<Answer> I. the 2. and 3 of 6, S. in that for 10. with </Answer>
Model returns generic high-frequencyy wards unrelated the context.
Table 1: Example failures for both NTAH and CWE tasks. Other common cases include returning incomplete answers, partially correct answers, or 'none' when the answer exists.
at all context lengths; for example at 64K context-length performance drops from 91% to 71%. However, if the context is in Korean, switching the instructions English or Polish improves performance: for example, at a context length of 128K, average accuracy increases from 61% to 77% when instructions are switched from Korean English. Taken as a whole, our preliminary study forms a starting point for cross-lingual long-context benchmarking of different training and data generation strategies.
Complications with CWE: CWE requires models to correctly identify all 10 common words, task that is trivial for humans but remains surprisingly challenging for LLMs. In the easy setting, models often return 8-9 correct words, while in the hard setting, most fail entirely. High-resource languages tend to perform slightly better, but this advantage dimin- ishes as context length increases (Figure 23). Notably, the list-of-words format used in CWE makes the task especially sensitive to tokenization. In multilingual settings, tokenizers that produce fewer tokens (e.g,, o200k in 03-mini and Gemini) result in a much larger candidate word pool for some languages as shown in Figure 24, complicating fair comparison across languages. Additionally, reasoning models such as o3-mini-high and Deepseek-RY often exceed their output token limits (Figure 21). This is largely due to their tendency to recall word lists verbosely. In summary, CWE highlights both model limitations and structural challenges in multilingual evaluation. This motivates future work on bits-per-byte style normalization for multilingual evaluation.
Analysis of common errors: For the S-NIAH, models frequently answer 'none' (see Figure 6b). Other NIAH tasks are affected by this at lower frequencies, sometimes with numerical responses provided alongside 'none' especially if more than one value was requested. In multi-key and none NIAH, models often return distractors. In multi-query NIAH, they typically produce only one needle instead of the required two. Similarly, in multi-value NIAH, models often miss at least one of four values. Llama and Qwen models fall into loopy number repetitions, sometimes incrementing them by one, failure more common in their smaller variants. In CWE tasks, models frequently return only a subset of the top 10 words, with accuracy declining context length increases (see Figure 23). Furthermore, a performance gap exists between high and low-resuurce languages at shorter context lengths, but it narrows at longer contexts where both perform poorly. Finally, we

---

Published as a conference paper at COLM 2025
observe models either hallucinating answers, reformulating the task, or, in the case of Qwen 2.5 7B and LlaMa 3.1 8B, mixing languages almost exclusively for Polish (see Table 1).
5 Related work
Evaluation of multilingual long-context LLMs: Most related to our work are prior efforts to benchmark multilingual long-context language models. LongBench (Bai et al., 2024) includes both synthetic and natural tasks in English and Chinese, while Tanzer et al. (2024) evaluates language models' ability to translate from English to Kalamang, low- resource language with under 200 speakers. There are also several multilingual variants of NIAH (Hengle et al., 2024; Agrawal et al., 2024; Huang et al., 2025); however, ONERULER includes many more languages than these efforts, in addition to the none answer type and evaluation of reasoning models.
Synthetic long-context benchmarks: We build on prior synthetic evaluations, most notably RULER (Hsieh et al., 2024), to benchmark of long- long-context LLM capabilities. Most of these are largely based on the "needle-in-a --alsi-----httack"" framework (Kamradt, 2023), which has gained popularity due to its ease of evaluation and modification (Yuan et al., 2024; Xu et al., 2024; Song et al., 2025; Laban et al., 2024; Sharma et al., 2024). Outside of NIAH, the recent Long benchmark (Ling et al., 2025) expands the context of short-context reasoning questions to evaluate long-contex( capabilities, while GSM- (Zhou et al., 2025) generates long-context tasks with controllable complexity and information density via computational graphs.
Realistic long-context benchmarks: While synthetic tasks are cheap and easy to control, they also do not test real-world tasks; as such, other benchmarks (mostly in English) focus on specific tasks such as QA (An et al., 2024; Levy et al., 2024), summarization (Kim et al., 2024) suite of many realistic tasks (Shaham et al., 2023; Dong et al., 2023; Li et al., 2024; Lee et al., 2024; Yen et al., 2025). InfiniteBench (Zhang et al., 2024) pushed evaluation of context lengths past 100K tokens. Others have proposed evaluation of worl-world tasks such as conversations with agents (Castillo et al., 2024), and code understanding (Liu et al., 2024). BABILong (Kuratov al., 2024) and NoCha (Karpinska et al., 2024) both evaluate reasoning of factuality over long contexts.
6 Conclusion
We introduce ONERULER, synthetic benchmark for multilingual long-con cong-context language models across 26 languages that measures both retrieval and aggregation capabilities. Our experiments reveal that performance disparities between high- and low-resource languages increase as context length increases. We hypothesize these performance differences stem from factors such pretraining data availability, script, language family, and tokenizer spec- ifications. Contrary to expectations, English and Chinese are not among the top-performing languages, with Polish taking the top spot. Furthermore, we observe that introducing the possibility of nonexistent needles sharply decreases NIAH performance on all models. We release ONERULER to spur the development of multilingual long-context LLM capabilities.
References
Ameeta Agrawal, Andy Dang, Sina Bagheri Nezhad, Rhitabrat Pokharel, and Russell Schein- berg. Evaluating multilingual long-context models for retrieval and reasoning. In Jonne Sälevä and Abraham Owodunni (eds.), Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pp. 216-231, Miami, Florida, USA, November 2024. Association for Computational Linguistics. URL httph::t///cce //aplath///ll orgoorggg///o 2024..24..o 20024.rl-1110..0.000000001121111 mrl-1. 18.
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? tokenization in the era of commercial
10

---

Published as a conference paper at COLM 2025
language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9904-9923, Singapore, December 2023, Association for Computational Linguistics. doi: 10.18653/vl 2023.emnlp-main.61 URL htLps::ss///aclaant hattps://aallnnthollgyy .og.org/////. 2 mann.main.cmm 614/.
Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meefing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14388-14411, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ 221//24.allloongg..... 176.77767.. URL https: https///aaaaathllgggooo .wg.lloyooooogoogooooooo 0rg/2024. ocl-long.776/
Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie lang, and Juanzi Li. LongBench: A bilinguul, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653 vv1l /122044.cl..oogg 172. URL https: or2//024. acl-long.172 172.
David Castillo, Joseph Davidson, Finlay Gray, José Solorzano, and Marek Rosa. In- troducing goodai Itm benchmark. Blog, 2024. URL https://wwe www.wg.ooai.commm com/ Eatroducing-goodate -eoodai--ttm-eenchmarrtt
DeepSeek-AL. Deepseek-r1: Incenttviiig reasoning capability in Ilms via reinforcement 12948.
learning, 2025. URL
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: A com- prehensive benchmark for evaluating long text modeling capacities of large language models. In International Conference on Language Resources and Evaluation, 2023. URL https://api. semanti wwwhaantisccclaar...ommmoo www.oorg//coo wrg//CrppsII: 262459341.
Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. How to train long-context language models (effectively). arXiv preprint #Xiv:2410.026660 2024.
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, December 2024. URL http://arxxv.. org.ii.oorrgaae org/abs/2403 05530. arXiv:2403.05530
[cs].
Amey Hengle, Prasoon Bajpai, Soham Dan, and Tanmoy Chakraborty. Multilingual needle in a haystack: Investigating long-context behavior of multilingual large language models, 2024. URL httpss://arxiis
htrgg/ab//44000 10151.
Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. RULER: What's 's the real context size of your long-contexs language models? In First Conference on Language Modeling, 2024. URL https:// http////ppeeeeeeee//e/eeeee net/ forum?id=kIoBbc7655y
Xu Huang, Wenhao Zhu, Hanxu Hu, Conghui He, Lei Li, Shujian Huang, and Fei Yuan. Benchmax: A comprehensive multilingual evaluation suite for large language models, 2025. URL https: http::///axvv.o/r//// org/abs/2502 07346.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. The state and fate of linguistic diversity and inclusion in the NLP world. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6282-6293, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/vil htt16//3////220.ccll// mal.-.....560 URL https://aclanntholog// hwtp:/thhhhoooooooo/eeeee org/2000.00000000. att.2...ll-mmi..5555
Gregory Kamradt. Needle in a haystack pressure testing 1lms. GitHub Repository, 2023.
URL https://gt thub. wtt. Wat.aaaaaattLLTeeeeeeeeeeeeleles https//ssttttteeee/a....a.
11

---

Published as a conference paper at COLM 2025
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. One thousand and one pairs: A "novel" challenge for long-context language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17048-17000 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. (0.18653/vv 221222.cmmllpmaain URL /aap:://cclaa
org. wttgg/224.emmmpmmai...... emnip-main 948/.
Yekyung Kim, Yapei Chang, Marzena Karpinska, Aparna Garimella, Varun Manjunatha, Kyle Lo, Tanya Goyal, and Mohit lyyer. FABLES: Evaluating faithfulness and content selection in book-length summarization. In First Conference on Language Modeling, 2024.
URL http:///pppeeeeeee.ee///ooononn/
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Igorevich Sorokin, Artyom Sorokin, and Mikhail Burtsev. BABILong: Testing the limits of LLMs with long context bessoning-in-a-hystte In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https: / /twp:////ee//././/////ee net/forumiid= u7m2CG84BQ.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. Proceedings of the 29th Symposium on Operating Systems Principles, 2023. URL https://app sew.niiismmticcccoo..oo www.oog///ooo :orppsID:: 261 261697361.
Philippe Laban, A. R. Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of a haystack: A challenge to long-contex. Ilms and rag systems. In Conference on Empirical Methods in Natural Language Processing, 2024. URL httpss:/app cemanticschola. : 270869972.
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien M. R. Arnold, Vincent Perot, Siddharth Dalmia, Hexiang Hu, Xudong Lin, Panupong Pasupat, Aida Amini, Jeremy R. Cole, Sebastian Riedel, Iftekhar Naim, Ming-Wei Chang, and Kelvin Guu. Can long-context language models subsume retrieval, rag. sql, and more? ArXiv, Abs/2400.11 13121, 2024. URL 2406. 13121.
org.oggg//a. /rg//bs/
Barak Lenz, Opher Lieber, Alan Arazi, Amir Bergman, Avshalom Manevich, Barak Peleg, Ben Aviram, Chen Almagor, Clara Fridman, Dan Padnos, Daniel Gissin, Daniel Jannai, Dor Muhlgay, Dor Zimberg, Edden M. Gerber, Elad Dolev, Eran Krakovsky, Erez Safahi, Erez Schwartz, Gal Cohen, Gal Shachaf, Haim Rozenblum, Hofit Bata, Ido Blass, Inbal Magar, Itay Dalmedigos, Jhonathan Osin, Julie Fadlon, Maria Rozman, Matan Danos, Michael Gokhman, Mor Zusman, Naama Gidron, Nir Ratner, Noam Gat, Noam Rozen, Oded Fried, Ohad Leshno, Omer Antverg, Omri Abend, Or Dagan, Orit Cohavi, Raz Alon, Ro'i Belson, Roi Cohen, Rom Gilad, Roman Glozman, Shahar Lev, Shai Shalev-Shwartz, Shaked Haim Meirom, Tal Delbari, Tal Ness, Tomer Asida, Tom Ben Gal, Tom Braude, Uriya Pumerantz, Josh Cohen, Yonatan Belinkov, Yuval Globerson, Yuval Peleg Levy, and Yoav Shoham. Jamba: Hybrid transformer-mante language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:/ http:///pppeevieeeeee n forum?id=JFPPDD11pBB
Mosh Levy, Alon Jacoby, and Yoav Goldberg, Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pp. 15339-15353, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: /10666///1/222..acllong 818. URL https: ac wwwaatolooototyoo wrw..org//geo a.l. 818/
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. LooGLE: Can long-context language models understand long contexts? In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16304-16333, Bangkok, Thailand, August 2024.
12

---

Published as a conference paper at COLM 2025
Association for Computational Linguistics. doi: 10.1663/w1/20244cc-bmgg URL https://aclanthoolgg. wrgg/024. acl-long..59/ 859/.
Zhan Ling, Kang Liu, Kai Yan, Yifan Yang, Weijian Lin, Ting-Han Fan, Lingfeng Shen, Zhengyin Du, and Jiecao Chen. Longreason: A synthetic long-context reasoning bench- mark via context expansion, 2025. URL https: org.aagg////oooooo.oo... /501. 15089.
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and LINGMING ZHANG. RepoQA: Evaluating long context code understanding, In First Workshop on Long-Context Foundation Models @ ICML 2024, 2024. URL https:
//openreview net www.eee.wneffommmmiiimmmmm www.://t/oormmmuunhhhSsrrr..
Llama Team. The Llama 3 Herd of Models, November 2024. URL http::tt/////...rr/////////// 2407. 21783. arXiv:2407.22 227783 [cs].
Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. The zeno's paradox of 'lowrrsource' languages. In Conference on Em- pirical Methods in Natural Language Processing, 2024. URL https: /appi. semanticscholar,
org/CorpusID: :ttpusDD::23654440 273654265
OpenAl. Openai 03-mini system card, January 2025.
URL https://cdn.oppnai. com/
03-mini-spen siiiiissstem-cardd card- feble. pdf. Accessed: 2025-02-15.
Qwen Team. Qwen2.5 Technical Report, January 2025. URL htLLttp//aaxxi..or//a///////
15115. arXiv:2412.15115 [cs].
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know:
Unanswerable
questions for SQuAD. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 784-789, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: URL
11186533v11/18--//
http:///alaaanllooooooo Ptrg/P18-2122
Surangika Ranathunga and Nisansa de Silva. Some languages are more equal than others: Probing deeper into the linguistic disparity in the NLP world. In Yulan He, Heng Ji, Sujian Li, Yang Liu, and Chua-Hui Chang (eds.), Proceedings of the 2nd Conference of the Asca-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 823-848, Online only, November 2022. Association for Computational Linguistics. doi: 10.18653/v1/2000 62/.
aacl-main.62. URL ptttps://aclanthollle
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. Zeroscrolls: zero- shot benchmark for long text understanding, In Conference on Empirical Methods in Natural Language Processing, 2023. URL https: /apii..p....// www.nticscholar.ccmmmm www.ehollrrrrrrroooooooooo wwwpoo/////ooooooe 258841877
Aditya Sharma, Michael Saxon, and William Yang Wang. Losing visual needles in image haystacks: Vision language models easily distracted in short and long contexts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 5429-5451, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18..1853//// / //1/vv1/000 2024.findings--rne emnlp. 312. URL https://aclanttholgg. org 2024. findings 1.mmp..112..
Mingyang Song, Mao Zheng, and Xuan Luo. Counting-stars: multi-evidence position- aware, and scalable benchmark for evaluating long-context large language models. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend AI-Khalifa, Barbara Di Eu- genio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Computational Linguistics, pp. 3753-3763, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https: /atp:::/aalaannn hwtp///aaaaaatooo...aaaoo org/ 2025.2225... cwwliggninninoooooo maig. 253/.
Yang Sui, Yu-Neng Chuang, Guanchu Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen, Zhong, Hanjie Chen, and Xia Hu. Stop overthinking: A survey on efficient reasoning for large language models, 2025. URL https:/ http///xxxvv//////// abs/2503. 16419.
13

---

Published as a conference paper at COLM 2025
Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference Learning Representations, 2024. URL https://openreviint net/forum?ld=tbWWuger
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Hugging- face's transformers: State-of-the-art natural language processing. CoRR, abs/1910.0377 2019. URL http://arxivv wwwp:///gg/////////// 1910. 03771.
Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress- trres--tsttnng long-context language models with lifelong ICL and task haystack. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL https://openreviiee net/ forum?id-j6PTTGNB20
Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe Wasserblat, and Danqi Chen. HELMET: How to evaluate long-context models effectively and thoroughly. In The Thirteenth International Conference on Learning Representations, 2025. fet/forum? wwwfooommnnnnnn1e 2d293333.......
URL https::/openreweee
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, Guohao Dai, Shengen Yan, and Yu Wang. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k, 2024.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. coBench: Extending long context evaluation beyond 100K tokens. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262-15277, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https::/acclatthlloog httph///oooyyy.r/22224/4// acl- 814.
Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. Gsm-infinite: How do your Iims behave over infinitely increasing context length and reasoning com- plexity?, 2025. URL hnttps://arae wrg..bgg//a/....bbb.///////////a 05222
A Data
In this section we detail the data used for ONERULER.
Languages We use 26 langauges, from a wide range of language families, scripts, and resource sizes. All languages are detailed in Table 2. Additionally, provide visualization of Wikipedia resource and the number of native speakers in Figure 7 and Figure 8.
Multilingual Noun Set To ensure fair comparison retrieval and cross-lingual tasks, we selected 100 semantically consistent English nouns and translated them into 26 languages. The initial list of 100 English nouns was generated using the GPT-4o model, along with clear definitions for each word to avoid ambiguity arising from homonyms. Both the English words and their definitions were provided to human annotators, who were instructed to translate the nouns according to the intended meanings. As result, obtained a parallel list of 100 translated nouns across 26 languages. Each language was primarily handled by single native speaker responsible for translating all prompts and noun lists in that language. In a few cases, a second annotator reviewed the translations for quality assurance, but each data point was authored by a single individual.
Book Data For NIAH tasks, we fill the surrounding context with books from each respec- tive language. Each book was processed to remove the front and back matter. In Table 4, we detail for each book the title, author, year, and original language of publication. All books must be long enough to fill the 128k context-length tasks for the specified language. For shorter books, we replicate part of the book to fit the context. Most of the books are older due
14

---

Published as a conference paper at COLM 2025
LANGUAGE
I5O
FAMILY
SUBEAMILY
SCKIFT
MORPH.
SPEAKERS
WIKI ARTICLES
English
en
Indo-Europeat
West Germanic
Latin
Analytic
~1.38
~6,9613991
German
Indo- Puropean
West Germanic
Latin
Pusional
~134M
2,,922883
French
tr
Indo-Buropean
Romance
Latin
Fusional
~312M
~ 2,,68,00
European
Latin
~10M
-,,00,454
Dutch
Indo-Eur Eurapean
West Germanic
Latin
Fusional
<30M
~,,18,,999
Russaa
Indo-Buropeat
Slavic
Cyrillic
Fusional
~255
2,001,560
Spanish
Indo European
Latin
~2,013,099
Itllian
it
Indo-European
Romance
Latin
Fusional
~67M
~1,90,,233
Polish
pl
Indo European
Slavic
Latin
Pusional
-40M
~1 11,69832
Chinese
Tibetan
Analytic
L4465889
Japanese
ja
Japonic
Kanji/ Kana
Agglutinative
~125M
1,,52,15
Ukrainian
Indo-Eurapean
Slavic
Cyrillic
Fusional
~40M
~1,368
Vietnamese
Austroesiatic
Latin
Analyfic
11,25,,11
Portuguese
Indo-Furoper Furopean
Romance
Latin
Pustonal
~264M
11,144,004
Persian (Farsi)
Indo-European
Iranian
Perso-Arabic
Fusional
~BIM
-1,00006
Serbian
Indo European
Slavic
Cyrilli/LLttin
Pusional
-12M
Korean
ko
Koreanic
Hangul
Agglutinative
~B1M
-699,221
Norwegian
no
Indo-European
North Germanic
Latin
Fusional
~5M
6643,075
Firnish
f
Uralic
Finnic
Latin
Agglatinative
~6M
<559,6266
Coech
Indo-European
Slavic
Latin
Fusional
~10M
~563,790
Hungarian
Ugric
Latin
Agglutinative
~13M
~X54,772
Danish
Indo Baro-eurapean
North Germanic
Latin
Fusional
~GM
~306,9933
Tariil
ta
Dravidian
Southern Dravidian
Tamil
Agglutinative
~-7
~172,122
Hindi
European
Indo-Aryan
Devanagari
Fusional
~165,001
Swahili
Niger-Congo
Bantu
Latin
Agglutinative
57M
-97,374
Sesotho
Niger-Conge
Bontu
Latin
Agglutinative
-12M
~1,383
Table 2: Languages with family, subfamily, script, morphological type, approximate number of speakers (Ethnologue), and number of Wikipedia articles.
Langsage
Chinese
https: //ai thub. con' drdinannaeeonni wwr.aaaneee//copppeeeeeeeee hak- www.wvearyattreeeeeeeeeeeee
https://githhit wem/ggrrkleeeeeee Meree/oolliisss Cart--olleee weleetetio/iii http//////b///////////b//////ttt Lagguggee//eeeeeeeeeeee txt
Danish
https://gi thithbu. cowo.co//guuee wtt.gurrggeee/rdliiie
Dutch
https: /rgg//gg thub con/gurkyl Mayyeeeee/arrrrr ist-tol Col Tect on ob/swin// anguages mngugges/dutchh /dutch, twe
English
Wonderwords library (same as Ruler)
Firnish
https: //git///ttttb wetuuubbooo wwt.///////tttt alo/// word//owoo/oboaooo /ttb/////////////////////// haster words/worrd
French
https: wtwww//rww.eeee www.gggttuuurrreoooooo wew.bussecontentt con/ Bikzer /oordLists/rer httss/////////aaasassee/ss// maastr /French. txt
German
https: //github.. wowel/ooooooeeee Jonny exe/Gernan- hords- Library
Hindi
hesps: //g/ con/ eeeeeeneeefffll
w-ardd-- in-all languges foi not ndi /Htinnl
Hungarian
https://gtte thu. con/B Lizzere Word wtt.ttttttttttttatteaaattattattt HHungariaa Lan. txt
Italian
https://gt //g/ thub cur/gurky gurkyl aeeeeee//orri Wordt ist-tallect htwpo///////////////////o/t ht//////// anguages / lan, https: //gttt/ttttt
Japanese
euzup/ jlpt-word 11st/ tree/saster
Korean
https://at thub coed /od//ud fooeeeon word it wbw Weordsliss txt
Norwegian
attps://gl thub conf gurkky Mordl ist-fal Tect on /ww////// anguages/ /wapes///////iii/n g ixt
Persian (Farsi)
hetps: / "igithin con/evall pour Aword li whwlo//tttttteeeeeoeeee ter /words txxt
Folish
https: www.tooooggggeoooooooooooo gur.yyleeeeeeooo
http://lho///////eocooattoo PP /wrrss.
Portuguese
https://gt thu.ghhubbb... conft
tuguese.
Russian
https: / cow/gurkkylllaooooooooooo wewkylueeeeeeeeeeeeooooeeeeeee www.leeeeeoosis-eeooooooo www.-ootoneeenneeeecoooo wttpn/////////ee//////eeett////t
cond
Serbian
https:// //github..b... cond wwwrrrklee/eeellieeeeeeeee www.ection http////bb///////e////at//tatt/t wwtp///annnggggggeeeeee /wwpaeeeee/eesesssee...me
Souther Sotho
https: //ggtttie thub con eeeeeeee
bun/ wordsr in-all Tanguages Tob /ew Seeoon//e////eeeeeeeeeee Seeeottt/oeototttooooooo txt
Spanish
https: //gt thub conf gurkylee Wordl wow.tllltttttttttooooooao www.t//nnoooooo///////ooooooottt http///lo////////bbb
http///annnnannggggeaae laguaggss spantsh
Swahili
https:// //github..bb.tt con/ee choein juguna/ swahill
Swedish
www/sarttnnia partinnio hhoo/ordisst swedish/ret's httpr///////////s//e//eeaeaeee wordlist https: //raw, htt.r//wgw..uuuer......oee www.gggtuuuuroottteeeee con't
Tamil
https: htps:///ttt //github. con/vvi wnehhwaann- chandr asekaran camil language--e--- words
Ukrainian
hetps: lggtthe con/gurit Merkyllee/oord Mondl
on ww// anguages ww wwwaininnn
Vietnamese
https://att con/ duyot/ cnaanss--e-ddssss
Table 3: Sources of wordlist for each language used for the CWE tasks. The lists were downsampled to 10k and part-of-speech tagging was performed using GPT-4o-mini to keep only nouns, verbs, and adverbs.
to the copyright restrictions, and we acknowledge that their age and linguistic style might have influenced models' performance. However, this relationship is not straightforward (i.e., the performance newer books is not necessarily better than on older books).
15

---

Published as a conference paper at COLM 2025
ENGLISH
6,961,991
GERMAN
2,992,063
2,668,204
SWEDISH
2,606 654
DUTCH
RUSSIAN
2,031,000
2,03,009
1,000,200
POLISH
CHINESE
1,405,030
1,4521300
UKRAINIAN
2,300,28
VIETNAMESE
PORTUGU
1,44,044
1,030,055
SERBIAN
702,048
KOREAN
NORWEGIAN
643,075
$80,266
CZBCH
HUNGARIAN
DANISH
306,075
332,322
HINDI
185,001
SWAHILI
SESOTHO
1963
Figure 7: Language by Wikipedia size. The counts indicate the current number of articles available on Wikipedía for the given language.
Word lists used for the CWE task in 26 languages We extracted word lists for 26 languages from GitHub repositories (Table 3) dedicated to language wordlists, using part-of-speech tagging to identify nouns, adjectives, and verbs.
B Generations
Prompt Templates FFor each task, we use consistent templates with minimal rewording. Translations were done at the instruction level, with each paragraph translated separately. Translators were informed of the task's purpose and the words to be replaced, ensuring the wording allowed for 'key" substitution without affecting grammar The correct instructions
16

---

Published as a conference paper at COLM 2025
Sinitic
West
1004,000,000
Slavic 151099000
LEUNSOOO
Indo Aiyy yan
Bannu 06,00,000
Romance 1261p0n,000
aL000.000
Southern Reccspoo
Vietic
Figure 8: Language family by the number of speakers as reported by Ethnologue.
Lang.
Translated Title (Original Title)
The Good Soldier Suejk, Vol.3 (Osudy dobrého vojáka Suejkn soétové sdiky)
Jaroslay Hašek
174576
da
Elderfloter and Hops (Hyld og Humlee Fortallinger)
Sophie Breum
1900
76082
de
The Magic Menntain (Der Zauberberg)
Thomas Mann
1924
598161
Little Women
Louisa May Alcott
1868
233668
Don Quinote
Miguel de Cervantes
1605
626105
fa
The Blnnd Owa
Sadegh Hedayat
1936
77451
fi
Sylei
Minna Canth
1893
60803
fr
Les Misérables
Victor Hugo
1862
970526
Chandrakanta
Devaki Nandan Khatri
King
(Milié kinilly)
Zoltán Ambrus
The Late Mattia Pascal
Luigi Pirandello
1904
144402
ja
Kakam =
Natsume Soseki
1914
185905
ko
A Bay's Sorrowu (.2: [) s)oo))
Yi Kwang-su
1917
12279
nl
The Diary of Younng Girl (Het Auhterluus)
Anne Frank
1947
136314
no
Kristin Laeransdatter, Vol. 2: The Wife
Sigrid Undset
1921
251016
pl
Nights and Days, Vol. (Noce duir)
Maria Dabrowska
1934
452847
pt
The Book of Disquiet (Licro Desososegu)
Fernando Pessoa
1982
248245
ru
War Peace (Bodza xmps)
Tolstoy
1112514
sr
Seconds of Elernity (Sekand vetwosti, istočnjakki
Dragutin Ili
Chaka
Thomas Mofolo
1925
119342
The Story Gösta Berling
Selma Lagerlit
1891
255416
My Life Fify Years After (Maisha yangu banda Miska Hamsini)
Shaaban Robert
1958
61490
ta
Ponniyán Selwan, Vol. 1: The First Floods
Kalki Krishnamurthy
1950
912523
uk
After Finisling School (Instylutaa)
Marko Vavehok
1862
41056
vi
Pure Heart (To lam)
Hoàng Ngoc Phách
1925
72267
zh
Arms (NB)
Lu Xun
1922
153415
Table 4: Complete list of books used for needle injections in the retrieval task. Each row contains the language the book was originally published in, the title, author name, published year, and token count. Tokens counts calculated using tiktoken (o200k).
and words were added to make the final prompts. The English versions of the prompts are as follows: (1) the Single-Niah task (Figure 9), (2) the multi-query task (Figure 12), (3) the multi-key task (Figure 10), (4) the none-NIAH task (Figure 13), (5) the multi-value task (Figure 11), and (6) the single NIAH task without the inclusion of none Figure 14 (used for ablation study). The CWE prompt is shown in (Figure 15). We include the English
17

---

Published as a conference paper at COLM 2025
Single NIAH Template
Please read and memorize the text below. ask you about later. <text>
[CONTEXT]
The special magic number for "[WORD 13" is: [NUMBER 1].
[CONTEXT]
</text>
<question>
What special magic numbers associated with "[WORD 1]" are mentioned the provided text? Please all that apply. no such numbers exist, please answer "none". </question>
Please provide your answer in the following format: <answer>List all numbers here</answer>
Figure 9; Prompt template for our "Single NIAH" task with one magic number. The bolded sentence introduces the possibility of no answer existing, which we show significantly hurts model accuracies, even for this Single NIAH task where the answer always exists.
Multi-Key NIAH Template
Please read and memorize the text below. will ask you about it later <text>
[CONTEXT]
The special magic number for "[WORD 1]" is: [NUMBER 1]. [CONTEXT]
The special magic number for "[WORD 2]" is: [NUMBER 2]. [CONTEXT]
The special magic number for "[WORD 3]" is: [NUMBER 3].
[CONTEXT]
The special magic number for [[WORD 4]" is: [NUMBER 4].
</text>
<question>What special magic numbers associated with "[WORD 3]" are mentioned in the provided text? Please list all that apply. If no such numbers exist, please answer "none". </question>
Please provide your answer in the following format: <answer>List all numbers here</answer>>
Figure 10: Prompt template for Multi-Key NIAH Task with multiple magic numbers.
prompts for understanding, but for experiments translated instruction prompts are used in the language of the context (except the cross-lingual setup). We ensured all variables could be substituted in templates without grammatical changes (e.g., singular/plure forms, gender variants, or grammatical cases). All instructions and needles are verified translations by native speakers of each respective language.
18

---

Published as a conference paper at COLM 2025
Multi-Value NIAH Template
Please read and memorize the text below. will ask you about it later <text>
[CONTEXT]
The special magic number for "[WORD 13" is:
[NUMBER 1].
[CONTEXT]
The special magic number for "[WORD 13" is: [NUMBER 2]. [CONTEXT]
The special magic number for "[WORD 13" is: [NUMBER 3].
[CONTEXT]
The special magic number for "[WORD 13" [NUMBER 4].
</text>
<question2What special magic numbers associated with "[WORD 13" mentioned in the provided text? Please list all apply. no such numbers exist, please answer "none". "iiee..</qestiom"
Please provide your answer in the following format: <answer>List numbers here</answer>
Figure 11: Prompt template for Multi-Value NIAH Task with multiple magic numbers.
Multi-Query NIAH Template
Please read and memorize the text below. will ask you about it later <text>
[CONTEXT]
The special magic number for "[WORD 1]" is: [NUMBER 1]. [CONTEXT]
The special magic number for "[WORD 2]" is: [NUMBER 2].
</text>
<question>What special magic numbers associated with "[WORD 1]" and "[WORD 2]" are mentioned in the provided text? Please list all that apply. If no such numbers exist, please answer "none".</que" </questioon
Please provide your answer in the following format: <answer>List all numbers here</answer>
Figure 12: Prompt template for Multi-Query NIAH Task with multiple magic numbers.
19

---

Published as a conference paper at COLM 2025
None NIAH Template
Please read and memorize the text below. will ask you about it later
<text>
[CONTEXT]
The special magic number for "[WORD 1]" is:
[NUMBER 1].
[CONTEXT]
The special magic number for "[WORD 2] is:
[NUMBER 2].
[CONTEXT]
The special magic number for "[WORD 3] is:
[NUMBER 3].
[CONTEXT]
The special magic number for [[WORD 4]" is: [NUMBER 4].
</text>
<question>What special magic numbers associated with "[WORD 5]" are mentioned in the provided text? Please list all that apply. If such numbers exist, please answer "none".</question <<qquestio>>
Please provide your answer in the following format: <answer>List all numbers here</answer>
Figure 13: Prompt template for None NIAH Task with no matching magic numbers.
Single NIAH No 'None' Template
Please read and memorize the text below. will ask you about it later.
<text>
[CONTEXT]
special magic number for "[WORD 1]" [NUMBER 1].
[CONTEXT]
</text>
<question>
What special magic numbers associated with "[WORD 1]" are mentioned provided text? Please list all that apply. If no such numbers exist, please answer "none".
</question>
Please provide your answer in the following format: <answer>List all numbers here</answer>
Figure 14: Prompt template for Single NIAH Task, without the inclusion of None in the prompt.
CWE Template
Below is numbered list of words. In these words, appear often than others. Memorize the ones that appear most often.
<list>:ENUMERATED LIST OF WORDS]</list>
<question> what are the 10 most common words in the list above? </question> Please provide your answer in the following format <answer> List the words here </answer>
Figure 15: Prompt template for Most Common Words Task.
20

---

Published as a conference paper at COLM 2025
Model
Size
Context Length
Huggingface Wolf et (2019) API
Cost
Gemini 1.5 Flush (Gemini Team, 2024)
1M
gemini-1.S-faash
$137.7
u3-mini-high (OpenAI, 2025)
200K
o3-mini-2002--01-3
$1932.5
Deepseek-RI R1 (DeepSeck-AL) AI, 2025)
6858
128k
drepseek-rl
$333.2
3 (Llama Team, 2024)
70B
128K
meta- 1.0.amma--.-70---stttte
1lama3.1 (Llama Team, 2024)
8B
128K
meta-llama/ /ate//MMtt-LLma33----------
Qwen2.5-72B (Qwen Team, 2025)
72B
128K
Qwen Qwen2.5-77B-nnstrret
Qwen2.5-7B (Qwen Team, 2023)
7B
128K
Qwen/ /Qweeee2222-77--mitrrrt
Table 5: Comparison of Open and Closed Language Models used in experiments
B.1 Model Configurations
For o3-mini, we set reasoning efforts to high and configured the maximum output to- ken length to 10k, with temperature fixed at 1.0 as per OpenAl defaults. For Deepseek- R1, we set the maximum output token length to 8k and used a temperature of 0.6. We evaluated all 26 languages Gemini 1.5 Flash and o3-mini, utilizing Batch API for o3-mini which provided 50% cost duction. For Deepseek-RI, we used the Fire- works API but limited evaluation English only due to budget constraints. Please re- fer to Table and Figure 16 for detailed model configurations and associated costs. For model configurations, we run inference on open-weights models using greedy de- coding with temperature set to 0.0, which we also applied to the Gemini-1.5-Flast model. We evaluate Llama and Qwen with vLLM (Kwon et al., 2023), using A100 for smaller models and A100s for larger ones.
Model
Param
Ctx len
Vocab
Gemini 1.5 Flash
1M
256K
03-mini-high
200K
200K
Deepseek-RR R1
671B/37B
128K
129k
Llama 3.3
70B
128K
128K
Llama 3.1
8B
128K
128K
Qwen 2.5
72B
128K
152K
Qwen 2.5
7B
128K
152K
and open- weights (c) models used in the experiments. We report models' parameters count, claimed context length, and vocabulary size.
Figure 16:
Closed-source
C Additional Results
C.1 Retrieval Results
In this section, we detail additional results for the Needle-in----aayttack task.
Average NIAH performance across all languages In Figure 17, we see the performance of each model averaged across all NIAH tasks and all languages, with gemini-1.5-flan outperforming all other models, especially at long context-lengths.
Number of incorrect answers marked as 'none' across all languages We show that the number of wrong answers labeled as 'none' in S-NIAH tasks with long context varies across models and languages, as illustrated in Figure 18 and Figure 19. The 03-mini-high model produces a significantly higher number of 'none' errors compared to other models. Interestingly, Gemini-1.5-flase also generates a notable number of 'none' errors for some high-resource languages. Surprisingly, although the Qwen model specialized in Chinese, it exhibits a large number of 'oone' errors in S-NIAH as well.
C.2 Reasoning Results
In this section, we detail additional results for reasoning models. In Figure 20 we show the average amounts of reasoning tokens for NIAH tasks from 03-mini. We observe that incorrect answers produce drastically higher amounts reasoning tokens than correct answers. In Figure 21, show the number of responses exceeding the set max number of
21

---

Published as a conference paper at COLM 2025
Performance on NIAH Tasks Across Context Length By Model
1.0
0.9
Micro Accuracy
0.6
model,
Ueme-3.1-08
0.5
Lamm--33-706
0wen2.5-728
0wen2.5-78
gemini-1.5-flash
03 mini
20000
40000
60000
80000
100000
120000
Context Length
Figure 17: By model performance across all languages on the NIAH tasks across context lengths. We find that LLama-3.3 has the largest decrease in performance across context lengths.
22

---

Published as a conference paper at COLM 2025
a3-mini
home enors Rema ring erorr
23-
22-
0.5
Language
gemini-1.5-fip
40
Bemaning enor
Bone enore
Average
10-
Language
Figure 18: The types of errors made in the S-NIAH at long-context(64K and 128K) across closed model and languages
output tokens for the English language. We note that both o3-mini-high and deepseek-rl often exceed the token limit, overreasoning for simple task.
C.3 Aggregation Results
We evaluate CWE in both the easy and hard settings over four context lengths: 8k, 32k, 64k, and 128k. As seen in Figure 5, average English accuracy for the CWE-easy task. The CWE-hard setting proves unsolvable with nearly 0% accuracy across all models. Performance of each model by context length can be found in Figure 22. All models evaluated perform much better on the CWE-easy task in the Sk length. The drop in performance amidst increasing context length indicates language models still have trouble with this kind of aggregation task in very long- context settings. The CWE-hard task near impossible for all models. For a task that would be so simple for humans, the near-total incapability of modern LLMs to complete the task in the harder setting highlights that synthetic tasks like CWE and NIAH, that can be made more challenging as model performance improves, necessary for ongoing evaluation of models.
all models is only 31.5%
long-context
Reasoning models "overthink" on simple aggregation tasks: On both CWE tasks, o3- mini-high fails to generate answers within its 10K output token limit for almost every sample across all languages and context sizes." 14 Notably, this overthinking persists even with smaller contexts, with reasoning outputs sometimes exceeding the length of the given context itself! This is not just unique to o3-mini-high: number of Deepseek-Re output token limit before reaching an answer the reasoning tokens reveals strategies unrelated to the task (e.g., behaviors (e.g., unrelated to Alll model configurations reported in subsection B.1.
also observe that significant responses in the CWE-hard task at long contexts exceed the 8k in Figure 21. Qualitative inspection of
23

---

Published as a conference paper at COLM 2025
Qwen2.5-728
20-
22-
Language
Qwen2.5-77
hone
32
10-
Language
Llama-3.3-708
kome emore Remaiiing ning Emor
20-
11
Average
Language
Llama-3.1
Remantis
40-
02-
efror
Language
Figure 19: The types of errors made in the S-NIAH at long-context(64K and 128K) across open-source model and languages
24

---

Published as a conference paper at COLM 2025
reasoning tokens true
reasoning tokens false
Average Reasoning Tokens
5000
4000
3000
2000
1000
32000
64000
128000
Context Length
Figure 20: Average reasoning token length for correct incorrect answers across context lengths in NIAH tasks
Farz Easy
15
Fard
Exoossive resparse CHUTE
12
5000
32000
84000
125000
5000
30000
14000
128000
Longth
(a)
o3-mini-
(b) Deepseek RI
Figure 21: Number of responses exceeding the maximum output tokens in CWE tasks, caused by reasoning overthinking, for 03-mini-high (26 languages) and Deepseek-R1 (English only).
the task. As whole, these results suggest that reasoning models should be better optimized for tasks requiring aggregating information across long contexts.
D One Tokenizer Does Not Rule Them All
One of the biggest challenges of the multilingual setting understanding the effects of the tokenizer on model performance. For example, one model may be able to take in 50,000 tokens in Tamil, but another may only be able to take in 25,000, changing the overall amount of information passed to the model. What would happen if we measured the effects of model using the information density (or amount of content given), instead of the token count? Below, we detail many of the same experiments as conducted earlier, but in the context of each model being given the same amount of information per model instead of the same amount of tokens.
Drastic Differences in Token Lengths: For each language, the token counts of other tokenizers when stabilizing for 128k by tiktoken cl cl100k_base show astounding variance, seen in Figure 24. We see that the same 128k per 11100k_bass results in large differences between other tokenizers, For example, Tamil has only 33,249 tokens when processed using the tiktoken o200k o200k_base, compared to 128,000 with cl 21100k_BBr base.
25

---

Published as a conference paper at COLM 2025
Example of Deepseek-R1's Reasoning for MV-NIAH
<think>
Okay, let's tackle this question. The user wants to know the special magic numbers associated with the word "field" mentioned in the provided text. I need to go through the text carefully and look for any instances where the word "field" is followed by a number.
First, I'll start by recalling that the text is from "Little Women" by Louisa May Alcott. The user provided lengthy excerpt covering several chapters. The task is to find any mention of a "special magic number for 'field" in the text.
I'll need to scan through the text for the exact phrase "The special magic number for 'field' is: [number]."" Since the user provided the entire text, can search for occurrences of this specific phrase.
Looking through the chapters, I remember that in some parts, there are notes or numbers inserted. Let me check each chapter section:
1. Chapter One: No mention of magic number here. The story starts with the sisters talking about Christmas. 2. Chapter Two: Focuses on Christmas morning and the girls' activities. No numbers related to "field." 3. Chapter Three: The Laurence boy. Here, there's a mention of magic number. The text says: "The special magic number for 'field' is: 2140194." So that's one number. 4. Chapter Four: Burdens. No mention here. 5. Chapter Five: Being Neighborly. No numbers here either. 6. Chapter Six: Beth Finds the Palace Beautiful. No magic numbers. 7. Chapter Seven: Amy's 's Valley of Humiliation. No numbers here. 8. Chapter Eight: Jo Meets Apollyon. No mention. 9. Chapter Nine: Meg Goes to Vanity Fair. Here, another instance: "The special magic number for 'field' is: 4946066." 10. Chapter Ten: The P.C. and P.O. No numbers here. 11. Chapter Eleven: Experiments. No mention. 12. Chapter Twelve: Camp Laurence. Another instance: "The special magic number for 'field' is: 4117625."
So, in total, there are three instances where the text explicitly states a special magic number for "field": 21 2140194, 4946066, and 4117625. Each of these is found in different chapters, need to list all three as the answer. </think>
Table 6: Example of Deepseek's 's reasoning for the MV-NIAH task, showing how it systemati- cally analyzes the text by dividing it into chapters and searching for the target pattern.
CVE, EASY Accurecy by Medel and Conbect Length
3.0
CIVE HARD Accurscy by Madal and Cornest Length
contect jergth
2003
0.
61030
0.3
129000
0.6
ACCENCY
0.6
Uame: 1.40
Uemad -11
Dredt
Deenz
persiu Schali
Unme 3.1
Usis 3:8
Oeonk 7/8
Beess 178
penss S-bas
a3 samool
Model Name
Model Neme
(a) CWEaasy
(b) CWE-hard
Figure 22: CWE Accuracy by Context Length
Similar findings hold despite dramatic differences in tokenized lengths: As seen in Figure 25, when controlling for amount of context information, Slavic languages (Russian, Ukrainian, Polish) perform highly, surpassing the performance of Romance languages like Italian, French, and Spanish. English performance is even lower, typically 8th. However, similar model ranking patterns remain: Gemini 1.5 Flash and Qwen 2.5 72B perform the best, and high-resource Latin/ Latin/Cyrillic languages outperform others. is interesting to note that languages such as Korean, Hindi, and Chinese are towards the low end of the performance
26

---

Published as a conference paper at COLM 2025
Remrrce type
Ilihhnowore
Number of Correct words
Number of Correct words
6
High ressree
66.8k
8544k
1.12kk
86.8k
Sl.gak
8.64k
& 128k
Context length
Context length
(a) CWE_easy task
(b) CWE-hard task
Figure 23: Number of correct words in cases where the model generated an answer but was ultimately incorrect at different context length. As context length increases, the model finds less correct words. High resource languages tend to return higher average numbers of correct words, which may be indicative of issues relating to tokenization.
spectrum, even though they often tokenized to shorter lengths when in comparison to English 1ata. 15 As context length increases, still see that the performance gap between high-resource and low-resource languages, consistent with our main results controlling for number of tokens.
19This is due to larger tokemization gains we have achieved for these languages with newer tokenizer (e.g,, cl cl10@k_basee _200k_baaal base.
27

---

Published as a conference paper at COLM 2025
Tokenizer
o-series
gemini
qwen
llama
Cs
99597
95345
126844
97213
da
112143
110238
127769
127416
de
108990
104431
127032
127480
-120k
en 126875
126767
127574
127534
es 112502
107578
126542
127106
fa 63703
59743
102384
64472
fi
102266
105860
127624
127308
fr
111580
111783
126984
127563
-100k
hi 45246
52152
118653
67682
hu
109443
103795
126862
125776
Language
it
116052
107961
127708
127654
ja
94280
66231
78423
84180
ko
83389
92233
92640
82528
nl
103367
108157
128293
127339
-ook
no 113099
111794
127782
127351
pl 111266
101113
119683
126416
pt
107893
108830
127048
127545
80315
77481
96679
87709
Sr
78706
77119
105755
96753
st
106866
115360
127479
126104
-0ok
110111
106941
127676
127044
SW
102751
110859
128756
126767
ta
33249
42124
103990
126674
uk
88636
86819
109761
88948
vi 76336
69835
73964
73678
40k
zh 87903
75722
75097
88357
Mean
95637
93703
114962
110177
Figure 24: For each language, the token counts of other tokenizers when stabilizing for 128k
by cl100.
28

---

Published as a conference paper at COLM 2025
Gemini 1.5
Qwenz.5
og-mini
LLaMA 3-3 70B
3.aMAA3-1 8R
Qwenz..5
T2k
yzk
satk
1256.
3.00
Language
0:9
0.54
DF4
0.50
hu
6:40
0.40
1090
10:4
0.66
0.96
0.71
0:59
01
5.95
0.74
0.00
0.05
0.07
0.71
0.50
5.46
v2-10
0.97
0:57
11-
5:50
0.97
554
699
0.07
0.79
0.79
0.72
0.54
da-
0.93
2.98
534
0.03
231
0.38
0.50
0.45
0.05
0.81
0.70
11-10
0.55
0.55
2:97
555
0.0
0.76
0.46
0.48
0.74
0.75
0.6
zh-
[5:1
0.80
0.90
001
0.33
0.38
0.69
0.88
8.90 0.85
0.3
60-
0.5h
0.81
535
0850
0.75
0.79
0.58
0.2
027
0.94
0.35
0.58
0.5
0.94
0.52
099
0.88
898
5:9
0.58
0.03
0.87
0.04
0.38
0.2
0.11
0.51
0.38
0:42
0.97
0:3
0.9
0.80
0.79
021
0.42
0.75
0.69
0.20
0.68
0.69
0.37
0.32
0.20
0:38
0.37
0.15
0.50
0.38
0000
0.75
0.00
0.1
0.2
0.15
0.50
0.25
010
Figure 25: Heatmap of average accuracy of NIAH tasks by language when controlling for input context length. In these tasks, models are provided with the same input. Input length was measured with tiktoken (cl100k).
29


Reviews:
[{"review_id": "Ci0cMlqHWg", "reviewer": "Reviewer_UQwv", "summary": "This paper describes an extensive experiment about testing LLMs with synthetic long-context tasks, based on the needle-in-a-haystack (NIAH) and the common-word-extraction (CWE) tasks. The authors present five variants of NIAH, including cases in which the needle is not present the text, so no result should be obtained, and they report that this addition makes the task particularly hard for many LLMs that would otherwise solve NIAH easily. They also present two variants of CWE classified as easy and hard.\n\nThey experiment with several open and closed models, and the benchmark is first generated in English and then the instructions and words to use are manually translated into 25 more languages of different families and scripts, which makes the experimental setup quite complete.\n\nThe paper presents interesting analyses of the results of the different NIAH variants, but here resides the major lack of this paper, because the results of CWE are not shown in the main document and only detailed in the appendices. As the CWE experiments are described inside the main section, one would expect at least a table with its main results and not just a brief comment about what happened, especially because it seems to be a much challenging task than NIAH for current LLMs. On the other hand, the paper would be just fine without the CWE experiments, as the different variants of NIAH and their analyses already have a lot of substance, but it's strange that the CWE experiments are described but not analyzed in the same way.\n\nSome other interesting information is also relegated to the appendices, which is understandable given the size of the paper, but this is perhaps a case of a paper that is better suited to a journal because the amount of content that needs to be compacted in 9 pages is too much.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The described benchmark is extensive with different languages covered and in different context sizes. It was originally built for English but manually translated to the other languages.\n\nThe experimental setup is solid covering open and closed LLMs, and the results analysis for the NIAH tasks with follow up experiments for particular behaviors are also very interesting.", "reasons_to_reject": "It is strange that the CWE experiments are drafted but the results are not presented on the main paper. They could be altogether removed, or otherwise at least the basic results incorporated on the main document.\n\nFigure 1 is strange, presenting results of slightly different experiments at the same time. At least both figures should be given the same vertical axis space so they can be compared side by side. Also it seems that for shorter context, the missing needle scenario is actually solved better by the models, which contradicts the analysis.", "questions_to_authors": "What is the actual text used as long context in each language? Is it also synthetic? Is it text originally in that language or is it a translation of some general long text? Are they Wikipedia articles?\n\nThe paper describes two separate subtasks \"Single NIAH\" and \"None NIAH\", but also it says that Single NIAH could have instances where the needle is not there. Can you describe in which way these are two different tasks or are they just the same in different scenarios?\n\nLine 89 says each annotator was paid 25 USD per language to translate (instructions + nouns), and then the total compensation was 492 USD. How do we arrive at this final number?\n\nIn line 231 the authors indicate their suspicion that some of the LLMs might already be finetuned with the NIAH task in mind, and using actual NIAH data (at least obtained from other works). Would this render the approach obsolete? Would it impact on other languages is it's only done in English?\n\nSmall correction on line 107: \"make sure to each language's\" -> \"make sure to follow each language's\"", "extra_scores": {}}, {"review_id": "7FSce8koYY", "reviewer": "Reviewer_mDZF", "summary": "The paper aims to extend a benchmark for multilingual long-context models (RULER) focusing on needle-in-the-haystack (NIAH), introducing a novel category where the needle is not present in the haystack, and common word extraction tasks towards instruction fine-tuned model evaluation. The curation process involves a two-step process with English generations that are then translated into 25 other languages with native speakers. Extensive experiments with multiple LLMs demonstrate the performance gap between low- and high-resource languages. The process and benchmark will prove useful to multilingual research benchmarking at large.", "questions": "", "limitations": "", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The benchmark covers 26 languages in total, and allows for cross-lingual performance evaluation. Moreover, native speakers translate the English-based benchmark, introducing cultural relevance to the benchmark.\n- The benchmark covers multiple types of possible NIAH tasks including ones where the answer does not exist, as well as different context lengths, making for a robust benchmark.\n- The takeaways are well studied and interesting, revealing important performance gaps.\n- It is interesting to note that the number of tokens vary across tokenizers. This study is reported in the Appendix, and provides interesting insights for multilingual settings.", "reasons_to_reject": "- An obvious limitation is the coverage of low-resource languages, especially those which are regional and culturally rich. However, finding native speakers for these language could be difficult.\n- Further details about native speakers and how many samples were created by each (along with details of whether multiple annotators were involved in the creation of a single data point) would strengthen the data curation section.\n- While the paper conducts relevant analysis on performance observations, a great addition would be evaluating domain-specific or noisy data, which might provide deeper insights into discrepancies for English, Polish and Chinese where observations are unexpected", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTFF4xuG6u", "reviewer": "Reviewer_p1ae", "summary": "In this paper, the authors propose a new needle in a haystack (NIAH) benchmark for evaluating long-context LMs called OneRuler. Their benchmark extends the original Ruler Benchmark and offers the following advantages: 1) a more comprehensive set of challenging tasks (see Figure 2) that highlight new challenges and shortcomings in models; 2) expert translations from English into 25 languages that allows them to systematically study performance across languages; 3) a focus on instruction-tuned models as opposed to base models (I think this last point is insufficiently motivated, I would like to see the authors motivate this more). \n\nWith this new dataset, they report many intriguing results. I note the following: A variant of NIAH that allows a model to predict no needle greatly reduces performance of the original single NIAH (Figure 1b). “Low resource” languages are strongly outperformed by “High resource” languages in aggregate (where the distinction between “high” and “low” is determined by Wikipedia rank). English is not the best performing model in aggregate and is outperformed by several slavic and Romance languages: Polish, Russian, French, Italian and Spanish (figure 4b); Chinese, another high-resource language, is ranked 4th from the bottom. They also systematically investigate the effect of mixing languages (Figure 5a) and find strange behavior involving reasoning models (Figure 5b). \n\nIn summary, I find this to be a really exciting study that addresses clear short-coming in current testing methods for long-context language models, notably the lack of multilinguality and the limited complexity of existing tasks. I also found the paper very easy to read.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A new and more comprehensive set of multilingual challenge datasets for testing long-context language models, ones that I am confident with have broad impact within this area of NLP and beyond (those engaged in general pre-training and post-training research). \n\n- A new set of intriguing results about the gap between high and low resources languages on these tasks that open up many new questions and possible directions of future research.", "reasons_to_reject": "- Their most controversial design decision relates to how they make the total token count uniform across all languages to account for differences in tokenization, as discussed in lines 90-98. If I understand correctly, this has the effect of truncating the total amount of information content that each language receives (e.g., it could be that an entire book in one language might be a single chapter in another language simply due to the tokenizer being more verbose in the latter model. In such a case, I would think that this would affect the relative complexity of each task potentially creating problems for having a fair comparison across languages). \n\nMy own intuition is that the first option they consider to deal with these mismatching tokenization issues, i.e., ensuring that the input text is of the same length, is more natural. Of course, the authors do report results for this setting too in Appendix D. However, I would like to see this potential issue of mismatching information content addressed.", "questions_to_authors": "- Did you do any systematic comparison between Figure 25 and Figure 3? E.g., are these results highly correlated across these two settings? Would it make sense to take an aggregate of these results? (Such questions relate to my comments above)\n\n- Is there an analogue to a bits-per-byte style metric (i.e., what’s  commonly used for normalizing tokenization mismatch in perplexity analysis) that one could employ here to deal with tokenization issues? \n\n- Do you have a table or non-inline version of the CWE results discussed starting on line 209? I am confused about where to look for these results. \n\n- Like above, I was confused about the results reported on lines 157 (“While most models perform near perfectly on vanilla NIAH for English”). Which plots or tables are showing this? It’s not clear.", "extra_scores": {}}, {"review_id": "7q3Z3tLhMu", "reviewer": "Reviewer_tfQ4", "summary": "This paper describes the benchmark \"one ruler\", containing two tasks for long-context evaluation, Needle in a haystack (NIAH) and common word extraction (CWE), for 26 languages. The NIHA task is novel in that it also contains the possibility of the needle not occurring in the context, which makes the task considerably harder, even when the needle is there. The paper describes the benchmark and experiments on it with five different LLMs, leading to some insights.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper describes a useful multilingual resource that allows interesting cross-lingual analysis.\n\nThe experiments are interesting and hint at many interesting aspects of these models.", "reasons_to_reject": "Most of the experiments just scratch the surface. I miss a clear analysis of the performance difference between the different subtasks, for instance. \n\nThe CWE analysis and description are minimal. I think the paper would have been stronger without including that part, and focusing only on NIHA, which is the most in-depth and interesting part. \n\nMost novels used for context are very old, from 1605 to the early 20th century. This probably impacts the results, since the language in many cases is different from the modern language the models are trained on, and the language in which the needles and instructions are formulated. This is not even mentioned in the paper. I understand that the reason for this is probably copyright, but it still needs to be discussed, especially since the novels differ in age and the amount of language change between the used languages likely differ as well. It is, for instance, worth noting that the top-performing language, Polish, has one of the most modern novels (1934). This issue may thus impact one of your main findings.", "questions_to_authors": "I'm not an expert on NIHA, but it seems slightly weird to me that all needles are formulated in the same way (\"the special magic number ...\"). Do the models not learn from this? How does this impact the overall task? Is this standard? \n\nHow do you decide where to insert the needles? Completely randomly, by some distribution, or some other model?\n\nFor the cross-lingual experiment (Figure 5A), how much do you think this is affected by script? Would you get the same results if you swapped Korean with a low-resource language with Latin script? \n\nHow common is it that models do not produce an answer (footnote 9)? Do you use the same subset of examples for all models, i.e. if a model does not produce an answer for an example, that example is removed for all models, or does this make the models being evaluated on different subsets? If the latter, what impact does this have on your analysis? Also, how is this handled cross-lingually? Does this mean that the subsets evaluated differ across languages? More elaboration on this issue is needed than a footnote.\n\nMinor comments:\nPlease do not add analysis to figure and table captions. Keep the analysis in the text, and let the captions describe the figure.", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper, the authors propose a new needle in a haystack (NIAH) benchmark for evaluating long-context LMs called OneRuler. Their benchmark extends the original Ruler Benchmark and offers the following advantages: 1) a more comprehensive set of challenging tasks that highlight new challenges and shortcomings in models; 2) expert translations from English into 25 languages that allow them to systematically study performance across languages.

With this new dataset, they report intriguing results including, A variant of NIAH that allows a model to predict no needle greatly reduces performance of the original single NIAH, “Low resource” languages are strongly outperformed by “High resource” languages in aggregate, and English is not the best performing model in aggregate and is outperformed by several slavic. They also systematically investigate the effect of mixing languages and find strange behavior involving reasoning models. 

Reviewers raised many great questions that need to be clarified in the final version. This includes clarifying the NIAH subtask definitions, improving Figure 1, moving CWE details to the main paper, annotation details, annotator compensation, the influence of book age and language style, tokenization, and various other small clarifications.

Reasons To Accept:
- A new and more comprehensive set of multilingual challenge datasets for testing long-context language models, ones that can have broad impact within this area of NLP and beyond (those engaged in general pre-training and post-training research).

- A new set of intriguing results about the gap between high and low resources languages on these tasks that open up many new questions and possible directions of future research.

Reasons To Reject:
- Nothing that can't be addressed in the final version
Decision: Accept