RULER: What’s the Real Context Size of Your Long-Context Language Models?
Authors: Cheng-Ping Hsieh
Simeng Sun
Samuel Kriman
Shantanu Acharya
Dima Rekesh
Fei Jia
Boris Ginsburg
Venue: COLM (2024)
Keywords: Long-context, Evaluation, Benchmark, Synthetic
TL;DR: We propose a novel synthetic benchmark RULER to evaluate long-context language models

Abstract: The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the “needle”) from long distractor texts (the “haystack”), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create a new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test to encompass variations with diverse types and quantities of needles. Moreover, RULER introduces new task categories multi-hop tracing and aggregation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models exhibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.

Published as a conference paper at COLM 2024
RULER: What's the Real Context Size of Your Long-Context Language Models?
Cheng- Ping Hsieh', Simeng Sun", Samuel Kriman, Shantanu Acharya Dima Rekesh, Fei Jia, Yang Zhang, Boris Ginsburg
NVIDIA
{chsieh, siimengs)@evidia.ccmm
Abstract
The wvedle-in-a-haystack (NIAH) test, which examines the ability to re- trieve piece of information (the "needle") from long distractor texts (the "haystack"), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval- test is indicative of only a superficial form of long-context understanding. To provide a more comprehensive evaluation of long-context LMs, we create new synthetic benchmark RULER with flexible configurations for customized sequence length and task complexity. RULER expands upon the vanilla NIAH test encompass variations with diverse types and quantities of needles. More- over, RULER introduces new task categories multi-hop tracing and aggre- gation to test behaviors beyond searching from context. We evaluate 17 long-context LMs with 13 representative tasks in RULER. Despite achieving nearly perfect accuracy in the vanilla NIAH test, almost all models ex- hibit large performance drops as the context length increases. While these models all claim context sizes of 32K tokens or greater, only half of them can maintain satisfactory performance at the length of 32K. Our analysis of Yi-34B, which supports context length of 200K, reveals large room for improvement as we increase input length and task complexity. We open source RULER to spur comprehensive evaluation of long-context LMs.
1 Introduction
Recent ad advancements in Al system engineering (Dao et al. 2022 Jacobs et al. 2023 Fu
et al. 2024) and language model designs (Chen et 20223 Xiong et al. 20233) have enabled
efficient scaling up of context length for language models (Liu al.) 2024a Young et al.,
2024). Previous works (A121 2024 X.AT 2024 Reid et al. [204 Anthropic 2024) commonly
adopt synthetic tasks, such as passkey retrieval (Mohtashami Jaggi 2023) and needle-in-a-
haystack (Kamradi) (2033 to evaluate long-context LMs. However, these evaluations are used inconsistently across works and reveal merely the retrieval capability, failing gauge
other forms of long-context understanding.
In this work, we propose RULER, a new benchmark evaluate long-contexx modeling capa- bilities for language models. RULER contains four task categories to test behaviors (Ribeiro) et 2020) beyond simple retrieval from context:
Retrieval: we extend the seedle-i--a-hasstaan (Kamradt 2023 NIAH) test to evaluate retrieval capability with diverse types and quantities of needles.
2. Multi-hop Tracing:
we propose variable tracking, minimal proxy task for coreference chain resolution to check the behavior of tracing entities with multi-hop connections.
3. Aggregation: we propose common /ommon//rrquent words extraction, proxy tasks for summariza- tion to test the ability to aggregate relevant information that spans long-range context.
Authors contributed equally.

---

Published as a conference paper at COLM 2024
Benchmark & Task
Avg Len
Type
Diverse Tasks
Min. Parametric Knowledge
Controllable
Context
ZeraSCROLLS
~10k
realistic
L-Eval
~8k
realistic
BAMBOO
~16k
realistic
/
X
LongBench
~-8k
hybrid
/
LooGLE
~20k
hybrid
/
InfiniteBench
2000
hybrid
/
Needle-i--a-haystack (NIAH)
any
synthetic synthetic
x
/
Passkey/ Line KV Retrieval
any
x
RULER (Ours)
any synthetic
V
Table 1: Comparison between existing long-context benchmarks and RULER. "Realistic" type refers to human-annotated while "synthetic" type refers to auto-generated. RULER includes diverse task domains beyond retrieval, reduces reliance on parametric knowledge with synthetic input, and offers flexibility to control the contexts for different sequence lengths and task complexities. In RULER, contexts can be adjusted by changing the volume or placement of relevant and distracted information.
4. Question Answering: we add distracting information to the input of existing short- context QA datasets to evaluate question answering capability at various context sizes.
Compared to existing realistic benchmarks (Table 1, RULER consists solely of synthetic tasks, which offer the flexibility to control sequence length and task complexity. The synthetic input in RULER reduces reliance parametric knowledge, which interferes with the utilization of long-context input in realistic tasks (Shaham et al.
2023
Bai et al. 2023).
Using RULER, we benchmark Gemini-1.5 (Reid et al. 2024 GPT-4 (OpenAl: Josh Achiam
al. 2023), and 15 open-source models with context length ranging from 4k to 128k. Despite achieving nearly perfect performance on the vanilla NIAH test, almost all models exhibit large degradation on more complex tasks in RULER as sequence length increases. While all models claim context size of 32k tokens or greater, our results indicate that only half of them can effectively handle sequence length of 32K by exceeding qualitative threshold. Moreover, almost all models fall below the threshold before reaching the claimed context lengths. To obtain fine-grained model comparisons, we aggregate performance from 4k to 128k with two weighted average scores where the weights simulate the length distribution of real-world use cases. The top two models Gemini-1.5 and GPT-4, consistently outperform other models regardless of the chosen weighting scheme.
We further analyze Yi-34B, which claims context length of 200K and achieves reasonably good performance on RULER among open-source models. Our results demonstrate large increase input length and task complexity. At large degradation in Yi's performance incomplete answers and fails to precisely locate the context sizes, Yi-34B often returns relevant information. Furthermore, observe two behaviors emerging with the scaling of context size across multiple models: the increased reliance on parametric knowledge and the increased tendency to copy from context for non-retrieval tasks. Our additional ablations demonstrate that training on longer sequences does not always lead to better performance on RULER, and that larger model sizes positively correlate with better long- context capabilities. Finally, we show that non- con-Transformer architectures, such as RWKV and Mamba, still lag behind Transformer by large margins on RULER.
Our contributions are as follows:
We propose new benchmark RULER for evaluating long-context language models via synthetic tasks with flexible configurations.
We introduce new task categories, specifically multi-hop tracing and aggregation, to test behaviors other than retrieval from long context.
We evaluate 17 long-context LMs using RULER and perform analysis across models and task complexities.
We open source RULER to spur future research in long-context language models
https:///ithhb. http::///ttttttccccc/cshshhhhant
2

---

Published as a conference paper at COLM 2024
2 Related Work
Long-context Language Models. Numerous long-context language models have been in-
troduced lately owing to the progress in engineering, architectural, and algorithmic designs.
Flash attention (Dao et al. 2022 Dao 2023) and Ring attention (Liu et al. 2023) significantly
reduce the memory footprint required for processing long context. Various sparse attention
mechanisms (Child et al. 2019; Jaszczur et al. 2021) such as shifted sparse attention (Chen
et al.
2024) dilated attention (Ding et al. 2023), and attention sinks (Han et al. 2023 Xiao
et al.
2024b) were employed to enable efficiett context scaling. Novel position embedding
methods were proposed to improve length extrapolation
Transformers (Vaswani al.
2017), including ALiBi (Press et al.) 2222)), xPOS (Sun et al. (2023)), and RoPE (Su et al. 2023
Variants (Chen et al. | Xiong et al. 2023 Peng et al. 2024 Liu et al. 2024b
et al.
2024 Zhu et al. 2024). Another line of research focuses
reducing context size. This can
be achieved by caching previous context using recurrence mechanism (Zhang et al. 2024a
Bulatov et al. 2023 Martins et al. [2022] Wu et al. 2022), retrieving relevant information
from context (Xu et al.2,
Mohtashami & Jaggi 2023: Wang et al.) 2024 Tworkowski
et al. 2024 Xiao et a or preserving the salient information via compression (Jiang
et al. 2023). Finally, novel architectures (Gu et al. 2022 Fu et al., 2023a Poli et al. 2023
Fu al.] 2023b Sun et al., 2023a Beck et al.) 2024 Sun et al. 2024) such as Mamba (Gu
& Dao 2023) and RWKV
Peng et al. 2023) have also been proposed to efficiently handle
long-context input.
Long-context Benchmarks and Tasks. Our work is closely related to other works on bench- marking long-context language models. ZeroSCROLLS (Shaham et al. 2023) covers ten real- istic natural language tasks, such long-document QA and (query-based summarization. L-Eval (An et al. (2024) also uses realistic data, which was filtered manually to ensure quality, LongBench (Bai et |l.] 2023) contains tasks in bilingual setting. InfiniteBench (Zhang et al., 2024b) includes tasks with length greater than 100K tokens. LTM (Castillo et al. 2024 targets the evaluation of long-term conversations. To isolate the effect of parametric knowledge, previous works (Dong et al. 2023: Li et al. 2023b) also propose to use documents posted online later than a certain cutoff date, or leverage extremely low-resource materials (Tanzer eet al. 2024). Compared to realistic benchmarks, synthetic tasks are more flexible to control the setup (e.g., sequence length and task complexity) and less affected by parametric knowl- edge. Recent works have primarily focused on retrieval-based synthetic tasks (Kamradt)
|023 Mohtashami & Jaggi 2023 Li al.) 2023a; Liu et al. | [0224d Lee et al.)
| 2024), with
few investigate other aspects, including tact reasoning (Kuratov et al. 2024 Karpinska et al.
2024),
long-range discourse modeling (Sun et al. 2022), question answering (Levy et al.
2024
Yuan et al. 2024), many-shot in-context learning (Agarwal et al. 2024 Bertsch et al.
2024
Xu et al. 2024b), and code understanding (Liu et al. 2024c).
3 The RULER Benchmark
RULER comprises tasks across four categories: refrieval, multi-hop tracing, aggregation, and question answering. Evaluation examples in RULER are automatically generated based on input configurations (see Table that define the length and complexity of each input. Within constrained domain as in RULER, the task complexity can be thought of as a function of the number of target output tokens and the signal-to-noie ratio in the context. We point readers (Goldman et al. 2024) for more comprehensive discussion on evaluation task design for long-context language models.
3.1 Retrieval: Needle-in-a-haystacck (NIAH)
Recent works (Reid et al. 2024 Anthropic 2023) commonly employ the needle-in-a-
haystack (Kamradt 2023
NIAH) test to evaluate long-context modeling capability. The NIAH test is reminiscent of the extensively studied (Hopfield) 1982 Graves et al.) 2014 Olsson et al., 2022 Arora et al. 2024) associative recall tasks, in which relevant information needs to be retrieved from context given a sufficient query. In RULER, we include multiple retrieval -based tasks, extending the vanilla NIAH test to evaluate models based three
3

---

Published as a conference paper at COLM 2024
Task
Configuration
Example
Single
type.key word
(essays)
NIAH
type_value number
One of the special magic numbers for long-context is: 12345.
(S-NIAH)
sype_haystac essay
What the special magic number for long-context mentioned the provided text?
size_ pre_haystack a context length
Answer: 12345
Multi-keys
num. num_keys 2
(essays)
NIAH
type key word
special magic numbers for long-context 12345. One of the special magic numbers for large-model is: 54321.
(MK-NIAH)
type.value number
type.hayslack essay size size_ haystaas & context length
What is the special magic number for long-context mentioned in the provided text?
Answer: 12345
Multi-values NIAH
num.values type key word type.value number
(essays)
One of the special magic numbers for long-context is: 12345. One special magic numbers for long-context 54321.
(MV-NIAH)
type.hayslack = essay size haystaa context length
What all special magic numbers for long-context mentioned in the provided text?
Answer: 12345 54321
Multi-queries
num queries = type key word type.value number type haystack essay size haystack & context length
(essays)
NIAH
special magic numbers for long-context 12345.
(MQ-NIAH)
One of the special magic numbers for large-model is: 54321.
What are all the special magic numbers for long-context and large-model mentioned in the provided text?
Answer: 12345 54321
Variable Tracking (VT)
chains
(noises)
num num.hops =
VAR X1 = 12345 VAR Y1 54321
noises context length
VAR X2 ... VAR
VAR X3 = X2 VAR Y3 Y2
Find all variables that assigned the value 12345.
Answer: X1 X2X2XX
Common Words Extraction
freg.ew frequee num ew = 10
ana bib cee aaa
gag ii
What are the 10 most common words in the above list?
(CWE)
num, ucw context length
Answer: aaa ece
Frequent Words
x =
ada bbb coc aaa ddd eee cee fif BEB aaa hih aaa ccc lii iii coded text?
Extraction
num_word context length
are most froquently appeared words Answer, ccc ini
(PWE)
Question Answering (QA)
dataset = SQuAD
Document
and
num document context length
Document 2:
bbb
Document 000 co.
Question: question
Answer:
Table 2:
Task examples with flexible configurations in RULER. We use different colors to highlight queries, keys, values, and distractors in our examples.
criteria. Concretely, the retrieval capability should be (1) agnostic to the type of the "needle" and the "haystack", (2) strong enough to disregard hard distractors, and (3) of high recall when multiple items need to be retrieved. Based tasks. The "needle" in each of these tasks is a key-value pair inserted into the "haystack" (long distractor texts). The query is located at the end of the sequence and serves as cue for matching the keys in the context and subsequently retrieving the associated values.
these criteria, develop four NIAH
Single NIAH (S-NIAH): This is the vanilla NIAH test where a single " "needle needs to be retrieved from the "haystack". The query/key /ky/valuu value can take the form of words, numbers digits), or UUIDs (32 digits). The "haystack" can be repeated noise sentence or Paul Graham essays [amradd (023).
Multi-keys NIAH (MK-NIAH): Multiple "needles" are inserted into the "haystack", and only one of them needs to be retrieved. The additional "needles" are hard distractors. The most challenging setting is version where the "haystack" is filled with distractor needles.
2Similar to Liu et al. 2024a), use "the special magic number for XXX is: YYY" the needle due to its extendability instead of the sentence about San Francisco proposed by Kamradt 2023).
Following Mohtashami & Jaggi (2023), we use "The grass is green. The sky is blue. The swn is yellow.
Here we go. There and back agam." as sentences.
4

---

Published as a conference paper at COLM 2024
CWE
log (Word Freq.)
FWE
Word Freq.
30
Common words
8
20
ommon
o=1
2-2.0
.=2
2
a-3.0
Word types
log (Word rank)
Figure 1: In aggregation tasks, sample words from vocabulary following the two distri- butions above. The common words extraction (CWE) samples from uniform distributions. In the frequent words extraction (FWE),, the frequency of each word is determined by its rank in the vocabulary and the parameter a of Zeta distribution.
Multi-values NIAH (MV-NIAH): Multiple "needles" sharing the same key are inserted into the "haystack". All values associated with the same key need to be retrieved.
Multi-queries NIAH (MQ-NIAH): Multiple "needles" are inserted into the "haystack". All "needles" with distinct keys need to be retrieved. This is the same multi-query associative recall task setup used by Arora et al. (2024). Together with MV-NIAH, these two tasks evaluate the retrieval capability without missing any critical information.
3.2 Multi-hop Tracing: Variable Tracking (VT)
Effective discourse comprehension (van Dijk & Kintsch 1983 is contingent upon successful recognition of newly mentioned entities and establishing the chain of references co-referring to the same entity [Karttunen] 1969) throughout the long context. We develop a new task variable tracking to emulate a minimal coreference chain resolution (Ng) 2010) task. This task checks the behavior of tracking relevant co-occurrence patterns and drawing skipped connections within long input. Specifically, variable X1 is initialized with value V, followed by linear chain of variable name binding statements (e.g, X2 = X1, X3 X2,,.., which are inserted at various positions of the input. The objective is return all variable names pointing the same value V. The task complexity be increased by adding more hops (i.e., the times of name binding) or more chains, similar to adding hard distractors in MK-NIAH.
3.3 Aggregation: Common Words (CWE) and Frequent Words Extraction (FWE)
In RULER, introduce new category proxy for summarization tasks where relevant information constitutes much larger portion the context, and the target output depends accurate aggregation of the relevant input. Concretely, by sampling words from a pre-defined (synthetic) word list. In the common word extraction task (CWE), words are sampled from discrete uniform distributions, with the number of common words fixed while the number of uncommon words increases with the sequence length. In the frequent words extraction task (FWE), words are sampled from Zeta distribu- tion Figure 1 shows an illustration of word frequency in the constructed input. A model needs to return the top-K frequent words in the context. In CWE, K equals to the number of common words. In FWE, we set K 3, as increasing leads to poor performance even at small context sizes for most models. The task complexity can be adjusted by varying the number of common words or the parameter of Zeta distribution.
construct an input sequence
3.4
Question Answering (QA)
The majority of existing QA datasets (Rajpurkar et al. 2018 Yang et al., 2018 Trivedi et al. 2022) designed to answer questions based short passages. These datasets
WWe draw inspiration from Zipf's Law (Kngslley Zipf 1932). Let N be the total number of words, which is determined by the context size, the trequency of the k-th ranked word (the k-th most frequently
appeared word) is KN where [(x) is the Zeta function. We set the top-ranked word to noise. [(K]
5

---

Published as a conference paper at COLM 2024
can be extended to simulate long-context input by adding distracting information. In this task category, we insert the golden paragraphs (i.e,, the paragraphs that contain answers) into paragraphs randomly sampled from the same dataset. This category is real-world adaptation (lvgi et al. 2023) of NIAH, where the question serves as the query, the golden paragraphs are the "needles", and the distracting paragraphs form the "haystack".
4 Experiments & Results
Models & Inference setup We select 17 long-context LLMs, including 15 open-source models and two closed-source model (Gemini-I.5-Pro and GPT-4), ) covering diverse model sizes (7B to 8x22B with MoE architecture) and claimed context lengths (32K to 1M). Com- plete information about these models is included in Appendix We evaluate all models using vLLM (Kwon et al. 2023), an LLM serving system with efficient KV cache memory management. For all models, we run the inference in BFloat16 on 8 NVIDIA A100 GPUs with greedy decoding.
Task configurations We test all models on 13 tasks ranging diverse complexities from the four categories of RULER. The test configurations have been selected (shown in Appendix B based task correlational study described in Appendix C
We select these tasks as most models perform decently at short context size of 4K tokens. Our main goal is to see whether models can maintain such good performance with the scaling of context length. For each task, we evaluate each model with 500 examples generated for each length from the series (4K, 8K, 16K, 32K, 64K, 128K), while complying with each model's necessary chat template To prevent the model from refusing to answer a query or generating explanations, we append the task input with an answer prefix and check the presence of the target output with recall-based accuracy.
Effective Context Size We notice large performance degradation in all models as we increase input length in RULER. To determine the maximum context size model can effectively handle, we grade each model with fixed threshold, passing which indicates satisfactory performance at the length of evaluation. We use the performance of Llama2-7b model at the 4K context length as the threshold. We report in Table 3 the maximum length exceeding the threshold the "effective length" along with the "claimed length".
Model Ranking Criteria While the threshold-based grading reveals the discrepancy between claimed and effective length, lacks details for fine-grained model comparisons. As such, we use weighted average score to aggregate model performance across various context sizes. We rank models under two weighting schemes: w (inc) and wAg. (dec) where the weight linearly increases and decreases with sequence length respectively. Ideally, the weight for each length should be determined by the length distribution of model usage, here we choose the two schemes to simulate the scenarios where longer sequences (inc) or shorter sequences (dec) dominate the distribution.
Main Results We include the results of 17 long-context LMs in comparison with the
Llama2-7B baseline in Table The performance at certain length is the average of all 13 tasks in RULER. The closed-soure model Gemini-1.5-Pro outperforms the rest of the models by a large margin, with the effective length greater than the maximum length have tested on. Pressure testing this model with harder version RULER can be interesting to follow up in the future. For the rest of the models, while they achieve nearly perfect performance on the passkey retrieval and the vanilla NIAH task (shown in Appendix E), all of them exhibit large degradation in RULER sequence length increases and they fail to maintain performance above the Llama2-7B baseline at their claimed length. The top-ranked open-source models (Llama3.1, Qwen2 and Nommand-R-pluss, share common configurations, such as having larger model sizes and using larger base frequencies in
5See Appendix D for model and tasks templates details. PPrfformance of base models and breakdown by task categories can be found in Appendix

---

Published as a conference paper at COLM 2024
Models
Claimed Length
Effective Length
4K
8K
16K
32K
64K
128K
Avg-
wAvg. (inc)
wAvg.
(dec)
Llama2 (7B)
4K
85.6
Ge5Pno-1.5-Pro
1M
128K
96.7
95.8
96.0
95.9
95.9
95.8
95.5
96.1(sst)
GPT-1
128K
61K
96.6
96.3
95.2
93.2
87.0
81.2
91.6
94.122d) 89.0pzd)
Ilama3.1 (20B)
128K
64K
96.5
95.8
95.4
94.8
88.4
66.6
89.6
93.7 93.7 85.5
Qwen2 (72B)
128K
32K
96.9
96.1
94,9
94.1
79.8
53.7
85,9
79 99.6)m)
(2.3
Bommmand-Rpplun (104B)
128K
32K
95.6
95.2
94.2
92.0
84.3
63.1
87.4
82.7 (7h)
(2.1(st))
GLM4 (9B)
1M
64K
94.7
92.8
92.1
89.9
86.7
83.1
89.9
88, (rd)
91.7 (6ch)
Llama3.1 (SB)
128K
32K
95.5
93.8
91.6
87.4
84.7
77.0
88.3
85.4
91.3
GradientAI/Llama3 (70B)
1M
16K
95.1
94.1
90.8
85.4
80.9
72.1
86.5
82.6
(0.3
Mixtral-6x22B (39B/141B)
64K
32K
95.6
94.9
93.4
90.9
84.7
31.7
81.9
73.5
90.3
Yi (34B)
200K
32K
93.3
92.2
91.3
87.5
77.3
87.5
(4t8))
(0.1(10m)
Phi3-medium (14B)
128K
32K
93.3
93.2
91.1
86.8
78.6
46.1
81.5
74.8(10th)
(
Mistral-v0.2 (7B)
32K
16K
93.6
91.2
87.2
75.4
49.0
13.8
68.4
55.6(13th)
81.2
LWM (7B)
1M
<4K
82.3
78.4
73.7
69.1
68.1
65.0
72.8
69. 69.91121t)
75.7 75.7
DBRX (36B/133 /132B)
32K
8K
95.1
93.8
83.6
63.1
2.4
0.0
56.3
38.0
74.7
Together (7B)
32K
4K
81.1
69.4
50.3
33.8
61.7
LongChat (7B)
32K
<4K
84.7
79.9
70,8
59.3
49.1
65.2 (3.1
(16n))
LongAlpaca (13B)
32K
<4K
60.6
57.0
56.6
43.6
0.0
0.0
36,3
24.7(171h)
47.9(170)
Table 3: Long Context Performance (%) of selected models evaluated at length from 4K to 128K. Each score is computed by averaging accuracy of 13 tasks in RULER. The performance exceeding the Llama2-7B performance at 4K (85.6%) is underlined. The effective context length is the maximum length passing this threshold. Weighted average score (w Awg.) aggregates performance across all context sizes, with the weights linearly increasing (inc) or decreasing (dec) to simulate length distribution of real-world usage. We put the rank of each model in the subscript. More details about the selected models are in Appendix A
RoPE (Xiong et al. 2023). Large training context window not always necessary for good long context pertormance- top-ranked open-source models contain both brute-force context scaling (Llama3.1 trained on 128K context length) and inference-time length extrapolation (Qwen2 trained on 32K context length). The less performant models also include those trained on much larger context size (e.g., LWM and GradientAl/Llamas both on 1M context length). Although LWM achieves a higher rank than Mistral-v0.2 when longer sequences receive larger weight (wAvg. inc) and shows less degradation as the context size increases, performs worse than Llama2-7B even at 4K. This result suggests a trade-off in evaluation between absolute performance on short sequences and the relative degradation with the scaling of context size. We provide more analysis on the model size and maximum training length in section 6
5 Task Error Analysis
We evaluate Yi-34B-200K with increased input lengths (up to 256K) on more complex tasks to understand the effect of task configurations and failure modes on RULER..
Non-robustness to "needle" types. Figure 2 (left) shows that while Yi achieves almost perfect performance when using needle of nomber pair in the standard passkey retrieval and vanilla NIAH, performance degrades when the needle takes other forms. We observe the largest degradation in the task of retrieving UUIDs, for which Yi sometimes fail to return the complete 32 digits given long (> 128K) input context.
Failure to ignore distractors. Figure 2 (middle-left) shows that increasing the number of distracting needles steadily lowers pertormance, with Yi dropping by 400 points at 256K in the extreme version, where the context is full of irrelevant needles (#K=FULL). Error analysis reveals that Yi fails effectively ignore the hard distractors given long input context, thus incorrectly retrieves values associated with the distractor keys. In the extreme version, Yi often returns values from the vicinity of the target, suggesting coarse match of the range but the lack of precision to locate the key when the target is in-distribution of the noises.
Return incomplete information. Consistent with previous works (Liu et al. 2024a Reid et al. 2024), we notice significant degradation in performance when the model needs to
7

---

Published as a conference paper at COLM 2024
Y1348 Single NIAH
Y134B MK-NIAM
Y:348 MV-NIAH
Yi348 -O-NIAH
1.00
1.00
100
Accurney
Accurncy
Passkey retieval
4k=1
Vanilla NIH
cKod
gV-1
#0=1
Swit V-U
4V-2
#0-2
Switl V-WV
60-4
Ked, V-u
C-U
2318
2308
2300
Sequence Length
Sequence Length
Sequence Length
Sequence Length
Figure 2: Performance of Yi-34B in the needle-in-a-haystacc (NIAH) tasks. By default, we use word-number as the key-value pair and Paul Graham essays as the haystack. Yi is not robust to the change of needle types and degrades with the increasing amount of distractors. (W: words; N: numbers; U: UUIDs; Full: entire haystack).
Yi348
100
1.00
100
100
90
90
90
75
Accuracy
Accuracy
70
70
25
SQUAD
hap=2
hop-10
hop-2
Hotpot
200k
2009 2.00
200k
2009 2550
Sequence Length
Sequence Length
Sequence Length
Sequence Length
Figure 3: Performance of Yi-34B in variable tracking (VT), frequent words extraction (FWE), and QA tasks across different task complexities. Yi shows large degradation and distinct trends with scaled context size in these non-retrieval tasks, demonstrating the need to evaluate behavior beyond retrieval from context.
retrieve multiple items from long input. For instance, increasing the number of queries riigut))right....... from 1 to 8 drops the performance by 15 points (Figure right). When the model needs to retrieve multiple values associated with the same key (Figure middle-right)) Yi often outputs duplicated answers without returning the complete set of values, implying uneven associations between the key and each of its values.
Tendency to copy from context. We notice that Yi has strong tendency to copy from context verbatim when scaling the input length. This tendency is most notable in variable tracking (VT) and common words extraction (CWE) where include one in-context demon- stration at the beginning of the sequence. Over 80% of Yi's output in the CWE task at 128K is simply string copied from the one-shot example, whereas the copying is nonexistent for short sequences. ? This copying behavior is also present in the LWM model and Alpca,, however it is less prevalent in other models, such as Mixtral. This finding further reinforces the need to test behaviors other than retrieval given long input context.
Unreliable tracking within context. For the variable tracking task, both adding more chains and more hops contribute to large degradation in Yi's performance. Yi consistently degrades in the more-hops setting as increase context size (Figure leeft), whereas the degradation in the more-chains setting is most significant for lengths greater than 128K (Figure 3 middle- left). Besides the aforementioned copying issue, Yi makes errors due to incorrectly returning empty strings or variables from other chains, implying lack of ability to reliably trace the same entity within long context. These errors are also frequently observed in models that do not exhibit the copying behavior.
We also experimented with removing the one-shot example. The model will simply copy the string of the beginning of the input, likely due the attention sinks (Xiao al. 2024b).
8

---

Published as a conference paper at COLM 2024
Langth Comparison
Length Comparison
Size Comparison
Architecture Comparison
Accurecy
Accurocy
60
DAN base 1M
LVN-1M
DAH base 5724
40
LYN 522K
LAH base 2365
548 200K
20
LVN 256K
LaH base 128K
2006
LVM-128K
V-58-200K
Namba slmm))
2508
2516
Sequence Longth
Sequence Leng th
Sequence Length
Sequ Length
Figure 4: (Left& & middle left): Comparison of MordeWorlddMode (LWM) series trained up to various context sizes with fixed parameter size of 7B. (Middle right): Comparison of Yi suite models with different parameter sizes with controlled training context length of 200K. (Right): Performance of non-Transformer architectures lags behind the Transformer baseline Llama2-7B by large margin. Length extrapolation is presented with dashed lines.
Failure to accurately aggregate. We observe two common failure modes in aggregation tasks: incorrect use of parametric knowledge and inaccurate aggregation. Models that do not exhibit the copying issue in the CWE task, sometimes ignore the contextual information and instead use parametric knowledge to answer the query, especially at large context sizes. For instance, Mistral (7b-instruct-v0.2) returns high frequency words, such "the", "an", "a", as output without counting the words in context. For the FWE task which demonstrates less the copying issue, Yi fails to correctly output the top frequent words as we decrease the a in Zeta distribution (Figure middde-right) Decreasing a leads to smaller difference in frequency among words, increasing the difficulty distinguish the top-frequent words.
Frequent hallucination in long-context QA. For the QA tasks, Yi's performance ap- proaches its no-context baseline as we extend the context with distracting paragraphs (Figure 3 right). The degradation stems primarily from hallucination and reduced reliance on contextual information. We notice that, at large context sizes, model predictions some- times are irrelevant to the question and can coincide with the answers of its no-context baseline. The overall worse performance in QA tasks confirms that the fuzzy matching between a query and a relevant paragraph in long context more challenging setting than the simplistic NIAH tests, where keys can be exactly located in context.
6 Model Analysis
Effect of training context length. Do models trained with larger context sizes perform better on RULER? We evaluate the suite of LargeWorldModels (Liu al. 2024a LWM) of equal parameter size and trained up to various context lengths. Figure|4] (leit & middle-left) shows that larger context sizes overall lead to better performance, but the ranking can be inconsistent for long sequences. For instance, the model trained with 1M context size (LWM-1M) is worse than the one with 512K at length of 256K, likely due to insufficient training for adjusting to the new base frequency in RoPE. Moreover, performance drops when models need to extrapolate to unseen lengths (e.g., LMW-128K given input of 256K), and almost linear degradation with input length on log scale within the max training context size.
observe abrupt
Effect of model size The top models in our main results are much larger than other models. To ablate the effect of model size, we evaluate Yi-34B-200k, Yi-9B-200k, and Yi-6B-200k, all trained up to the same context length using the same data blend. Figure 4 (middle-right) shows that the 34B model is significantly better than the 6B model on RUILER for both performance length of 4K and the relative degradation, suggesting the benefit of scaling model sizes for better long-conted modeling.
Effect of architecture We evaluate the effective context length for two models with non- Transformer architectures: RWKV-v5 (Peng et al., 2023) and Mamba-2.8B-slimg, (Gu &

---

Published as a conference paper at COLM 2024
We find that both models demonstrate significant degradation when extending context size to 8K, and both underperform the Transformer baseline Llama2-7B by large margins up till the length of 4K, beyond which Llama2 shows poor length extrapolation performance 1iguuee4right
Dao 2023
Conclusion
We present RULER, synthetic benchmark for evaluating long-context language models. RULER contains diverse task categories, retrieval, multi-hop tracing, aggregation and question answering, providing flexible and comprehensive evaluation of LLM's long-contens ca- pabilities. We benchmark 17 fong-context LMs using RULER with context sizes ranging from 4K to 128K. Despite achieving perfect results in the widely used geedle-i--ahhasslan test, almost all models fail to maintain their performance in other tasks of RULER as we increase input length. We observe common failure modes at large context sizes, including the failure to ignore distractors and ineffective utilization of long context (e.g,, simply copy from context or use parametric knowledge instead). We show that RULER challenging for even the top-ranked open-source models as we increase task complexity. Our analysis further reveals the large potential for improvement on RULER and the benefit of scaling model sizes in achieving better long context capabilities.
8 Limitations
Despite covering more task categories than retrieval-oriented benchmarks, RULER limited in multiple ways which we describe in detail below.
Lack of position controlling. Current RULER reports single number metric for each input length without providing the depth-level performance. The depth-level performance was evaluated by the NIAH test (Kamradt) 2023) and recent works such as LV-Eval (Yuan et al., 2024) and can be effective in revealing the lost-in-the-middle (Liu et al. 2024d) phenomenon. We are aware of this issue and plan to support the position controlling of the key information in our codebase.
Lack of correlation with realistic long-context tasks. While tasks such as variable tracking and frequent words extraction were proposed serve proxies for real long-conter natural language tasks, the lack of -asy-to-evaluate realistic long-context tasks prevents us from verifying the validity of these proxies. Due this limitation, we emphasize that RULER can be used convenient behavioral checks of long-context language models, however it should not be preferred over more realistic settings, such as NoCHA (Karpinska et al. 2024), which also emphasize other capabilities such as reasoning and instruction-tollownng
Lack of evaluation short context. In the current RULER task suite, we include tasks that most models perform reasonably well at 4k context size, and aim to observe performance degradation with the scaling of context size. This should not be misread perfect LM capabilities at 4k context size. In fact, recent works, such as FlenQA (Levy et al., 2024), have demonstrated degrading performance when increasing their task input length to just few thousand tokens. While increasing the task complexity in RULER leads to much worse performance shorter context size, we did not include these results in this paper.
Lack of verification of prompt robustness. Language models can be sensitive to the prompt format, however we did not extend comprehensive study on the prompt robust- ness beyond preliminary testing in the early stage of this work. We also did not heavily experiment with few fixed hyperparameters in the existing tasks, such as the length of variable names in variable tracking and the synthetic vocabulary size in common word extraction and frequent word extraction.
10

---

Published as a conference paper at COLM 2024
References
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone, ar preprint Xiio:2404.142199 2024.
Rishabh Agarwal, Avi Singh, Lei M Zhang, Bernd Bohnet, Stephanie Chan, Ankesh Anand, Zaheer Abbas, Azade Nova, John Co-Reyes, Eric Chu, al. Many-shot in-context learning. arviv preprint ar aXXeo:440..11018 2024.
A121. Introducing jamba: Ai21's groundbreaking ssm-transformer model, 2024. URL https://www a121 www.bloo/////occoocccooooocooooo hawpl/////ooonnnnnggjiaaaa
Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language models. In ICLR,, 2024.
Anthropic.
Long context prompiing for Claude 2.1. Blog, 2023. URUntpp://www.nttrrop// comm:indee//lladdee-11-prrreeeeo
Anthropic. Introducing the next generation of claude, 2024. URL https://www.anthropi.
comm/news/CLaude-3ffaan
Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models. In ICLR, 2024.
Yushi Bai et al. LongBench: A bilingual, multitask benchmark for long context understand- ing. arXiv:2308.1 14508 2023.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prud- nikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory. Xrv preprint ar arXio:2405.04517 2024.
Amanda Bertsch, Maor Ivgi, Uri Jonathan Berant, Matthew R Gormley, and Graham Neubig. In-context learning with long-context models: An in-depth exploration. arXiv preprint (rXiv22405.00200 2024.
Aydar Bulatov, Yuri Kuratov, and Mikhail S Burtsev. Scaling Transformer to 1M tokens and beyond with RMT. ar Xrv::2304. 11062, 2023.
David Castillo, Joseph Davidson, Finlay Gray, José Solorzano, and Marek Rosa. In- troducing GoodAí LTM benchmark.
Blog, 2024. URL https://ww..gee gwodai..goo...oommoooooom com/
introducing-goodai 1esi-m1mmbbbeeee tm-tbbnbbbhk///
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. In JCLR, 2023.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. LongLoRA: Efficient fine-tuning of long-context large language models. In ICLR, 2024.
Rewon Child, Scott Gray, Alec Radford, and Tlya Sutskever. Generating long sequences with sparse Transformers. arXto:1904.115500 2019.
Cohere. Command r, 2024.
URL
r-plus
model-details]
Tri Dao. Atestonten Faster attention with better parallelism and work partitioning. arxiv:2307.086911,2000 2023.
Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In NeurlPS, 2022.
11

---

Published as a conference paper at COLM 2024
Databricks. Introducing dbrx: A new state-of-the-art open 11m, 2024. URL https://ww/
http:///iii...oooooooooo///tiiee o staerrddeeecine----eee---tt---ee Lm
Jiayu Ding et al. LongNet: Scaling Transformers to 1,000,000,000 tokens. arXio:2307.02486, 2023.
Yiran Ding et al. LongRoPE: Extending LLM context window beyond 2 million tokens. ar XfXo:2402..3753, 2024.
Zican Dong, Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. Bamboo: com- prehensive benchmark for evaluating long text modeling capacities of large language 2023.
models. arXiv:2309:.
Daniel Y. Fu, Tri Dao, Khaled K. Saab, Armin W. Thomas, Atri Rudra, and Christopher Ré. Hungry Hungry Hippos: Towards language modeling with state space models. In ICLR, 2023a.
Daniel Y. Fu et al. Simple hardware-efficient long convolutions for sequence modeling. ICML, 2023b.
Yao Fu et al. Data engineering for scaling language models to 128k context. arXit:2402.10171, 2024.
Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yuevan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiao Du, Zhenyu Hou, and Zihan Wang. Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.
Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. Is it really long context if all you need is retrieval? towards genuinely difficult long context nlp. ar Xiv preprint ar Sit:2407.0040) 2024.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural Turing machines. N10::4110,5411, 2014.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. #rXio:2312.00552 2023.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In ICLR, 2022.
Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, and Sinong Wang. Lm-infinite: Simple on-the-fly length generalization for large language models. xrXiv:2308.16137, 2023.
John Hopfield. Neural networks and physical systems with emergent collective computa- tional abilities. Proc of the National Academy of Sciences of the United States of America, 79 2554-8, 8, 1982.
Maor Ivgi, Uri Shaham, and Jonathan Berant. Efficient long-text understanding with short- text models. Transactions of the ACL, 11:284-299, 2023.
Sam Ade Jacobs et al. DeepSpeed Ulysses: System optimizations for enabling training of extreme long sequence Transformer models. arXio:2309.14509 2023.
Sebastian Jaszczur et al. Sparse is enough in scaling transformers. In NeurIPS, 2021.
Albert Jiang et al. Mixtral of experts. arXit:2401.04088. 2024.
12

---

Published as a conference paper at COLM 2024
Huiqiang Jiang et al. LongLImLingua: Accelerating and enhancing LLMs in long context scenarios via prompt compression. arXio:2310.06839, 2023.
Gregory Kamradt. Needle In A -aystack pressure testing LLMs. Github, 2023. URL
https://github.cm com/ BelaaraatttLMMMeestteeeeeeeaaaa http:/aasstta///ee/////nnann
Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit lyyer. One thousand and one pairs: A" novel" challenge for long-context language models. aiv preprint drXio:2406.16264, 2024.
Lauri Karttunen. Discourse referents. In COLING, 1969.
George Kingsley Zipf. Selected studies of the principle of relative frequency in language. Harvard university press, 1932.
Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Sorokin, Artyom Sorokin, and Mikhail Burtsev. Babilong: Testing the limits of Ilms with long context reasoning-in a-haystack. ar Xiv preprint ar arXiv:2406.10114 2024.
Woosuk Kwon et al. Efficient memory management for large language model serving with paged attention. In Proc. of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, Yi Luan, Sébastien MR Arnold, Vincent Perot, Siddharth Dalmia, et al. Can long-context language models subsume retrieval, rag, sql, and more? ar preprint arXiv:2406.13111, 2024.
Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv preprint ar arXi::2402.14488 2024.
Dacheng Li, Rulin Shao, et al. How long can open-source LLMs truly promise on context length?, 2023a. 2023a. URL httpss://mmyss http://mmww...rrr//lo////222---0 26-
Jiaqi Li, Mengmeng Wang, Zilong Zheng, and Muhan Zhang. Loogle: Can long-context language models understand long contexts? #rXie:2311.04938 2023b.
Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise Transformers for near-infinite context. In ICLR, 2023.
Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with Ring Attention. arxio:2402.09268. 2024a.
Jiaheng Liu et al. E2-LLM: Efficient and extreme length extension of large language models. 0iii:2401.0695: 2024b.
Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. Repoqa: Evaluating long context code understanding. Xiv preprint ar Xix::2466.06022 2024c.
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the ACL, 12:157-173, 2024d.
Pedro Henrique Martins, Zita Marinho, and Andre Martins. co-former: Infinite memory Transformer. In Proc. of the 60th Anmual Meeting of the ACL (Volume 1: Long Papers), 2022.
Meta.AI. Llama 3 model card. 2024a. /ttpr//te////ttt.o.oooooaaaaaaaa
main/MODEL CARD. md)
Meta.AI. Llama 3.1 model card. 2024b.
URL https:://githbb.coo//eett
mdl
Mistral.AI La plateforme, Utt.hh/pp///mssrrraaaaaa////aaap
13

---

Published as a conference paper at COLM 2024
Amirkeivan Mohtashami and Martin Jaggi. Landmark attention: Random-access infinite context length for Transformers. In Workshop on Efficient Systems for Foundation Models ICML, 2023.
Vincent Ng. Supervised noun phrase coreference research: The first fifteen years. In Proc. of the 48th Annual Meeting of the ACL, 2010.
Catherine Olsson et al. In-context learning and induction heads. Transformer Circuits Thread, 2022. https://transformettt /tt/://afformeerdiiiitsspppp http//////...oeeeeeeeeeaaaaaaaaa heads/index.html.
OpenAI: Josh Achiam et al. GPT-4 technical report. arX1to2303.08774, 2023.
Bo Peng et al. RWKV: Reinventing RNNs for the transformer era. In EMNLP, 2023.
Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. YaRN: Efficient context window extension of large language models. In ICLR, 2024.
Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher Ré. Hyena hierarchy: Towards larger convolu- tional language models. In ICML, 2023.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In ICLR, 2022.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD, Proc. of the 56th Annual Meeting of the ACL (Volume 2: Short Papers), 2018.
Machel Reid et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv:2403.0553) 2024.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proc. of the 58th Annual Meeting of the ACL, 2020.
Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In EMNLP, 2023.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: Enhanced Transformer with rotary position embedding. arXiv:2104.09864 2023.
Simeng Sun, Katherine Thai, and Mohit lyyer. ChapterBreak: A challenge dataset for long-range language models. In Proc. of the 2022 Conference of the North American Chapter of the ACL: Human Language Technologies, 2022.
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to Transformer for large language models. ar arXiv:2307.08621, 2023a.
Yutao Sun, Li Dong, Barun Patra, Shuming Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, and Furu Wei. A dength-extrapolltable Transformer. Proc. of the 61st Annual Meeting of the ACL (Volume 1: Long Papers), 2023b.
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang. Shuming Ma, Quanlu Zhang, Jianyong Wang, and Furu Wei. You only cache once: Decoder-decoder architectures for language models. ar Xie preprint ar Xrx::2405.05254, 2024.
Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi, A benchmark for learning to translate new language from one grammar book. In ICLR, 2024.
Together.AL. Preparing for the era of 32k context: Early learnings and explorations, 2023a. URL https://www/ www..ttoeett..aai.ee aitblrraai////////e Barlllogglllmaaa 1aap://llaaaa22--aa-----
14

---

Published as a conference paper at COLM 2024
Together.A. LLama-277-32--instuutt- and fine-tuning for llama-2 models with together api, 2023b. URL https:///ww.
Hugo Touvron et al. drXiv:230.009288 2023.
Llama 2: Open foundation and fine-tuned chat models.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the ACL, 10: 539-554, 2022.
Szymon Tworkowski et al. Focused Transformer: Contrastive training for context scaling. NeurlPSS, 36, 2024.
Teun A. van Dijk and Walter Kintsch. Strategies of discourse comprehension. In Academic Press, 1983.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, and Furu Wei. Augmenting language models with long-term memory. NeurIPS, 36, 2024.
Thomas Wolf et al. Huggingface's Transformers: State-of-the-art natural language process- ing. arXto:1910.03371 2019.
Qingyang Wu, Zhenzhong Lan, Kun Qian, Jing Gu, Alborz Geramifard, and Zhou Yu. Memformer: A memory-augmented Transformer for sequence modeling, In Findings of the ACL: AACL-IJCNLP, 2022.
X.AI. Announcing grok-155, 2024. URL
Chaojun Xiao et al. InfLLM: Unveiling the intrinsic capacity of LLMSs for understanding extremely long sequences with training-free memory. arXim:2402.04617 2024a.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient stream- ing language models with attention sinks. In ICLR, 2024b.
Wenhan Xiong al. Effective long-context scaling of foundation models. arXiv:2309.16039 2023.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In ICLR, 2024a.
Xiaoyue Xu, Qinyuan Ye, and Xiang Ren. Stress-testing long-context language models with lifelong icl and task haystack. arXiv preprint wxXi:22477.16695, 2024b.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. ar preprint arXiv:2407.10671, 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhut- dinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
Alex Young et al. Yi: Open foundation models by 01.AI. arXin:2403.04652,P 2024.
Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang, Shiyao Li, Minghui Zhuang, Zheyue Tan, Zhuyu Yao, Dahua Lin, Boxun Li, et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. ar preprint ar arxi:22402.0111
2024.
Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, and Zhicheng Dou. Soaring from 4k to 400k: Extending LLM's context with activation beacon. arXiv:2401.03462, 2024a.
15

---

Published as a conference paper at COLM 2024
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. oobench: Extend- ing long context evaluation beyond 100k tokens. arXfv:2402.13718, 2024b.
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, and Sujian Li. PoSE: Efficient context window extension of LLMs via positional skip-wise training. In ICLR, 2024.
16

---

Published as a conference paper at COLM 2024
A
Models
We select in total 37 models for evaluation and analysis. Our results in the main text only include aligned models (GPT-4, Gemini-1.5, and 15 open-source models). Besides the aligned models, we also evaluate 7 open-source base models using RULER. We use the performance of Llama2-7b (base) and Llama2-7b (chat) context length of 4K the threshold for determining effective context size. In our analysis section, we evaluate in total 11 models, including model series Yi and LWM, as well as models of novel architectures, including Mamba and RWKV.
Model
Aligned
Size
Context Length
Huggingface Wolf et 2019 API
GPT-4 OpenAL Jesh Achiam et al. 2025
128K
spt-4-1186-press previev 1ontni-1.5-pr9
GeminF13
Reld crepp 2023
1M
Llama3.1
Mets 25
70B
128K
meta- Saaaslllma//// Meta-LLama- 1-700-Instruc
Laam3.11
88
MMta- 1.-m----------rttruuc
1048
CohereFowA1 wttpr/////illlmmmmeeeeee.
Owen2
Fanket
2021
728
128K
Yi Yourig 4 (M/43
348
200K
O1-ai/ Y1-348-200
Mistral:85221:
[f 2024
398/1418
32K
mistralai/ Maaaaai//issall-/ Saatral--k228--nstmtt---0
Mistral-v0.2
Misuor
7B
32K
mistralai (istral-7B-instruutt0000
GLMH
THUDM/ /tlt////mmmm----uttt--
GradientAd/ Llanna
gradientat/
Sestrrcc-rraientte
Phil-mediium (bbline 4074
143
128K
microsott/ Phi-3-mediuu-- 128k-instruct
LIVM [Li 7.
7B
1M
LargeWorldMMudel LLtWWLL---eeet-hute
DBRX (Larabor S3 2022
36D 26D/1320
IM
databricks, / /bbx in-tincttcct
Together TogehersAi 20230
7B
32K
togethercemputer Llamma2-7B--3---nnae
LongChal
7B
LongAlpaca
Chee
Yukang/ LangAlpaca-130
Mixtral-ase
liancetal
2744
8x78
mistralii MMixtrl-8s<7--0111
Mistral-base
7H
32K
apinddlee Mistral-7B--0---d
LWV--base
Lu ocede 2013
7B
1M
SorgeWWordMMoee/Deee LLWM--Tax- IM
LongLoRA
2024
78
100K
Yukang/ Llamm-2-7--oonglare 100k--t
Pemg
7H
NousReeeanchh /aamm---lllamgnn 2--m--22----44
Together-Keet
logethencempeter/ 2
Jamba-base
Alz1 2029
52B
256K
si2llabs/ 22llabs/Jambb-v0)
lamaa2 (chal) Inivroo et 20133
7B
metn Hama/ lama---7h--calll
1lama2 (base) Touven
7B
4K
metn llama/ LLlamaaaaa2222 hd
Yi seriees
Youngetal 2021
SB,98
200K
01-ai/ (Y-(68 98)-200K
IWM
7B
128K
256K,552K2
LargeWorldMadse LLW (128K
LWM--base
32K, S12K
LargeWorlddMddel Btt.ee/eLLLL-MMM/00.....
Mamba
Gar Dao
2,88
state-sacces mwmmaam222mmmmmmm 8b-slimp)
RWKV
Pengger sl
7B
RWKV CCVV/v5.. Eagle-7D-HIP
Table 4: Information of evaluated and analyzed models in RULER.
17

---

Published as a conference paper at COLM 2024
B Task Configurations
RULER is designed to be contigurable to allow for diverse sequence lengths and task complexities. For each task, there arises combinatorially large number of configurations one can adopt. In the main text, we evaluate the models with 13 representative tasks spanning the four categories of RULER. Our task selection process is described in the next appendix section.
Retrieval: In S-NTAH, we include the passkey retrieval (Mohtashami & Jaggi 2023 and the vanilla NIAH (Kamradt) 2023), both use word-number as key-value and differ only by the background haystack. Additionally, we change the value type to UUID, for the purpose of testing model robustness retrieving long strings from context. For MK-NIAH, we add three distractor needles into the haystack. We also include existing setups from previous works: line retrieval (Li et al., 2023a) and key-value retrieval (Liu al. 2024d) with the haystack filled entirely with distractor needles. For MV-NIAH and MQ-NIAH, we test 4 values and queries respectively.
Multi-hop tracing: For VT, we insert chain with name-binding hops, totally variable names need to be returned.
Aggregation: For CWE, in total 10 common words need to be returned, each appears 30 times whereas the uncommon words appear times each. For FWE, we set a to 2.0 in Zeta distribution for sampling synthetic words.
QA: For QA, we augment SQuAD (Rajpurkar et al. 2018 and HotpotQA (Yang et al. 2018 to simulate long-context scenario. They are representative of single-hop and multi-hop question answering tasks respectively.
Configurations
Task
Subtask-1
Subtask-2
Subtask-3
type key word type. number type haystack repeat passkey retrieval
: word type-value number type haystack essay
type key word type_value uuid type haystack essay
Single NIÀH
~aanilla NIAH
num_keys 4
num = full haystack type word
num full haystack type.key key uuid
type type.key word type.value number type.haystack= = essay
type
MK-NIAH
type_value number
type.value = uuid
line retrieval
~V retrieval
MV-NIAH
num.values 4, type word, type.value number, haystakk essay
MQ-NIAH
num-queries 4, type_key word, type. number, type.haystack essay
VT
num_chains 1, num =4
CWB
freq.ow= = 30, freq_uew = 3, num_cw 10
FWE
2.0.0
QA
dataset SQuAD
dataset HotpotQA
Table 5: Our total 13 task configurations in RULER.
18

---

Published as a conference paper at COLM 2024
C Task Correlation Analysis
RULER is designed under the assumption that tasks across different categories are able to reveal distinct model behaviors. We conduct a preliminary correlational study to confirm the validity of task categories and guide the selection of representative tasks. We evaluate eight open-sourced models at various context sizes across 18 task configurations. Each task can then be represented with vector of model performance at various context sizes. The 18 task vectors are then clustered via agglomorative clustering algorithm, using correlation coefficient as the distance metric. As shown in Figure while certain tasks exhibit moderate correlations with others, tasks in each the four categories (NIAH, VT, AG, QA) form cohesive clusters of their own without redundancy. We further eliminate few tasks that correlate highly with other tasks within the same cluster, and finalize 13 tasks for later large scale evaluation.
a=2.0 (FWE)
am3.5 (FWE)
squad (QA)
hotpotqa (QA)
chain=2, hop=2 (VT)
chain-1, hop-4 (VT)
#q-4 (MQ-NIAH)
#v-4 (MV-NIAH)
(MV-NIAH)
#q=2 (MQ-NIAH)
kmW, V-U, hmessay (S-NIAH)
k-W, v-W, h-essay (S-NIAH)
#k=4 (MKNIAH)
k=W, V=N, h=essay (5-NIAH)
k=W, V=N, h=repeat (S-NIAH)
#k-full (MKNIAH)
#k-full, K-U, V- (MKNIAH)
a=2.0 (FWE)
a (FWE)
squad (QA)
hotpatqa (QA)
chain=2, hop=2 (VT)
chain=1, 1, hop (VT)
CWE
#q=4 (MO- NIHH
#und (MV-NIAH)
#y=2 (MV NIHH)
keW, you, hesssay (S-NIAH)
# (MO- NIAH)
W veW. hw heessay (-NIAH)
kw Ver, hwessay (S (-AHHH))
ke ft. herepeat (S-NIHH)
#ked NIAH)
#k=full, KrU, vou (MK-NIAH)
#ksttull NIAH)
Figure Correlation heatmap among 18 tasks with diverse task configurations. We remove redundant tasks (in red) and only preserve 13 representative tasks in RULER. (W: words; N: numbers; U: UUIDs; Full: entire haystack)
19

---

Published as a conference paper at COLM 2024
D Prompt Templates
We decompose the input prompt template into the model template in Table the task template in Table The model template is the model chat format while the task template combines instruction, context, and query. To prevent models from refusing to answer our questions, we append the input with an answer prefix to elicit model responses. For VT and CWE, we use one task sample as in-context demonstration.
Model
Template
GPT-4
(task_template) Do not provide any explanation. Please directly give me the answer. (task.answer.prefix
Yi/Base
(task.template) (task_answer_prefix)
BOS TOKEN)
START-OF TERTTUIN.KEEN
USER.OKEEN TOKENI) (task template)
Command-R
END OF TURN TOKEN))
START LFT TFT.UUNNTOOEEN CHATBOT
LWM/LongChat
(system.promp) task.answer.preeht prefix
USER: (lask.template)
ASSISTANT:
[gMASK] sop (/usr))
GLM
(task.template } [assisttntt]
(task.answer.preehf
(Juser))
task.template (/endl)
Phi3
|assistant])
(tsk_ansserr_reefix]
(lim start])system
system_.rompt|(ime
(jim_start)) user
Qwen/DBRX
task.template| (lim_end])
iimstaars (ask_answer.preeii)]
(lbegin of text)) (|start header id)) useru/eeeenee (/end header idl)
(task template ) (/eot id))
Glama//Llamas
(|start header ( assissantt( header id|)
(task.answer.prefix)
Llama2/Others
[NSTT] (task,template template) /INST]]
aanswer prefix)
Table 6: Model chat templates. We append task answer prefix in model response to prevent models from refusing to answer our questions. The addition of answer preffx does not break the models' chat template.
20

---

Published as a conference paper at COLM 2024
Task Template:
Some special magic numbers are hidden within the following text. Make to memorize it I will quiz you about the numbers afterwards.
The grass green. The sky is blue. The sun is yellow. Here we go. There and back again.
S-NIAH Subtask-1
One of the special magic numbers for (word) is: (number).
What the special magic number for (word) mentioned in the provided text?
Task Answer Prefix:
The special magic number for (word) mentioned in the provided text is
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize it I will quiz you about the numbers afterwards.
Paul Graham Hssays.
S-NIAH Subtask-2
One of the special magic numbers for {word) is: (number).
What is the special magic number for (word) mentioned in the provided text?
Task Answer Prefix:
The special magic number for (word) mentioned in the provided text is
Task Template:
Some special magic words are hidden within the following text. Make sure to memorize it. I will quiz you about the words afterwards.
Paul Graham Hssays.
S-NIAH Subtask-3
One of the special magic words for {word] | is: {word).
What the special magic word for (word) mentioned in the provided text?
Task Answer Prefix:
The special magic word for (word) mentioned in the provided text is
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize I will quiz you about the numbers afterwards.
Paul Graham Hssays.
One of the special magic numbers for [word-1} is: [number-1
MK-NIAII Subtask-1
One of the special magic numbers for [word-2] is: (number-2.
One of the special magic numbers for [word-3) is: (number-3) }.
One the special magic numbers for [word-4) is: (number-4).
What the special magic number for (word-4) mentioned the provided text?
Task Answer Prefix:
The special magic number for (word-4) mentioned the provided text
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize
it I will quiz you about the numbers afterwards.
One of the special magic numbers for (word-1) 1) ist (number-1).
One of the special magic numbers for [word-2) ( ist (number-2).
MK-NIAH Subtask-2
One of the special magic numbers for [word-x) is: (number-x).
One of the special magic numbers for (word-n-1) is: (number-n-1).
One of the special magic numbers for (word-n) is: (number-n).
What is the special magic number for (word-x) mentioned in the provided text?
Task Answer Prefix:
The special magic number for (word-x) mentioned in the provided text is
Task Template:
Some special magic uuids are hidden within the following text. Make sure to memorize will quiz you about the uuids afterwards.
One the special magic uuids (uuid-1) [uuid-1).
One the special magic uuids (uuid-2] is: [uuid-2].
MK-NIAH Subtask-3
.... One of the special magic uuids for [uuid-x) {uuid-x)......
One the special magic uuids (uuid-n-1) is: (uuid-n-1).
One the special magic uuids [uuid-n] is: [uuid-n).
What the special magic number for (uuid-x) mentioned the provided text?
Task Answer Prefix:
The special magic number for (uuid-x) mentioned the provided text
Table 7: S-NIAH and MK-NIAH templates.
21

---

Published as a conference paper at COLM 2024
Task Template:
Some special magic numbers are hidden within the following text. Make to memorize I will quiz you about the numbers afterwards.
Paul Graham Essays.
One of the special magic numbers for [word] is: [number-] 1
One of the special magic numbers for [word] is:
[number-2]
MV-NIAH
One of the special magic numbers for (word) is: (number-3
One of the special magic numbers for [word) is: (number-4).
What are all the special magic numbers for (word) mentioned in the provided text?
Task Answer Prefix:
The special magic numbers for (word) mentioned in the provided text are
Task Template:
Some special magic numbers are hidden within the following text. Make sure to memorize will quiz you about the numbers afterwards.
Paul Graham Essays.
One the special magic numbers for (word-1} is: (number-1
One the special magic numbers for [word-2) ) is: (number-2)
One the special magic numbers for (word-3) is: (number-3)
MQ-NIAII
One the special magic numbers for [word-4) is: (number-4)
What are the special magic numbers for {word-1], (word-2), (word-3), and {word-4)
mentioned the provided text?
Task Answer Prefix:
The special magic numbers for (word-1), (word-2), (word-3), and (word-4) mentioned the provided text are
Task Template:
(one task example)
Memorize and track the chain(s) of variable assignment hidden in the following text.
The grass is green. The sky is blue. The sun is yellow. Here go. There and back again.
VAR [X1 (uuberr)
VAR X2 = X1]
VT
VAR X3 = [X2)
VAR [4] (X3
VAR (55 = (X4)
Question: Find all variables that are assigned the value (number) in the text above.
Task Answer Prefix:
Answer: According to the chain(s) of variable assignment in the text above, 5 variables are assigned the value (number), they are:
Task Template:
(one task example)
Below numbered list of words. In these words, some appear more often than others. Memorize the that appear most often.
CWE
word-a word-b 3. word-c 4. word-a 5. word-d word-a word-e word-f
Question: What are the 10 most common words in the above list?
Task Answer Prefix:
Answer: The top words that appear most often the list
Task Template:
Read the following coded text and track the frequency of each coded word. Find the three most frequently appeared coded words. word-a word-b word-d word-e... word-a word-f
word-c word-a word-g ... word-h... word-a word-i .... Question: Do not provide any explanation. Please ignore the dots '...'. What are the three most frequently appeared words in the above coded text?
FWE
Task Answer Prefix:
Answer: According to the coded text above, the three most frequently appeared words are:
Table 8: MV-NIAH, MQ-NIAH, VI, CWE, and FWE templates.
22

---

Published as a conference paper at COLM 2024
Task Template: Answer the question based on the given documents. Only give me the answer and do output any other words.
The following are given documents.
Document 1:
[document-1] 1]
Document
Single Hop
[document-x]
QA
Document n:
[document-n]
Answer the question based on the given documents. Only give the answer and do not output any other words.
Question: question
Task Answer Prefix: Answer:
Task Template:
Answer the question based on the given documents. Only give me the answer and do not output any other words.
The following given documents.
Document
(document-1)
Document
[document-x]
Multi Hop
Document y:
QA
(document-y)
Document n:
[document-n]
Answer the question based on the given documents. Only give me the answer and do not output any other words.
Question: question
Task Answer Prefix:
Answer:
Table 9: QA templates.
23

---

Published as a conference paper at COLM 2024
E
Passkey Retrieval and Vanilla NIAH Results
Claimed Length
Models
4K
SK
16K
32K
64K
128K
Avg.
Gemini-1.5
1M
100.0
100.0
100.0
100.0
100.0
100.0
100.0
GPT-4
128K
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Llama3.1 (7OB)
128K
100.0
100.0
100.0
100.0
100.0
97.8
99.6
Llama3.1 (8B)
128K
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Qwen2 (72B)
128K
100.0
100.0
100.0
100.0
100.0
96.6
99.4
Command-R-plasn (104B)
128K
100.0
100.0
99.8
99.8
100.0
97.2
99.5
GLM4 (9B)
1M
100.0
100.0
100.0
100.0
100.0
100.0
GradientAI/Llamas (70B)
1M
100.0
100.0
100.0
100.0
100.0
93.6
98.9
Mixtral-8x22B (39B/141B)
64K
100.0
100.0
99.6
0.0
(3418)
200K
100.0
100.0
100.0
100.0
100,
Phi3-medium (14B)
128K
100.0
100.0
100.0
100.0
100.0
88.0
98.0
Mistral-v0.2 (7B)
32K
100.0
100.0
100.0
100.0
99.6
69.6
94.9
LWM (7B)
1M
100.0
100.0
100.0
100.0
100.0
100.0
100.0
DBRX (36B/1328)
32K
100.0
100.0
100.0
100.0
0.0
0.0
66.7
Together (7B)
32K
100.0
100.0
100.0
100.0
0.0
0.0
66.7
LongChat (7B)
32K
100.0
100.0
100.0
99.4
0.0
0.0
66.6
LongAlpaca (13B)
32K
88.2
88.6
86.4
82.4
0.0
0.0
57.6
Mixtral-base (8x7B)
32K
100.0
100.0
100.0
100.0
100.0
46.8
91.1
Mistral-base (7B)
32K
100.0
100.0
100.0
100.0
99.6
70.8
95.1
lamba-base (52B)
256K
100.0
100.0
100.0
100.0
100.0
100.0
100.0
LWM-base (7B)
1M
99.8
100.0
99.6
99.6
98.2
96.0
98.9
LongLoRA-base (7B)
100K
99.6
99.4
99.0
99.4
99.4
0.0
82.8
Yarn-base (7B)
128K
100.0
100.0
99.0
100.0
99.2
39.6
89.6
Together-base (7B)
32K
100.0
100.0
99.8
100.0
0.0
0.0
Table 10: Performance of selected aligned and base models across length 4K to 128K in passkey retrieval of RULER. Almost all models have perfect score at their claimed length.
Claimed Length
Models
4K
8K
16K
32K
64K
128K
Avg.
Gemini-1.5
1M
100.0
100.0
100.0
100.0
100.0
GPT-
128K
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Llama3.1 (70B)
128K
100.0
100.0
100.0
100.0
100.0
99.6
99.9
Ilama3.1 (8B)
128K
100.0
100.0
100.0
100.0
100.0
99.6
99.9
Qwen2 (728)
128K
100.0
100.0
100.0
100.0
99.8
56.4
92.7
Command-R-pp -oamand-R-plus (35B)
128K
100.0
100.0
100.0
100.0
99.8
86.0
97.6
GLM4 (9B)
128K
100.0
100.0
100.0
100.0
100.0
100.0
100.0
KradientAAl/Llamee (70B)
1M
100.0
100.0
100.0
99.6
99.2
97.8
99.4
Mixtral-8x22B (39B/141)
64K
100.0
100.0
100.0
100.0
99.6
24.2
87.3
Yi (34B)
200K
100.0
100.0
100.0
100.0
100.0
100.0
100.0
Phi3-medium (14B)
128K
100.0
99.8
100.0
99.8
99.8
73.8
95.5
Mistral-v0.22779 (7B)
32K
100.0
100.0
100.0
97.0
70.0
7.4
79.1
LWM (7B)
1M
100.0
100.0
100.0
100.0
100.0
100.0
100.0
DBRX (36B/132B)
32K
100.0
100.0
90.0
0.0
Together (7B)
32K
100.0
100.0
100.0
0.0
LongChat (7B)
32K
100.0
100.0
97.6
0.0
LongAlpaca (13B)
32K
90.2
90.2
88.4
0.0
0.0
58.7
Mixtral-base (8x7B)
32K
100.0
100.0
100.0
100.0
85.2
34.8
86.7
Mistral-base (7B)
32K
100.0
100.0
100.0
100.0
94.8
0.4
82.5
Jamba-base (52B)
256K
100.0
100.0
98.8
99.8
99.8
86.4
97.5
I.W--base (71)
1M
100.0
99.4
97.8
98.6
98.2
98.6
98.8
LongLoRA-base (7B)
100K
99.8
100.0
100.0
99.8
100.0
0.0
83.3
Yarn-base (7B)
128K
97.4
97.8
91.4
85.4
86.6
20.0
79.8
Together-basc (7B)
32K
100.0
100.0
100.0
99.8
0.0
0.0
66.6
Table 11: Performance of selected aligned and base models across length 4K to 128K in vanilla NIAH of RULER. Almost all models have perfect score at their claimed length.
24

---

Published as a conference paper at COLM 2024
F
Additional Results
Claimed Length
Effective Length
wAvg. (inc)
wAvg.
Models
4K
8K
16K
32K
64K
128K
Avg.
(dec)
Llama2-7B (basc)
4K
79.4
Mixtral-base (8x7B)
32K
32K
91.8
91.0
89.5
85.8
66.9
29.0
75.7
66.4 66.4
85
Mistral-base (7B)
32K
16K
91.6
89.8
86.3
77.2
52.3
8.0
67.5
54.7
80.4
Jamba-base (52B)
256K
4K
81.2
75.4
68.8
65.3
61.0
514
67.2
62.5ppd
71.8(ath)
LWM-bbse (7B)
<4K
77.5
74.0
69.6
64.6
61.3
59.0
67.7
70.9/500 64.4(zed)
LongLoRA-base (7B)
81.9
50.4
73.6
65.1
0.0
60.6
49.2 (9h2
72.0(ard)
Yarn-base (7B)
<4K
77.3
67.5
59.0
47.3
13.9
50.6
40.7 (0.7
60.5,700
Together-base (7B)
4K
84.6
68.3
57.9
0.0
48.2
32.3
64.2
Table 12: Performance of selected base models across length 4K to 128K by averaging 13 task scores in RULER.
Claimed Length
Effective Length
wAvg. (inc)
wAvg. (dec)
Models
SK
16K
32K
64K
128K
Avg.
Llama2-7B (chat)
4K
96.9
Gemini-1.5
1M
> -28K
99.8
99.9
99.6
99.7
99.7
99.6
99.7
99.7(1st)
99.7 (1at)
Ilama3.1 (8B)
128K
98.7
92.6
97.5(3nd)
99.4(2nd)
GLM4 (9B)
1M
64K
99.4
99.2
99.5
99.4
97.3
94.4
98.2
98. 98.9(3d)) 9/x5)
Llama3.1 (70B)
128K
64K
100.0
100.0
99.9
99.6
98.5
78.9
96.1
98.8 93.5/5m
GPT-4
128K
32K
99.9
99.9
98.7
98.3
90.9
84.8
95.4
92.9(6th)
97. 97.9(5a)
Command-R-plus (104B)
128K
99.9
99,4
97.9
89.6
65.7
92.1
87.3(am))
96.9(6)))
CradientAl/Lllmaal (70B)
1M
16K
98.8
98.3
94.5
91.2
84.9
94.4
92.1
96.8(tt)
Yi (34B)
200K
16K
98.2
96.8
97.3
95.1
93.0
90.2
95.1
93.8
96.4
Qwen2 (72B)
128K
32K
100.0
99.9
99.9
99.4
84.5
48.0
88.6
81.3
95.9
Phi3-medium (14B)
128K
98.5
96.6
95.4
91.9
51.3
88.7
82.6rict))
94.9(10th)
Mixtral-8x22B (39B/141B)
64K
16K
99.1
96.7
89.9
23.8
84.4
74.8
94.1
LWM (7B)
1M
<4K
92.5
92.1
87.6
83.7
84.1
83,4
87.2
85,5g9h)
89.0 89.0
Mistral-v0.2 (7B)
32K
4K
98.
96.2
94.3
85.5
51.1
10.7
72.6
58 (8380)300)
86.5
DBRX (36B/132B)
32K
99.4
99.0
93.5
73.4
0.5
0.0
61.0
41.6q1th
80.3
Together (7B)
32K
<4K
96.22
89,9
82.3
80.2
0.0
58.1
40.2
76.0
LongChat (7B)
32K
<4K
93.3
92.2
81.1
67.3
0.0
0.0
55.7
37.6mam)
73.7 (ea))
LongAAlpaca (13B)
32K
<4K
74.9
72.2
70.8
53.2
0.0
0.0
45.2
30.7
59.7
Llama2-7B (base)
4K
Mixtral-base (8x7B)
32K
32K
99.7
98.4
94.8
72.1
29.1
82.3
71.82ad)
92.8(1sb)
Mistral-base (7B)
32K
16K
99.3
97.5
95.7
89.8
56.8
10.2
74.9
61.224thh
88.6(2n4)
Jamba-base (52B)
256K
<4K
86.4
80.5
73.7
72.3
68.1
56.9
73.0
68.5,300
77.4ga
LWM-base (7B)
IM
<4K
87.7
84.5
79.6
76.1
74.2
81.8
79.1
84.444m
LongLoRA-base (7B)
100K
16K
95.6
81.5
76.2
0.0
73.5
60.6çsdo
86.5(add)
Yarn-base (7B)
128K
<<4K
89.9
86.1
78.4
59.0
49.5
17.5
63.4
75.1 51.7
Together-base (7B)
32K
8K
95.4
91.5
86.1
75.1
0.0
0.0
58.0
39.9
76.2
Table 13: Performance of selected aligned and base models across length 4K to 128K by averaging 8 task scores in Retrieval (NIAH) of RULER.
25

---

Published as a conference paper at COLM 2024
Models
Claimed Length
Effective Length
4K
8K
16K
32K
64K
128K
Avg.
wAvg. (inc)
w'vg,
(dec)
Llama2-7B (chat)
4K
89.7
0
GPT-4
128K
128K
100.0
100.0
100.0
100.0
100.0
99.6
99.9
99.9 99.9
100.0
Gemini-1.5
-128K
10000
10001
100.0
99.6
10000
99.9
99.9 (149
10010ppd)
Command-R (104B)
128K
128K
100.0
100.0
0,000
10000
99.9
97.2
99.5
99.2
99.8,sa)
CLM4 (9B)
1M
>128K
99.9
99.6
99.8
99.8
99.6
97.7
99.4
99.1
99.7 99.7(th)
Qwen2 (72B)
128K
64K
100.0
100.0
100.0
100.0
95.2
79.0
95.7
(2th(sth)
98.5
Llama3.1 (70B)
128K
64K
100.0
100.0
100.0
100.0
99.9
59.2
93.2
98.0 88.3
Llama3.1 (8B)
128K
64K
99.9
99.7
99.7
98.8
97.6
70.4
94.4
90.7
98.0
GradientAl/Llanen (70B)
1M
61K
100.0
100.0
100.0
100.0
99.7
56.2
92.6
(7.4
(7.9
Yi (34B)
200K
64K
99.8
99.2
98.8
94.5
92.5
76.8
93.6
90.3
96.9
Mixtral-8x22B (39B/141B)
64K
64K
100.0
100.0
99.8
98.6
96.4
0.0
82.5
30.3
94.7 94.7
Phi3-medium (148)
128K
98.4
82.1
53.6
26.0
76.5
64. (11h)
(8.9
Mistral-v0.2 (7B)
32K
16K
98.9
96.0
92.2
85.0
74.5
0.0
74.4
60.9
87 (2th
LongChat (7B)
32K
8K
97 6
93.5
83.4
62.4
0,0
0.0
56.2
37 4(14))
75.0
DBRX (368/1328)
32K
72.5
45.8
0.0
0.0
52.9
33.3
72-5
LWM (7B)
1M
<4K
84.4
80.1
67.2
52.2
45.9
15.2
57
46.5
68.6 68.6
Together (7B)
32K
<4K
89
88.8
48.3
16.6
0.0
0.0
40.5
22. 22,8
58. ((0th)
LongAlpaca (138)
32K
<4K
18.2
17.0
0.0
0.0
7.6
8. 8.8 6.5()16)
Llama2-7B (base)
4K
58.8
Mixtral-base (8x7B)
32K
64K
100.0
99.9
100.0
98.4
57.3
43.3
88.1
95.8y1s0 80.5
Mistral-base (7B)
32K
64K
99.0
98.4
96.5
89.1
0.0
78.2
65, 65.4(eth)
91.0(2nd)
Jamba-base (52H)
256K
128K
87.5
87.6
86.2
88.1
86.0
77.8
85.5
86.7(3e)) 84.3
LWM--ase (7B)
1M
128K
80.2
82.7
79.3
76.4
20.7
66.1
75.9
73.3
78.5
LongLoRA-base (7B)
100K
64K
92.5
87.
73.1
56.0
69.2
0.0
63.0
75.8 50.
Yarn-base (7B)
128K
4K
84.6
43.6
24.8
43.0
20.9
0.0
36.1
24.9
47
47.4mm)
Together-base (7B)
32K
16K
95.0
90.6
69.6
43.2
0.0
0.0
49.7
31.3
68. 68.1)
Table 14: Performance of selected aligned and base models across length 4K to 128K in Multi-hop tracing (VT) of RULER.
Models
Claimed Length
Effective Length
4K
16K
32K
64K
128K
Avg-
wAvg. (inc)
wAvg. (dec)
Llama2-7B-chat
4K
84.8
Gemini-1.5
1M
1228K
97.7
97.7
97.6
98.6
97.3
90.9
96.6
95.8
97. 4mw
GPT-4
128K
99.0
98.3
98.0
95.0
90.1
79.7
93.4
90.4 90.4
96.3
Qwen2 (72B)
128K
32K
99.3
98.0
93.1
97.4
78.5
70.3
89.4
64.7(3xd)
94.2(3ed)
Mixtral-8x22B (39B/141B)
64K
32K
97.8
96.9
94.8
88.2
83.3
69.7
88.5
84.0(4i))
(2.9(44)
Command-R R-pus (104B)
128K
32K
98.2
96.9
95.2
90.3
82.5
59.5
87.1
81.3(5m)
92.8(5t)
Llama3.1 (70B)
128K
32K
99.9
98.3
98.4
97.1
66.3
39.8
83.3
73.8
92.8(6th)
Phi3-medium (14B)
128K
16K
90.8
95.1
90.3
82.4
62.1
43.8
77.4
69.3
85.677th
Yi (34B)
200K
91.4
86.2
75.3
43.4
74.3
66.0
62, 62,6
GLM4 (9B)
1M
93.5
85.2
78.5
68.1
58.3
49.7
72.2
64.8(9th)
79.619th)
GradientAI/Llanme (70B)
1M
SK
96.4
94.7
74.9
57.0
45.1
41.4
68.3
57.7 (1.7
78.8
Llama3.1 (8B)
128K
8K
97.0
90.1
79.2
54.1
43.5
36.2
66.7
55.5
77.5 77.8
Mistral-vO.2 (7B)
32K
8K
94.3
90.4
77.4
48.5
42.4
33.7
64.4
13.1
75.8
DBRX (36B/132B)
32K
94.5
94.7
73.7
48.7
4.1
0.0
52.6
34.3
70.9
Together (7B)
32K
<4K
82.3
64.5
43.3
34.8
37.5
22.9
52.1
LongChat (7B)
32K
<4K
74.3
50.7
46.7
51.1
0.0
0.0
37.1
24.8
49.5
I.WM (7B)
1M
<4K
61.3
43.6
38.3
32.8
29.1
29.1
39.0
14
44.0
LongAlpaca (13B)
32K
<4K
33.0
27.0
26.0
23.2
0.0
0.0
18.2
12.3
24. 227
Llama2-7B (base)
73.1
Mixtral-base (8x7B)
32K
32K
96.5
93.1
87.8
24.3
77.5
66.9(14)
88:
Mistral-base (7B)
32K
16K
94.8
93.1
81.6
53.3
36.7
9.2
61.4
46.5
76.3
Jamba-base (52B)
256K
4K
75.9
63.5
51.7
38.5
33.3
28.0
48.5
40.3
56.6
LWM-base (7B)
1M
<4K
67.1
48.4
36.0
26,3
21.5
18.7
36.3
28.4(50))
44.2
LongLoRA- A-base (7B)
100K
<4K
70.3
64.4
50.7
39.9
29.4
0.0
42.4
31.3)44h)
53.6(440
Yam-base (7B)
128K
<4K
70.6
49.2
28.9
20.5
17.0
2.1
31.4
20.7
42.0goah)
Together-base (7B)
32K
<4K
69.1
53.0
19.9
20.6
27.1
15.
39.1
Table 15: Performance of selected aligned and base models across length 4K to 128K by averaging task scores in Aggregation (CWE/FWE) of RULER.
26

---

Published as a conference paper at COLM 2024
Models
Claimed Length
Effective Length
4K
16K
32K
64K
128K
Avg.
wAvg. (inc)
wAvg.
(dec)
Llama2-7B (chat)
49.7
Gemini-1.5
1M
81.9
75.9
75.9
74.1
77.2
76.3
78.0
GPT-4
128K
128K
79.0
78.0
76.0
68.0
61.6
59.0
70.3
(6.5
74.0)znd)
Qwen2 (728)
128K
64K
80.8
76.9
74.1
66.9
54.5
47.2
66.7
(1.0(s)))
72.5(sad)
GradientAI/Llaman (70B)
1M
> 128K
73.6
73.9
72.4
69.9
66.0
59.8
69.6
67. (3rd)
72.1(4m)
Llama3.1 (70B)
128K
64K
77.2
74.8
72.3
70.4
64.2
47.6
67.8
63. 4(7th)
72. 72.1
GLM4 (9B)
128K
74.7
71.3
71.9
68.5
63.6
69.4
6/2ad)
71.1(6th)
Mixtral-8x22B (39B/ 14)B)
64K
64K
76.6
73.5
71.8
66.5
39.7
40.8
64.8
59. 59.4(9th)
70.2
Yi (34B)
200K
128K
72.7
71.5
68.4
66.2
64.1
59.9
67.1
65.0(5th)
69. (19h)))))
Llama3.1 (8B)
128K
128K
74.1
70.1
67.3
65.8
63.7
58.8
66.6
64.3
68.9
Command-R-plus (104B)
128K
64K
73.4
72.3
69.
65.9
57.0
39.2
62.9
57 37.6)100)
68. 68.1
Phiß-medium (14B)
128K
70.9
67.2
66.1
59.3
38.0
59.3
(4.3
64.3
Mistral-v0.2 (7B)
7244
70.0
65.7
57.6
34.4
52.2
42.5(
62.0g12t)
LWM
1M
>128
61.2
57.8
56.7
55.4
52.6
56.4
57.7 57.7(12th)
DBRX (36B/132))
32K
16K
76.0
69.4
59.4
45.0
9.6
0.0
43.2
29.6
56.9
Together (7B)
32K
16K
61.1
58.3
54.2
45.6
0.0
0.0
36.5
24.9
48.2
LongAAlpcca (13B)
32K
16K
57.2
53.5
49.7
39.0
0.0
0.0
33.2
44.1 (10ch)
LongChat (7B)
32K
54.5
47.6
34.0
0.0
31.6
21.0
(273
Llama2-7B (base)
4K
48.6
Mixtral-basee (8x7B)
32K
4K
50.8
47.7
45.3
41.3
34.4
26.4
41.0
37 37.0(3nd)
44.9(3d))
Mistral-baa bastal-bbseeeee (7B)
32K
8K
53.5
51.0
48.4
44.7
32.8
2.2
38.8
31.3
46.3yamd)
Jamba-base (52B)
256K
32K
62.7
60.6
57.9
52.6
47.5
39.6
53.5
49.7
57.3
LWM-base (7B)
<4K
42.7
40.2
38.7
37.1
37.3
34.6
38.4
37.2 37.2
39.6(4t))
LongLoRA-bor (7B)
100K
34.5
32.1
29.4
26.1
26.0
21.3(6ih)
30.6G6/)
Yam-base (7B)
128K
<4K
29.7
23.5
28.6
29.7
25.5
18.1
25.9
24.6(5hh
27. 17.1
Together-bail (7B)
32K
4K
52.0
47.5
44.6
33.6
0.0
0.0
29.6
19,8
39.5
Table 16: Performance of selected aligned and base models across length 4K to 128K by averaging 2 task scores in Question Answering of RULER.
27


Reviews:
[{"review_id": "UGcnbhY30u", "reviewer": "Reviewer_cgKY", "summary": "The paper introduces a new synthetic benchmark, RULER, designed to evaluate the long-context modeling capabilities of language models (LMs). The authors argue that existing tests, such as the needle-in-a-haystack (NIAH), are insufficient to comprehensively assess LMs' understanding of long contexts. RULER expands beyond simple retrieval tasks to include multi-hop tracing, aggregation, and question answering, allowing for a more nuanced evaluation of model performance across various context lengths and complexities. The study involves testing ten long-context LMs with 13 representative tasks and finds that despite near-perfect accuracy in NIAH, all models experience significant performance drops as context length increases. The paper's analysis reveals that only a few models can maintain satisfactory performance at 32K tokens, and there is substantial room for improvement, especially as input length and task complexity increase.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer cgKY for their encouraging and thoughtful comments. \n\nWe agree with the reviewer that evaluating the latest models optimized for long-context scenarios with RULER can give us more insights. As such, we provide additional evaluations of Gemini-1.5-pro, Command-R-plus, Mixtral-8x22B, Phi3, DBRX, Qwen along with results from our submitted version in the table below. We plan to evaluate Claude 3 and Moonshot after finalizing the evaluation pipeline of these two models. \n\n|Models|Claimed Length|Effective Length|4K|8K|16K|32K|64K|128K|Avg.|wAvg. (inc)|wAvg. (dec)|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|Llama2 (7B)|4K||85.6|\nGemini-1.5-pro1M | 128K| >128K|96.7|95.8| 96.0| 95.9|95.9|94.4|95.8|95.5 (1st)|96.1 (1st)|\nGPT-4-1106-preview|128K|64K|96.6|96.3| 95.2| 93.2|87.0|81.2|91.6|89.0 (2nd)|94.1 (2nd)|\nCommand-R-plus(104B)|128K|32K| 95.6| 95.2| 94.2|92.0|84.3|63.1|87.4|82.7 **(5th)**|92.1 **(3rd)**|\nCommand-R (35B)|128K|32K| 93.8| 93.3| 92.4|89.5|84.9|76.0|88.3|85.5 **(3rd)**|91.1 **(4th)**|\nMixtral-8x22B (39B/141B)|64K|32K| 95.6| 94.9| 93.4|90.9|84.7|31.7|81.9|73.5 **(6th)**|90.3 **(5th)**|\nYi (34B)|200K|32K| 93.3| 92.2| 91.3|87.5|83.2|77.3|87.5|84.8 **(4th)**|90.1 **(6th)**|\nMixtral-8x7B (12.9B/46.7B)|32K|32K| 94.9| 92.1| 92.5|85.9|72.4|44.5|80.4|72.8 (7th)|87.9 (7th)|\nMistral (7B)|32K|16K| 93.6| 91.2|87.2|75.4|49.0|13.8|68.4|55.6 **(11th)**|81.2 **(8th)**|\nChatGLM (6B)|128K|4K|87.8|83.4|78.6|69.9|56.0|42.0|69.6|62.0 (10th)|77.2 (9th)|\nLWM (7B)|1M|<4K|82.3|78.4|73.7|69.1|68.1|65.0|72.8|69.9 **(8th)**|75.7 **(10th)**|\nPhi3 (3.8B)|128K|4K|86.7|78.1|75.6|70.3|58.9|43.3|68.8|62.2 **(9th)**|75.5 **(11th)**|\nDBRX (36B/132B)|32K|8K|95.1|93.8|83.6|63.1|2.4|0.0|56.3|38.0 (12th)|74.7 (12th)|\nQwen (72B)|32K|8K|94.9|93.8|78.0|67.8|0.0|0.0|55.7|37.5 (13th)|74.0 (13th)|\nTogether (7B)|32K|4K|88.2|81.1|69.4|63.0|0.0|0.0|50.3|33.8 (14th)|66.7 (14th)|\nLongChat (7B)|32K|<4K|84.7|79.9|70.8|59.3|0.0|0.0|49.1|33.1 (15th)|65.2 (15th)|\nLongAlpaca (13B)| 32K|<4K|60.6|57.0|56.6|43.6|0.0|0.0|36.3|24.7 (16th)|47.9 (16th)|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. RULER provides a more thorough evaluation of LMs by including diverse tasks that go beyond simple retrieval, offering a better understanding of how models handle long contexts. The benchmark allows for adjustments in sequence length and task complexity, making it adaptable for various research needs and future model evaluations.\n\n2. The paper presents detailed results and analysis across different models and context sizes, giving insights into the limitations and performance degradation of current LMs.\n\n3. The study reveals that most models struggle with complex tasks in long contexts, despite their claims of handling large context sizes, which is valuable information for both researchers and practitioners.", "reasons_to_reject": "1. The study focuses on a selection of models, which might not represent the entire spectrum of available LMs, possibly missing insights from a broader set. Actually, there are lots of models that specially optimized for long context scenarios, such as Claude and Moonshot, which the performance needs to be discussed.", "questions_to_authors": "na", "extra_scores": {}}, {"review_id": "UgT4nmt0HZ", "reviewer": "Reviewer_GXoR", "summary": "This paper presents RULER, a new benchmark to comprehensively evaluate the capabilities of long-context language models through synthetic tasks with flexible configurations. These tasks expand the needle-in-a-haystack test which is currently popular. Empirical results reveal that current long-text models exhibit large degradation on these complex tasks, and training on longer sequences does not always lead to better performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer GXoR for their encouraging comments and thoughtful advice.\n\nWe agree with the reviewer that there are similar long-context benchmarks and thus have included a brief comparison in section 1 Table 1. \n- Compared to existing synthetic evaluation such as passkey retrieval and needle-in-a-haystack, RULER covers diverse task categories such as tracing and aggregation, testing LM capabilities beyond simple in-context recall. \n- Compared to existing realistic long-context benchmarks, RULER reduces the impact of parametric knowledge by maximizing the reliance on the contextual information. Evaluating on input that has large overlap with the pre-training data (e.g., book data used by other benchmarks) entangles context utilization with memorized parametric knowledge, whereas testing on fully synthetic data allows us to test a model’s behavior while minimizing the memorization effect.\n- Compared to existing realistic long-context benchmarks, RULER also has the key advantage of allowing for flexible context length and varied task complexity. Existing benchmarks mostly adopt fixed input length, which can be too short to test “long”-context language models. In contrast, input can be adjusted to arbitrary length in RULER. Moreover, the number of distractors and the amount of relevant information can also be controlled by task hyperparameters, which lead to different task complexities. \n\nReviewer mentions a potential gap between the synthetic nature of RULER and the intricacy of real-world long-context applications. We acknowledge the limitation of synthetic tasks and would like to emphasize in the revised version that RULER can serve as diagnostic tests of LM capabilities (e.g., retrieval, tracing, and aggregation), however it cannot replace real-world long-context tasks. We agree with the reviewer that it can be interesting to see the correlation between RULER and other real-world long-context tasks, since many tasks in RULER are heavily motivated by realistic natural language tasks (e.g., variable tracking motivated by coreference chain resolution, and frequent word extraction motivated by summarization) after all. Due to the scarcity of high-quality naturalistic long-context tasks that are easy to evaluate, we decide that this particular direction is out-of-scope of our paper and can be a promising direction to pursue in the future.\n\nWe thank the reviewer for bringing these two points up and will add more detailed discussions in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Providing a novel benchmark that incorporates diverse tasks to test long-context abilities of large language models.\n- Thorough evaluation and error analysis of a range of models, providing a comprehensive understanding of their performance characteristics.\n- Well-written paper with clear motivation, illustration of tasks, and presentation of results.", "reasons_to_reject": "- There are many similar benchmarks, and the significance of this paper is not prominent. Further discussion may be needed on the advantages of this paper's benchmark and why these advantages are important.\n- The synthetic nature of tasks may not fully capture the challenges of real-world long-context applications. It may sound easy to construct more complex synthesis tasks, but we need more insights about real long-context requirements. For example, How well do the synthetic tasks in RULER correlate with performance on real-world long-context applications? Is it necessary for long-context models to complete these complex tasks such as extreme long chain of binding statements?", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "80vlY9jQXV", "reviewer": "Reviewer_HSfg", "summary": "The manuscript introduces RULER, a comprehensive synthetic benchmark designed to evaluate the real-world capabilities of long-context language models (LLMs). The authors argue that traditional benchmarks do not adequately measure true contextual understanding, especially across longer text spans. To address this gap, RULER integrates complex tasks including multi-hop tracing and information aggregation, aiming to provide a more accurate assessment of LLMs' long-context processing abilities.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer HSfg for their thoughtful comments on our submission.\n\nThe reviewer mentions the absence of a comparative study between RULER and real-world applications. We would like to mention that real-world long-context tasks usually have fixed input lengths, thus it can be difficult to perform evaluation like we do with RULER, where we evaluate models with increased context size while maintaining the same and known upper-bound performance. Moveover, existing long-context tasks employ imperfect automatic metrics (e.g. ROUGE for book-level summarization), whereas gold human evaluation for long-context tasks can be expensive and the topic itself warrants a separate research work 1]. We agree with the reviewer that the correlation between performance on RULER and other real-world applications can be fruitful to explore. However, due to the scarcity of high-quality naturalistic long-context tasks that are easy to evaluate, we decide that this particular direction is out-of-scope of our paper and can be a promising direction to pursue in the future.\n\nWe agree with the reviewer that additionally profiling inference speed and memory footprint can give valuable information about performance-efficiency trade-off. We will add the support of comparing inference efficiency in our open-source code repository in the future.\n\nReference:\n[1] BooookScore: A systematic exploration of book-length summarization in the era of LLMs", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The evaluation is thorough, testing ten different LLMs across 13 tasks that vary in complexity and length. This comprehensive testing provides a clear picture of how well these models perform under rigorous and varied conditions.\n2) The manuscript offers valuable insights into the performance degradation of LLMs as context size increases. These insights are crucial for understanding the limitations of current LLMs and could guide future research and development in the field.", "reasons_to_reject": "1) The dataset constructed in this article primarily extends the range of tasks and enriches the current evaluation dataset. However, it shows limited innovation in terms of novel methodologies or insights into language model capabilities.\n2) While RULER's synthetic tasks provide controlled conditions for testing, the absence of a comparative analysis with benchmarks that use realistic data might limit understanding of how these findings translate to real-world applications.\n3) It is recommended to supplement the analysis with details on the usage of graphics memory and inference speed for different contexts and models to provide a more comprehensive evaluation of model efficiency.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H5OA0sHXQQ", "reviewer": "Reviewer_4oYN", "summary": "This paper provides a new synthetic long-context LLM benchmark dubbed RULER. Compared to previous benchmarks, diverse newly designed tasks have been applied with flexible context lengths, characterized by different difficulty levels. Extensive experiments have been conducted on several long-context LLMs under different task settings.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 4oYN for their thoughtful comments.\n\nThe reviewer raises concern on the template used for all models. We employ the same template to enforce a controlled evaluation setup for all models. While the diversity of the prompts can be a valid concern for evaluating large language models, the robustness to various prompts is an orthogonal research direction which we believe to be out-of-scope of our paper. However, we agree with the reviewer that evaluating models on additional prompts can further strengthen our results. \n\nTo answer the questions raised by the reviewer:\n1. In the current version of RULER, we purposefully select the synthetic tasks on which the majority of the models perform decently with short input length (<4k) so that the major differences we observe are due to the scaling of context size. We did not include harder tasks in RULER (e.g., those pressure testing Yi-34B in the analysis section) as those tasks will unfairly put some models in disadvantage even at short sequences.\n2. As discussed in section 5, the reviewer is correct that the number of distractors indeed also impacts the complexity of a task. Within a constrained domain as in RULER, we believe the task complexity is a function of at least two factors: the number of target output tokens and the signal-to-noise ratio in the context. The number of target output tokens can be changed by changing the number of returned needles, the number of hops in variable tracking, and the number of top-frequent words to be returned in CWE/FWE. The signal-to-noise ratio can be controlled by the number of distractors in NIAH and variable tracking, the frequency of common words in the CWE task and the hyperparameter alpha in the FWE task. We plan to add more detailed relevant discussion in the next version.\n3. In our preliminary experiments, we observe very small variances evaluating the same model on different synthetic vocabularies since we fix the size of the synthetic vocabulary and the length of each synthetic word. While these two hyperparameters can also be varied, we keep them fixed in our experiments to have a controlled setup. It can be interesting to investigate how these two hyperparameters impact model performance. We thank the reviewer for bringing this up and will add relevant discussion in our revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.  A new benchmark with more difficult tasks has been provided. The benchmark could better distinguish the long-context capability of LLMs compared to existing benchmarks,  which might contribute to future long-context research.\n2. Extensive experiments have been conducted, with deep analysis with demonstration of RULER's capability.\n3. Insightful findings on the long-context ability of LLMs might be helpful for future researchers.", "reasons_to_reject": "1. The authors could strictly control variables while synthesizing the dataset. For example, the task of Retrieval, where keys and values are put in the context might be also influential to the performance of LLMs. \n2. The synthetic datasets for one task all have the same template, which might not be diverse enough.", "questions_to_authors": "1. How can we make sure the synthetic datasets do not have more preference for certain LLMs, regardless of their long-context learning ability?\n2. How to define the complexity of a task? It seems to be more complicated than simply defining it as the length of the context, some other factors might contribute such as distraction level. How to control this factor?\n3. For the aggregation task, does LLM on different vocabularies have different performances?", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a novel synthetic benchmark called RULER for evaluating the long-context processing abilities of LLMs. Unlike the "needle in a haystack" benchmark, which is too easy, RULER consists of more comprehensive and challenging tasks, including multi-hop tracing and aggregation. The findings are straightforward: most LLMs are not fully functional at their nominal context size, but a few perform well at a context size of 32k. Even though some models support larger context sizes, they struggle with handling those large inputs.

The reviewers found this work:

- Provides extensive and thorough evaluations of LLMs on diverse tasks.
- Offers detailed analysis.
- Is well written.
- Proposes a benchmark and findings that will be helpful for LLM researchers and practitioners.

Overall, we recommend acceptance.
Decision: Accept