YAML Metadata Warning:empty or missing yaml metadata in repo card

Check out the documentation for more information.

BeyondRetrievalBench

Testing Functional Memory and Memory Updating in Long-Context LLMs

Abstract

As the context length supported by large language models (LLMs) continues to increase, their memory capacity has correspondingly improved. LLMs with strong long-term memory are able to effectively store and utilize information from distant and extremely long-range inputs. However, existing benchmarks primarily evaluate retrieval-based memory — such as locating and reproducing specific facts from context — rather than functional memory, which refers to memory that abstracts from surface information into representations useful for downstream tasks.

Human memory, in contrast, is inherently imprecise and highly compressed. We argue that the most valuable form of functional memory captures the most generalizable aspects of experience, which are often the underlying rules governing observed phenomena. Moreover, such functional memory must support continuous composition and updating over streams of incoming information, corresponding to a model's capacity for continual learning.

To study this, we introduce a new benchmark designed to test three complementary capabilities:

Memory Retention — the ability to preserve rule-relevant information over long contexts;
Memory Composition — the ability to integrate distributed evidence across multiple context segments;
Memory Revision — the ability to update previously learned rules when new information contradicts prior observations.

Our benchmark is constructed using procedurally generated rule-based tasks across multiple domains, including numerical transformations, string operations, and set manipulations. Each task requires models to infer deterministic rules from examples and apply them to novel inputs, making shallow pattern matching insufficient. Empirical evaluation reveals that even state-of-the-art long-context LLMs struggle with multi-stage composition and especially rule revision. These results highlight a critical gap between current long-context capabilities and true executable memory, and suggest new directions for model architectures and training paradigms.

Introduction

Long-context large language models have rapidly expanded the amount of information that can be supplied to a model at inference time. Context windows that once contained only a few thousand tokens now extend to hundreds of thousands or even millions of tokens. This shift has changed how LLMs are used: rather than relying only on parametric knowledge stored in model weights, modern systems increasingly operate by conditioning on long documents, codebases, conversation histories, structured logs, tool outputs, and other external context streams.

The dominant interpretation of long-context ability has been retrieval-centric. A model is considered to have strong long-context memory if it can locate information placed far back in the prompt and reproduce or use that information when queried. This view has motivated a family of "needle-in-a-haystack" style evaluations, where a fact or passage is embedded inside a long distractor context and the model must recover it. Such evaluations are important: they test whether attention mechanisms, positional encodings, inference systems, and context management strategies allow information to remain accessible over long distances.

However, retrieval is only one form of memory. In many real scenarios, the most useful information is not a surface fact that can be copied from the context, but a latent structure that must be inferred from multiple observations. A scientist may remember not every individual measurement, but the law that best explains them. A programmer may not remember each line of a codebase, but the design invariant that governs later changes. A user may not expect an assistant to recite every previous example, but to infer a preference, policy, or rule and apply it correctly in future cases. In these settings, memory is functional: it compresses observed experience into a representation that supports downstream behavior.

We define functional memory as memory that stores and uses abstracted, task-relevant structure rather than merely preserving surface tokens. In this paper, we focus on a particularly controlled and verifiable form of functional memory: deterministic rule induction from examples. A model is given examples of an unknown operation, followed by a long distractor context, and is later asked to apply the operation to a novel input. Solving the task requires inferring the underlying rule from the examples, retaining that inferred rule across a long context, and executing it correctly at test time.

This setup exposes limitations that are not captured by standard retrieval benchmarks. A model can succeed at retrieving an exact sentence while failing to infer the rule implied by a set of examples. Conversely, a model may infer a rule locally but fail to preserve it after a long distractor. More importantly, realistic memory is not static: new observations arrive continuously and may need to be composed with or used to revise previous beliefs. A benchmark for functional long-context memory should therefore test not only retention, but also composition and updating.

Comparison with Existing Benchmarks

Benchmark	Exact text match	Multi-hop reasoning	Max hops	Max length	Real-world task	Data leakage risk	Rule induction	Forgetting / Updating
NIAH	Yes	No	/	~128K	No	Yes	No	No
RULER	Yes	No	/	≤128K	No	Yes	No	No
MRCR	Partial	No (multi-needle retrieval)	/	≥1M	Partial	Yes	No	No
LongBench	Partial	Partial	≤5	≤32K	Yes	Yes	Partial (few-shot, not primary)	No
LongBench v2	Partial	Partial	≤10	≤2M	Yes	Yes	Partial (many-shot, not primary)	No
L-Eval / Ada-LEval	Partial	Partial	≤16	≤200K	Yes	Yes	Partial (GSM 16-shot, not primary)	No
BeyondRetrievalBench	No	Yes	variable	variable	No	No	Yes (primary)	Yes (primary)

Benchmark Design

Functional Memory vs. Retrieval Memory

We distinguish functional memory from retrieval memory by whether the model must preserve surface content or inferred structure. Retrieval memory can be evaluated by asking whether a model can recover a fact that appears verbatim or near-verbatim in the context. Functional memory requires a model to infer a transformation, compress the examples into an abstract representation, and execute that representation on new inputs.

For example:

Retrieval task: "The textbook states: 1 + 1 = 2. [long distractor...] Q: What is 1 + 1?"

Functional memory task: "The textbook states: f(1, 2) = 3; f(1, 4) = 5; f(2, 2) = 4; [M more examples...] [long distractor...] Q: What is f(1, 1)?" (the answer 2 never appears in the examples)

Task 1: Memory Retention

Memory Retention evaluates whether a model can infer a rule from examples near the beginning of the context and apply it after a long distractor. Each instance contains one operator and a set of demonstrations. The final query appears after a distractor of controlled length.

Structure:

[Instruction prefix]
[Examples of operator f]
[Long distractor text]
[Question: apply f to a new input]

Formally, let $f$ be a hidden deterministic operator sampled from a domain-specific rule generator. The context contains demonstrations:

$\left(x_i,\ f(x_i)\right), \quad i = 1, \dots, M$

After a distractor sequence $D$, the model receives a query input $x^*$ and must output $f(x^*)$.

Key variables: number of examples $M$, context length.

Task 2: Memory Composition

Memory Composition evaluates whether a model can integrate multiple pieces of rule-relevant information distributed across the context. A typical instance contains multiple operators $f_1, \dots, f_K$, each introduced in a separate segment with its own examples. The final query requires applying a sequence or composition of these operators.

For example, the final answer may require:

$y = f_K(f_{K-1}(\cdots f_1(x) \cdots))$

or, in the multi-input case:

$y = f_3(f_1(x_1),\ f_2(x_2))$

Structure:

[Instruction prefix]
[Examples of operator f1]  [Distractor]
[Examples of operator f2]  [Distractor]
...
[Examples of operator fK]  [Distractor]
[Question: apply a composition or sequence of operators]

Key variables: number of operators $K$, context length.

Task 3: Memory Revision

Memory Revision evaluates whether a model can update a previously inferred rule when later examples contradict or modify it. Each instance contains an initial rule $f^{\text{old}}$, followed later by examples of an updated rule. The final query asks for the current rule's output, so the correct answer is $f^{\text{new}}(x^*)$, not $f^{\text{old}}(x^*)$.

Structure:

[Instruction prefix]
[Initial examples of operator f_old]
[Distractor]
[Later examples indicating an updated operator f_new]
[Distractor]
[Question: apply the current version of f]

Formally:

$( x_i,\ f^{\text{old}}(x_i) ) \quad \text{for } i = 1, \dots, M_{\text{old}}$

followed later by:

$( z_j,\ f^{\text{new}}(z_j) ) \quad \text{for } j = 1, \dots, M_{\text{new}}$

Key variables: number of initial examples $M_{\text{old}}$, number of update examples $M_{\text{new}}$, edit distance between $f^{\text{old}}$ and $f^{\text{new}}$.

Domains

Each task is instantiated across four symbolic domains:

Numerical Patterns Operators are defined over integers, including arithmetic combinations, modular operations, min/max, digit-level transformations, or piecewise expressions. Example:

f(x1, x2) = 2 * x1 + 3 * x2
→ "Input numbers 4 and 7 produce 29 under operation f."

String Transformations Operators are defined over symbolic strings: reversals, rotations, prefix/suffix extraction, concatenation, wrapping, repetition, etc. Example:

f(x) = concatenate(reverse(x), first_character(x))

Set Operations Operators are defined over finite sets: union, intersection, difference, symmetric difference, filtering, element-wise relabeling, etc. Example:

f(A, B) = sorted((A ∪ B) \ (A ∩ B))

File System Agent Operators describe file-system manipulation rules applied to directory trees, testing whether models can infer and execute agent-style procedures. Example:

f: When applied to a folder, move all files whose names contain "draft" into review/,
   and rename them by replacing "_draft" with "_needs_review".

Controls

Each task is evaluated under three conditions:

Condition	Description	Prompt structure
All	Standard evaluation with examples and long distractor	`[prefix] [examples] [distractor] [question]`
Zero-Example	No valid demonstrations provided; measures prior knowledge exploitation	`[prefix] [distractor] [question]`
Zero-Distance	Examples placed immediately before the query; isolates degradation due to memory distance	`[prefix] [examples] [question]`

Experiment Setup

Dataset Summary

Task	Domain	Number of Cases
MemoryRet	NumPat / StrTran / SetOps / FileSysAgent	90 each
MemoryCom	NumPat / StrTran / SetOps / FileSysAgent	450 each
MemoryRev	NumPat / StrTran / SetOps / FileSysAgent	1080 each

Key Conditions

language = "chinese"   # or "english"
task_types = ["MemoryRet", "MemoryCom", "MemoryRev"]
mode_lists = ["zero_example", "zero_distance", "all"]
domain_lists = ["NumPat", "StrTrans", "SetOps"]

num_cases = 10
task_type_setting = {
    "MemoryRet": {"K_s": [1],           "M_s": [70], "edit_Ks": [None]},
    "MemoryCom": {"K_s": [5,10,20,40,60], "M_s": [70], "edit_Ks": [None]},
    "MemoryRev": {"K_s": [5,10,15],     "M_s": [70], "edit_Ks": [1,3,5,10,15]},
}

context_len_s = [32, 64, 128, 200, 400, 700, 1000]  # in units of ~100 tokens

Getting Started

Prerequisites

Python 3.9+
Access to an LLM API (for data generation and evaluation)
Configure model endpoints and API keys as needed in each script

Repository Structure

BeyondRetrievalBench/
├── config.py                      # Central configuration file
├── gen_prompts_chinese.json       # Prompts for generating Chinese-language data
├── gen_prompts_english.json       # Prompts for generating English-language data
│
├── 1_gen_domain_rules.py          # Step 1: Generate operators
├── 2_gen_exps_of_rules.py         # Step 2: Generate examples for each operator
├── 3_fix_error_exps.py            # Step 3: LLM-based error correction for examples
├── 4_gen_distractors.py           # Step 4: Generate distractor text
├── 5_gen_question_answer.py       # Step 5: Generate question–answer pairs
├── 6_chi_eng_trans.py             # Step 6: Translate Chinese data to English (optional)
│
├── 7_eval_model_MemoryRet.py      # Evaluate models on Memory Retention
├── 7_eval_model_MemoryCom.py      # Evaluate models on Memory Composition
├── 7_eval_model_MemoryRev.py      # Evaluate models on Memory Revision
├── 8_checked_is_correct_run_after_7.py  # Extended answer checking (regex + LLM judge)
├── 9_visualize_results.ipynb      # Result visualization
│
├── seed_dataset/                  # Pre-generated seed data
└── eval_dataset/                  # Evaluation dataset

Step-by-Step Guide

1. Configure Parameters

Edit config.py to set the key parameters for data generation and evaluation:

language: "chinese" or "english"
domain: one of NumPat, StrTrans, SetOps, FileSysAgent
number of operators, examples per operator, distractor lengths, etc.
task types and control modes for evaluation

2. Data Generation

Run the following scripts in order:

# Step 1: Generate operators (rules)
python 1_gen_domain_rules.py

# Step 2: Generate examples for each operator
python 2_gen_exps_of_rules.py

# Step 3: Use an LLM to verify and correct generated examples
python 3_fix_error_exps.py

# Step 4: Generate distractor text
python 4_gen_distractors.py

# Step 5: Generate question–answer pairs (num_cases set in config.py)
python 5_gen_question_answer.py

Tip: Steps 2, 3, and 5 involve LLM calls. We recommend using a strong model (e.g., Opus4.6，Fable5) for better example and answer quality. Each script allows you to specify the model directly inside the file.

For English data, you have two options:

Option A (direct): Use gen_prompts_english.json directly in each script (requires completing the English prompt templates).
Option B (translate, recommended): Generate Chinese data with gen_prompts_chinese.json, then translate using:

python 6_chi_eng_trans.py

This approach guarantees consistency between Chinese and English versions and allows cross-lingual comparison.

The data generated after running steps 1–6 will be saved in ./seed_dataset. Pre-generated data is also available on HuggingFace: https://huggingface.co/alych9/BeyondRetrievalBench. And the multi-options version to support evaluating pretrained models: https://huggingface.co/alych9/BeyondRetrievalBench_Pretrain

hf download alych9/BeyondRetrievalBench \
    --repo-type model \
    --include "seed_dataset/*" \
    --local-dir ""

3. Evaluation

Run the evaluation scripts for each task type. Each script supports sample-level checkpoint resumption — if a run is interrupted, simply re-run the script and it will skip already-evaluated samples and continue from where it stopped.

# Evaluate Memory Retention
python 7_eval_model_MemoryRet.py

# Evaluate Memory Composition
python 7_eval_model_MemoryCom.py

# Evaluate Memory Revision
python 7_eval_model_MemoryRev.py

Set the model(s) to evaluate directly inside each script.

After running the evaluation scripts, results will be saved to two directories: ./eval_dataset and ./dialogs. ./eval_dataset contains the full model inputs for each sample. ./dialogs stores the ground-truth answers, the model's responses, and the correctness scores.

Pre-evaluated results for several models are also available for download from HuggingFace (same as seed_dataset): https://huggingface.co/alych9/BeyondRetrievalBench

4. Post-processing and Answer Verification

After all 7_*.py scripts have finished, run the extended answer checker:

python 8_checked_is_correct_run_after_7.py

This script applies additional regex-based matching and calls an LLM judge for cases where regex matching is insufficient. Run this after all evaluation is complete.

5. Visualize Results

Open and run 9_visualize_results.ipynb in Jupyter after step 4 is complete. The notebook generates plots across task types, domains, context lengths, and control conditions.

Citation

If you use BeyondRetrievalBench in your research, please cite:

@article{beyondretrievalbench2025,
  title     = {Beyond Retrieval Bench: Testing Functional Memory and Memory Updating in Long-Context LLMs},
  year      = {2025},
}

License

See LICENSE for details.

Downloads last month: -; Downloads are not tracked for this model. How to track

Inference Providers NEW

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support