Minerva: A Programmable Memory Test Benchmark for Language Models
Authors: Menglin Xia
Victor Rühle
Saravan Rajmohan
Reza Shokri
Venue: ICML 2025 poster (2025)
Keywords: LLM evaluation
LLM capability
context utilization
memory benchmark
Primary area: general_machine_learning->evaluation

Abstract: How effectively can LLM-based AI assistants utilize their memory (context) to perform various tasks? Traditional data benchmarks, which are often manually crafted, suffer from several limitations: they are static, susceptible to overfitting, difficult to interpret, and lack actionable insights--failing to pinpoint the specific capabilities a model lacks when it does not pass a test. In this paper, we present a framework for automatically generating a comprehensive set of tests to evaluate models' abilities to use their memory effectively. Our framework extends the range of capability tests beyond the commonly explored (passkey, key-value, needle in the haystack) search, a dominant focus in the literature. Specifically, we evaluate models on atomic tasks such as searching, recalling, editing, matching, comparing information in context memory, performing basic operations when inputs are structured into distinct blocks, and maintaining state while operating on memory, simulating real-world data. Additionally, we design composite tests to investigate the models' ability to perform more complex, integrated tasks. Our benchmark enables an interpretable, detailed assessment of memory capabilities of LLMs.

Minerva: A Programmable Memory Test Benchmark for Language Models
Menglin Xia
Victor Rühle!
Saravan Rajmohan' Reza Shokri
Abstract
paper, we address fundamental question that is critical to improving AI assistants: What specific capabiliiies do large language models demonstrate in utilizing their memory?
How effectively can LLM-based Al assistants uti- lize their memory (context) to perform various tasks? Traditional data benchmarks, which are often manually crafted, suffer from several limi- tations: they are static, susceptible to overfitting, difficult to interpret, and lack actionable insights- failing to pinpoint the specific capabilities model lacks when it does not pass test. In this paper. we present a framework for automatically generat- ing a comprehensive set of tests to evaluate mod- els' abilities to use their memory effectively. Our framework extends the range of capability tests beyond the commonly explored (passkey, key- value, needle in the haystack) search, a dominant focus in the literature. Specifically, we evaluate models atomic tasks such as searching, recall- ing, editing, matching, comparing information in context memory, performing basic operations when inputs are structured into distinct blocks,
One common approach to testing models is through data benchmarks. However, evaluating model capabilities using static data benchmarks-based on some user queries, their data, and expected outcomes- be costly, imprecise, and lacks scalability. Additionally, performing tasks in realistic scenarios often requires multiple capabílities, making it challenging to identify which specific capability model lacks when it fails data benchmark. This limitation reduces the effectiveness of these benchmarks in designing better models. Moreover, blindly optimizing models to improve on such benchmarks risks overfitting. often rendering the data benchmarks obsolete over time.
To address some of these concerns, recently, there have been many attempts to test models using automatically generated benchmarks, but these efforts have primarily focused on evaluating basic search capabilities (e.g., passkey or key- value search) in long-contexts (Kamradt, 2023; Liu et al., 2024; Zhang et al., 2024; Anthropic, 2024; Wu et al., 2024; Li et al., 2024; Hsieh et al., 2024). In this paper, we go beyond simple search tasks and introduce framework to test a comprehensive range of memory-related capabilities in LLMs. Note that, we deliberately avoid conflating the memory usage capabilities with the complex reasoning abil- ities (e.g., complex mathematical or logical reasoning) in language models, as the latter is a separate skill that is cur- rently the focus of extensive study (Clark et al., 2018; Cobbe et al., 2021; Hendrycks et al., 2021b; Suzgun et al., 2022).
and maintaining state while operating on mem- ory. simulating real-world data. Additionally, we design composite tests to investigate the models' ability to perform more complex, integrated tasks. Our benchmark enables an interpretable, detailed assessment of memory capabilities of L.LMs.
1. Introduction
What capabilities should we expect from AI assistants? The Al assistants are provided with a (large) input context con- taining all the available information that is potentially rele- vant to the user's request (e.g.. all the prior emails, messages, and confirmed calendar events). This input, commonly re- ferred as the context (for the LLM), encapsulates what the AI assistant knows about the world in which it is tasked to operate. This representation of the world, expressed in natural language, functions as the model's memory. In this
What are memory- ussge capabilities? We define these as the abilities to retrieve relevant information, compose it for the instructed task, and recall key details when synthesizing the output. This process also involves creating associations between the instruction and the stored information, as well as among different parts of the memory itself. Without ex- tracting these relationships, the memory remains flat and formless, rendering it less useful. Consequently, the model must be able to recognize differences, identify similarities, and take appropriate actions based on them. We design a series of atomic tests aimed at evaluating each of these indi- vidual capabilities in isolation, to the extent that isolating such capabilities is possible.
Equal contribution 'M365 Research, Microsoft NNational Uni- versity of Singapore. Correspondence to: Menglin Xia <mol- lyxia@ microsoft.com>> Reza Shokri <reza @ www@@coommmmmm.o nus. sdu.s
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Minerva: A Programmable Memory Test Benchmark for Language Models
To evaluate the more complex scenarios in memory usage, we construct composite tests that reflect real- scenar- ios, where memory is divided into multiple compartments (i.e., information relevant to distinct contexts). The model is expected to recognize the boundaries of these compartments, trace the knowledge contained within them, and perform operations such as information retrieval, memory associ- ation, and other tasks while respecting these boundaries. The complexity increases further when there is interaction between compartments. In such cases, information must flow across boundaries- for example, when stories about two parallel events converge at particular moment, when inter- actions occur between different events, or when information known to certain entities in memory is shared with others. Examples include AI assistants managing calendar events, tracking financial transactions, or suggesting medical di- agnoses. Handling these scenarios is highly challenging, yet it is essential for AI assistants to achieve practical and reliable performance. The core challenge is due to the fact that context memory, as provided to the model, flattens data from multiple parallel and potentially interrelated memory compartments. This requires the model to disentangle the content by leveraging the available labels and clues, and tracking the state of relevant information throughout the memory scanning, while performing the task.
this memory and execute potentially complex retrieval tasks. Accordingly, we expect the Al assistant to demonstrate spe- cific capabilities in memory utilization, including accurate retrieval, effective synthesis of relevant information, and adaptability to evolving context. These capabilities include:
Information Retrieval and Localization: The ability to efficiently locate, search for, and extract relevant information from the memory (i.e., input data) based on specific instructions or user queries.
The capability to perform modifications, computations, and logical oper- ations on the imput data, including identifying pattems, recognizing repetitions, and understanding relation- ships within the memory.
Processing and Basic Reasoning:
Content Transfer and Synthesis: The ability to copy, rephrase, or generate synthesized output by integrating both original and modified elements from the input.
Structural Awareness and Organization: The capac- ity to interpret the spatial, structural, or organizational layout of the memory, such as distinguishing labeled fragments of text, sets, lists, or hierarchical structures.
In our benchmark, we focus on isolating the atomic memory- related capabilities of LLMs. Success or failure in these tests provides clear and interpretable assessment of the strengths and limitations of the models. We design mul- tiple atomic tests to measure fundamental skills without interference from other factors. These tests are simple, tar- geted, and structured to assess specific abilities with elarity. Our benchmark includes the existing basic tests, notably the tests and its variations, but it goes beyond the search methods and includes a diverse set of atomic capabilities. In our benchmark, we focus on some fundamental capabilities: search, recall and edit, match and compare, spot the differences, compute on sets and lists, and stateful processing.
Table 1 presents an overview of the types of memory tests included in our benchmark. For each test, we use ef- ficient parametric programs to generate randomized test cases. We run a comprehensive evaluation of several major open-source and black-box models (e.g., GPT-4(o), Cohere, Gemma, LLaMA, Mistral, Phi). Our experimental results show that while models perform relatively well on simple search tasks, they exhibit significant disparities across con- text utilization capabilities even at a context length of 4k tokens. This indicates that strong performance in basic retrieval does not necessarily translate to other context pro- cessing abilities. Our framework goes beyond search-based tests by incorporating atomic tests that pinpoint distinct ca- pabilities, providing more nuanced picture of the strengths and weaknesses of models in context processing. Moreover, composite tests, which combine multiple atomic capabili- ties, resulted in substantial performance drops for all models. These tests present the limitations of current models and provide valuable insights for guiding future model training and development.
We also develop composite tests to evaluate how effectively models can perform more complex, integrated tasks. These tests assess the integration of multiple atomic capabilities to simulate real-world scenarios. By combining elements such as retrieval, reasoning, synthesis, and structural aware- ness, composite tests measure how well an Al assistant can coordinate different skills to execute complex operations. We provide two class of composite tests: processing data blocks, and composite-state tracking (theory of mind). Our objective here is to test the composition of various atomic operations and the ability to interpret segments of data (e.g.. messages or paragraphs associated with a particular person or topic). These composite tests evaluate if the model can sistant leveraging the LLM can instruct the model to parse
The code and data will be available at //gittp::://tttt com/ wwwoooottiiieeeeeeeeeereeee
2. Benchmark for Memory Tests
We define the entirety of context data available to a large language model (L..M) as its memory. Users of an AI as-
make sense of the "spatial" structure of the memory, and also keep track of its "temporal" changes (e-g., information
2

---

Minerva: A Programmable Memory Test Benchmark for Language Models
that gets updated across many emails). This can become particularly challenging for the current architecture of major LLMs because the context has a flat structure.
models show stronger all-around performance across the capabilities. In contrast, other models excel at the search task but struggle significantly in other areas, leading to a widening performance gap compared to stronger models. This is especially evident in the Stateful Processing tasks, where models exhibit steep performance drops. Even within the GPT-4(o) models, there were noticeable variations in performance across different tasks, despite them being the best- pest-performing models. This suggests that strong perfor- mance in simple retrieval tasks does not imply effective context processing, highlighting that using NIAH-like tests alone for evaluating context utilization is not sufficient to capture the full spectrum of model capabilities. Our frame- work instead reveals significant variability in performance across distinct capability categories, offering a more nu- anced understanding of model limitations.
Table presents the list and description of representative tests in each category. Appendix A presents the exact tem- plates for all our tests. Our benchmark differs from tradi- tional data benchmarks by allowing the generation of fresh, randomized test cases for each category. Each test acts as programmable script that measures the model's capabil- ity while adjusting the hyperparameters that influence test difficulty. The programmable tests also enable us compos- ing them easily, which is one of the key advantages of our benchmark. New categories, and new tests, can be easily added to this framework enabling a more diverse set of tests.
3. Evaluation
The following sections analyze each test type in detail, high- lighting key insights from the evaluations.
3.1. Experimental Setup
We use the proposed framework to evaluate nine widely used language models on a fixed snapshot of I 1110 randomly gen- erated test samples. For all tests, we fixed the context length to 4k tokens, except in the Stateful Processing category, where the context length depends on the number of opera- tion steps. We set the number of steps as 200 for quantity state and 100 for set state, corresponding an approximate context length of 1.5k tokens. For evaluation, we use ex- act match accuracy for binary tasks, ROUGE-L(Lin, 2004) for tests that require sequence overlap measurement, and Jaccard similarity (Jaccard, 1901) for set overlap. Further details on the number of examples, hyperparameter configu- rations, and evaluation metrics for the tests are provided in Appendices B and C.
3.3. Analysis on Atomic Tests
Search All models performed relatively well on Search tasks, which is unsurprising given the 4k context length. However, even at this length, model performance varied sig- nificantly depending on the specific search type (see Table 2). For example, in the binary String Search task, models handled individual word searches well but struggled with subsequence searches, where queries consisted of multi- word sequences. The performance drop can be attributed to two factors: (1) length of query affects the difficulty of precise memory access; (2) negative samples are created by replacing a single word in present subsequences, making absent longer subsequence more distracting.
The evaluated models are divided into two groups:
Figure 2 further analyzes subsequence search performance for GPT-4o, Mistral, and Phi-3-medium. These models ex- hibit distinct error patterns as the length of the subsequence increases: GPT-4o has no false negative errors (it never misses a present subsequence) but makes more false posi- tive errors as the subsequence length grows, suggesting it overestimates presence in more ambiguous cases. Mistral also makes no false negative errors but exhibits decreasing false positive rate, implying it struggles more with shorter distractors. Phi-3-medium, in contrast, makes few false pos- itive eтrors (rarely identifies an absent sequence as present), but struggles more with false negatives, indicating a general tendency to deny presence. These differing patterns sug- gest that the models may employ different search strategies, affecting their susceptibility to different types of errors.
Black-box models: GPT-4-turbo, GPT-4o, GPT-44-mini, and Cohere-command--plls.
Open-source models: dostral-b-instruct-ie Phi-3-small- 1 128k-instruct (7B), I.LAMA-3. -8b-instrrct, Gemma-2-9b, and Phi-3-medium-11 128k-instrutt -ndn----iit (14B).
We set the max output token to 4096, temperature to 0, and top-p to I for all model inference.
3.2. Model Performance Overview
Figure 1 summarizes the overall performance of the eval- uated models on the memory test snapshot within 4k con- text length. Notably, this context length is usually consid- ered short for context utilization benchmarks, and many models are expected to perform perfectly at this length. However, our evaluation reveals significant disparities in performance across the capabilities, even within this man- ageable context length. Overall, the GPT-4turrbo//PT-40
For Batch Search and Key- Value Search tasks (analogous to multi-NIAH and NIAH, respectively), models like Mistral, Phi-3, and Cohere show a notable performance drop, reveal- ing their limitations in handling multiple memory accesses effectively.
3

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Search
String search (binary test)
Is the string x in memory? a could be a word or sequence. What is the word or phrase that is paired with keyword x? For each of the keywords in the batch Ti 42, ... Fh perform the search and return the batch of corresponding responses.
Key-value search
Batch search
Recall and Edit
Snapshot
Share a snapshot of the entire memory (as a verbatim copy). Share the entire memory after replacing all occurrences of a with y. Share the entire memory after overwriting the words that are on particular positions (e.g., every hthh word) with y.
Replace all
Overwrite positions
Functional updates
Update every x with the output of a function (x)
Match and Compare
Compare positions (binary test)
Does x appear before y?
Find duplicates
Which word/string has duplicates in memory?
Count
How many times repeated in memory?
Check association (binary test)
Check if both æ and y are associated with the same tag in memory
(assuming every term associated with tag).
Spot the Differences
Compare two lists
Give two lists X and Y of the same length (e.g., same number of words), report the difference (i.e., X ----- -Y.
Identify the odd group
Given multiple sets, identifyy which one is different (assuming n identical sets with shuffled elements, and set with some differences).
Patch the difference
A sequence of words is repeated multiple times, and then there is a partial sequence. What is the next kt element in the sequence?
Compute on Sets and Lists
Group membership
Given sets, identify which set includes x.
Group association
Check if x and belong to the same set. Given k lists, return the last element in each list.
Iterate
Stateful Processing
Quantity
Keep track of the total quantity of items, based on a sequence of addition and subtraction operations (e.g.. "add 10, subtract 2, add 7.."".
Set
Keep track of the items in a set, based on a sequence of addition and removal operation (e.g., "add apple, pear; add orange; remove apple; add lime,
Processing Data Blocks Search, recall, and edit
The input contains alternating labeled lists of elements (e.g., "LI: a, c; L2: h, f, i; Ll: d, z, k; ..."). For given list label (e.g.. L1 or L2) and specified element within that list, return all the elements that appear after that specified element in the same list.
Somposite-State Tracking (Theory of Mind)
State tracking across data blocks
Perform "Stateful Processing" for multiple agents, and report the final set state for each agent. The input provides a list of operations by k agents over time, including both independent actions (add/remove) and interactive actions (swap) (e.g.. "Alice: add apple, pear, remove orange, add banana; Bob: add peach, berry, remove kiwi; Charley: add lime: Bob: remove peach, swap berry with Alice for banana;...")
Table 1. List of memory tests. We divide the tests into different categories based on the core expected capability for passing the test. Most the initial tests are atomic, i.c., the expected capability cannot be broken down into other meaningful capabilities. The tests at the bottom of the list are composite tests and require the model to have multiple atomic capabilities at the same time in order succeed.
Recall and Edit Figure 3 presents the results for the Re- call and Edit tasks. While models performed well on ba-
sie recall (Snapshot), their performance dropped sharply when tasked with making regular edits. A closer analysis

---

Minerva: A Programmable Memory Test Benchmark for Language Models
aut-4-turbo
got-4o
got-4o-mini
cohere
mistral- phi-3-small
phi-3-medium
gemmma-2-9
Search
Search
Recall Edit
Stateful Processing
Rocall and Gdit
Statéful Processing
Match and Comoaae
Computy Sets nad Lists
Match and Compare
Compute Sets nad Lists
Spot Differences
Spot the Differences
Performance of black-box models.
(b) Performance of the open-source models.
Figure I. Overall performance of nine models on snapshot within 4k context length of Minerva.
Models
Word
Subsequence
Key-value
Batch
gpt-4o (pos) gpt-4o (nog) Un mistral-7b (pos)
mistral 7h (neg) phi medium (pos) (8 phi-3 meedium (neg)
gpt-I-urbo
0.94
0.94 (-0.00)
1.00
1.00 (-0.00)
gpt-du
1.00
0.82 (80.88)
1.00
1,00 1.00 (-0.00)
gpt
0.64 (0344)
1.00
(:0.04)
coher-coomman--pla
1.00
0.85 (50.15)
0.98
0.87 (-0.11)
0.8
mistral-7b
0.78
0.80 (+0.02)
0.92
0,47 (-0.45)
phi-3-small
0.94
0.84 (:0.10)
0.94
0.77 (-0.17)
0.6
phi-3-medium
1.00
(-0.45)
1.00
0,72 (-0.28)
gemma-2-96
1.00
0.60 (-0.40)
1.00
1,00 (40.00)
Ilama-3,1-8 1-8b
1.00
0.57 (-0.43)
1.00
0.99 (-0.01)
Table 2. Results for the Search tasks. The four columns represent: String Search (with word), String Search(with subsequence), Key- value Search, and Batch Scarch. Numbers in parentheses indicate comparative performance differences between String Search (with subsequence word) and Batch Search vs. Key-Value Search.
8
16
32
64
Figure 2. Analysis String Search (with subsequence) across increasing subsequence lengths. This figure examines the behavior of models positive samples (where the subsequence is present) and negative samples (where the subsequence is absent).
Model
String Search (word) Snapshot
gpt-4-turba
1.00 (0.06)
1.00 (0.04)
gpt-4o
1.00 (0:00)
1.00 (0.00)
gpt-4a-mini
0.94 (-0.04)
1.00 (0.00)
that these models frequently deviated from instructions over longer sequences, suggesting difficulties maintaining con- sistent rule applications over extended contexts.
cohere
1.00 (0.00)
1,00 (0.26)
mistral-7b
1.00 (0.22)
0,96 (0.00)
phi-3-small
1.00 (0.06)
0,99 (0.04)
phi-3-medium
0.98 (-0.02)
0.87 (-0.09)
Additionally, conducted separate ablation study Snapshot and String Search. In this study, we replaced meaningful words in the context with gibberish tokens con- sisting of randomly generated alphabetical characters. As shown in Table 3, performance remained largely unchanged, suggesting that semantic meaning was not significant dis- tractor in these tasks.
gemma-2-9b
0.96 (-0.04)
0.96 (0.05)
1lama-3.1-8h
0.98 (-0.02)
1.00 (0.00)
Table 3. Ablation study with gibberish context.
of the generated outputs reveals that models struggled with maintaining coherence during edits, often getting trapped in repetitive word loops. For the Functional Update task, we deliberately selected simple numerical updates, such "Subtract from every number," to ensure the edits were within the models' capabilities. Nevertheless, when compar- ing performance on Snapshot (with numbers) to Functional Updates, all models exhibited a steep decline, especially for smaller ones. Analysis of generated outputs revealed
Match and Compare As shown in Figure model per- formance in the Match and Compare tasks was relatively consistent across different model sizes. Given that counting is well-known weakness in LLMs, it is unsurprising that all models struggled significantly with the counting task, though GPT models performed slightly better than others. However, models generally succeeded identifying the
5

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Model
Group membership
Group association (Group membership)
Group assoc. (alternating)
Iterate
(Compared against)
(Sub-string search)
(Grrup association)
(Iterate (last))
gpt-4-turbo
0.96 (0.02)
0.75 (-0.21)
0.68 (-0.07)
0.83 (-0.17)
gpt-4o
0.98 (-0.02)
0.65 (-0.33)
0.52 (-0.13)
0,86 (-0.14)
gpt-4o-mini
0.96 (-0.02)
0,68 (-0.28)
0.52 (-0.16)
0,67 (-0.33)
cohere-command-pluss
0.93 (-0.07)
0.70 (-0.23)
0.72 (0.02)
0.10 (-0.9)
mistral-7b
0.50 (-0.28)
0.57 (0.07)
0.52 (-0.05)
0.04 (-0.25)
phi-3-small
0.52 (-0.42)
0.55 (0.03)
0.68 (0.13)
0.04 (-0.7)
phi-3-medium
0.60 (-0.4)
0.72 (0.12)
0.50 (-0.22)
0.04 (-0.69)
gemma-2-96
0.80 (-0.2)
0.60 (-0.2)
0.62 (0.02)
0.14 (-0.65)
11ama-3.1-8b
0.84 (-0.16)
0.78 (-0.06)
0.82 (0.04)
0.05 (-0.43)
Table 4. Results for Compute on Sets and Lists. The numbers in parentheses indicate the performance difference compared to the corresponding tasks they are evalauted against.
Eipe--etzzt Bape-tol Typt- oott
mittabille de placiarall Inphid-anchuss He
reliably identify it. This may stem from a combination of their inherent inclination to agree and the difficulty in recognizing relative comparisons and associations.
08
0U5
00
gpt-4- gpt-4o III got-4o-mini Ia cohere
04
M mistral-7h
D (pii-3--mmall 01 phi-3 I gemmm-2-99h
1ama--3.1 1-88b
02
Stcppbed (oentnn
Replace
Cvereese poolin
Stophett burdect
focctionl optstes
Rophace
Ovenento pootlatt
I Peactien apacces
0.8
0.6
Figure 3. Results for the Recall and Edit tasks.
0.4
4tuto 18ge. do Dgg.oooninnB.otter
lu miani-h Dapte Soedtar 2-gr B1
0.2
0.5
Compare two lists
Identify the odd group
Patch the difference
04-
D.6
0.4
0.4
Figure 5. Results for Spot the Differences tasks.
0.2
0.8
0-
Compans pottere
Veet tettestt
Count
Chock monccation
Conppss Pe5tiAAe
0
19nd dafieess
Conti
Cleck sosecition
Spot the Differences As shown in Figure 5, performance across all models are poor on Compare Two Lists, suggest- ing inherent difficulties cross-referencing information across long contexts, even for larger models. GPT-4o and the LLaMA model significantly outperform the others in the Identify the Odd Group task, highlighting general weakness in detecting contextual differences by the other models. However, an 8B LLaMA model outperforms both equivalently-sized models and even GPT-4 in this task, sug- gesting that model size alone was not the determining fac- tor. This indicates that architectural differences, training objectives, or specific inductive biases may contribute to improved performance in comparative memory utilization.
Figure 4. Results for the Match and Compare tasks.
duplicates (in Find duplicates), and primarily struggled with the counting aspect, which requires tracking and updating an integer state, a skill that is more similar to stateful pro- cessing. This suggests that relying solely on counting- based tests (Song et al., 2024) could overly bias the evaluation and fail to capture broader model capabilities. The results also indicate that models exhibit some ability to recognize relative positions and group associations, but their accuracy remains limited (ranging between 0.6-0.8). A closer amination of model generations reveals an overwhelming tendency for the models to produce false positive errors- models often answer "yes" when the correct answer is "no", while making very few false negative errors. This means that when the relationship is correct, the models can more
Compute on Sets and Lists The tasks in this category require models to recognize and process group structures within the context, and performance gradually declines as the complexity of the task increases (see Table 4). For in- stance, in comparing the Group Membership task with the
6

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Model
Quantity state
Set state
Model
Processing Data Blocks
Theory of Mind
gpt-4-turbo
0.8
0.80
gpt-4-turbo
0.31
0.26
gpt-40
1.0
0.65
gpt-4o
0.37
0.38
gpt-4o-mini
0.7
0.24
gpt-4o-mini
0.31
0.21
cohere
0.0
0.58
cohere
0.26
0.18
mistral-7b
0.0
0.08
mistral-7b
0.18
0.16
phi-3-small
0.0
0.13
phi-3-small
0.21
0.20
phi-3-medium
0.0
0.11
phi-3-medium
0.18
0.10
gemma-2-9b
0.0
0.24
gemma-2-9b
0.26
0.12
18mb
0.0
0.13
1lama-3.1-88
0.15
0.03
Table 5. Results for Stateful Processing tasks.
Table 6. Results for the composite tests.
spe-4ituboo
colore
77
21e do
phi 3-small
2pr-4-thho oohere
7o
small
models struggle (near zero accuracy). For set state tracking, larger models generally perform better.
0.8
0.6
0.6
We conducted an ablation study to examine how the number of operation steps influences performance of five selected models (Fig. 6). For quantity state tracking, GPT-4(o) mod- els perform well within fewer than 200 steps but experience sharp decline in accuracy beyond this threshold. For set state tracking, the performance decline is more gradual. The differences in performance drop between the two tasks can be attributed to the nature of the two tasks. While tracking an integer state might seem simpler than tracking a set, it actually requires the model to maintain and apply every op- cration sequentially to compute the final value. In contrast, for set state, the fixed size of the set makes more recent operations more relevant to the final state, reducing the need for exhaustive step-by-step tracking. Nevertheless, even in this scenario, all models show a clear inability to handle longer or more complex operation sequences effectively. Interestingly, GPT-4 model outperformed GPT-4o at this task, suggesting potential optimization trade-offs may have affected its ability to manage set-based updates.
0.4
0.4
0.2
0:2
50200 100
800
1,200
1,600
50200 400
800
1,200
1,600
Stop
Figure 6. Effect of context length (number of operation steps) on performance in the quantity state (left) and set state (right) tasks.
String Search task, where the former requires identifying which list word belongs to rather than simply determining its presence, the performance of open-source models drops considerably. Similarly, in comparing the Group Associa- tion task with the Group Membership task, where the former requires determining whether two words belong to the same group, all models exhibit noticeable decline in perfor- mance. The decline becomes even more pronounced when comparing the Group Associution (alternating) variant of the task to the standard Group Association task. Here, the context involves alternating repeated groups rather than sim- ple group structures, which further challenges the models' abilities to handle partitioned contexts effectively.
Overall, while larger models like GPT-4(o) exhibit some ability to track state over time, their effectiveness rapidly deteriorates as task complexity increases. Smaller models, in particular, struggle to track operations over time, pointing to significant gaps in their ability to manage and process sequential dependencies critical for state tracking tasks.
An interesting observation was found during the Iterate task. In an ablation study, we modified the task to require re- turning the first words in each list instead of the last words (making it more similar to the Batch Search task). The performance sharply declines when models are asked to re- turn the last words, despite their strong infornatiio-fetching capabilities. This suggests that, while the models can re- trieve information effectively, they struggle accurately recognize and process partitions within the context.
3.4. Results on Composite Tests
The composite tests significantly challenge the models by combining multiple atomic capabilities into a single test. In the Processing Data Blocks task. the context is fixed at 4k tokens, while for the Theory of Mind task, the number of operation steps is set to 100. As shown in Table 6, model performance on both tasks are generally low, showing a broad inability to handle the more complex scenarios. Per- formance across all models drop substantially on composite tasks compared to their performance on individual capability tasks, such as search, recall, and group processing.
Stateful Processing Table 5 presents the results for the Stateful Processing tasks, where performance gaps among models are the most pronounced. The GPT-4(o) models perform well on integer state tracking, while most other
7

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Interestingly, some smaller models, like Mistral and Phi-3- small, exhibit slightly better performance the Theory of Mind task than on the set state tracking task. This anomaly likely stems from their already weak state tracking ability, which limits their performance across both tasks. Addition- ally, these models tend to generate longer answers in the set state task which reduces the set overlap.
the words 'AAA' and 'BBB' belong the same list (Var 2)".
As shown in Table 8, performance differences between prompt variants are generally small, suggesting that instruc- tion interpretation is not the primary bottleneck in these tasks, rather, the main challenge lies in actually executing the task correctly. Nevertheless, we recognize that more intensive prompt engineering could potentially affect model performance. Given these findings, we standardized the prompts to single, simple version (as shown in Appendix A) for all experiments in this paper to ensure consistency and comparability across models. However, because the benchmark is programmable, researchers can easily swap in alternate prompts to explore additional prompt settings. Appendix D provides additional examples on more tests for prompt variations.
Notably, even the most capable models, such as GPT-4- turbo and GPT-40, struggle, showing that scaling model size alone is not enough for solving these composite tasks. Additionally, the variation in performance among smaller models suggests that their limitations stem not only from size but also from underlying architectural or training dif- ferences. This indicates that smaller models require more targeted care to bridge the gap in effective memory use.
3.5. Extending the benchmark to other configurations
4. Related Work
Our benchmark is fully programmable and supports flexi- ble experimentation across a wide range of configurations, including varying context lengths, evaluation criteria, and prompt phrasing. In this section, we illustrate how the benchmark can be adapted beyond the default setup used in the main paper. These examples highlight its versatility in probing model behavior under diverse conditions.
LLM Evaluation with Benchmarks The evaluation of LLMs has traditionally relied on static benchmarks, from early benchmarks for perplexity- -based evaluation (Marcus et al., 1993) datasets focused on specific downstream tasks such as question answering (Kwiatkowski et al., 2019), summarization (Gliwa et al., 2019), math reasoning (Cobbe et al., 2021), and code generation (Chen et al., 2021). As L. began to address a broader range of tasks across var- ious domains (Wu et al., 2023; Wang et al., 2023), more comprehensive benchmark suites (Hendrycks et al., 2021a; Zhong et al., 2023) were developed to assess general ca- pabilities rather than individual task performance. Recent advancements in LLM evaluation have introduced the con- cept of LLM-as a-jdgee- enabling the use of open-ended benchmarks without predefined answers (Zheng et al., 2023). However, these benchmarks remain static in nature and can easily get overfit. Recently, platforms like ChatBot Arena (Zheng et al., 2023) utilize crowdsoureing to rank LLM responses and provides more dynamic evaluations. How- ever, its reliance on human annotation makes it less scal- able. Moreover, despite their utility, existing benchmarks primarily assess downstream applications that usually re- quire multiple capabilities, making it difficult to debug and understand model weaknesses.
Context length In the main experiments, we fixed the context length to 4K tokens to emphasize that models al- ready exhibit notable failures this moderate length across many memory tasks. However, the benehmark is scalable to longer contexts. Table 7 presents additional results for two representative tasks Functional updates and Counting evaluated at various context lengths up to 16K tokens.
In both cases, we observe that model performance begins to degrade significantly well before reaching what typi- cally considered a "long" context window. These failures reveal underlying limitations in how models manage long- range memory beyond simple retrieval. In contrast, mod- els tend to maintain strong performance on retrieval-style tasks (e.g., String search) even at extended lengths, making such tasks less effective at distinguishing model capabili- ties. Additional results and task examples are provided in Appendix D.
Tests and Benchmarks for Evaluating Context Utiliza- tion As L.MMS become capable of processing increasingly long inputs, designing automated tests to evaluate their abil- ity to utilize context has become an area of active research. A notable example is the needle-in-a- hayssaa (NIAH) task'. where small piece of information (the "needle") is hid- den within a long document, and the model needs to re-
Prompt Variation We also investigated the model perfor- mance sensitivity to minor variations in prompt phrasing. Specifically, we tested different phrasings of task instruc- tions while keeping the underlying task logic unchanged. For instance, in the String search task, we compared "Given the context, determine if XXX is present (Var 1)" versus "Is XXX present in the context? (Var 2y". Similarly, in the Group association task, we tested "Determine if word 'AAA' and word 'BB'' are in the same list (Var 1)" versus "Check
hhttpa://githee com/ hatm//gaaaatt//IMMMM Need1 e InAHaystack
8

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Model
500
1K
2K
4K
8K
Model
1K
2K
4K
8K
16K
gpt-4o
1.00
1.00
0.99
0.93
0.59
gpt-4-turbo
0.52
0,44
0.40
0.32
0.28
gpt-4o-mini
0.69
0.66
0.42
0.24
0.10
cohere
0.36
0.24
0.20
0.20
0.12
phi-3-small
0.49
0.45
0.21
0.07
0.03
phi- m-edium
0.20
0.16
0.12
0.08
0.04
(a) Functional updates (ROUGE-L)
(b) Counting (Exact match)
Table 7. Performance across context lengths two representative tasks.
Model
Var 1
C
Var 2
Cl
Model
Yar 1
Class
Var 2
Clus%
gpl-40
1.00
(0.93, 1.00)
1.00
(0.93, 1.00)
gpi-4o
0.65
(0.50, 0.78)
0.63
(0.47, 0.76)
gpi-4o-mini
0.98
(0.90, 1.00)
0.98
(0.90, 1.00)
cohere
0.70
(0.55, 0.82)
0.75
(0.60, 0.86)
phi-3-medium
1.00
(0.93, 1.00)
1.00
(0.93, 1.00)
phi-3-small
0.55
(0.40, 0.69)
0.55
(0.40, 0.69)
(a) String search (Exact match)
(b) Group association (Exact match)
Table Prompt variation performance on the String scarch task (with 50 samples) and the Group association task (with 40 samples). Variation and Variation 2 differ slightly in phrasing but preserve task intent.
trieve it. Similar tests include key-value retrieval (Liu et al., 2024) and passkey retrieval (Mohtashami & Jaggi, 2023). The simplicity and interpretability of NIAH have made it a standard for evaluating LLM context utilization, and it has since inspired various methods for improving long-context processing (Mohtashami & Jaggi, 2023; Ding et al., 2024; Xiong et al., 2023; Behrouz et al., 2024).
2007; Nasreddine al., 2005). By isolating distinct abil- ities while minimizing confounding factors like attention, memory span, and reasoning (Kane et al., 2007), such tests provide detailed profiles of cognitive functions, guiding in- terventions and shaping broader theories of human thought. Inspired by this approach, we design atomic tests that sys- tematically isolate core aspects of L.LM context processing, aiming for a fine-grained understanding of LLM memory- usage capabilities analogous to memory testing in humans.
However, these tests focus solely basic information re- trieval, without capturing more complex aspects of context processing. To address this limitation, other tests have been proposed. Needlebench (Li et al., 2024) extends simple retrieval tasks to include multi-needle reasoning and a an- cestral trace task which requires navigating chains or graphs of information. Song et al. (2024) introduce the Count- ing Stars task, which involves tallying numbers of stars embedded in phrases. Ruler (Hsich et al., 2024) proposes additional tasks such as variable tracking and frequent word extraction. While these tests increase task complexity or broaden the range of evaluated tasks, they remain limited in scope for systematically evaluating contextual processing.
5. Conclusions
AI assistants powered by LLMs are expected to handle nu- merous operations involving memory. However, simple tests reveal that they often fall short of meeting user expectations, even in basic retrieval and processing tasks. For instance, retrieving vacation schedules for each team member from a message history that includes evolving plans over time proves challenging. To enable targeted improvements, it is essential to establish a comprehensive benchmark that tests each capability in isolation while also allowing for pro- grammable composition to evaluate more complex scenarios. Our benchmark provides straightforward yet effective ap- proach to achieving this goal. We primarily focus on short- context scenarios to demonstrate that current limitations are not solely attributable to the models' challenges with parsing long contexts. Addressing these issues demands attention beyond merely solving the "attention" problem.
Beyond individual tests, several benchmarks explicitly tar- get long-context processing, including InftyBench (Zhang et al., 2024), L-Eval (An et al., 2024), and LongBench (Bai et al., 2024). These benchmarks use NIAH-like tasks along- side question answering, summarization, and code genera- tion over long contexts. However, like other benchmarks, they remain static and primarily measure end-to-end perfor- mance rather than systematically dissecting capabilities.
Impact Statement
Analogies to Human Cognitive Testing Memory tests are widely used in cognitive research to assess specific func- tions. Such assessments often involve evaluating short term memory via recall tests (Crannell& & Parrish, 1957; Towse et al., 2008), inductive reasoning via pattern recognition tasks, or attention via cooructtoo-folloiinggoo et al.,
This paper contributes to the advancement of Machine Learning by introducing a systematic and programmable evaluation framework for assessing the contextual process- ing capabilities of large language models. Our work pro- vides insights into model strengths and limitations in han-

---

Minerva: A Programmable Memory Test Benchmark for Language Models
dling various atomic and composite tasks, offering struc- tured way analyze model behavior. These contributions can guide future research in improving model efficiency and reliability.
W. Evaluating large language models trained code. 2021.
Clark, P., Cowhey, I,, Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try are, the al2 reasoning challenge, 2018. URL https: http:://: /ttttp://axxxv.oorr abs// 1803. 05457.
We do not foresee any direct ethical concerns or negative so- cietal consequences arising from this work. Our evaluation methodology is designed to be model-agnostic and does not involve sensitive data or high-stakes applications.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L.,, Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 2021.
References
An, C., Gong, S., Zhong, M., Zhao, X., Li, M., Zhang, J., Kong, L., and Qiu, X. L-eval: Instituting standard- ized evaluation for long context language models. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Pro- ceedings of the 62nd Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pupers), pp. 14388-14411 14 Bangkok, Thailand, Au- gust 2024. Association for Computational Linguisties. URL https: long.
Crannell, and Parrish, J. A comparison of immediate memory span for digits, letters, and words. The Journal of Psychology, 44(2):319
1957.
Ding, Y., Zhang, L. L.,, Zhang. C., Xu, Y., Shang, N., Xu, J., Yang, F., and Yang, M. Longrope: Extending Ilm context window beyond 2 million tokens, 2024.
doi: 10.18653w11/2224cc11oogggg //aclannthology.orggg///o
Anthropic. Introducing the next generation of com/news/ -laude-3-frrl 3- ly, 2024. Accessed: 2024-03-27.
Gliwa, B., Mochol, I., Biesek, M., and Wawer, A. Sam- sum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. Asso- ciation for Computational Linguistics, 2019. doi: 10. 1/15349/411--549 URL http://dx. http/xxxxxoooooo org/ 10. 10.18653/v1/019-5444
claude. https:/ //ws:///wwww./ anthropicc.............. https://www.aattrropp..c
Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. LongBench: A bilingual, multitask benchmark for long context understanding. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meet- ing of the Association for Computational Linguistics (Vol- ume 1: Long Papers). pp. 11-33137, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10. 10.18663//1/20222.clllooooo 172. URL https: //aclanthology.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. URL https:// openreview net forum? dididdd7755 (KwjmIIBIII JMI
Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, Measuring math- ematical problem solving with the math dataset. NeurIPS, 2021b.
Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time, 2024. URL htthttsss//axii arg/abs/2500 00663.
Hsieh, C.-P., Sun, S., Kriman, S., Acharya, S., Rekesh, D., Jia, F., and Ginsburg, B. RULER: What's the real context size of your long-context language models? In First Conference on Language Modeling, 2024. URL https: / //ooreeviees ew. net. forum? didk 65y.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J.. Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri. R.. Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavar- ian, M.. Winter, C., Tillet, P., Such, F. P., Cummings. D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss., A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C. Carr. A. N., Leike, J., Achiam, J.. Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M.. Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba,
Jaccard, P. Étude comparative de la distribution florale dans une portion des alpes et des jura. Bull Soc Vaudoise Sci Nat, 37:547-579, 1901.
Kamradt, G. Needle in a haystack pres- sure testing 1lms. GitHub, 2023. URL L
10

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Kane, R. L.., Roehuck-Spencer, T., Short, P., Kabat, M., and Wilken, J. Identifying and monitoring cognitive deficits in clinical populations using automated neuropsychological assessment metrics (anam) tests. Archives of Clinical Neuropsychology. 22(Suppl_1):SI 15--116,,, 2007.
whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
Towse, J. N., Cowan, N., Hitch, G. J., and Horton, N. The recall of information from working memory: Insights from behavioural and chronometric perspectives. Experi- mental Psychology, 55(6):371 2008.
Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C.,, Epstein, D., Polosukhin, I., Devlin, J., Lee, K., et al. Natural questions: benchmark for ques- tion answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L.,, and Anandkumar, A. Voyager: An open- ended embodied agent with large language models, arXiv preprint arXiv: Arxiv-2305 2309. 2023.
Li, M., Zhang, S., Liu, Y., and Chen, K. Needlebench: Can IIms do retrieval and reasoning in million context window?, 2024. URL https: / wawaaaivvv...o. org/ /bs/ 2407 11963.
Wu, Q. Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L.. Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang. C. Autogen: Enabling next- gen Ilm applications via multi-agent conversation, 2023, URL https: arwiv.iiiivv///.i org/ abs 2008. 08155.
Lin, C.-Y. ROUGE: A package for automatic evalua- tion of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004, Asso- ciation for Computational Linguistics. URL https: //aclanthologe wwg/oogg///00o
Wu, Y., Hee, M. S., Hu, Z., and Lee, R. K.-W. Longgen- bench: Benchmarking long- mor-form generation in long con- text 1lms, arXiv preprint arXiv:2409. 02006, 2024.
Xiong, W., Liu, J., Molybog, I., Zhang. H., Bhargava, P., Hou, R., Martin, L., Rungta, R., Sankararaman, K. A., Oguz. B., Khabsa, M., Fang, H., Mehdad, Y., Narang, S., Malik, K., Fan, A., Bhosale, S., Edunov, S., Lewis, M., Wang, S., and Ma, H. Effective long-context scaling of foundation models, 2023. URL org/ abs 2209 16039.
Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. Lost the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157- URL https: tacl-1.9/
173, 2024. doi: 10.1162/acl //aclanthology wtg///2//2222....../ 2024. 2tt.1/2......11.1...9999 9/.
Marcus, M. P., Santorini, B., and Marcinkiewicz, M. A. Building large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313- 330, 1993. URL J93-2004/.
Zhang, X., Chen, Y., Hu, S., Xu, Z., Chen, J., Hao, M., Han, X., Thai, Z., Wang, S., Liu, Z.,, and Sun, M. Infty- bench: Extending long context evaluation beyond 100K tokens. In Ku, L.-W., Martins, A.. and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 15262-15277, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10.18653ww1/0022.cc-loogo acl- acc.ooo 814/.
org/
Mohtashami, A. and Jaggi, M. Random-accos access infi- nite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL htttps://pperreeee id- 4w0Vy.
URL https:
Nasreddine, Z. S., Phillips, N. A., Bédirian, V., Charbon- neau, S., Whitehead, V., Collin, I., Cummings, J. L., and Chertkow, H. The montreal cognitive assessment, moca: brief screening tool for mild cognitive impairment. Jour- nal of the American Geriatrics Society, 53(4):695-699, 2005.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing. E., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging I.LMaas-a-judge with MT-bench and chatbot arena. In Thirty-sevnnth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: / / openrevieww net/ / forum? d=wcHHGDla...
Song, M., Zheng, M., and Luo, X. Counting-stars: A multi-evidence, position-awand aware, and scalable benchmark for evaluating long-context large language models, 2024. URL https: http://axvvvv..///e//ee//eeea../ org/ abs/ 11802.
Zhong. W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human- benchmark for evaluating foundation mod- els, 2023. URL https: /arxivvvvvv www.i.wwrgg//.....o.o...o..ooott abs/2304 06364.
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le. Q. V., Chi, E. H., Zhou, D.,, , and Wei, Challenging big-bench tasks and

---

Minerva: A Programmable Memory Test Benchmark for Language Models
A. Test Templates
In this appendix, we provide the templates of the test prompts. Placeholder context words such as "aaa, bbb, ccc," etc., are used for illustration purposes. During testing, these context words are uniformly sampled from an English dictionary. Variable tokens in the instruction part marked with the bold font.
Search
Task name
Prompt
String search (with word)
Context:
aaa, bbb, ccc,
Instruction:
Given the context, determine the word "bbb" is present in the context. Answer with "yes" or 'no".
Answer:
String search (with subsequence)
Context:
aaa, bbb, ccc,
Instruction:
Given the list of words in the context, determine if the sequence "bbb, xxx, ddd" appears in the context. Answer with 'yes' or 'no'.
Answer:
Key-value search
Context:
aaa:bbb, eecıddd,
Instruction:
Given a list of word pairs formatted as "word_l: word 2 in the context, return the second word associated with the provided first word. For the first word "aaa", the corresponding second word is:
Batch search
Context:
aaa:bbb, cec:ddd,
Instruction:
Given a list of word pairs formatted as "word word 2" in the context, return the second word associated with the provided first words. For the first words: aaa, cee, the corresponding second words are:
Recall and Edit
Task name
Prompt
Snapshot
Context:
aaa, bbb, cCC,
Instruction:
Repeat the previous context exactly as is, without making any additions or deletions.
Answer:
Replace all (x to y)
Context:
aaa, bbb, aaa, ccc, aaa, ddd,
Instruction:
Repeat the previous context and replace the word "aaa" with "zzz" each time it appears.
Answer:
Replace all (x to null)
Context:
aaa, bbb, aaa, ccc, aaa, ddd,
Instruction:
Repeat the previous context but skip the word "aaa" each time it appears.
Answer:
12

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Overwrite (nth y)
positions
Context:
aaa, bbb, ceC,
Instruction:
Repeat the previous context and replace every third word with "zzz".
Answer:
Overwrite positions
Context:
(nth null)
aaa, bbb, ccc, ...
Instruction:
Repeat the previous context and skip every other word.
Answer:
Functional updates
Context:
111, 222, 333,
Instruction:
Add 3 to every number in the previous context.
Answer:
Match and Compare
Task name
Prompt
Compare positions
Context:
aaa, bbb, ccc,
Instruction:
Given the list words in the context, determine the relative positions two words.
Does the word "aaa" appear before the word "cec" the list? Answer "yes" or "no".
Answer:
Find duplicates
Context:
aaa, bbb, aaa,
Instruction:
word is repeated multiple times in the context. Your task is to identify the word that is repeated.
The repeated word is:
Count
Context:
aaa, bbb, aaa,
Instruction:
Count the number of times the word "aaa" appeared in the context.
Answer: The word "aaa" appeared
Check association
Context:
aaarattribute bbb:attribute 2, cec: attribute 2. ddd: attribute 1,
Instruction:
Given the list of words and their respective attributes in the format of "word:attribute" determine if the word "aaa" and the word "ggg" have the same attribute. Answer with "yes" or "no".
Answer:
Spot the Differences
Task name
Prompt
13

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Compare lists
Context:
List 1: aaa, bbb, cec,
List 2: aaa, ddd, occ,
Instruction:
There are two lists of words in the context. The first list contains the original words. The second list is similar to the first but has some words replaced with different ones. first/second list that are different from those
Your task is identify the words the other list. Provide the different words as your answer. Answer:
Identify group
the odd
Context:
List 1: aaa, bbb, cee,
List 2: bbb, aaa, oec,
List 3; aaa, zzZ, CCC,
List 4: ccc, aaa, bbb,
Instruction:
Given the lists of words in the context, identify the list that is different from the others. Provide the list number as your answer. For example, the Nth list is different, provide
"List N as your answer.
Answer:
Patch the difference
Context:
aaa, bbb, ccc, aaa, bbb, cec,
Instruction:
Given the sequence of words that follows a specific pattern in the context, predict the Nth word that appears after the final word in the given sequence. Answer: The Nth word that appears after the final word in the given sequence is
Compute on Sets and Lists
Task name
Prompt
Group membership
Context:
List 1: aaa, bbb, cee,
List 2: ddd, eee, fff.
Instruction:
Given the lists of words in the context, determine which list contains the word "fff". If the word is not present in either list, answer "no"". Answer:
Group association
Context:
List 1: aaa, bbb, ccc,
List 2: ddd, eee, fff.
Instruction:
Given the lists of words in the context, determine if word "aaa" and the word "eec" are the same list. Answer with "yes" or "no".
Answer:
14

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Group association (al-
Context:
ternating)
Role A: aaa, bbb,
Role B: ecc, ddd,
Role A: eee, fff,
Role B: ggg, hhh,
Instruction:
Given the context with alternating roles and their respective context words, determine if the word "aaa" and the word "ggg" are in the same role. Answer with "yes" "no".
Answer:
Iterate
Context:
List 1: aaa, bbb, oce, ...
List 2: ddd, eee, fff,, ...
Instruction:
Given the lists of words in the context, identify and recall the last word from each list. Provide your answer as a list of these words separated by commas.
Answer:
Stateful Processing
Task name
Prompt
Quantity state
Context:
Begin with the number xx. Perform the following operations:
1. Add xx 2. Subtract xx 3.
Instruction:
In the context, you are given an initial number and a series of operations to perform on that number. Your task is to determine the final result the operations. Write your final answer after the text "FINAL ANSWER:". For example, "FINAL ANSWER: 42".
FINAL ANSWER:
Set state
Agent actions:
Agent draws aaa, bbb, ccc
Agent discards bbb, ccc
Agent draws ddd, fff
Agent discards ddd
Instruction:
Given the actions of the agent, your task is determine the final list of words the agent ends up with after series of actions. Write your final answer after the text "FINAL ANSWER:". For example, "FINAL ANSWER: wordl, word2, word3".
FINAL ANSWER:
Processing Data Blocks
Task name
Prompt
15

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Processing
Data
Context:
Blocks
Role 1: aaa, bbb, cee,
Role 2: ddd, eec, fff, The context consists of series of alternating roles, each associated with list of words. Your task is identify and recall all the words from the role labeled "Role 2" that appear after the word "zzz" in the sequence. Please write your answer after the text "Answer:". For example, "Answer: wordl, word2, word3".
Role 3: ggg, hhh, iii,
Role 1: iij. kkk,
Instruction:
Answer:
Composite-State Tracking (Theory of Mind)
Task name
Prompt
Theory of Mind
Agents actions::
Agent A starts with the following words: aaa, bbb, ccc.
Agent B starts with the following words: ddd, eee, fff,
Agent B starts with the following words: ggg, hhh, iii...
Agent B swaps the following words "ddd" with Agent for the following words "hhh".
Agent discards the following words: cec, zzz,
Agent draws the following words: xxx, yyy....
Instruction:
Given the actions of the agents, your task is to determine the final list of words each agent ends up with after a series of actions. Write your final answer after the text "FINAL ANSWER:". For example, "FINAL ANSWER: Agent A: word1, word2, word3 nAgent B: word4, word5". FINAL ANSWER:
B. Task Details
This appendix provides details for each task, including the number of examples, evaluation metrics, and configurable hyperparameters. The context length is fixed at 4k for almost all tasks, apart from Stateful Processing, where the context is determined by number of operation steps and set to 200 for quantity state and 100 for set state, which maps to around 1.5k context tokens.
Here an example of number of examples calculation String search (with word): (query depth) * (labels) * (samples per parameter setting) 50.
Table 10: Task Overview with Hyperparameters, Number Examples, and Evaluation Metrics
Task Name
Hyperparameters
of Examples
Metric
Search
String search (word)
query depth [0, 0.25, 0.5, 0.75, 1], label [positive, negative], samples 5 sequence length [8, 16. 32, 64), label [posi- tive, negative], samples = 10
50
exact_match
String search (sequence)
80
exact match
Key-value search
query depth = [0, 0.25, 0.5, 0.75, 1]. samples 10
50
exact_match
Continued on next page
16

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Task Name
Hyperparameters
of Examples
Metric
Batch search
batch size [4, 8, 16, 32], samples 5
20
rouge-L.recall
Number of Entries for Category: 200
Recall and Edit
Snapshot (words)
samples 10
10
rouge-L rouge-L
Replace all
density [0.2, 0.4, 0.6, 0.8] [random word, null]. samples 5
40
Overwrite positions
nth =2. 3, 4]. y= [random word, null], samples
30
rouge-L
=5
Snapshot (numbers)
samples 10
rouge-L rouge-I.
Functional updates
function type = [add (3), subtract (1), multiply (2)), samples
Number of Entries for Category: 105
Match and Compare Compare positions
query depth [0, 0.25, 0.5, 0.75, 1J. query 2
75
exact_match
depth [0. 0.25, 0.5, 0.75, 1J. samples =
Find duplicates
repetition = [2. 4, 8, 16, 32], samples 5 repetition = [2. 4, 8, 16, 32], samples =
25
exact_match
Count Check association
25
exact_match
n attribute = [2, 4, 8, 16, 32), label [positive,
50
exact_match
negative], samples =
Number of Entries for Category: 175
Spot the Differences Compare lists
different words [1, 5, 10, 20), chosen
80
rouge-L.recan recall
list = [first, second], samples = 10
Identify the odd group
words per group [25, 50, 75, 100], percentage of difference = [0. 0.25, 0.5), samples = pattern length = [2, 15, 30], cut off depth = [0, 0.5, I], nth = [1, 6], samples= =
60
exact_match
Patch the difference
120
exact match
Number of Entries for Category: 260
Compute on Sets and Lists Group membership
number of groups [4, 8, 16. 32], query depth
100
exact_mater
[0, 0.25, 0.5, 0.75, 1], samples 5 number of groups [4, 8, 16, 32], label [pos-
Group association
40
exact_match
itive, negative], samples = 5
Group association (alternating)
number of groups = [2, 4, 8, 16, 32], number of turms = 10, label = [positive, negative], sample
50
exact_match
=
Iterate
number of groups [4, 8, 16, 32], samples 5
20
rouge-L
Number of Entries for Category: 210
Stateful Processing
Set state
number of steps 100, set size [5, 10, 15, 20), samples 10 number of steps 200, samples = 10
40
jaccard,similarity
Quantity state
10
exact match
Number of Entries for Category: 50
Continued on next page
For Patch the difference task with pattern length 2, there is only cut off percentage options; therefore the total number of data points is instead of 135.
17

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Task Name
Hyperparameters
of Examples
Metric
Composite
Processing data blocks
number of blocks [2, 4, 8, 16, 32), number of turns 10, samples 5
50
rouge-L
Theory of mind
number of steps 100, number of agents = [2, 3, 4], samples = [10, 20]
60
jaccard similarity
Number of Entries for Category: 110
Total Number of Entries: 1110
C. Evaluation Metrics
In this appendix section, provide details about the evaluation metrics we have used in the tests.
Exact Match: The exact match accuracy measures whether the generated answer exactly matches the reference answer. is computed follows:
{
reference answer generated answer,
Exact Match
otherwise.
ROUGE-L/ ROUGE-L-recall: ROUGE (Recall-Oriente Understudy for Gisting Evaluation) (Lin, 2004) measures the verbatim overlap between the reference and the generated answers. ROUGE-L specifically looks for the longest common subsequence (LCS) between the texts, which reflects the structure of the text and the longest sequence of matching words. ROUGE-L recall focuses on the ability of the model recall the content from the reference answer, and it emphasizes matching the longest subsequences.
ROUGE-L-recall can be defined as:
(CS[[gemrated.aanww referencce length reference answer
ROUGE-L-recall
ROUGE-L is computed as the Fl-score, which combines both precision and recall to provide more balanced measure of overlap.
Jaccard Similarity: Jaccard similarity measures the overlap between two by comparing the intersection and union of the sets. It is computed as:
LA n B
Jaccard Similarity
A U B
where A and B are sets representing the elements in the generated and reference answers, respectively. This metric is used for tasks involving set-based comparisons or when the goal is measure the similarity between two sets of elements (c.g., word sets).
D. More Examples for Context Length Variation and Prompt Variation
In this section, we present additional examples illustrating the effects context length and minor variations in prompt instructions on selected models.
Length variation Table 11 reports results from three additional tasks: string search (word), replace all, and iterate. The results reaffirm that while models tend to perform well on the search task even at long context lengths, their performance on other tasks degrades significantly at much shorter lengths. This highlights a fundamental limitation how effectively these models utilize long-range context for non-search tasks.
18

---

Minerva: A Programmable Memory Test Benchmark for Language Models
Model
2000
4000
8000
16000
32000
Model
1000
2000
4000
8000
16000
gpt-4o
1.00
1.00
1.00
1.00
0.98
gpt-4o
1.00
1.00
0.99
0.81
0.48
gpt-4o-mini
0.98
0.98
0.94
0.90
0.78
gpt-4o-mini
0.99
0.91
0.84
0.71
0.42
phi-3-small
1.00
0.94
0.90
0.98
0.98
phi-3-small
0.87
0.67
0.49
0.32
0.06
(a) String scarch (word)
(b) Replace all
Model
1000
2000
4000
8000
16000
gpt-4o
1.00
0.97
0.86
0.70
0.57
gpt-4o-mini
0.91
0.87
0.67
0.43
0.28
phi-3-small
0.15
0.09
0.04
0.01
0.01
(c) Iterate
Table 11. Model performance across varying context lengths for three tasks.
Prompt variation. We provide additional results examining how small changes in prompt phrasing affect model perfor- mance across three tasks (see Table 12). In general, we find that minor wording changes (e.g., replace all and quantity state) do not significantly affect performance. This suggests that task accuracy is primarily driven by the model's underlying capability to process memory rather than sensitivity to prompt wording. However, more substantial prompt changes, such as shown in check association task, can lead to notable differences in performance across models. To minimize the impact of prompt tuning, we standardize all prompts to the versions specified in the main paper.
Model
Var
Cly5%
Var 2
Cly5%
Model
Var
Cla5%
Var 2
Cla5%
gpt-4o
0.99
(0.93, 1.00)
0.98
(0.93, 1.00)
gpt-4o
0.72
(0.58, 0.83)
0.76
(0.63, 0.86)
gpt-4o-mini
0.84
(0.71, 0.94)
0.83
(0.71, 0.94)
gpt-4o-mini
0.60
(0.46, 0.72)
0.58
(0.42, 0.69)
phi-3-small
0.49
(0.32, 0.63)
0.51
(0.35, 0.65)
phi-3-small
0.70
(0.56, 0.81)
0.60
(0.46, 0.72)
(a) Replace all (ROUGE-L)
(b) Check association (Exact match)
Model
Var
Cla5%
Var 2
Cla5%
gpt-4o
1.00
(0.72, 1.00)
1.00
(0.72, 1.00)
gpt-4o-mini
0.70
(0.40, 0.89)
0.80
(0.49, 0.94)
phi-3-small
0.00
(0.00)
0.00
(0.00)
(c) Quantity state (Exact match)
Table 12. Effect of prompt variations on model performance across three tasks. We report accuracy (or ROUGE-L) and 95% confidence intervals.
Prompt variants used in the table above:
Replace all
Var I: "Repeat the previous context and replace the word with bbb."
Var 2: "Copy the previous context but replace the word aaa with bbb."
Check association
Var 1: "Given the context with words and their assigned attributes in the format of word: ATT N, determine if
the word aaa has the same attribute as the word bbb?"
Var 2: "Given the context"
Quantity state
Var 1: "Your task to determine the final result of the operations."
Var 2: "Determine the final result after the operations."
19


Reviews:
[{"review_id": "SIYsWl21Z3", "reviewer": "Reviewer_H8dt", "summary": "This paper introduces a framework for systematically evaluating the memory utilization capabilities of language models. Expanding beyond conventional memory tests—such as passkey retrieval, key-value lookup, and needle-in-the-haystack search—the proposed framework assesses models on a broader range of atomic tasks, including searching, recalling, editing, matching, and comparing information within structured memory contexts. Additionally, the authors design composite tests to examine models' ability to maintain state across operations. By simulating real-world data structures, this benchmark provides a more interpretable and granular assessment of large language models' memory handling capabilities.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thoughtful feedback and the chance to clarify our contributions.\n\n**Insights from the benchmark**\n\nOur main goal is not to rank known models, but rather to introduce tests to evaluate different functionalities needed for LLM (agents). Any model can be (1) tested using our benchmark, (2) updated/retrained/fine-tuned to improve its functionalities WITHOUT the problem of being overfitted on the benchmark, and (3) have older and newer versions of the model compared fairly by rerunning the benchmark with fresh randomness.\n\nWhile some of our results align with prior benchmarks, we disagree that our work merely reinforces prior knowledge. It is true that GPT-4o performed the best in our tests, as expected, but our findings provide meaningful insights beyond the simplification of “GPT-4o is better than GPT-4o-mini”. Specifically: (1) Models of similar sizes (particularly open-source ones) show notable performance variations across different types of memory tasks (See Figure 3-5) (2) Performance gaps vary by task, e.g. while models perform similarly in information fetching tasks, more challenging tasks like \"spot the difference\" reveal larger gaps. (3) Smaller models can outperform larger ones on certain tasks. Notably, LLaMA-3.1-8B and phi-3-small outperformed GPT-4-turbo in “patch the difference”, despite being smaller. This suggests that factors beyond sheer model size, such as architectural choices and training data, play a significant role in memory-related capabilities. Generating insights doesn’t require contradicting prior benchmarks. Instead, we aim to uncover nuanced model behavior patterns that inform development and evaluation strategies. This goes beyond using benchmarks merely for ranking models.\n\nWe also would like to clarify on the following regarding our experiment setups:\n\n**1. Variation in context length**\n\nWe mainly fixed the context length to 4K tokens to highlight that models already struggle in many memory tests at this length. We have also shown in one of our ablation studies (Figure 6) how the number of operation steps/context length affects performance. We also have other ablation results that show a similar trend: performance was near perfect at shorter lengths, but significantly lower well before reaching what is typically considered \"long context\". We can add more results in the paper to clarify these findings. Additionally, since our benchmark is fully programmable and transparent, it can be extended to evaluate models at different context lengths.\n\n**2. Evaluation criteria**\n\nFor the substring presence test, exact match means that if the model’s response (yes/no) matches with presence/absence of the substring, it receives a score of 1, otherwise 0. We believe this is a fitting metric for a binary classification task.\n\n**3. Variations in prompt wording**\n\nAs discussed in our response to Reviewer RVnF, we conducted a preliminary experiment to assess the impact of wording variations and found that performance differences are minimal for instruction rephrasing. This supports our decision to fix prompts to a single simple version (in Appendix A) for consistency and comparability across models. So, based on this result, we observe that the main challenge is not in “understanding” the instruction, but rather in “performing the task”. All that said, note that (1) our benchmark is not a static dataset and a simple script can be added to our benchmark to generate many variations; (2) we will add confidence intervals around the results to show also robustness to instruction variations. \n\n**4. Deterministic setting**\n\nWe deliberately used a deterministic setting for consistency and controlled comparisons. While we understand the concern that this might disadvantage certain models, we argue that: (1) A deterministic setting provides more stable and reproducible evaluation. (2) This choice is an evaluation setting, not a limitation of the benchmark itself. While we used a deterministic setup in this study, our benchmark is fully configurable. Users can modify decoding parameters to introduce stochasticity and explore its effects on different memory tasks.\n\n**5. Separation between proprietary and open-source models**\n\nWe separate proprietary and open-source models because these groups tend to be more comparable within themselves. For example, the open-source models we evaluated are of similar sizes, making side-by-side comparisons more meaningful. This grouping allows for clearer analysis while still enabling cross-category comparisons where relevant.\n\n**Final remarks on experiment setups**\n\nOur benchmark is designed to be fully programmable, extendable to different context lengths, evaluation criteria, prompting strategies, or other configurations. We will open-source our code and provide a snapshot of our data to encourage further research. However, for the sake of reproducibility, we fixed the experimental settings as described in the paper to ensure consistency in our results.", "claims_and_evidence": "The paper claims to introduce a novel framework for evaluating the memory utilization capabilities of large language models (LLMs). While the authors do propose an extended set of tests beyond traditional memory benchmarks, it is unclear whether the framework provides fundamentally new insights or if it merely reinforces existing knowledge about model performance. Looking at the performance results, it appears that models that perform well on standard NLP benchmarks (e.g., GPT-4o outperforming GPT-4 Turbo, which in turn outperforms GPT-4o-mini) also perform well on this benchmark. This raises the question of whether the proposed benchmark truly captures unique aspects of memory utilization or if it is simply correlated with overall model quality. Additional evidence, such as cases where model rankings differ from traditional benchmarks, would strengthen the claim of novelty.", "methods_and_evaluation_criteria": "The authors employ a variety of atomic and composite tests to assess models' abilities to retrieve, edit, and manipulate information within structured contexts. However, there are some concerns regarding the design of the evaluation criteria. For example, in the String Search task, the prompt asks whether a certain substring (e.g., “bbb”) is present, with expected answers being \"yes\" or \"no.\" However, the evaluation metric used is exact_match, which does not seem appropriate in this context.\n\nAdditionally, the authors set the temperature to 0 and top-p to 1, ensuring deterministic outputs. While this suggests an intent to evaluate models under controlled conditions, it is unclear whether this decision might disadvantage certain models or affect specific types of tests. For instance, tasks requiring reasoning or more flexible memory retrieval might be better evaluated with slightly higher randomness. The paper should clarify the implications of this choice on model performance.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental setup is well-structured, but there are areas where further exploration could enhance the analysis:\n\n1. The paper does not discuss whether the authors experimented with varying context lengths, which could have provided deeper insights into how memory performance scales with longer or shorter inputs. Given that real-world applications involve diverse context lengths, it would be valuable to understand whether model performance remains stable across different settings.\n\n2. It is unclear whether the models' performance depends on how instructions are phrased in the prompts. The authors should clarify whether variations in prompt wording lead to performance shifts and, if not, why that would be the case. Understanding how models respond to different instructions is critical for assessing robustness.\n\n3. The authors emphasize the distinction between proprietary (black-box) and open-source models (Line 184), but it is unclear why this distinction is particularly relevant in the context of memory benchmarking. Are there fundamental differences in how proprietary and open-source models handle memory? Additional clarification on this point would be useful.", "relation_to_broader_scientific_literature": "The work is relevant for the research community.", "essential_references_not_discussed": "The authors have discussed sufficient literature on the topic.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper provides a structured approach to evaluating LLMs’ memory abilities, introducing a broader set of tasks beyond traditional benchmarks.\nThe inclusion of composite tests that assess state retention while operating on memory is valuable for understanding sequential dependencies.\n\nWeaknesses: \n\nThe novelty of the benchmark is questionable, given that performance trends largely align with existing NLP benchmarks.\nThe lack of variation in context length experiments limits insights into how well models generalize memory performance across different scales.\nThe deterministic setting (temperature = 0) may not be suitable for all types of memory-related tasks and could influence results in ways that are not discussed in the paper.", "other_comments_or_suggestions": "Minor\nLine103-104, seems to have some structural irregularity. I don’t see the bullet points following from the earlier discussion. Please rewrite. \nLine 74, 84 (right) - remove boldening of words", "questions_for_authors": "Mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CXSbYk1ZC", "reviewer": "Reviewer_7bBC", "summary": "This paper presents a framework for automatically generating a broad set of tests to evaluate LLMs' memory usage. Going beyond simple search, the benchmark assesses tasks like recalling, editing, matching, and tracking information across distinct data blocks. Experiments reveal that while models handle basic retrieval, they often fail more complex operations, highlighting important gaps and guiding future improvements.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s comments and the opportunity to clarify our contributions.\n\n**Position of the paper and fitness with ICML**\n\nThe reviewer raises the concern that our paper does not introduce new theoretical concepts. The paper does not aim to be theoretical. The nature of memory analysis in LLMs is empirical. However, we believe that a rigorous and comprehensive benchmarking framework is a significant contribution. Using static data benchmarks to test needle-in-the-haystack type of analysis is not sufficient especially in a rapidly evolving field like LLMs. Our study provides:\n\n1. **A well-defined taxonomy of memory-related tasks** that categorizes different types of memory demands in LLMs, providing a structured way to assess and analyze LLM memory capabilities, as described in Section 2. \n\n2. **New insights into LLM memory behavior** across various tasks, revealing critical limitations that have not been studied before. As discussed in Section 4 line 434 - 436 (left) and 385-394 (right), previous benchmarks and tests focus mainly on information retrieval, and lack a comprehensive evaluation on memory. Many other benchmarks are also static dataset benchmarks.\n\n3. **A programmable, extensible benchmark** that enables future research on memory efficiency, potentially informing both theoretical studies and model development. This mitigates the risk of overfitting on benchmarks, and biasing the empirical evaluations. \n\nHistorically, benchmarks have played a pivotal role in advancing ML research. For example, ImageNet reshaped computer vision, and MMLU influenced LLM evaluation. Similarly, our benchmark exposes key limitations in memory-intensive tasks, which could inspire future advancements. Moreover, evaluation is explicitly within the scope of ICML’s call for papers.\n\n**Clarifying the distinction between simple search and other tasks**\n\nThanks for the suggestion and we will add additional explanations and examples to better illustrate this distinction in the paper. \n\nBy \"simple search,\" we refer to tasks that require only the ability to locate and extract relevant information from memory without additional processing (see line 105, left). In our benchmark, this includes tasks like string search, key-value search, and batch search (see Table 1 and Appendix A). For example, given a keyword x, the model retrieves the associated value y. This \"locate-and-extract\" ability resembles traditional search tasks, and thus we refer to them as simple search. This type of task has dominated prior work on LLM memory utilization, such as the Needle-in-a-Haystack task (see Section 4, line 419, left). \n\nHowever, real-world applications of memory demand more than just retrieval. Our benchmark explicitly evaluates tasks that require models to recall, edit, match, compare, and track state, which extend beyond this simple search (see Table 1 and line 55-67, right). For instance: A writing assistant that rephrases or edits a paragraph operates differently from a system that merely retrieves a fact embedded in a document. A financial analysis tool that verifies consistency across multiple reports needs to compare and reconcile stored information rather than just extract it. A personalized assistant managing ongoing tasks must track state changes over multiple interactions instead of retrieving isolated details. In practice, memory-intensive tasks often require a composition of these abilities. Our benchmark not only evaluates these capabilities but also provides a structured taxonomy to identify model strengths and weaknesses systematically.", "claims_and_evidence": "Please refer to Strengths And Weaknesses.", "methods_and_evaluation_criteria": "Please refer to Strengths And Weaknesses.", "theoretical_claims": "Please refer to Strengths And Weaknesses.", "experimental_designs_or_analyses": "Please refer to Strengths And Weaknesses.", "relation_to_broader_scientific_literature": "Please refer to Strengths And Weaknesses.", "essential_references_not_discussed": "Please refer to Strengths And Weaknesses.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper presents a wide range of memory-related tasks and thoroughly evaluates them.\n\n- By incorporating visual results, the authors make it easier for readers to compare model performance and understand outcome disparities.\n\n\n**Weaknesses**\n\n- While extensive in empirical evaluation, the work does not introduce significant new theoretical concepts or frameworks for memory usage.\n\n- The paper positions itself as going \"beyond simple search,\" but does not clearly illustrate the threshold that separates simple retrieval from more advanced memory tasks; additional concrete examples would clarify these distinctions. \n\n- The paper's core focus on LLM memory benchmarking may not align closely with the calls for papers in ICML.", "other_comments_or_suggestions": "Please refer to Strengths And Weaknesses.", "questions_for_authors": "Please refer to Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "120Qrcoqjm", "reviewer": "Reviewer_8FYn", "summary": "Paper presents a new benchmark for evaluating LLMs' long-context problem-solving abilities. The proposed benchmark include atomic tasks (searching, recalling, editing, matching, etc) that evaluate models on tasks that go beyond those commonly explored (passkey, key-value, needle in the haystack). Experiments present a comprehensive and detailed analysis of different models' ability to use memory.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer’s positive assessment of our work and thoughtful feedback.\n\nOne key takeaway from our experiments is that different models exhibit high variance across atomic memory tasks, reinforcing the need for a diverse evaluation suite. In addition, we found that a major failure pattern is that models perform well on direct retrieval (e.g., finding an exact string in a long context) but degrade significantly when required to store, update, and apply memory dynamically. We believe this might be due to the mainstream attention-based architecture, which makes point-wise access (e.g., retrieving a specific token) relatively easy but lacks explicit memory mechanisms for storing the information and updates. Future models may benefit from architectures that explicitly model memory, such as those explored in the recent Titan paper. \n\nBeyond memory retention, models also struggle with comparing, integrating, or transforming stored information, particularly when relevant facts are dispersed across different parts of the context rather than appearing in a single block, or when the task requires keeping track of updates to an entity’s attributes over time. These results suggest that attention alone may not be enough for effective long-context understanding. Future improvements might require structured memory systems, such as hierarchical memory representations to help models better handle complex memory tasks.\n\nWe will expand this discussion in the paper to further highlight how our evaluations can inform future model development.", "claims_and_evidence": "The main claim is that the proposed approach provides a more comprehensive and nuanced evaluation of an LLM's context utilization, which I believe is supported by the experiments.", "methods_and_evaluation_criteria": "Presents a wide range of different and relevant atomic tasks for evaluating long-context capabilities in LLMs, including search, recall, edit, match, compare, spot the difference, stateful processing, processing data blocks. Paper also evaluates models on composite tasks that combine multiple atomic abilities, and show that there is a significant deterioration of abilities in this setting.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments evaluated a range of different models (both black-box and open-source) on the proposed benchmarks. The evaluations show that different models perform differently on different atomic tasks (i.e. A model being better of some tasks doesn't mean it will be better on another), which highlights the need to a wide range of tasks to evaluate different kind of long-context retrieval abilities.", "relation_to_broader_scientific_literature": "While a number of prior works have proposed benchmarks for context utilizations, they mainly focus on basic information retrieval or are relatively limited in scope. The proposed new benchmark presents a more comprehensive suite of atomic evaluations on different kinds of tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "One of the claims of the paper is that the proposed benchmarks provide more a more detailed view into the long-context capabilities of the model, which can provide better guidance for future model training and development. It would be nice to add a bit more discussions about the specific kinds of improvements the evals revealed.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v9jS4YwDVC", "reviewer": "Reviewer_RVnF", "summary": "Minerva introduces a \u0010thorough evaluation set to test the memory abilities of different LLMs. The evaluation set is generated with parametric programs and covers a wide breadth of different memory skills such as information retrieval and localization, processing, content transfer and structural awareness. Tests are also divided into atomic and composite where atomic tests individual skills while composite relies on the combination of atomic skills. Different LLMs are evaluated and the results provide a more detailed picture on the different memory skills of LLMs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and positive assessment of our paper. Below, we address your main concerns:\n\n**Use of in-context examples**\n\nWe intentionally did not include in-context learning examples, as our goal is to evaluate models' inherent memory capabilities, rather than their ability to perform few-shot learning. We think that including in-context examples would conflate memory with adaptation to in-context cues, making it harder to isolate the specific skills we aim to measure.\n\n**Prompt variation and sensitivity**\n\nWe recognize that LLMs can be sensitive to prompt formulation. However, our primary goal in this work is to establish a standardized benchmark for evaluating memory capabilities, rather than optimizing performance through prompt engineering.\n\nTo assess the impact of prompt variations, we conducted a preliminary experiment where we tested different wordings for the same atomic task. We found that, as long as no heavy prompt tuning is involved, the impact on performance remains minimal. For example, in the word presence task, we obtained identical results from multiple models for the prompt variant \"Is xxx present in the context?\" compared to \"Given the context, determine if xxx is present in the context.\" This suggests that minor wording differences do not substantially alter model performance.\n\nGiven these findings, we believe that fixing the prompts to a single, simple version (as shown in Appendix A) ensures consistency and comparability across models. While prompt tuning could enhance performance for specific models, that is a separate research question beyond the scope of this work.\n\n**Number of trials**\n\nThe results reported in the paper are based on a fixed snapshot of the benchmark, meaning we used our program to generate a predefined set of 1110 evaluation instances, which we assume corresponds to what the reviewer means by \"one trial.\" However, we generated multiple instances for each task (see Appendix B), ensuring a more diverse representation. To maintain reproducibility and consistency, we fixed the experimental settings as described in the paper.\n\nWe also want to emphasize that our benchmark can be used to generate any number of test samples and is fully extendable to different context lengths. We will open-source our code and also provide the exact snapshot of our data used in this experiment to facilitate further experimentation with alternative configurations.", "claims_and_evidence": "The main claim is that current benchmarks do not capture memory-usage capabilities and the evidence is shown through the new evaluation set as well as well as the different results.", "methods_and_evaluation_criteria": "The method in this case is an evaluation set and it is measured with several different criteria including ROGUE, exact match and Jaccard similarity. Given the nature of the task, these metrics \nmake sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "There are many experiments across the different tasks. The main limitation is that it is unclear how much context or in-context examples were given to the different models as well as different prompt variants. Given the sensitivity of LLMs to these factors, it is an important aspect to look at.", "relation_to_broader_scientific_literature": "Deigning new benchmarks to test LLM abilities is a common topic. The most related are needle in a haystack benchmarks but Minerva goes much deeper in testing the abilities of memory of LLMs.", "essential_references_not_discussed": "Not that I am aware of", "other_strengths_and_weaknesses": "Strengths:\n- The topic of memory utilization is extremely important as it is crucial to many different tasks. In general, designing new benchmark is also important given the saturation of current ones. \n- The test set is fairly comprehensive and shows strengths and weaknesses of different models (rather than all models performing similarly). These results are useful for the broader community. \n- The test set is flexible making it easier to add to. The benchmark is also easy to extend and can prevent models from over fitting to it. \n\nWeaknesses:\n- As mentioned earlier, the main weakness is how the models were evaluated. Given how sensitive models are to different prompts and effectiveness of in-context learning, it would make sense to provide several examples for the different tasks as that might change the performance, especially for the more complex tasks. For the composite tasks, the examples could come from the output of the atomic tasks.\nOverall, this paper presents an extendable evaluation set that examines the memory capabilities of different LLMs. Given that there is not an evaluation set like this, I think it should be accepted as it provides insight that would otherwise be difficult to obtain.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. How many trials were done (if more than one)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Minerva, a framework for evaluating LLMs' memory capabilities beyond simple retrieval tasks. The benchmark assesses a comprehensive set of atomic tasks (searching, recalling, editing, matching) and more complex composite tasks that require maintaining state while operating on memory.
Strengths
- Offers a programmable framework that prevents overfitting through dynamic test generation
- Provides a structured taxonomy of memory-related tasks for systematic evaluation
- Reveals nuanced insights into model performance across different memory capabilities
- Demonstrates that models perform well on direct retrieval but struggle with more complex operations

Weaknesses
- Primary experiments use a fixed context length (4K tokens), though additional results show scaling behavior
- Limited exploration of prompt variations, though preliminary tests suggest minimal sensitivity
- Primarily empirical rather than theoretical contribution (though appropriate for this work)

The contribution addresses an important gap in LLM evaluation, offering  value to the research community by enabling systematic assessment of how models utilize context memory. The experimental results could inform future architectural improvements. While some aspects could be strengthened, the core contribution represents a valuable addition to the field of LLM evaluation.
Decision: Accept (poster)