Assessing Episodic Memory in LLMs with Sequence Order Recall Tasks

## Abstract

Abstract Current LLM benchmarks focus on evaluating models’ memory of facts and semantic relations, primarily assessing semantic aspects of long-term memory. However, in humans, long-term memory also includes episodic memory, which links memories to their contexts, such as the time and place they occurred. The ability to contextualize memories is crucial for many cognitive tasks and everyday functions. This form of memory has not been evaluated in LLMs with existing benchmarks. To address the gap in evaluating memory in LLMs, we introduce Sequence Order Recall Tasks (SORT), which we adapt from tasks used to study episodic memory in cognitive psychology. SORT requires LLMs to recall the correct order of text segments, and provides a general framework that is both easily extendable and does not require any additional annotations. We present an initial evaluation dataset, Book-SORT, comprising 36 36 36 36 k pairs of segments extracted from 9 9 9 9 books recently added to the public domain. Based on a human experiment with 155 155 155 155 participants, we show that humans can recall sequence order based on long-term memory of a book. We find that models can perform the task with high accuracy when relevant text is given in-context during the SORT evaluation. However, when presented with the book text only during training, LLMs’ performance on SORT falls short. By making it possible to evaluate more aspects of memory, we believe that SORT will aid in the emerging development of memory-augmented models. † † Code: https://github.com/bridge-ai-neuro/SORT † † Dataset: https://huggingface.co/datasets/memari/booksort

## 1 Introduction

Large language models (LLMs) have impressive performance on many benchmarks that test factual or semantic knowledge learned during training or in-context (Hendrycks et al., ; Ryo et al., ; Logan IV et al., ; Petroni et al., ; Yu et al., ; Sun et al., ).
While these advances are noteworthy, the type of long-term knowledge that these datasets test is only one of several types that naturally intelligent systems store, retrieve, and update continuously over time (Norris, ; Izquierdo et al., ; McClelland et al., ). Current evaluation tasks do not assess episodic memory, which is a form of long-term knowledge thought to be important for cognitive function in humans and animals. In contrast to semantic memory, episodic memory links memories to their contexts, such as the time and place they occurred. This ability to organize memory based on spatial and temporal details enables us to reconstruct events that occurred in the possibly distant past, predict the future, and relate information across multiple events that are separated by time windows spanning a lifetime, capabilities crucial for many cognitive tasks and everyday functions.

The ability to link temporal context to stored information may be key to improving LLM performance on several tasks. More human-like episodic memory may improve models’ continual learning and adaptation to shifting data distributions, performance on tasks requiring long contexts (e.g., long chat exchanges with a user), and source attribution via knowledge of where and when a memory was acquired, which could help to reduce or identify hallucinations.

To address the gap in evaluating memory in LLMs, we propose the Sequence Order Recall Task (SORT), which we adapt from tasks in cognitive psychology that are used to assess long-term episodic memory in humans and animals (Eichenbaum, ; Davachi & DuBrow, ). Specifically, SORT requires a model to recall the correct order of sequential data, such as segments of text.

We provide a specific instantiation of SORT that requires models to recall the correct order of two segments sampled from text, along with a corresponding evaluation dataset–Book-SORT. Book-SORT contains over $36$k pairs of text segments from $9$ books, with variations in segment length ($20$ and $50$ words) and distance between segments (up to $16$k words). We chose books that were very recently released from U.S. copyright to minimize the possibility that LLMs were pre-trained on these texts. This allowed us to test three common methods of giving a language model access to a specific text: (1) during inference in-context, (2) during inference via retrieval augmented generation (RAG), and (3) during training via fine-tuning with a language modeling objective. Furthermore, we provide a human evaluation from $155$ participants who had finished reading a whole book and were tested with no additional access to the book, showing that humans can recall segment order with up to $70\%$ accuracy based on their long-term memory of the book. While the ceiling performance on SORT is $100\%$ (assuming that texts do not contain duplicate segments), our human data provides an important reference point to compare and contrast long-term memory across models and humans.

When given access to excerpts from the books in-context, we find that models achieve up to $95\%$ accuracy with relevant $250$-word excerpts but degrade quickly as longer excerpts are presented. When models use RAG instead, they can recall sequence order only with limited performance below $65\%$. Finally, models fine-tuned with a language modeling objective on the book texts do not significantly improve their SORT performance, showing that parametric memory in current transformer models supports semantic but not episodic long-term memory.

Our main contributions can be summarized as follows:

- •
proposal of the self-supervised task SORT, which requires LLMs to recall the correct order of segments from a sequence and can be used to assess capabilities in LLMs that would be supported by episodic memory in humans
- •
a new dataset Book-SORT composed of $36$k samples from $9$ public domain books and an evaluation framework that is easily extendable to new datasets
- •
first-of-its-kind human evaluation ($N=155$) showing that humans are capable of recalling the order of text from an entire book based on long-term memory
- •
a comprehensive evaluation of open-source and closed language models on Book-SORT, showing that current models: i) have good in-context memory performance, when all necessary information is presented in the prompt and the prompt is short; ii) quickly lose the ability to recall sequence order as the excerpt provided in-context gets longer, even though the excerpt still easily fits within the context window; (iii) fail to recall segment order based on parametric memory formed via fine-tuning with a language modeling objective; (iv) perform worse on SORT with retrieval augmented memory than with in-context memory.

## 2 Related Work

Evaluation of parametric semantic memory in LLMs.
Benchmarks such as MMLU (Hendrycks et al., ), T-REx (Elsahar et al., ), LAMA (Petroni et al., ), WICE (Ryo et al., ), KoLA (Yu et al., ), and others (Sun et al., ) test models’ retrieval and reasoning ability on different domains, such as recalling a chemistry fact.

Other benchmarks that partially evaluate LLM semantic memory are those that require reasoning using temporal (Ning et al., ; Zhou et al., ; Feng et al., ) (e.g. lunch happens before dinner), causal (Srivastava et al., ) (e.g. she is eating, therefore she is hungry), or other commonsense knowledge (e.g. food is edible) (Ismayilzada et al., ) acquired during pretraining. In contrast to these benchmarks, our work proposes a task that involves judgments regarding temporal context information about text segments that either (a) are available through in-context memory or (b) were otherwise previously presented to the model, e.g. via fine-tuning or Retrieval Augmented Generation, and is agnostic of the specific semantic content of these segments.

Evaluation of in-context memory in LLMs. Among other conditions, we evaluate in-context memory, in which the model has in-context access to all relevant text for the task. This relates to works that evaluate a model’s ability to reason over its context input, such as Needle In A Haystack (Kamradt, ) and FLenQA (Levy et al., ).

Previous datasets and benchmarks that evaluate performance over long context lengths, such as Long Range Arena (Tay et al., ), SCROLLS (Shaham et al., ), and MULD (Hudson & Al Moubayed, ), are also relevant.
The evaluation of in-context memory with SORT differs from these works by focusing on order information, which is key to episodic memory in humans. Additionally, we use SORT to evaluate parametric memory which contains information beyond the current context.

Tasks related to SORT. Previously proposed tasks that most closely relate to SORT are BART’s denoising training objective (Lewis et al., ), which permutes the order of sentences in a document and learns to reconstruct the correct order, and BERT’s next sentence prediction objective (Devlin et al., ), which learns to predict whether two sentences follow each other in a text. SORT differs from these tasks, as it is not intended as a training objective, and it can include text segments with an arbitrary distance between each other in a document, possibly exceeding the context input length of the model.
In ChapterBreak (Sun et al., ), long segments ending at a chapter boundary taken from a book are presented to an LLM along with multiple segments of chapter beginnings from the same book. The task for the LLM is then to tell which one is the directly following chapter and which are not. This suffix-identification task aims to evaluate narrative-understanding based reasoning about books, while we propose SORT as an evaluation for episodic memory in LLMs, involving both a model and a memory-insertion method. By evaluating a SORT baseline in which the models do not have access to relevant source texts, we show that memory is needed for SORT and general narrative-reasoning ability is not enough.

## 3 Sequence Order Recall Task

We introduce a novel evaluation task: recalling the order of parts of a sequence, which we term the Sequence Order Recall Task (SORT). SORT is adapted from recency judgment tasks used in cognitive psychology to evaluate episodic memory in humans and animals (Eichenbaum, ; Davachi & DuBrow, ). In this task, a sequence is presented to a participant. Then, after some delay, the participant is asked to judge the order in which two segments of the sequence appeared. We adapt this task to test memory in models. The general task can be applied to any sequential domain, including video and audio. Here we focus on the text domain to evaluate LLMs (Fig. 1).

Figure: Figure 1: Overview of the Sequence Order Recall Task (SORT) to evaluate how models can access memory of temporal order. Left: Example task prompt for SORT. A prefix to the prompt can be given to assess in-context forms of memory. Right: Methods to insert memory of specific texts into a model.
Refer to caption: x1.png

Formal description of SORT.
The general form of the task can be described as follows. Let $\mathbf{X}\in\mathbb{R}^{T\times F}$ be sequential data, where $\bf{T}$ is the number of time-steps (e.g. token in a text) and $\bf{F}$ is the number of features (e.g. vocabulary size). We define start indices $\mathbf{t_{j}}$ and $\mathbf{t_{k}}$ for pairs of segments of length $\mathbf{L}\in\mathbb{N}^{+}$ in $\mathbf{X}$, such that both $\mathbf{t_{j}<t_{k}}$ and $\mathbf{t_{j}+L\leq t_{k}}$. Using these, we extract non-overlapping segments from the original sequence $\mathbf{X}$ as $\mathbf{\widetilde{X}_{i}=X[t_{i}:t_{i}+L-1,:]}$. The order of segments $\mathbf{\widetilde{X}_{j}}$ and $\mathbf{\widetilde{X}_{k}}$ is randomized, yielding $\mathbf{[\widetilde{X}_{A}\;\widetilde{X}_{B}]}$, which is then given as part of a model’s input. The task for a model $\mathbf{\mathcal{M}_{\theta}}$ is to infer whether $\mathbf{t_{A}<t_{B}}$, i.e. in SORT, the task of a model is to predict which of two non-overlapping subsequences $\mathbf{\widetilde{X}_{A}}$ and $\mathbf{\widetilde{X}_{B}}$ has the lower starting index in $\mathbf{X}$. The task can be used to evaluate a variety of methods to include document-specific memory in models. To assess in-context memory, i.e. memory based on text presented in-context, the segments are preceded by $\mathbf{X}$ in the model’s input. When assessing retrieval-augmented generation methods, instead of prepending $\mathbf{X}$, passages of $\mathbf{X}$ are retrieved and prepended. For the assessment of parametric long-term memory, $\mathbf{X}$ is not part of a model’s input, instead the model’s parameters $\mathbf{\theta}$ are a function of $\mathbf{X}$ via pre-training or fine-tuning: $\theta=f(\mathbf{X})$.

The general form of SORT is the following input, which can be preceded by additional context to insert a memory:

$$ $\displaystyle I_{SORT}=[P_{context}\;P_{task}\;P_{label_{A}}\;\mathbf{ \widetilde{X}_{A}}\;P_{label_{B}}\;\mathbf{\widetilde{X}_{B}}\;P_{question}P_{ answer}],$ (1) $$

where $\bf{P_{context}}$ can either be relevant context, such as (parts of) the source sequence $\mathbf{X}$ to assess in-context memory (stored in activation slots), or an empty string when parametric memory (stored in weights) is assessed; $\bf{P_{task}}$ instructs the model for the sequence order recall task to read two segments and describes the objective: answering which of the two labeled segments appears first in $\mathbf{X}$; $\bf{P_{label_{A}}}$ and $\bf{P_{label_{B}}}$ are the labels (e.g. the characters “A” and “B”) for the first and second segment presented in the task $\mathbf{\widetilde{X}_{A}}$ and $\mathbf{\widetilde{X}_{B}}$; $\bf{P_{question}}$ repeats the SORT objective as a question; finally, $\bf{P_{answer}}$ provides the beginning of the answer string as “Answer: Segment”.

### 3.1 Evaluating Large Language Models on sort

We greedily sample an answer token $\mathbf{a=argmax(\mathcal{M}_{\theta}(I))}$ from the model $\mathbf{\mathcal{M}}_{\theta}$, which is parameterized by $\mathbf{\theta}$, and decode the sampled answer token $\mathbf{a}$ as either "A" or "B".

The answer is evaluated as correct if it corresponds to the segment that truly appears first in $\mathbf{X}$.
For proprietary (OpenAI) models that do not allow completing assistant responses with prepended text, we omit $\bf{P_{answer}}$. In this case we resort to generating a sequence of 25 tokens, and parse the generated text for A or B responses.

Prompt selection. Using a single prompt formulation across all models may bias the results. To prevent this, we compiled a set of $12$ prompts that vary formulations in $\bf{P_{context}}$ and $\bf{P_{task}}$. For each model, we evaluate each prompt on a held-out dataset of 400 samples and used the best performing prompt for each model. The full prompts and further details on prompt selection are given in Appendix B.2-B.3.

Baseline without book-specific memory. We want to ensure that performance on SORT is due to text-specific memory and not due to temporal order reasoning supported by more semantic forms of memory such as commonsense knowledge (e.g. lunch happens before dinner). We isolate the effects on SORT that are due to text-specific memory by contrasting performance between a baseline model that does not have access to the specific text and a model that has access to the sequences in one of various ways in which memory can be inserted.

### 3.2 Inserting text-specific memory into models

We evaluate three methods to insert text-specific memory into models: (1) via in-context presentation, (2) via fine-tuning with a language modeling objective, and (3) via retrieval augmented generation of short chunks of text in a book.

In-context presentation. When assessing in-context memory, $\bf{P_{context}}$ in Eq. 1 contains relevant excerpts from the source text along with the book title. The prompt includes the instruction to carefully read the text from the book (a list of used prompts is shown in Appendix LABEL:appendix:prompt-list). To test in-context memory, We make sure that excerpts contain both segments and vary the length of excerpts in our experiments.

Finetuning with a language modeling objective. Instead of presenting text from the books in the same prompt in which the SORT task is given, we are interested in parametric memory of the texts. In this condition, $\bf{P_{context}}$ in Eq. 1 is an empty string. To insert parametric memory of the source texts into a model, we fine-tune the model with a next-token prediction objective on the books, split into chunks of 5000 words and contextualized by the books’ titles. Since we need to preserve the models’ ability to understand and follow the task instructions, we fine-tune on a dataset that additionally includes 3,500 random instruction-following examples that are unrelated to SORT. This helps to prevent catastrophic forgetting during continued finetuning (Luo et al., ). We finetune on 8 A100 GPUs with an initial learning rate of 5e-6 and a batch size of 192. Full details of the fine-tuning setup are given in Appendix E and our code will be available.

Retrieval Augmented Generation.
To include memory of text via retrieval augmented generation (RAG), we built a typical naive RAG pipeline that relies on two separately pretrained models for the retriever and the reader (Gao et al., ). The retriever returns text passages from a database to serve as task context for the LLM (i.e. as $\bf{P_{context}}$, Eq. 1).

The retrieval database contained text embeddings of all passages from Book-SORT (Sec. 4). We used the LangChain recursive text splitter to chunk Book-SORT text into $\sim$1024 character, non-overlapping passages (average 183 words). Each passage was then encoded into a 1024-d vector using a high-performing, open-source text retrieval model (BGE-v1.5, Xiao et al. ()). To retrieve the passages, we used the Faiss (Douze et al., ) library to conduct an exact nearest neighbor search. The search returned the $k=2$ nearest neighbors. We maintained this similarity order when inserting the retrieved passages into the prompt, i.e. the most similar passage appears first in $\mathbf{P_{context}}$. As described in Section 3.1, we selected a single prompt for each model based on the model’s performance on the held-out validation set across 10 different possible prompts (see Appendix B.4).

## 4 Book-sort Dataset and Evaluation

We created an English language dataset to evaluate episodic memory in humans and LLMs. The selected sequence data considered several factors: (1) we chose long texts (mean length = 72,700 words) that exceed the context windows of most transformer LLMs; (2) we used books to enhance memorability for human readers and facilitate our human evaluation experiment; (3) we selected books from *Project Gutenberg* that recently entered the U.S. public domain to avoid ethical and copyright issues, and minimize pre-training contamination in LLMs. Within these constraints, we aimed to maximize content diversity, including narrative fiction novels, a physics text, and an extended essay. Further details on the $9$ books in the Book-SORT dataset are available in Appendix A.1.

### 4.1 Book-SORT Creation

We constructed a dataset that varies across factors that can affect human or model performance on SORT. Based on prior reports on LLMs (Liu et al., ), we first varied (1) $L_{E}$, the length of the text excerpt presented in context. Since the typical standard context length of the LLMs in our study was 4096 tokens, we set $L_{E}=\{250,1000,2500\}$ words. For models with extended context windows, we also created datasets where $L_{E}=\{10000,20000\}$ words, which excluded one book that was too short. Our pilot experiments on humans suggested two other factors that would affect task performance: (2) $L_{S}$, the length of the segments from the text, and (3) $D_{S}$, the distance between the segments in the original text. To mirror the human experiments, we set $L_{S}=\{20,50\}$ words. We then created 4 different distance bins $D_{S}=\{d_{0},d_{1},d_{2},d_{3}\}$, whose values were bounded by the excerpt length $L_{E}$ (Appendix Table 4).

Within each unique combination of the first two factors $L_{E}$ and $L_{S}$, we randomly sampled 110 excerpts from each of the 9 books (i.e. 100 samples for SORT evaluation, and 10 samples for prompt selection per book). All excerpts and segments began at a sentence boundary. Within each combination of $L_{E},L_{S}$, we randomly sampled 4 different segment pairs, one from each distance bin $D_{S}$. This minimized the possibility that observing an effect of distance on SORT performance would be due to differences in the semantic content of the text segments. Finally, for all 110 trials within each of these 3 factors, we counterbalanced the correct answer. This yielded a well-controlled and easily extendable dataset of about $36K$ text segment pairs for SORT evaluation.

### 4.2 Human long-term memory evaluation

As a reference point (but not a performance ceiling), we further provide a human evaluation from $155$ participants who had recently finished reading one of the $9$ books in the Book-SORT dataset, *The Murder of Roger Ackroyd* (Christie, ). This evaluation assessed long-term memory, as the average time between reading and testing was $7.5$ days, far surpassing short-term memory duration (Hasson et al., ). There is no previously reported data on long-term memory for entire books from large samples, so we designed an experiment to collect this data. Given the difficulty of recruiting participants to read lengthy books specifically for an experiment, we used a creative recruiting strategy: inviting members of the online reading community *Goodreads* who had recently finished *The Murder of Roger Ackroyd*. Participants completed an online survey within $30$ days of finishing the book. The expected compensation for participation was $\$12$ and the study was approved by the IRB at Anonymized University. We provide 1570 segment pair samples from 155 participants. Further details about this one-of-a-kind study are provided in Appendix A.3.

### 4.3 Models

We evaluate a selection of open models covering a broad range of scores on popular benchmarks such as MMLU (see Table 5) ranging from 7b to 8x22b parameter transformer models. Initial experiments with non-instruction-tuned models resulted in chance performance on Book-SORT (see Appendix D), which we attribute to the lack of instruction tuning(^1^11 (Zhang et al., ) provides an overview of instruction tuning approaches), and thus focus on evaluating instruction-tuned models in this work. We have selected models from different model families including Llama3 (AI@Meta, ), Llama2 (Touvron et al., ), Mistral (Jiang et al., ), Mixtral (Jiang et al., ), Gemma (Team et al., ) and OpenAI GPTs (Achiam et al., ). For our experiments on finetuning as a method for inserting memory into models, we focus on two models Mistral-v0.2-7b-Instruct and Llama3-8b-Instruct because they allow full-parameter fine-tuning with 8 A100 GPUs.

## 5 Results

We present empirical findings for a baseline without text-specific memory of the books in Book-SORT, as well as three methods to include memory, using 9 open-source models and 2 closed language models.

### 5.1 Baseline

SORT requires memory specific to books in Book-SORT. To validate that it is not possible to achieve high performance on Book-SORT without memory of the specific books that are included in the dataset, we evaluate models before they have access to the books. We find that segment pairs with a very short and with a very long distance in the book allow for above-chance-performance (see Appendix C.1), indicating that some of these segment pairs can be ordered based not on memory but rather on temporal-order reasoning or common-sense. However, performance is below 60% for all models and segment lengths, confirming that SORT requires memory for the particular books being queried to yield high levels of performance.

**Table 1: Baseline: SORT performance before models are exposed to the books in Book-SORT.**
|  | Segment length 20 | Segment length 50 |
| --- | --- | --- |
| Llama3-70b-inst | $0.52\pm 0.007$ | $0.54\pm 0.007$ |
| Llama3-8b-inst | $0.51\pm 0.008$ | $0.52\pm 0.007$ |
| Mixtral-8x22b-inst | $0.52\pm 0.007$ | $0.55\pm 0.007$ |
| Mixtral-8x7b-DPO-inst | $0.52\pm 0.008$ | $0.54\pm 0.008$ |
| Llama2-70b-inst | $0.51\pm 0.007$ | $0.51\pm 0.008$ |
| Gemma-1.1-7b-inst | $0.51\pm 0.008$ | $0.51\pm 0.007$ |
| Mistral-v0.2-7b-inst | $0.51\pm 0.007$ | $0.51\pm 0.008$ |
| Mistral-v0.1-7b-inst | $0.50\pm 0.008$ | $0.50\pm 0.008$ |
| Llama2-7b-inst | $0.50\pm 0.008$ | $0.49\pm 0.008$ |
| GPT-3.5-turbo | $0.52\pm 0.009$ | $0.52\pm 0.012$ |
| GPT-4 | $0.53\pm 0.008$ | $0.57\pm 0.007$ |

Figure: Figure 2: Human long-term memory performance on SORT for different segment lengths and distances between segments. Shaded areas depict bootstrapped 95% confidence intervals. Significant difference from chance is marked with asterisks (^∗p-value$<$0.05,^∗∗p-value$<$0.01).
Refer to caption: x2.png

### 5.2 Human Experiment

Humans can perform in SORT based on long-term memory. The results from human long-term memory (LTM) experiments, depicted in Figure 2, demonstrate that humans can perform in SORT based on long-term memory. The average accuracy is $0.64$ for segments of 50 words and $0.56$ for segments of 20 words).
Human performance is higher for pairs of segments that have a greater distance in the book, with a peak accuracy of $0.76$ for distances greater than 25,000 words and 50-word segments. Binomial tests show that beyond a distance of 4000 words, humans perform statistically significantly better than chance. Note that we present these results as evidence that one possible information processing system–a human–can perform SORT based on long-term memory. Importantly, these results do not present the ceiling performance on the memory task that we propose. The expected ceiling performance on SORT is 100%, assuming that the books do not contain duplicated segments of text; the odds of exact duplication decrease as segment length increases.

### 5.3 In-context memory

Models generally perform well on SORT based on in-context memory. Nearly all models achieve above 77% accuracy when given in-context access to relevant excerpts from the books, reaching up to 95% (Table 2). This indicates that very large models are not necessary to perform this task effectively, as demonstrated by the Llama3-8b model outperforming larger models such as Llama3-70b and Mixtral-8x7b-DPO.

In-context memory performance increases with greater distance between segments. We further evaluate the effect of another factor which may influence the model performance–the distance between the text segments in the excerpt. Figure 3(b) shows an increasing trend in accuracy as the distance between segments increases. This improvement in accuracy, which we also observed in our human experiment (Fig. 2), is consistent across excerpt lengths and is observed across all models (see Appendix C.2).

In-context memory performance decreases with increasing excerpt length. Average performance on longer excerpts (Table 2, SORT-extend) is substantially lower than in the standard context lengths, despite the presence of longer segment distances.
For increasing excerpt lengths, we see a consistently monotonic decrease in average accuracy (Figures 3(a) and 3). This is consistent with previous findings on length generalization in LLMs (Liu et al., ; Levy et al., ; Hsieh et al., ).

Additional analyses. Further analyses are presented in Appendix C.2. Like humans, models handle longer segments (50 words) slightly more effectively than shorter segments (20 words), with an improvement of up to $4\%$. We found no significant differences across books from different domains (Table 11-12).

**Table 2: Mean of in-context memory performance with 95% bootstrapped confidence interval. SORT-extend shows performance with excerpts of lengths 10000 and 20000 words, which exceeds most models’ context lengths.**
| Model name | Parameters | Max context | SORT | SORT-extend |
| --- | --- | --- | --- | --- |
| Llama3-70b-inst | 70b | 8k | $0.92\pm 0.020$ | / |
| Llama3-8b-inst | 8b | 8k | $0.93\pm 0.007$ |  |
| Mixtral-8x22b-inst | 8x22b | 64k | $0.95\pm 0.020$ | $0.79\pm 0.038$ |
| Mixtral-8x7b-DPO-inst | 8x7b | 32k | $0.89\pm 0.030$ | $0.56\pm 0.058$ |
| Llama2-70b-inst | 70b | 8k | $0.77\pm 0.040$ |  |
| Gemma-1.1-7b-inst | 7b | 8k | $0.85\pm 0.010$ |  |
| Mistral-v0.2-7b-inst | 7b | 32k | $0.85\pm 0.032$ | $0.65\pm 0.045$ |
| Mistral-v0.1-7b-inst | 7b | 8k | $0.77\pm 0.013$ |  |
| Llama2-7b-inst | 7b | 4k | $0.56\pm 0.014$ |  |
| GPT-3.5-turbo | unknown | 16k | $0.86\pm 0.010$ | / |

Figure: (a) By excerpt lengths
Refer to caption: x3.png

### 5.4 Parametric Memory via Finetuning

Full parameter fine-tuning on books with a language modeling objective did not improve SORT performance. For Llama3-8b-Instruct and Mistral-7b-v0.2-Instruct, we do not observe any difference in performance on SORT after memory is inserted via fine-tuning on large chunks of book-text. A pairwise statistical analysis across epochs of fine-tuning, relative to two baselines that either exclude the books from the fine-tuning dataset or instead include only summaries of the books, shows no substantial improvement (see Appendix E).

### 5.5 Retrieval Augmented Memory

RAG based memory leads to worse performance than in-context memory. RAG performance is between 55% and 67% for all distances between segments and tested models
(Figure 4(a)), which is substantially lower than the in-context memory performance. This difference in performance follows from the fact that standard forms of RAG do not necessarily preserve the order of retrieved passages, whereas the excerpt provided for in-context memory does have the passages in the correct order (and additionally contains the text that connects the passages, which may help in making the order judgment). When the relevant passages are retrieved and presented in the correct order, RAG performance improves substantially. Interestingly, we find that Llama3-8b-Instruct model outperforms the much larger Mixtral-8x22b-Instruct and Llama3-70b-Instruct on SORT with an accuracy around $90\%$ across all distances between segments (Figure 4(b)).

Figure: (a) Vanilla RAG
Refer to caption: x5.png

## 6 Discussion

We provide a new evaluation task, SORT, for assessing episodic memory in large language models, that can be used with any text data and without the need for annotation. We created Book-SORT, a dataset for SORT based on books that were recently added to the public domain and we validated that book-specific memory is indeed needed to achieve high performance on Book-SORT. We evaluated three different ways to include memory of specific texts in a model to assess whether they support a key function of episodic memory. Below, we discuss our results for these methods in relation to episodic memory in humans.

Is in-context memory a form of episodic memory? Several links have been drawn between in-context memory in transformers and models of episodic memory in humans (Ji-An et al., ; Whittington et al., ; ; Ellwood, ), and our results, which show that in-context memory supports sequence order recall, could be interpreted as further evidence for in-context memory acting as episodic memory in LLMs.
However, our results also show that in-context sequence order recall performance degrades with increasing context length, which would not be the case with episodic memory.
This discrepancy stems from a key difference between in-context memory in models and episodic memory in humans and animals, which is that in-context memory in LLMs can directly attend to all tokens in the context window, whereas the episodic memory system in humans and animals stores past experiences in synaptic form, and requires an additional retrieval step before episodic memory content can be attended to. The reliance on synaptic storage and retrieval is what enables the episodic memory system in humans and animals to make use of a sequence-length invariant mechanism with a fixed computational cost to remember past experiences over a lifetime. This sequence-length invariant property of the episodic memory system in humans and animals allows it to generalize to arbitrarily long sequences, while attention over all tokens in a growing sequence eventually leads to generalization failure for in-context memory and, at the same time, comes with a sharply increasing computational cost.
Based on these considerations, we believe that, although both the episodic memory system in animals and in-context memory in transformer models perform a kind of similarity-based lookup of past experiences, in-context memory’s access to activations is more analogous to working memory in humans (O’Reilly et al., ), but with a capacity that vastly exceeds human working memory.

Is parametric memory in transformers a form of episodic memory? High performance on benchmarks including MMLU suggests that parametric memory in LLMs learned via a language modeling objective can support semantic forms of memory (e.g. when recalling knowledge to answer factual questions). Our evaluation on SORT showing close to chance performance after finetuning suggests that current forms of parametric memory do not support functions similar to those of episodic memory. This suggests that different learning methods and architectures (e.g. with a separate memory system) may be needed for functioning parametric forms of episodic memory.

Is retrieval augmented memory a form of episodic memory? Since it avoids the problems of context-length generalization and increasing computational costs observed for in-context memory, Retrieval Augmented Generation presents a potentially strong way to include memory of episodes via a retrieval process and subsequent in-context presentation. However, our results suggest that there is a lot of room for improvement over the performance of vanilla RAG. The weak performance of vanilla RAG on SORT arises from the fact that it is decontextualized–all that it retrieves is independent parts of the text. By contrast, current theories of episodic memory posit that episodic memory contents are bound to a drifting temporal context; later, when some content is retrieved, the temporal context associated with that content is also retrieved (Howard & Kahana, ; Polyn et al., ). One aspect of retrieved temporal context–absent from vanilla RAG–is required for sequence order recall. Order-preserving (OP) variants of RAG (Yu et al., ) can increase performance on SORT, as suggested by our results shown in Figure 4(b). However, OP-RAG maintains contextual information only about the sequential order, and it does not bind any other temporal context to the independently retrieved passages. The core difference between current RAG systems and episodic memory remains: they do not present a method to bind temporal context to the content of memories.

Limitations. Current high performing LLMs do not disclose their training data, which means that care needs to be taken in selecting suitable data to include in a SORT dataset. To minimize the probability that models have been trained on books used for our SORT evaluation, we curated Book-SORT based on books that were not publicly available when models were trained. However we cannot rule out the possibility that the books in this set were used in training of a model, which (if true) would require us to interpret results as indicating the effectiveness of additional rather than initial memory-insertion. Furthermore the reliance on instruction-following can limit the applicability to both non-instruction-tuned models and models that have poor instruction-following ability.
Lastly, we provided two examples of more long-term memory-insertion via fine-tuning and Retrieval Augmented Generation for two models, Llama3-8b-Instruct and Mistral-7b-v0.2-Instruct, and leave more extensive studies on how to induce episodic memories without relying on complete in-context presentation to future work.

Future work. Improving long-term memory in LLMs is an emerging area of research (Liu et al., ; Borgeaud et al., ; Fournier et al., ; Phang et al., ; Wang et al., ; Zhong et al., ; ), and SORT can be used to assess improvement in an crucial aspect of an important form of memory in new models. Specifically, improving episodic memory in models may improve models’ continual learning, performance on tasks at long contexts such as extended chat exchanges with a user, and source attribution via knowledge of where and when a memory was acquired. Recent efforts have highlighted the potential of augmenting LLMs with additional episodic memory mechanisms (Fountas et al., ; Das et al., ), and we expect that SORT can be used to evaluate these classes of models, once such a model with a sufficiently strong instruction-following ability is released. Another possibility is to identify new and better methods to insert episodic memory of texts into existing models. Additionally, SORT can be extended to other types of inputs, such as audio and video, which can be used to evaluate episodic memory in multimodal models in the future.

Conclusion. The ability of LLMs to retain and retrieve long-term knowledge is crucial for their continued integration in many applications. Therefore, a more comprehensive and systematic evaluation of these abilities is needed. We believe that the new evaluation framework SORT offers a promising path for future research aimed at better understanding and improving these capabilities in foundation models.

Ethics Statement. To avoid ethical issues concerning copyright, we based Book-SORT on books that were recently added to the public domain. Our human experiment with 155 participants was approved by the IRB at Anonymized University and participants were compensated.

Reproducibility Statement. We will publicly release the Book-SORT dataset as well as all code to generate new SORT datasets and evaluate models on SORT. For open models, evaluation on Book-SORT is deterministic due to greedy sampling and the use of an answer prefix.

Acknowledgements
Funded in part by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – GRK 2853/1 “Neuroexplicit Models of Language, Vision, and Action” - project number 471607914.

## Appendix A Additional details on Book-SORT data set

Preprocessing book text. We wrote custom Python code to only retain the book text that formed a continuous narrative. We stripped the front and back matter of the book, and extracted chapter titles if they existed. 8 of the 9 books contained individual section or chapter breaks. For these 8 books, we parsed the text corresponding to each chapter. Chapter titles or section headings (e.g. ‘VI’ to indicate section six) were removed, and all remaining text was concatenated. This string was split into words (assuming simple whitespace separators with Python string.split()) to produce a final text array for each book. This text array was sampled for the Book-SORT dataset.

### A.1 Book selection

We provide details about the $9$ books in Book-SORT in Table 3.

**Table 3: Project Gutenberg metadata on Book-SORT books.**
| ID | Title | Author | Word count | Release | Pub | LoCC* | Subjects |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 69087 | The Murder of Roger Ackroyd | Christie, Agatha | 69,720 | 10/2/2022 | 1926 | PR | Detective and mystery stories; Fiction: Private investigators - England, Murder - Investigation, Belgians - England |
| 72578 | Tom Swift and His Talking Pictures | Appleton, Victor | 43,853 | 1/1/2024 | 1928 | PZ | Adventure stories; Motion pictures |
| 72600 | The Trumpeter of Krakow | Kelly, Eric Philbrook | 59,081 | 1/2/2024 | 1928 | PZ | Juvenile fiction: Middle Ages, Poland - History - Casimir IV, 1447-1492 |
| 72869 | Meet the Tiger | Charteris, Leslie | 79,946 | 2/4/2024 | 1928 | PR | Fiction: Private investigators - England; Detective and mystery stories |
| 72958 | Hunting for Hidden Gold | Dixon, Franklin W. | 42,354 | 2/14/2024 | 1928 | PZ | Juvenile fiction: Brothers, Gold mines and mining, Montana, Robbers and outlaws; Mystery and detective stories |
| 72963 | The Nature of the Physical World | Eddington, Arthur Stanley, Sir | 104,530 | 2/15/2024 | 1928 | Q | Physics - Philosophy; Science - Philosophy |
| 72972 | Money for Nothing | Wodehouse, P.G. (Pelham Grenville) | 82,331 | 2/16/2024 | 1928 | PR | Humorous stories; Fiction: Swindlers and swindling, Greed |
| 73017 | Pomona; or, the Future of English | De Selincourt, Basil | 9,273 | 2/22/2024 | 1928 | PE | English language |
| 73042 | The Well of Loneliness | Hall, Radclyffe | 163,217 | 2/26/2024 | 1928 | PR | Fiction: Lesbians - England - Social conditions |

*LoCC = Library of Congress classification.

### A.2 Between-segment distances

The segment distance $L_{S}$ for Book-SORT is sampled from one of four distance bins. The right edge of each bin is given in Table 4. Distance is computed between the beginning of the first segment and the beginning of the second segment. The minimum distance $L_{S}$ therefore produces adjacent, non-overlapping segments.

**Table 4: Right edge of each distance bin used to create samples for Book-SORT.**
|  | Minimum | Bin0 | Bin1 | Bin2 | Bin3 |
| --- | --- | --- | --- | --- | --- |
| $L_{E}\leq 2,500$ | $L_{S}$ | $L_{E}/4$ | $L_{E}/3$ | $L_{E}/2$ | $L_{E}/0.8$ |
| $L_{E}\geq 10,000$ | $L_{S}$ | 1000 | $L_{E}/4$ | $L_{E}/2$ | $L_{E}/0.8$ |

### A.3 Human study details

##### Participant compensation.

Participants were compensated via a lottery system with a chance to win a gift card to a popular book store. The expected value of the compensation came out to $\$12$ per hour.

##### Study design.

Each participant completed an online survey. First, the participant consented to the study, read a brief set of instructions, and completed a brief survey, including a question regarding when the participant finished reading the book. The complete set of survey questions is listed below. Each participant was then asked to answer "Which segment occurred first in the book?" for $10$ randomly chosen text segment pairs from a total set of $540$ unique segment pairs sampled from the whole book. We chose to present a sample number of trials to each participant to minimize interference effects from repeated memory retrieval (Kliegl & Bäuml, ). The presentation order of the text segments was randomized across participants. In the end, each participant was asked $4$ simple questions about the book plot to verify that the participant had indeed read the book. Each participant was only allowed to participate in the study once.

##### Demographics questions.

The human participants were asked the following set of demographics questions before beginning the experiment:

- 1.
I have finished the book The Murder of Roger Ackroyd [Options: True/False]
- 2.
On what date did you finish the book? [Calendar question type]
- 3.
Did you read or listen to the book? [Options: Read/Listen]
- 4.
Was this your first time reading / listening to the book? [Options: Yes / No]
- 5.
What is your age? [Options: 18-25, 25-35, 35-45, 45-55, 55-65, 65+]
- 6.
What gender do you identify with? [Options: Female/Male/Other]
- 7.
What is your experience with the English language? [Options: Native / Fluent / Advanced / Intermediate / Beginner]
- 8.
How many books did you read or listen to in the past year? [Options: 1-2 / 3-5 / 6-10 / 10+]

We use the responses above to determine the number of days that have passed since finishing the book, and make this information available in the human dataset together with the responses.

##### Inclusion criteria.

We include data from participants who answered at least $3$ of $4$ plot questions correctly, and finished reading the book within $30$ days of participating in the study. These inclusion criteria result in $155$ participants.

## Appendix B Model and prompting details

### B.1 Model details

We listed all models we used in this paper and their download links from HuggingFace in Table 5. For the OpenAI models, we used the gpt-3.5-turbo-0125 version of GPT-3.5, and gpt-4-turbo-2024-04-09 for GPT-4. Models were selected to cover a broad range of performance on more semantic/knowledge-based tasks such as those included in MMLU.

**Table 5: Model Details**
| Name in HuggingFace | Name in Paper | MMLU score |
| --- | --- | --- |
| [Llama-3-70B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-70B-Instruct) | Llama3-70b-inst | 80.06 |
| [Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct) | Llama3-8b-inst | 66.60 |
| [Mixtral-8x22B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x22B-Instruct-v0.1) | Mixtral-8x22b-inst | 77.77 |
| [Nous-Hermes-2-Mixtral-8x7B-DPO](https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPO) | Mixtral-8x7b-DPO-inst | 72.28 |
| [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | Mistral-v1-7b-inst | 60.10 |
| [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | Mistral-v2-7b-inst | 60.07 |
| [Llama-2-70b-chat](https://huggingface.co/meta-llama/Llama-2-70b-chat) | Llama2-70b-inst | 68.90 |
| [Llama-2-7b-chat](https://huggingface.co/meta-llama/Llama-2-7b-chat) | Llama2-7b-inst | 45.30 |
| [gemma-1.1-7b-inst](https://huggingface.co/google/gemma-1.1-7b-it) | Gemma-1.1-7b-inst | 64.30 |

### B.2 Prompting

For our experiments with Book-SORT, we created a total of 12 prompts that are composed of two parts. The prompts differ in how they phrase the tasks. The first part contains instructions to read the text excerpt from the book as well as a placeholder for the actual excerpt. The second part of the prompt contains the description of SORT, including a mention of the book or document title as well as two segments from that document. We found that current open LLMs fail at the task even with in-context access to the text, if they are asked to tell which segment appeared second or last. For this reason, we ran all experiments with the placeholder <position> set to "first". All of these prompts were preceded by the same generic system prompt: "You are a helpful, respectful and honest assistant."

**Table 6: Selection of 12 prompts used for prompt validation**
| No. | Reading instruction | SORT instruction |
| --- | --- | --- |
| 1 | "Please take some time to thoroughly read and comprehend this extract from the book <booktitle>. The passage is as follows: <excerpt>" | "You will be shown pairs of text fragments from <booktitle>. Please select which of two fragments appeared <position> in the book. You will be shown 10 such pairs. <segments> Which fragment appeared <position> in the book, <label_0> or <label_1>?" |
| 2 | "I need you to thoroughly read and comprehend this extract from the book <booktitle>. The passage is as follows: <excerpt>" | "In this exercise, your objective is to identify the text segment, either <label_0> or <label_1>, that appeared <position> in <booktitle>. Please read the segments carefully to determine their order of appearance in <booktitle> and respond with either <label_0> or <label_1>: <segments> Which of these, <label_0> or <label_1>, was <position> in <booktitle>?" |
| 3 | "I need you to thoroughly read and comprehend this extract from the book <booktitle>. The passage is as follows: <excerpt>" | "Your task is to recall which text segment, either <label_0> or <label_1>, appeared <position> in the book <booktitle>. Please read the segments carefully to remember in which order they appeared in <booktitle> and respond with either <label_0> or <label_1>: <segments> Which of these, <label_0> or <label_1>, was <position> in the book <booktitle>?" |
| 4 | "I need you to thoroughly read and comprehend this extract from the book <booktitle>. The passage is as follows: <excerpt>" | "You will be shown two text segments, labeled as <label_0> and <label_1>. Please recall in which order they appeared in the book <booktitle> and tell me which one came <position>. Please read the segments carefully: <segments> Which of these two parts of the book, <label_0> or <label_1>, came <position> in the book <booktitle>?" |
| 5 | "I need you to thoroughly read and comprehend this extract from the book <booktitle>. The passage is as follows: <excerpt>" | "I will show you two short parts from a book, labeled as <label_0> or <label_1>. Your task is to tell me which of them appeared <position> in the book <booktitle>. Please read both segments carefully and try to remember where in the book they come from: <segments> Which of these, <label_0> or <label_1>, appeared <position> in the book <booktitle>?" |
| 6 | "I need you to thoroughly read and comprehend this extract from the book <booktitle>. The passage is as follows: <excerpt>" | "This is your task: Given two segments from a book, labeled as <label_0> and <label_1>, please tell me which of them appeared <position> in <booktitle>. Read both segments carefully and try to remember where in <booktitle> they appeared: <segments> Which of these, <label_0> or <label_1>, comes <position> in the book <booktitle>?" |
| 7 | "Please carefully read this excerpt from the book <booktitle>. This is the relevant passage: <excerpt>" | "You will be shown pairs of text fragments from <booktitle>. Please select which of two fragments appeared <position> in the book. You will be shown 10 such pairs. <segments> Which fragment appeared <position> in the book, <label_0> or <label_1>?" |
| 8 | "Please carefully read this excerpt from the book <booktitle>. This is the relevant passage: <excerpt>" | "In this exercise, your objective is to identify the text segment, either <label_0> or <label_1>, that appeared <position> in <booktitle>. Please read the segments carefully to determine their order of appearance in <booktitle> and respond with either <label_0> or <label_1>: <segments> Which of these, <label_0> or <label_1>, was <position> in <booktitle>?" |
| 9 | "Please carefully read this excerpt from the book <booktitle>. This is the relevant passage: <excerpt>" | "Your task is to recall which text segment, either <label_0> or <label_1>, appeared <position> in the book <booktitle>. Please read the segments carefully to remember in which order they appeared in <booktitle> and respond with either <label_0> or <label_1>: <segments> Which of these, <label_0> or <label_1>, was <position> in the book <booktitle>?" |
| 10 | "Please carefully read this excerpt from the book <booktitle>. This is the relevant passage: <excerpt>" | "You will be shown two text segments, labeled as <label_0> and <label_1>. Please recall in which order they appeared in the book <booktitle> and tell me which one came <position>. Please read the segments carefully: <segments> Which of these two parts of the book, <label_0> or <label_1>, came <position> in the book <booktitle>?" |
| 11 | "Please carefully read this excerpt from the book <booktitle>. This is the relevant passage: <excerpt>" | "I will show you two short parts from a book, labeled as <label_0> and <label_1>. Your task is to tell me which of them appeared <position> in the book <booktitle>. Please read both segments carefully and try to remember where in the book they come from: <segments> Which of these, <label_0> or <label_1>, appeared <position> in the book <booktitle>?" |
| 12 | "Please carefully read this excerpt from the book <booktitle>. This is the relevant passage: <excerpt>" | "This is your task: Given two segments from a book, labeled as <label_0> and <label_1>, please tell me which of them appeared <position> in <booktitle>. Read both segments carefully and try to remember where in <booktitle> they appeared: <segments> Which of these, <label_0> or <label_1>, comes <position> in the book <booktitle>?" |

### B.3 Per-model results on prompt selection sweep

To identify the prompts that work best for each model, we take 400 segment-pair samples that we excluded from the main evaluation and evaluate models’ in-context memory with all prompts shown in Table LABEL:appendix:prompt-list. To select the best prompt we considered both the proportion of A and B responses, which should be around $0.5$, and the accuracy.
We report the best selected prompts in Table 10 with numbers referring to the prompts presented in Table LABEL:appendix:prompt-list.

**Table 7: Selected prompts for each model.**
| Model Name | Best Prompt |
| --- | --- |
| Llama3-70b-inst | 4 |
| Llama3-8b-inst | 3 |
| Mixtral-8x22b-inst | 4 |
| Llama2-70b-inst | 7 |
| Gemma-1.1-7b-inst | 8 |
| Mistral-v0.2-7b-inst | 3 |
| Mistral-v0.1-7b-inst | 2 |
| Llama2-7b-inst | 10 |
| GPT-3.5-turbo | 7 |
| GPT-4 | 7 |

### B.4 RAG prompt selection

There were two different prompts to select for the retrieval-augmented generation experiments: the retrieval prompt (i.e. the search query), and the LLM prompt.

#### B.4.1 Retrieval prompt (search query)

The goal of retrieval in our RAG experiments is to find the text passages that will provide the most information about the segments for the sequence ordering task. After we created the vector database of all the text passages from Book-SORT, we formulated several different search queries (Table 8). We then ran retrieval using a validation subset of Book-SORT (50-word segments, 250-word excerpts from all books). The retrieval used the same database and text embedding model as described in the RAG portion of Section 3.2. The best search query was simple and only consisted of the segment text (query 8, Table 8). This search query is used for all RAG experiments.

**Table 8: The search queries for the RAG experiment and their average retrieval recall@10 on a validation subset of Book-SORT (250 word excerpts, 50 word segments).**
| No. | Search Query Text | Recall@10 |
| --- | --- | --- |
| 0 | "Please determine the order in which the following text segments appeared in <booktitle>: <segments>" | 0.728 |
| 1 | "We need to put text segments from <booktitle> in order. These are the segments: <segments>" | 0.817 |
| 2 | "Please find these text segments from <booktitle>: <segments>" | 0.869 |
| 3 | "Please find these text segments from <booktitle> to provide context for the next task: <segments>" | 0.875 |
| 4 | "Which text chunks from <booktitle> contain the following segments? <segments>" | 0.802 |
| 5 | "Which text excerpts from <booktitle> contain the following segments? <segments>" | 0.799 |
| 6 | "Which text chunks from <booktitle> overlap with these text segments: <segments>" | 0.782 |
| 7 | "<booktitle> contains this text: <segments>" | 0.865 |
| 8 | "<segments>" | 0.906 |
| 9 | "<booktitle> <segments>" | 0.858 |

#### B.4.2 RAG LLM prompts

We followed a procedure similar to the one outlined in Section B.2. We created a total of 10 modifications to the reading instructions from Table LABEL:appendix:prompt-list.

**Table 9: RAG prompt modifications.**
| No. | RAG Reading Instruction |
| --- | --- |
| 0 | "Here are some relevant excerpts from the book <booktitle>: <context>" |
| 1 | "The following excerpts from the book <booktitle>may be helpful context for the task. Context: <context>" |
| 2 | "Context: <context>" |
| 3 | "Searching a book database found these relevant text snippets: <context>" |
| 4 | "The following search results may be useful context: <context>" |
| 5 | "I will show you some relevant text found by searching a database of books: <context>" |
| 6 | "Please read some text deemed relevant for the task before performing the task. Relevant text: <context>" |
| 7 | "Please read these search results carefully to help you perform the task. Search results: <context>" |
| 8 | "Your objective may become easier with the use of these search results: <context>" |
| 9 | "This context may be helpful: <context>" |

#### B.4.3 Per-model results on RAG prompt selection

For a given LLM, we modified the reading instruction of the best prompt from Table 10 with each of the 10 options in Table 9. We then ran a sweep over the same 400 segment-pair samples detailed in Section B.3 and found the instruction that resulted in the highest performance on this held-out dataset.

**Table 10: Best RAG instruction prompts for each model.**
| Model Name | Best RAG Instruction No. |
| --- | --- |
| Llama3-70b-inst | 7 |
| Llama3-8b-inst | 7 |
| Mixtral-8x22b-inst | 3 |
| Mistral-v0.2-7b-inst | 6 |

## Appendix C Additional details on Book-SORT  results

### C.1 Memory-less baseline results

Figure 5 shows performance on Book-SORT without any memory-insertion of the books used in Book-SORT. We find that performance is higher in segment pairs that are very proximal or very distant in the book, indicating that it might be easier to sort these pairs based on temporal order reasoning. Performance without additional memory-insertion is generally low, showing that memory is needed for SORT.

Figure: (a) Segment length 20
Refer to caption: x7.png

### C.2 In-context memory full results

In this section, we provide a comprehensive overview of the in-context memory results across various models in Table 11 and Table 12. The table below illustrates the accuracy of different models on multiple books at segment lengths of 20 and 50 words. We observe that, while models generally perform slightly better with longer segments (50 words) compared to shorter ones (20 words), the improvement is modest, averaging up to $4\%$.

**Table 11: Accuracy and Difference of Various Models on Multiple Books at Excerpt Lengths of 20 and 50, with in-context memory (Part 1)**
| Model name | Book | SORT S20 | SORT S50 | SORT-Extend S20 | SORT-Extend S50 |
| --- | --- | --- | --- | --- | --- |
| Llama3-8b-inst | 69087 | 0.89$\pm$0.03 | 0.92$\pm$0.03 |  |  |
| Llama3-8b-inst | 72578 | 0.91$\pm$0.02 | 0.93$\pm$0.02 |  |  |
| Llama3-8b-inst | 72600 | 0.92$\pm$0.03 | 0.94$\pm$0.02 |  |  |
| Llama3-8b-inst | 72869 | 0.92$\pm$0.03 | 0.94$\pm$0.02 |  |  |
| Llama3-8b-inst | 72958 | 0.92$\pm$0.02 | 0.94$\pm$0.02 |  |  |
| Llama3-8b-inst | 72963 | 0.92$\pm$0.03 | 0.94$\pm$0.02 |  |  |
| Llama3-8b-inst | 72972 | 0.92$\pm$0.03 | 0.94$\pm$0.02 |  |  |
| Llama3-8b-inst | 73017 | 0.91$\pm$0.03 | 0.94$\pm$0.02 |  |  |
| Llama3-8b-inst | 73042 | 0.92$\pm$0.03 | 0.94$\pm$0.02 |  |  |
| Llama2-70b-inst | 69087 | 0.74$\pm$0.12 | 0.90$\pm$0.08 |  |  |
| Llama2-70b-inst | 72578 | 0.75$\pm$0.12 | 0.90$\pm$0.09 |  |  |
| Llama2-70b-inst | 72600 | 0.71$\pm$0.13 | 0.91$\pm$0.09 |  |  |
| Llama2-70b-inst | 72869 | 0.71$\pm$0.13 | 0.91$\pm$0.09 |  |  |
| Llama2-70b-inst | 72958 | 0.71$\pm$0.13 | 0.90$\pm$0.09 |  |  |
| Llama2-70b-inst | 72963 | 0.72$\pm$0.13 | 0.89$\pm$0.10 |  |  |
| Llama2-70b-inst | 72972 | 0.70$\pm$0.13 | 0.88$\pm$0.10 |  |  |
| Llama2-70b-inst | 73017 | 0.70$\pm$0.13 | 0.87$\pm$0.10 |  |  |
| Llama2-70b-inst | 73042 | 0.71$\pm$0.13 | 0.88$\pm$0.10 |  |  |
| Llama2-7b-inst | 69087 | 0.56$\pm$0.05 | 0.56$\pm$0.05 |  |  |
| Llama2-7b-inst | 72578 | 0.57$\pm$0.05 | 0.55$\pm$0.05 |  |  |
| Llama2-7b-inst | 72600 | 0.57$\pm$0.05 | 0.56$\pm$0.04 |  |  |
| Llama2-7b-inst | 72869 | 0.57$\pm$0.05 | 0.56$\pm$0.04 |  |  |
| Llama2-7b-inst | 72958 | 0.57$\pm$0.05 | 0.56$\pm$0.04 |  |  |
| Llama2-7b-inst | 72963 | 0.57$\pm$0.05 | 0.57$\pm$0.05 |  |  |
| Llama2-7b-inst | 72972 | 0.57$\pm$0.05 | 0.56$\pm$0.05 |  |  |
| Llama2-7b-inst | 73017 | 0.57$\pm$0.05 | 0.56$\pm$0.05 |  |  |
| Llama2-7b-inst | 73042 | 0.57$\pm$0.05 | 0.56$\pm$0.05 |  |  |
| Llama3-70b-inst | 69087 | 0.90$\pm$0.08 | 0.92$\pm$0.09 |  |  |
| Llama3-70b-inst | 72578 | 0.92$\pm$0.08 | 0.92$\pm$0.09 |  |  |
| Llama3-70b-inst | 72600 | 0.92$\pm$0.08 | 0.93$\pm$0.09 |  |  |
| Llama3-70b-inst | 72869 | 0.93$\pm$0.07 | 0.93$\pm$0.08 |  |  |
| Llama3-70b-inst | 72958 | 0.93$\pm$0.07 | 0.94$\pm$0.08 |  |  |
| Llama3-70b-inst | 72963 | 0.92$\pm$0.08 | 0.93$\pm$0.09 |  |  |
| Llama3-70b-inst | 72972 | 0.91$\pm$0.08 | 0.93$\pm$0.09 |  |  |
| Llama3-70b-inst | 73017 | 0.92$\pm$0.08 | 0.94$\pm$0.09 |  |  |
| Llama3-70b-inst | 73042 | 0.91$\pm$0.09 | 0.94$\pm$0.08 |  |  |

**Table 12: Accuracy and Difference of Various Models on Multiple Books at Excerpt Lengths of 20 and 50, with in-context memory (Part 2)**
| Model name | Book | SORT S20 | SORT | SORT-Extend S20 | SORT-Extend S50 |
| --- | --- | --- | --- | --- | --- |
| Mixtral-8x7b-DPO-inst | 69087 | 0.86$\pm$0.10 | 0.87$\pm$0.13 | 0.63$\pm$0.18 | 0.49$\pm$0.14 |
| Mixtral-8x7b-DPO-inst | 72578 | 0.88$\pm$0.10 | 0.90$\pm$0.10 | 0.63$\pm$0.18 | 0.57$\pm$0.14 |
| Mixtral-8x7b-DPO-inst | 72600 | 0.89$\pm$0.10 | 0.91$\pm$0.10 | 0.63$\pm$0.18 | 0.58$\pm$0.15 |
| Mixtral-8x7b-DPO-inst | 72869 | 0.90$\pm$0.09 | 0.92$\pm$0.10 | 0.61$\pm$0.17 | 0.55$\pm$0.15 |
| Mixtral-8x7b-DPO-inst | 72958 | 0.90$\pm$0.09 | 0.93$\pm$0.09 | 0.57$\pm$0.16 | 0.57$\pm$0.15 |
| Mixtral-8x7b-DPO-inst | 72963 | 0.89$\pm$0.10 | 0.92$\pm$0.10 | 0.56$\pm$0.16 | 0.55$\pm$0.15 |
| Mixtral-8x7b-DPO-inst | 72972 | 0.89$\pm$0.10 | 0.91$\pm$0.09 | 0.55$\pm$0.16 | 0.54$\pm$0.15 |
| Mixtral-8x7b-DPO-inst | 73017 | 0.87$\pm$0.10 | 0.91$\pm$0.10 | 0.55$\pm$0.14 | 0.54$\pm$0.14 |
| Mixtral-8x7b-DPO-inst | 73042 | 0.87$\pm$0.10 | 0.91$\pm$0.09 | 0.57$\pm$0.14 | 0.55$\pm$0.14 |
| Mixtral-8x22b-inst | 69087 | 0.92$\pm$0.08 | 0.93$\pm$0.09 | 0.73$\pm$0.13 | 0.73$\pm$0.11 |
| Mixtral-8x22b-inst | 72578 | 0.92$\pm$0.08 | 0.95$\pm$0.07 | 0.76$\pm$0.12 | 0.76$\pm$0.12 |
| Mixtral-8x22b-inst | 72600 | 0.93$\pm$0.08 | 0.96$\pm$0.07 | 0.77$\pm$0.12 | 0.78$\pm$0.11 |
| Mixtral-8x22b-inst | 72869 | 0.93$\pm$0.08 | 0.97$\pm$0.07 | 0.78$\pm$0.12 | 0.80$\pm$0.11 |
| Mixtral-8x22b-inst | 72958 | 0.93$\pm$0.08 | 0.97$\pm$0.06 | 0.79$\pm$0.12 | 0.80$\pm$0.11 |
| Mixtral-8x22b-inst | 72963 | 0.92$\pm$0.09 | 0.97$\pm$0.06 | 0.78$\pm$0.12 | 0.78$\pm$0.12 |
| Mixtral-8x22b-inst | 72972 | 0.92$\pm$0.09 | 0.97$\pm$0.07 | 0.78$\pm$0.12 | 0.79$\pm$0.12 |
| Mixtral-8x22b-inst | 73017 | 0.93$\pm$0.09 | 0.97$\pm$0.07 | 0.78$\pm$0.12 | 0.79$\pm$0.12 |
| Mixtral-8x22b-inst | 73042 | 0.93$\pm$0.09 | 0.97$\pm$0.07 | 0.78$\pm$0.12 | 0.79$\pm$0.12 |
| Mistral-v2-7b-inst | 69087 | 0.85$\pm$0.10 | 0.87$\pm$0.11 | 0.64$\pm$0.15 | 0.66$\pm$0.13 |
| Mistral-v2-7b-inst | 72578 | 0.85$\pm$0.11 | 0.87$\pm$0.10 | 0.63$\pm$0.15 | 0.65$\pm$0.14 |
| Mistral-v2-7b-inst | 72600 | 0.86$\pm$0.11 | 0.87$\pm$0.10 | 0.64$\pm$0.14 | 0.67$\pm$0.14 |
| Mistral-v2-7b-inst | 72869 | 0.85$\pm$0.11 | 0.87$\pm$0.11 | 0.64$\pm$0.15 | 0.68$\pm$0.13 |
| Mistral-v2-7b-inst | 72958 | 0.86$\pm$0.10 | 0.88$\pm$0.11 | 0.65$\pm$0.15 | 0.68$\pm$0.14 |
| Mistral-v2-7b-inst | 72963 | 0.83$\pm$0.11 | 0.88$\pm$0.11 | 0.64$\pm$0.14 | 0.68$\pm$0.14 |
| Mistral-v2-7b-inst | 72972 | 0.84$\pm$0.11 | 0.88$\pm$0.10 | 0.63$\pm$0.14 | 0.68$\pm$0.14 |
| Mistral-v2-7b-inst | 73017 | 0.83$\pm$0.11 | 0.88$\pm$0.10 | 0.63$\pm$0.14 | 0.68$\pm$0.14 |
| Mistral-v2-7b-inst | 73042 | 0.83$\pm$0.11 | 0.88$\pm$0.10 | 0.63$\pm$0.14 | 0.68$\pm$0.14 |
| Mistral-v1-7b-inst | 69087 | 0.74$\pm$0.04 | 0.82$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 72578 | 0.75$\pm$0.04 | 0.81$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 72600 | 0.74$\pm$0.04 | 0.80$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 72869 | 0.74$\pm$0.04 | 0.81$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 72958 | 0.74$\pm$0.04 | 0.81$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 72963 | 0.74$\pm$0.04 | 0.80$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 72972 | 0.75$\pm$0.04 | 0.80$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 73017 | 0.74$\pm$0.04 | 0.80$\pm$0.03 |  |  |
| Mistral-v1-7b-inst | 73042 | 0.75$\pm$0.04 | 0.80$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 69087 | 0.82$\pm$0.03 | 0.88$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 72578 | 0.83$\pm$0.04 | 0.89$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 72600 | 0.83$\pm$0.04 | 0.88$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 72869 | 0.84$\pm$0.04 | 0.89$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 72958 | 0.84$\pm$0.04 | 0.89$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 72963 | 0.84$\pm$0.04 | 0.88$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 72972 | 0.84$\pm$0.04 | 0.87$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 73017 | 0.83$\pm$0.04 | 0.87$\pm$0.03 |  |  |
| Gemma-1.1-7b-inst | 73042 | 0.84$\pm$0.04 | 0.87$\pm$0.03 |  |  |
| GPT-3.5-turbo | 69087 | 0.86$\pm$0.03 | 0.88$\pm$0.03 |  | 0.69$\pm$0.04 |
| GPT-3.5-turbo | 72578 | 0.87$\pm$0.03 | 0.89$\pm$0.03 |  | 0.69$\pm$0.04 |
| GPT-3.5-turbo | 72600 | 0.87$\pm$0.03 | 0.89$\pm$0.03 |  | 0.67$\pm$0.04 |
| GPT-3.5-turbo | 72869 | 0.87$\pm$0.03 | 0.90$\pm$0.03 |  | 0.67$\pm$0.04 |
| GPT-3.5-turbo | 72958 | 0.87$\pm$0.03 | 0.90$\pm$0.03 |  | 0.67$\pm$0.04 |
| GPT-3.5-turbo | 72963 | 0.86$\pm$0.03 | 0.89$\pm$0.03 |  | 0.67$\pm$0.04 |
| GPT-3.5-turbo | 72972 | 0.86$\pm$0.03 | 0.88$\pm$0.03 |  | 0.67$\pm$0.04 |
| GPT-3.5-turbo | 73017 | 0.85$\pm$0.03 | 0.88$\pm$0.03 |  | 0.67$\pm$0.04 |
| GPT-3.5-turbo | 73042 | 0.85$\pm$0.03 | 0.88$\pm$0.03 |  | 0.67$\pm$0.04 |

### C.3 Results per book

In Fig. 6, we provide the baseline results without text-specific memory separately for each of the $9$ books in Book-SORT.

In Fig. 7, we provide the in-context memory results separately for each of the $9$ books in Book-SORT.

Figure: (a) Segment length 20
Refer to caption: x10.png

Figure: (a) Segment length 20
Refer to caption: x12.png

### C.4 Relationship between in-context memory results and distance between segments across excerpt lengths

In Fig. 8 and Fig 9, we show the average accuracy by the distance between segments for all the excerpt lengths and segment lengths.

Figure: (a) Excerpt length 250, segment length 20
Refer to caption: x14.png

Figure: (a) Excerpt length 10000, segment length 20
Refer to caption: x21.png

### C.5 Baseline performance

In Fig. 10, we provide the SORT results based on parametric memory for all models across various segment distances. Due to the recent addition of the texts in Book-SORT to the public domain, we expect that models were not trained on these texts, i.e. they should not have text-specific memory. Performance is higher for segment pairs that have a short distance and a high distance in the books, indicating that these are more likely to be sort-able without episodic memory, based on temporal order reasoning.

Figure: (a) Segment length 20
Refer to caption: x26.png

## Appendix D Book-SORT  results from additional models

### D.1 Base models

We chose 2 base models to evaluate, Llama3-8b and Mistral-7b, whose fine-tuned versions (Llama3-8b-inst and Mistral-v2-7b-inst) performed well on SORT based on in-context memory. Figure 11 shows that both the base models got around chance performance across all the excerpt lengths and segment lengths.

Figure: (a) Llama3-8b
Refer to caption: x29.png

### D.2 State-space models

We tested an instruction-tuned version of the state space model RWKV (Peng et al., ), available in Huggingface as RWKV/rwkv-raven-7b. The results of the prompt sweep on SORT with in-context memory yielded a performance of 51% – very close to chance levels. A possibility for this is a larger sensitivity to prompting, e.g. this model might require instructions to be given in a different order. We assume that this is due to insufficient instruction tuning. While it could be interesting to see the performance of a state-space model with memory other than in-context, we leave this question to future work.

## Appendix E Finetuning of Llama3-8b-Instruct

##### Fine-tuning details.

We fine-tuned Llama3-8b-Instruct and Mistral-7b-v0.2-Instruct on a single node with 8 A100 GPUs. The books (without pre-processing beyond removing Project Gutenberg related text, i.e. including chapter signifiers) are split into chunks of 5000 words and contextualized in the same way in which excerpts are presented in-context in our experiments, i.e. together with the book-title in a user prompt along with a preceding system prompt. For the instruction data, we exclude the following task types: "experience", "stylized_response", "joke", "trivia", "roleplay", "riddle" and "greeting". Samples containing both book-chunks and instruction-following examples are padded to the maximum length in a batch. The effective batch size in our experiments is 192. We choose a moderately low initial learning rate of 5e-6 with cosine decay and a small amount of weight decay set to 1e-4. The chunks of books comprise a total of 116 independent samples. Together with 3 500 instruction samples from the OpenHermes dataset (Teknium, ), this means 19 steps of gradient descent are taken in one epoch. We fine-tuned both models for a total of 5 epochs.

##### Inclusion of instruction data to avoid catastrophic forgetting.

Fine-tuning an instruction-tuned model on specific data can lead to catastrophic forgetting (Luo et al., ), such that only a few steps of gradient descent can be enough to undo previous behavioral alignment (Qi et al., ; Zhan et al., ). To retain the general ability to follow instructions, and to allow for control condition fine-tuned models in which the book text is not part of the training data, we include $3,500$ instruction samples from the OpenHermes2.5 dataset on Huggingface (Teknium, ) (see Appendix E for details). Therefore the baseline without text-specific memory to compare with is not only the respective initial model before fine-tuning, but the same model fine-tuned on the same $3,500$ instruction samples but excluding the 116 samples of book chunks.

### E.1 Perplexity analysis of fine-tuned models

To confirm that fine-tuning on the books makes a model learn about the segments, we compare the perplexities of the two segments shown in SORT without source text presented in-context. We find that when the models are finetuned on data that includes the chunks of the books, they have a substantially lower perplexity for both segments, compared with the models fine-tuned only on the instruction data (see figure 12). Note that the scale of these perplexity values highlights that our task is likely out of distribution, presumably with little to no similar instruction data seen during pre-training and fine-tuning.

Figure: Figure 12: Perplexity of the two segments after fine-tuning of Mistral-7b-v0.2-Instruct and Llama3-8b-Instruct, when presented in the absence of in-context access to source excerpts.
Refer to caption: x31.png

### E.2 Comparison of SORT performance after fine-tuning using McNemar’s Test

We find that even though the book-text finetuned Llama3-8b model has a form of memory of the books’ texts, the epoch-matched performance between the models fine-tuned without the book-chunks does not differ statistically for any epoch (Figure 13). For this analysis we use McNemar’s test since we have an exact match of presented samples for both the memory-finetuned model and the baseline that does not form any memory of the text (Figure 12). We find high p-values, indicating no difference in performance between models fine-tuned with and without the book text (Figure 14), neither for Llama3-8b-Instruct, nor for Mistral-7b-v0.2-Instruct.

Figure: Figure 13: Accuracy of Llama3-8b-Instruct and Mistral-7b-v0.2-Instruct across epochs of finetuning on data including and excluding relevant book-text. Figure 14 shows that differences between accuracies shown here are not statistically significant (p>0.05).
Refer to caption: x32.png

Figure: (a) Mistral-7b-v0.2-Instruct
Refer to caption: x33.png

### E.3 Comparison of SORT performance after fine-tuning using a pairwise t-test

Testing the binary correctness evaluated based on a greedily sampled token does not allow us to draw conclusions about sub-threshold effects of fine-tuning on task performance.
To test whether the models fine-tuned on the books is better than the models that are fine-tuned without chunks from the books, we performed a pairwise t-test on a continuous measure of accuracy based on the token log-probabilities. We compute the likelihood of the correct answer by taking the log ratio of the correct answer among all answers that can be mapped to either A or B, i.e. we are interested in $\text{log}\left(\frac{p(a=y)}{(p(a=A)+p(a=B)}\right)$, where $y$ is the correct answer.

The results shown in figure 15 suggest that fine-tuned models do improve over the base model, with the book text condition performing better than the others after one epoch of training with statistical significance ($p<0.01$). Even though there is an effect, the magnitude is very small, as can be seen in Figure 16, and this positive effect could also be attributed to interleaving the instruction data with samples including longer texts ($5,000$ words) compared to just the instruction samples.

Figure: (a) Mistral-7b-v0.2-Instruct
Refer to caption: x35.png

Figure: Figure 16: Log-probability of the correct answer for fine-tuned models across epochs. Figure E.3 shows statistical significance between conditions and epochs for this data.
Refer to caption: x37.png

### E.4 In-context memory performance of fine-tuned models

Despite the inclusion of instruction data in fine-tuning, the accuracy with source excerpts presented in-context of SORT decreased from 0.93 to $0.90$ after a single epoch and to $0.88$ after three epochs of fine-tuning for Llama3-8b-Instruct. For the instruction-data only baseline of Llama3-8b-Instruct, the performance degraded slightly less with an accuracy of $0.91$ after the first epoch of fine-tuning.

## Appendix F Code and Data

We provide the code to create SORT datasets and evaluate models on SORT in a [public GitHub repository](https://github.com/bridge-ai-neuro/SORT).
Our evaluation code currently supports the OpenAI API, Huggingface Transformers (Wolf et al., ) and vLLM (Kwon et al., ) for distributed inference. Our initial Book-SORT dataset can be accessed via [Huggingface Datasets](https://huggingface.co/datasets/memari/booksort).

##### License.

We make our code and data openly available under a permissive BSD-3 license for code. Data including Book-SORT is available under a CC0 license.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 4
questions: I am sorry I could not provide a more positive evaluation. Clearly, a lot of work went into this paper, and it pains me to recommend a rejection. I deeply hope that the authors can slightly re-focus the paper, making it more focused on a specific problem or question. And if the question is the nature of LLM memory as compared to human memory, a much deeper discussion of characteristics and properties of different types of human memory would be necessary. It's also important to clearly decide on the distinction between having a memory and being able to access it. For example, some model might be able to recite Hamlet, but be unable to answer questions about it. Should we treat such a model as having the memory of Hamlet or not? In any case, the fine-tuning experiments need to be improved, as right now they are, in my view, the weakest part of the paper. Again, I am sorry I can't provide a more positive assessment at the moment. 

I hope the authors continue their work and I'd be happy to see a revised version of this paper published one day.

I will also, of course, read the other reviews and the rebuttal. I am not sure how likely it is, but I will be open to reevaluate my assessment if it turns out that I overlooked some important considerations.

Questions:

What is the criterion for having an episodic memory? If a model has it "implicitly" (see the Hamlet example above) -- does it count?

What was the reason behind trying to maintain instruction-answering capabilities of the LLMs instead of switching to a simple Supervised Fine-Tuning classification task to see if the information is internalized?

What is the main goal of the paper? Specifically, do the authors see it as more of an inquiry into the nature and characteristics of LLM memory, or is it primarily a benchmark to evaluate the newly released models' memory capacity?

Minor:
On line 162, the authors say that the segments are "preceded by X", which would mean providing the whole book. But this contradicts the actual experiments where only book samples are actually presented due to context length limitations.

** Extra sources **
If we treat this paper as primarily a useful benchmark for assessing trainsformers' memory capability, NarrativeQA (https://huggingface.co/datasets/deepmind/narrativeqa) and, even more so, NarrativeXL (https://arxiv.org/pdf/2305.13877), datasets propose a different approach to a similar problem of assessing long-term memory in transformers, and use a similar methodology. These and related works should probably be discussed to better define the role of the present contribution.
soundness: 2
strengths: The paper addresses a highly important topic of assessing memory capabilities in LLMs.

I appreciate the attention to statistical detail reporting, and the generally thorough reporting of the experiments.

The paper is well-written.

The inclusion of human baselines, especially with the clever involvement of Goodreads users, is a great addition to the paper.
summary: The authors propose a new dataset, called SORT, for assessing episodic memory in LLMs, centered around the task of recalling the correct order of two short book snippets. The authors report extensive evaluations of modern LLMs on the proposed dataset, studying both in-context and "parametric" memory (memory encoded in weights and acquired during training), as well as externally augmented memory options. They find that modern LLMs can use in-context memory quite well, but struggle when the relevant context is only presented during training.
weaknesses: ** Significance and novelty **

I believe that the work, overall, is sufficiently novel. At the very least, it provides a unique and potentially useful dataset.
I'm however, very conflicted on the significance of this work.

First, the authors do not discuss whether the notion of Episodic memory in LLMs is valid at all. "Episodic memory" is not clearly defined in the paper, other than type of memory "which links memories to their contexts, such as the time and place they occurred".

If we take this definition, many general-knowledge questions can be seen as probing episodic memory. E.g. questions in the form of "what was the movie in which a <SCENE DESCRIPTION>?", and "Describe that scene in more detail", or "When was this movie created" which modern LLMs are well-equipped to answer.

Usually, in psychology, episodic memory is distinguished from declarative memory by being related to personal experiences as opposed to stated facts. This distinction is extremely moot in the case of LLMs. This seriously undermines the premise of the paper.

Moreover, the key psychological literature the authors cite actually refers to a specific subset of human episodic memory, namely temporal memory. This creates confusion. If accepted, the authors should consider changing "episodic" to "temporal" throughout the article. It's worth noting that preserving temporal order is not a necessary attribute of episodic memory. In fact, humans routinely mix up the order of their personally significant memories; a simple way to see that is to try to reliably order all of one's childhood memories.

Still, the problem is deeper than this mismatch in terms. With no clear distinction between "personal" and "stated" memories in LLMs, what is the importance of studying temporal memory in LLMs in the way that authors suggest? Why, and to what extent, do we equate personal experiences with the order in which a some information is presented in the data?

If this is a way to address the issue of correcting LLM memories - why isn't there a deeper discussion of alternative methods and of how exactly this dataset will benefit the effort. If this is about whether or not LLMs can remember the order of things presented in contiguous chunks in training data - we know very well that LLMs are very good at that, e.g. that common mathematical sequences can often be retrieved if included in the training data. If it's about in-context handling of long sequences - why isn't there a more thorough discussion of other datasets that look at long-context tasks?

** Quality of the experimental support **

When it comes to human evaluation, the experiment seems well-organized and adds a lot of value to the paper.

I am, however, very troubled by many fine-tuning results and experimental decisions. 

*Data ratio
The ratio of the new data to instruction data is very strange: the authors keep 3500 instruction following prompts "to avoid catastrophic forgetting", when there are only 116 independent book samples. It's very likely that in these conditions, the models simply do not have enough incentive to learn book-specific information. Additionally, the instruction fine-tuning stage behind Llama and Mixtral models is much more extensive than the 3500 instruction samples. Therefore, most likely, doing full fine-tuning on such a restricted sample severely hurts model performance across the board, making results hard to interpret. As I mention later in this review, it's not entirely clear why the authors even try to "preserve the models’ ability to understand and follow the task instructions" instead of simply fine-tuning a classification head for their specific task.

*Unusual training stage order:
Instruction fine-tuning and raw language modeling stages are mixed. I don't know how it affected results, but again, it makes interpretation harder. Instead, the authors could have chosen a much simpler two-step process: First, a language modeling fine-tuning stage on the books of interest (main condition) or on irrelevant books (control). Second, transfer learning fine-tuning with a classification head, treating SORT as a binary classification task. This would remove the need to keep a bunch of instruction-tuning samples and would clarify the results substantially. I understand that not all types of evaluations can be performed on these models, but that would more definitively answer whether LLMs can remember temporal information from their training set.

*Concerning training dynamics:
Another issue is that according to figure 16, the log probability of the correct answer is dropping with each new epoch. This is deeply concerning, and suggests that something is fundamentally wrong with the environmental set-up.

*Apparent failure to overfit:
It is well-known that LLMs are extremely good at verbatim memorization, and can overfit to memorize extreme amount of information. I am absolutely certain that the LLMs considered in the paper can be fine-tuned to memorize 9 books verbatim. The most likely reason that we didn't see improved performance is, then, either 1) inadequate training setup 2) inadequate methods of probing/information extraction (e.g. the model has the memory but its zero-shot skills are so degraded that it can't use this memory). Perhaps the authors are interested not only in whether a given "episodic" memory is implicitly present in LLM's parameters, but also whether it can it be used via zero-shot prompting? But that seems like a substantially different research question.

The last quality issue, not related to fine-tuning, is that the dataset is destined to become obsolete. I.e. as the public domain books in question enter the training sets of new LLMs, much of the dataset's value will be lost, and the reported results won't be comparable to an attempted replication on newer LLMs. I don't think this is a disqualifying consideration, but it does negatively affect the significance and potential impact of this work.

Overall, these issues largely invalidate the conclusions on whether parametric memory in transformer can be seen as a form of episodic memory.

A minor note: I believe it'd be beneficial to the paper to add an extra classification baseline where a model is finetuned to predict which segment comes earlier in a book, without any context. It's the same as "Baseline" in the paper, but with fine-tuning. The reason it's important is that it could be that zero-shot instructions are not enough to extract this information from the model. Of course, since there are only 9 books in the dataset, it'd be necessary to either do cross-validation (fine-tune on 8 books, evaluate on 1).


** Summary **

Overall, I highly resonate with the importance of creating new datasets to thoroughly evaluate LLM memory capacity. I deeply appreciate the inclusion of a human-subject study and a thorough approach to evaluation. That being said, unfortunately, I can't recommend the work for acceptance.

The main issue is the positioning of this work in the broader context. What exactly is the problem we are trying to address? Is this a useful benchmark that should use to test memory capabilities of new models, or is it more of an investigation into the nature of episodic memory in transformers? I feel that the paper attempts to tackle both questions at once, but, as a result, does not sufficiently address either. Another substantial problem is the experimental support: I find that many of the fine-tuning experiments are not sufficiently clearly constructed and executed.

My true overall score for this submission is 3.5, or 4. I think that even if published as is, it will provide some positive value to the community. That being said, unfortunately, in its present form, it is not strong enough for ICLR.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: - Older versions of GPT4 used to know by heart some of the books on the gutenberg project ("the Murder of Roger Ackroyd" is present there). Did you ensure that the models did not know the book? (e.g. by asking questions?) 

- Why is the evaluation on GPT4-o missing from the second part? Would it be possible to include GPT-4 results or explain their omission?

- In which way your work is different from simply testing positional encoding capabilities of LLMs? Would it be possible to add analysis distinguishing positional encoding from true temporal understanding?
soundness: 2
strengths: - **Valuable Human Evaluation**: The paper conducts a large-scale human evaluation involving 155 participants who recently read a book. This provides valuable data on human long-term memory performance in recalling the order of text segments.

- **Highlighting Limitations of Parametric Memory**: Although simple, the paper rightfully brings attention to the limitations of current LLMs' parametric memory in handling one of the episodic memory tasks (here segment order). This is important for raising awareness and fostering further research in this area.
summary: The paper introduces a new evaluation task called the Sequence Order Recall Task (SORT) to assess episodic memory in large language models (LLMs). SORT requires models to recall the correct order of two text segments extracted from longer sequences. The authors create a dataset named Book-SORT, consisting of 36,000 pairs of text segments from nine books recently added to the public domain. They also conduct a human evaluation with 155 participants who had recently read one of the books, demonstrating that humans can recall some segment orders based on long-term memory, especially when the segments are far apart. The paper evaluates various LLMs on SORT using different methods of memory insertion, such as in-context learning, fine-tuning, and retrieval-augmented generation. The main finding is that models perform relatively well when given relevant context but fail to recall the order when using only their parametric memory.
weaknesses: - **Little coverage of episodic memory** The correct order of text segments is one tiny aspect in a large spectrum of episodic memory capabilities: (i) places/spaces in which events occured, the (ii) intricate sorting of events and entities that are tracked along long periods of time. For example, we are able not only to sort the major events of our life, but also the major events that happened within the lifetime of a six-months long project, or the major events that happened in a movie, or a novel. We are able to sort, and recall the spatial and action details etc.


- **Reported events vs. "encountered" events** For humans, and LLMs in particular (for which language is also a *reporting tool* of episodic events), there are other aspects that matter: For example, what are the actual order of events as they happened in time (as reported by language), not only as they were told to the reader/LLM sequentially. Does it make sense to ask a model which book appeared first in its training set? but if so, based on which epoch? Are LLMs equipped to answer this type of questions? Wouldn't this task simply only test the positional encoding/embedding abilities of LLMs? because that's the only fundamental signal that allows them to know what comes before what.


- **Benchmark contamination** Although recently added to the public domain, the books are old and there are chances that they are part of the training data. I appreciate that the authors did the effort of verifying that the models failed to answer using only their parametric memory (e.g. Table1), but one cannot rule out that pre-training could have helped in the in-context task. 


- **No significant technical innovation in methodology**

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. Given the results in Table 2, it appears that larger context windows may positively impact model accuracy on the SORT task. However, many of the models used in this study are limited to an 8k context window. Could the authors consider incorporating models designed for extended context lengths, such as LongLLaMA[1], to assess whether increasing the context window further improves SORT accuracy?
2. In the RAG setup, how sensitive are the SORT results to the choice of embedding model and vector dimensionality?
3. When using in-context learning for the SORT task, does the model exhibit a "Lost in the Middle" effect, where it struggles to differentiate the order of segments that appear in the middle of longer contexts?

[1] Tworkowski S, Staniszewski K, Pacek M, et al. Focused transformer: Contrastive training for context scaling[J]. Advances in Neural Information Processing Systems, 2024, 36.

[2] Liu N F, Lin K, Hewitt J, et al. Lost in the middle: How language models use long contexts[J]. Transactions of the Association for Computational Linguistics, 2024, 12: 157-173.
soundness: 3
strengths: 1.  The paper offers a simple yet effective task, easily understandable and extensible to various settings. The general formulation of SORT enables adaptation to multiple domains and sequential data types, such as video or audio, enhancing clarity and future applicability.
2. The authors have open-sourced both the Book-SORT dataset and the code, ensuring reproducibility and enabling further research by the community.
3. The paper features multiple experiment setting, covering in-context memory, Retrieval-Augmented Generation (RAG), and fine-tuning methods.
summary: The paper introduces a new evaluation framework, Sequence Order Recall Tasks (SORT), designed to assess episodic memory in large language models (LLMs). Unlike traditional benchmarks that focus on semantic memory, SORT requires models to recall the order of text segments, assessing memory that links content to its context. The authors created Book-SORT, a dataset of 36,000 text segment pairs from nine public domain books, and conducted human experiments, showing that humans can remember sequence order from long-term memory with notable accuracy. Their findings indicate that while models perform well with in-context memory, they struggle with SORT when only trained on book content, highlighting current LLMs' limitations in true episodic memory.
weaknesses: 1. Although the SORT task is positioned as an assessment of episodic memory, the simple task of sequence order recall does not fully capture the complexities of episodic memory as seen in humans. Episodic memory typically involves more than the temporal order of events; it also includes contextual elements such as the significance, location, or source of memories. As currently designed, SORT assesses temporal ordering without requiring models to associate segments with rich context or situational details.
2. The authors note in the related work that sequence order prediction tasks have been used in prior language models, such as BART, where sequence reordering is part of the training objective. SORT may therefore be too easy to offer a strong evaluation of advanced memory capabilities in LLMs.
3. While the paper includes a broad range of experimental settings (in-context memory, RAG, fine-tuning), it does not clearly identify which specific factors most significantly improve performance on the SORT task. Table 2 indicates that with in-context settings, models can achieve near-perfect accuracy (e.g., Mixtral-8x22b-inst reaches 0.95 accuracy). This high performance suggests that in-context memory alone might be sufficient for solving SORT.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: * Have you considered long-context models as a way of showing the model the entire book in context which may be more similar to the actual setup in humans?
* Why is GPT-4 present in the baseline tests but missing in RAG and In-context setups? It seems to have performed very well in the baseline.
soundness: 3
strengths: * The authors builds a new task for assessing episodic memory which is a novel test for LLMs. This is done in a scalable way which can be used to generate more information if needed, and is furthermore carried out in a way that minimizes contamination.
* By using a no-context baseline, the authors show that the benchmark cannot be solved from temporal reasoning alone without knowing the context (~50% accuracy).
* Human evaluations are used to provide a human baseline on this task, which is very effective in understanding the limitations and successes of LLMs. 
* The authors compare the most common strategies used to adapt LLMs to specific tasks, and thus are able to compare and contrast them, which is also helpful when designing solutions based on LLMs that require this capability (specifically, pointing out that RAG naively does not perform well and that pretraining on book data does not help). 
* The authors compare several smaller and larger models (mostly open source) on this task which shows that performing well on this task is not just a matter of model size. 
* Authors perform prompt-optimization per model to reduce biases
Overall, many comparisons across different scenarios allow for a comprehensive understanding of LLM performance on this task.
summary: The paper proposes a new benchmark, SORT, for testing the episodic memory of and temporal order reasoning of LLMs, which is lacking in the current evaluation benchmarks. This test is inspired by cognitive sciences, and composed of pairs of randomly selected non-overlapping passages from fiction books, with a length of 20-50 words, which the model is provided with, along with contextual information in the form of an excerpt from the book, or without it. It is then asked to decide which of them precedes the other. Several setups are tested: Zero-shot prompting with and without context, with RAG, and finetuned on book data as a way to insert memory. The performance of the several models is compared to that of human baselines on this task (evaluated using 155 participants). 
The results show that without context models fail entirely in this task (with performance in the range of 50% - random chance). When given context, models succeed very well, but their performance degrades as the length of the excerpt increases. Finetuning on language modeling task with the book data does not improve the results much.
Due to these results, specifically degradation of performance with context length, authors surmise that the transformer’s context is more similar to working memory than episodic memory. Naive RAG can also not be considered episodic memory since it does not preserve order, and indeed performance is not high with this implementation. Order preserving RAG, with an oracle retriever, is successful.
weaknesses: * It is very clear that the in-context scenario is relatively simple for LLMs. The paper thus hypothesizes that this is more similar to working memory. It would be interesting to compare to baseline human performance when shown the in-context SORT task, as a way of strengthening the claim. 
* Other than the fine tuning setup, the model cannot be said to have seen the entire book in any of the setups (this may be possible with long-context). This is markedly different from the human baseline, which also relies on long-term memory which makes comparison (and possibly relating the task to episodic memory) a bit more difficult.

Meta-review: This submission assesses the abilities of a set of language models (LMs) to perform "sequence-order recall", that is, recalling the order of elements of a previously seen sequence. The authors describe the capacity that performs this task in LMs "episodic memory", and give an informal definition of this capacity in LMs. The target of the submission is to relate the "episodic memory" capacity in LMs and humans by testing LMs and humans in matched tasks of "sequence-order recall". The comparison demonstrates that in-context sequence-order recall in LMs is superhuman but degrades with context length, suggesting that it is operating more like "working memory" and in conflict with recent work that connects in-context learning and episodic memory. This is an interesting discussion point. Other forms of LM memory (fine-tuning and RAG) are tested to see their performance on sequence-order recall tasks (SORT). The submission shortly concludes that these forms of memory are also unlike human/animal episodic memory.

The evaluation of "sequence-order recall" to provide evidence for "episodic-memory" like capabilities is novel and interesting. However, the relationship between episodic memory in humans and the sequence-order recall tasks presented here is muddy: As Reviewer GiyF, the capability to recall sequence order is not exactly episodic memory in humans. Moreover, the same control variables are not in use across the human and LM studies (namely, context lengths in Figs. 3, 4 are not varied for humans), making it difficult to conclude that the same mechanisms are at play to substantiate a direct comparison. A revision of the submission should appropriately scope the discussion of episodic memory in LMs as directly related to a memory capacity in humans alongside the narrowness of the evaluation task.

Decision: Reject