Venue: NAACL 2025

# An Efficient Context-Dependent Memory Framework for

LLM-Centric Agents

 Pengyu Gao\({}^{\dagger}\)

Independent Researcher

piri.gao@outlook.com &Jinming Zhao12

Qiyuan Lab

zhaojinming@qiyuanlab.com &Xinyue Chen3

Yanjing University of Aeronautics and Astronautics

cxy_nuaaa2012@nuaa.edu.cn &Yilin Long

Peking University

yilinlong@stu.pku.edu.cn

Corresponding Author

 Equal contribution.Pengyu Gao: CDMem implementation, ALFWorld experiments, writing.Jimming Zhao: CDMem proposal, paper refinement.Xiuyue Chen: ScienceWorld experiments, paper refinement.[https://github.com/piri-gao/CDMem](https://github.com/piri-gao/CDMem)

Footnote 1: footnotemark:

###### Abstract

In human cognitive memory psychology, the context-dependent effect helps retrieve key memory cues essential for recalling relevant knowledge in problem-solving. Inspired by this, we introduce the context-dependent memory framework (CDMem), an efficient architecture mimimg human memory processes through multistage encoding, context-aware storage, and retrieval strategies for LLM-centric agents. We propose multistage memory encoding strategies for acquiring high-quality multilevel knowledge: expert encoding compresses raw trajectories from a domain-expert perspective, short-term encoding consolidates experiences from current tasks, and long-term encoding reflects insights from past tasks. For memory storage and retrieval, we design a graph-structured, context-dependent indexing mechanism that allows agents to efficiently and accurately recall the most relevant multilevel knowledge tailored to the current task and environmental context. Furthermore, the proposed CDMem framework is an online learning architecture, enabling agents to efficiently learn and update memory while adapting to novel environments and tasks in real-world applications. We conducted extensive experiments on two interactive decision-making benchmarks in the navigation and manipulation domain, ALFWorld and ScienceWorld. Using GPT-4o-mini, our method surpasses state-of-the-art online LLVM-centric approaches, achieving success rates of 85.8% and 56.0%, respectively. We hope this work will serve as a valuable reference for the academic and industrial communities in advancing agent-based applications. The codes are available1.

Footnote 1: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote†: footnotetext: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: Corresponding author

+
Footnote †: footnotet: Corresponding author


reflection mechanism, enabling agents to summarize experiences from past trajectories and apply them to subsequent trials or tasks. Recently, some LLM-centric agents have been designed with memory modules to facilitate information storage and retrieval by organizing environmental knowledge into categories or generating state-aware guidelines (Chen et al., 2024; Fu et al., 2024). Memory storage is often inefficient, hindering quick access and selective retrieval, which results in slower and less accurate decisions (Zhong et al., 2024). These methods often struggle to store and retrieve complex information effectively, especially in dynamically changing contexts, preventing agents from leveraging relevant information.

The human brain encodes, stores, and retrieves memories in a context-dependent manner, associating them with specific environments or tasks to quickly identify key cues and activate relevant memory (Smith and Kosslyn, 2007; Xue, 2022). Inspired by this, we propose the context-dependent memory framework (CDMem), an efficient architecture miming human memory processes through multistage encoding, context-aware storage, and retrieval strategies for LLM-centric agents.

We introduce a multistage memory encoding strategy to learn high-quality, multilevel knowledge. First, expert encoding compresses raw trajectories from a domain-expert perspective, mimicking how human experts effectively organize and summarize information after completing a trial. Next, short-term encoding consolidates successes and failures from recent trials within the current task. Finally, long-term encoding integrates insights from past tasks and updates memory indexes to maintain relevance and accuracy. Furthermore, as illustrated in Figure 1, we propose a context-dependent storage indexing mechanism that structures multilevel knowledge within a graph. During retrieval, the agent identifies the current task and environment types and then utilizes the key cues to accurately access relevant exemplars, task experiences, and insights. This precise retrieval process enhances the LLM-centric agent's ability to address and solve the current task effectively. Additionally, the proposed CDMem is an online learning framework that enables agents to efficiently learn and update memory while adapting to novel environments and tasks in real-world applications.

To summarize, our contributions are as follows:

* We propose an efficient online memory paradigm for LLM-centric agents: a context-dependent memory learning, storage, and retrieval framework (CDMem) inspired by the human memory mechanism, particularly suited for developing domain-specific agents in industrial applications.
* We propose an efficient multistage memory learning method, including expert encoding, short-term memory encoding, and long-term memory encoding, to learn multilevel and high-quality knowledge from past tasks;
* We design a context-dependent graph-based indexing that allows agents to efficiently and accurately retrieve the most relevant knowledge through environmental and task-specific cues;
* We conduct extensive experiments on two interactive decision-making benchmarks (ALFWorld and ScienceWorld), We demonstrate that our method outperforms state-of-the-art online LLM-centric memory-based methods, achieving significant performance improvements.

## 2 Related Work

### LLM for Reasoning and Decision-Making

The introduction of Chain-of-Thought (CoT) (Wei et al., 2022) has significantly enhanced the reasoning capabilities of LLM. Building on this, several works (Kojima et al., 2022; Yao et al., 2023b; Wu et al., 2023) have demonstrated the immense potential of LLM in reasoning and decision-making, surpassing traditional reinforcement learning methods in specific scenarios. Tree-of-Thought (Yao et al., 2023a) and Graph-of-Thought (Besta et al., 2024) further enhanced CoT by extending the linear CoT structure to tree-based and graph-based structures, respectively. Many other works have applied the reasoning and decision-making capabilities of LLM to various domains, including robotics (Ahn et al., 2022; Liang et al., 2023), gaming (Wang et al., 2024b, 2023; Zhu et al., 2023), and game theory (Zhang et al., 2024; Guo et al., 2023). In these complex domains, fully leveraging learned experiences and dynamically forming new experiences based on real-time environmental feedback is crucial for decision-making 



### Memory Storage and Retrieval of Agents

Designing an effective memory mechanism is essential for improving the performance of decision-making agents. MemoryBank Zhong et al. (2024) proposed a long-term memory mechanism that addresses the lack of long-term memory in LLM by incorporating storage, retrieval, and update mechanisms combined with the Ebbinghaus forgetting curve theory. ChatDB Hu et al. (2023) introduced databases as symbolic memory for LLM and proposed the Chain-of-Memory (CoM), enhancing the complex reasoning capabilities of LLM. TiM Liu et al. (2023)improved the performance of LLM in long-term interactions by storing historical thoughts, updating memory through operations such as insertion, forgetting, and merging, and utilizing locality-sensitive hashing for efficient retrieval. Nevertheless, these works all rely on direct retrieval from the entire memory pool without constructing more efficient indexing mechanisms, which leads to inefficient retrieval and insufficient accuracy.

### Memory Self-Learning of Agents

ReflexionShinn et al. (2023) converts environmental feedback into textual statements and stores them in memory, allowing the agent to utilize this memory in subsequent trials to improve task performance. RetroformerYao et al. (2024) and ReflectRLZhou et al. (2024) further enhanced Reflexion by incorporating reinforcement learning to train specific components, effectively embedding part of the memory into model parameters to improve the agent's reasoning capabilities. In-Memory LearningWang et al. (2024) proposed a framework that constructs memory and enables agent self-improvement through induction, revision, and inference. ExpelZhao et al. (2024) introduced an offline learning agent that collects experiences through trial-and-error interactions with the environment during the training phase, storing them in an experience pool for later extraction of insights. During the evaluation phase, the agent uses these insights and successful trajectories to assist decision-making. AutoGuideFu et al. (2024) ex

Figure 2: **Illustration of the Context-Dependent Memory (CDMem) framework.** (a) The agent interacts with the environment to generate a trial history. (b) The expert encoding module compress the raw trial history and extract expert history including environment, task goal and expert trajectory description. (c) The short-term memory encoding based on the compressed output of (b) to generate successful shortcuts, experiences of defined failure types, and environmental summary. (d) The long-term memory encoding captures cross-task insights, including environmental insights as well as success- and failure-related task insights, based on accumulated short-term memories every \(B\) tasks. (e) Organize the knowledge learned from the previous steps into a graph-structured storage index according to task type and environment type. (f) At the inference stage, given a new trial with a description of the environment and task goal. (g) Retrieve relevant knowledge from the context-dependent memory, including exemplars, environmental insights, as well as success- and failure-related task insights. (h) Organize the retrieved knowledge, current trial, and task experiences to form the prompt (similar to working memory in the human brain), and use it to make action decisions through the LLM.



tracts a set of state-aware guidelines through offline learning, providing more targeted guidance to the agent based on the current state during testing. Unlike Expel and AutoGuide, CDMem is an online memory-based method that continuously self-improves through real-time memory updates.

## 3 Method

When humans perform tasks in a specific environment, they use environment- and task-specific cues to retrieve relevant memories efficiently. After completing a task, they organize and consolidate these memories for future use. Inspired by this process, we propose the Context-Dependent Memory (CDMMem) framework, which includes three key components: memory encoding, context-dependent memory storage, and context-dependent memory retrieval. This framework efficiently retrieves relevant exemplars, experiences, and insights based on the current environment and task instructions. Memory encoding uses a novel multistage memory learning strategy, while memory storage and retrieval rely on a context-dependent indexing structure. Detailed method descriptions with pseudo code are provided in the Appendix.

### Multistage Memory Encoding

#### 3.1.1 Expert Encoding

Memories formed by domain experts are typically more concise and organized than those of non-experts. This is because expert encoding efficiently groups raw trajectories into knowledge chunks. For instance, a professional chess player will deduce the tactics used in a game, thereby remembering the arrangement of the pieces, whereas a novice would attempt to remember the position of each chess piece from the outset. Inspired by this, we introduce the Expert Encoding Module \({\cal M}_{\rm expert}\) which takes a raw trajectory \(\tau\) as input and outputs compressed expert observations and actions \(E_{\rm expert}\).

\[E_{\rm expert}={\cal M}_{\rm expert}(\tau) \tag{1}\]

Expert observations provide a concise description of the environment, summarizing object locations and their properties within the current environment. Expert actions are well-organized trajectories from an expert's perspective, which omit unnecessary details and consolidate similar actions into a single statement to reduce redundancy.

#### 3.1.2 Short-term Memory Encoding

When an agent repeatedly attempts a task in a specific environment, it reflects on its actions, creating memories tailored to that task and environment, like human short-term memory. To model this process, we introduce the Short-term Memory Encoding Module \({\cal M}_{\rm short}\), which takes raw trajectories and expert encoding as inputs and generates short-term memories as output.

\[E_{\rm short}={\cal M}_{\rm short}(\tau,E_{\rm expert}) \tag{2}\]

This module is similar to the reflection process in ReflexionShinn et al. (2023), but with two key improvements inspired by human memory:

**(a) Reflection on Successful and unsuccessful Trajectories** Unlike Reflexion, which only reflects on unsuccessful trajectories, our approach separately reflects on both successful and unsuccessful ones. For successful trajectories, the agent reflects on which actions were necessary and which were not, removing unnecessary steps to create a "Successful Shortcut." This represents the shortest path to complete the task and helps the agent focus on essential planning. For unsuccessful trajectories, the agent analyzes the error type, such as planning, search, or operation failures, and then adjusts its plan accordingly.

**(b) Environmental Memory.** Beyond learning the experiences through reflection, the short-term memory encoding module also learns memories of the current environment via different aspects. When the agent makes attempts in the same or similar environment, the environment memories can help the agent to effective and efficient understanding of the environment.

#### 3.1.3 Long-term Memory Encoding

When an agent performs different tasks in various environments, similarities between these tasks and environments may emerge. These similarities can be summarized into high-level, abstract memories spanning tasks and environments. These memories are recalled not only when the agent encounters the same task or environment but also when it faces similar ones, demonstrating strong generalization and persistence, much like human long-term memory. To capture this, we designed the Long-term Memory Encoding Module \({\cal M}_{\rm long}\), which takes short-term memories as inputs and generates environmental and task insights as output 

\[env\_insights,task\_insights\\ ={\cal M}_{\rm long}(\tau,E_{\rm short}) \tag{3}\]

**(a)Environmental Insights.** Using ALFWorld and ScienceWorld as examples, environmental insights focus on encoding object properties, such as "a microwave can heat food," rather than summarizing object positions, as in short-term memory. This approach mirrors human memory patterns, where object positions are not easily generalized across environments.

**(b)Task Insights.** Similar to short-term memory encoding, task insights separately summarize positive and negative examples. For successful memories, the agent combines multiple successful shortcuts to create a general plan for a task category. For unsuccessful memories, the agent consolidates reflections on errors to identify common mistakes and their remedies across tasks.

### Context-Dependent Memory Storage

We propose the Context-Dependent Memory (CDMem) framework, featuring a context-dependent indexing structure that includes both environment-dependent and task-dependent indices. This structure uses environmental and task-specific cues to improve long-term memory retrieval. This subsection outlines the memory storage process.

The context-dependent indexing structure consists of two dictionaries: the Env-Index for indexing environmental long-term memories and the Task-Index for task-related long-term memories. The Task-Index is further divided into two subdictionaries: Success and Failure, which store summarized successful and unsuccessful short-term memories. The dictionary keys represent environment or task descriptions, while the values contain pairs of short-term memories and their corresponding raw trajectories.

When a new short-term memory is created, the corresponding dictionary is updated based on the environment and task. The system searches for matching environment or task descriptions. If a match is found, the memory and its trajectory are added to the list. If no match exists, a new key is created with an empty list, and the memory is added. Once the list reaches a batch size, the Long-term Memory Encoding Module processes the entries into long-term memories, storing environmental insights in the Env-Index and task insights (Success or Failure) in the Task-Index.

### Context-Dependent Memory Retrieval

During inference, the agent retrieves and organizes information based on the current task and environment using the context-dependent indexing structure. To retrieve the corresponding insights and raw trajectories, we propose a prompt-based Index Matching Module \({\cal M}_{\rm match}\), which takes the current task and environment as input and outputs the best-matching environment and task types. The next step is determining which insights and raw trajectories to recall as exemplars.

**(a) Retrieval and recall exemplars.** The agent retrieves relevant trajectories (CD-exemplars) from both the Env-Index and Task-Index, then prioritizes those in the intersection, where both task and environment match. If more exemplars are needed, the system recalls additional trajectories from the Task-Index or falls back on default exemplars from Reflexion. The Env-Index is not considered at this stage.

**(b) Retrieval and re-ranking insights.** To filter and rank insights, we propose a non-LLM-based sorting algorithm. It calculates the cosine similarity between each insight and short-term memories in the current environment or task, then sums the scores to prioritize the most relevant insights.

## 4 Experiments

### Setting

We validated the effectiveness of CDMem and conducted analyses on typical complex interactive reasoning tasks, navigation, and manipulation of situated virtual textual environments: ALFWorld and ScienceWorld. In ALFWorld, agents interact with different rooms to complete household tasks. Following the setting in ReactYao et al. (2023), we selected 134 environments from ALFWorld as our test set. These 134 environments are composed of 9 rooms and 6 task types. In ScienceWorld, which is similar to ALFWorld, the tasks are more complex. We selected tasks that could be completed within 20 steps to form our test set, which includes a total of 50 tasks.

#### 4.1.1 Baselines

Since this study primarily focuses on how agents generate and utilize memory during online interactions, we choose Reflexion, Expel and AutoGuide as baselines. To ensure a fair comparison, we implemented online versions of Expel and AutoGuide, referred to as "Expel-Online" and "AutoGuide 

Online". Furthermore, we also selected the React algorithm, which does not involve memory, as a baseline to reflect the model's performance without using any memory methods.

#### 4.1.2 Implementation

We conducted 5 trials in each environment of ALFWorld and ScienceWorld, with a maximum of 20 interaction steps per trial and environment. All methods use two exemplars. All experiments were run three times, and the experimental results were averaged. We evaluated our model with two configurations:(a) GPT-4o-mini: All components use GPT-4o-mini. (b) GPT-4o: Memory-related components use GPT-4o, while other components use GPT-4o-mini.

### Main Results

In both ALFWorld and ScienceWorld, CDMem achieved significant improvements over the baselines(see Figure 3).With Configuration 1, CDMem achieved a success rate of 85.8%, a 9% improvement over the AutoGuide-Online. Similarly, in ScienceWorld, CDMem scored 56.02, exceeding AutoGuide-Online's score of 44.84 by more than 10 points. With Configuration 2, CDMem's success rate on ALFworld reached 90.0%, nearly a 10% increase over Expel-Online's 80.3%.

### Ablation Studies

We verified the modules of CDMem contribute to performance improvement on ALFWorld shown in Table 1 and on ScienceWorld shown in Table 2. It can be seen that each component of CDMem plays an important role.

**(a)Role of Expert Encoding** Expert encoding compresses the raw trajectory and allows larger batch sizes during updates. Second, expert encoding improves the accuracy of successful shortcut

\begin{table}
\begin{tabular}{c c} \hline Method & Success Rate\% \\ \hline CDMem w/o expert & \(72.3\) \\ CDMem w/o task-encoding & \(84.3\) \\ CDMem w/o env-encoding & \(74.3\) \\ CDMem w/o CD-Exemplars & \(80.6\) \\ CDMem & \(85.8\) \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline Method & Score \\ \hline CDMem w/o task-encoding & 40.9 \\ CDMem w/o env-encoding & 54.7 \\ CDMem & 56.0 \\ \hline \end{tabular}
\end{table}
Table 1: Ablation Study on ALFWorld.(a) CDMem w/o expert: CDMem without expert encoding;(b) CDMem w/o task-encoding: Removal of task insights from the long-term encoding; (c) CDMem w/o env-encoding: Removal of environmental insights from the long-term memory encoding; (d)CDMem w/o CD-Exemplars: Instead of using context-dependent memory for exemplars, using fixed exemplars in ReflexionShinn et al. (2023).

Figure 4: The curve of the relationship between success rate and the number of trials.

Figure 3: Main Results. Average task success rates on ALFWorld with two model configurations and average scores on ScienceWorld with the base model configuration.

 

summarization. Rather than directly extracting shortcuts from successful trajectories, the agent first summarizes expert actions. This helps the agent focus only on identifying unnecessary actions when summarizing successful shortcuts.

**(b)Role of Short-term Memory** A well-designed reflection mechanism enables significant improvement in an agent's performance on subsequent trials of the same task. In our tests on ALFWorld (see Figure 4), CDMem showed a more substantial improvement between trial 0 and trial 1 compared to other methods.

**(c)Role of Environmental Insights** In ALFWorld, removing expert encoding and environmental insights had the largest impact and reduced about 10-points of SR. However, in the Science-World environment, removing environmental insights had only a 2-point impact. Moreover, we found that environmental insights significantly reduce hallucinations in the agent's behavior. For example, in ALFWorld, these insights provide accurate contextual information for item (such as microwave) usage.

**(d)Role of Task Insights** Task insights are similar to the status guidelines in AutoGuide. Compared to Expel, which uses all available insights, task insights are more focused and relevant to the current task, offering more accurate guidance. Similar to the role of environmental insights, task insights are also essential in mitigating hallucinations in the agent's behavior. For example, in ALFWord, successful task insights outline the correct sequence of actions for a task, helping the agent avoid performing actions that fall outside the planned steps due to hallucinations.

**(e)Role of CD-Exemplar** Although the fixed exemplars provided by Reflexion are also task-dependent, CD-Exemplars represent the actual interaction trajectories of the agent with the environment, making them more valuable as references for the agent. As a result, this led to an improvement of nearly 5% in ALFWorld.

### Computational Cost

We compared the computational cost of CDMem with Reflexion. Specifically, using the GPT-4o-mini configuration, we conducted five trials across 20 randomly selected environments from ALFworld to compare the computational cost shown in Table 3, which includes the number of API calls per individual sample, the total number of API calls for the selected dataset, and the corresponding monetary cost for processing the dataset.

## 5 Conclusion

We introduce CDMem, an efficient online memory framework for LLM-centric agents inspired by human memory mechanisms and designed for domain-specific industrial applications. Our approach incorporates a multi-stage memory learning method--expert encoding, short-term memory encoding, and long-term memory encoding--to effectively capture and organize knowledge from past tasks. We also introduce a context-dependent graph-based indexing structure that allows agents to retrieve relevant knowledge efficiently. We demonstrate that CDMem significantly outperforms state-of-the-art methods through extensive experiments, achieving notable performance improvements.

## References

* Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, and Karol Hausman et al. 2022. Do as I can, not as I say: Grounding language in robotic affordances. In _Conference on Robot Learning_, volume 205, pages 287-318.
* Besta et al. (2024) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. 2024. Graph of thoughts: Solving elaborate problems with large language models. In _Proceedings of the AAAI Conference on Artificial Intelligence_, pages 17682-17690.
* Chen et al. (2024) Minghao Chen, Yihang Li, Yanting Yang, Shiyu Yu, Binbin Lin, and Xiaofei He. 2024. Automanual: Constructing instruction manuals by llm agents via interactive environmental learning. In _Advances in Neural Information Processing Systems_, volume 37, pages 589-631.
* Fu et al. (2024) Yao Fu, Dong-Ki Kim, Jaekyeom Kim, Sungryull Sohn, Lajanugen Logeswaran, Kyunghoon Bae, and Hongglak Lee. 2024. Autoguide: Automated generation and selection of context-aware guidelines for large language model agents. In _Advances in Neural Information Processing Systems_, volume 37, pages 119919-119948.

\begin{table}
\begin{tabular}{l c c} \hline Computational Cost & Reflexion & CDMem \\ \hline API Calls per Sample & 2 & 4 \\ Total API Calls (Dataset) & 781 & 1155 \\ Monetary Cost (Dataset) & $0.33 & $0.51 \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of Computational Costs 

Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2023. Suspicion-agent: Playing imperfect information games with theory of mind aware GPT-4. _arXiv preprint arxiv:2309.17277_.
* Hu et al. (2023) Chenxu Hu, Jie Fu, Cenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. 2023. Chatdb: Augmenting llms with databases as their symbolic memory. _arXiv preprint arXiv:2306.03901_.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In _Advances in Neural Information Processing Systems_, volume 35, pages 22199-22213.
* Liang et al. (2023) Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as policies: Language model programs for embodied control. In _International Conference on Robotics and Automation_, pages 9493-9500.
* Liu et al. (2023) Lei Liu, Xiaoyan Yang, Yue Shen, Binbin Hu, Zhiqiang Zhang, Jinjie Gu, and Guannan Zhang. 2023. Thinkin-memory: Recalling and post-thinking enable llms with long-term memory. _arXiv preprint arXiv:2311.08719_.
* Loftus and Loftus (2019) Geoffrey R Loftus and Elizabeth F Loftus. 2019. _Human memory: The processing of information_. Psychology Press.
* Shinn et al. (2023) Noah Shinin, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In _Advances in Neural Information Processing Systems_, volume 36, pages 8634-8652.
* Smith and Kosslyn (2007) Edward E Smith and Stephen Michael Kosslyn. 2007. _Cognitive psychology: Mind and brain_. Pearson-/Prentice Hall.
* Wang et al. (2024a) Bo Wang, Tianxiang Sun, Hang Yan, Siyin Wang, Qingyuan Cheng, and Xipeng Qiu. 2024a. Innermory learning: A declarative learning framework for large language models. _arXiv preprint arxiv:2403.02757_.
* Wang et al. (2024b) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2024b. Voyager: An open-ended embodied agent with large language models. _Trans. Mach. Learn. Res._, 2024.
* Wang et al. (2023) Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023. Describe, explain, plan and select: Interactive planning with LLMs enables open-world multi-task agents. In _Advances in Neural Information Processing Systems_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. In _Advances in Neural Information Processing Systems_, volume 35, pages 24824-24837.
* Wu et al. (2023) Yue Wu, So Yeon Min, Shrimai Prabhumoye, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom Mitchell, and Yuanzh Li. 2023. Spring: Studying the paper and reasoning to play games. In _Advances in Neural Information Processing Systems_.
* Xue (2022) Gui Xue. 2022. From remembering to reconstruction: The transformative neural representation of episodic memory. _Progress in Neurobiology_, 219:102351.
* Yao et al. (2023a) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. In _Advances in Neural Information Processing Systems_, volume 36, pages 11809-11822. Curran Associates, Inc.
* Yao et al. (2023b) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. React: Synergizing reasoning and acting in language models. In _International Conference on Learning Representations_.
* Yao et al. (2024) Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, and Devansh Arpit et al. 2024. Retroformer: Retrospective large language agents with policy gradient optimization. In _International Conference on Learning Representations_.
* Zhang et al. (2024) Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. 2024. Agent-pro: Learning to evolve via policy-level reflection and optimization. In _Proceedings of the Annual Meeting of the Association for Computational_, pages 5348-5375.
* Zhao et al. (2024) Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. 2024. Expel: Llm agents are experiential learners. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 19632-19642.
* Zhong et al. (2024) Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 38, pages 19724-19731.
* Zhou et al. (2024) Runlong Zhou, Simon S. Du, and Beibin Li. 2024. Reflect-rl: Two-player online RL fine-tuning for lms. In _Proceedings of the Annual Meeting of the Association for Computational_, pages 995-1015.
* Zhu et al. (2023) Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. 2023. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. _arXiv preprint arXiv:2309.17277_.

 

Appendix

### Environment Details

We conduct experiments on the CDMem algorithm using two virtual textual environments: ALFWorld and ScienceWorld. Since the test set for ALFWorld is the same as those used in Reflexion and Expel, this section primarily introduces the construction of the ScienceWorld test set. The tasks in ScienceWorld are divided into 30 task types, each containing multiple variants. We select 10 types of tasks where the average length of the oracle agent's trajectories is **less than 20**. For each Task Type, we choose the top 5 variants, resulting in a total of 50 environments, which are shown in Table 4

### Implementation Details

In Table 5, we provide the specific version of the models used in the experiments.

### Prompt Templates

We present our prompt templates for different modules in Figures 5-11.

#### a.4 Additional Experiments and Analyses

##### a.4.1 Role of Expert Encoding

We analyze the role of expert encoding and find that applying expert encoding before short-term memory encoding results in more accurate summaries of successful shortcuts than directly using raw trajectories. We believe that the action sequence summarized by the expert encoding provides a reference for short-term memory encoding, reducing the cognitive load on the LLM when identifying redundant actions. This appearance is similar to the effectiveness of the "think step by step" approach. Here is an example of the appearance in Figure 12.

#### a.4.2 Effect of batch size

CDMem supports a larger batch size in the insights extraction phase than Expel and AutoGuide. We conducted batch size experiments on ALFWorld, and the results indicate that even with a batch size as large as 11, insights can still be effectively extracted. The experimental results are shown in the Table 6

### Pseudo code of methods

The pseudo code for the multistage memory encoding and storage, memory retrieval, and results rerank are shown in Algorithm 1-3

\begin{table}
\begin{tabular}{c|c} \hline Batch size & Success Rate\% \\ \hline
1 & 85.8 \\
3 & 84.6 \\
5 & 86.4 \\
7 & 86.2 \\
9 & 83.6 \\
11 & 84.3 \\ \hline \end{tabular}
\end{table}
Table 6: Batch size experimental results on ALFWorld

\begin{table}
\begin{tabular}{c|c} \hline Model & Version \\ \hline GPT-4o & gpt-4o-2024-05-13 \\ GPT-4o-mini & gpt-4o-mini-2024-07-18 \\ Embedding model & text-embedding-3-small \\ \hline \end{tabular}
\end{table}
Table 5: Version numbers of the models.

 

* [10] M. C. Bader, A. D. Krizhevsky, and R. G. Hinton. Learning to predict the best representation of a given image by using deep neural networks. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 257-268, 2013.
* [11] J. L. Friedman, S. W. Teh, and I. Goodfellow. Imagenet classification with deep convolutional neural networks. In _Advances in Neural Information Processing Systems (NIPS'94)_, pages 139-140, 2004.
* [12] Y. Bengio, P. Salakhutdinov, and V. Vanhoucke. Generative adversarial nets for large scale data. In _Advances in Neural Information Processing Systems (NIPS'94)_, pages 1035-1040, 2004.
* [13] H. Chen, X. Li, and Z. Sun. Deep residual learning for image recognition. In _International conference on computer vision and pattern recognition_, volume 1, pages 1035-1040, 2004.

[MISSING_PAGE_POST]

 



## 6 Conclusion

In this paper, we have proposed a novel approach to the design of the system and its applications. The main idea is that it can be used in order to design an object-oriented

[Instruction] You were unsuccessful in completing the task in this trial. Now, Your job is to build memory in two aspects:

1. Given the current trials's stem location information/items refer to mug, lettuce, bread, alarm dock, etc) and past environmental memory (summary of item locations in this environment), summarize them to form new environmental memory and output it.

2. Given interactive trajectory of current trial, environmental memory, expert actions(summary of action trajectories), and past reflectors(reflections you made in past trials). you must first consider what types of failure you meet and output corresponding reflections.There are three types of failure:

Planning Failure: The task planning has issues, such as missing steps or misunderstandings. Output the reflection of current planning issues and the correct plan.

Search Failure: Continuously searching for an item but unable to find it. Output the item's location already searched and the reflection of the future search plan. For example, if you tried A and B but forget C, devise a plan to achieve C with environment-specific actions.

Operation Failure: The expected feedback was not received after acting, such as returning with "nothing happens," which means the current observation doesn't match the current action. For example, attempting to take something from cabinet 1 while at the location of cabinet 4 and then returning "nothing happens." Output the reflection of the failed reason and correct actions.

[Exemplers] There are three exemplars to help you better understand the memory you should build.

**[fewshots]**

Input

Based on instruction and reference exemplars, it is your turn to build memory.

*** Input *** Current Trial {current_trial}

Current Trial Rem Location; {cur_item_location}

Environmental Memory; {environmental_memory}

Expert Actions; {except_actions}

 Past Failure Memory; *= Output *** Environmental Memory.

Figure 8: Our prompt template for short-term memory encoding of failure trajectory.

Figure 9: Our prompt template for long-term memory encoding of successful task insights.

 

[Instruction]

Given multiple experiences of expert observations/functions of containers, containers refer to drawer, shelf, sinkbasin, fridge, etcl and environmental insights you made in past trials, your job is to summarize them in new environmental insights. For example, given two expert observations, "I can clean mug with sinkbasin" and "I can clean egg with sinkbasin," and your past summary, "I can cool items with a fridge," a new summary can be "I can clean items with sinkbasin, I can cool terms with a fridge."

(Exemplars)

There are two exemplers to help you better understand the insights you should summarize.

**(env_freshots)**

(Input)

Based on instruction and reference exemplars, it is your turn to make a summary.

*** Input ***

Expert Observation: **(expr_obs)**

Post Insights: **(env_insights)**

*** Output ***

Figure 12: Role of Expert Encoding.The left column summarizes the successful shortcuts using only the raw trajectory, while the right column utilizes both the raw trajectory and expert encoding.

 

```
1:for\(i=1\) to \(N\)do
2:for\(j=1\) to \(K\)do
3:\(\tau^{i,j}=Interact\_with\_environment()\)
4:\(E^{i,j}_{\text{expert}}\) = \(\mathcal{M}_{\text{expert}}(\tau^{i,j})\)
5:\(E^{i}_{\text{short}}\) = \(\mathcal{M}_{\text{short}}(\tau^{i,j},\)\(E^{i-1}_{\text{short}})\)
6:\(task\_type\), \(env\_type\) = \(\mathcal{M}_{\text{match}}(tasks[j])\)
7:\(Task\)-\(Index[task\_type]\)[trajs].add(\(\tau^{i,j},\)\(E^{i}_{\text{short}})\)
8:\(Env\)-\(Index[env\_type]\)[trajs].add(\(\tau^{i,j},\)\(E^{i}_{\text{short}})\)
9:if\(len(Task\)-\(Index[task\_type]\) % \(bs==0\)then
10:\(task\_insights^{\text{old}}\)  = \(Task\)-\(Index[task\_type]\)[insights]
11:\(task\_insights^{\text{new}}\)  = \(\mathcal{M}_{\text{long}}(E^{\text{batch}}_{\text{short}},task\_insights^{ \text{old}})\)
12:\(Task\)-\(Index[task\_type]\)[insights]  = \(task\_insights^{\text{new}}\)
13:endif
14:if\(len(Env\)-\(Index[env\_type])\) % \(bs==0\)then
15:\(env\_insights^{\text{old}}\)  = \(Env\)-\(Index[env\_type]\)[insights]
16:\(env\_insights^{\text{new}}\)  = \(\mathcal{M}_{\text{long}}(E^{\text{batch}}_{\text{short}},\)\(env\_insights^{\text{old}})\)
17:\(Env\)-\(Index[env\_type]\)[insights]  = \(env\_insights^{\text{new}}\)
18:endif
19:endfor 

```
1:Task \(task\_tasks\), Index matching module \({\cal M}_{\rm match}\), environment dictionary \(Env\text{-}Index\), Task dictionary \(Task\text{-}Index\), Number of exemplars needed \(M\), Number of insights needed \(2L\), Default exemplar list \(default\_exemplers\), Rerank algorithm \(Rerank\) Output: environmental insights \(env\_insights\), Task insights \(task\_insights\), exemplars \(CD\_exemplers\)
2:\(CD\_exemplers=\emptyset\)
3:\(task\_type,env\_type={\cal M}_{\rm match}(task)\)
4:\(similar\_task\_trajs,similar\_task\_short\_memories\\  \text{=}Task\text{-}Index[task\_type]\)[trajs]
5:\(similar\_env\_trajs,similar\_env\_short\_memories\\  \text{=}Task\text{-}Index[env\_type]\)[trajs]
6:\(task\_insights=Task\text{-}Index[task\_type]\)
7:\(env\_insights=Env\text{-}Index[env\_type]\)
8:\(task\_insights=Remark(task\_insights,\\ similar\_task\_short\_memories\,))
9:\(env\_insights=Remark(env\_insights,\\ similar\_env\_short\_memories\,))
10:\(intersection\\  \text{=}~{}similar\_task\_trajs\cap similar\_env\_trajs\\ 10:Add exemplars to \(CD\_exemplers\) in order of priority: \(intersection,similar\_task\_trajs\), \(default\_exemplers\)
11:return\(task\_insights[:L],env\_insights[:L]\), \(CD\_exemplers\)
```

**Algorithm 2** Memory Retrieval

```
1:Input: Insights \(sorted\_insights\) Short Memories \(short\_memories\) Output: Sorted insights \(insights\)
2:for each \(insight\) in \(insights\)do
3:\(similarity\_scores\) = Faiss(\(short\_memories, insight\))
4:\(ranking\_weight\) = sum(\(similarity\_scores\))
5:endfor
6: Sort \(insight\) in descending order according to their respective the \(ranking\_weight\) as \(sorted\_insights\)
7:return\(sorted\_insights\)
```

**Algorithm 3** Rerank Algorithm