KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems
Authors: zixuan wang, Bo Yu, Junzhe Zhao, Wenhao Sun, Sai Hou, Shuai Liang, Xing Hu, Shaoshan Liu, Yinhe Han, Yiming Gan


Reviewer paper summaries: The paper introduces a memory system designed to enhance the planning capabilities of AI agents performing complex, long-sequence tasks in indoor environments. This system called KARMA distinguishes between long-term and short-term memory modules. The long-term memory captures static representations of the environment through 3D scene graphs, while the short-term memory dynamically updates with changes in object states and positions. This dual-memory setup significantly improves the agents' task execution efficiency and success rate by enabling more accurate and context-aware planning.
Summary of strengths: This research is centered on improving LLM agents by incorporating memory capabilities, a crucial element for developing truly autonomous agents capable of executing complex tasks in dynamic environments. Although various strategies have been attempted, this challenge remains unresolved. The ideas in the paper are articulated clearly and are well-backed by relevant recent literature. The findings indicate a marked improvement in success rates attributable to short-term memory and an increase in task efficiency due to long-term memory enhancements. Additionally, ablation studies have been performed to explore various methods of omitting information from short-term memory.
Summary of weaknesses: Short-term and long-term memory are distinct from one another. Long-term memory primarily acts like a map of spatial representations rather than holding detailed facts about the environment. This setup might suffice for a limited virtual setting with specific tasks, but ideally, there should be a way to convert short-term memories into long-term ones, along with mechanisms for retrieving information from long-term memory. Merely integrating all long-term memory into a model's context could overwhelm the model if the environment becomes too expansive.

The authors acknowledged that relying on vector similarity to search for relevant information in short-term memory might not be the most precise approach.

The authors considered as baselines only different variants of KARMA (with and without different types of memory). However, there exist other variants of embodied agents enhanced with a memory, see e.g.
https://aclanthology.org/2023.findings-emnlp.226.pdf
Comments, suggestions and typos: The authors note that increasing the memory hit rate enhances the agent's task execution efficiency. However, while the hit rate pertains to short-term memory, the main change in task execution efficiency was observed due to long-term memory. How are these results related to each other?
Overall assessment scores: 3