MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory
Authors: Junyeong Park
Junmo Cho
Sungjin Ahn
Venue: ICLR 2025 Poster (2025)
Keywords: Generalist Agents
Minecraft
Place Event Memory
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Significant advances have been made in developing general-purpose embodied AI in environments like Minecraft through the adoption of LLM-augmented hierarchical approaches. While these approaches, which combine high-level planners with low-level controllers, show promise, low-level controllers frequently become performance bottlenecks due to repeated failures. In this paper, we argue that the primary cause of failure in many low-level controllers is the absence of an episodic memory system. To address this, we introduce MrSteve (Memory Recall Steve), a novel low-level controller equipped with Place Event Memory (PEM), a form of episodic memory that captures what, where, and when information from episodes. This directly addresses the main limitation of the popular low-level controller, Steve-1. Unlike previous models that rely on short-term memory, PEM organizes spatial and event-based data, enabling efficient recall and navigation in long-horizon tasks. Additionally, we propose an Exploration Strategy and a Memory-Augmented Task Solving Framework, allowing agents to alternate between exploration and task-solving based on recalled events. Our approach significantly improves task-solving and exploration efficiency compared to existing methods. We will release our code and demos on the project page: https://sites.google.com/view/mr-steve.

Published as a conference paper at ICLR 2025
MRSTEVE: INSTRUCTION-FOLLOWNNG AGENTS IN MINECRAFT WITH WHERE-WHEE MEMORY
Junyeong Park',, Junmo Cho' Sungjin Ahn KAIST& & NNw York University
ABSTRACT
Significant advances have been made in developing general in environments like Minecraft through the adoption of L.LM--ugmented hierar- chical approaches. While these approaches, which combine high-level planners with low-level controllers, show promise, low-level controllers frequently become performance bottlenecks due repeated failures. In this paper, we argue that the primary cause of failure in many low-level controllers is the absence of an episodic memory system. To address this, we introduce MrSteve (Memory Recall Steve), a novel low-level controller equipped with Place Event Memory (PEM), form of episodic memory that captures what, where, and when information from episodes. This directly addresses the main limitation of the popular low-level controller, Steve-1. Unlike previous models that rely on short-term memory, PEM organizes spatial and event-based data, enabling efficient recall and navigation in long- hong-horizon tasks. Additionally, we propose an Exploration Strategy and a Task Solving Framework, allowing agents to alternate between exploration and task-solving based on recalled events. Our approach significantly improves task- solving and exploration efficiency compared existing methods. We will release our code and demos on the project page: https://iteeggoogll.com/////////
general-purpose purpose embodied Al
INTRODUCTION
The emergence of large-scale foundation models has driven significant advances in developing general-purpose embodied Al agents capable of generalizing across broad spectrum of tasks in complex. open, and real-world-lik. environments (Johnson et al.) 2016) Guss et al.] 2019 Fan et al. 2022b; Hafner 2022; Albrecht et al. 2022, Voudouris et al., 2023). While simulating such environments for effective learning and evaluation remains major challenge, Minecraft has become a leading testhed, offering demanding, open-ended environment with rich interaction possibilities. Its procedurally generated world presents agents with challenges like exploration, resource management, tool crafting, and survival, all requiring advanced decision-making and long-horizon planning. For instance, the task of obtaining a diamond requires agents to locate diamond ore and craft an iron pickaxe A. This process involves finding, mining, and refining iron to execute detailed long-term planning over roughly 24,000 environmental steps (Li et al. 2024).
requiring the agent
Solving such tasks through Reinforcement Learning (RL) approaches from seratch is nearly infeasible; however, recent M-a-uggeetted hierarchical methods have demonstrated a promising avenue (Huang al. 2022a b. Wang et al. 2023a). These methods feature a division between high-level planners and lowelevel controller policies. High-level planners, driven by Large Language Models (LLMs) or Multimodal Large Language Models (MLLMs), propose subgoals by utilizing the reasoning abilities and prior knowledge inherent in LLMs (Brown et al. 2020 Touvron et al. 2023 OpenAl 2024).
These subgoals, conveyed textual instruction form, are then sequentially passed to learned, dnstruction-folloviii low-level controller for execution (Wang et al. 2023c Li et al.) 2024).
For this framework to be effective, is essential that both the high-level planner and the low-level controller improve in tandem. However, previous research has primarily focused on enhancing
high-level planning, e.g., via maintaining skill library (Zuu et al. 2023 Wang et al. 2023b Qin et al.
* 'Equal contribution. Correspondence Junyeong Park and Sungjin Ahn.
Contact: (jyp10993 0997, ( n. ahn) @ka st.acc.k

---

Published as a conference paper at ICLR 2025
2024: Li et al. 2024), often assuming that low-level controllers will efficiently execute the subgoals provided by the high-level planner. However, this assumption frequently does not hold in practice, and the low-level controller becomes significant performance bottleneck (Cai et al. 2023b).
In this regard, we specifically focus on limitations in Steve-l (Lifshitz et al. 2024), the most widely used low-level instruction-folle followingg controller framework. Steve-1 is foslluctionfollowinng policy obtained by fine-tuning the Video Pre-Training (VPT) (Baker et al. 2022) model. A primary limitation we focus on its constrained episodic memory capability. Steve- is based on Transformer-XL. (Dai) et al. 2019), which leverages relatively short-term memory, retaining only the last 128 hidden states. Given Minecraft's simulation speed of 20Hz, this memory span amounts to only a few seconds of gameplay. While it can be increased, the quadratic complexity and FIFO-only memory structure of transformers make them significantly inefficient for long-horizon tasks.
As result, when the agent requires information beyond this short memory span, it tends to forget past events within the episode and reverts to inefficient random exploration for each new task, consuming excessive time. For example, when given a task like "Find a Cow", the agent is unable to recall, 'I've seen it before near the river in the north'. Ideally, a low-level agent would instead maintain an episodic memory of meaningful events and recall relevant information. See Figure I for more detail illustration. Moreover, Steve- not only lacks the ability to recall such memories but also the ability to navigate directly to the associated locations, which could help avoid unnecessary exploration. Instead, Steve-1 relies on a "go explore" instruction, randomly exploring until it stumbles upon the resource by chance. This inefficiency in executing low-level primitives is not addressed by high-level planners, which focus on optimizing the sequence of high-level skills (i.e., the plan) but do not optimize the execution of the primitives themselves. We elaborate more on this in Appendix A
In this paper, we introduce an enhanced low-level controller agent, MrSteve (Memory Recall Steve), designed to address the limitations of Steve- The key innovation of MrSteve is the integration of Place Event Memory (PEM) which is the instantiation of What-Where- While previous approaches have explored episodic memory, they primarily target high-level planners, such as building libraries of high-level skills and plans, which do not directly improve the low-level controller's performance. We argue essential for the low-level controller to possess memory capabilities. To address this, PEM manages memory more effectively, surpassing the limitations of the non-scalable FIFO memory found in transformers. PEM stores spatial and event-based information, allowing the agent to hierarchically organize and retrieve details about locations and events it has previously encountered. For PEM to be fully effective, the agent must also move directly to the desired location along with the ability modulate between exploration and goal-directed navigation- and-execution, a capability lacking in Steve- Therefore, we introduce the second component of MrSteve: the Exploration Strategy and Memory-Augmented Task Solving Framework. Built upon the PEM structure, this framework enables the agent to alternate between exploration-wher no relevant information is stored-and task-solving by recalling past events when applicable. This is made possible and effective with our new navigation policy, VPT-Nav.
Episodic Memory.
Our contributions are as follows. First, we point out the limitations of Steve- the most widely used cnstruction-folloiing controller, and show how its bottlenecks can be addressed with MrSteve. Second, we introduce Place Event Memory (PEM), a novel hierarchical memory system that orga- nizes spatial and event-based data for efficient querying and storage, even under limited memory capacity. Third, we propose an Exploration Strategy and Task Solving Module built on PEM that enables efficient exploration while maintaining high task-solving performance in Minecraft. Last, we demonstrate that our agent significantly outperforms existing baselines in both exploration and long sequence of tasks solving. We will release the code for further research.
2 RELATED WORKS
Low-Level Controllers in Minecraft Earlier works (Guss et al., 2019 Lin et al.) 2021 Mao
ct al., 2022 Cai et al.) 2023a Hafner et al. 2023: Zhou et al. 2024a) introduced policy models for simple tasks in Minecraft. MineCLIP (aan et al. 2022b leveraged text-video data to train contrastive ldno-language model as a reward model, while VPT (Baker et al. 2022 was pre-trained on unlabelled videos without text-based instruction input. Steve- (Lifshitz et al. 2024) extended
VPT by incorporating text instructions to generate low-level actions based on human demonstration
data. GROOT (Cai et al. 2023b) used reference video instead of text for goal-conditioned behavior
2

---

Published as a conference paper at ICLR 2025
"obtaiin log"
Explore
Execute
Unseen
Mobtain beet"
Recall
Explored
Execute
Explore
Seen
Navigate
Success
Figure 1: Sparse Sequential Task Solving Scenario. The first task is to obtain a log. The agent explores to find a tree. While scarching. the agent observes cow but continues focusing acquiring the log. Once the log is obtained, the next task is to obtain water bucket. Remembering that already explored the forward direction while searching
the tree, agent chooses to explore the right. After gathering final task obtain meat, which can he acquired from the cow. Recalling the cow's location, the agent navigates there and completes the task by ohtaining the meat. Note that each task takes a few thousand steps to achieve. This scenario highlights the significance of episodic memory for efficient exploration and task-solving in an open-ended world where task-relevant resources are sparsely distributed.
water bucket, the
cloning. Recently, MineDreamer (Zhou et al. 2024b) leveraged Steve-1 generating subgoal images with MILM and Diffusion based on text and current observation for improved control. However, these agents lack episodic memory, forcing agents to start new tasks from scratch. MrSteve addresses this by integrating episodic memory, making more effective in sequential tasks.
LLM-Augmented Agents The development of LLMs has significantly advanced agents in Minecraft (Wang et al. 2023a b). These works utilize pre-trained LLMs zero- shot planners (Brown et al.) 2020 Touvron et al. (203)) leveraging their powerful reasoning capabilities to generate subgoal plans or executable code. Broadly, this line of research can be divided into two approaches: one that uses
LLMss for code generation to interact with the environment directly (Wang et al.) 2023a 7huu et al.)
2023 Qin et al. 2024 Liu et al. 2024). and another that generates text-based subgoals which are then executed by gow.-caooooooo condiiiood low-level controller, such Steve- programmed heuristics (Nottingham et al. 2023 Yuan et al. 2023 Li et al. 2024). In the latter approach, to ensure LLMs tocus on high-level semantic reasoning, the low-level controller must efficiently execute subgoals. While combining LLM as high-level planner with MrSteve is one possible direction, we focus on enhancing low-level controller's capabilities based the new type of memory in this work.
Memory Agents Memory systems in agents primarily aim to retrieve robust and accurate high- level plans for long-horizon tasks (Zhang al. 2023 Song al. 2023 Kagaya et al. 2024 Sun et al. 2024 Shinn et al. 2024). Existing works store successful task's 's text instruction and its plans in language often with observations for robust retrieval, which is useful when plans for the new task already exist in memory. Voyager (Wang et al. 2023a) uses an unimodal storage of achieved skill codes in the form of text. GITM (Zhu et al.] (023) integrates text-based knowledge and memory for higher reasoning efficiency and stores entire skill codes after goal is achieved. Recently, MP5 QQn et al. 2024) and JARVIS-1 (Wang et al. 2023b) enhance planning by storing plans and whole multimodal observations in the abstracted memory, allowing for situation- Optimus-1 (Li et al.) 2024) introduces a multimodal experience pool that summarizes all multimodal information during agent's execution of the task improving storage and retrieval efficiency. However, these memory systems store the sequence of high-level skills or plans for high-level planners, which not optimized for low-level controllers. We address this problem with Place Event Memory.
retrieval, while
METHOD
In this section, we describe our agent, MrSteve (Memory Recall Steve). We begin with the problem setting, followed by step-by-step construction of our agent's main modules.
Problem Setting In this work, we define a sparse sequential task scenario where the agent is continuously given tasks 100 through text instructions (e.g., Obtain water bucket) from the
3

---

Published as a conference paper at ICLR 2025
(a) Agent Overview
(b) Place Event Memory
Pooc Cluster Event Cluster
Enveonmen:
MiSteve
Explore
Agent's Trajectory
Selver Module
Bounc-Bucc
WP Bickee
110
Mode Selector
Riace Cluster Event Cluster
Place Event Memory
Place Cluster
Text Instruction
Memory Module
Figure 2: MrSteve and Place Event Memory. (a) MrSteve takes agent's position, first person view, and text instruction, and utilizes Memory Module and Solver Module to follow the instruction. (b) MrSteve leverages Place Event Memory for exploration and task execution, which stores the novel events from visited places.
environment subgoal plans by LLM. Additionally, assume that task-relevant resources (e.g., water, cow) rarely exist and are sparsely distributed in the environment, making it essential to memorize novel events from visited places for future tasks shown in Figure] When an episode begins, for every time step I, the agent is provided with the observation X, = {&,, Fe, (). which consists of the pixel observation & "xx' representing the first person view of the environment, the positional information 4 = (coord,, coord,, coord,, yaw, pitch) R$, which denotes the agent's relative 3D position and camera angles with respect to initial position lo. and time
Instruction Following Policy In sparse sequential task, naive approach is to employ Steve-1 1 (ifshhitz et al. (2024). an cestruction-following policy Fins(aa/hg Tn) that generates low-level controls (mouse and keyboard) in Minecraft, Here, h, is past pixel observation sequence 1- While past observations are processed by Transformer-XX XI. layers Steve-1. I, the model is ineffective at recalling observations from a few thousand steps ago (Lampinen al. 202T). Additionally, the Transformer's quadratic complexity makes it significantly inefficient to process thousands of observations. This makes Steve-1 poorly suited for sparse sequential task, as it cannot recall visited places or task- relevant resources seen in the past. To address this, we propose MrSteve which stores novel events from visited places for efficient sparse sequential task-solving.
instruction following policy that consists of Memory Mod- ule and Solver Module, shown in Figure a In Memory Module, use the mem- ory called Place Event Memory M, that stores novel events from visited places (Figure (b)).
MrSteve a eeemory-augmente
Based on Place Event Memory, Mode Selector in Solver Module decides between Explore mode and Execute mode. When task-relevant resource exists in the memory, Ex- plore mode is selected, and the agent explores with our hierarchical exploration method. a task-relevant resource exists in the memory, Execute mode is selected, then the agent navigates to the resource's position and executes Tiss (i.e., Steve-1) to solve the task. Algorithm outlines the task-solving loop of MrSteve, which repeats every fixed step when a new task is given (More details in Appendix D). With these modules, MrSteve can efficiently explore and recall task-relevant resources from the memory solve sparse sequential task in Figure [ In the following sections, we describe how each module in MrSteve constructed. We begin by constructing Memory Module.
Algorithm MrSteve Single Loop
Require:
Memory M,, and task Ta 1: candidates + Read MM, Tx)
2: if candidates + 0 then
3:
X1,4= OneOO(candidateen
4:
Navigate with TL-Nav
5:
Execute with Tinst
6:
else
7:
Explore with TH-Catt TL-Nav
8: end if
3.1 MEMORY MODULE: CONSTRUCTION OF PLACE EVENT MEMORY
The simplest form of memory FIFO Memory, denoted as M, with capacity N. At every time step, instead of storing X, in Mr, we can extract a semantic representation from the video a Ha video encoder to store an experience frame I, = {er, 4,, t}. where é = Enc,, term e the video embedding time step t. When the memory exceeds its capacity, the oldest frame is removed. For memory read, we calculate the cosine similarity between the task embedding f. (n,,(T,) and the video embedding e, in M, to retrieve task-relevant frames. Here,
with For simplicity,
4

---

Published as a conference paper at ICLR 2025
(a) Selector
VPT-Nav
Pick evp-ievent clusters from su(s.) (L.)) - CLIP, (ck.)
Policy
Net selected
LoRA
T:XLg
Be 4y(%)) tor 4x(%)) Explore
alignment score: (x(T) CLIP,(T)) CLIP,(es)
Figure Mode Selector and VPT-Nav in MrSteve. Mode Selector with Place Episodic Memory. It decides agent's mode (Explore or Execute) based on whether task-relevant resource the memory. It uses a hierarchical read operation. (b) Architecture of Goal-Conditioned VPT Navigator.
video encoder Enc,, text encoder Enc,, and H = 16 from MineCLIP (Fan et al. 2022a). which is CLIP (Radford et al. 2021 trained on web videos of Minecraft gameplay and associated captions.
While FIFO Memory offers benefits from its simple memory operations, it has two drawbacks. First, the computational complexity of the read operation grows linearly with the memory size. Second, the bias toward removing the oldest frames can be problematic in sparse sequential task as in the Figure I scenario, where task-relevant frames from visited places are lost.
Place Memory To address these issues, Place Memory (Cho al. 2024) divides the agent's positions {b))==0 in the trajectory into clusters of distinct places, where each place cluster is assigned a FIFO Memory. Here, agent's 's top-down location and its head direction. Place Memory is represented as Mr = { {M& | where Mj is the k-tth place cluster with center position EMP,- and center embedding eM2-- Here, eM, the video embedding whose position is closest to M, This structure improves the efficiency of the read operation by extracting top-k place clusters with their center embeddings first, then fetching relevant frames from these clusters. Furthermore, when memory capacity is limited, the oldest frame removed from the largest place cluster, allowing the agent to retain memories in diverse places.
term & (coordg, coordy. yaw) agent's 's position, which is concatenation of
While Place Memory prioritizes storing experience frames across diverse places, its FIFO structure within each cluster still loses novel experience frames in the past. For instance, if an agent stays in a place where zombies burn and disappear for a long time, the place cluster removes the frames of burning zombies that can crucial in upcoming tasks. This highlights the importance of focusing on visually distinct experience frames rather than storing them sequentially, which can be redundant.
Place Event Memory resolve this issue, we introduce Place Event Memory built on Place Memory, which captures distinct events that occur within each place cluster (Figure ())). While Place Memory uses agent's position to cluster experience frames, there is no criterion for clustering frames to form events. To tackle this, use the cosine similarity of video embeddings from MineCLIP for criterion.
J Specifically, each place cluster My is subdivided into event clusters, denoted as { where each E represents the i-th event cluster in k-th place cluster, characterized by center embedding E These event clusters are newly created and updated as the place cluster accumulates a certain number of additional experience frames. For generating event clusters, MP-Means algorithm (Dinari & Freifeld 2022) is applied on video embeddings of these frames, generated by MineCLIP, and the resulting cluster centers become center embedding of each cluster. If the cosine similarity of the cluster centers between a newly created cluster and an existing cluster is higher than threshold e (we define that two event clusters are indistinct), the two clusters are merged prevent redundancy and ensure distinct event clusters within each place cluster. When memory capacity is exceeded, the oldest frame in the largest event cluster removed, thus the memory can retain diverse places and distinct events within each place. More details on Place Event Memory can be found in Appendix
3.2 SOLVER MODULE: MODE SELECTOR, EXPLORATION, AND NAVIGATION
In this section, we introduce the remaining components in Solver Module, which are Mode Selector, and hierarchical policies TTI.at NayNay for episodic exploration, and goal-reaching navigation.
5

---

Published as a conference paper at ICLR 2025
Mode Selector Mode Selector decides between Explore and Execute mode by checking whether task-relevant resource exists in the memory. If the resource exists, the agent chooses Execute mode, or Explore mode otherwise. When Place Event Memory is employed, Mode Selector first picks top-k event clusters with task alignment score 8i (T, ) = (T,) CLIP, (e between task embedding, and center embedding of event cluster. Then, it calculates task alignment scores on experience frames in top-k event clusters and gathers frames with scores higher than task threshold & as shown in Figure Baa). We note that leveraging Place Event Memory offers computational efficiency with hierarchical read operation compared to FIFO Memory, which calculates the alignment scores on whole frames in the memory. We provide a comparison of memory query time in Appendix for further insights.
Hierarchical Episodic Exploration We propose memory -ased hierarchical exploration method that allows the agent to efficiently explore the environment while minimizing revisits to previously ex- plored positions. This is achieved through high-level goal selector Nav (9t l M,) and a low-level goal achiever (L, (.) where h1 128:tt and Et and 9r are the agent's s current location, and goal location in (coord,,, coord,,) ), respectively. We introduce a Count-Based exploration strategy Tang et al. 2017 Chang et al. 2023) for the high-level goal selector.
9t),
Yamauchi 1998
Specifically, L x L visitation grid map my is used with the agent's starting location set as the center of the map. The locations of the agent's ''s trajectory are discretized and marked on the grid. The goal selector then divides the visitation map into grid cells of size G x G, and selects the location of the grid cell with the lowest visitation count as the goal, 9t- If multiple grid cells have the same minimum count, the cell closest the current location E is chosen. This approach directs the agent toward unexplored locations, while minimizing unnecessary revisits. The size of grid cell can be dynamically adjusted to balance between broader exploration and finer local searches. Additionally, in an infinitely large map, the visitation map can be easily expanded by adding new grids, and further hierarchies on visitation maps can be introduced for efficiently managing explored locations.
Conl-Conditioned VPT Navigator Once the goal location is selected by the high-level goal selector, it is crucial for the agent to navigate to the goal accurately. However, navigating complex terrains (e.g., river, mountain) requires human prior knowledge, where pure RL policy trained from scratch in prior work (Yuan et al., 2023) often shows suboptimal navigation ability. To address this, we use the VPT as our starting policy, and fine-tune it for goal-conditiones navigation policy. We name this policy as VPT-Nav. In VPT-Nav, we add goal embedding Gz gt) in the output of TrXL, in VPT with LoRA adaptor (Hu et al. 2021a) as in Figure ()). We used PPO (Schulman et al.) 2017) for fine-tuning goal encoder G LoRA parameters, policy Tg, and value Ue with reward based on the distance the goal location. We note that our VPT--av introduces several differences from prior VPT fine-tuning methods, thoroughly investigated in Appendix L
3
4 EXPERIMENTS
This section presents a step-by-step validation of our agent MrSteve across various environments and conditions. We begin by evaluating the exploration and navigation ability of MrSteve, which is crucial in sparse sequential tasks (Section 4.1. Then, we demonstrate MrSteve's 's capability to solve A-B-A task sequentially where the memory is necessary to solve the task A twice (Section 4.2). Additionally, we show that the proposed Place Event Memory outperforms other memory variants, particularly when memory capacity is limited (Section 4.3) Lastly, we showcase the generalization of MrSteve to long-horizon sparse sequential task (Section 4.4). Each baseline and task is explained in each of the experiment sections with more details in Appendix
4.1 EXPLORATION & NAVIGATION FOR SPARSE SEQUENTIAL TASK
In this section, we evaluate the exploration and navigation ability of our agent. To verify this, we placed an agent in 100 x 100 block map with complex terrains such as mountains and a river and gave 6K steps to wander around the map. Since successful exploration in Minecraft involves covering as much of the map as possible while minimizing revisits to previously visited locations, we measure two metrics: Map Coverage and Revisit Count. Map Coverage is calculated by dividing the map into 11 x 11 grid cells and measuring the percentage of cells covered by the agent's 's trajectory. Revisit Count measures the average number of times the agent visits the same grid cell.

---

Published as a conference paper at ICLR 2025
Table 1: Map Coverage and Revisit Count of different exploration policies. Our exploration method (High-Level:
Count-Basse Bastd, Low-Level: VPT-Nav) performs the best.
High-Level Low-Level
Count-based
RNN-based
Steve-1
VPT-Nav (Ours)
DQN
VPT-Nav
DQN (Plan4MC)
Map Coverage (1)
84.42 + 0.06
31.83 + 0.11
29.82 + 0.07
16.36 = 0.04 6.2 1.73
50.77 + 0.13 2.68 1.36
Revisit Count (L)
0.38 0.6
4.47 1.43
4.72 + 1.73
(Ours)
(Plan4MC) High-Level: MNN--based Low-Level: DQN
High-Level: Count-based
High-Level: Count-based Low-Level: DQN
High-Level: RNN-based Low-Level: VPT-Nav
STEVE-1 prompted "Go Explore"
Low-Level: VPT-Nav
Figure 4: Agent's trajectories of length 6K steps on 100 x 100 block map with different exploration methods. The lefimost figure is the agent's 's trajectory from our exploration method.
To demonstrate that our proposed hierarchical episodic exploration with Count- Bauut high-level goal selector and low-level goal achiever VPT-Nav is more effective for exploration in the given map, we compared our method with the following baselines: Steve-1 (Lifshitz et al.
2024). and exploration method from PM (Yuan et al. 2023). j. For Steve- 1, we provided the "Go Explore" instruction to assess its exploratory behavior as in prior works (Cai al. 2023b Zhou et al.
2024b). Plan4MC, on the other hand, employs hierarchical approach where the high-level RNN policy selects the next goal location based on past locations, and a low-level DQN policy is used for goal-reaching. Since the input-output space of our method and PIan4MC is identical, interchanging high-level and low-level policies between two approaches is allowed so that we can evaluate the benefits of each component.
As shown in [ and Figure 4 we can see that our exploration method outperforms other baselines. While Steve- I showed decent performance in Map Coverage, it repeatedly visits previously explored places because of lack of memory. In the case of hierarchical exploration, high-level RNN policy struggled with memorizing visited places as the trajectory gets longer, resulting in high Revisit Counts. Additionally, the low-level DQN policy had difficulty navigating complex terrain, such as mountains and rivers, showing low Map Coverage. On the other hand, the Count-Based goal selector that directs the agent to the least-visited locations as goals and the VPT-Nav that effectively reaches those goals resulted in strong exploratory behavior. Furthermore, to show the robustness of VPT-Nav, report its navigation capability in diverse terrains in Appendix L
4.2 SEQUENTIAL TASK SOLVING WITH MEMORY IN SPARSE CONDITION
In this section, we demonstrate our agent's capability to solve sparse sequential tasks based on exploration methods studied in Section 4.1 To evaluate this, we introduce ABA ABA-Sparse task, which consists of A-B-A tasks given sequentially with text instructions. Task A involves gathering a sparse resource, which can be either water bucket collecting a dense resource, chosen from log dirt leaves total 20 tasks. The agent spawns in a 100 100 block map, and the A resource exists in a single location, while the B resource can be found in multiple locations. The agent is given 12K steps with unlimited memory capacity to complete all tasks. Since finding the sparse resource is challenging, the task requires an efficient exploration algorithm. Moreover, after solving tasks A and B, memory becomes crucial to return to the location of resource A within the time limit. We measure success rates and task duration for evaluation.
beef wool
milk ①. Task B requires S seeds 4², or sand
making
To verify the benefits of efficient exploration and the memory, we compared the following agents: Steve-1, 1, MrSteve with exploration method from Plan4MC and FIFO Memory (PMC-MMSSeve-FFM) and our agent MrSteve with Count-Based goal selector and VPT-Nav for exploration and Place Event Memory. We also test MrSteve with different memory variants, MrSteve-FM, MrSteve-FM.,
7

---

Published as a conference paper at ICLR 2025
Wool-Dirt-Wood
Beef-Log- Beef
Water-Seed-Watee
Milk-Sand-Milf
1.0
1.0
1.0
1.0
Success
0.6
0.6
0.6
0.6
0.2
0.2
0.2
0.2
0.0
Task Duration (x104)
5.3
5.35
3.5
5.4
2.2
2.4
Task 2.0
Task
Task
Task
Task
Task
A
MrSteve
MrSteve-EM
MMSStee.PM
0
MrStowe-FM
PMC-MrStews-FM
Steve-1
Figure Success Rate and Task Duration of different agents in -BA-See tasks. Task A refers to the first task in the A- A-B-A task sequence, while Task A' refers to the final A task in the A-B-A task sequence. We note that MrSteve, as well as its memory variants, outperforms Steve-1, 1, which lacks the memory. Additionally, while Steve-1 1 takes a similar amount of time to solve both task and task A', MrSteve solves task A' much faster. The full results on all 20 tasks in Appendix H and investigations about memory variants in Appendix O
MrSteve-PM, which use FIFO Memory, Event Memory, and Place Memory, respectively. Here, Event Memory is Place Event Memory without place clusters, which stores the frames based on visual similarity. We explain the details of Event Memory in Appendix E.3
As shown in Figure 5 it is clear that MrSteve outperforms Steve- This is because MrSteve find task-relevant resources faster with efficient exploration and store the location of the task A resource, allowing it to revisit the location and solve task A again within a limited time. This is evident from the task duration in Figure where MrSteve shows a shorter task duration than Steve-I on the first A task. When solving the second A task, MrSteve exhibits a much shorter task duration compared to the first A task, while Steve- I takes similar or even greater number of steps. While other memory- augmented baselines showed similar performance MrSteve, PMC-MrSSeve-FNM performed worse due a suboptimal exploration method, making it difficult to find the sparse resource. We report the full results on all 20 tasks in Appendix H
4.3
TASK SOLVING WITH MEMORY
We demonstrated in Section 4.2 that memory is essential in solving sparse sequential tasks when there no limitation memory capacity. However, in real-world scenarios where memory capacity is limited, memorizing visited places and novel events becomes important. In this section, show that Place Event Memory can benefit in this scenario. To verify this, we introduce three Memory Tasks, which are Find Water, Find Zombies' Death Spot, and Find First- Visited House. In all tasks, the agent begins with an exploration phase, followed by a task phase. In the exploration phase, the agent follows a fixed exploratory behavior. In the subsequent task phase, the agent is given a MineCLIP embedding = Enc., (tr {ir]1²] as a task embedding instead of text instruction, where i, is the pixel observation seen in the exploration phase. Then, this becomes an image goal navigation task where an agent should navigate to the location of the given image.
In all tasks, the exploration phase 3K steps, and the agent's memory capacity is limited to 2K. In Find Water task, the agent stays near water for 0.5K steps. then travels to random location. In the task phase, the agent should return to the water (Figure ((a)). This task evaluates whether an agent can memorize water frames in the past. In Find Zombies Death Spot, the agent sees burning zombies for 1K steps (zombies burn for 0.5K steps then disappear), then travels. The task is to return to where zombies burned (Figure b)). This task evaluates whether an agent can memorize distinct events (zombies burn and disappear) in the same place. In Find First- Visited House task, the agent sees the first house for 0.1K steps, then goes the second house and stays until 2K step, then travels. The task is to return to the first house Bic)ur/B(c).. This task evaluates whether an agent can memorize two visually similar houses in two different places.
8

---

Published as a conference paper at ICLR 2025
(a) Find Water
Stay
travel
02
0)
Find Zombies' Death Spot
stay for long time before travel
0.
styy
travel
Succes
Zombies Burn (1 - O)
Zombies Gone (1 1K)
After Travel (1 - =3K)
(c) Find First-Visited House
stay for long time before travel
Rate
0.6
travel
travel
0.6
Midree
Stoee
House
2nd House 113)
After Travel
MMSineFM
Figure 6: The overview of Memory Tasks, and Success Rate for each Memory Task from different agents. Memory Tasks are basically navigation tasks reaching the location the previously seen experience frame. observe that MrSteve which uses Place Event Memory shows high success rates in all tasks.
To evaluate how each memory type performs in the Memory Tasks, we tested Steve-1, MrSteve, MrSteve-FM, MrSteve-EM, and MrSteve-PM. In Figure 6 the performance of each memory type illustrated. In all tasks, Steve-1 which lacks memory, showed worst performance since it has to find the targets from scratch. In Find Water task, MrSteve-FM does not have the water frames in memory task phase, so it should explore find the water showing about 60% success rate. In contrast, other agents store the water frames by allocating place cluster event cluster in water location, allowing the agent to recall the frames and easily return to water, showing high success rate. In Find Zombies' Death Spot task, MrSteve-PM loses burning zombies' frames since Place Memory removes the frames the largest place cluster, which is zombies' place, showing about 60% success rate. However, MrSteve- EM and MrSteve store the burning zombies' frames novel event, allowing the agent to easily return to zombies' spot, showing high success rate, In Find First- Visited House task, MrSteve-EM loses frames of first house, since clusters two visually similar houses as the same event, showing about 50% success rate. However, MrSteve-PM and MrSteve store the two houses in different place clusters, enabling the agent return to the first house, showing high success rate. These results suggest that Place Event Memory demonstrates its strength in memory- limited settings, where memorizing both visited places and novel events is crucial for task completion.
4.4 LONG-HORIZON SPARSE SEQUENTIAL TASK SOLVING WITH MEMORY
In this section, to see how MrSteve generalizes to long-horizon tasks, we introduce two sparse sequential tasks. For both tasks, the agent plays in 200 x 200 block map for 500K steps (About 7 hours of gameplay) with 20K steps of memory capacity. The first is Long-Instruction task, where the agent continuously given random tasks from Obtain X. Here X can be water beef wool log dirt or seeds 4:. If the agent fails to complete the task within 20K steps, the task is changed. This task requires efficient exploration in large map, and managing memory memorize places with task-relevant resources to continuously solve the given tasks.
The second task is Long-Navigation task similar Memory Tasks in Section It has exploration phase of 16K steps and task phase. In the exploration phase, the agent observes six events in different places: 1) burning zombies, 2) river, 3) sugarcane blow up. 4) spider spawn, 5) tree, and 6) house, spending 2K steps at each place. In the task phase, the image goal is continuously given randomly selected from the frames in the early steps of the event. For instance, the task is to reach sugarcane place, the image of sugarcane place before blow up is set as an image goal. This task requires managing memory to retain distinct events different places.

---

Published as a conference paper at ICLR 2025
The results are shown in Figure
Succeeded Tasks
Long-Instruction Task
Long- -angiaaiigt Task
For Long-Instruction task, we ob- serve that MrSteve, and MrSteve-PM solved over 80 tasks, showing their
100
80
80
60
60
40
capabilities of retaining task-relevant resources in different places effec- tively. MrSteve-EM solved around 50 tasks, suggesting event-based memory less effective than place-based based mem- ory. This is because similar events in different places, like cows and sheeps living in visually similar forests, are in the same event cluster, possibly los- ing task-relevant frames. For the remaining baselines, they either have suboptimal exploratory behaviors or keep losing the task-revelant frames, solving less than 50 tasks. For Long-Navigation task, MrSteve, and MrSteve-EM solved around 70 tasks, showing the ability to retain novel events occured in different places. In the case of MrSteve-PM, it removes the frames in early time steps of each place cluster, losing novel events (e.g., sugarcane before blow up), thus solving less than 20 tasks. For the remaining baselines, they lose or cannot retain frames in the early stage of an episode solving less than 10 tasks. These results suggest that MrSteve demonstrates its strength in long-horizon tasks.
20
20
100K 200K 300K 400K 500K
0
100K 200K 300K 400K SD0K
Time Steps
Time Steps
MrSteve
MrSteve PM
PMC-MrSSeee-FFM
MrSteve-EM MrSteve-FM FM
Steve-1
Figure 7: The performance in Long-Intruction task and Long- Navigation task. MrSteve performs well in both tasks.
5 LIMITATIONS
This work focuses on improving Steve-1 through the introduction of What-Where- When episodic memory, significantly enhancing the agent's ability to retain and recall past events for more efficient task-solving. Our experiments demonstrate that MrSteve exhibits significantly enhanced performance when integrated with LLM-augmented agents for high-level planning. However, current limitations prevent the high-level planner from accessing PEM in the low-level controller. Future work could explore enabling PEM access for high-level planners, which could generate more accurate plans by leveraging the agent's 's episodic memories, further enhancing the system's 's capabilities for complex, long-horizen tasks.
We list up a few more limitations. First, our experiments limited to surface-level exploration in the Minecraft environment, omitting underground navigation, which a crucial aspect the game mining plays central role. However, map could easily be extended to include vertical dimensions. Additionally, while our VPT-Nav demonstrated strong navigation abilities in plains biomes, more challenging terrains, such crossing cliffs that require skills like building bridges, were not addressed. Lastly, we used exact position data in Minecraft, which may limit the model's adaptability robotics tasks where positional information is often noisy. One possible direction is adapting MrSteve in environments with noisy positions.
hierarchical exploration methods based visitation
6 CONCLUSION
In this paper, we introduced MrSteve (Memory Recall Steve), novel low-level controller designed to address the limitations of current LLM-augmented hierarchical approaches in general-purpon embodied AI environments like Minecraft. We argued that the primary cause of failures in many low-level controllers is the absence of an episodic memory system. To overcome this, we equipped MrSteve with Place Event Memory (PEM), a form of episodic memory that captures and organizes what, where, and when information from episodes. This allows for efficient recall and navigation in long-horizon tasks, directly addressing the limitations of existing low-level controllers like Steve-1, 1, which rely heavily on short-term memory. Additionally, we proposed an Exploration Strategy and a Memory-Augment Task Solving Framework, enabling agents to effectively switch between explo- ration and task-solving based on recalled events. Our results demonstrate significant improvements in both task-solving and exploration efficiency compared existing methods. We believe that MrSteve opens new avenues for improving low-level controllers in hierarchical planning and are releasing our code facilitate further research in this field.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We acknowledge potential societal concerns related to our work. While our agent, in its current form, designed for use in virtual environments such Minecraft, the techniques and advancements made here could be extended to broader autonomous systems. There is possibility that, if adapted for real- world applications. such systems might be misused for unethical purposes, including unauthorized surveillance or actions that infringe on individual privacy. We encourage responsible usage and further research into safeguards to prevent such outcomes.
REPRODUCIBILITY STATEMENT
To facilitate the reproducibility of our work, we have provided the pseudo codes, model architecture, and hyperparameters in Appendix D and We will also release the source code for our models and experiments.
ACKNOWLEDGEMENT
This work was supported by GRDC (Global Research Development Center) Cooperative Hub Program RSS-2024-00436163 Brain Pool Plus Program (No. 202HH133A2001106655 and Basic Research Lab (No. 2224--00444222 through the National Research Foundation (NRF) funded by the Ministry of Science and ICT (MSIT). The authors would like thank the members of Machine Learning and Mind Lab (MLML) for helpful comments.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23756 2022.
Josh Albrecht, Abraham J. Fetterman, Bryden Fogelman, Ellic Kitanidis, Bartosz Wróblewski, Nicole Sco, Michael Rosenthal, Maksis Knutins, Zachary Polizzi, James B. Simon, and Kanjun Qiu. Avalon: A Benchmark for RL Generalization Using Procedurally Generated Worlds. In NeurlPS Datasets and Benchmarky Track, 2022.
Bowen Baker, Ilge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances Neural Information Processing Systems, pp. 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:200s 14 14165, 2020.
Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13734-13744 2023a.
Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Groot: Learning to follow instructions by watching gameplay videos. arXiv preprint 2rXiv:23 2023b.
Matthew Chang, Theophile Gervet, Mukul Khanna, Sriram Yenamandra, Dhruv Shah, So Yeon Min, Kavit Shah, Chris Paxton, Saurabh Gupta, Dhruv Batra, Roozbeh Mottaghi, Jitendra Malik, and Devendra Singh Chaplot. Goat: Go any thing, 2023. URL. https: //tatt//avvva/ org/obrgga........
231 64430
Junmo Cho, Jaesik Yoon, and Sungjin Ahn. Spatially-aware transformers for embodied agents. In The Twelfth International Conference on Learning Representations, 2024. URL https
7/ //ppnrevi ew net.

---

Published as a conference paper at ICLR 2025
Zihang Dai, Zhilin Yang. Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer- XL: Attentive language models beyond a fixed-length context. 2019.
Or Dinari and Oren Freifeld. Revisiting DP-means: Fast scalable algorithms via parallelism and delayed cluster creation. The 38th Conference on Uncertainty in Artificial Intelligence, 2022.
Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang. De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with intemet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022a. URLhttps: http:s///pppeeeiievwe www. www//eettouaaa.
_d=rc8o_3881 T
Linxi Fan, Guanzhi Wang, Yunfan Jiang. Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In l'irtty-sith Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022b. httttss://pppereeeee
net. forum?
William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset minecraft demonstrations. arXiv preprint arXiv:1907. 2019.
Danijar Hafner. Benchmarking the spectrum of agent capabilities. 2022.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint wrXiv:2301.04104 2023.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp, 2019. URL MRkLhhtss://axxttt org. 19t.a./999....0 00751
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021a. URL https:
//arxiv.org/ httpg//ab//2.....
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021b.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint ar arXiv:2201.07207, 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models, 2022b. URL 05608
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In Ijcai, volume 16, pp. 4246-4247 2016.
Tomoyuki Kagaya, Thong Jing Yuan, Yuxuan Lou, Jayashree Karlekar, Sugiri Pranata, Akira Kinose, Koki Oguri, Felix Wick, and Yang You. Rap: Retrieval-augmmented planning with contextual memory for multimodal Ilm agents, 2024.
Andrew Kyle Lampinen, Stephanic C Y Chan, Andrea Banino, and Felix Hill. Towards mental time travel: hierarchical memory for reinforcement learning agents. 2021.
Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus- I: Hybrid multimodal memory empowered agents excel long-horizon tasks, 2024. URL
https: /ttpss:////iiii wttp:/wwwww/p/o.//..ooeo 03615
12

---

Published as a conference paper at ICLR 2025
Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila Mcllraith. Steve-1: A generative model for settto-beehavor in minecraft. Advances in Neural Information Processing Systems, 36, 2024.
Zichuan Lin, Junyou Li, Jianing Shi, Deheng Ye, Qiang Fu, and Wei Yang. Juewu-me: Playing minecraft with sample-efficient hierarchical reinforcement learning. arXiv preprint arXiv:2112.04907, 2021.
Shunyu Liu, Yaoru Li, Kongcheng Zhang, Zhenyu Cui, Wenkai Fang, Yuxuan Zheng, Tongya Zheng, and Mingli Song. Odyssey: Empowering agents with open-world skills. arXiv preprint arXiv:2407. 15325 2024.
Hangyu Mao, Chao Wang. Xiaotian Hao, Yihuan Mao, Yiming Lu, Chengjie Wu, Jianye Hao, Dong Li, and Pingzhong Tang. Seihai: sample-efficient hierarchical ai for the minerl competition. In Distributed Artificial Intelligence: Third International Conference, DAI 2021, Shanghai, China, December 17-18, 2021, Proceedings 3, pp. 38 Springer, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep rein- doi: 10.3////aa URL
forcement learning. Nature, 518(7540):529 533, 2015.
https://doi.or// www.a/o..ogg...o.. wttp://...138//aattaato http:://w.....a.......////aaaaa.
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. arXiv preprint arX00 2023. URL https: //tp:i/aaiivv.v/aa org/ abs https//21...0.00...
OpenAl. Gpt-4 technical report, 2024.
Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, and Jing Shao. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp.
16307-16316, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https
//axxiv.org/abs/22100 00020
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint trXiv:1707.06:: 2017.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
Zhiyuan Sun, Haochen Shi, Marc-Alexandre Côté, Glen Berseth, Xingdi Yuan, and Bang Liu. Enhancing agent learning through world dynamics modeling, arXiv preprint MrXiv:2407.17695 2024.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning, 2017. /tthttss::// httttpps//axxiv.ooo//bb////////e 04717
Qwen Team. Qwen2.5: party of foundation models, September 2024. URL https: 1m github. tt/bllg///////oooo qww..22.5/ww//////////
13

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ana Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023. URL wtt.thttt:///attit org/
13971
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Ma- org/papers/v9
chine Learning Research, 9(86):5579-266 2008.
Vaanderaatten0Ba.huu html
Konstantinos Voudouris, Ibrahim Alhas, Wout Schellaert, Matthew Crosby, Joel Holmes, John Burden, Niharika Chaubey, Niall Donnelly, Matishalin Patel, Marta Halina, José Hernández- Orallo, and Lucy G. Cheke. Animal-ai 3: What's new why you should care, 2023. URL
https://arxix:
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv: Arxiv-2305. 16291, 2023a.
Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma and Yitao Liang. Jarvis-I: :: Open- world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv: 2311 110099997 2023b.
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01 2023c.
Brian Yamauchi. Frontier-band based exploration using multiple robots. In Proceedings of the Second International Conference on Autonomous Agents, AGENTS '98, pp. 47-53, New York, NY, USA, 1998. Association for Computing Machinery. ISBN 19831, doi: 01144/280765.280777777 URL https::/doi .or.oi..rrroooo 10. 2145/2807655 280773.
Haoqi Yuan, Chi Zhang. Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Plan4MC: Skill reinforcement learning and planning for open-world Minecraft tasks. arXiv preprint Arv:::2303. 16563, 2023.
Haoqi Yuan, Zhancun Mu, Feiyang Xie, and Zongging Lu. Pre-training goal-based models for sample-efficient reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp://///peeeeeeeeeeemme wtwpeeeewwwweeeeeeeeeeeeeeeeeeee
net wwt.://orommmmo//...... Imm.ee1122mLLL9
Jesse Zhang. Jiahui Zhang, Karl Pertsch, Ziyi Liu, Xiang Ren, Minsuk Chang, Shao-Hua Sun, and Joseph Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. In 7th Annual Conference on Robot Learning, 2023. URL https:: / net/forum? i id= aaaaa00FRaaaaam
Bohan Zhou, Ke Li, Jiechuan Jiang, and Zongqing Lu. Learning from visual observation via offline pretrained state-to-go transformer. Advances in Neural Information Processing Systems, 36, 2024a.
Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao. Minedreamer: Learning to follow instructions via chain-of-imaginatioo for simulated-word control. arXiv preprint arXiv:2403. /2037, 2024b.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with bext- knowledge and memory. arXiv preprint 1.Xxi:23055.7144 2023.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS OF MEMORY SYSTEM IN LLM-AUGMENTED AGENTS
Memory systems in agents primarily aim retrieve robust and accurate high-level plans for long- horizon tasks (Wang et al. 2023a b Zhu al. 2023 Qin al.) 2024 Li ct al. 2024). Existing works use an abstracted memory that stores the succeeded task with its plans, and often with history of observations for reliable retrieval, which is helpful when the similar plan in other tasks already exists in memory. However, these types of memory systems are not well-suited for low-level controllers for the following reasons:
Issues with Managing Memory Recent memory systems in Minecraft, when saving successful plans or skills, the history of observations for solving the task is all stored in FIFO manner (Wang) jet al. 2023b) or only task-relevant frames are stored in the plan (Li et al., 2024). However, computation complexity for retrieving the experience frames that are relevant to new task is computationally expensive or even impossible (because the latter may not store the experience frames despite the agent observing it).
Lack of Mechanism for Retrieving Experience frames Current memory systems store the text instructions of succeeded tasks as keys and their plans to complete those tasks as values. The retrieval process begins by matching a task query to the task keys in memory and then filtering further using the current scene's 's similarity to the stored frames before retrieving the final plan. These memory systems are targeted to retrieve the succeeded plan, but they lack mechanism for utilizing the experience frames in the memory, which could be crucial for future tasks.
Consider the following example: Suppose the agent is tasked with collecting wood. While searching for a tree, let's assume the agent came across cow in the forest. Once the wood is collected, the memory will store the successful plan and its corresponding observations. However, if the agent later tasked with finding the cow, there would be no memory key related to the cow, making impossible to retrieve the relevant frames. While some heuristics to calculate the similarity of the task embedding for "find cow" and the visual representations memory using MineCLIP is possible, it is computationally expensive since the similarity should be calculated for all stored frames.
Thus, we need a new type of memory system for the low-level controller to efficiently store novel events (such encountering the cow) as they explore the environment, even when such events are not directly relevant to the current task. Also, the memory should hierarchically organize these novel events that they be efficiently retrieved later. In this paper, we propose memory system called Place Event Memory (PEM), which organizes experiences by both location and event. PEM allows the agent to store diverse novel events across various locations, making future retrieval more efficient. We argue that PEM, when combined with current memory systems that store successful plans, will enable more effective retrieval of task-relevant information.
B COMPUTATION OVERHEAD
Our study was performed on an Intel server equipped with 8 NVIDIA RTX 4090 GPUs and 512GB of memory. The inference time for tasks under 20K steps for rumning a single episode was approximately 30 minutes on single GPU. For long- hong-horizon tasks that take 500K steps, approximately 12 hours were required for running a single episode on single GPU. The VPT-Nav training took roughly 23 hours on a single GPU.
C ENVIRONMENTS DETAILS
C.I ENVIRONMENTS SETTING
All tasks are implemented using MineDojo (Fan et al., 2022b). We utilize MineDojo's success checker, where the success each task is determined based on changes in the agent's 's inventory. Hence, the agent succeeds in the task if the corresponding target item appears in the inventory. If the agent exceeds the time limit of each task or dies before completing assigned tasks, indicating the agent failed the task.
For all tasks, we assume that the agent has access to both first-person view pixels and its positional data. The raw pixel observation provided in the shape (160 x 256 3). The agent's position pt
15

---

Published as a conference paper at ICLR 2025
Map
Mар 2
Map 3
Figure 8: Topdown View of Three ABA-Sparse Task Maps. The first map used in the ABA-Sparse tasks, including the water bucket task. Trees distributed the left side of the map, and water exists only in the upper right corner. The second map was used in tasks, not including located on the left side of the map, and on the right side, there are cows and sheep, with mountain separating them. The last map was used in tasks, including the sand task. Its overall layout is identical to the second map, except for additional water pond at the top. The map's 's edges surrounded high walls, making it impossible to access anything other than the resources visible the top-down view.
water bucket
sand task. Trees are
is represented as vector of shape (5, ). where the first three components correspond to the (ar, y, =) coordinates, and the last two components represent the pitch and yaw angles of the agent's camera. We note that privileged observations, such LiDAR or voxel data, are provided. The agent operates in a keyboard and mouse action space following VPT (Baker et al. 2022). This action space consists of button input states paired with mouse movements.
Crafting items, which requires long-horizon planning, is not considered in our method. To eliminate the need for crafting, the appropriate item necessary for solving task is provided to the agent the beginning each new task. For instance, if the task is to "obtain water bucket W the agent starts with empty bucket in its main hand. Additionally, we apply the following rules:
/ di ff cully peaceful: This rule prevents the occurrence of hostile mobs, such as zombies and spiders, and death by starvation.
/gamerule doWeatherCyele false: This rule keeps the weather clear reduce the noise from heavy rain.
C.2 TASK DETAILS
In this section, we describe the details for each task in Experiment section (Section 4.. The basic environment settings follow those outlined in Appendix C.I unless specified otherwise.
C.2.1 EVALUATION PROTOCOLS
Except for the Long-Horizen tasks in Section 4.4 we ran 100 episodes for each agent using different random seeds to evaluate performance. For the Long- Horizon tasks, we reported the average success rate with the standard error over episode runs for each agent.
C.2.2 EXPLORATION NAVIGATION TASK DETAILS
For this task, we used Map in Figure with 100 100 size, surrounded by high walls. The map includes complex terrains such mountains and water pond, which requires robust low-level navigation policy for successful exploration. For each episode, the agent spawns the center the map and explores for 6,000 steps.
16

---

Published as a conference paper at ICLR 2025
C.2.3 ABA-Sparss TASKS DETAILS
In these tasks, the agent is asked to complete three sequentially given tasks, where the first and third tasks are identical. The target item for the first task, denoted as A, is one of four sparsly distributed items: water
beef wool or milk T In the second task, denoted as B, the taget item is one of five items: log dirt seeds 42, leaves 8 or sand T The agent has unlimited memory capacity and is allowed maximum of 12,000 steps to complete three tasks. The task only changes to the next one upon successful completion of the current task.
If the first task, A, is to obtain water bucket the agents spawn at the center of Map 1, shown in Figure On this map, most of the surface is covered with dirt, while grass, which provides seeds, is widely distributed. A water pond is located the upper right corner the map, but it only becomes visible when agents are nearby.
the first task, A, to collect beef or wool the agents spawn at the center of Map 2 in Figure 8 Similarly, if the second task, B is to collect sand T the agents start at the center of Map 3 in Figure In both maps, trees are scattered on the left side of the map, while sheep ? and cows te found on the right side. A sand mountain runs through the middle, separating the trees from the sheep and cows. Dirt is only present on the far left and right sides of the map.
C.2.4 MEMORY TASKS
In the three Memory Tasks, the agent explores for 3,000 steps before being asked to complete an image goal navigation. Unlike the ABA-Sparse Tasks, the agent has limited memory capacity of 2,000 frames. Consequently, an agent utilizing the FIFO memory forgets the memory from the first 1,000 frames after the exploration phase.
Find Water Task The agent is initially spawned near a water pond and remains in its vicinity for 500 steps. During this period, one observation is selected as goal image for a subsequent navigation task. After the initial 500 steps, the agent moves to a random location for the remainder of the exploration phase. After 3K steps, the agent begins to navigate back the water pond observed the start of the episode.
Find Zombies' Death Spot Task At the beginning, the agent burning zombies for the first 1K steps. Approximately 500 steps after the start of the episode, the zombies disappear, resulting in the agent observing two distinct scenes in the same location. A goal image for navigation is selected from the observation where the zombies are burning. After 1K steps, the agent starts to travel for the rest of the exploration phase. Once the exploration phase finishes, the agent returns to the place where the zombies were burning.
Find Vistteisiied House Task In this task, there are two distinet houses that look similar to one another. The agent starts near one of the houses, where it stays for 100 steps before moving to the other house, where it remains for 2K steps. For the remainder of the exploration phase, the agent travels to random location. After the exploration phase, the agent is asked to go to the first house it has visited.
C.2.5 LONG-HORIZON TASKS
Long-Instruction Task In this task, the agent is required to complete series of tasks sequentially on 200 x 200 block-sized map in Figure 9 The order of tasks within the sequence is randomized, with each task being one of six possible types: water bucket beef O wool wood dirt and seeds 4:. We evaluated the agent's 's performance by measuring the number of successfully completed tasks over 500K steps. If the agent fails to complete a given task within 20,000 steps, that task is canceled, and new is assigned.
Long-Naver Navigaaang Task This task is basically image goal navigation in a 200 x 200 block sized map in Figure 9 Before the task phase, the agent undergoes 16K-step exploration phase where it observes six landmarks: 1) zombie burning = 2) water + 3) sugarcane explosion # 4) spider spawn M. 5) tree and 6) house, spending 2K steps at each location. In the task phase, the agent is given a random start image of one of these landmarks and must navigate to it. The dynamic nature of some landmarks, such as the burning zombie, exploding sugarcane, and spider spawn makes important to store novel events from the exploration phase.
17

---

Published as a conference paper at ICLR 2025
Long-Instruction Task
Long-NNaigation Task
Figure Topdown View of Two Long-Horizon Task Maps. Both maps are size 200 200 blocks. (Left) This map is used for the Long-Instrunn Instruction task. Trees are located in the lower left and middle bottom of the map. while sheep inhabit the upper left area, and cows exist in the right bottom. A water pond can be found in the upper right area of the map. (Right) This map is utilized for the Long-Navee -Navigatttn task. Agents traverse between six scenes the map. Three of them are dynamic scenes: burning zombies, popping sugarcanes, and spawning spiders, which are positioned at the first, third, and fourth places on the map, respectively. The other three are static scenes: water pond, trees, and house, located at the second, fifth, and sixth places on the map, respectively.
18

---

Published as a conference paper at ICLR 2025
D MRSTEVE ALGORITHM
Algorithm 2 MrSteve Algorithm
Require: ym from Environment env, Memory Menn and new task on at time step ",
1: mode +- null
2: made <- 0
3: Jr f null
4: reached & false
5: t şinit
6: Xe + X Mj t M,m
7: loop
8:
Write( M,, X₂)
9: T mode I then
10:
mode +- null
11:
Tiode +
12:
9t null
13:
reached + false
end if
15:
Mode Selector
if mode is null then candidates + Read Tx)
17:
if candidates * 0 then
19:
X PickOne(candidates) Jt Position (X,)
21:
mode EXECUTE
22:
else
23:
mode EXPLORE
24:
end
25:
end if
26:
h4 X2-128:1
27:
Explore Mode
28:
if mode is EXPLORE LORE then
St TH-Nav (gt Cg M,)
30:
O1 TL-Naw (at (t,
end
32:
Execute Mode
33:
if mode is EXECUTE then
34:
if the agent reaches then
35:
reached + true end if
36:
37:
reached then a: Tis (a:lhe Tu)
38:
39:
else
40:
a: Nar
41:
end M++1 +
43:
end if
if the task T, succeeded timeout then
45:
break
46:
end
47:
Tmode f- Tmode +1
48:
X + env.step(a)); Mc-1 + M
49:
t ++++1
50:
end loop
We provide more details in Algorithm This algorithm is executed when a new task given to the MrSteve and finished if it completes the task or exceeds time limit (Line 44-46).
19

---

Published as a conference paper at ICLR 2025
Agent State Variables When new task Tn is given, the agent state variables initialized (Line 1-4). mode indicates which module executed and Tmode denotes elapsed times for the current mode execution. 9t is a target location for the navigation, and reached indicates whether the agent has reached the target location in the Execute Mode.
The Agent State Variables reset if the agent remains in the current mode for L steps (Lines 9-13). the Mode Selector retrieves a wrong experience frame, where no task-relevant resource is present the corresponding location, the agent can have trouble and fail to complete the given task for long time. In addition, the agent might encounter previously unobserved task-relevant resources while navigating the location of the previously retrieved memory. In both scenarios, changing the navigation target can be helpful. Consequently. the agent queries the memory to search for new candidates the agent executes the current mode for I. 600 steps, equivalent 30 seconds in-game time.
Memory Write Frequency Line 8, MrSteve writes the current observation the memory, regardless of the current mode.
Mode Selector The Mode Selector decides between two modes according to the existence task- relevant resources in the memory. First, if no mode has been selected, MrSteve queries the memory (Line 16-17). If task-relevant resource exists, MrSteve picks one of them and sets the navigation target 9: the location of the picked one (Line 18-21). Otherwise, the Explore Mode is selected (Line 23).
Explore Mode In the Explore Mode, MrSteve explores the least-visited locations using WH-Nav and TL-Nav (Line 28-31). We provide more details in Appendix F2
Execute Mode In the Execute Mode, MrSteve navigates to the selected target location Ir first. Once it reached, Steve-1 is executed follow the task instruction Tp (Line 33-43).
20

---

Published as a conference paper at ICLR 2025
E PLACE EVENT MEMORY, PLACE MEMORY, EVENT MEMORY DETAILS
We provide the Algorithms on Write Read operations of Place Event Memory and other memory variants, which are Place Memory and Event Memory. We also provide the specifications of each memory in the following section
E.1 PLACE EVENT MEMORY WRITE & READ OPERATIONS
Algorithm Place Event Memory Write Operation at time step /
Require:
Assume memory at time step f. as M. {M&) where Mj is k-tth place cluster with center position f, and center embedding 6M, and each place cluster Mg has event clusters {E where E is i-th event cluster with center embedding E Each place cluster has dummy deque Q2 that stores recent R frames in that cluster, and update frequency timer rk- Additional variables are Memory Capacity N, MineCLIP image encoder CLIP,, and experience x,= {,,,4, (,,44 I) time step f. MineCLIP threshold
1: # Memorr Add
2: Get experience frame 2; {er, 4. where e, CL)
3: if & E PLACE_CLUSS CLUSTER then
4: Find place cluster M, where PLACE(() Lad,
5: Add to dummy deque Qj
6: Update frequency timer Ta = rk +1
7:
rr R then
8:
{E = DP-Mcans(Q))
9:
[B,,e MERGE CLUNGEECLUUTTRS((EEC e for I..... do add cluster-True for p l dj do Ep then
11:
13:
e/ e
Merge E., EJ add bid_cluster=Fale; break
15:
end if end for add Tluster=True then
17:
18:
19:
Create new event cluster E2 #1 = E,, with center embedding BB Cu
Add created cluster F2 M,
21:
end end for
22:
23:
Fh = Q
24:
end if
25:
else
Create new place cluster MK+1 with center position PMx+1 PLACE(&:). and center embedding MM
Create new dummy deque QK+1 = {xi}
28: Add created cluster MK+1 = (QK+1) to M,
29: end if
30: # Memory Remove
31: if len (M) N then
32: Find event cluster E where i, arg max E.)
33:
Remove the oldest frame in 1it
34:
end if'
We further elaborate on how # Memory Add in Algorithm 3 operates. First, we get the experience frame Zs, and check &, belongs to one of place clusters in Mr (Line 3). Here, PLACE CLUSTER/ fixed-size square area for each
indicates the whole 2D space that memory M, covers. Since place cluster, can be seen as covered area of set of squares. If r, not in current place clusters, new place cluster is created with dummy deque (Line 26-28). Dummy deque here short memory
21

---

Published as a conference paper at ICLR 2025
for each place cluster used for clustering the events. If 2, belongs some place cluster (Line PLACE() maps l to its place), it is added to dummy deque in the place cluster, and increase update frequency timer in that place cluster by (Line 4-6). When update frequency timer equals to R. DP-Means algorithm (Dinari & Freifeld 2022) is applied to experience frames in dummy deque for event clustering. After applying DP-Means algorithm, we get set of events and its center embeddings (Line 8). However, we found that DP-Means tends to make different clusters even when the agent observes the same scenes. Thus, we applied MERGE CLUSTERS() DP-Means output clusters to merge clusters that have high alignments in center embeddings (Line 9). After this, for each newly created event cluster from DP-Means (Line 10), if some existing event cluster is aligned (Line 13), two clusters are merged (Line 14). If newly created cluster does not belong to any existing event clusters, then add to the place cluster as a new event cluster (Line 19-20).
Event Cluster Details We use the DP-Means algorithm for clustering the events. DP-Means algorithm Bayesian non-pan parameerteti extension of the K-means algorithm based on small variance asymptotic approximation of the Dirichlet Process Mixture Model. It doesn't 't require prior knowledge on the number of clusters K. To run this algorithm, we first set the initial number clusters K' and cluster the data with K-Means++ initialization (K' can be 1), then DP-Means algorithm automatically re-adjust the number of clusters based on the data points and cluster penalty parameter Thus, DP-Means algorithm behaves similarly K-means with the exception that a new cluster is formed whenever a data point is farther than 5 away from every existing cluster centroid.
Using clustering algorithm in Minecraft was previously done in Yuan al. (2024), where K-Means algorithm is used to cluster 100 goal states in massive Minecraft dataset. However, they applied the clustering on the reduced dimension of MineCLIP representation with t-SNE (van der Maaten & Hinton 2008). In this work, we directly apply DP-Means algorithm in MineCLIP representation 1.
space setting iitial number of clusters K' 5, and
Place Cluster Details We provide a detailed explanation of how place clusters are formed. Each place cluster stores the agent's experience frames based on its 2D ground position (zz y) and head direction angle yaw. The size of a place cluster and its yaw range are defined by parameters C' (in Minecraft place cluster is
block) and W (0); 180%), respectively, which distinguish different clusters. centered at specific position (æ, y) with a center yaw An experience frame is assigned to place cluster the agent's 's position falls within the range (æ - CC22, x + C/2) for x and (y - C/22 y C/2) for y, and the yaw angle satisfies (w W/2, W/2). We use relative position and yaw for center of the place cluster, implying that the center of the initial place cluster has position (0, 0), and yaw 0.
Algorithm 4 Place Event Memory Read Operation at time step
Require: Assume memory time step f as M, { where Mr is k-th place cluster with center position EM,, and center embedding 6M2- and each place cluster Mr has event clusters {E Jim1 where Et is i-th event cluster with center embedding e Additional variables are MineCLIP image encoder CLIP,,. and text encoder CLIP,, task instruction Tn at time step t, and
task threshold h.
E. ) for all i, 2: Sort 85k (T,)) for all event clusters and select top-K event clusters
1: Calculate 8ix (Tx) (T,) CLIP, (e
3: Gather all experience frames x, [e,,4,, t} from the top-K event clusters
4: cand = Q
5: for each experience frame x, in the selected top-K event clusters do
6: (T,(Tx) = (T,) (e))
7: if s2. (Tn)
then
Add 24 to cand 9: end if
8:
10: end for
11:
return cand
Exploiting hierarchical structure of Place Event Memory For memory read operation, we can exploit hierarchical structure of place event memory for better computation efficiency. Suppose we have N1 place clusters and exactly N₂ event clusters for each place cluster. Then, read operation in Algorithm has computational complexity of N2). However, if we first attend only to place clusters (calculate su from center embeddings em, from place clusters), then attend event
22

---

Published as a conference paper at ICLR 2025
clusters from extracted top-k place clusters, the computational complexity becomes O + kN2). However, center embedding ead, may not be sufficient summarization of the place cluster, since center embedding not capture all different events occurred in that place. Thus, we use the read operation as in A Aggoithhm
23

---

Published as a conference paper at ICLR 2025
E.2
PLACE MEMORY WRITE & READ OPERATIONS
Algorithm 5 Place Memory Write Operation time step
Require: Assume memory at time step / as M2 {M&Je= where Mr is k-th place cluster with center position EM,- and center embedding eM Each place cluster Mj. is FIFO Memory. Additional variables are Memory Capacity N, MineCLIP image encoder CLIP,, and experience
X₁ {ir, l. t time step t.
1: # Memory Add
2: Get experience frame 24 {ee, 4. t} where e, (L)
3: if & E PLACE_ CEEEEEERUSSRRRRE then
4: Find place cluster M, where PLACE(e)) = eM;
5: Add to Mj
6: else
7:
Create new place cluster MK+1 with center position (MK+1 PLACE(&)). and center embedding MM Et
8: Add created cluster MK+1 {ara } to Mr
9: end if
# Memory Remove
11: if (en( > N then
Find place cluster M₁ where arg max len(Me
13: Remove the oldest frame in M2
end if
Algorithm 6 Place Memory Read Operation at time step t
Require: Assume memory time step f Mr {M& ) where Mr is k-th place cluster with center position fM and center embedding EM, Each place cluster Mr is a FIFO Memory. Ad- ditional variables are MineCLIP image encoder CLIP,, v. and text encoder CLIP,, task instruction at time step f, and task threshold h.
Calculate 8y. (T, = (T.) CLIP,(EM,) for k:
2: Sort (x((T)) for all place clusters and select top-K place clusters
3: Gather all experience frames Ie- = {e,, 4, t from the top-K place clusters
4: cand [}
5: for each experience frame x in the selected top-K place clusters do $z, (T,) (T,): (e.P,(e.)
6:
7: if (2.22(TT) (Tx) ) > then
Add de to cand 9: end
8:
end for
11: return cand
24

---

Published as a conference paper at ICLR 2025
E.3 EVENT MEMORY WRITE & READ OPERATIONS
Algorithm Event Memory Write Operation at time step /
Require:
Assume memory at time step as M, {Fx) where Eg is k-th event cluster with center embedding eE,, and each event cluster a FIFO Memory. Memory has dummy deque Q that stores recent R. frames, and update frequency timer Additional variables Memory Capacity N, MineCLIP image encoder CLIP., and experience X, = {iz, 4, tJ at time step t.
MineCLIP threshold
1: Memory Add
2: Get experience frame 2, {e,, 4, where e, (L)
3: Add x2 to dummy deque Q
4: Update frequency timer r
5: if r R. then
6:
{E,, e4 DP-Means(Q,)
7:
{E, = MERGE CLERSERSS (E,, e( for 1....,, do add clussttrreree for p 1,.... do if' e EE, > c then Merge E. Ex add cluster=False; break end if end for add aaadddccusse cluster-=True then
8:
9:
11:
13:
15:
17:
Create new event cluster Ex-1 = E,, with center embedding 1 Cu Add created cluster E to M,
18:
end if 20: end for
19:
21: Tk 0
22: end if
23: #
Memory Remove
24: if (Mn) > N then
Find event cluster Eş where & arg max len Ex)
26: Remove the oldest frame in E₁
end if
Algorithm 8 Event Memory Read Operation at time step
Require: Assume memory at time step as M, = {Ex } where Eg k-th event cluster with center embedding e and each event cluster Eş is FIFO Memory. Additional variables are MineCLIP image encoder CLIP,,. and text encoder CLIP,, task instruction Tn at time step t, and task threshold h.
1: Calculate St((T)) = CLIP, CLIP, for all k:
2: Sort (Tx))) for all event clusters and select top-K event clusters
3: Gather all experience frames I2 {ex, 4, t from the top-K event clusters
4: cand = f
5: for each experience frame x, in the selected top-K K event clusters do
6: (T.(TT) = CLIP:(T.)) (ex),(ee)
7: if 82, (T,,)TT)) > h then
Add Je to cand 9: end if
8:
10: end for
11:
return cand
Discussions MineCLIP threshold As in Event Memory's Write Operation in Algorithm 7
clusters generated by DP-Meams algorithm are either merged with an existing event cluster, or added event cluster. This is determined by MineCLIP threshold e, which is the criterion for separating
25

---

Published as a conference paper at ICLR 2025
event clusters. We note that using proper value for threshold is important for Event Memory to work reasonably across the tasks. is too small, Event Memory will cluster experience frames with only few clusters, which may not be visually distinctive. If e is too large, in extreme case, Event Memory will make event cluster for each experience frame. When memory capacity is exceeded, will randomly remove the oldest frame (since all event clusters have the same size), which behaves as a FIFO Memory.
Drawback in Event Memory Event Memory removes the frame in the largest event cluster, when memory capacity is exceeded. This indicates that the memory can retain visually distinct events. However, since it does not consider position information in clustering, similar visual frames in different places can be clustered into the same event cluster. This can be fatal since task-relevant frames in different places can be removed from the memory, which can be crucial in upcoming tasks. This drawback is shown in Find First- Visited House task in Section and Long-Inte long-Instruction task in where MrSteve-FM showed suboptimal performances.
Section 4.4
E.4 SXYEE-PARAMETE FOR MEMORY
We provide the specifications for each memory type used in the experiments. The task threshold is set to 22. 74 by default, but can be adjusted to enhance retrieval accuracy.
Table 2: Specifications for each memory type.
Memory Type
Parameter
Value
Place Cluster Size C
Place Cluster Yaw Y
Place Event Memory
Update Frequency P
100
top-K K
30
MineCLIP Threshold
73.5
Task Threshold h
22.74
Update Frequency nk
100
top-K
Event Memory
MineCLIP Threshold c
73.5
Task Threshold h
22.74
Place Cluster Size C
Place Cluster Yaw Y
60
Place Memory
top-K
30
Task Threshold h
22.74
26

---

Published as a conference paper at ICLR 2025
(a) Explore Mode
(b) Execute Mode
Mace Event Memory
Count-based HightLevel Policy
Place Event Merory
Armked
9n.t
ag
St
Expare langet
Goal Arrival Check
Finetuned VPT
Navigation
VPT
Navigation
a1
4
Ot
O
4.
Figure 10: The model architectures of Explore Mode and Execute Mode Solver Module.
SOLVER MODULE DETAILS
MODE SELECTOR
In Section 3.2
described the Mode Selector when combined with Place Event Memory. When the memory type changes such as Place Memory or Event Memory, different read operation should be employed. We provide the Memory Read operation of Place Memory, and Event Memory in Appendix E.2. and E.3 respectively. In case of FIFO Memory, for read operation, task alignment scores on whole frames in the memory is required.
F.2 EXPLORE MODE
In explore mode, hierarchical structure is employed. For high-level goal selector, we take the similar approach introduced in Chang et al. (2023), where the target problem is indoor navigation with robots. Their global policy exploits semantic map output exploration goal. For the goal selection, uses frontier-based exploration (Yamauchi 1998), which selects the closest unexplored region as the goal. In Figure 10 a). the overview of hierarchical episodic exploration is illustrated. From Place Event Memory, we contruct a visitation map by marking the agent's 's visited locations. addition to marking agent's visited locations, region towards agent's head direction (yellow sector shows agent's current FoV). From the visitation map, the next goal is chosen, and the goal, and current observation, position given to VPT-Nav for generating low-level action. In experiments, for the tasks with map size 100 x 100, we used visitation map size L 120, and grid cell size G 15. For the tasks with map size 200 200, used visitation map size L 240, and grid cell size G 30.
also marked the agent's FoV in the visitation map with sector
F.3 EXECUTE MODE
The model architecture Execute mode illustrated in Figure 10 b). When the Mode Selector selects the Execute Mode, the experience frame of task-relevant resource is retrieved from the memory. MrSteve then navigates to the goal location the experience frame, and then adjust camera orientation using yaw and pitch from the frame ensure it faces the observed of the retrieved experience frame again. Once the camera adjustment is complete, Steve-1 is executed to follow the task instruction T..-
F.4 MINECLIP STEVE-1 PROMPTS FOR TASKS
The prompts used in our experiments are listed in Table MrSteve sets to one of two different prompts depending on the agent's status. When the agent is the mode selection phase, the MineCLIP prompts are used the query for the memory to determine whether the task-relevant resource exists in the memory. contrast, the Steve-1 prompts are utilized during the execution mode.
Relying solely on Steve-1 prompts can lead to difficulties in calculating alignment score between the prompt and memory records. Since MineCLIP (Fan et al. (022b) computes the alignment between videos and textual descriptions, the alignment score is higher when the textual description well reflects the agent's action shown in the video. If video contains scenes relevant to the current
27

---

Published as a conference paper at ICLR 2025
Table 3: Prompts used in MrSteve for each task.
MineCLIP
Steve-1
log
near tree
cut a tree
beef
near cow
kill cow
dirt
near dirt
get dirt
sand
near sand
get sand
4 seed
grass
collect seeds
wool
near sheep
kill sheep break leaves
leaves
near tree
milk bucket
near cow
get milk bucket
water bucket
near water
get water bucket
task but lacks behavior indicative of task completion, the alignment score will be low. For instance, consider a scenario where the "obtain water bucket O taas is given for the first time, and the agent has previously encountered water. Although the agent is Near water in that scene, it has not yet performed the action of Obtaining water bucket. As result, the prompt "obtain water bucket" would not accurately describe that memory, and that scene could be disregarded during the mode selection process. In our experiments, we manually defined two types of prompts for each task. However, this process can be automated using LLMs, which prompts an LLM to extract the more suitable language instructions to complete a task.
G EXPLORATION EXPERIMENTS DETAILS
STEVE-1 prompted with "Go Explore"
High-Level: Plan4MC Low-Level: Plan4MC
High-Level: Count-based
Low-Level: VPT-Navigator
High-Level: Plan4MC Low-Level: VPT-Navigator
High-Level: Count-based Low-Level: Plan4MC
Figure 11: Agent's trajectories length 6K steps on 100 100 sized map with different exploration methods Left figure the agent's 's trajectory from our exploration method.
We provide further details evaluation metries and analysis of the exploration results. For evaluation metrics, we measured Map Coverage and Rivisit Count. Map Coverage calculated by dividing the map into 11 x 11 grid cells and measuring the percentage of cells covered by the agent's trajectory.
28

---

Published as a conference paper at ICLR 2025
Revisit Count measures the average number of times the agent visits the same grid cell only for the cells that have visitation counts larger than 300. We measured Revisit Count only for highly visited grid cell since the agent needs some amount of time to escape the grid cell.
Since Steve-I (ifshitz et al., 2024) has no memory module, it tends to visit the same place multiple times. We also observed that Steve-1 1 exhibits behavior of moving straight ahead when task instruction "Go Explore" given, often colliding with walls and then following along the wall instead of avoiding it, which harms the Map Coverage and Revisit Count.
PIan4MC Yuan et al. 2023) employs a hierarchical exploration policy. The high-level policy, based LSTM model, takes the agent's (Ir, y) coordinates along its trajectory and outputs the next direction to move (north, south, cast, or west). The environment is discretized into an 11 x 11 grid, and the policy is trained using PPO (Schulman al. 2017) to maximize the number of unique grid cells visited. The low-level policy is trained using DQN (Mnih et al. 2015) and follows the Mine structure from the MineDojo framework. It receives goal position and current observation in pixel and outputs actions in the MineDojo action space. However, despite the high-level RNN policy, it struggled to keep track of visited locations as the trajectory grew longer, leading to a high Revisit Count. Furthermore, the low-level DQN controller faced difficulties navigating complex terrain, such as mountains and rivers, resulting in lower Map Coverage.
H ABA-Sparse TASKS FULL RESULTS
WaterDDit-Water
Beef-Dirt-Beef
Wool-Dirt-Wool
Milk-Dirt-Mik
1.0
1.0
10
14-
Success Rate
04-
344-
50
6.0
0.
Water-Seed-Water
Beef-Seed-Beef
Wool-Seed-Wool
Milk-Seed-Milk
1.4
16-
14-
Success Rate
08
12
0:
02
32-
0.0
Water-Log-Water
Beef-Log-Beef
Wool-Log--oool
Milk-Log-Milk
1.0
Success Rate
6.8
64-
68-
64
04
2.4
02
00
Water-Leave Water
Leev- Beef
Wool-Leaves Wool
Milk-Leaves Milk
1.8
1.0
1.0
1.8
Success Rate
0.
3.4
0.0
Water-Sand-Wate
Beef-Sand-Beef
Wool-Sand-Wool
Milk-Sand-Mik
Success Rate
0.
62-
02-
02
McStrur
MrStrun-FM
MM
McStrorFM
PMG-MGGmee.FM
Figure 12: ABA-Sparse Tasks Full Result.
We report the results of all combinations the ABA-Sparse tasks in Figure 12 The experiments conducted under the conditions described in Section 4.2
We note that MrSteve and
29

---

Published as a conference paper at ICLR 2025
its memory variants consistently outperform Steve-1 in all tasks. In of PMC-MrSteve-FMM performed better than Steve-I in most tasks, while it failed on some tasks due to a suboptimal exploration strategy.
I MRSTEVE IN RANDOMLY GENERATED MAPS
Water-Log-Water
Beef-Log-Beef
Wool-Log-Woon
SEQ(4)
Success Rate
62-
6:2-
0.2
24-
M2-
MrStavw
Mr-tewe-EM
MrSteae-P
MrStoswFM
PMG-AMSSwweFFM
Steve=
Figure 13: The performance different agents ABA-Random tasks. MrSieve consistently outperforms
Steve-1 in randomly generated map.
In this experiment, we investigate whether MrSteve benefits over Steve-1 in sequential task when the map randomly generated. To verify this, we created random Plains maps using the Minedojo environment (Fan et al. 2022b), and set up tasks similar to the ABA-Sparse tasks described in Section We call this task, ABA- ABA-Random task.
4.2
In this experiment, task A could involve collecting either water, beef, or wool, while task B involves gathering logs. Additionally, we introduce a sequential task named SEQ (4), which requires the agent to solve four consecutive tasks: log, water, wool, and beef. For the ABA-Random tasks, the agent was given 12K steps, and for the SEQ (4) task, 16K steps were allowed. We evaluated the following agents: MrSteve, MrSteve-EM, MrSteve-PM, MrSteve-FM, PMC-MrSteve-FM, and Steve-1.
As shown Figure 13 MrSteve and its memory variants consistently showed higher success rate than Steve-1 across all tasks. This is because when Steve-1 is finished with task B, it tries to solve final task A from scratch, making it difficult to complete all tasks limited time. However, MrSteve could retain the experience frames of task in memory. making it efficient to solve all tasks in time. This result suggests that augmenting memory plays a crucial role in solving sequential tasks, in randomly generated map.
J ABLATION STUDIES ON PLACE EVENT MEMORY
In this section, we study the robustness of our proposed agent MrSteve by exploring the effects of top-k selection in Mode Selector and the size of a place cluster for Place Event Memory. For this, used one of the tasks from ABA-Sparse task in Section 2 which is Wool-Log- Woo! task. The success rate of MrSteve with different top-k's and place cluster sizes are illustrated in Figure [4] For top-k experiment, we tested five I values, which are 1, 2, 4, 10, and 30. From the results, we found that using small did not largely affect the performance. For cluster size experiment, we tested five cluster sizes, which are 1, 2, 4, 6, and 12 in Minecraft blocks. From the results, we see that using bigger cluster size slightly lowered the performance since the center embedding of the place cluster may not be good summarization vector the place when cluster size is large, making the agent difficult to recall task-relevant frames in the memory.
Success zgt
0.8
0.6
0.6
0.4
0.4
0.2
0.0
10
top-k
Cluster Size
Figure 14: Success Rates of MrSteve with different top- I's, and cluster sizes Wool-Log- Wool task.
30

---

Published as a conference paper at ICLR 2025
Table 4: Querying Time and FLOPs for each memory type and & values. We report the averages and standard errors over IK query operations.
Memory Type
top-k
Time per Query (ms)
FLOPs (x 10")
6.24 0.01
1.591 0.001
6.47 0.02
1.610 0.002
Place Event Memory
6.89 = 0.02
1.667 0.003
10
7.92 = 0.03
1.791 + 0.004
30
11.24 = 0.07
2.210 + 0.010
1
11.28 = 0.35
1.603 + 0.054
2
18.95 0.56
2.711 0.073
Event Memory
42.41 = 1.20
3.734 0.088
10
53.82 = 1.11
7.476 0.150
30
117.16 2.02
12.694 + 0.201
1
3.86 + 0.05
1,251 0.001
2
4.21 + 0.08
1,281 + 0.002
Place Memory
4
4.72 0.08
1.338 1 0.003
10
5.98 + 0.09
1.504 0.005
30
9.85 = 0.12
2.024 + 0.011
FIFO
438.46 0.10
52.481 +. 0.000
K COMPUTATIONAL COMPLEXITY OF QUERY OPERATION
We provide the computational complexity of query operation of each memory type in Table 4 ensure fair comparison, we generated an episode with a 100K-step trajectory, allowing each type of memory to process identical observations. After constructing each type of memory with this trajectory, we randomly selected 1,000 observation embeddings from the trajectory for the query operations. The query time represents the elapsed time during the query function call. FLOPs denotes the number of floating-pont operations required for the MineCLIP score calculation. For FIFO memory, MineCLIP scores computed once during task alignment score calculation. For the other memory types, MineCLIP scores are calculated twice: during the top-k selection and for the task alignment score.
To
The number of clusters affects the complexity of top-k selection and the number of frames per cluster influences the complexity of the task alignment score calculation. Place Event Memory consists of approximately 3,000 clusters, with each cluster containing an average of 30 frames. Place Memory, on the other hand, includes about 2,300 clusters, each holding an average of 43 frames. Although Place Memory provides the fastest querying time, Place Event Memory has comparable speed while achieving similar or superior success rates in most tasks.
Event Memory comprises around 582 clusters, with an average of 172 frames per cluster. Its clustering approach relies on visual discriminative features, which means visually similar places are grouped together, even if they are spatially distant. As result, Event Memory has significantly fewer clusters compared to Place and Place Event Memory.
Overall, the three hierarchical memory types are computationally efficient and lightweight compared to FIFO Memory, which calculates task alignment scores on whole 100K frames. contrast, Place Event Memory with 1: = 30 evaluates roughly 3,900 frames per query (3,000 for top-k selection and 30 frames per cluster across 30 clusters), resulting in a performance that is approximately 40 times faster and requires about 24 times fewer FLOPs than FIFO Memory.
L GOAL-CONDI VPT NAVIGATOR DETAILS AND INVESTIGATION
L. GOAL-CONDITIONED VPT NAVIGATOR
DETAILS
When the goal location LG is selected by high-level goal selector, it is important for the agent to navigate to the goal location accurately. Since navigating complex terrains (e.g.. river, mountain) requires human prior knowledge, VPT initial policy, and fine-tune it for goal-conditioned
31

---

Published as a conference paper at ICLR 2025
navigation policy. We name this model VPT-Nav. To see how VPT-Nav model works, first describe the components of VPT as follows:
Image Encoder:
2t 18 (ix)
Transformer-XL:
Tt-Ta (
Policy Head:
āg ~ my (
In previous works, there were different approaches in fine-tuning VPT for specific purpose. First is goal-conditioned behavior cloning. In Steve-1 (Lifshitz et al., 2024). authors added linear projection of text embedding the image embedding Iy and fine-tuned the whole VPT model for behavior cloning from human demonstration data. This makes Steve-1 I a text-conditioned VPT. In GROOT (Cai al. 2023b), authors used gated cross-attention layers (Alayrac al. 2022) Transformer-Xl
(Tr-LL) to condition the video of some tasks (e.g., log wood). GROOT was trained for behavior cloning from reference videos of human plays working as condiiiond VPT.
The second is RL fine-tuning for the single task. In DECKARD (Notingham et al. 2023 and PTGM
(Yuan et al. 2024). VPT was fine-tuned for single task with PPO (Schulman et al. 2017) 7) by attaching adaptor (Houlsby et al. 2019) in Tr-XL layers, and value head i2 vol (z)). When fine-tuning, only the adaptors, policy head, and value head were updated. For learning stability, those works used different KL loss in PPO objective, which is KL loss between fine-tuning policy and VPT policy to keep the VPT's prior knowledge.
Since target is to train goal goal-conditioned navigation policy, one way achieve this is to naively combining ideas from I) goal-conditioned behavior cloning, and 2) RL fine-tuming for the single task. We first make goal embedding G(1,, IG) from the agent's location 4, and goal location lo with goal encoder G(-), then add it to image embedding zt- Second, we attach the adaptor in Tr-XL. layers, and value head. Then, we fine-tune the whole model only adaptors and policy, value heads with PPO.
However, we found that this naive approach showed suboptimal navigation behavior in complex terrains such as mountain and river. We speculated that this is because RL objective is rather weak learning signal compared to behavior cloning, which causes hardship in giving information of goal embedding to the policy head. Also, some information of goal embedding may corrupted while it is added to image embedding, and processed by Tr-XL layers. Thus, we made the following modifications in the model architecture. First, instead of giving goal embedding Tr-XL. input, we added the goal embedding to policy head input. Second, we used recently proposed adaptor for Tr-XL, which is LoRA (Hu al., 2021b). The following changes enabled VPT-Nav to exhibit optimal navigation behavior. We provide an investigation of this model architecture search in Appendix L.2
With the changes in previous paragraph, the modified VPT for navigation with the learning parameters & has the following components:
Image Encoder:
2, (b,)
Transformer-XI.
4- TrXT.p (F: (FL-T,"".,
Goal Conditioning:
2f = (1, +
(2)
Policy Head:
à ~ (a(a)()))
Value Head:
0. = op(2)),
Here, goal encoder G is 4-layer MLP, and value head is randomly initialized single linear layer. Parameter ub in Tr-XL indicates LoRA parameters. We use PPO for training with reward based on the increase decrease in Euclidean distance between locations of the goal and the agent. When the agent reaches the goal location within 3 blocks, it is considered a success, and an additional reward of 100 given. The detailed parammtetsssrsss for training will be found in Table
L.2 MODEL ARCHITECTURE SEARCH FOR ROAL-CONDITIONER VPT NAVIGATOR
In the previous section, we discussed the naive way of combining the idea of goal-conditioned behavior cloning and RL finetuning for training goal-conditioned navigator. In this section, we conduct model architecture search VPT model to find the optimal coal-conditioned navigation model. To do this, we focused on three key design choices: 1) the input location of goal embedding. 2) training parameters, and 3) different Tr-XI. adaptors. For the location of goal embedding,
32

---

Published as a conference paper at ICLR 2025
Table 5: Hyper parameters for the Goal-Conditionee Navigation VPT Training.
Name
Value
Initial VPT Model
r1- fwwoom--oonnattoom -nn-tiin----
Discount Factor
0.999
Rollout Buffer Size
40
Training Epochs per Iteration
Vectorized Environments
Learning Rate
10-1
KL Loss Coefficient
10"
KL Loss Coefficient Decay
0.999
Total Iteration
400K
Steps per Iteration
500
GAE Lambda
0.95
Clip Range
0.2
consider the three tactics: (1) add the goal embedding to the image embeddings and input to Tr-XL (TrXL-Cond.), (2) add the goal embedding to Tr-XL output (Head-Cond.), and (3) add the goal embedding both at the input and output of Tr-XL (Dual-Cond.). For training parameters, we use full-fnetuning or finetuning only part of the model. For the Tr-XI, adaptors, we consider the adaptor from Houlsby et al. (2019), and LoRA adaptor (Hu et al., 2021b). In Figure 15 we illustrate the different model architectures for goal-conditioned VPT finetuning models.
For the model search, we summarized the navigation performance (SPL and Success Rate (SR)) of different model architectures in Table Interestingly, we found that adding goal embedding to image embeddings (TrXL-Cond.) showed suboptimal behavior, while adding goal embedding to output of Tr-XL (Head-Cond., Dual-Cond.) showed good performance indicating that propagating learning signal to goal embedding through Tr-XL is difficult from RL objective. We note that using LoRA adaptor showed higher performance than adaptors from Houlsby et al.] (2019). In conclusion, using Head-Cond. with LoRA finetuning performed the best. Additionally, we tested one additional model that does not update Tr-XL, which showed comparable result to the best performing model. This architecture benefits from using a smaller number of learning parameters and lower gradient computations. Thus, we use the VPT-Nav trained with one of two architectures, which are Head-Cond. with LoRA finetuning and Head-Cond. with No Tr-XL Update.
Value
Palcy
TrXLg
TrXL,
TrXL
TrXLg
TrXL-Cond. LoRA Fine-Tuning
Dual-Cond. LoRA Fine-Tuning
Head-Cond. LoRA Fine-Tuning
Head-Cond. No TXXL Update
Figure 15: Four Types of Navigation Goal-Conditioning
L.3 VPT NAVIGATOR ABLATION STUDY
We investigate how KL loss in PPO objective (i.e., KL loss between fine-tuning VPT and original VPT) affects the performance of VPT-Nav in diverse environments (Table 7 We measured the SPL and Success Rate (SR) of the navigators navigation tasks where the goal location is within 10-20 blocks away from agent's ss location. While Heuristic method performs best in Flat, Plains, and Mountain tasks, VPT-Nav with KL coefficient 10 showed high performances in all tasks. In the
case of a low-level navigator in Plan4MC (Yuan al. 2023). which uses DQN policy, we observed
33

---

Published as a conference paper at ICLR 2025
Table 6: VPT-Nav Performance of different Goal Con-itondii Methods. Top-1 performances are bolded.
Conditioning
Fine-tuning
Flat
Plains
Mountain
River
Full
0.050 (5%)
0.423 (46%)
0.000 (0%)
0.000 (0%)
TXXL
Houlsby et al.
(0019
0.488 (56%)
0.305 (34%)
0.052 (32%)
0.000 (0%)
LoRA
0.481 (55%)
0.475 (51%)
0.058 (42%)
0.000 (0%)
Houlsby et al (0199
0.883 (95%)
0.828 (90%)
0.125 (949)
0.066 (31%)
Dual
LoRA
0.798 (96%)
0.762 (90%)
0.169 (100%)
0.287 (100%)
No TrXL Update
0.849 (96%)
0.780 (88%)
0.157 (100 %)
0.274 (87%)
Head
Houlsby et (2019)
0.729 (85%)
0.841 (91%)
0,159 (100%)
0.052 (23%)
LoRA
0.904 (98 %
0.880 (95 %) 0.150 (100%)
0.307 (100%)
suboptimal behavior. We note that VPT-Nav is robust to tasks with difficult terrains such as Mountain and River since VPT (Baker et al. 2022) is trained from human demonstration data, which has high-quality navigation ability. Also, using large KL coefficient (e.g., 10 10- harmed the overall performance, while using a small KL coefficient (e.g., 0) harmed the robustness of the navigator in complex tasks (e.g., River).
Table 7: Performance of different Low-Level Navigators, Top-2 performances bolded.
SPL (Success Rate)
Flat
Plains
Mountain
River
Heuristic
0.962 (100%)
0.906 (94%) %
0.188 (100%
0.000 (0%)
Low-level Navigator in Plan4MC
0.416 (61%)
0.326 (47%)
0.010 (10%)
0,000 (0%)
VPT-Nav (KI. =0) =)
0.806 (90%)
0.774 (85%)
0.131 (97%)
0.034 (13%)
VPT-Nav (KL 10
0.849 (96%)
0.780 (88%)
0.157 (100%)
0.274 (87 %)
VPT-Nav (KL 3)
0.762 (95%)
0.651 (78%)
0,015 (17%)
0.012 (6%)
VPT-Nav (KI. =
0.692 (84%)
0.702 (83%)
0.078 (87%)
0.014 (7%)
34

---

Published as a conference paper at ICLR 2025
M LLM-AUGMENTED AGENT WITH MRSTEVE
Table 8: Long-Horizon Planning Tasks details. The required subgoals denote the length of the shortest plan for each task. The items that the low-Icvel controller should obtain are listed in Items Obtain column.
Task
Max Steps
Required Subgoals
Items to Obtain
oak_stairs
3000
logx3
sign
3000
5
logx3
fence
3000
5
logx3
bed
6000
5
logx3, wool
painting
6000
logx2, wool
carpet
6000
2
wool
item item_frame
6000
6
log leatherx 1
leather_bood boots
6000
5
log 1, leatherx4
leather_chestpe cheettlaaeeeeeeeeeeeeeee
6000
log 1, leatherx
leather_helmet
6000
logx1 1, leatherx
leather_ legging
6000
5
log 1, 1eatherx 7
Table 9: Success Rate of two low-level controllers with DEPS (Wagg 2023c planner.
Task
Steve-1
MrSteve
oak_stain
67%
80%
sign
53%
60%
fence
40%
50%
bed
27%
50%
In this section, we investigate synergy between MrSteve and LLM-augmented hicrarchical agent framework with long-horizon planning tasks listed in Table g We follow DEPS (Wang et 2003c) as LLM -ugg-neeeetedd high-level planner with two low-level controllers: Steve-1 and MrSteve.
Once the target object for task is given, the high-level planner asks the LLM to make the initial plan P. = (TZ,) )= for the task, where TH, i-th subgoal at timestep in the textual form. After the initial plan is given, the low-level controller executes the subgoal sequentially. Each subgoal is m mine or craft type, where craft subgoals are exccuted heuristically via MineDojo functional actions, whereas mine subgoals are executed by low-level controllers.
LLM-generated initial plans can be inaccurate, so the low-level controllers often fail to execute subgoals. For instance, we found that the initial plans frequently omit the subgoal for creating crafting table, which is prerequisite item for the tasks in Table g except the carpet task. To address this problem, DEPS framework introduced the replanning procedure with the descriptor and explainer modules. The descriptor module makes a text prompt representing the inventory of the agent. The explainer modules ask the L.LM the reason of failure based on the text prompt from the descriptor module. Based on the explanation, the high-level planner asks the LLM to revise the plan and the low-level controller exccutes subgoals from the revised plan.
Additionally, because the LLM does not observe the environment, the order of subgoals in a plan can be suboptimal. When some subgoals share the same prerequisite so they can be executed in any order, it is more efficient to execute subgoal, which can be completed faster than the other subgoals, first. This can prevent wasting time by giving up subgoals that could be completed quickly, pursuing subgoal whose completion time is uncertain, and then going back to search for the previously quicker task again after exccuting the subgoal. In DEPS, this procedure called elector module is implemented with the horizon prediction module in GSB (Cai et al. 2023a). The horizon prediction module (1,,, T takes the current observation 8 and the textual subgoals [ \. and predicts the time complete each subgoal. Based the time prediction, subgoal to be executed is sampled from the distribution as follows:
exp(-u(sti1 T(,))
Selector(T,,5st,P) E, exp( exp(-juiss (18,, T (())
35

---

Published as a conference paper at ICLR 2025
Table 10: Success Rate of two low-level controllers with the Ground-Truth plan for each task.
Task
Steve-1
MrSteve
oak stairs
80%
83%
sign
70%
67%
fence
67%
70%
bed
37%
60%
painting
60%
73%
carpet
43%
60%
item_frram frame
53%
63%
leather_hood _hooss
13%
33%
leather chestplate
3%
17%
leather helmet
20%
20%
leather leggings
0%
13%
We report success rates of DEPS framework with Steve-1 and MrSteve in Table We Qwen2.5- 72B (Team 2024 as LLM in the high-level planner. DEPS with MrSteve shows comparable performance outperforms compared to DEPS with Steve-1. Especially, there huge performance gap between the two low-level controllers in the bed task, which requires killing three sheeps. We observe that when the agent hit sheep, it runs away from the agent and the agent chases until the agent gets wool items. After this, Steve-1 easily forgets the place where other sheeps exist and tries to find sheep. However, MrSteve avoids this redundant exploration by utilizing the episodic memory. Hence, these results highlight the importance of episodic memory for low-level controllers in LLM-angmented hierarchical agent frameworks.
In Table 10
also report success rates of hierarchical agents with Steve-1 and MrSteve and the optimal plan for each plan. In this setting, because there is no instability from the LLM, the performance is solely determined by performance of low-level controllers. Based on this experiment, although the goal optimal, Steve-1 shows lower performance compared to MrSteve, indicating low-level controllers is performance bottleneck of hierarchical agent frameworks.
36

---

Published as a conference paper at ICLR 2025
N
HIERARCHICAL EPISODIC EXPLORATION
Table 11: Performance Comparison of Two Exploration Methods. We report success rates and the average and standard error of execution times for the explore mode in ABA ABA-Sparse tasks.
Task-Conditinn Conditioned Exploration
Task- Free Exploration
Success Rate
Explore Mode Length
Success Rate
Explore Mode Length
Beef-Log-Bee
93%
1202.64 $7.49
92%
1512.00 1 62.37
Beef-Leaves-Beet Beef
90%
1269.23 1 66.57
91%
1496.10 1 69.43
Wool-Sand Sand-Wool
98%
924.98 + 56.85
93%
1350.00 = 78.80
Wool-Dirt Sool-Drtt-Wol Wool
84%
1303.21 117.04
84%
1452.00 = 89.92
Milk-Sand- Sand-Milk
86%
1197.29 71.44
83%
1526.48 + 63.31
Milk---caves-Milk 59%
1224.24 + 68.79
62%
1579.35 + 64.61
In this section, we investigate an advanced ver- sion of hierarchical episodic exploration. The current exploration method selects the next ex- ploring position based on the visitation map gen- erated from PEM, while it does not use the infor- mation inherited in PEM. Hence, the current ploration method is task-free exploration. While not using information from PEM worked pretty well in practice, using knowledge from PEM may have better explorative behaviors. Thus, we implemented new task-conditioned explo- ration method which exploits information stored in PEM and evaluated two exploration methods on exploration and task-solving tasks.
Visited Pasture Ratio
0.4
0.1
Task-Free
Task Conditioned
0.0
250
500
750
1000
1250
1500
1750
2000
Timestep
Figure 16: Given the task is "find cow", the ratio of pastures among the locations explored time pre- sented for two agents, each employing a differrnnt explo- ration method.
The two exploration methods have the same primary goal: visit the least visited places first. The difference between the two explo- ration methods is how to select one of the least-visited places. The task-free exploration method randomly selects the next exploring position among the least-visited locations. In contrast, the cask-conditioned exploration method selects the next exploring position among the least-visited locations that estimated to be related to the task based on the information inherited in PEM.
To implement the cask-conditioned exploration method, the high-level exploration policy first accesses all event clusters when selecting the next exploring position. Using MineCLIP, the task-relevant score for each event cluster is estimated by calculating the alignment score between the center embedding of the event cluster and the text prompt. If we use the text prompts for MineCLIP listed in Table 3 the alignment score for locations where the target object has not yet been observed will tend to be low, even if the location is relevant the task, potentially hindering the exploration of those areas.
To address this problem, we used text prompts that describe places related to the task. For instance. the text prompt for log is "near pasture", while the text prompt for sand desert." After that, Event clusters with alignment scores exceeding the threshold are collected, and a task-relevance map constructed to represent the agent's 's FoVs of these event clusters, similarly to the visitation map building method described in Appendix E2
set "near
Additionally, the 3 box blur filter applied to the task-relevance map introduce an inductive bias, assuming that if something is near X, it is likely to also be X. Finally, based on the task- relevance map, the high-level exploration policy selects the next exploring position by prioritizing locations that the least visited, most task-relevant, and closest to the agent.
Figure shosh the proportion of task-relevant locations among the places explored by the agent, demonstrating that the task-conditioned exploration method explores task-relevant locations more quickly in the early stages. We used Map 2 in Figure g and set the task as "find cow," and therefore measured the proportion of pastures, which are relevant this task. In the early stages of the episode, the condicondiio exploration method explored pastures more than the fask-free exploration method. After approximately 2000 steps, however, the exploration tendencies of both methods became similar.
37

---

Published as a conference paper at ICLR 2025
Table I1 shows performance comparison between the two exploration methods in ABA- tasks. We report success rates and exccution times for the explore mode from MrSteve with the task-free and fask-conditioned exploration methods. The success rates between the two exploration methods are comparable, while the task-conditioned exploration is finished earlier than the task-free exploration.
Through the results of the two experiments, confirmed that fully utilizing PEM not only aids in task-solving but also optimizes exploration more effectively.
O ABA-Sparse TASKS WITH MEMORY CONSTRAINTS
Wool-Dirt-Woo
Beaf-Log-Bett Beef
Milk-Sand-Milk
2.0
1.0
Success Rate
50
61k
0.5K
1K
24
120
5.0
0.3
0.5C
1K
24
0.0
01K
Wool-Sand-Wool
Beef-Leaves-Beef
Milk-Leaves-Milk
Success Rate
1.0
1.0
0.8
0.1K
0.5K
1K Memory Limit
24
12C
0.1K
0.5c
1K
24
12K
01K
65c
12k
Memory Limit
Memory Limit
MrSteve
MrSteve
MrSteve:
MrSteve-Fii FM
Steve-1
Figure Success Rate Comparison between MrSteve with its memory variants and Steve-1 in ABA--BSpaaa tasks with memory constraints. We tested different memory capacities (0.1K to 12K) for each model. In all tasks, the performance of MrSteve-FM decreases as memory capacity gets smaller. We note that MrSteve is robust to memory capacities across tasks, while MrSteve-PM, and MrSteve-EM, EM showed performance degradation in Beef-Log- Beef and Beef-Leavcs-BB tasks.
In this section, we investigate how MrSteve and its memory variants perform in ABA- ABarree tasks in Section 4.2 when memory capacity is limited. We evaluated each model with different memory capacities ranging from 0.1 12K, which is the maximum episode length. In all tasks, we observe that the performance of MrSteve-FM decreases as the memory capacity gets small. This is because FIFO Memory in MrSteve-FM losts relevant frames first task while solving task B. While MrSteve showed robust performance to constrained memory capacities across tasks, MrSteve-PMr and MrSteve-EM showed degraded performances in Beef-Log when memory capacity is 0.1K. This indicates the robustness of PEM to memory capacities in ABA-Sparse tasks.
and Beef-Leaves- -Beff tasks
P PLACE EVENT MEMORY INVESTIGATION
Although our place event memory enables efficient querying by clustering experience frames, it stores not only the center embedding of each event cluster but also all the frames that constitute the event clusters. Since the frames within each cluster contain highly similar information, storing all of them can increase redundancy in the memory system, potentially degrading storage efficiency and retrieval performance. Therefore, we attempted to optimize the memory-storie method of PEM in the simplest way possible to investigate whether reducing the storage of redundant information could achieve better efficiency.
We implemented the modified PEM, which stores only center embeddings. The write and read operations of the modified PEM are the same the original PEM. However, once event clustering executed, the modified PEM stores only the center embedding of each event cluster and removes the embeddings of other frames. In experiment settings, event clustering is performed when the dummy deque of a place cluster has 100 frames, and until then, frames are stored in the dummy deque of place cluster, with each place cluster capable of holding up 100 frames. However, even
38

---

Published as a conference paper at ICLR 2025
Wool-Dirt-Wool
Beef-Log-
Milk-Sand-Miilk
Success Rate
0.6-
0.6
02
0.2
0.2
Wool-Sand-Wool
Beev-Leaves- Beef
Milk-Leaves-Mir
Success Rate
0.
0.4
6.4
25% Mrštove (Center Embed)
MrSteve EM MrStevs-PM
MrStove FM
MrSteve
Stove-
Figure 18: Success Rates Comparison between MrSteve with its memory variants and Steve-1 in ABA-Sparse tasks. We additionally evaluated MrSteve with modified place event memory. named MrSteve (Center Embed), that stores center embeddings only.
in such cases, memory read operation is highly optimized by accessing only the oldest frame in each place cluster during querying operations.
Figure 18 shows the performance comparison between MrSteve with its memory variants and Steve-1 in ABA-Sparse tasks. Surprisingly, the performance of the simplest optimized PEM, named MrSteve (Center Embed), was not dropped significantly. Nevertheless, the computational cost for the querying operation can be reduced. After 12K environmental steps. 305.52 event clusters, on average, were generated across all tasks, with standard error of 4.98. The original PEM stores all 12K frames and each event cluster in PEM holds approximately 40 frames on average. In the end, PEM calculates the alignment scores between frames from the top-30 relevant event clusters and the task instruction, resulting in 1.2K comparisons. However, the modified PEM calculates the alignment scores for around 0.3K frames, making it more efficient.
This experiment result demonstrates the potential to further optimize the PEM memory-storing method. However, our simplest approach resulted in a slight performance drop. Hence, optimizing memory to further reduce redundancy without losing important frames is worthwhile direction for future work.
39


Reviews:
[{"review_id": "8VM8FMNVyn", "reviewer": "Reviewer_29NL", "summary": "In the presented work, the authors propose an improvement over the widely used STEVE-1 low-level controller used in Minecraft challenges, addressing its limited episodic memory and inefficiency in handling long-horizon tasks. In particular, the authors propose Place Event Memory, a novel approach to storing \"what-where-when\" information. Utilizing this memory, the authors propose a new navigation policy, VPT-Nav, capable of balancing when exploration is needed and when direct task-solving can be done due to recalling prior information from the hierarchical PEM memory. Minecraft is a great testbed for dynamic environments and provides a challenging task for policies targeted at solving such settings where long-horizon task planning is needed.", "questions": "- In the example where meat needs to be found, what happens if navigating back to the location where cows were previously seen at does not have cows anymore? Is there a way to update/forget information in the PEM? \n- In Figure 5, are the results, particularly comparing Mr.Steve, Steve-EM, Steve-PM, and Steve-FM, statistically significant (particularly for Wool-Dirt-Wool and Milk-Sand-Milk)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The introduction is well-written and intuitively understandable.\n- The event-based clustering in the PEM memory seems novel an intuitive \n- The results of Mr.Steve are strong and supportive of the claims made by the authors.", "weaknesses": "- In the PEM memory, clusters are updated based on some threshold c concerning the similarity of events/locations. However, it is unclear how this is handling events that happen dynamically. Even in the \"burning zombies\" example given in the PM section, this event would only happen when it's early morning in the game (thus night-zombies are burning), yet the PEM memory doesn't include the game time, meaning that such an event would be unreliable and there doesn't seem to be a way to capture this. \n- The goal of Hierarchical Episodic Exploration seems to be to also prevent re-visiting places that have previously been seen. Given that the location of the agent is part of the place embeddings e_t, that seems feasible, however, if two environments are visually similar, that would mean that the agent would still explore as its location is a strong bias. However, this is a little in contrast to the goal of, for example, finding wood where searching yet another desert (just because the global position is different) would be suboptimal and should be avoided. So, in these settings, the global location would, on the one hand, need to be a strong separator to prevent searching the same environments, yet, at the same time, a weak separator because we wouldn't want to search the same biome for too long. How is this balance handled? I think a discussion on which factors exactly contribute to the creation of a new cluster would be beneficial for the paper.\n- While the results in section 4.3 are supportive of the author's claims, it would be great if the same settings as in section 4.2 could be tested to demonstrate the impact of varying global memory limitations. E.g. which of the methods can solve the milk-sand-milk task with the least amount of global memory? Such a comparison would make the contribution much stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QtZMr9M9vF", "reviewer": "Reviewer_UwZe", "summary": "Authors present Mr.Steve, an extension of the Steve-1 instruction-following agent, improving exploration and memory abilities in Minecraft settings. The core contribution of authors is the design of a Place Event Memory system, which allows creating a hierarchical memory: a set of map checkpoints is constructed from the agent trajectory through clustering, and in each checkpoint multiple events are memorized. The memory is built and queried based on CLIP embeddings from a previous work, allowing to compute similarities between a language or visual instruction and image-based memories. Additionally, authors finetune an existing transformer-based trajectory embedding network to learn a goal-conditioned policy and implement a count-based exploration system. Authors conduct multiple experiments to showcase the performance of the overall system, which significantly outperforms Steve-1", "questions": "see above", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This work tackles an important subject of research: how to design efficient language-conditioned agents in complex environments.\n\nThe paper is very well written. Experiments are in-depth and clearly described. Results are impressive.\n\nI enjoyed reading it.", "weaknesses": "The current version of the paper could be more pedagogical in some parts.\nFor instance: How is are the text and video encoder aligned ? More explanation about this should be featured in the paper (i.e. more explanation about MineClip I guess).\n\nLikewise, some details regarding the DP Means algorithm, used to figure out checkpoint locations, could be useful. E.g. how does it compare to the well known K-Means ? Does DP Means selects autonomously the number of clusters ? If no, how is the system detecting how many clusters/checkpoints to create for a given agent trajectory ?\n\nl.243 \"This structure improves the efficiency of the read operation by extracting top-k place clusters with their center embeddings first, then fetching relevant frames from these clusters\"\n--> This hierarchical decomposition assumes the “center embedding” is sufficient to figure out which FIFO memory to read. This looks like a strong assumption, e.g. the center embedding could correspond to the agent looking away from the object of interest. But maybe it is enough if clustering is well done ?\n\n### Novelty/Significance\n\nWhile very well executed, this work only tackles a Minecraft scenario, on a few individual contributions are proposed and efficiently combined: a hierarchical memory system, a count based exploration mechanism, and goal-conditioned navigation agent. Count based exploration and xy_goal-directed navigation are well known areas. I am not an expert regarding memory systems for decision-making agents, but similar systems might have been proposed in the past. I am also not expert enough to assess whether the considered baselines are sufficient.\n\nI am looking forward to the discussion period to update my score, but from this first review I recommend acceptance, despite my aforementioned concerns. This work efficiently showcases how to combine and scale known components in a complex and relevant setting.\n\n\nminor:\n\nl.100 \"(Hafner et al., 2023; Guss et al., 2019; Cai et al., 2023a; Mao et al., 2022; Lin et al., 2021; Zhou et al., 2024a)\" \n--> make sure for any multi-citation to order by year from old to new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCDzOLizsH", "reviewer": "Reviewer_UxTw", "summary": "The paper tries to address the forgetting problem in long-horizon embodied control tasks, such as Minecraft, by introducing a new episodic-memory approach called ‘Place Event Memory’ (PEM). This method organizes past observations based on their locations within the environment (Place Memory). To minimize memory redundancy, each group within the Place Memory is further refined into an event-based memory by clustering observations according to their latent embeddings. The memory is utilized to more efficiently locate previously visited objects, particularly in long-horizon tasks. The paper also proposes an exploration-exploitation strategy allowing agents to alternate between exploration and task-solving based on recalled events. Experiment results showed that the proposed method outperforms traditional method (short-term memory and traditional FIFO memory) in long-horizon tasks.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is technically sound and clearly describes the algorithm. The proposed approach (Place Event Memory, PEM) efficiently manages memory by integrating place-based and event-based memory, effectively reducing redundancy while preventing confusion from similar observations at different locations. This mechanism enables agents to recall previous memories during long sequences of tasks, thereby enhancing task efficiency.\n2. The results on long-horizon tasks are good.\n3. The presentation of the paper is well-executed.", "weaknesses": "1. The work requires further improvement: \n    - The authors refer to their proposed approach as “Place Event Memory” (PEM), where event memory is implemented by clustering according to the latent embeddings encoded by MineCLIP. However, many frames during the tasks do not constitute meaningful events. For instance, when searching for an object, it may take several steps to navigate in the environment. These observations may not be considered as events (and these observations are also not trained in MineCLIP, I’m uncertain about how they will be clustered). Therefore, it may not be appropriate to call it “event memory”, and there's still some unnessary memory storage.\n    - For exploration, the authors use a visitation map and always select positions with the lowest visit frequency. This rule-based method is not efficient and does not leverage the knowledge saved in PEM. Why did the authors not consider using the same mechanisms for both exploration and exploitation?\n    - The experimental results require further explanation. In Section 4.4, place-based memory achieves better performance on long-instruction tasks, while event-based memory performs better on long-navigation tasks. This result seems counterintuitive. Can the authors further explain these experimental outcomes?\n   - The key idea of this work is to record the positions of visited objects, framing the task as a lifelong navigation problem. There has been significant research on navigation, particularly regarding navigating in unseen environments and exploring to locate target objects while recording previously seen ones (e.g., building a semantic map). Why did the authors not include a comparison with these works?\n2. symbol expression: On page 4, the notation $O_t=\\{o_t, l_t, t\\}$ is used. In Markov Decision Processes (MDPs), uppercase $O$ typically represents the observation space, while lowercase $o_t$ represents an element within that space. This notation should be clarified.\n3. spelling errors: There are some spelling errors in the appendix. For example, on page 22, “Evene Cluster Details” should be corrected to “Event Cluster Details.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vf2HHx8D5C", "reviewer": "Reviewer_86n7", "summary": "This work approaches the failures of low-level controllers for navigation in embodied AI environments. It argues that one of the crucial reasons for these failures is the lack of memory for recalling objects and events, which could lead to redundant exploration. For this purpose, it proposes Place Event Memory (PEM), an episodic memory mechanism that tracks entities alongside with their locations and associated events. By using PEM, they employ a count-based exploration strategy, which alternates exploration/exploitation behaviors based on the recalled events. The work also proposes a new low-level navigator built on top of the TrXL architecture. Experiments in simple minecrafts navigation tasks demonstrate gains over prior methods and baselines with either no or simpler mechanisms\nof memory.", "questions": "As mentioned in my points above:\n\n- What are the advantages of the proposed method in comparison with other methods for the SLAM/robot navigation problem?\n\n- How does the proposed method function in the case of non-static entities, as pointed out in my second concern?\n\n\n**Summary of the Review**:\n\nThe work is well written, the method is very clearly presented, and the evaluation setup is well diversified, covering important aspects of what is proposed. Nonetheless, it is unclear if the considered navigation tasks are challenging enough to reflect the adopted motivation. It is also unclear if the method is advantageous over the classic robot navigation method and how the work is placed in comparison with this literature. I believe these are crucial questions to be answered by the work in order to understand what we can take from it.\n\n**References**\n\n[1] Simultaneous localization and mapping (SLAM). Wikipedia, available in: https://en.wikipedia.org/wiki/Simultaneous_localization_and_mapping\n\n[2] Street et. al. Multi-Robot Planning Under Uncertainty with Congestion-Aware Models. AAMAS, 2020.\n\n[3] Garg et. al. RoboHop: Segment-based Topological Map Representation for Open-World Visual Navigation. ICRA, 2024.\n\n**POST-REBUTTAL**\n\nI appreciate the authors' efforts on the rebuttal phase to clarify reviewers' concerns.\nAfter reading the comments, I believe my concerns were only partially addressed, particularly the first concern related to the comparison with other approaches for SLAM. The rebuttal limited to contrast with other recent work, but without any empirical comparison nor considering classic work on the Robotics literature.  My second concern was acknowledged by the authors during the rebuttal, and I understand it is a natural limitation of the work to be addressed in future iterations. My third concern was not addressed and was unclear why it was not possible to use the 100 seeds already executed to compute the error bars.\n\nI am slightly increasing my score to borderline acceptance to reflect the additional evidence and clarifications raised by the authors in response to all reviewers. Still, I recommend authors to consider classic SLAM approaches as future baselines for this line of work. This weakness is a consensus among the reviewers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work is well written and structured, easy to follow, with good illustrations; the proposed method and motivation are clear.\n\n- The particular choice of tasks is very smart since it shows different scenarios where only place memory or only event memory are not enough, justifying a combined mechanism as proposed by the work. This provides good positive evidence for the scenarios where the method is effective.\n\n- The evaluation setup also comprises different setups and evaluation criteria to evaluate the method in terms of exploration and under circumstances of sparsity, constrained memory, and long-horizon navigation.", "weaknesses": "- The major criticism comes from the fact that the paper is motivated by advancing methods for general-purpose embodied AI, but the evaluated scenarios are simplified versions of the SLAM [1] problem. The only considered task is navigation, and the work assumes access to a grid map with the perfect localization of entities. Entities are also assumed to be static (see the point below). In this setup, the only challenge seems to be identifying the different objects, which is actually easily solvable by visual foundational models (MineCLIP). In this setup, one could easily build and maintain a topological map of the environment as in prior work [2, 3] and perform exploration and navigation on top of it. Therefore, it is unclear whether (1) the tasks are challenging enough to reflect general-purpose embodied AI problems and (2) what are the advantages of the proposed method over classic SLAM methods. These points should be clarified in the paper.\n\n- As mentioned in the previous point, entities are assumed to be static, which assumes the entities in the map have fixed location and state. Nevertheless, in open-world settings this does not hold true, i.e., a “zombie” may move around or “be burnt” while the agent is not observing. These changes in the entities would add an uncertainty component in the memory that is disregarded by the proposed method - in other words, memory may become stale over time and the proposed mechanism does not have a way to update/discard these memories. Again, the work should clarify this potential limitation.\n\n- The Success Rate bar plots (in Figures 5, 6, 12, 13, 14) should bring the error bars (confidence intervals) so that it is possible to analyze statistical significance of the reported results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)