Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks
Authors: Zaijing Li
Yuquan Xie
Rui Shao
Gongwei Chen
Dongmei Jiang
Liqiang Nie
Venue: NeurIPS 2024 poster (2024)
Keywords: Multimodal Agent
Multimodal Large Language Models
Multimodal In-context Learning
Primary area: robotics
TL;DR: We propose a powerful agent with hybrid multimodal memory architecture, Optimus-1, in Minecraft.

Abstract: Building a general-purpose agent is a long-standing vision in the field of artificial intelligence. Existing agents have made remarkable progress in many domains, yet they still struggle to complete long-horizon tasks in an open world. We attribute this to the lack of necessary world knowledge and multimodal experience that can guide agents through a variety of long-horizon tasks. In this paper, we propose a Hybrid Multimodal Memory module to address the above challenges. It 1) transforms knowledge into Hierarchical Directed Knowledge Graph that allows agents to explicitly represent and learn world knowledge, and 2) summarises historical information into Abstracted Multimodal Experience Pool that provide agents with rich references for in-context learning. On top of the Hybrid Multimodal Memory module, a multimodal agent, Optimus-1, is constructed with dedicated Knowledge-guided Planner and Experience-Driven Reflector, contributing to a better planning and reflection in the face of long-horizon tasks in Minecraft. Extensive experimental results show that Optimus-1 significantly outperforms all existing agents on challenging long-horizon task benchmarks, and exhibits near human-level performance on many tasks. In addition, we introduce various Multimodal Large Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show that Optimus-1 exhibits strong generalization with the help of the Hybrid Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.

Optimus-1 Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks
Zaijing Li Yuquan Xie*, Rui Shao"; Gongwei Chen', Dongmei Jiang", Liqiang Niel
Harbin Institute of Technology, Shenzhen
Peng Cheng Laboratory
[1zj 14011, xieyuquan200166 1 0ar.:000,,rrraohoo rshaojiimaan niel Magaan com com
Task: Craft stone sword.
Hybrid Multimodal Memory
Experience-Driie Driven Reflector
Obs
Abstracted Multimodal
Experience Pool
Retrieve: HemooryExperience
Tayk Oit stane palaoo
Goal: Mine
Action Controller
Reflection Mine
Viuse info heaet 14
Siree
Action: 55
Hierarchical
Knowledge Graph
Knowledge-Guiie Guided Planner
Mission Successil!
Obs
Retrieve: Memory/Knowledgn
Plan:
Figure 1:
An illustration of Optimus-1 performing long- horizon tasks in Minecraft. Given the task Planner incorporates knowledge from Hicrarchical Directed
"Craft stone sword", Knowledge-Guide Knowledge Graph into planning, then Action Controller executes these planning sequences step-by- step. During the execution of the task, the Experience-Driver Reflector is periodically activated and retrieve experience from Abstracted Multimodal Experience Pool make reflection.
Abstract
Building -unproe agent long-standing vision in the field of artificial intelligence. Existing agents have made remarkable progress in many domains, yet they still struggle to complete long-horizon tasks in an open world. We attribute this to the lack of necessary world knowledge and multimodal experience that can guide agents through variety long- mogghorizon tasks. In this paper, we propose a Hybrid Multimodal Memory module to address the above challenges. It 1) transforms knowledge into Hierarchical Directed Knowledge Graph that allows agents to explicitly represent and learn world knowledge, and 2) summarises histor- ical information into Abstracted Multimodal Experience Pool that provide agents
*Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

with rich references for in-context learning. On top of the Hybrid Multimodal Memory module, multimodal agent, Optimus-1, constructed with dedicated Knowledge-guided Planner and Experience- better planning and reflection in the face of long-horizon tasks in Minecraft. Exten- sive experimental results show that Optimus-1 significantly outperforms all existing agents on challenging level performance on many tasks. In addition, we introduce various Multimodal Large Language Models (MLLMs) as the backbone Optimus-1. Experimental results show that Optimus-1 exhibits strong generalization with the help of the Hy- brid Multimodal Memory module, outperforming the GPT-4V baseline on various tasks. Please see the project page https///cybertronagent wwwpoonnggntgggguuu.... github.io/Optimni. 1. I.github.io/.
Reflector, contributing to
task benchmarks, and exhibits near human-
Introduction
Optimus Prime faces complex tasks alongside humans in Transformers to protect the peace of the planet. Creating an agent [44, 13] like Optimus that can perceive, plan, reflect, and complete long-horizon tasks in open world has been a longstanding aspiration in the field of artificial intelligence [22, 36, 37, 27, 58]. Early research developed simple policy through reinforcement learning [7] or imitation learning [1, 25]. A lot of work [47.50] have utilized Large Language Models (LLMs) as action planners for agents, generating exccutable sub-goal sequences for low-level action controllers. Further, recent studies [52, 33] employed Multimodal Large Language Models (MLLMs) 39, 56] as planner and reflector. Leveraging the powerful instruction- foollioon and logical reasoning capabilities of (Multimodal) LLMs [24], LLM-based agents have achieved remarkable success across multiple domains [14, 9, 10, 55J. Nevertheless, the ability of these agents to complete long-horizon tasks still falls significantly short of human-level performance.
According to relevant studies [28, 42, 46), the human ability to complete long-horizon tasks in an open world relies long-term memory storage, which is divided into knowledge and experience. The storage and utilization of knowledge and experience play a crucial role in guiding human behavior and enabling humans to adapt flexibly to their environments in order to accomplish long- tasks. Inspired by this theory, we summarize the challenges faced by current agents as follows:
Insufficient Exploration of Structured Knowledge: Structured knowledge, encompassing open world rules, object relationships, and interaction methods with the environment, essential for agents to complete complex tasks [34, 44J. However, MLLMs such as GPT-4V lack sufficient knowledge in Minecraft. Existing agents [1, 25, 7] only learn dispersed knowledge from video data and are unable to efficiently represent and learn this structured knowledge, rendering them incapable of performing complex tasks.
Lack of Multimodal Experience: Humans derive successful strategies and lessons from information on historical experience [8, 32], which assists them tackling current complex tasks. In a similar manner, agents can benefit from in-context learning with experience demonstrations [43, 54]. How- ever, existing agents [47, 51, 33] only consider unimodal information, which prevents them from learning from multimodal experience as humans do.
To address the aforementioned challenges, we propose Hybrid Multimodal Memory module that consists of Hierarchical Directed Knowledge Graph (HDKG) and Abstracted Multimodal Experience Pool (AMEP). For HDKG, we map the logical relationships between objects into a directed graph structure, thereby transforming knowledge into high-level semantic representations. HDKG efficiently provides the agent with the necessary knowledge for task execution, without requiring any parameter updates. For AMEP, we dynamically summarize and store the multimodal information (e.g., environment, agent state, task plan, video frames, etc.) from the agent's 's task execution process, ensuring that historical information contains both global overview and local details. Different from the method of directly storing successful cases as experience [52], AMEP considers both successful and failed cases as references. This innovative approach of incorporating failure cases into in-context learning significantly enhances the performance of the agent.
2

---

On top of the Hybrid Multimodal Memory module, we construct a multimodal composable agent, Optimus-1. As shown in Figure Optimus-I consists of Knowledge Guided Planner, Experience- Driven Reflector, and Action Controller. Too enhance the ability of agents to cope with complex environments and long-horizon tasks, Knowledge-Guiee Guided Planner incorporates visual observation into the planning phase, leveraging HDKG to capture the knowledge needed. This allows the agent to efficiently transform tasks into executable sub-goals. Action Controller takes the sub-goal and the current observation as imputs and generates low-level actions, interacting with the game environment to update the agent's state. In open-world complex environments, agents are prone to be erroneous when performing long- monghorizon tasks. To address this, we propose Experience-Driie Reflector, which periodically activated to retrieve relevant multimodal experiences from AMEP. This encourages the agent to reflect on its current actions and refine the plan.
hewwedde-Guiiedd
We validate the performance of Optimus-1 in Minecraft, popular open-world game environment. Experimental results show that Optimus-1 exhibits remarkable performance on long-horizon tasks, representing up to 30% improvement over existing agents. Moreover, we introduce various Multi- modal Large Language Models (MLLMs) as the backbone of Optimus-I. Experimental results show that Optimus-1 has a to times performance improvement with the help Hybrid Multimodal Memory, outperforming powerful GPT-4V baseline on lots of long-horizon tasks. Additionally, we verified that the plug-and and-play Hybrid Multimodal Memory can drive Optimus-1 1 to incrementally improve its performance in self-evold evolutio manner. The extensive experimental results show that Optimus-1 I makes major step toward a general agent with a human-like level of performance. Main contributions of our paper:
We propose Hybrid Multimodal Memory module which is composed of HDKG and AMEP. HDKG helps the agent make the planning of long-horizon tasks efficiently. AMEP provides refined historical experience and guides the agent to reason about the current situation state effectively.
On top of the Hybrid Multimodal Memory module, we construct Optimus-1, which consists of Knowledge-Guidee Planner, Experience-Driver Reflector, and Action Controller. Optimus-1 outperforms all baseline agents on long-horizon task benchmarks, and exhibits capabilities close to the level of human players.
Driven by Hybrid Multimodal Memory, various MLLM-based Optimus-1 have demonstrated 2 to 6 times performance improvement, demonstrating the generalization of Hybrid Multimodal Memory.
2 Optimus-1
In this section, we first elaborate on how to implement the Hybrid Multimodal Memory in Sec 2.1. As core innovation, plays a crucial role in enabling Optimus-1 execute long-horizon tasks. Next, we give an overview of Optimus-1 framework (Sec 2.2), which consists of Hybrid Multimodal Memory, Knowledge-Guud Guided Planner, Experience-Drivee Driven Reflector, and Action Controller. Finally, we introduce non-pan -on-aammetricc learning approach to expand the hybrid multimodal memory (Sec 2.3), thereby enhancing the success rate of task execution for Optimus-1.
2.1 Hybrid Multimodal Memory
In order endow agent with long-term memory storage mechanism [28, 46), we propose the Hybrid Multimodal Memory module, which consists of Abstracted Multimodal Experience Pool (AMEP) and Hierarchical Directed Knowledge Graph (HDKG).
2.1.1 Abstracted Multimodal Experience Pool
Relevant studies [23, 29, 17, 15] highlight the importance of historical information for agents completing long-horizon tasks. Minedojo [7] and Voyager [47] employed unimodal storage of historical information. Jarvis-1 1 [52] used multimodal experience mechanism that stores task planning and visual information without summarization, posing challenges to storage capacity and retrieval speed. To address this issue, we propose AMEP, which aims dynamically summarize all multimodal information during task execution. It preserves the integrity of long-horizon data while enhancing storage and retrieval efficiency.
3

---

Video
visual
info
Image Buffer
Knowledge
updase
Hierarchical Directed Knowledge Graph
Mine
Visual Encodier
CLIP
Craft stone pichaxe
Text Encoder
Abstracted Multimodal Experience Pool
Filter
pickaxe.
Obs:
hotbar: ampty
Plan:
(a)
(b)
Figure 2: (a) Extraction process of multimodal experience. The frames are filtered through video buffer and image buffer, then MineCLIP [7] is employed to compute the visual and sub-goal sim- ilarities and finally they are stored Abstracted Multimodal Experience Pool. (h) Overview of Hierarchical Directed Knowledge Graph. Knowledge is stored as directed graph, where its nodes represent objects, and directed edges point to materials that can be crafted by this object.
Specifically, depicted in Figure to conduct the static visual information abstraction, the video stream captured by Optimus-1 during task execution is first input to a video buffer, filtering the stream fixed frequency of frame per second. Based the filtered video frames, further perform a dynamic visual information abstraction, these frames are then fed into an image buffer with window size of 16, where the image similarity is dynamically computed and final abstracted frames are adaptively updated. To align such abstracted visual information with the corresponding textual sub-goal, we then utilize MineCLIP [7]. pre-trained video-text alignment model, calculate their multimodal correlation. When this correlation exceeds a threshold, the corresponding image buffer and textual sub-ge sub-goal are saved as multimodal experience into a pool. Finally, we further incorporate environment information, agent initial state, and plan generated by Knowledge- Guided Planner, into such a pool, which forms the AMEP. In this way, we consider the multimodal information of each sub-goal, and summarise it to finally compose the multimodal experience of the given task.
Growledge-Guided
2.1.2 Hierarchical Directed Knowledge Graph
In Minecraft, mining and crafting represent a complex knowledge network crucial for effective task planning. For instance, crafting a diamond sword & requires two diamonds @ and one wooden stick /, while mining diamonds requires an iron pickaxe 7), which involving further materials and steps. Such knowledge is essential for an agent's ability to perform long-horizon complex tasks. Instead of implicit learning through fine-tuning [33, 60], we propose HDKG, which transforms knowledge into graph representation. It enables the agent to perform explicit learning by retrieving information from the knowledge graph.
As shown in the Figure 2, we transform knowledge into graph D(V, E), where nodes set V represent objects, and directed edges set point to nodes that can be crafted by this object. An edge E E in the D can be represented as (u, v), where u. v E V. The directed graph efficiently stores and updates knowledge. For given object ar, retrieving the corresponding node allows extraction of sub-graph DD (V,, Eÿ) E DD where nodes set Vi and edges set & can be formulated as:
V; = {u E E [
& (e (u, v) E V u E U E V2),
(I)
Then by topological sorting, we can get all the materials and their relationships needed complete the task. This knowledge is provided to the Knowledge-Guice Guided Planner way to generate more

---

Hybrid Multimodal Memory
Hierarchical Directed Knowledge Graph
Abstracted Multimodal Experience Pool
Env: forest
Task: Mine cobblestone.
Visual Info: health: full, food: hullhotbar:: empty.
Obs:
Plan:
Knowledge-Guided Planner
Dxperience-Driven Reflector
Sub-Goals
Reflection
[replan]
Obs:
Current Obs:
[continue]
Knowledge:
Sub-Goal: Mine 11 cobbiestone.
[complete]
In-context Cases
Replan:
Continue:
Action Controller
Complete:
Task: Craft diamond.
Environment
Figure 3: Overview framework of our Optimus-1. Optimus-1 I consists of Knowledge-Guide Planner, Experience-Driver Reflector, Action Controller, and Hybrid Multimodal Memory architecture. Given the task "craft stone sword", Optimus-1 incorporates the knowledge from HDKG into Knowledge- Guided Planning, then Action Controller generates low-level actions. Experience- periodically activated to introduce multimodal experience from AMEP to determine the current task can be executed successfully. If not, will ask the Knowledge-Guud Guided Planner refine the plan.
Reflector is
reasonable sequence of sub-goals. With HDKG, significantly enhance the world knowledge of the agent in train-free manner.
2.2 Optimus-1: Framework
Relevant studies indicate that the human brain is essential for planning and reflection, while the cerebellum controls low-level actions, both crucial for complex tasks [40, 41]. Inspired by this, we divide the structure of Optimus-1 into Knowledge-Guiie Guided Planner, Experience-Driven Reflector, and Action Controller. In given game environment with long-horizon task, the Knowledge-Guud Guided Planner senses the environment, retrieves knowledge from HDKG, and decomposes the task into executable sub-goals. The action controller then sequentially executes these sub-goals. During execution, the Experience-Drive Driven Reflector activated periodically, leveraging historical experience from AMEP to assess whether Optimus-1 can complete the current sub-goal. If not, it instructs the Knowledge-Guided Planner to revise plan. Through iterative interaction with the environment, Optimus-1 ultimately completes the task.
Knowledge-Guided Planner. Open-world environments vary greatly, affecting task execution. Previous approaches [51] using LLMs for task planning failed to consider the environment, leading the failure of tasks. For example, an agent cave aims catch fish. lacks visual information to plan conditions on the current situation, such "leave the cave and find river". Therefore, we integrate environmental information into the planning stage. Unlike Jarvis-1 [52] and MP5 [33], which convert observation to textual descriptions, Optimus-1 directly employs observation visual conditions generate environment-related plans, i.e., sub-goal sequences. This results in more comprehensive and reasonable planning. More importantly, Knowledge-Guice Guided Planner retrieves the knowledge needed to complete the task from HDKG, allowing task planning be done once, rather than generating the next step in each iteration. Given the task t, observation o, the sub-goals
5

---

sequence 91 92, 93. ... In can be formulated as:
91, 92, -...................... In = po L, Py)()))
where n the number of sub-goals, Py denotes sub-graph retrieved from HDKG, pe denotes MILM. In this paper, we employ OpenAl's GPT-4V as Knowledge- ahowledge-Guided Guided Planner and Experience- Reflector. We also evaluate other alternatives of GPT-4V, such as open-source models like Deepseek- VL [26] and InternL IntemLM-XCoompser2-V
[6] in Section 3.4.
Action Controller. It takes the sub-goal and the current observation as inputs and then generates low-level actions, which are control signals for the mouse and keyboard. Thus, it can interact with the game environment to update the agent's state and the observation. The formulation is as follows:
ag - Pr (o,
where dg denotes low-level action at time k; Px denotes action controller. Unlike generating code [47, 33, 50], generating control actions for the mouse and keyboard [1, 25, 52, 3] more closely resembles human behavior. In this paper, we employ STEVE-1 [25] as Action Controller.
Experience-Driven Reflector. The sub-goals generated by Knowledge-Guided Planner are interde- pendent. The failure of any sub-goal halts the execution of subsequent ones, leading to overall task failure. Therefore, reflection module is essential identify and rectify errors promptly. During task execution, the Experience-Driee experience from AMEP, and then analyzing the current state of Optimus-1. 1. The reflection results of Optimus-1 are categorized as COMPLETE, CONTINUE, REPLAN. COMPLETE indicates successful execution, prompting the action controller to proceed to the next sub-goal. CONTINUE signifies ongoing execution without additional feedback. REPLAN denotes failure, requiring the Knowledge- Guided Planner to revise the plan. The reflection r generated by Experience- Driven Reflector can be formulated as:
Reflector activates at regular intervals, retrieving historical
p.).)).
where Pt denotes multimodal experience retrieved from AMEP. Experimental results in Section 3.3 demonstrate that the Experience-Drivee Reflector significantly enhances the success rate of long-horizon tasks.
During task execution, even in cases where task failure necessitates REPLAN, multimodal experiences are stored in AMEP. Thus, during the reflection phase, Optimus-I can retrieve the most relevant cases from each of the three scenarios COMPLETE, CONTINUE, and REPLAN from AMEP as references. Experimental Results in Section 3.3 demonstrate the effectiveness of this innovative method of incorporating failure cases into in-context learning.
2.3 Non-parametric Learning of Hybrid Multimodal Memory
To implement the Hybrid Multimodal Memory and enhance Optimus-1's 's capacity, we propose non- parametric learning method named "free exploration-tedd teacchrr guidance". In the free exploration phase, Optimus-1's 's equipment and tasks are randomly initialized, and it explores random environments, acquiring world knowledge through environmental feedback. For example, learns that "a stone sword x can be crafted with wooden stick / and two cobblestones Additionally, successful and failed cases are stored in the AMEP, providing reference experience for the reflection phase. We initialize multiple Optimus-1, and they share the same HDKG and AMEP. Thus the memory is filled up efficiently. After free exploration, Optimus-I has basic world know knowledge and multimodal experience. In the teacher guidance phase, Optimus-1 needs to learn small number of long-horizon tasks based on extra knowledge. For example, it learns "a diamond sword x is obtained by a stick / and two diamonds sword". During the teacher guidance phase, Optimus-1' 's memory is further expanded and gains the experience of executing complete long-horizon tasks.
storing this in the HDKG.
from the teacher, then perform the task "craft diamond
Unlike fine-tuning, this method enhances Optimus-1 incrementally without updating parameters, in self-evolution manner. Starting with an empty Hybrid Multimodal Memory, Optimus-1 iterates between "free exploration-teacher guidance" learning and unseen task inference. With each iteration, its memory capacity grows, enabling mastery of tasks from easy to hard.

---

Table Main Result Optimus-1 on
tasks benchmark. We report the average success rate (SR), average number of steps (AS), and average time (AT) on each task group. the results of each task can be found in the Appendix F. Lower AS and metrics mean that the agent is more efficient at completing the task, while +oo indicates that the agent is unable to complete the task. Overall represents the average result on the five groups of Iron, Gold, Diamond, Redstone, and Armor.
Group
Metric
GPT-3.5
GPT-4V
DEPS
Jarvis-1
Optimus-1
Human-level
1
40.16
41.42
77.01
93.76
98.60
100.00
Wood
AT 4
56.39
55.15
85.53
67.76
47.09
31.08
AS +
1127.78
1103.04
1710.61
1355.25
841.94
621.59
SR 1
20.40
20.89
48.52
89.20
92.35
100.00
Stone
AT
135.71
132.77
138.71
141.50
129.94
80.85
4
2714.21
2655.47
2574.30
2830.05
2518.88
1617.00
SR
0.00
0.00
16.37
36.15
46.69
86.00
Iron
AT
+00
+00
944.61
722.78
651.33
434.38
AS 4
+.00
+.00
8892.24
8455.51
6017.85
5687.60
SR 1
0.00
0.00
0.00
7.20
8.51
17.31
Gold
AT 4
1:00
1:00
1:00
787.37
726.35
557.08
AS 4
+000
+00
+
15747.13
15527.07
13141.60
SR 1
0.00
0.00
0.60
8.98
11.61
16.98
Diamond
AT +
1:00
+:00
1296.96
1255.06
1150.98
744.82
AS +
+.00
+00
23939.30
25101.25 .25
23019.64
16237.54
SR 1
0.00
0.00
0.00
16.31
25.02
33.27
Redstone
AT
1:00
1:00
1occ
1070.42
932.50
617.89
4
+00
+000
+
17408.
12709.99
12357.00
SR 1
0.00
0.00
9.98
15.82
19.47
28.48
Armor
4
4.00
+.00
997.59
924.60
824.53
551.30
AS +
+00
+00
17951.95
16492.96
16350.56
1 11026.00
Overall
SR 1
0.00
0.00
5.39
16.89
22.26
36.41
Experiments
3.1 Experiments Setting
Environment. To ensure realistic gameplay like human players, we employ MineRL [11] with Minecraft 1.16.5 our simulation environment. The agent operates fixed speed of 20 frames per second and only interacts with the environment via low-level action control signals of the mouse and keyboard. For more information about the detailed descriptions of the observation and action spaces, please refer to the Appendix B.
Benchmark. We constructed benchmark of 67 tasks to evaluate the Optimus-I's ability to complete long-horizoo tasks. As illustrated in Table 5, we divide the 67 Minecraft tasks into 7 groups according to recommended categories in Minecraft. Please refer Appendix D for more details.
Baseline. We compare Optimus-1 1 with various agents, including 2 GPT-4V, DEPS [51], and Jarvis-1 [52] on the challenging long-horizon tasks benchmark. In addition, we employed 10 volunteers to perform the same task on the benchmark, and their average performance served as human-level baseline. Please refer to Appendix D.2 for more details about human-level baseline. For a more comprehensive comparison, we also report Optimus- performances on the benchmark used by Voyager [47], MP5 [33], and DEPS [51] in the Appendix F.2. Note that we initialize Optimus-1 with an empty inventory, while DEPS [5]] and Jarvis-I [52] have tools their initial state. This makes it more challenging for Optimus-1 to perform the same tasks.
Battss//oppna.coom/ersscccyyggu

---

Table 2: Ablation study results. We report average Table 3: Ablation study AMEP. We report success rate (SR) on each task group. P., R., the average success rate (SR) on each task group. E. represent Planning, Reflection, Knowledge, and Zero, Suc., and Fail. represent retrieving from Experience, respectively.
AMEP without getting the case, getting the success case, and getting the failure case, respectively.
Ablation Setting
Group
Wood
Stone
Iron
Gold
Diamond
Ablation Setting
Task Group
14.29
0.00
0.00
0.00
0.00
Zero
Suc.
Wood
Stone
Gold
Diamond
42,95
25.67
0.00
0.00
0.00
92.00
79.26
36.32
4.25
3.25
55.00
47.37
18.11
2.08
V
95.00
84.29
46.98
9.36
7.89
95.00
81.10
45.47
7.50
6.39
38.33
3.49
97.49
94,26
53.33
11.54
9,59
97.49
94.26
53.33
11.34
9.59
fall into
STEVE-1
water
Ours
Abstracted MultiModal
Experience Pool
Current Goal: chop tree
drop in cave
STEVE-1
Abstracted
Ours
MultiModal
Experience
Current Goal: go fishing
Figure 4: Illustration of the role of reflection mechanism. Without the help of reflective mechanisms, STEVE-1 [25] often gets into trouble and fails complete the task. While Optimus-1, 1, with the help of the Experience-Driven Reflector, leverages the AMEP to retrieve relevant experience, reflect current situation and correct errors. This improves Optimus-1 's success rate on long-horizen tasks.
Evaluation Metrics. The agent always starts in survival mode, with an empty inventory. We conducted least 30 times for each task using different world seeds and reported the average success rate to ensure fair and thorough evaluation. Additionally, we add the average steps and average time of completing the task evaluation metrics.
3.2 Experimental Results
The overall experimental results on benchmark are shown in Table I, see accuracy for each task in Appendix F. Optimus-1 I has a success rate near 100% on the Wood Group Compared with Jarvis-1, Optimus-1 1 has 29.28% and 53.40% improvement on the Diamond Group and Redstone Group respectively. Optimus- achieves the best performance and the shortest elapsed time among all task groups. It reveals the effectiveness and efficiency our proposed Optimus-1 I framework. Moreover, compared with all baselines. Optimus-1 performance closer (average 5.37% improvement) to human levels on long-horizon task groups.
3.3 Ablation Study
We conduct extensive ablation experiments on 18 tasks, experiment setting can be found in Table 6. As shown in Table we first remove Knowledge- Driven Planner and Experience- Driven Reflector, the performance of Optimus-1 on all task groups drops dramatically. It demonstrates the necessity of Knowledge-Guidd
Planner and Experience-Drivee Reflector modules for performing long-horizon tasks. As for Hybrid Multimodal Memory, remove HDKG from Optimus-1. 1. Without the help of world knowledge, the performance of Optimus-1 decreased by an average of 20% across all task groups. We then removed AMEP, this resulted in the performance of Optimus-1 1 decreased by an

---

Success Rate
lood
Stona
stamord
22.6
52.50
6P1-de Marrery
83.07
15.2
beep Seepeeek-nn 1t, Marnoyy
Peepaeok- Nemory
Mamory
60.
51.3
43.74
610
300
4.6
3.9
3.8
4,88
Veed Group
Stane
Zren Group
Gold Group
Plannes
Group
Epoch
Epoch
Epoch
(a) Generalisation of Hybrid Multimodal Memory
(h) Sclf-Evolution
Figure 5: (a) With the help of Hybrid Multimodal Memory, various MLLM- Optimus-1 have demonstrated 2 to times performance improvement. (b) Illustration of the change Optimus-1 success rate on the unseen task over 4 epochs.
average of 12%. Finally, performed ablation experiments the way of retrieving cases from AMEP, As shown in Table 3, without retrieving cases from AMEP, the success rate shows an average of 10% decrease across all groups. It reveals that this reflection mechanism, which considers both success and failure cases, has a significant impact on the performance of Optimus-1. To illustrate the role of the reflection mechanism, we have shown some cases in Figure 4.
Generalization Ability
In this section, explore interesting issue: whether generic MILMs effectively perform various long-horiae long-horizon complex tasks in Minecraft using Hybrid Multimodal Memory. As shown in Figure 5, We employ Deepseck-VL. VL [26] and DeeemLL-XCoopossss Planner and Experience-Drrvee low performance on long-horizon tasks due the lack of knowledge and experience of Minecraft. With the assistance of Hybrid Multimodal Memory, the performance of MILMs has improved by times across various task groups, outperforming the GPT-4V baseline. This encouraging result demonstrates the generalization of the proposed Hybrid Multimodal Memory.
VL [6] as Khowledge-Guide Guided Reflector. The experimental results show that the original MLLM has
3.5 Self-Evolution via Hybrid Multimodal Memory
As shown in Section 2.3, we randomly initialize the Hybrid Multimodal Memory of Optimus-1, 1, then update it multiple times by using the "free exploration-teachet guidance" learning method. We the epoch to 4, and the number of learning tasks to 160. At each period, Optimus-1 performs free exploration 150 tasks and teacher guidance learning on the remaining 10 tasks, we then evaluate Optimus-1 1 's learning ability on the task groups same as ablation study. Experimental results are shown Figure 5, It reveals that Optimus-1 keeps getting stronger through the continuous expansion of memory during the learning process of multiple periods. Moreover, it demonstrates that MLLM with Hybrid Multimodal Memory can incarnate an expert agent in a self-evolution manner [45].
4 Related Work
4.1 Agents in Minecraft
We summarise the differences of existing Minecraft agents in the Appendix D.3. Earlier work [30, 57, 2, 3] introduced policy models for agents to perform simple tasks in Minecraft, MineCLIP [7] used text-video data train contrastive video- video-language model as a reward model for policy, while VPT [1] pre-traines preitra on unlabelled videos but lacked instruction input. Building on VPT and MineCLIP, STEVE-1 [25] added text input to generate low-level action sequences from human instructions and images. However, these agents struggle with complex tasks due to limitations in instruction comprehension and planning. Recent work [50, 47, 61] incorporated L.LMs as planning and reflection modules, but lacked visual information integration for adaptive planning. MP5

---

[33], MinaDreamer [60], and Jarvis-1 [52] enhanced situation attuttio-aware planning by obtaining textual descriptions of visual information, yet lacked detailed visual data. Optimus-1 addresses these issues by directly using observation as situation- conditions in the planning phase, enabling more rational, visually informed planning. Additionally, unlike other agents requiring multiple queries for task refinement, Optimus-1 generates a complete and effective plan in one step with the help of HDKG, This makes Optimus-1 planning more efficient.
4.2 Memory Agents
In the agent-environment interaction process, memory is key to achieving experience accumulation
[21], environment exploration [16], and knowledge abstraction [59]. There are two forms to represent
memory content in L.-M-based agents: textual form [17, 15, 31] and parametric form [5, 29, 48, 20]. In textual form, the information is explicitly retained and recalled by natural languages. In parametric form, the memory information [38] is encoded into parameters and implicitly influences the agent's actions. Recent work [49, 53, 12] has explored the long-term visual information storage [18, 19] and summarisation in MILM. Our proposed hybrid multimodal memory module is plug-and-play and can provide world knowledge and multimodal experience for Optimus-1 efficiently.
Conclusion
In this paper, propose Hybrid Multimodal Memory module, which consists of two parts: HDKG and AMEP. HDKG provides the necessary world knowledge for the planning phase of the agent, and AMEP provides the refined historical experience for the reflection phase of the agent. On top of the Hybrid Multimodal Memory, we construct the multimodal composable agent, Optimus-l, in Minecraft. Extensive experimental results show that Optimus-1 outperforms all existing agents on long-horizon tasks. Moreover, we validate that Multimodal Memory and without additional parameter updates, can exceed the powerful GPT-4V baseline. The extensive experimental results show that Optimus-1 makes a major step toward general agent with numan-like level of performance.
MLLMs, based on Hybrid
Limitation and Future Work
In the framework of Optimus-1, 1, are dedicated to leverage proposed Hierarchical Directed Knowl- edge Graph and Abstracted Multimodal Experience Pool can used to enhance the agent's ability to plan and reflect. For Action Controller, we directly introduce STEVE-1 [25] generator of low-level actions. However, limited by STEVE- I's ability to follow instructions and execute complex actions, Optimus-1 is weak in completing challenging tasks such "beat ender dragon" and "build
house". Therefore, potential future research direction is to enhance the instruction following and action generation capabilities of action controller.
In addition, most of the work, including Optimus-1, utilize multimodal large language model for planning and reflection, which then drives an action controller to perform the task. Building an end-to-end vision-language-action agent will be future work.
Acknowledgement
This study is supported by National Natural Science Foundation of China (Grant No. 62236003 and 62306090), Shenzhen College Stability Support Plan (Grant No. GXWD202208 17 17444280055, Natural Science Foundation of Guangdong Province of China (Grant No. 2024A1515010141 and Major Key Project of Peng Cheng Laboratory (Grant No. PCL2023A08).
10

---

References
[1] Bowen Baker, 1gge Akkaya, Peter Zhokov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-2444 -26644, 2022.
[2] Shaofei Cai, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Open-world multi-task con- trol through goal-aware representation learning and adaptive horizon prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13734-13744, 2023.
[3] Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Groot: Learning to follow instructions by watching gameplay videos. In The Twelfth International Conference Learning Representations, 2023.
[4] Gongwei Chen, Leyang Shen, Rui Shao, Xiang Deng, and Liqiang Nie. Lion: Empowering multimodal large language model with dual-level visual knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[5] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 6491-6506, 2021.
[6] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, al. xnoopse-xcomppoer2: Mastering free-form text-image composition and comprehension in langugee large model. arXiv preprint arXiv:2401.16420, 2024.
[7] Linxi Fan, Guanzhi Wang. Yunfan Jiang, Ajay Mandlekar, Yuncong Yang. Haoyi Zhu, Andrew Tang, De-An Huang. Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. Advances in Neural Information Processing
Systems, 35:18343-18362, 2022.
[8] Mariel K Goddu and Alison Gopnik. The development of human causal learning and reasoning. Nature Reviews Psychology, pages 1-21, 2024.
[9] Maitrey Gramopadhye and Daniel Szafir. Generating executable action plans with environmennaally-wwaan language models. In 2023 IEEE/RSJ International Conference Intelligett Robots and Systems (IROS), pages 3568-3575. IEEE, 2023.
[10] Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint wwxiiv2307.128555 2023.
[11] William H Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. Minerl: A large-scale dataset of minecraft demonstrations. arXiv preprint arXiv: 1907. 13400, 2019.
[12] Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: Memory-augmentee large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.05726. 2024.
[13] Jiangyong Huang, Silong Yong, Xiaojian Ma. Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, and Siyuan Huang. An embodied generalist agent in 3d world. arXiv preprint arXiv:2311.12871 2023.
[14] Wenlong Huang, Picter Abbecl, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147. PMLR, 2022.
[15] Ziheng Huang, Sebastian Gutierrez, Hemanth Kamana, and Stephen MacNeil. Memory sand- box: Transparent and interactive memory management for conversational agents. In Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Sofware and Technology. pages 1-3, 2023.
11

---

[16] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. In NeurlPS 2022 Foundation Models for Decision Making Workshop, 2022.
[17] Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng. Lianmin Zheng, Joseph Gonzalez, lon Stoica, Xuezhe Ma, and Hao Zhang. How long can context length of open-source Ilms truly promise? In NeurlPS 2023 Workshop on Instruction Tuning and Instruction Following, 2023.
[18] Xiaojie Li, Shaowei He, Jianlong Wu, Yue Yu, Liqiang Nie, and Min Zhang. Mask again: Masked knowledge distillation for masked video modeling. In Proceedings of the ACM International Conference Multimedia, page 2221-22231 ACM, 2023.
[19] Xiaojie Li. Jianlong Wu, Shaowei He, Shuo Kang, Yue Yu. Liqiang Nie, and Min Zhang, Fine- grained key-value memory enhanced predictor for video representation learning. In Proceedings of the ACM International Conference on Multimedia,, page 2264-22117 ACM, 2023.
[20] Xiaojie Li, Yibo Yang, Xiangtai Li, Jianlong Wu, Yue Yu, Bernard Ghanem, and Min Zhang. Genview: Enhancing view quality with pretrained generative model for self-supervised learning. In Proceedings of the European Conference on Computer Vision. Springer, 2024.
[21] Xiaojie Li, Yibo Yang, Jianlong Wu, Bernard Ghanem, Liqiang Nie, and Min Zhang. Mamba- fscil: Dynamic adaptation with selective state space model for few-shot incremmnatt learning, arXiv preprint arXhv:2407.06136 2024.
[22] Zaijing Li, Fengxiao Tang, Ming Zhao, and Yusen Zhu. Emocaps: Emotion capsule based model for conversational emotion recognition. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1610-1618, 2022.
[23] Zaijing Li, Ting-En Lin, Yuchuan Wu, Meng Liu, Fengxiao Tang. Ming Zhao, and Yongbin Li. Unisa: Unified generative framework for sentiment analysis. In Proceedings of the 31st ACM International Conference on Multimedia, pages 6132-6142, 2023.
[24] Zaijing Li, Gongwei Chen, Rui Shao, Dongmei Jiang, and Liqiang Nie. Enhancing the emotional generation capability large language models via emotional chain-of-thought. arXiv preprint arXiv:2401.06836 2024.
[25] Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila Mcllraith. Steve-1: A generative model for text-to-belt -et-vviob in minecraft. Advances in Neural Information Processing Systems, 2023.
[26] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepseek-vi. towards real-world vision- vision-language understanding. arXiv preprini 24i:22403005525 2024.
[27] Qi Lv, Xiang Deng, Gongwei Chen, Michael Y Wang, and Liqiang Nie. Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl. NeurIPS, 2024.
[28] Simon Makin. The amyloid hypothesis on trial. Nature, 559(7715):84 2018.
[29] Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. Fast model editing at scale. In International Conference on Learning Representations, 2021.
[30] Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generaliza- tion with multi-task deep reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning, pages
PMLR, 2017.
[31] Arka Pal, Deep Karkhanis, Manley Roberts, Samuel Dooley, Arvind Sundararajan, and Sid- dartha Naidu. Giraffe: Adventures in expanding context lengths in llms. arXiv preprint a30iv:2308. 10882, 2023.
[32] Eileen Parkes. Scientific progress is built on failure. Nature, 10, 2019.
12

---

[33] Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, and Jing Shao. Mp5: A multi-modda open-ended embodied system minecraft via active perception. arXiv preprini arXiv:2312.07472, 2023.
[34] Maria Abi Raad, Arun Ahuja, Catarina Barros, Frederic Besse, Andrew Bolt, Adrian Bolton, Bethanie Brownfield, Gavin Buttimore, Max Cant, Sarah Chakera, et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404. 10179, 2024.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference machine learning, pages
PMLR, 2021.
[36] Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C Yuen. Multi-adversarial discriminative deep domain generalization for face presentation attack detection. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10023-10031, 2019.
[37] Rui Shao, Tianxing Wu, and Ziwei Liu. Detecting and grounding multi-modal media manipula- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6904-6913, 2023.
[38] Rui Shao, Tianxing Wu, Jianlong Wu, Liqiang Nie, and Ziwei Liu. Detecting and grounding mudti-modal media manipulation and beyond. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[39] Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models. In NeurIPS, 2024.
[40] Shan H Siddiqi, Konrad Kording, Josef Parvizi, and Michael Fox. Causal mapping of human brain function. Nature reviews neuroscience, pages 361-375, 2022.
[41] JF Stein. Role of the cerebellum in the visual guidance of movement. Nature, pages 217-221, 1986.
[42] Tim Stuart, Andrew Butler, Paul Hoffman, Christoph Hafemeister, Efthymia Papalexi, William M Mauck, Yuhan Hao, Marlon Stoeckius, Peter Smibert, and Rahul Satija. Com- prehensive integration of single-cell data. cell, 177(7):1888-194)) 2019.
[43] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang. Qiying Yu, Zhengxiong Luo, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang. et Generative multimodal models are in-context learners. arXiv preprint arXiv:2312. 1.t.:2262..1.66. 2023.
[44] Weihao Tan, Ziluo Ding. Wentao Zhang, Boyu Li, Bohan Zhou, Junpeng Yue, Haochong Xia, Jiechuan Jiang, Longtao Zheng, Xinrun Xu, et al. Towards general computer control: A multimodal agent for red dead redemption ii as a case study. arXiv preprint arXiv:2403.03186, 2024.
[45] Zhengwei Tao, Ting-En En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou A survey on self-evolution of large language models. arXiv preprint arXiv:2404. 14387, 2024.
[46] Deniz Vatansever, Jonathan Smallwood, and Elizabeth Jefferies. Varying demands for cognitive control reveals shared neural processes supporting semantic and episodic memory retrieval. Nature communications, 12(1):2134, 2021.
[47] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305. 16291, 2023.
[48] Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng. Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, and Huajun Chen. Detoxifying large language models via knowledge editing. arXiv preprint 24Xiv:2403.1444 2024.
13

---

[49] Shihao Wang, Zhiding Yu, Xiaohui Jiang, Shiyi Lan, Min Shi, Nadine Chang, Jan Kautz, Ying Li, and Jose M Alvarez. Omnidrive: A holistic Ilm-agent framework for autonomous driving with 3d perception, reasoning and planning. arXiv preprint arXiv:2405.0111
2024.
[50] Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma. and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint 2302.0..00000166 2023.
[51] Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. arXiv preprint arXiv:2302.01 01 56000, 2023.
[52] Zihao Wang. Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-l: Open- woen-world multi-task agents with memory-augmented multimodal language models. arXiv preprint 1xXiv:2311.05997, 2023.
[53] Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang- Longvlm: Efficient long video understanding via large language models. arXiv preprint arXiv:2404. 033844 2024.
[54] Xu Yang, Yongliang Wu, Mingzhuo Yang, Haokun Chen, and Xin Geng. Exploring diverse in-context configurations for image captioning. Advances in Neural Information Processing Systems, 36, 2024.
[55] Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. arXiv preprint (txxx::33/2.177// 2023.
[56] Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: Enhancing multimodal large language model answer questions in dynamic audio-visual scenarios. In ECCV, 2024.
[57] Haoqi Yuan, Chi Zhang, Hongcheng Wang, Feiyang Xie, Penglin Cai, Hao Dong, and Zongqing Lu. Skill reinforcement learning and planning for open-world long-horizon tasks, arXiv preprint 2rX1v:2303.16563, 2023.
[58] Haoyu Zhang, Meng Liu, Zixin Liu, Xuemeng Song, Yaowei Wang. and Liqiang Nie. Multi- factor adaptive vision selection for egocentric video question answering. In Proceedings of the 41 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 59310-59328, PMLR, 2024.
[59] Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. A survey on the memory mechanism of large language model based agents. arXiv preprint arXiv:2404.135000 2024.
[60] Enshen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao. Minedreamer: Learning to follow instructions via chain-of-imaginatiio for wirulated-world control. arXiv preprint arXiv:2403. 120337 2024.
[61] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, et al. Ghost in the minecraft: Generally capable agents for open- world enviroments via large language models with text- knowledge and memory. arXiv preprint arXiv:2305.0 /7144, 2023.
14

---

A Broader Impact
With the increasing capability level of Multimodal Large Language Models (MLLM) comes many potential benefits and also risks. On the positive side, we anticipate that the techniques that used to create Optimus-1 could be applied to the creation of helpful agents in robotics, video games, and the web. This plug-and-play architecture that we have created can quickly adapted to different MILMs,, and the proposed methods also provide a viable solution for other application areas in the agent domain. However, on the negative side, it is imperative to acknowledge the inherent stochastic nature of MLLMs in text generation. If not addressed carefully, this could lead to devastating consequences for society. Prior to deploying MLLMs in conjunction with the Hybrid Multimodal Memory methodology, comprehensive assessment of their potential risks must be undertaken. We hope that while the stakes are low, works such as ours can improve access to safety research on instruction foolowiig models in multimodal agents domains.
B Minecraft
Minecraft is an extremely popular sandbox video game developed by Mojang Studios 3. It allows players to explore a blockly, procedurally generated 3D world with infinite terrain, discover and extract raw materials, craft tools and items, and build structures or earthworks (shown in Figure 6). Minecraft is a valuable and representative environment for evaluating long-horizon tasks, offering greater diversity and complexity compared to other environments. Unlike web/app navigation [55] and embodied manipulation [16], Minecraft is an open world with a complex and dynamic environment (79 biomes, including ocean, plains, forest, desert, etc.). To complete long-horizon tasks, agents must achieve multiple sub-goals (c.g., 15 sub-goals to craft a diamond sword). making the construction of a Minecraft agent quite challenging. Many studies [47, 33, 52] have chosen Minecraft as the environment for validating performance on long-horizon tasks. Extensive experimental results in the paper show that Optimus-1 outperforms all baselines. Therefore, we chose Minecraft open-world environment to evaluate the ability of agents to perform monghhoizoon tasks.
B.1
Basic Rules
Biomes. The Minecraft world is divided into different areas called "biomes". Different biomes contain different blocks and plants and change how the land is shaped. There are 79 biomes in Minecraft 1.16.5, including ocean, plains, forest, desert, etc. Diverse environments have high requirements for the generalization of agents.
Time. Time passes within this world, and a game day lasts for 20 real-world minutes. Nighttime is much more dangerous than daytime: the game starts at dawn, and agents have 10 minutes of game time before nightfall. Hostile or neutral mobs spawn when night falls, and most of these mobs are dangerous, trying to attack agents. How to survive in such a dangerous world is an open problem for Minecraft agents research.
Item, In Minecraft 1. 1.55, there are 975 items can be obtained, such as wooden pickaxe >, iron sword x. Item can be obtained by crafting or destroying blocks or attacking entities. For example, agent can attack cows @ to obtain leather X and beef Agent also can use 1 stick / and 2 diamonds to craft diamond sword x.
Gameplay progress. Progression primarily involves discovering and utilizing various materials and resources, each of which unlocks new capabilities and options. For instance, crafting wooden pickaxe A enables the player to mine stone a which can be used to create a stone pickaxe > and a furnace * these, in turn, allow for the mining and smelting of iron ore a Subsequently, an iron pickaxe > permits the extraction of diamonds @ and a diamond pickaxe > can mine virtually any block in the game. Similarly, cultivating different crops allows for the breeding of various animals, each providing distinct resources beyond mere sustenance. Enemy drops also have specific applications, with some being more beneficial than others. By integrating resources from mining, farming, and breeding, players can enchant their equipment. The collection and crafting of materials also facilitate construction, enabling players build diverse structures. Beyond practical
15

---

(a)
(b)
Dnvanbory
(c)
(d)
Figure 6: The screenshots in Minecraft. (a). The world has different complex terrains, including plains, river, forest and mine. (b). The agent can use crafting table to craft tools and items with recipes. (c). The agent can use the furnace to smelt ore to obtain precious ingot. (d). The agent can grow wheat near the river.
considerations such secure bases and farms, the creative aspect of building personalized structures constitutes significant part of the Minecraft experience.
Freedom. In Minecraft, player do anything they imagine. Player can craft tools, smelt ore, brew potions, trade with villagers and wandering traders, attack mobs, grow crops, raise animals in captivity, etc. Player even infinite possibilities.
redstone to build computer. This is world of freedom and
More Challenge than Diamond @ Progression beyond the Overworld is fairly limited: Eventually, you can build a nether portal to reach the Nether, where you can get materials for more complex crafting, the resources brew potions, and the top tier of tools and armor. The Nether materials also let you reach the End dimension, where you must defeat the Ender Dragon to unlock the outer End Islands, where you get an elytra that lets you fly, and shulker boxes for more storage.
B.2 Observation and Action Spaces
Observation. Our observation space completely consistent with human players. The agent only receives an RGB image with dimensions of 640 360 during the gameplay process, including the hotbar, health indicators. food saturation, and animations of the player's 's hands. is worth helping the agent more clearly in extremely dark environments, we have added night vision effect for the agent, which increases the brightness of the environment during the night.
Action Spaces. Our action space is almost similar to human players, except for craft and smelt actions. It consists of two parts: the mouse and the keyboard. The keypresses are responsible for controlling the movement of agents, such as jumping, forward, back, etc. The mouse movements are responsible for controlling the perspective agents and the cursor movements when the GUI opened. The left and right buttons of the mouse are responsible for attacking and using placing items. In Minecraft, precise mouse movements are important when completing complex tasks that need open inventory crafting table. In order to achieve both the same action space with MineDojo
16

---

Table Our action space.
Index
Action
Human Action
Description
Forward
key W
Move forward.
Back
key S
Move back.
Left
key A
Strafe left.
Right
key D
Strafe right.
Jump
key Space
Jump. When swimming, keeps player afloat.
Sneak
key left Shift
Slowly move in
direction of movement.
Sprint
key left Ctrl
Move quickly the direction of current movement.
Attack
left Button
Destroy blocks (hold down); Attack entity (click once).
Use
right Button
Place blocks, entity, open items other interact actions defined by game.
[1-9]
keys
Selects appropriate hothur item.
Open/Close Inventory
key E
Opens the Inventory. Close any open GUI.
12
Yaw
move Mouse X
Tuming: aiming: camera movement novemment.Rangingg from -180 to +180.
13
Pitch
move Mouse Y
Tuming; aiming; camera movement Ranging from 180 to +180. Execute crafting recipe obtain
Craft
Smelt
Execute smelting recipe to obtain new item.
[7]. we abstract the craft and the smelt action into action space. The detailed action space is described Table 4.
B.3 Long-horizon Tasks
Long-horizon Tasks are complex tasks that require world knowledge to solve and consist of multiple indispensable subtask sequences. In Minecraft, technology has six levels, including wood stone iron @ golden @ diamond @ and netherite * Wooden tools can mine stone-level blocks, but can't mine iron-level and upper-level blocks. Stone tools can mine iron-level blocks, but can't mine diamond-level and upper-level blocks. Iron-level tools can mine diamond-leve blocks, but can't 't mine netherite-level blocks. Diamond-leve tools can mine any level blocks.
For example, the agent now wants to complete the task "Craft iron sword x The agent needs to craft wood-level tools mine craft tools, the agent needs crafting table a To smelt iron ore into iron ingot @ the agent needs furnace Moreover, craft crafting table needs 4 planks, and craft furnace needs cobblestone. In summary, the agent needs to obtain many raw materials, wood-level and stone-level tools, 1 crafting table, 1 furnace, and most importantly, iron ingots. The process of this task is shown in Figure 7.
and craft stone-level tools mine iron ore In order to
C Theory
In this section, we briefly introduce the relevant theory of cognitive science. For more details, please refer the original articles.
Our ability understand and predict the world around us depends on our long-term memory stores, which have historically been divided into two distinct systems [28, 42, 46]. The semantic memory system provides a conceptual framework for describing the similar meanings of words and objects as they are encountered in different contexts (e.g., bee is a flying insect with yellow and black stripes that produces honey), whereas the episodic memory system records our personal experiences characterized by the co-occurrence of words and objects at different times and places (c.g., being stung by a bee while cating honey at a picnic last weekend). These information stores and the interactions between them play crucial role in guiding our behaviour and giving us the flexibility to adapt to the various demands of our environment.
In this paper, inspired by the above theory, we divide the agent memory module into two parts: knowledge and experience. Based on this, we propose Hierarchical Directed Knowledge Graph and Abstracted Multimodal Experience Pool to enable the agent to acquire, store, and utilize knowledge and experience during the execution of tasks. Extensive experimental results demonstrate the effectiveness of the proposed methodology.
17

---

Chop logs
(b) Craft planks
Craft sticks
Craft crafting table
(c) Craft 1 wooden pickaxe
(f) Mine cobblestone
BS
(g) Craft furnace
(h) Craft stone pickaxe
Dig down deeper to find iron ore
(j) Mine iron ores
(k) Smelt iron ingots
(I) Craft iron sword
Figure 7: Processing of task "Craft 1 iron sword". Optimus-1 needs thousands of steps to complete this task. To craft and smelt precisely, the mouse movements action can't 't have any error.
D Benchmark Suite
D.1 Benchmark
We constructed a benchmark 67 tasks evaluate Optimus- I's ability complete long- horizon tasks in Minecraft. According to recommended categories in Minecraft, we have classified these tasks into 7 groups: Wood C Stone C Iron @ Gold <, Diamond @ Redstone Armor t. The statistics for henchmark are shown in Table 5. Due to the varying complexity of these tasks, we adopt different maximum gameplay steps (Max. Steps) for each task. The maximum steps are determined by the average steps that human players need to complete the task. Due to the randomness of Minecraft, the world and initial spawn point of the agent could vary a lot. In our benchmark setting, We initialize the agent with empty inventory, which makes it necessary for the agent to complete series of sub-goals (mining materials, crafting tools) in order to perform any tasks. This makes every task challenging, even for human players.
Note that Diamonds are very rare item that only spawns in levels chance of spawning in Minecraft 1.16.5. Diamonds are usually found near level 9, or in man-made or natural mines no higher than level 16. To mitigate the significant impact of diamond generation probability the agent's 's likelihood of successfully completing the task, have adjusted the
and have 0.0846%
18

---

Table Setting of 7 groups encompassing 67 Minecraft long-horizon tasks.
Group
Task Num.
Example Task
Max. Steps
Initial Inventory
Avg. Sub-goal Num.
Wooden
10
Craft wooden axe
3600
Emply
Stone
Craft one stone pickaxe
7200
Empty
Iron
Cruft iron pickaxe
12000
Empty
Golden
Mine gold and smelt into golden ingot
36000
Empty
Redstone
Craft piston
36000
Empty
17
Diamond
Dig down and mine diamond
36000
Emply
15
Armor
13
Craft one iron helmet
36000
Empty
16
Table 6:
We evaluate Optimus-1 on these tasks in ablation study which are the subset of our
benchmark.
Group
Task
Sub- Lub-Goal Num.
Max. Step
Initial Inventory
Craft wooden axe
5
3600
Empty
Wooden
Craft crafting table
3
3600
Empty
Craft stone pickaxe
10
7200
Empty
Stone
Craft a stone axe
10
7200
Empty
Craft furnace
9
7200
Empty
Craft iron pickaxe
13
12000
Empty
Craft a bucket
13
12000
Empty
Iron
Craft a rail
13
12000
Empty
Craft iron sword
12
12000
Empty
Craft shears
12
12000
Empty
Craft golden pickaxe
16
36000
Empty
Golden
Craft a golden axe
16
36000
Empty
Smelt golden ingot
15
36000
Empty
Craft diamond pickaxe
15
36000
Empty
Craft diamond
16
36000
Empty
Diamond
Craft a diamond hoe
15
36000
Empty
Craft diamond sword
15
36000
Empty
Dig down and mine a diamond
15
36000
Empty
diamond generation probability to 20%, spawns levels 2 to 16. This setting applies to human players as well.
In the ablation study, we select the subset of our benchmark as the evaluation set (shown Table 6). The environment setting is the same as the benchmark.
D.2 Baselines
Existing Baseline. On the one hand, we employ GPT-3.5 and GPT-4V as baseline, which are evaluated without integrating hybrid multimodal memory modules. During the planning phase, they generate plan for the action controller based on task prompt (and observation). During the reflection phase, they generate reflection results in zero-shot manner. On the other hand, we compare existing SOTA Agents [51, 52] in Minecraft.
Human-level Baseline. To better demonstrate agent's performance level in Minecraft, we hired 10 volunteers play the game as a human-leve baseline. The volunteers played the game with the same environment and settings, and every volunteer asked to perform the each task on the benchmark 10 times. Ultimately, we used the average scores results of the human-level baseline are shown in Table 1, To ensure the validity of the experiment, we ensured that each volunteer had at least 20 hours Minecraft gameplay before conducting the experiment. For each volunteer, we pay $25 as reward.
volunteers as the human-level baseline. The
19

---

Table 7: Statistics for various Minecraft agents.
Agent
Pub.
Env.
Input
Output
Planning Reflection
Knowledge
Experience
VPT [1]
NeurPPS
MineRL
low-level action
MineDOJO [7]
NeurIPS' 22
MineDOJO
T+V
low-level action
STEVE-1 [25]
NeurlPS' 23
MineRL
T+V
low-level action
Voyager [47]
NeurIPS 23
Mineflayer
T+V
code
DEPS [51]
NeurIPS'
MineDOJO
T+V
code
GROOT [3]
ICLR' 24
MineRL
T+V
low-level action
MP5 [33]
CVPR' 24
MineDOJO
T+V
code
Jarvis-1 [52]
MineRL
T+V
low-level action
Optimus-1
MineRL
T4V
low-level action
D.3 Minecraft Agents
In this section, we summarise the differences between existing Minecraft agents. As shown in the Table earlier work [1, 7, 25, 3] constructed Transformer-base policy network as agent. Recent work [47, 51, 52, 33] introduces the Multimodal Large Language Model, which empowers the agent to complete long-hone long-horizon tasks by exploiting the powerful language comprehension and planning capabilities of L.M.
In the Mineflayer and Minedojo environments, agents [7, 51, 33] can accomplish sub-goals by calling APIs (in the form of codes), which is a different behavioral pattern from humans. In MineRL [11], agents [1, 25, 3, 52] must generate low-level actions perform tasks, which is more challenging to accomplish long-horizon tasks.
Moreover, existing agents lack knowledge and experience, and their performance in Minecraft is still vastly gapped from the human level. In this paper, we introduce Hybrid Multimodal Memory, which empowers Optimus-1 with hierarchical knowledge and multimodal experience. This makes Optimus-1 I significantly outperform all existing agents on challenging long-horizon tasks benchmark, and exhibits near human-level performance many tasks.
E
Implementation Details
E.1
Hybrid Multimodal Memory
E.1.1 Abstracted Multimodal Experience Pool
Relevant studies [5, 29, 17, 15] have demonstrated the importance of memory for agents complete long-horizon tasks. To implement the memory mechanism, Minedojo [7] and Voyager [47] only considered unimodal storage of historical information. Jarvis-1 [52] considered multimodal memory mechanism to store task planning and visual information as experience, but it stores all historical information without summarisation. This approach stores all visual images, which poses huge challenge in storage size and retrieval efficiency. To solve the problem, we propose the Abstracted Multimodal Experience Pool structure, which summarizes all historical information during the agent's s execution of the task, which maintains the integrity of long sequential information and greatly improves the storage and retrieval efficiency of the experience.
As shown in Figure 2, we first input the visual image stream to the video buffer, which filters the image stream Empirically, we set the frequency of filtering to second/frame, meaning that the video buffer takes one frame per second from the original image stream to compose the filtered image stream. We found that above this frequency makes the visual information redundant (too much similarity between images), and below this frequency does not preserve enough complete visual information.
fixed frequency. It makes the length of the image stream substantially shorter.
Then, we feed the filtered frames into an image buffer with a window size of 16. We dynamically compute the similarity between images in the image buffer, when new image comes in, we compute the similarity between the new image and the most recent image, and then we remove the image with the highest similarity in order keep the image buffer's window size to 16.
Subsequently, we introduce MineCLIP [7], pre-trained model of video-text alignment with structure similar to CLIP [35], our visual summariser. For given sub-goal, it calculates the
20

---

correlation between the visual content within the current memory bank and the sub sub-goal, and when this correlation exceeds pre-set threshold, the frames within the memory bank are saved as the visual memories corresponding to that sub-goal. Finally, we store the visual memories with the sub goal's textual description into the Abstracted Multimodal Experience Pool. In addition, we incorporate the environment information, agent initial state, plan from Knowledge-Guice Guided Planner, etc. into the experience memory of the given task. In this way, we consider the history information of each sub-goal and summaries and summarise it to finally compose the multimodal experience of the given task.
Note that we also store these visual memories as failure cases when the feedback from the reflection phase is REPLAN. Therefore, when Optimus-1 executes a long-horizon task, it can retrieve past successes and failures as references and update memory after the task is finished. In the reflection phase, Optimus-1 1 retrieve the most relevant cases from Abstracted Multimodal Experience Pool, which contains the three scenarios COMPLETE, CONTINUE, and REPLAN, to help the agent better assess which state the current situation belongs to. This approach of considering both successful and failed cases for in-context learning is inspired by related research [8, 32], and its effectiveness is validated in Section 3.3.
E.1.2 Hierarchical Directed Knowledge Graph
As shown in the Figure 2. crafting a diamond sword x requires two diamonds + and a wooden stick /, while mining diamonds requires an iron pickaxe 7, which turn requires additional raw materials and crafting steps. We transform this mine and craft knowledge into a graph structure, where the nodes of the graph are objects, and the nodes point to objects that can be crafted or completed by that object. With directed graph, we show that connections between objects are established, and that this knowledge can be stored and updated efficiently. For given object, we only need to retrieve the corresponding node to extract the corresponding subgraph from the knowledge graph. Then by topological sorting, we can get the antecedents and required materials for the object, and this information is provided to the Knowledge-Guiee Guided Planner as way to generate more reasonable sequence of sub-goals. With Hicrarchical Directed Knowledge Graph, we can significantly enhance the world knowledge of the agent in train-free manner, as shown in the experimental results in Section 3.3.
Our HDKG can be efficiently updated and expanded. When adding new nodes, the HDKG can be updated by simply merging the nodes and relationships into the graph. This method involves local linear modifications to the graph rather than altering the entire graph, making the process efficient and time-saving. For example, when M new nodes and edges are added, the HDKG can be updated with M+N times of operations. Moreover, an HDKG containing 851 objects (nodes) requires less than MB of memory. Thus, the HDKG can be efficiently updated and maintained.
E.2 Hybrid Multimodal Memory Driven Optimus-1
In order to implement the proposed Hybrid Multimodal Memory and to progressively increase the capacity of Optimus-1 in self-evolution manner, we propose non-parametric learning method named "free exploration-teacee
guidance".
In the free exploration phase, we randomly initialize the environment, materials, and tasks. For the task "craft a wooden pickaxe", we provide initial materials (three planks, two sticks), and then Optimus-1 (only the action controller activated) attempts to complete the task. If the environment feedback indicates the task is successful, the knowledge (3 planks, 2 sticks - wooden pickaxe) is added to the HDKG, Note that we randomly initialize materials and their quantities, which means that the task may not always succeed. As a result, each free exploration may not acquire the corresponding knowledge, but it can record the relevant experience (whether successful or fail). In the free exploration phase, Optimus-1 1 learns simple atomic operations, such as crafting sticks in the Wooden Group and mining diamonds in the Diamond Group.
In the teacher guidance phase, Optimus-1 need to learn a small number of long-horizon tasks based on extra knowledge. For example, during the free exploration phase, Optimus-1 mastered crafting stick / and mining diamond @ but did not know that "a diamond sword X is obtained by stick / and two diamonds
So provide some task plans, which will serve extra knowledge to guide
21

---

Optimus-1 I to complete the task of "craft diamond sword". We built the following automated process get the task plan needed for "free exploration".
We randomly select tasks for each Group groups in total) that are not included in the benchmark.
For each selected task, we use script to automatically obtain the crafting relationships from the Minecraft Wiki 4. Taking the task "craft wooden sword" example, script to automatically obtain the crafting relationships: I wooden stick, 2 planks, 1 crafting table + I wooden sword, log - planks, planks - sticks, 4 planks + crafting table.
the
These relationships are converted into a directed acyclic graph through an automated script. By performing topological sort, the graph can be converted into tuples of materials and their quantities: (wooden sword, 1), (crafting table, 1), (wooden stick, 1) (planks, 8), (log, 2).
We prompt GPT-4 construct plan in order from basic materials to advanced materials. Finally, we get the plan: 1. Get two logs 2. Craft eight planks 3. Craft crafting table 4. Craft a wooden stick Craft a wooden sword
During the teacher guidance phase, Optimus-1 's memory is further expanded and it gains the experience of executing complete acquire advanced knowledge and learn multimodal experiences through complete long-horizon tasks.
tasks. Teacher guidance phase allows Optimus- I to
E.3 Backbone of Optimus-1
Optimus-1 consists of Knowledge-Guide Guided Planner, Driveen Reflector, and Action Con- troller. In this paper, we employ OpenAl's GPT-4V as Knowledge-Guide Guided Planner and Experience- Driven Reflector, and STEVE-1 [25] as Action Controller. We also employ open-source models like Deepseek-VI. VI. [26] and InternI InternlMM-CCopossee XCComposer2-V [6] as Knowledge-Ginn Guided Planner and Drieree Reflector.
All experiments to perform different tasks at the same time, and this parallelized inference greatly improves our experimental efficiency. In the free exploration and teacher guidance phases, there is no need to access OpenAl's API, and the learning process takes approximately 16 hours on 4x A 100 80G GPUs. During the inference phase, takes about 20 hours on 4x A100 80G GPUs.
implemented 4x NVIDIA A100 GPUs. We employ multiple Optimus-1
Throughout the experiment, we spent about $5, 000 to access the GPT-4V APL. However, we also offer more cost-effective solutions. As shown in Figure 5, if we employ Deepseek-VE VL [26] or InternL M--XComposer2-VL. [6] as Optimus-13s I's backbone, we can get comparable performance with low-cost!
E.4 Prompt for Optimus-1
We show the prompt templates for Experience- Drivven Reflector and Action Controller as follows.
System: You are a MineCraft game expert and you can guide agents to complete complex tasks. User: For a given game screen and task, you need to complete <geal inference> and <visual inference>.
<goal inference>: According the materials required complete the task.
you need infer the weapons, equipment,
<visual inference>: According to the game screen, you need infer the following aspects: health bar, food bar, hotbar, environment.
I will give you example as follow: [Example]
<task>: craft a stone sword. <goal inference>: stone suord <visual inference>
health bar: full
food full
'ntps://minnerrat.wiik
22

---

hotbar: empty
environment: forest
Here is a game screen and task, you MUST output in example format.
<task>: {task).
<game screen>: (image)
Assistant:
User: Now you need to make plan with the help of <yisusl info? and <craft graph>. <visual info>: Consists of the following aspects: health bar, food bar, hotbar, environment.
Based the current visual information, you need to consider whether prequel steps needed to ensure that agent can complete the task.
<craft graph>: top-domn list of all the tools and materials needed to complete the taak, sill give you an example of planning under specific visual conditions as follou: [Exemple]
(example)
Here game screen task, you output example Remember <task planning?
MUST output example format.
<task>: {task)
<game screen?: (image}
<craft graph>: (graph)
Assistant:
Listing 1: Prompt for Sewwledge-Guider Planner.
System: You are a MineCraft game expert and you can guide agents to complete complex tasks. Agent executing the task: (taak). Given images about agent's environment (forest, cave, ocean, etc.,) which the agent located, then determine whether the agent's current situation is done, continue, replan.
before executing
and
state, you
should first detection
<done>:
Comparing the image before the task was performed, the current image reveals that the task is complete.
<continue>: Current image reveals that the task is NOT complete, but agent is in good state ( good health, not hungry) with high likelihood to complete task.
<replan>: Current image reveals that the task is XOT complete, and agent in state (bad
health, or hungry) or situation (in danger, or in trouble), need for replanning. For replan, you need further determine whether "in_vater", "drop_down"
agent's predicament is dowm" that the agent has fallen into cave mountain river, while "in_vater" means that the agent is in the ocean and needs to return to land immediately.
trapped
User: P'll give you some examples to illustrate the different situations. Each example consista of two images, shere the first image is the state of the agent before performing the task and the second image is the current state of the agent.
[Examples]
<don>>: (imagel), [image2]
<continue>: {imagel), (angeee,,mmmmeeee (image2) <replan>: (imagel), [image2)
Now given two images about agent's state before executing the task and its current state, you MUST and ONLY output in following format:
Enviroment: <environment>
Situation: keituation>
(if situation is replan) Predicament: <predicament>
Listing 2: Prompt for Experience- Driven Reflector.
F Additional Experimental Results
F.1
Full Results on Our Benchmark
We list the results of each task on the benchmark below, with details including task name, sub-goal numbers, success rate (SR), average number of steps (AS), average time (AT), and eval times. All tasks are evaluated in Minecraft 1.16.5 Survival Mode. Note that each time Optimus-1 performs task, we initial it with an empty initial inventory and a random start point, This makes it challenging for Optimus-1 I to perform each task.
23

---

Table The results of Optimus-1 on various tasks in the Wood group. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
Task
Sub-Goal Num.
SR
AS
AT(s)
Eval Times
Craft a wooden shovel
95.00
995.58
49.78
40
Craft wooden pickaxe
5
100.00
1153.91
57.70
30
Craft a wooden axe
5
96.67
1010.28
50.51
30
Craft wooden hoe
5
100.00
1042.80
52.14
30
Craft a stick
4
97.14
372.97
18.65
70
Craft crafting table
3
98.55
448.63
22.43
69
Craft wooden sword
5
100.00
1214.90
60.74
30
Craft a chest
4
100.00
573.80
28.69
30
Craft a bowl
4
100.00
744.30
37.21
30
Craft a ladder
100.00
820.30
41.02
30
Table 9: The results Optimus-1 on various tasks in the Stone group. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
Task
Sub-Goal Num.
SR
AS
AT(s)
Eval Times
Craft stone shovel
8
90.32
2221.00
111.05
31
Craft a stone pickaxe
10
96.77
2310.09
115.50
31
Craft a stone axe
10
96.88
2112.59
105.63
32
Craft stone hoe
8
94.64
2684.60
134.23
56
Craft a charcoal
9
88.57
3083.35
154.17
35
Craft a smoker
9
90.24
3118.89
155.94
41
Craft stone sword
8
94.29
2067.92
103.40
35
Craft a furnace
9
93.75
2842.71
142.14
32
Craft a torch
8
85.71
2109.00
105.45
95
Moreover, in MineRL [11] environment, 'steps' refers the number of interactions between the agent and the environment, occurring at a frequency 20 times per second. For example, an agent takes 2 seconds to complete the task "chop a tree", it interacts with the environment 40 times, resulting recorded steps number of 40. Experimental results show that Optimus-1 11's 's average task completion step (AS) is significantly lower than other baselines.
F.2 Results on Other Benchmark
For more comprehensive comparison with current Minecraft Agents, we also report Optimus- I's performances the benchmark used by Voyager [47], MP5 [33], and DEPS [51] below. Due to the different environments and settings, agents perform tasks with varying degrees difficulty. For
example, Optimus-1 I requires low-level action to perform any task in MineRL [11], and initialize its inventory to be empty. While Voyager [47] performs tasks in Mineflayer environment only
through encapsulated code, MP5 [33] performs tasks in MineDOJO [7] environment only needs specific control signal to craft tools, no low-level actions (mouse movement and click) are needed.
Optimus-1' 's success rate in completing tasks with these baselines is shown in the Table 15 and Table 16, and Optimus-1 's efficiency in unlocking the tech tree Minecraft is shown in the Figure These results reveal that Optimus-1 outperforms variety of powerful baseline agents, even in challenging environmental settings!
24

---

Table 10: The results of Optimus-1 I on various tasks in the Iron group. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
Task
Sub Goal Num.
SR
AS
AT(s)
Eval Times
Craft an iron shovel
13
54.79
5677.35
637.81
73
Craft an iron pickaxe
13
59.42
6157.39
591.81
69
Craft an iron
13
54.29
6026.26
676.97
70
Craft an iron hoe
13
52.70
6650.97
743.82
74
Craft a bucket
13
54.29
6124.61
591.35
70
Craft hopper
14
46.67
7242.14
710.17
60
Craft rail
13
42.19
6713.07
754.48
64
Craft an iron sword
12
57.14
5625.49
633.91
70
Craft a shears
12
53.62
5058.00
570.35
69
Craft smithing table
12
44.93
5317.39
594.81
69
Craft a tripwire hook
13
48.57
4968.74
562.66
70
Craft a chain
13
44.93
5764.42
645.33
69
Craft iron bars
12
42.00
6508.43
723.13
50
Craft iron nugget
12
30.99
4697.23
525.29
71
Craft blast furnace
14
25.71
7760.67
711.05
35
Craft a stonecutter
13
34.78
5993.38
675.52
46
Table 11: The results of Optimus-1 on various tasks the Gold group. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
Task
Sub Goal Num.
SR
AS
AT(()
Eval Times
Craft golden shovel
16
9.80
13734.75
686.74
51
Craft golden pickaxe
16
13.75
9672.00
783.60
80
Craft a golden axe
16
4.44
10158.75
707.94
45
Craft a golden hoe
16
3.33
13120.50
756.03
27
Craft golden sword
16
3.33
9792.00
789.60
26
Smelt and craft golden ingot
15
16.42
9630.27
681.51
67
Table 12:
The results of Optimus-1 on various tasks in the Diamond group. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
Task
Sub Goal Num.
SR
AS
AT(s)
Eval Times
Craft a diamond shovel
18.75
23696.75
1184.84
64
Craft a diamond pickaxe
15
15.71
32189.50
1609.46
70
Craft a diamond axe
16
4.00
21920.50
1096.03
75
Craft diamond hoe Craft diamond sword
15
4.61
24031.00
1201.55
65
15
14.52
27555.50
1377.78
62
Dig down and mine diamond
9.09
20782.13
1039.11
64
Craft a jukebox
14.58
25056.00
1252.80
48
25

---

Table 13: The results of Optimus-1 on various tasks in the Redstone group. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
Language Instruction
Sub-Goal Num.
SR
AS
AT(s)
Eval Times
Craft piston
28.57
6457.10
822.85
35
Craft redstone torch
16
29.63
6787.87
939.39
27
Craft an activator rail
18
15.68
8685.62
934.28
51
Craft compass
23
15.00
14908.67
845.43
40
Craft a dropper
16
37.50
7272.80
1063.64
40
Craft note block
16
24.32
6727.89
936.39
37
Table 14: The results Optimus-1 on various tasks in the Armor group. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
Task
Sub Goal Num.
SR
AS
AT(()
Eval Times
Craft shield
14
43.33
7229.00
861.45
30
Craft iron chestplate
14
47.22
7230.24
851.51
36
Craft iron boots
14
23.81
6597.33
729.87
42
Craft iron leggings
14
6.67
9279.00
763.95
30
Craft iron helmet
14
58.14
6287.11
814.36
43
Craft diamond helmet
17
2.08
7342.00
867.10
48
Craft diamond chestplate
17
2.70
7552.00
777.60
37
Craft diamond leggings
17
9.68
7664.67
883.23
31
Craft diamond boots
17
16.67
10065.60
803.28
30
Craft golden helmet
17
12.50
11563.25
778.16
32
Craft golden leggings
17
14.60
10107.33
805.37
41
Craft golden boots
17
6.06
10311.00
915.55
33
Craft golden chestplate
17
9.67
10407.58
820.38
31
125
Minecraft Tech Tree
100
Wooden
Redstone Diamond
Nurber Distinct Itene
50
>
>
>
60
100
Our
Voyager
AutoGPT
Figure 8: An illustration of Optimus-1 unlocking the tech tree Minecraft.
26

---

Table 15: Result on Process-Dependeee Tasks compared with MP5 [33]. SR, AS, AT denote success rate, average number of steps, and average time (seconds), respectively.
MP5 [33]
Optimus-1
Task Level
SR
SR
AS
AT(()
log
96.67
100.00
586.58
29.33
sand
96.67
94.32
1540.33
77.02
planks
96.67
100.00
571.06
28.55
Basic Level
stick
96.67
97.14
372.97
18.65
crafting table
93.33
98.55
448.63
22.43
Average
96.00
98.00
703.91
35.20
bowl
93.33
100.00
744.30
37.21
boat
93.33
92.86
1170.00
58.50
chest
90.00
100.00
573.80
28.69
Wooden Level
wooden sword
86.67
100.00
1214.90
60.74
wooden pickaxe
80.00
100.00
1153.91
57.70
Average
88.67
98.57
971.38
48.56
cobblestone
80.00
95.29
1492.00
74.60
furnace
80.00
93.75
2842.71
142.14
stone pickaxe
80.00
96.77
2310.09
115.50
Stone Level
iron
60.00
50.00
3017.00
150.85
glass
80.00
81.11
3870.75
193.54
Average
76.00
83.38
2706.51
135.32
iron ingot
56.67
59.42
4697.23
634.86
shicld *
56.67
43.33
7229.00
661.45
bucket
53.33
54.29
6124.61
606.23
Iron Level
iron pickaxe
50.00
59.42
6157.39
607.87
iron door
43.33
48.28
5528.00
676.40
Average
52.00
52.94
5947.25
637.36
diamond ore *
30.00
9.09
20782.13
1039.10
mind redstone
20.00
25.12
6787.87
739.39
compass
16.67
15.00
14908.67
745.43
Diamond Level
diamond pickaxe
23.33
15.71
32189.50
1609.48
piston
20.00
28.57
6457.10
622.85
Average
22.00
18.70
14963.83
948.19
27

---

Table 16: Results (success rate) on 8 META TASK groups compared with DEPS [50].
Masa
Task Object
InnerMonologue
Code-as-Policy
DEPS [50]
Ours
planks
83.3
83.3
83.3
100.0
stick
83.3
83.3
86.7
97.1
chest
0.0
50.0
76.7
100.0
Basic MT1
sign
0.0
43.3
86.7
94.3
boat
26.7
56.7
73.3
92.9
trapdoor
56.7
56.7
76.7
96.2
bowl
23.3
46.7
80.0
100.0
trafting_table
70.0
70.0
90.0
98.5
wooden_piie Liccden_pickxeeee
80.0
80.0
80.0
100.0
wooden_sword
83.3
83.3
86.7
100.0
sonden_shovel
76.7
76.7
90.0
95.0
Tool(Si
furnace
40.0
40.0
66.7
93.7
MT2
stone_picke stone_pickaxe
36.7
53.3
73.3
96.7
stone_axe
30,0
30.0
70.0
96,8
stone_
36.7
56.7
66.7
94.6
stone_shovel
36.7
36.7
66.7
90.3
stone stone_sword
53.3
36.7
80.0
94.2
bed painting
6.7
6.7
43.3
90.0
16.7
16.7
86.7
92.2
Hunt and Food
carpet
0.0
13.3
43.3
91.3
MT3
cooked_ porkchop
0.0
0.0
50,0
90.0
cooked_bes beef
0.0
0.0
63.3
90.0
cooked_mutton
0.0
0.0
66.7
90.0
stone_stairs
36.7
16.7
66.7
90.3
stone_slab
16.7
33.3
73.3
91.2
Dig-down
lever
46.7
46.7
83.3
91.0
MT4
coal
6.7
0.0
20.0
86.5
torch
6.7
0.0
13.3
85.7
Srather_boot
13.3
13.3
60.0
68.2
leather_chhe chhsttta__ctttteeeeeeee
0.0
6.7
36.7
64.2
leather Deatherhhllmet helmet
6.7
0.0
70.0
65.9
Deather_legging
20.0
0.0
56.7
65.5
Equipment MT5
iron chessplee
0.0
0.0
0.0
47.2
iron_ iron_leggings iron_helmet
0.0
0.0
3.3
6.6
0.0
0.0
3.3
58.1
iron_boots boots iron
0.0
0.0
20.0
23.8
shield
0.0
6.7
13.3
43.3
bucket
0.0
3.3
6.7
54.3
shears
0.0
0.0
30.0
53.6
iron pickaxe
6.7
0.0
10.0
59.4
Tool Complex
iron_axc
0.0
0.0
16.7
54.3
MT6
iron_hoe
0.0
0.0
13.3
52.7
iron_ iron_shovel
0.0
0.0
13.3
57.8
iron_swon sword
0.0
3.3
6.7
54.7
iron bars
0.0
0.0
42.0
hopper
0.0
0.0
6.7
46.7
Iron-Stage
iron_ door
0.0
0.0
3.3
48.3
MT7
tripwire hook
6.7
0.0
30.0
48,6
rail
0.0
0.0
6.7
42.2
Challenge MT8
diamond
0.0
0.0
0.6
9.1
28

---

G
Case Study
This section introduces several cases comprchensively demonstrate Optimus-11s capabilities.
Figures 9, 10, and 11 demonstrate the superiority of our reflection mechanism, which dynamically adjusts the plan based on the current game progress.
Figure illustrates Optimus-1 I's replanning ability. When Optimus-1 realizes it cannot complete task (such craft failure shown in the figure), it will replan the current task and continue execution.
Figures 10 and 11 showcase Optimus-1 's ability to make judgments based on visual signals.
When Optimus-1 determines that it has completed a task (such as "kill a cow " in Figure 10), will finish the current task and move on to the next one. If Optimus-1 I discovers that it has not yet completed the task and the task has not failed(as shown in Figure 11), it will continue executing the task.
Figures 12 and 13 illustrate the advantages of planning with knowledge. With the Hierarchical Directed Knowledge Graph, we can generate a high-quality plan in one step and dynamically adjust the plan based on current visual signals.
Figure 12 demonstrates the importance of knowledge. For long-horizon task such as "Mine diamond @," Optimus-1 I first generates a plan based on the Hierarchical Directed Knowledge Graph. However, this plan needs to be adjusted based the current visual signals. For example, in this figure, Optimus-1 appears cave, so the primary task is not to "chop tree" but to "leave the cave" first. Only after exiting the cave can Optimus-1 proceed with the initial plan.
Figure 13 demonstrates the high efficiency of our method. Agents like MP5 [33] and Voyager [47] use an iterative planning approach, which is very time-consuming, generating the final plan step by step. During this process, agent does not take any action. As shown in Figure 13, zombie is gradually approaching the agent, but the agent still iterating its plan. Optimus-1, however, generates the plan in one step based on the Hierarchical Directed Knowledge Graph and makes reasonable plans based on the current visual signals.
1. Mine logs
2. Craft 8 planks
3. Craft sticks
0,
Task: Craft 1 wooden pickaxe
Hierarchical Directed
Knowledge Graph
0g
0g
BEES
Plan
4. Craft Crafting table
Craft wooden pickaxe (falled)
6. Replan: Mine Log, Craft planks, Craft wooden pickaxe
Figure 9: The process of completing the task "Craft wooden pickaxe". Optimus- gives wrong planning, When Optimus-1 realizes it cannot complete the task, it will replan the current task.
29

---

1. Mine Logs
2. Craft 12 planks
3. Craft sticks
Task: Find cow and kill it
Hierarchical Directed
Knowledge Graph
Plan
4. Craft Crafting table
5. Craft wooden sword
6.
cow
Figure 10: The process of completing the task "Find a cow and kill it". Hierarchical Directed Knowledge Graph indicates that having wooden sword will make the task easier to complete. Therefore, Optimus-1 first crafts wooden sword and then proceeds to find and kill cow.
1. Mine 10 Zogs
2. Mine 10 logs
3. Mine 10 Zogs
Ogi
Task: Chop obtain 10 logs
Hierarchical Directed
Knowledge Graph
+
Plan
4. Continue: Mine 10 Zogs
5. Mine 10 lags
6. Done: Mine 10 Zogs
Figure 11: The process of completing the task "Chop to obtain 10 logs". Hierarchical Directed Knowledge graph indicates that no tools are needed to complete this goal. After finding a tree, Optimus-1 starts chopping down. The task requires substantial amount of wood, midway through, Optimus-1 performs a reflection. The task is not yet complete but is progressing smoothly, and the result of the reflection is to continue.
30

---

1. Leave the cave
Leave the cave
2. Craft wooden pichaxe
Task: Mine diamond
Hlerarchical Directed
Knowledge Graph
0,
Plan
Leave the cave
3. Craft stone pichaxe
Craft 1. iron pichaxe
Mine 1 diamond
The process of completing the task "Mine diamond O". Mining diamonds @ is a highly complex task. Diamonds can only be mined with an iron pickaxe 7, an iron pickaxe 7 must be crafted first. Crafting an iron pickaxe >> requires iron ingots @,, which are smelted from iron ore C Mining iron ore requires a stone pickaxe >. Crafting a stone pickaxe > requires stone which in turn must be mined with a wooden pickaxe >. Crafting a wooden pickaxe > requires wooden planks and sticks /. All these crafting processes require a crafting table T and smelting requires furnace In this case, the agent spawns at cave, so Optimus- I must leave the cave to chop logs.
Figure 12:
Task: Craft 1 iron sword
MP5
Questio::
How tron sword??
Qunetion How cral stone picbass?
Quastan: How to olown sticc
Answer saord.
Engot,
Answer: cssolestone
Answer: Using planks
attchs stone pichuxe.
Question: How to octain tren Yinoo??
planks?
Answor: Mine Ezon cir, and smetl inf Lzon Engot,
Answer. Using stone picluzee to nine Eron are.
Arswer: Using logs to craft planks.
Hlerarchical Directed
zombie arriving...
Knowledge Graph
How swozd??
Ours
Answert sunny And
Figure 13: In this example, zombie is slowly approaching the agent. Agents like MP5 [33] and Voyager [47] uses an iterative planning strategy to generate the plan, which consumes a great deal of time and puts the agent in danger. While Optimus-I directly generates plan in step based the knowledge graph. Using the current visual information, it makes a plan to "run to a sunny place," allowing the agent avoid danger then begin to achieve sub-goals.
31

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We clearly state the claims and contributions of this paper in the abstract and introduction.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We clearly state the limitations of this paper in the Section
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer[[NA]
Justification: In this paper, we do not propose new theories or principles.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe the proposed methods and experimental setup in detail in the Section 2 and Section 3, and the implementation details in Appendix F.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We release our code and project in: http::///beerttooonnnnngghihhhhp 1. Iggithub.io/.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our method requires no parameter training. But we detail the benchmark and environment settings in Section 3 and Appendix E.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For each evaluation task, ran it at least 30 times and calculated the average accuracy to minimise random errors and uncertainty factors.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
32

---

Justification: We report the computer resource in Appendix F.3.
9. Code Of Ethies
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : http::p.....pplcccthhhhcccceeeee
Answer: [Yes]
Justification: The research conducted in the paper complies with the NeurIPS Code Ethics in all respects.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We state the Broader Impacts of the paper in the Appendix A.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We describe the safeguards in place responsibly release models with high risk in Appendix A.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We state the models used in Section 2 and comply with all licences.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We hired volunteers to play Minecraft and presented the details in the Appendix E.2.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We hired volunteers to play Minecraft as human-level baseline. We only record the games results, and volunteers are not exposed to any risks.
33


Reviews:
[{"review_id": "fRuD9ADTOj", "reviewer": "Reviewer_3HCc", "summary": "Optimus-1 introduces an innovative Hybrid Multimodal Memory module that combines a Hierarchical Directed Knowledge Graph (HDKG) and an Abstracted Multimodal Experience Pool (AMEP) to address knowledge and experience management in long-horizon tasks.", "questions": "See weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Proposes a hybrid multimodal memory module, which includes HDKG and AMEP, offering an innovative approach to managing knowledge and experience in long-horizon tasks.\n* Experimental results demonstrate that Optimus-1 significantly outperforms all existing agents on long-horizon tasks and achieves near-human-level performance in many tasks.\n* The HDKG maps knowledge into a directed graph structure, enabling the agent to efficiently retrieve and utilize knowledge without needing to update parameters.\n* The AMEP summarizes not only successful cases but also failure cases, significantly enhancing the agent's learning effectiveness.", "weaknesses": "* Improving long-horizon task performance through memory modules is common in LLMs; applying this directly to VLMs is not very novel.\n* Reaction time and decision efficiency might be issues; the current experimental results still show a large number of steps.\n* Although it performs well in the Minecraft environment, its performance in real-world applications, such as software manipulation and web navigation, has not been verified.\n* As new knowledge and tasks emerge, effectively updating and maintaining the knowledge (HDKG) remains a challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Improving long-horizon task performance through memory modules is common in LLMs; applying this directly to VLMs is not very novel.\n\nA1: **Incorporating multimodal memory into VLMs presents significant challenges compared to apply unimodal memory to LLMs.** Long-horizon tasks require the model to save and utilize past information, which is particularly crucial in a multimodal environment. The diverse structures and characteristics of information from different modalities make it difficult to  effectively preserve multimodal historical information over extended time periods.\n\n**To address this, we propose a novel Hybrid Multimodal Memory module that incorporates structured knowledge and multimodal experiences, implementing the dynamic summarization of multimodal information to reduce storage and retrieval costs.** It includes 1) a novel memory module called Hierarchical Directed Knowledge Graph, which is highly structured and easily updated, enabling concise representation and storage of complex knowledge; 2) a novel method for constructing Abstracted Multimodal Experience Pool that dynamically summarizes long-sequence multimodal information, encompassing both global overviews and local details of multimodal experiences.\n\n**Table 7 in the manuscript** shows that existing agents do not incorporate memory modules with both knowledge and experience, resulting in inferior performance on long-horizon tasks compared to Optimus-1 (**Table 1, Table 15, Table 16 in the manuscript**).\n\n\n> Q2: Reaction time and decision efficiency might be issues; the current experimental results still show a large number of steps.\n\nA2: (1) For reaction time, **Table 1 in the manuscript** shows that Optimus-1's average task completion time (AT) is significantly lower than other baselines and approaches human-level performance. For example, on the Wood Group, Optimus-1 takes an average of **47** seconds, human baseline takes **31** seconds, whereas DEPS [1] takes **85** seconds.\n\n(2) For decision efficiency, unlike existing agents [1] [2] [3] that require multiple interactions with (M) LLMs for task planning, Optimus-1 completes task planning with a single interaction, thus achieving much higher decision efficiency than existing agents. For example, to complete the planning of task “craft iron pickaxe”, MP5 [3] requires **11 interactions** with MLLM, whereas Optimus-1 completes the planning in just **one interaction**.\n\n(3) In MineRL environment, 'steps' refers to the number of interactions between the agent and the environment, occurring at a frequency of 20 times per second. For example, if an agent takes 2 seconds to complete the task “chop a tree”, it interacts with the environment 40 times, resulting in a recorded steps number of 40. **Table 1 in the manuscript** shows that Optimus-1's average task completion step (AS) is significantly lower than other baselines.\n\n\n> Q3: Although it performs well in the Minecraft environment, its performance in real-world applications, such as software manipulation and web navigation, has not been verified.\n\nA3: (1) **Minecraft is a valuable and representative environment for evaluating long-horizon tasks, offering greater diversity and complexity compared to other environments**. Unlike web/app navigation [4] and embodied manipulation [5], Minecraft is an open world with a complex and dynamic environment (79 biomes, including ocean, plains, forest, desert, etc.). To complete long-horizon tasks, agents must achieve multiple sub-goals (e.g., 15 sub-goals to craft a diamond sword), making the construction of a Minecraft agent quite challenging. Many studies [2] [3] [6] have chosen Minecraft as the environment for validating performance on long-horizon tasks. Extensive experimental results (**Table 1, Table 15, Table 16, Figure 8 in the manuscript**) show that Optimus-1 outperforms all baselines. Therefore, conducting experiments in the Minecraft environment is sufficient to demonstrate the contributions of this paper.\n\n(2) **Our proposed Hybrid Multimodal Memory is easily adaptable to other domains**. Take the app agent [4] as an example, the key step involves transforming the knowledge structure from Minecraft's object synthesis relationships into logical relationships between buttons or operations. Once these logical relationships are established, HDKG can easily be adapted to the app environment. As for AMEP, it can be simplified to store the task prompt, image, and action for each atomic operation. Adapting our method to other domains remains as future work.\n\n\n> Q4: As new knowledge and tasks emerge, effectively updating and maintaining the knowledge (HDKG) remains a challenge.\n\nA4: **Our HDKG can be efficiently updated and expanded**. When adding new nodes, the HDKG can be updated by simply merging the nodes and relationships into the graph. This method involves local linear modifications to the graph rather than altering the entire graph, making the process efficient and time-saving. For example, when M new nodes and N edges are added, the HDKG can be updated with M+N times of operations. Moreover, an HDKG containing 851 objects (nodes) requires less than 1 MB of memory. Thus, the HDKG can be efficiently updated and maintained.\n\n[1] Wang et al. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. 2023.\n\n[2] Wang et al. Voyager: An open-ended embodied agent with large language models. 2023.\n\n[3] Qin et al. MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception. 2024.\n\n[4] Zhang et al. AppAgent: Multimodal Agents as Smartphone Users. 2023.\n\n[5] Jiang et al. VIMA: General Robot Manipulation with Multimodal Prompts. 2023.\n\n[6] Baker et al. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gumwHbSQOh", "reviewer": "Reviewer_397y", "summary": "The paper tackles the long-horizon tasks in Minecraft by building a pipeline based on multimodal LLM. Specifically, it proposes to store multimodal memory during agent exploration and a knowledge graph that stores the causal relations between objects and tasks. Additionally, a self-reflection mechanism is used to improve the textual actions. Finally, the textual actions are executed by a pre-trained action controller to output the low-level actions.", "questions": "See \"weaknesses\" section above.", "limitations": "The limitations are described in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The core idea is intuitive that multimodal experience can help with long-horizon tasks for embodied agents, where they can be retrieved at test time to help with decision making.\n- The paper is overall written clearly and sufficient details are provided in appendix.", "weaknesses": "- There seem to be too many individual components in the paper, most of which are claimed to be important components in the paper, which makes it unclear to what extent the claimed contribution generalizes to broader settings beyond Minecraft (or even just beyond the evaluated tasks in Minecraft), as it is also highly plausible that the entire pipeline is carefully designed specifically for the evaluated tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: There seem to be too many individual components in the paper, most of which are claimed to be important components in the paper, which makes it unclear to what extent the claimed contribution generalizes to broader settings beyond Minecraft (or even just beyond the evaluated tasks in Minecraft), as it is also highly plausible that the entire pipeline is carefully designed specifically for the evaluated tasks.\n\n(1) **The core innovative module of Optimus-1 is Hybrid Multimodal Memory**. It includes 1) a novel memory module called Hierarchical Directed Knowledge Graph, which is highly structured and easily updated, enabling concise representation and storage of complex knowledge; 2) a novel method for constructing Abstracted Multimodal Experience Pool that dynamically summarizes long-sequence multimodal information, encompassing both global overviews and local details of multimodal experiences. Building on it, the Knowledge-guided Planner utilizes the HDKG to enhance task planning capabilities, while the Experience-Driven Reflector leverages the AMEP to improve reflection abilities. Both represent improvements over planner and reflector of existing agents. \n\n(2) **Optimus-1 can adapt to different settings and generalize to various Minecraft tasks**, consistently outperforming all baselines across multiple benchmarks [1] [2] [3] (**Figure 8, Table 15, Table 16 in the Appendix**). \n\n(3) **Our proposed Hybrid Multimodal Memory is easily adaptable to other domains**. Take the app agent [4] as an example, the key step involves transforming the knowledge structure from Minecraft's object synthesis relationships into logical relationships between buttons or operations. Once these logical relationships are established, HDKG can easily be adapted to the app environment. As for AMEP, it can be simplified to store the task prompt, image, and action for each atomic operation. Adapting our method to other domains remains as future work.\n\n[1] Wang et al. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. 2023.\n\n[2] Qin et al. MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception. 2024.\n\n[3] Wang et al. Voyager: An open-ended embodied agent with large language models. 2023.\n\n[4] Zhang et al. AppAgent: Multimodal Agents as Smartphone Users. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1NuXYNXoRn", "reviewer": "Reviewer_mEzT", "summary": "The paper proposes \"Optimus-1\" a Mulit-Modal LLM based agent. They evaluate their agent extensively on MineCraft and demonstrate superior performance compared to previous work. \n\nThe key components of the approach are: \n1. A memory module consisting of structured memories (DAG) and multi-modal experience replay (with negative and positive examples)\n2. A planing module\n3. A reflection module\n4. A \"low-level\" execution module\n\nThe novelty of the work consists of:\n1. The DAG memory  \n2. the positive & negative samples in the replay memory\n3. Putting everything together in this way\n\nMultiple ablation studies are made.", "questions": "Questions:\n1. How are teacher demonstrations obtained? How costly is this? How many human annotators and hours are needed for this?\n2. How does the agent perform without teacher demonstrations?\n3. What is the performance without negative samples in AMEP?\n4. How long does the training / exploration phase take? How costly is it? (How many GPU hours on 4x A100 did it take?) How much is an evaluation?\n5. How long would your method take to setup in a new environment altogether (and what would be the rough steps)? What is another good environment?", "limitations": "Limitations:\n1. The authors speak about the limitations of one of the modules (the action generator), however, the teacher phase limitation mentioned above is not discussed. It seems that the method strongly depends on expert demonstrations and these are expensive to obtain in new environments outside of Minecraft.\n\n2. Evaluations on other envs than Minecraft would have been interesting to discuss.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. A lot of work went into running these experiments and a lot of results are presented (incl. open-source MLLMs)\n2. A lot of work went into designing the evaluation benchmark\n3. A novel agent that performs better than previous work (and seems that can be applied to more tasks).", "weaknesses": "Weakness:\n1. \"teacher guidance learning\" for HDKG -> It seems that teacher demonstrations are needed to make the agent perform the hard tasks. \na. The paper (even incl. appendix) does not explain where these demonstrations come from and what the impact of these is. (Or how costly these demonstrations are to obtain).\nb. Looking at ablation studies one can see that this Knowledge Module is key and performance drops by 20% (however, it is unclear how much the \"teacher\" contribution is. It seems that without the teacher contribution the model might perform quite poorly compared to baselines.) Therefore this indicates that expert human demonstrations are needed to actually make the agent work, as opposed to the \"LLM\" doing the reasoning and work.\nc. Similarly, all further evaluations (incl. against other benchmarks in the appendix) are therefore somewhat questionable.\n\n2. The actual learning mechanism of the DAG memory (HDKG) is not described in sufficient detail to replicate the work. It would be good to cite more references or explain this part in more detail. \n\n3. Positive and Negative examples in AMEP. While the paper argues this is an important contribution, it seems that it is nowhere evaluated if only positive examples would be more succesful.\n\n4. Computational costs are only marginally mentioned ($5000 for OpenAI, and only 4xA100, but no amount of hours). It is not clear how much the \"exploration\" / learning phase is (time and money), what about the teacher phase? \n\n5. Evaluation outside of Minecraft would be interesting as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: How are teacher demonstrations obtained? How costly is this? How many human annotators and hours are needed for this?\n\nA1: (1) We obtain teacher demonstrations from Minecraft Wiki. For each task group, we randomly select 5 tasks that are not included in the benchmark. We then create corresponding task plans based on content (synthesis relationship between objects) from the Minecraft Wiki. Taking the task “craft a wooden sword” as an example, we obtain the crafting relationships from the Minecraft Wiki: {1 wooden stick, 2 planks, 1 crafting table → 1 wooden sword}, {1 log → 4 planks}, {2 planks → 4 sticks}, {4 planks → 1 crafting table}. These relationships are transformed into the plan: {1. Get two logs 2. Craft eight planks 3. Craft a crafting table 4. Craft two sticks 5. Craft a wooden sword}. The plan serves as teacher demonstrations for action controller to perform the task.\n\n(2) This method is very low-cost. This process does not require additional human annotators, and it takes about 2 hours to obtain all teacher demonstrations.\n\n(3) We will include these details in the Appendix.\n\n> Q2: How does the agent perform without teacher demonstrations? It is unclear how much the teacher demonstrations contribute to HDKG.  Teacher demonstrations are needed to actually make the agent work, as opposed to the \"LLM\" doing the reasoning and work.\n\nA2: (1) **Table 2 in the manuscript shows that without teacher demonstrations, the performance of Optimus-1 decreases** (e.g., 9.5% -> 1.8% on Diamond Group)**.** \n\n(2) **The construction of the HDKG is indispensable without teacher demonstrations**.  For example, in the free exploration phase, Optimus-1 learns the methods for crafting/mining basic materials (e.g., craft sticks and mine diamonds). Without demonstrations (plan), it cannot learn the synthesis methods for advanced items (e.g., a diamond sword is obtained by a stick and two diamonds). This limits Optimus-1's ability to complete challenging long-horizon tasks (such as craft a diamond sword).\n\n(3) **Obtaining teacher demonstrations is efficient and low-cost.** Additionally, compared to parameter-update learning, our non-parametric learning method needs only a few demonstrations (plans) as learning data. This allows for the rapid expansion of the HDKG, enabling better reasoning and planning for MLLMs.\n\n> Q3: The actual learning mechanism of the DAG memory (HDKG) is not described in sufficient detail to replicate the work.\n\nA3: (1) Due to the diversity and complexity of knowledge in Minecraft, we propose \"free exploration-teacher guidance\" approach instead of manually constructing a knowledge graph.\n\n(2) During the free exploration phase, Optimus-1 will randomly initialize the environment, materials, and tasks. It will freely explore basic tasks, such as chop down trees, mine stones with pickaxes, craft stone swords, etc. When the environment feedback indicates that a task is completed, the corresponding relationship (e.g., {1 wooden stick, 2 planks} → {1 wooden sword}) is updated into the HDKG. \n\n(3) During the teacher guidance phase, after Optimus-1 completes a long-horizon task, advanced synthesis relationship (e.g., {1 wooden sticks, 2 diamonds} → {1 diamond sword}) is updated in the HDKG. \n\n(4) We will add a detailed description of “free exploration-teacher guidance\" approach in the Appendix.\n\n> Q4: What is the performance without negative samples in AMEP?\n\nA4: **Table 3 in the manuscript** shows the ablation study for AMEP. It demonstrates that removing negative samples results in a decrease on the success rate (e.g. 94% -> 84% on Stone Group). This reveals that incorporating both success and failure cases into in-context learning significantly enhances the performance of the agent.\n\n> Q5: How long does the training / exploration phase take? How costly is it? (How many GPU hours on 4x A100 did it take?) How much is an evaluation?\n\nA5: (1) In the free exploration and teacher guidance phases, there is no need to access OpenAI's API, which keeps costs low. We instantiate multiple Optimus-1 in parallel, sharing the same memory, and the learning process takes approximately 16 hours on 4x A100 80G GPUs.\n\n(2) Evaluating Optimus-1 on benchmark costs approximately $900. We parallelize the evaluation, which takes about 20 hours on 4x A100 80G GPUs. While figure 5 in the manuscript demonstrates that with Hybrid Multimodal Memory, open-source MLLMs’ performance near the GPT-4V’s. This reveals that we can achieve excellent performance with open-source MLLMs at a very low cost.\n\n> Q6: How long would your method take to setup in a new environment altogether? What is another good environment?\n\nA6: **Our method is highly adaptable to various environments**. Take the app agent as an example, the key step involves transforming the knowledge structure from Minecraft's object synthesis relationships into logical relationships between buttons or operations. Once these logical relationships are established, HDKG can easily be adapted to the app environment. As for AMEP, it can be simplified to store the task prompt, image, and action for each atomic operation. Adapting our method to other domains remains as future work.\n\n> Q7: The teacher phase limitation mentioned above is not discussed. It seems that the method strongly depends on expert demonstrations and these are expensive to obtain in new environments outside of Minecraft.\n\nA7: As stated in A1 and A2, teacher demonstrations (plans) are easy to collect, efficient, and low-cost. In a new environment, such as an app agent, it is only necessary to collect plans of atomic operations.\n\n> Q8: Evaluations on other envs than Minecraft would have been interesting to discuss.\n\nA8: In the future, we will extend the Hybrid Multimodal Memory to other domains. However, we believe that the current environment and experiments sufficiently demonstrate the contribution and effectiveness of our work. Please refer to the response to reviewer 3HCc, A3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F018UJNteE", "reviewer": "Reviewer_AfEc", "summary": "This paper presents Optimus-1, a multimodal agent that focuses on Minecraft tasks. Specifically,  Optimus-1 is equipped with a Hybrid Multimodal Memory including: 1) a Hierarchical Directed Knowledge Graph that stores the world knowledge through free exploration and teacher guidance; 2) an Abstracted Multimodal Experience Pool that enables Optimus-1 to reason about the current situation by using past experience. Based on the multimodal memory, Optimus-1 adopts a Knowledge-Guided Planner and an Experience-Driven Reflector to generate better plan and reflect periodically in long-horizion tasks. Experiments results illustrates the effectiveness of the multimodal memory of Optimus-1 in long-horizion tasks.", "questions": "1.  In the construction of AMEP, given the video stream, how to adaptively update the abstracted frames in the buffer? How is the threshold of MineCLIP determined?\n2. In the free-exploration phase, what is the efficiency of Optimus-1 to learn an entry of world knowledge such as \"a stone sword can be crafted with a wooden stick and two cobblestones\" through random exploration?\n3. In the reflection phase, how to retrieve past experience from AMEP?  Is the retrieval based on the image similartiy, task goal, or a combination of both? How does the fail case contribute to the final success of the task?\n4. In Table 1, how is GPT-4V evaluated?  Does GPT-4V have the same world knowledge as Optimus-1?", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a multi-modal memory mechanism that includes a hierarchical world knowledge graph and a multi-modal past experience pool. The memory is later utilized by the multi-modal planner and reflector module of the Optimus-1. \n2. Good experiment results illustrate the effectiveness of Optimus-1 compared to other strong baselines. Sufficent ablation studies of the effectiveness of the modules in Optimus-1 as well as the necessity of both success and failure cases for reflection.\n3. The paper is well written.", "weaknesses": "1. The details of the construction procedure of AMEP are unclear (e.g. How to maintain the image buffer by computing image similarity? What is the threshold used in MineCLIP? )\n2. How Optimus-1 acquires world knowledge through free-exploration is not specified.\n3. There are other minecraft agents  (Jarvis-1, etc.) using multi-modal memory. The paper claims the efficiency of the memory storage and retrieval compared to Jarvis-1, but this is not quantitatively evaluated. \n4. It seems that the low-level action controller will not be updated through reflection, which limits the effectiveness of the reflection pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: In the construction of AMEP, given the video stream, how to adaptively update the abstracted frames in the buffer? How is the threshold of MineCLIP determined?\n\nA1: (1) **As described in Section 2.1.1**, for video streams, we filter video streams at 1-second intervals and store them in a variable-length video buffer. These filtered images sequentially enter an image buffer with a window size of 16. When the image buffer reaches its capacity and a new image is added, we calculate the cosine similarity between frames and remove one frame from the pair with the highest similarity. Through this process, we can dynamically preserve abstracted frames in the image buffer.\n\n(2) We empirically set the threshold for MineCLIP at 0.7.\n\n> Q2: How Optimus-1 acquires world knowledge through free-exploration? what is the efficiency of Optimus-1 to learn an entry of world knowledge such as \"a stone sword can be crafted with a wooden stick and two cobblestones\" through random exploration?\n\nA2: (1) During the free exploration phase, Optimus-1 will randomly initialize the environment, materials, and tasks. It will freely explore basic tasks, such as chop down trees, mine stones with pickaxes, craft stone swords, etc. When the environment feedback indicates that a task is completed, the corresponding relationship (e.g., {1 wooden stick, 2 planks} → {1 wooden sword}) is updated into the HDKG.\n\n(2) We use task decomposition and parallelized methods to enable Optimus-1 to learn world knowledge efficiently.  Given the initial materials, Optimus-1 only needs to perform one sub-goal (chop a tree, mine iron ore, craft a stone sword by a wooden stick and two cobblestones, etc.) each time, which enables Optimus-1 to quickly complete the task and then learn knowledge. Furthermore, we initialize multiple instances of Optimus-1, which share the same HDKG and AMEP. This allows Optimus-1 to efficiently learn such knowledge.\n\n> Q3: There are other minecraft agents (Jarvis-1, etc.) using multi-modal memory. The paper claims the efficiency of the memory storage and retrieval compared to Jarvis-1, but this is not quantitatively evaluated.\n\nA3: Jarvis-1 [1] stores all images without summarization, while our approach summarizes each sub-goal and only retains 16 images per sub-goal, significantly improving storage efficiency. Take the example of \"craft a wooden pickaxe\" shown in its demo website, Jarvis-1 executes 1,139 steps, storing 1,139 images. In contrast, we only store  5 sub-goals × 16 images = 80 images, resulting in a 14x increase in storage efficiency. Under a smaller memory storage, the retrieval efficiency will naturally be higher.\n\n> Q4: It seems that the low-level action controller will not be updated through reflection, which limits the effectiveness of the reflection pipeline.\n\nA4: The action controller does not affect the effectiveness of the reflection pipeline. The purpose of reflection is to correct the planner. The reflector evaluates the current situation based on multimodal experience to determine whether the planner needs to replan. If needed, the planner is requested to generate a new plan, which is then executed by the action controller. If not, the action controller continues executing the current sub-goal. Therefore, the action controller does not affect the effectiveness of the reflection pipeline.\n\n> Q5: In the reflection phase, how to retrieve past experience from AMEP? Is the retrieval based on the image similartiy, task goal, or a combination of both? How does the fail case contribute to the final success of the task?\n\nA5: (1) During the reflection phase, we use text matching method to retrieve task goal to obtain success and failure cases. When there are multiple similar cases, we select the one with the highest image similarity.\n\n(2) Failure cases serve as in-context examples for the Reflector, assisting in evaluating whether the current task goal can be achieved under the present situation. Due to the complexity and diversity of the environment, it’s challenging to determine the success of the current task based solely on success cases. The inclusion of failure cases allows the agent to assess the current state through a diverse comparison. **Table 3 in the manuscript** reveals that incorporating both success and failure cases into in-context learning significantly enhances the performance on long-horizon tasks.\n\n> Q6: In Table 1, how is GPT-4V evaluated? Does GPT-4V have the same world knowledge as Optimus-1?\n\nA6: (1) **In Table 1 in the manuscript**, GPT-4V is evaluated without integrating hybrid multimodal memory modules. During the planning phase, GPT-4V generates a plan for the action controller based on observation and task. During the reflection phase, it generates reflection results in a zero-shot manner. \n\n(2) Optimus-1 is built upon the GPT4V foundation with the Hybrid Multimodal Memory. **Table 2 in the manuscript** shows that the removal of structured knowledge from Optimus-1 (based on GPT-4V) leads to a significant decrease in success rates across all task groups. It proves that GPT4V lacks sufficient knowledge of the Minecraft environment.\n\n(3) We will add a detailed description of the baseline settings in the Appendix.\n\n[1] Wang et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znnHT8EEdX", "reviewer": "Reviewer_wzp9", "summary": "This paper introduces Optimus-1, a retrieval augmented generation method that enables minecraft agents to excel in long-horizon tasks. The proposed method is based on a Hybrid Multimodal Memory Module that consists of an Abstracted Multimodal Experience Pool and a Hierarchical Directed Knowledge Graph. Those two key components are mainly proposed to solve the challenge of insufficient structured knowledge and lack of multimodal experience. Empirical evaluations show that such a memory module can effectively improve the performance of a minecraft agent in long-horizon planning tasks. Ablative experiments are conducted to validate the effective of each component.", "questions": "See weakness", "limitations": "Limitations are not adequately discussed in the paper. One potential limitation is that the performance of RAG methods may be capped by the performance of the base model.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "This paper addresses an important problem that transforms LLM into autonomous decision making agents.\n\nThe proposed method achieves a significant improvement over prior state-of-the-art in Minecraft, particularly exceling at long-horizon tasks such as getting diamonds.", "weaknesses": "The writing of this paper can be improved. In particular, I would suggest a major rewriting of the method section that 1) follows a top-down organization to first talk about high-level ideas of the method before going to the details such as \"1 frame per second\" and \"a window size of 16\", and 2) reduces unnecessary complications of the terminology including Hybrid Multimodal Memory Module, Hierarchical Directed Knowledge Graph, etc.\n\nMajor claims in the introduction are not supported by empirical evidence. It is unclear whether GPT4V etc does not have sufficient structured knowledge and related works in multi-modal agents do employ multi-modal experiences [1].\n\nIt seems that the use of Hierarchical Directed Knowledge Graph is limited to Minecraft where there is a strict Directed Graph relation between different objects. It is unclear whether this method can be helpful in the general setting.\n\nThe performances are evaluated on a set of custom benchmarks, and it will be good to have results on prior benchmarks reported by the baseline methods such as Voyager and Jarvis.\n\nMinor:\nLine 25 \"Early research [1, 6, 17] developed simple agents by constructing policy networks.\" seems unclear what those policy networks are.\n\nLine 30, what long horizon tasks are the authors talking about?\n\nLine 36 grammar error \"Insufficient of Structured Knowledge\", what is empirical evidence for existing agents do not have structured knowledge?\n\nLine 70 and line 71, two 30% seems repetitive.\n\nLine 108, where is the subgoal coming from?\n\nLine 115-116, what are those reflection phases?\n\nIn line 206, why did the authors contruct a new benchmark instead of following the prior benchmarks on minecraft?\n\n[1] Zhang, C., Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., … Yu, G. (2023). AppAgent: Multimodal Agents as Smartphone Users.\n\n[2] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023\n\n[3] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: The writing of this paper can be improved.\n\nA1: Thank you for your valuable suggestions. In the method section, we first introduce the core contribution of this paper, Hybrid Multimodal Memory, including its motivation, innovations, and components. Based on it, we developed a novel agent framework, Optimus-1. We follow a top-down organization to introduce Optimus-1’s components. Following your suggestions, we will improve the organization and logic of the paper to minimize potential misunderstandings for readers.\n\n> Q2: It is unclear whether GPT4V etc does not have sufficient structured knowledge and related works in multi-modal agents do employ multi-modal experiences [1].\n\nA2: (1) **Table 2 in the manuscript** shows that in the Minecraft environment, the removal of structured knowledge from Optimus-1 (based on GPT-4V) leads to a significant decrease in success rates across all task groups. It proves that GPT4V lacks sufficient knowledge of the Minecraft environment.\n\n(2) App Agent does not employ multimodal experiences. During the stage of free exploration, it records the effects of actions applied to different UI elements in the text modality. In contrast, Optimus-1 not only records the tasks, environmental information, agent initial state, and plan in the text modality, but also the abstract visual information in the image modality.  Furthermore, It dynamically summarizes long-sequence multimodal information, significantly reducing memory and retrieval costs.\n\n> Q3: It is unclear whether this method can be helpful in the general setting.\n\nA3: In the future, we will extend the Hybrid Multimodal Memory to other domains. However, we believe that the current environment and experiments sufficiently demonstrate the contribution and effectiveness of our work. Please refer to the response to reviewer 3HCc, A3.\n\n> Q4: The performances are evaluated on a set of custom benchmarks, and it will be good to have results on prior benchmarks reported by the baseline methods such as Voyager and Jarvis.\n\nA4: (1) Our benchmark is extensive and comprehensive, involving the most common long-horizon tasks in Minecraft. Moreover, we add the average steps (AS) and average time (AT) of completing the task as evaluation metrics, to better evaluate the efficiency of the agent. Furthermore, we have constructed a baseline for human evaluation, which represents a major contribution compared to the previous benchmark.\n\n(2) We have evaluated the performance of Optimus-1 on prior benchmarks reported by DEPS [1], Voyager [2], MP5 [3] (**Table 15, Table 16, Figure 8 in the Appendix**). Extensive experimental results demonstrate that Optimus-1 outperforms all baselines.\n\n> Q5: Minor: Line 25 \"Early research [1, 6, 17] developed simple agents by constructing policy networks.\" seems unclear what those policy networks are.\n\nA5: Policy networks in line 25 refer to models based on the Transformer architecture, trained through reinforcement learning/imitation learning. We will revise the wording to avoid potential misunderstandings by readers.\n\n> Q6: Line 30, what long horizon tasks are the authors talking about?\n\nA6: In Minecraft, long-horizon tasks refer to complex tasks that require the agent to continually interact with a complex environment to complete a long sequence of sub-goals. **We provide a detailed explanation and examples in Appendix C.3**.\n\n> Q7: Line 36 grammar error \"Insufficient of Structured Knowledge\"\n\nA7: We will revise it to 'Insufficient Exploration of Structured Knowledge.\n\n> Q8: Line 70 and line 71, two 30% seems repetitive.\n\nA8: We will remove “Optimus-1 closes the gap between agents and human player performance by 30%”.\n\n> Q9: Line 108, where is the subgoal coming from?\n\nA9: As stated in line 152-153, sub-goals are plans generated by the Knowledge-Guided Planner. We will revise the description of sub-goals in line 108 to avoid misunderstanding by readers.\n\n> Q10: Line 115-116, what are those reflection phases?\n\nA10: As stated in line 173, reflection results from Experience-Driven Reflector are categorized as COMPLETE, CONTINUE, and REPLAN. We will revise the description of reflection phases in line 115 to avoid misunderstanding by readers.\n\n> Q11: Limitations are not adequately discussed in the paper. One potential limitation is that the performance of RAG methods may be capped by the performance of the base model.\n\nA11: **We discuss the limitations in Appendix B**. **Figure 5 in the manuscript** indicates that the proposed hybrid multimodal memory (using RAG technology) is adaptable to GPT-4V and open-source multimodal large language models (MLLMs). Various MLLM-based versions of Optimus-1 have demonstrated performance improvements ranging from 2 to 6 times.\n\n[1] Wang et al. Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. 2023.\n\n[2] Wang et al. Voyager: An open-ended embodied agent with large language models. 2023.\n\n[3] Qin et al. MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Response to all Reviewers**

We would like to thank all reviewers (#3HCc, #397y, #mEzT, #AfEc, #wzp9) for their time and efforts in providing constructive feedback. We are very encouraged that the reviewers found the manuscript is well-written and easy to follow (R#397y, R#AfEc), proposed Hybrid Multimodal Memory is novel (R#3HCc, R#AfEc, R#397y, R#mEzT, R#wzp9), proposed Optimus-1 outperforms over prior state-of-the-art (R#3HCc, R#mEzT, R#AfEc, R#wzp9), with comprehensive experiments (R#mEzT, R#AfEc). We have built an official repository for providing well-structured open-source codes (released upon the acceptance). 

We have responded to your questions and comments inside each individual review. We hope these responses will offer a more thorough understanding of our paper. If your concerns have been resolved, we would greatly appreciate it if you could consider giving us a higher rating.

Meta-review: This paper introduces Optimus-1, designed to handle long-horizon tasks in Minecraft. The core innovation is a Hybrid Multimodal Memory module that manages knowledge and experience in such tasks. Additionally, a self-reflection mechanism enhances textual actions, which are then executed by a pre-trained action controller to produce low-level actions.

**Strengths**

* Novel concept of a hybrid multimodal memory module, which facilitates the management of knowledge and experience in long-horizon tasks.

*  Optimus-1 significantly outperforms all existing agents on long-horizon tasks in Minecraft.

**Weaknesses/suggestions**

* Limited Domain: Some components of the proposed pipeline are somewhat domain-specific (specialized to Minecraft), which raised concerns among reviewers regarding the generality of the work. The authors addressed these concerns by presenting additional results on mobile device control.

I think the paper makes a nice contribution that the community will find valuable. However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera ready version.
Decision: Accept (poster)