Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks

Authors: zaijing  li
yuquan  xie
rui  shao
gongwei  chen
dongmei  jiang
liqiang  nie
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 07:56:06

Abstract: Building a general-purpose agent is a long-standing vision in the field of artificial intelligence. Existing agents have made remarkable progress in many domains, yet they still struggle to complete long-horizon tasks in an open world. We attribute this to the lack of necessary world knowledge and multimodal experience that can guide agents through a variety of long-horizon tasks. In this paper, we propose a Hybrid Multimodal Memory module to address the above challenges. It 1) transforms knowledge into Hierarchical Directed Knowledge Graph that allows agents to explicitly represent and learn world knowledge, and 2) summarises historical information into Abstracted Multimodal Experience Pool that provide agents with rich references for in-context learning. On top of the Hybrid Multimodal Memory module, a multimodal agent, Optimus-1, is constructed with dedicated Knowledge-guided Planner and Experience-Driven Reflector, contributing to a better planning and reflection in the face of long-horizon tasks in Minecraft. Extensive experimental results show that Optimus-1 significantly outperforms all existing agents on challenging long-horizon task benchmarks, and exhibits near human-level performance on many tasks. In addition, we introduce various Multimodal Large Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show that Optimus-1 exhibits strong generalization with the help of the Hybrid Multimodal Memory module, outperforming the GPT-4V baseline on many tasks.

Main paper: 
 introduction: 
 Optimus Prime faces complex tasks alongside humans in Transformers to protect the peace of the planet. Creating an agent [44, 13] like Optimus that can perceive, plan, reflect, and complete long-horizon tasks in an open world has been a longstanding aspiration in the field of artificial intelligence [22, 36, 37, 27, 58] . Early research developed simple policy through reinforcement learning [7] or imitation learning [1, 25] . A lot of work [47, 50] have utilized Large Language Models (LLMs) as action planners for agents, generating executable sub-goal sequences for low-level action controllers. Further, recent studies [52, 33] employed Multimodal Large Language Models (MLLMs) [4, 39, 56] as planner and reflector. Leveraging the powerful instruction-following and logical reasoning capabilities of (Multimodal) LLMs [24] , LLM-based agents have achieved remarkable success across multiple domains [14, 9, 10, 55] . Nevertheless, the ability of these agents to complete long-horizon tasks still falls significantly short of human-level performance.According to relevant studies [28, 42, 46] , the human ability to complete long-horizon tasks in an open world relies on long-term memory storage, which is divided into knowledge and experience. The storage and utilization of knowledge and experience play a crucial role in guiding human behavior and enabling humans to adapt flexibly to their environments in order to accomplish long-horizon tasks. Inspired by this theory, we summarize the challenges faced by current agents as follows:Insufficient Exploration of Structured Knowledge: Structured knowledge, encompassing open world rules, object relationships, and interaction methods with the environment, is essential for agents to complete complex tasks [34, 44] . However, MLLMs such as GPT-4V * lack sufficient knowledge in Minecraft. Existing agents [1, 25, 7] only learn dispersed knowledge from video data and are unable to efficiently represent and learn this structured knowledge, rendering them incapable of performing complex tasks.Lack of Multimodal Experience: Humans derive successful strategies and lessons from information on historical experience [8, 32] , which assists them in tackling current complex tasks. In a similar manner, agents can benefit from in-context learning with experience demonstrations [43, 54] . However, existing agents [47, 51, 33] only consider unimodal information, which prevents them from learning from multimodal experience as humans do.To address the aforementioned challenges, we propose Hybrid Multimodal Memory module that consists of Hierarchical Directed Knowledge Graph (HDKG) and Abstracted Multimodal Experience Pool (AMEP). For HDKG, we map the logical relationships between objects into a directed graph structure, thereby transforming knowledge into high-level semantic representations. HDKG efficiently provides the agent with the necessary knowledge for task execution, without requiring any parameter updates. For AMEP, we dynamically summarize and store the multimodal information (e.g., environment, agent state, task plan, video frames, etc.) from the agent's task execution process, ensuring that historical information contains both a global overview and local details. Different from the method of directly storing successful cases as experience [52] , AMEP considers both successful and failed cases as references. This innovative approach of incorporating failure cases into in-context learning significantly enhances the performance of the agent.On top of the Hybrid Multimodal Memory module, we construct a multimodal composable agent, Optimus-1. As shown in Figure 1 , Optimus-1 consists of Knowledge-Guided Planner, Experience-Driven Reflector, and Action Controller. To enhance the ability of agents to cope with complex environments and long-horizon tasks, Knowledge-Guided Planner incorporates visual observation into the planning phase, leveraging HDKG to capture the knowledge needed. This allows the agent to efficiently transform tasks into executable sub-goals. Action Controller takes the sub-goal and the current observation as inputs and generates low-level actions, interacting with the game environment to update the agent's state. In open-world complex environments, agents are prone to be erroneous when performing long-horizon tasks. To address this, we propose Experience-Driven Reflector, which is periodically activated to retrieve relevant multimodal experiences from AMEP. This encourages the agent to reflect on its current actions and refine the plan.We validate the performance of Optimus-1 in Minecraft, a popular open-world game environment. Experimental results show that Optimus-1 exhibits remarkable performance on long-horizon tasks, representing up to 30% improvement over existing agents. Moreover, we introduce various Multimodal Large Language Models (MLLMs) as the backbone of Optimus-1. Experimental results show that Optimus-1 has a 2 to 6 times performance improvement with the help of Hybrid Multimodal Memory, outperforming powerful GPT-4V baseline on lots of long-horizon tasks. Additionally, we verified that the plug-and-play Hybrid Multimodal Memory can drive Optimus-1 to incrementally improve its performance in a self-evolution manner. The extensive experimental results show that Optimus-1 makes a major step toward a general agent with a human-like level of performance. Main contributions of our paper:• We propose Hybrid Multimodal Memory module which is composed of HDKG and AMEP.HDKG helps the agent make the planning of long-horizon tasks efficiently. AMEP provides refined historical experience and guides the agent to reason about the current situation state effectively.• On top of the Hybrid Multimodal Memory module, we construct Optimus-1, which consists of Knowledge-Guided Planner, Experience-Driven Reflector, and Action Controller. Optimus-1 outperforms all baseline agents on long-horizon task benchmarks, and exhibits capabilities close to the level of human players.• Driven by Hybrid Multimodal Memory, various MLLM-based Optimus-1 have demonstrated 2 to 6 times performance improvement, demonstrating the generalization of Hybrid Multimodal Memory.2 Optimus-1In this section, we first elaborate on how to implement the Hybrid Multimodal Memory in Sec 2.1. As a core innovation, it plays a crucial role in enabling Optimus-1 to execute long-horizon tasks. Next, we give an overview of Optimus-1 framework (Sec 2.2), which consists of Hybrid Multimodal Memory, Knowledge-Guided Planner, Experience-Driven Reflector, and Action Controller. Finally, we introduce a non-parametric learning approach to expand the hybrid multimodal memory (Sec 2.3), thereby enhancing the success rate of task execution for Optimus-1. 
hybrid multimodal memory: 
 In order to endow agent with a long-term memory storage mechanism [28, 46] , we propose the Hybrid Multimodal Memory module, which consists of Abstracted Multimodal Experience Pool (AMEP) and Hierarchical Directed Knowledge Graph (HDKG).Relevant studies [23, 29, 17, 15] highlight the importance of historical information for agents completing long-horizon tasks. Minedojo [7] and Voyager [47] employed unimodal storage of historical information. Jarvis-1 [52] used a multimodal experience mechanism that stores task planning and visual information without summarization, posing challenges to storage capacity and retrieval speed. To address this issue, we propose AMEP, which aims to dynamically summarize all multimodal information during task execution. It preserves the integrity of long-horizon data while enhancing storage and retrieval efficiency. Specifically, as depicted in Figure 2 , to conduct the static visual information abstraction, the video stream captured by Optimus-1 during task execution is first input to a video buffer, filtering the stream at a fixed frequency of 1 frame per second. Based on the filtered video frames, to further perform a dynamic visual information abstraction, these frames are then fed into an image buffer with a window size of 16, where the image similarity is dynamically computed and final abstracted frames are adaptively updated. To align such abstracted visual information with the corresponding textual sub-goal, we then utilize MineCLIP [7] , a pre-trained video-text alignment model, to calculate their multimodal correlation. When this correlation exceeds a threshold, the corresponding image buffer and textual sub-goal are saved as multimodal experience into a pool. Finally, we further incorporate environment information, agent initial state, and plan generated by Knowledge-Guided Planner, into such a pool, which forms the AMEP. In this way, we consider the multimodal information of each sub-goal, and summarise it to finally compose the multimodal experience of the given task.In Minecraft, mining and crafting represent a complex knowledge network crucial for effective task planning. For instance, crafting a diamond sword requires two diamonds and one wooden stick , while mining diamonds requires an iron pickaxe , which involving further materials and steps. Such knowledge is essential for an agent's ability to perform long-horizon complex tasks. Instead of implicit learning through fine-tuning [33, 60] , we propose HDKG, which transforms knowledge into a graph representation. It enables the agent to perform explicit learning by retrieving information from the knowledge graph.As shown in the Figure 2 , we transform knowledge into a graph D(V, E), where nodes set V represent objects, and directed edges set E point to nodes that can be crafted by this object. An edge e ∈ E in the D can be represented as e = (u, v), where u, v ∈ V. The directed graph efficiently stores and updates knowledge. For a given object x, retrieving the corresponding node allows extraction of a sub-graph D j (V j , E j ) ∈ D, where nodes set V j and edges set E j can be formulated as:EQUATIONThen by topological sorting, we can get all the materials and their relationships needed to complete the task. This knowledge is provided to the Knowledge-Guided Planner as a way to generate a moreAbstracted Multimodal Experience Pool Hierarchical Directed Knowledge Graph Environment reasonable sequence of sub-goals. With HDKG, we can significantly enhance the world knowledge of the agent in a train-free manner.Relevant studies indicate that the human brain is essential for planning and reflection, while the cerebellum controls low-level actions, both crucial for complex tasks [40, 41] . Inspired by this, we divide the structure of Optimus-1 into Knowledge-Guided Planner, Experience-Driven Reflector, and Action Controller. In a given game environment with a long-horizon task, the Knowledge-Guided Planner senses the environment, retrieves knowledge from HDKG, and decomposes the task into executable sub-goals. The action controller then sequentially executes these sub-goals. During execution, the Experience-Driven Reflector is activated periodically, leveraging historical experience from AMEP to assess whether Optimus-1 can complete the current sub-goal. If not, it instructs the Knowledge-Guided Planner to revise its plan. Through iterative interaction with the environment, Optimus-1 ultimately completes the task.Knowledge-Guided Planner. Open-world environments vary greatly, affecting task execution.Previous approaches [51] using LLMs for task planning failed to consider the environment, leading to the failure of tasks. For example, an agent in a cave aims to catch fish. It lacks visual information to plan conditions on the current situation, such as "leave the cave and find a river". Therefore, we integrate environmental information into the planning stage. Unlike Jarvis-1 [52] and MP5 [33] , which convert observation to textual descriptions, Optimus-1 directly employs observation as visual conditions to generate environment-related plans, i.e., sub-goal sequences. This results in more comprehensive and reasonable planning. More importantly, Knowledge-Guided Planner retrieves the knowledge needed to complete the task from HDKG, allowing task planning to be done once, rather than generating the next step in each iteration. Given the task t, observation o, the sub-goals sequence g 1 , g 2 , g 3 , ..., g n can be formulated as:EQUATIONwhere n is the number of sub-goals, p η denotes sub-graph retrieved from HDKG, p θ denotes MLLM.In this paper, we employ OpenAI's GPT-4V as Knowledge-Guided Planner and Experience-Driven Reflector. We also evaluate other alternatives of GPT-4V, such as open-source models like Deepseek-VL [26] and InternLM-XComposer2-VL [6] in Section 3.4.Action Controller. It takes the sub-goal and the current observation as inputs and then generates low-level actions, which are control signals for the mouse and keyboard. Thus, it can interact with the game environment to update the agent's state and the observation. The formulation is as follows:EQUATIONwhere a k denotes low-level action at time k, p π denotes action controller. Unlike generating code [47, 33, 50] , generating control actions for the mouse and keyboard [1, 25, 52, 3] more closely resembles human behavior. In this paper, we employ STEVE-1 [25] as our Action Controller.Experience-Driven Reflector. EQUATIONwhere p ϵ denotes multimodal experience retrieved from AMEP. Experimental results in Section 3.3 demonstrate that the Experience-Driven Reflector significantly enhances the success rate of long-horizon tasks.During task execution, even in cases where task failure necessitates REPLAN, multimodal experiences are stored in AMEP. Thus, during the reflection phase, Optimus-1 can retrieve the most relevant cases from each of the three scenarios COMPLETE, CONTINUE, and REPLAN from AMEP as references. Experimental Results in Section 3.3 demonstrate the effectiveness of this innovative method of incorporating failure cases into in-context learning.To implement the Hybrid Multimodal Memory and enhance Optimus-1's capacity, we propose a nonparametric learning method named "free exploration-teacher guidance". In the free exploration phase, Optimus-1's equipment and tasks are randomly initialized, and it explores random environments, acquiring world knowledge through environmental feedback. For example, it learns that "a stone sword can be crafted with a wooden stick and two cobblestones ", storing this in the HDKG. Additionally, successful and failed cases are stored in the AMEP, providing reference experience for the reflection phase. We initialize multiple Optimus-1, and they share the same HDKG and AMEP. Thus the memory is filled up efficiently. After free exploration, Optimus-1 has basic world knowledge and multimodal experience. In the teacher guidance phase, Optimus-1 needs to learn a small number of long-horizon tasks based on extra knowledge. For example, it learns "a diamond sword is obtained by a stick and two diamonds " from the teacher, then perform the task "craft diamond sword". During the teacher guidance phase, Optimus-1's memory is further expanded and it gains the experience of executing complete long-horizon tasks.Unlike fine-tuning, this method enhances Optimus-1 incrementally without updating parameters, in a self-evolution manner. Starting with an empty Hybrid Multimodal Memory, Optimus-1 iterates between "free exploration-teacher guidance" learning and unseen task inference. With each iteration, its memory capacity grows, enabling mastery of tasks from easy to hard.Table 1 : Main Result of Optimus-1 on long-horizon tasks benchmark. We report the average success rate (SR), average number of steps (AS), and average time (AT) on each task group, the results of each task can be found in the Appendix F. Lower AS and AT metrics mean that the agent is more efficient at completing the task, while +∞ indicates that the agent is unable to complete the task. Overall represents the average result on the five groups of Iron, Gold, Diamond, Redstone, and Armor.Metric GPT-3.5 GPT-4V DEPS Jarvis- Benchmark. We constructed a benchmark of 67 tasks to evaluate the Optimus-1's ability to complete long-horizon tasks. As illustrated in Table 5 , we divide the 67 Minecraft tasks into 7 groups according to recommended categories in Minecraft. Please refer to Appendix D for more details.Baseline. We compare Optimus-1 with various agents, including GPT-3.52 , GPT-4V, DEPS [51] , and Jarvis-1 [52] on the challenging long-horizon tasks benchmark. In addition, we employed 10 volunteers to perform the same task on the benchmark, and their average performance served as a human-level baseline. Please refer to Appendix D.2 for more details about human-level baseline. For a more comprehensive comparison, we also report Optimus-1's performances on the benchmark used by Voyager [47] , MP5 [33] , and DEPS [51] in the Appendix F.2. Note that we initialize Optimus-1 with an empty inventory, while DEPS [51] and Jarvis-1 [52] have tools in their initial state. This makes it more challenging for Optimus-1 to perform the same tasks. Evaluation Metrics. The agent always starts in survival mode, with an empty inventory. We conducted at least 30 times for each task using different world seeds and reported the average success rate to ensure fair and thorough evaluation. Additionally, we add the average steps and average time of completing the task as evaluation metrics. 
experimental results: 
 The overall experimental results on benchmark are shown in Table 1 , see the accuracy for each task in Appendix F. Optimus-1 has a success rate near 100% on the Wood Group . Compared with Jarvis-1, Optimus-1 has 29.28% and 53.40% improvement on the Diamond Group and Redstone Group , respectively. Optimus-1 achieves the best performance and the shortest elapsed time among all task groups. It reveals the effectiveness and efficiency of our proposed Optimus-1 framework. Moreover, compared with all baselines, Optimus-1 performance was closer (average 5.37% improvement) to human levels on long-horizon task groups.We conduct extensive ablation experiments on 18 tasks, experiment setting can be found in Table 6 . As shown in Table 2 , we first remove Knowledge-Driven Planner and Experience-Driven Reflector, the performance of Optimus-1 on all task groups drops dramatically. It demonstrates the necessity of Knowledge-Guided Planner and Experience-Driven Reflector modules for performing long-horizon tasks. As for Hybrid Multimodal Memory, we remove HDKG from Optimus-1. Without the help of world knowledge, the performance of Optimus-1 decreased by an average of 20% across all task groups. We then removed AMEP, this resulted in the performance of Optimus-1 decreased by an average of 12%. Finally, we performed ablation experiments on the way of retrieving cases from AMEP. As shown in Table 3 , without retrieving cases from AMEP, the success rate shows an average of 10% decrease across all groups. It reveals that this reflection mechanism, which considers both success and failure cases, has a significant impact on the performance of Optimus-1. To illustrate the role of the reflection mechanism, we have shown some cases in Figure 4 .In this section, we explore an interesting issue: whether generic MLLMs can effectively perform various long-horizon complex tasks in Minecraft using Hybrid Multimodal Memory. As shown in Figure 5 , We employ Deepseek-VL [26] and InternLM-XComposer2-VL [6] as Knowledge-Guided Planner and Experience-Driven Reflector. The experimental results show that the original MLLM has low performance on long-horizon tasks due to the lack of knowledge and experience of Minecraft.With the assistance of Hybrid Multimodal Memory, the performance of MLLMs has improved by 2 to 6 times across various task groups, outperforming the GPT-4V baseline. This encouraging result demonstrates the generalization of the proposed Hybrid Multimodal Memory.As shown in Section 2.3, we randomly initialize the Hybrid Multimodal Memory of Optimus-1, then update it multiple times by using the "free exploration-teacher guidance" learning method. We set the epoch to 4, and the number of learning tasks to 160. At each period, Optimus-1 performs free exploration on 150 tasks and teacher guidance learning on the remaining 10 tasks, we then evaluate Optimus-1's learning ability on the task groups same as ablation study. Experimental results are shown in Figure 5 . It reveals that Optimus-1 keeps getting stronger through the continuous expansion of memory during the learning process of multiple periods. Moreover, it demonstrates that MLLM with Hybrid Multimodal Memory can incarnate an expert agent in a self-evolution manner [45] .4 Related Work 
agents in minecraft: 
 We summarise the differences of existing Minecraft agents in the Appendix D.3. Earlier work [30, 57, 2, 3] introduced policy models for agents to perform simple tasks in Minecraft. MineCLIP [7] used text-video data to train a contrastive video-language model as a reward model for policy, while VPT [1] pre-trained on unlabelled videos but lacked instruction as input. Building on VPT and MineCLIP, STEVE-1 [25] added text input to generate low-level action sequences from human instructions and images. However, these agents struggle with complex tasks due to limitations in instruction comprehension and planning. Recent work [50, 47, 61] incorporated LLMs as planning and reflection modules, but lacked visual information integration for adaptive planning. MP5 [33] , MineDreamer [60] , and Jarvis-1 [52] enhanced situation-aware planning by obtaining textual descriptions of visual information, yet lacked detailed visual data. Optimus-1 addresses these issues by directly using observation as situation-aware conditions in the planning phase, enabling more rational, visually informed planning. Additionally, unlike other agents requiring multiple queries for task refinement, Optimus-1 generates a complete and effective plan in one step with the help of HDKG. This makes Optimus-1 planning more efficient.In the agent-environment interaction process, memory is key to achieving experience accumulation [21] , environment exploration [16] , and knowledge abstraction [59] . There are two forms to represent memory content in LLM-based agents: textual form [17, 15, 31] and parametric form [5, 29, 48, 20] .In textual form, the information is explicitly retained and recalled by natural languages. In parametric form, the memory information [38] is encoded into parameters and implicitly influences the agent's actions. Recent work [49, 53, 12] has explored the long-term visual information storage [18, 19] and summarisation in MLLM. Our proposed hybrid multimodal memory module is plug-and-play and can provide world knowledge and multimodal experience for Optimus-1 efficiently. 
conclusion: 
 In this paper, we propose Hybrid Multimodal Memory module, which consists of two parts: HDKG and AMEP. HDKG provides the necessary world knowledge for the planning phase of the agent, and AMEP provides the refined historical experience for the reflection phase of the agent. On top of the Hybrid Multimodal Memory, we construct the multimodal composable agent, Optimus-1, in Minecraft. Extensive experimental results show that Optimus-1 outperforms all existing agents on long-horizon tasks. Moreover, we validate that general-purpose MLLMs, based on Hybrid Multimodal Memory and without additional parameter updates, can exceed the powerful GPT-4V baseline. The extensive experimental results show that Optimus-1 makes a major step toward a general agent with a human-like level of performance. 
limitation and future work: 
 In the framework of Optimus-1, we are dedicated to leverage proposed Hierarchical Directed Knowledge Graph and Abstracted Multimodal Experience Pool can be used to enhance the agent's ability to plan and reflect. For Action Controller, we directly introduce STEVE-1 [25] as a generator of low-level actions. However, limited by STEVE-1's ability to follow instructions and execute complex actions, Optimus-1 is weak in completing challenging tasks such as "beat ender dragon" and "build a house". Therefore, a potential future research direction is to enhance the instruction following and action generation capabilities of action controller.In addition, most of the work, including Optimus-1, utilize a multimodal large language model for planning and reflection, which then drives an action controller to perform the task. Building an end-to-end vision-language-action agent will be future work. 
 
 Appendix: With the increasing capability level of Multimodal Large Language Models (MLLM) comes many potential benefits and also risks. On the positive side, we anticipate that the techniques that used to create Optimus-1 could be applied to the creation of helpful agents in robotics, video games, and the web. This plug-and-play architecture that we have created can be quickly adapted to different MLLMs, and the proposed methods also provide a viable solution for other application areas in the agent domain. However, on the negative side, it is imperative to acknowledge the inherent stochastic nature of MLLMs in text generation. If not addressed carefully, this could lead to devastating consequences for society. Prior to deploying MLLMs in conjunction with the Hybrid Multimodal Memory methodology, a comprehensive assessment of their potential risks must be undertaken. We hope that while the stakes are low, works such as ours can improve access to safety research on instruction-following models in multimodal agents domains.Minecraft is an extremely popular sandbox video game developed by Mojang Studios3 . It allows players to explore a blockly, procedurally generated 3D world with infinite terrain, discover and extract raw materials, craft tools and items, and build structures or earthworks (shown in Figure 6 ). Minecraft is a valuable and representative environment for evaluating long-horizon tasks, offering greater diversity and complexity compared to other environments. Unlike web/app navigation [55] and embodied manipulation [16] ,Minecraft is an open world with a complex and dynamic environment (79 biomes, including ocean, plains, forest, desert, etc.). To complete long-horizon tasks, agents must achieve multiple sub-goals (e.g., 15 sub-goals to craft a diamond sword), making the construction of a Minecraft agent quite challenging. Many studies [47, 33, 52] have chosen Minecraft as the environment for validating performance on long-horizon tasks. Extensive experimental results in the paper show that Optimus-1 outperforms all baselines. Therefore, we chose Minecraft as open-world environment to evaluate the ability of agents to perform long-horizon tasks. Action Spaces. Our action space is almost similar to human players, except for craft and smelt actions. It consists of two parts: the mouse and the keyboard. The keypresses are responsible for controlling the movement of agents, such as jumping, forward, back, etc. The mouse movements are responsible for controlling the perspective of agents and the cursor movements when the GUI is opened. The left and right buttons of the mouse are responsible for attacking and using or placing items. In Minecraft, precise mouse movements are important when completing complex tasks that need open inventory or crafting table. In order to achieve both the same action space with MineDojo [7], we abstract the craft and the smelt action into action space. The detailed action space is described in Table 4 .Long-horizon Tasks are complex tasks that require world knowledge to solve and consist of multiple indispensable subtask sequences. In Minecraft, technology has six levels, including wood , stone , iron , golden , diamond , and netherite . Wooden tools can mine stone-level blocks, but can't mine iron-level and upper-level blocks. Stone tools can mine iron-level blocks, but can't mine diamond-level and upper-level blocks. Iron-level tools can mine diamond-level blocks, but can't mine netherite-level blocks. Diamond-level tools can mine any level blocks.For example, the agent now wants to complete the task "Craft iron sword ". The agent needs to craft wood-level tools to mine stone , and craft stone-level tools to mine iron ore . In order to craft tools, the agent needs a crafting table . To smelt iron ore into iron ingot , the agent needs a furnace . Moreover, craft crafting table needs 4 planks, and craft furnace needs 8 cobblestone. In summary, the agent needs to obtain many raw materials, wood-level and stone-level tools, 1 crafting table, 1 furnace, and most importantly, 2 iron ingots. The process of this task is shown in Figure 7 .In this section, we briefly introduce the relevant theory of cognitive science. For more details, please refer to the original articles.Our ability to understand and predict the world around us depends on our long-term memory stores, which have historically been divided into two distinct systems [28, 42, 46] . The semantic memory system provides a conceptual framework for describing the similar meanings of words and objects as they are encountered in different contexts (e.g., a bee is a flying insect with yellow and black stripes that produces honey), whereas the episodic memory system records our personal experiences characterized by the co-occurrence of words and objects at different times and places (e.g., being stung by a bee while eating honey at a picnic last weekend). These information stores and the interactions between them play a crucial role in guiding our behaviour and giving us the flexibility to adapt to the various demands of our environment.In this paper, inspired by the above theory, we divide the agent memory module into two parts: knowledge and experience. Based on this, we propose Hierarchical Directed Knowledge Graph and Abstracted Multimodal Experience Pool to enable the agent to acquire, store, and utilize knowledge and experience during the execution of tasks. Extensive experimental results demonstrate the effectiveness of the proposed methodology We constructed a benchmark of 67 tasks to evaluate Optimus-1's ability to complete long-horizon tasks in Minecraft. According to recommended categories in Minecraft, we have classified these tasks into 7 groups: Wood , Stone , Iron , Gold , Diamond , Redstone , Armor . The statistics for benchmark are shown in Table 5 . Due to the varying complexity of these tasks, we adopt different maximum gameplay steps (Max. Steps) for each task. The maximum steps are determined by the average steps that human players need to complete the task. Due to the randomness of Minecraft, the world and initial spawn point of the agent could vary a lot. In our benchmark setting, We initialize the agent with an empty inventory, which makes necessary for the agent to complete a series of sub-goals (mining materials, crafting tools) in order to perform any tasks. This makes every task challenging, even for human players.Note that Diamonds are a very rare item that only spawns in levels 2 to 16 and have a 0.0846% chance of spawning in Minecraft 1.16.5. Diamonds are usually found near level 9, or in man-made or natural mines no higher than level 16. In order to reduce the huge impact that diamond generation probability has on agent's likelihood of completing a task, we have adjusted the diamond generation probability to 20%, spawns in levels 2 to 16. This setting applies to human players as well.In the ablation study, we select the subset of our benchmark as the test set (shown in Table 6 ). The environment setting is the same as the benchmark.Existing Baseline. On the one hand, we employ GPT-3.5 and GPT-4V as baseline, which are evaluated without integrating hybrid multimodal memory modules. During the planning phase, they generate a plan for the action controller based on task prompt (and observation). During the reflection phase, they generate reflection results in a zero-shot manner. On the other hand, we compare existing SOTA Agents [51, 52] in Minecraft.Human-level Baseline. To better demonstrate agent's performance level in Minecraft, we hired 10 volunteers to play the game as a human-level baseline. The volunteers played the game with the same environment and settings, and every volunteer asked to perform the each task on the benchmark 10 times. Ultimately, we used the average scores of 10 volunteers as the human-level baseline. The results of the human-level baseline are shown in Table 1 . To ensure the validity of the experiment, we ensured that each volunteer had at least 20 hours of Minecraft gameplay before conducting the experiment. For each volunteer, we pay $25 as reward. MineRL T+V low-level action Voyager [47] NeurIPS' 23 Mineflayer T+V code DEPS [51] NeurIPS' 23 MineDOJO T+V code GROOT [3] ICLR' 24 MineRL T+V low-level action MP5 [33] CVPR' 24 MineDOJO T+V code Jarvis-1 [52] -MineRL T+V low-level action Optimus-1 -MineRL T+V low-level actionIn this section, we summarise the differences between existing Minecraft agents. As shown in the Table 7 , earlier work [1, 7, 25, 3] constructed Transformer-based policy network as agent. Recent work [47, 51, 52, 33] introduces the Multimodal Large Language Model, which empowers the agent to complete long-horizon tasks by exploiting the powerful language comprehension and planning capabilities of LLM.In the Mineflayer and Minedojo environments, agents [7, 47, 51, 33] can accomplish sub-goals by calling APIs (in the form of codes), which is a different behavioral pattern from humans. In MineRL [11] , agents [1, 25, 3, 52] must generate low-level actions to perform tasks, which is more challenging to accomplish long-horizon tasks.Moreover, existing agents lack knowledge and experience, and their performance in Minecraft is still vastly gapped from the human level. In this paper, we introduce Hybrid Multimodal Memory, which empowers Optimus-1 with hierarchical knowledge and multimodal experience. This makes Optimus-1 significantly outperform all existing agents on challenging long-horizon tasks benchmark, and exhibits near human-level performance on many tasks.Relevant studies [5, 29, 17, 15] have demonstrated the importance of memory for agents to complete long-horizon tasks. To implement the memory mechanism, Minedojo [7] and Voyager [47] only considered unimodal storage of historical information. Jarvis-1 [52] considered a multimodal memory mechanism to store task planning and visual information as experience, but it stores all historical information without summarisation. This approach stores all visual images, which poses a huge challenge in storage size and retrieval efficiency. To solve the problem, we propose the Abstracted Multimodal Experience Pool structure, which summarizes all historical information during the agent's execution of the task, which maintains the integrity of long sequential information and greatly improves the storage and retrieval efficiency of the experience.As shown in Figure 2 , we first input the visual image stream to the video buffer, which filters the image stream at a fixed frequency. It makes the length of the image stream substantially shorter. Empirically, we set the frequency of filtering to 1 second/frame, meaning that the video buffer takes one frame per second from the original image stream to compose the filtered image stream. We found that above this frequency makes the visual information redundant (too much similarity between images), and below this frequency does not preserve enough complete visual information.Then, we feed the filtered frames into an image buffer with a window size of 16. We dynamically compute the similarity between images in the image buffer, when a new image comes in, we compute the similarity between the new image and the most recent image, and then we remove the image with the highest similarity in order to keep the image buffer's window size to 16.Subsequently, we introduce MineCLIP [7] , a pre-trained model of video-text alignment with a structure similar to CLIP [35] , as our visual summariser. For a given sub-goal, it calculates the correlation between the visual content within the current memory bank and the sub-goal, and when this correlation exceeds a pre-set threshold, the frames within the memory bank are saved as the visual memories corresponding to that sub-goal. Finally, we store the visual memories with the sub goal's textual description into the Abstracted Multimodal Experience Pool. In addition, we incorporate the environment information, agent initial state, plan from Knowledge-Guided Planner, etc. into the experience memory of the given task. In this way, we consider the history information of each sub-goal and summaries and summarise it to finally compose the multimodal experience of the given task.Note that we also store these visual memories as failure cases when the feedback from the reflection phase is REPLAN. Therefore, when Optimus-1 executes a long-horizon task, it can retrieve past successes and failures as references and update memory after the task is finished. In the reflection phase, Optimus-1 retrieve the most relevant cases from Abstracted Multimodal Experience Pool, which contains the three scenarios COMPLETE, CONTINUE, and REPLAN, to help the agent better assess which state the current situation belongs to. This approach of considering both successful and failed cases for in-context learning is inspired by related research [8, 32] , and its effectiveness is validated in Section 3.3.As shown in the Figure 2 , crafting a diamond sword requires two diamonds and a wooden stick , while mining diamonds requires an iron pickaxe , which in turn requires additional raw materials and crafting steps. We transform this mine and craft knowledge into a graph structure, where the nodes of the graph are objects, and the nodes point to objects that can be crafted or completed by that object. With directed graph, we show that connections between objects are established, and that this knowledge can be stored and updated efficiently. For a given object, we only need to retrieve the corresponding node to extract the corresponding subgraph from the knowledge graph. Then by topological sorting, we can get the antecedents and required materials for the object, and this information is provided to the Knowledge-Guided Planner as a way to generate a more reasonable sequence of sub-goals. With Hierarchical Directed Knowledge Graph, we can significantly enhance the world knowledge of the agent in a train-free manner, as shown in the experimental results in Section 3.3.Our HDKG can be efficiently updated and expanded. When adding new nodes, the HDKG can be updated by simply merging the nodes and relationships into the graph. This method involves local linear modifications to the graph rather than altering the entire graph, making the process efficient and time-saving. For example, when M new nodes and N edges are added, the HDKG can be updated with M+N times of operations. Moreover, an HDKG containing 851 objects (nodes) requires less than 1 MB of memory. Thus, the HDKG can be efficiently updated and maintained.In order to implement the proposed Hybrid Multimodal Memory and to progressively increase the capacity of Optimus-1 in a self-evolution manner, we propose a non-parametric learning method named "free exploration-teacher guidance".In the free exploration phase, we randomly initialize the environment, materials, and tasks. For the task "craft a wooden pickaxe", we provide initial materials (three planks, two sticks), and then Optimus-1 (only the action controller activated) attempts to complete the task. If the environment feedback indicates the task is successful, the knowledge {3 planks, 2 sticks → wooden pickaxe} is added to the HDKG. Note that we randomly initialize materials and their quantities, which means that the task may not always succeed. As a result, each free exploration may not acquire the corresponding knowledge, but it can record the relevant experience (whether successful or fail). In the free exploration phase, Optimus-1 learns simple atomic operations, such as crafting sticks in the Wooden Group and mining diamonds in the Diamond Group.In the teacher guidance phase, Optimus-1 need to learn a small number of long-horizon tasks based on extra knowledge. For example, during the free exploration phase, Optimus-1 mastered crafting stick and mining diamond , but did not know that "a diamond sword is obtained by a stick and two diamonds ". So we provide some task plans, which will serve as extra knowledge to guide Optimus-1 to complete the task of "craft diamond sword". We built the following automated process to get the task plan needed for "free exploration":• We randomly select 5 tasks for each Group (7 groups in total) that are not included in the benchmark.• For each selected task, we use a script to automatically obtain the crafting relationships from the Minecraft Wiki4 . Taking the task "craft a wooden sword" as an example, we use the script to automatically obtain the crafting relationships: 1 wooden stick, 2 planks, 1 crafting table → 1 wooden sword, 1 log → 4 planks, 2 planks → 4 sticks, 4 planks → 1 crafting table.• These relationships are converted into a directed acyclic graph through an automated script. By performing a topological sort, the graph can be converted into tuples of materials and their quantities: (wooden sword, 1), (crafting table, 1), (wooden stick, 1) (planks, 8), (log, 2).• We prompt GPT-4 to construct a plan in order from basic materials to advanced materials.• Finally, we get the plan: 1. Get two logs 2. Craft eight planks 3. Craft a crafting table 4 . Craft a wooden stick 5. Craft a wooden sword During the teacher guidance phase, Optimus-1's memory is further expanded and it gains the experience of executing complete long-horizon tasks. Teacher guidance phase allows Optimus-1 to acquire advanced knowledge and learn multimodal experiences through complete long-horizon tasks.Optimus-1 consists of Knowledge-Guided Planner, Experience-Driven Reflector, and Action Controller. In this paper, we employ OpenAI's GPT-4V (gpt-4-turbo) 5 as Knowledge-Guided Planner and Experience-Driven Reflector, and STEVE-1 [25] as Action Controller. We also employ opensource models like Deepseek-VL [26] and InternLM-XComposer2-VL [6] as Knowledge-Guided Planner and Experience-Driven Reflector.All experiments were implemented on 4x NVIDIA A100 GPUs. We employ multiple Optimus-1 to perform different tasks at the same time, and this parallelized inference greatly improves our experimental efficiency. In the free exploration and teacher guidance phases, there is no need to access OpenAI's API, and the learning process takes approximately 16 hours on 4x A100 80G GPUs.During the inference phase, it takes about 20 hours on 4x A100 80G GPUs.Throughout the experiment, we spent about $5, 000 to access the GPT-4V API. However, we also offer more cost-effective solutions. As shown in Figure 5 , if we employ Deepseek-VL [26] or InternLM-XComposer2-VL [6] as Optimus-1's backbone, we can get comparable performance with low-cost!We show the prompt templates for Experience-Driven Reflector and Action Controller as follows.System: You are a MineCraft game expert and you can guide agents to complete complex tasks. User: For a given game screen and task, you need to complete <goal inference> and <visual inference>. <goal inference>: According to the task, you need to infer the weapons, equipment, or materials required to complete the task. <visual inference>: According to the game screen, you need to infer the following aspects: health bar, food bar, hotbar, environment. I will give you an example as follow:[Example] <task>: craft a stone sword. <goal inference>: stone sword <visual inference> health bar: full food bar: full hotbar: empty environment: forest Here is a game screen and task, you MUST output in example format. <task>: {task}. <game screen>: {image} Assistant: ============================== User: Now you need to make a plan with the help of <visual info> and <craft graph>. <visual info>: Consists of the following aspects: health bar, food bar, hotbar, environment.Based on the current visual information, you need to consider whether prequel steps needed to ensure that agent can complete the task. <craft graph>: a top-down list of all the tools and materials needed to complete the task. I will give you an example of planning under specific visual conditions as follow:[Example] {example} Here is a game screen and task, you MUST output in example format. Remember <task planning> MUST output in example format. <task>: {task} <game screen>: {image} <craft graph>: {graph} Assistant:Listing 1: Prompt for Knowledge-Guided Planner.System: You are a MineCraft game expert and you can guide agents to complete complex tasks.Agent is executing the task: {task}. Given two images about agent's state before executing the task and its current state, you should first detection the environment (forest, cave, ocean, etc.,) in which the agent is located, then determine whether the agent's current situation is done, continue, or replan. <done>: Comparing the image before the task was performed, the current image reveals that the task is complete. <continue>: Current image reveals that the task is NOT complete, but agent is in good state ( good health, not hungry) with high likelihood to complete task. <replan>: Current image reveals that the task is NOT complete, and agent is in bad state (bad health, or hungry) or situation (in danger, or in trouble), need for replanning. For replan, you need to further determine whether the agent's predicament is "drop_down" or "in_water". "drop_down" means that the agent has fallen into a cave or is trapped in a mountain or river, while "in_water" means that the agent is in the ocean and needs to return to land immediately.User: I'll give you some examples to illustrate the different situations. Each example consists of two images, where the first image is the state of the agent before performing the task and the second image is the current state of the agent.[Examples] <done>: {image1},{image2} <continue>: {image1},{image2} <replan>: {image1},{image2}Now given two images about agent's state before executing the task and its current state, you MUST and ONLY output in following format: Enviroment: <environment> Situation: <situation> (if situation is replan) Predicament: <predicament> Listing 2: Prompt for Experience-Driven Reflector.We list the results of each task on the benchmark below, with details including task name, sub-goal numbers, success rate (SR), average number of steps (AS), average time (AT), and eval times. All tasks are evaluated in Minecraft 1.16.5 Survival Mode. Note that each time Optimus-1 performs a task, we initial it with an empty initial inventory and a random start point. This makes it challenging for Optimus-1 to perform each task. Moreover, in MineRL [11] environment, 'steps' refers to the number of interactions between the agent and the environment, occurring at a frequency of 20 times per second. For example, if an agent takes 2 seconds to complete the task "chop a tree", it interacts with the environment 40 times, resulting in a recorded steps number of 40. Experimental results show that Optimus-1's average task completion step (AS) is significantly lower than other baselines.For a more comprehensive comparison with current Minecraft Agents, we also report Optimus-1's performances on the benchmark used by Voyager [47] , MP5 [33] , and DEPS [51] below. Due to the different environments and settings, agents perform tasks with varying degrees of difficulty. For example, Optimus-1 requires low-level action to perform any task in MineRL [11] , and we initialize its inventory to be empty. While Voyager [47] performs tasks in Mineflayer6 environment only through encapsulated code, MP5 [33] performs tasks in MineDOJO [7] environment only needs a specific control signal to craft tools, no low-level actions (mouse movement and click) are needed.Optimus-1's success rate in completing tasks with these baselines is shown in the Table 15 and Table 16 , and Optimus-1's efficiency in unlocking the tech tree in Minecraft is shown in the Figure 8 . These results reveal that Optimus-1 outperforms a variety of powerful baseline agents, even in challenging environmental settings! This section introduces several cases to comprehensively demonstrate Optimus-1's capabilities. When Optimus-1 determines that it has completed a task (such as "kill a cow " in Figure 10 ), it will finish the current task and move on to the next one. If Optimus-1 discovers that it has not yet completed the task and the task has not failed(as shown in Figure 11 ), it will continue executing the task. • Figure 12 demonstrates the importance of knowledge. For a long-horizon task such as "Mine 1 diamond ," Optimus-1 first generates a plan based on the Hierarchical Directed Knowledge Graph. However, this plan needs to be adjusted based on the current visual signals. For example, in this figure, Optimus-1 appears in a cave, so the primary task is not to "chop a tree" but to "leave the cave" first. Only after exiting the cave can Optimus-1 proceed with the initial plan. • Figure 13 demonstrates the high efficiency of our method. Agents like MP5 [33] and Voyager [47] use an iterative planning approach, which is very time-consuming, generating the final plan step by step. During this process, agent does not take any action. As shown in Figure 13 , a zombie is gradually approaching the agent, but the agent is still iterating on its plan. Optimus-1, however, generates the plan in one step based on the Hierarchical Directed Knowledge Graph and makes reasonable plans based on the current visual signals. Figure 10 : The process of completing the task "Find a cow and kill it". Hierarchical Directed Knowledge Graph indicates that having a wooden sword will make the task easier to complete. Therefore, Optimus-1 first crafts a wooden sword and then proceeds to find and kill a cow. The process of completing the task "Mine 1 diamond ". Mining diamonds is a highly complex task. Diamonds can only be mined with an iron pickaxe , so an iron pickaxe must be crafted first. Crafting an iron pickaxe requires iron ingots , which are smelted from iron ore .Mining iron ore requires a stone pickaxe . Crafting a stone pickaxe requires stone , which in turn must be mined with a wooden pickaxe . Crafting a wooden pickaxe requires wooden planks and sticks . All these crafting processes require a crafting table , and smelting requires a furnace . In this case, the agent spawns at a cave, so Optimus-1 must leave the cave to chop logs.Question: How to craft iron sword?Answer: Run to a sunny place first. And thenhttps://openai.com/research/gpt-3.5https://www.minecraft.net/en-us/article/meet-mojang-studioshttps://minecraft.wiki/https://openai.com/index/gpt-4v-system-card/https://github.com/PrismarineJS/mineflayer 


Decision: This paper introduces Optimus-1, designed to handle long-horizon tasks in Minecraft. The core innovation is a Hybrid Multimodal Memory module that manages knowledge and experience in such tasks. Additionally, a self-reflection mechanism enhances textual actions, which are then executed by a pre-trained action controller to produce low-level actions.

**Strengths**

* Novel concept of a hybrid multimodal memory module, which facilitates the management of knowledge and experience in long-horizon tasks.

*  Optimus-1 significantly outperforms all existing agents on long-horizon tasks in Minecraft.

**Weaknesses/suggestions**

* Limited Domain: Some components of the proposed pipeline are somewhat domain-specific (specialized to Minecraft), which raised concerns among reviewers regarding the generality of the work. The authors addressed these concerns by presenting additional results on mobile device control.

I think the paper makes a nice contribution that the community will find valuable. However, I encourage the authors to think carefully about how to reflect the comments or resolve the questions from reviewers in the camera ready version.

Reviews: review: paper_summary: paper_summary: Optimus-1 introduces an innovative Hybrid Multimodal Memory module that combines a Hierarchical Directed Knowledge Graph (HDKG) and an Abstracted Multimodal Experience Pool (AMEP) to address knowledge and experience management in long-horizon tasks.
strength_weakness: strength_weakness: * Improving long-horizon task performance through memory modules is common in LLMs; applying this directly to VLMs is not very novel.
* Reaction time and decision efficiency might be issues; the current experimental results still show a large number of steps.
* Although it performs well in the Minecraft environment, its performance in real-world applications, such as software manipulation and web navigation, has not been verified.
* As new knowledge and tasks emerge, effectively updating and maintaining the knowledge (HDKG) remains a challenge.
questions: questions: See weaknesses
limitations: limitations: Yes
paper_summary: paper_summary: The paper tackles the long-horizon tasks in Minecraft by building a pipeline based on multimodal LLM. Specifically, it proposes to store multimodal memory during agent exploration and a knowledge graph that stores the causal relations between objects and tasks. Additionally, a self-reflection mechanism is used to improve the textual actions. Finally, the textual actions are executed by a pre-trained action controller to output the low-level actions.
strength_weakness: strength_weakness: - There seem to be too many individual components in the paper, most of which are claimed to be important components in the paper, which makes it unclear to what extent the claimed contribution generalizes to broader settings beyond Minecraft (or even just beyond the evaluated tasks in Minecraft), as it is also highly plausible that the entire pipeline is carefully designed specifically for the evaluated tasks.
questions: questions: See "weaknesses" section above.
limitations: limitations: The limitations are described in the paper.
paper_summary: paper_summary: The paper proposes "Optimus-1" a Mulit-Modal LLM based agent. They evaluate their agent extensively on MineCraft and demonstrate superior performance compared to previous work. 

The key components of the approach are: 
1. A memory module consisting of structured memories (DAG) and multi-modal experience replay (with negative and positive examples)
2. A planing module
3. A reflection module
4. A "low-level" execution module

The novelty of the work consists of:
1. The DAG memory  
2. the positive & negative samples in the replay memory
3. Putting everything together in this way

Multiple ablation studies are made.
strength_weakness: strength_weakness: Weakness:
1. "teacher guidance learning" for HDKG -> It seems that teacher demonstrations are needed to make the agent perform the hard tasks. 
a. The paper (even incl. appendix) does not explain where these demonstrations come from and what the impact of these is. (Or how costly these demonstrations are to obtain).
b. Looking at ablation studies one can see that this Knowledge Module is key and performance drops by 20% (however, it is unclear how much the "teacher" contribution is. It seems that without the teacher contribution the model might perform quite poorly compared to baselines.) Therefore this indicates that expert human demonstrations are needed to actually make the agent work, as opposed to the "LLM" doing the reasoning and work.
c. Similarly, all further evaluations (incl. against other benchmarks in the appendix) are therefore somewhat questionable.

2. The actual learning mechanism of the DAG memory (HDKG) is not described in sufficient detail to replicate the work. It would be good to cite more references or explain this part in more detail. 

3. Positive and Negative examples in AMEP. While the paper argues this is an important contribution, it seems that it is nowhere evaluated if only positive examples would be more succesful.

4. Computational costs are only marginally mentioned ($5000 for OpenAI, and only 4xA100, but no amount of hours). It is not clear how much the "exploration" / learning phase is (time and money), what about the teacher phase? 

5. Evaluation outside of Minecraft would be interesting as well.
questions: questions: Questions:
1. How are teacher demonstrations obtained? How costly is this? How many human annotators and hours are needed for this?
2. How does the agent perform without teacher demonstrations?
3. What is the performance without negative samples in AMEP?
4. How long does the training / exploration phase take? How costly is it? (How many GPU hours on 4x A100 did it take?) How much is an evaluation?
5. How long would your method take to setup in a new environment altogether (and what would be the rough steps)? What is another good environment?
limitations: limitations: Limitations:
1. The authors speak about the limitations of one of the modules (the action generator), however, the teacher phase limitation mentioned above is not discussed. It seems that the method strongly depends on expert demonstrations and these are expensive to obtain in new environments outside of Minecraft.

2. Evaluations on other envs than Minecraft would have been interesting to discuss.
paper_summary: paper_summary: This paper presents Optimus-1, a multimodal agent that focuses on Minecraft tasks. Specifically,  Optimus-1 is equipped with a Hybrid Multimodal Memory including: 1) a Hierarchical Directed Knowledge Graph that stores the world knowledge through free exploration and teacher guidance; 2) an Abstracted Multimodal Experience Pool that enables Optimus-1 to reason about the current situation by using past experience. Based on the multimodal memory, Optimus-1 adopts a Knowledge-Guided Planner and an Experience-Driven Reflector to generate better plan and reflect periodically in long-horizion tasks. Experiments results illustrates the effectiveness of the multimodal memory of Optimus-1 in long-horizion tasks.
strength_weakness: strength_weakness: 1. The details of the construction procedure of AMEP are unclear (e.g. How to maintain the image buffer by computing image similarity? What is the threshold used in MineCLIP? )
2. How Optimus-1 acquires world knowledge through free-exploration is not specified.
3. There are other minecraft agents  (Jarvis-1, etc.) using multi-modal memory. The paper claims the efficiency of the memory storage and retrieval compared to Jarvis-1, but this is not quantitatively evaluated. 
4. It seems that the low-level action controller will not be updated through reflection, which limits the effectiveness of the reflection pipeline.
questions: questions: 1.  In the construction of AMEP, given the video stream, how to adaptively update the abstracted frames in the buffer? How is the threshold of MineCLIP determined?
2. In the free-exploration phase, what is the efficiency of Optimus-1 to learn an entry of world knowledge such as "a stone sword can be crafted with a wooden stick and two cobblestones" through random exploration?
3. In the reflection phase, how to retrieve past experience from AMEP?  Is the retrieval based on the image similartiy, task goal, or a combination of both? How does the fail case contribute to the final success of the task?
4. In Table 1, how is GPT-4V evaluated?  Does GPT-4V have the same world knowledge as Optimus-1?
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces Optimus-1, a retrieval augmented generation method that enables minecraft agents to excel in long-horizon tasks. The proposed method is based on a Hybrid Multimodal Memory Module that consists of an Abstracted Multimodal Experience Pool and a Hierarchical Directed Knowledge Graph. Those two key components are mainly proposed to solve the challenge of insufficient structured knowledge and lack of multimodal experience. Empirical evaluations show that such a memory module can effectively improve the performance of a minecraft agent in long-horizon planning tasks. Ablative experiments are conducted to validate the effective of each component.
strength_weakness: strength_weakness: The writing of this paper can be improved. In particular, I would suggest a major rewriting of the method section that 1) follows a top-down organization to first talk about high-level ideas of the method before going to the details such as "1 frame per second" and "a window size of 16", and 2) reduces unnecessary complications of the terminology including Hybrid Multimodal Memory Module, Hierarchical Directed Knowledge Graph, etc.

Major claims in the introduction are not supported by empirical evidence. It is unclear whether GPT4V etc does not have sufficient structured knowledge and related works in multi-modal agents do employ multi-modal experiences [1].

It seems that the use of Hierarchical Directed Knowledge Graph is limited to Minecraft where there is a strict Directed Graph relation between different objects. It is unclear whether this method can be helpful in the general setting.

The performances are evaluated on a set of custom benchmarks, and it will be good to have results on prior benchmarks reported by the baseline methods such as Voyager and Jarvis.

Minor:
Line 25 "Early research [1, 6, 17] developed simple agents by constructing policy networks." seems unclear what those policy networks are.

Line 30, what long horizon tasks are the authors talking about?

Line 36 grammar error "Insufficient of Structured Knowledge", what is empirical evidence for existing agents do not have structured knowledge?

Line 70 and line 71, two 30% seems repetitive.

Line 108, where is the subgoal coming from?

Line 115-116, what are those reflection phases?

In line 206, why did the authors contruct a new benchmark instead of following the prior benchmarks on minecraft?

[1] Zhang, C., Yang, Z., Liu, J., Han, Y., Chen, X., Huang, Z., … Yu, G. (2023). AppAgent: Multimodal Agents as Smartphone Users.

[2] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023

[3] Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. arXiv preprint arXiv:2311.05997, 2023.
questions: questions: See weakness
limitations: limitations: Limitations are not adequately discussed in the paper. One potential limitation is that the performance of RAG methods may be capped by the performance of the base model.

Comments: title: Responses to Reviewer 3HCc
Thank the authors for the rebuttal
Responses to Reviewer 3HCc
Response to Authors
Responses to Reviewer mEzT
Response to Reviewer 397y
Response
Response to Reviewer 397y
Responses to Reviewer mEzT
Responses to Reviewer 3HCc
Reply to Authors
Responses to Reviewer wzp9
Responses to Reviewer mEzT (3)
Response to Reviewer 397y
Responses to Reviewer 3HCc
Responses to Reviewer wzp9 (2)
Responses to Reviewer wzp9 (1)
Responses to Reviewer mEzT (2)
Responses to Reviewer mEzT (1)
Response to authors
Thanks for the rebuttal
comment: We are still in contact with the area chair and have not yet received permission to share the code. We sincerely hope you can recognize the efforts and sincerity we have put into addressing your concerns. We would greatly appreciate it if you could take these aspects into consideration and give us a higher score.

Thanks again for the time and effort you invested in the evaluation of our paper.
I have read the rebuttal and other reviews. Most of my concerns have been solved, so I will maintain my original score as 6: Weak Accept.
Thank you very much for taking the time to discuss with us despite your busy schedule. We would like to share the code with you, but we have noticed that NeurIPS's rebuttal guideline clearly states, 'All the texts you post (rebuttal, discussion, and PDF) should not contain any links to external pages.' Therefore, we will discuss the feasibility of sharing the code link with the area chair, and we hope you can understand our concerns.

Moreover, we would be grateful if you could focus more on our contribution of proposing the hybrid multimodal memory module and constructing the agent Optimus-1, which outperformed all powerful baselines in executing long-horizon tasks in Minecraft. This has been acknowledged by all reviewers (#3HCc, #397y, #mEzT, #AfEc, #wzp9).

To address your concerns, we have done our best to verify the effectiveness of the proposed method on the AppAgent benchmark and supplemented experiments on AitW at your additional request. These experimental results demonstrate the generalization of Optimus-1 in general scenarios. We sincerely hope you can see the efforts and sincerity we put into addressing your concerns. We have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).

Additionally, we are pleased to have addressed the concerns of other reviewers (#397y, #mEzT, #wzp9), and they have raised their scores. We would greatly appreciate it if you could take these aspects into consideration and give us a higher score.

Thanks again for the time and effort you invested in the evaluation of our paper.
Thanks for the experiments results. Would you like to share the implementation code and comprehensive trajectory records in an anonymous link, I will check with it and increase my score if the implementation is correct.
We deeply appreciate the time and effort you invested in the evaluation of our paper. 

We will revise the paper based on your and the Reviewer wzp9's suggestions, including: 
1. Add the comparison with existing Minecraft agents (e.g., Voyager) in Appendix E.3. 
2. Add a detailed explanation (how to construct and apply) of the HDKG and AMEP in Appendix F.1. 
3. Add experimental results in the app agent scenario in the Appendix. 
4. Revising the methodology section and other parts. 
5. Incorporating clarifications and implementation details discussed with the reviewers into the Appendix.

Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). 

Thank you again for your valuable suggestions, which are crucial for improving the quality of our paper.
Thank you for this detailed clarification and answer!

The score is raised from 7 to 8. 

We would recommend adding all of the above into the paper to make the paper clearer. Specifically:
1. The comparision against previous work (such as Voyager-1)
2. The detailed explanation of the various steps and how they work and what they involve.
3. Additional discussion above.
Thanks for your reply! We are pleased to have addressed your concerns. We will add these experiments in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).
Thank you for the response, and I appreciate the efforts for the new experiments. These have addressed my concerns and I'm raising my recommendation accordingly.
We would be grateful if you could take time out of your busy schedule to discuss with us. We are very keen to engage in deeper discussions with the reviewers.

**We further conducted experiments on AitW [1]**. We followed the environment settings of AppAgent and conducted comparative experiments on AitW. **As shown in Table below, Optimus-1 outperforms the PaLM 2, GPT-4V and AppAgent.** AitW is a popular, general benchmark that can demonstrate an agent's ability to operate apps in real-world scenarios. So the experimental results we provided below are sufficient to demonstrate Optimus-1's generalisation in real-world scenarios. We will include these results in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).

Tab 1: Experiments on the subset of AitW. We report the partial match scores for AitW Standard split.

| Method | Match Scores |
| --- | --- |
| PaLM 2 | 39.6 |
| GPT-4V | 50.5 |
| AppAgent | 52.4 |
| Optimus-1 | **58.3** |

We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating.

[1] Rawles et al. Android in the Wild: A Large-Scale Dataset for Android Device Control. 2023.
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows:

> Q1: How could you illustrate the automatic plan acquisition step more clearly? What is required beforehand (eg a Wiki, in what format?) what are the exact steps (i.e. input and output of each component).

A1: 

**Step 1: We randomly select 5 tasks for each Group (7 groups in total) that are not included in the benchmark.**

**Step 2:  For each selected task, we use a script to automatically obtain the crafting relationships from the Minecraft Wiki**. The pseudocode for the script is as follows:

```python
def get_information_from_wiki(item):
	html = get_html_from_wiki(f"https://minecraft.wiki/w/{item}")
	status, recipe = parser_crafting_from_html(html)
	if status == "failed": # means atomic item
		breaking = parser_breaking_from_html(html)
		can_break_tools = parser_tools_from_breaking(breaking)
		save_breaking(item, can_break_tools)
		return
	save_recipe(item, recipe)
	
	for sub_item in recipe:
		get_information_from_wiki(sub_item)
	
	
# get item's knowledge from wiki
item = "wooden_sword"

get_information_from_wiki(item)
```

Taking the task “craft a wooden sword” as an example, we use the script to automatically obtain the crafting relationships: {1 wooden stick, 2 planks, 1 crafting table → 1 wooden sword}, {1 log → 4 planks}, {2 planks → 4 sticks}, {4 planks → 1 crafting table}. 

**Step 3: These relationships are converted into a directed acyclic graph through the script below**. 

```python
def get_knowledge_graph(item):
	status, recipe = read_recipe(item)
	if stauts == "failed":
		tools = read_breaking_tools(item)
		for tool in tools:
			tool_graph[tool][item] = True
		return
	for sub_item in recipe:
		craft_graph[sub_item][item] = recipe.number[sub_item]
		get_knowledge_graph(sub_item)
		

# get item's knowledge graph
item = "wooden_sword"
craft_graph = {}
tool_graph = {}
get_knowledge_graph(item)

# merge 2 graph into an unify knowledge graph
kg = merge(item, craft_graph, tool_graph)
```

By performing a topological sort, the graph can be converted into tuples of materials and their quantities: (wooden sword, 1), (crafting table, 1), (wooden stick, 1) (planks, 8), (log, 2)

**Step 4: We prompt GPT-4 to construct a plan in order from basic materials to advanced materials**: 

```python
System Prompt: You are an expert in Minecraft and you can efficiently make plans for me to complete challenging tasks.

User Prompt: I need to complete the task {Task}. Here are the materials needed and their quantities {Materials}. 
Please make a feasible plan for me in the order from basic materials to advanced materials:
```

Finally, we get the plan: {1. Get two logs 2. Craft eight planks 3. Craft a crafting table 4. Craft a wooden stick 5. Craft a wooden sword}

We have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance). And we will add these implementation details in the Appendix.

> Q2: Does Optimus-1 exceed the capabilities of the teacher alone? How meaningful are these new capabilities?

A2: Actually, 'teacher' refers to the internal knowledge of the Minecraft environment. In the free exploration phase, Optimus-1 acquires basic knowledge through environmental feedback, while in the teacher guidance phase, Optimus-1 obtains advanced knowledge through the automated process described in A1. This knowledge cannot directly transform into the capabilities in Minecraft. So we constructed Optimus-1, which includes modules such as the Knowledge-guided Planner, Experience-Driven Reflector, and Action Controller, to transform this knowledge into the capabilities for the agent to execute long-horizon tasks in Minecraft, through the reasoning and reflection of multimodal large language model.

**Figure 5(b) in the manuscript** shows that Optimus-1's performance continually improves during the multi epoches 'free exploration-teacher guidance' learning process. This indicates that Optimus-1 can utilize past memories (knowledge and experience) to gradually enhance its performance on unseen tasks. **Table 2 in the manuscript** shows that the performance of Optimus-1 significantly decreases after removing such knowledge (e.g., from 9.5% to 1.8% on the Diamond Group). Therefore, knowledge has a significant impact on Optimus-1's capabilities to execute long-horizon tasks.

We hope that these explanation can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating.
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows:

> Q: AppAgent can also achieve the similar results with "Watching Demos". Would you like to explore the performance of Optimus-1 in AitW[1] and Osworld[2], it would be more challenging and persuasive.

A: (1) **The experimental results in the Table below show that Optimus-1 outperforms AppAgent, even when compared to AppAgent with Watching Demos.**

Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks.

| Method | Success Rate |
| --- | --- |
| GPT-4 | 48.9% |
| AppAgent-Auto. Exploration | 73.3% |
| AppAgent-Watching Demos | 84.4% |
| Optimus-1 | **86.7%** |

(2) **We further conducted experiments on AitW [1]**. We followed the environment settings of AppAgent and conducted comparative experiments on AitW. **As shown in Table below, Optimus-1 outperforms the PaLM 2, GPT-4V and AppAgent.** It is important to note that our method is train-free, and due to time constraints, the experiments were conducted under insufficient “free exploration-teacher guidance” learning conditions. So it is unfair to compare it to baselines that fine-tuned on AitW dataset. We will include these results in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).

Tab 2: Experiments on the subset of AitW. We report the partial match scores for AitW Standard split.

| Method | Match Scores |
| --- | --- |
| PaLM 2 | 39.6 |
| GPT-4V | 50.5 |
| AppAgent | 52.4 |
| Optimus-1 | **58.3** |

(3) Since conducting experiments on OSWorld [2] requires a virtual machine and the environment configuration is quite complex, we are unable to provide experimental results for Optimus-1 on OSWorld in such short time. However, AitW is a popular, general benchmark that can demonstrate an agent's ability to operate apps in real-world scenarios. So the experimental results we provided above are sufficient to demonstrate Optimus-1's generalisation in real-world scenarios. We would like to provide experiments in more scenarios in future versions to demonstrate the generalisation of the proposed method. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).

We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating.

[1] Rawles et al. Android in the Wild: A Large-Scale Dataset for Android Device Control. 2023.

[2] Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. 2024.
Thanks for the additional experiments. Actually, AppAgent can also achieve the similar results with "Watching Demos". Would you like to explore the performance of Optimus-1 in AitW[1] and Osworld[2], it would be more challenging and persuasive.

[1] Android in the Wild: A Large-Scale Dataset for Android Device Control
[2] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
Thank you very much clarifying your work.

It sounds like excellent work indeed. Perhaps, some parts need further clarification and simplification. (Such as the initial plan acquisition step, at this stage it would be hard to replicate this step from your paper and discussion).

Follow questions:
1. How could you illustrate the automatic plan acquisition step more clearly? What is required beforehand (eg a Wiki, in what format?) what are the exact steps (i.e. input and output of each component).
2. Does Optimus-1 exceed the capabilities of the teacher alone? How meaningful are these new capabilities?
We are pleased to have addressed your concerns. We will incorporate your valuable suggestions in future revisions and add more experiments in the Appendix. Additionally, we have built an official repository to provide well-structured open-source codes and a project page (to be released upon acceptance).
Thank the authors for the additional response on the clarity of the paper and results on real-world app navigation. I am more convinced of the proposed method being general and increase the score. I hope that those new results and rewriting can be reflected in a future version of the paper.
### Additional experimental results

As you and some reviewers mentioned the feasibility of adapting Optimus-1 to other domains, we added additional experiments to demonstrate the generalisation of our method. We applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [1] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents.

Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks.

| Method | Success Rate |
| --- | --- |
| GPT-4 | 48.9% |
| AppAgent | 73.3% |
| Optimus-1 | **86.7%** |

We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating.
### Additional experimental results

To demonstrate that our Hybrid Multimodal Memory is easily adaptable to other domains, we applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [1] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents.

Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks.

| Method | Success Rate |
| --- | --- |
| GPT-4 | 48.9% |
| AppAgent | 73.3% |
| Optimus-1 | **86.7%** |

We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. 

We welcome the reviewer to continue the discussion with us during the discussion phase. If you have any further questions, please feel free to contact us.

P.S.: We just realized that it was not visible to you due to a setting error. We sincerely apologize for any inconvenience this may have caused and for not responding to your question in a timely manner.
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows:

Q1:  I look forward to seeing some experiments in software manipulation or web navigation based on Optimus-1 framework and also open to further modify the score.

A1: To address your concerns about the generalisation of our method in real-world scenarios, we applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [1] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents.

Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks.

| Method | Success Rate |
| --- | --- |
| GPT-4 | 48.9% |
| AppAgent | 73.3% |
| Optimus-1 | **86.7%** |

We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. If you have any further questions, please feel free to contact us.

[1] Zhang et al. AppAgent: Multimodal Agents as Smartphone Users. 2023.
This comment connects to the Responses to Reviewer wzp9 (1)

> Q5: It is unclear whether such designs can generalize to more realistic settings like real-world web navigation agents or robotics.

A5: (1) To address your concerns about the generalisation of our method in real-world scenarios, we applied Optimus-1 to the app agent scenario. We followed the environment and settings of AppAgent [2] and conducted comparative experiments on its benchmark (9 apps with a total of 45 tasks). The experimental results in the table below show that Optimus-1 outperforms AppAgent and GPT-4 baselines. This reveals that Optimus-1 can generalize to more realistic settings, such as real-world app navigation agents.

Tab 1: Experiments on the benchmark of AppAgent. We report the success rate of the agent in completing 45 tasks.

| Method | Success Rate |
| --- | --- |
| GPT-4 | 48.9% |
| AppAgent | 73.3% |
| Optimus-1 | **86.7%** |

(2) Proposed Hybrid Multimodal Memory is a general architecture. As mentioned in lines 31 to 35 in the manuscript, we were inspired by the theory that 'humans benefit from knowledge and experience when performing long-sequence tasks' , and propose a novel Hybrid Multimodal Memory structure that incorporates both knowledge and experience into the memory mechanism. We argue that it is general and can be adapted according to different environments. In the experiments mentioned above in the app scenario, the key step is obtaining the logical relationships between buttons or actions and converting them into a knowledge graph. Once these logical relationships are established, HDKG can easily be adapted to the app environment. As for AMEP, it can be simplified to store the task prompt, images, and actions for each atomic operation.

We hope that the results of these experiments can address your concerns, and we would greatly appreciate it if you could consider giving us a higher rating. If you have any further questions, please feel free to contact us.

[1] Wang et al. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. 2023.

[2] Zhang et al. AppAgent: Multimodal Agents as Smartphone Users. 2023.
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows:

> Q1: Currently the description of the technical section is very convoluted and can be better organized. 

A1: We appreciate your valuable and constructive feedback, which will be pivotal in enhancing the quality of our work. To present the methodology with clear logic, we first introduce the proposed hybrid multimodal memory module, followed by a detailed description of the Optimus-1 architecture. The hybrid multimodal memory is coupled with the operation mechanism of Optimus-1, while the current version lacks detailed descriptions of sub-goals, the reflection module, etc. We will add these descriptions in Section 2.1.1 to ensure logical coherence and avoid reader confusion. Additionally, in each subsection, we will organize the content in a top-down manner, following the sequence of motivation, high-level idea of the method, and implementation details. We will revise the manuscript based on your suggestion. We also note that the reviewers found the manuscript well-written and easy to follow (R#397y, R#AfEc), so these revisions will not affect the main contribution of our work.

> Q2: Some of the terminologies can be simplified.

(1) **These terminologies reflect the characteristics and innovations of proposed methods**. For example, the Hierarchical Directed Knowledge Graph links knowledge at different levels (wooden, stone, diamond) through **Directed** graphs, forming a **Hierarchical** knowledge graph, which differs from previous knowledge graphs. The Abstracted Multimodal Experience Pool **Abstracts** long-sequence multimodal historical information into multimodal experiences, whereas existing multimodal memory mechanisms do not summarize multimodal information.

(2) **We can use abbreviations in the manuscript to describe these terminologies**. Although we introduced the Hierarchical Directed Knowledge Graph (HDKG) and the Abstracted Multimodal Experience Pool (AMEP) in Section 1 of the manuscript, we have frequently used the full names in subsequent sections out of concern that readers might forget or misunderstand the meanings of HDKG and AMEP. We will consider using abbreviations in the manuscript to describe these terminologies to facilitate easier reading for the readers.

> Q3: Difference between Hybrid Multimodal Memory and just Multimodal Memory.

A3: **Hybrid Multimodal Memory is different from existing multimodal memory**. As stated in the Introduction section of the manuscript, the hybrid multimodal memory module consists of structured knowledge (graphs) and multimodal experiences (text, image sequences). It stores multiple heterogeneous contents in a **mixed manner** and dynamically summarizes long-sequence multimodal information. 

In contrast, existing agents only store text and images as multimodal memory and do not summarize them. For example, Jarvis-1 [1] stores text and image sequences without summarizing multimodal information. 

Moreover, the comparison of memory mechanisms in existing Minecraft agents is shown in **Table 7 of the manuscript**.

> Q4: Ensure the major claims in the paper are scientific as supported by the experiments.

A4: Thank you for your suggestions. We will revise the claim in the Introduction section in the manuscript: “Existing Multimodal Large Language Models such as GPT-4V lack sufficient knowledge in Minecraft”, and results of **Table 2 in the manuscript** support this claim.

**The rest of the responses are in the next comment**.
This comment connects to the **Responses to Reviewer mEzT (1)**

> Q4: Could you compare your method in more detail to works such as Voyager, specifically in terms of manual effort needed for the various phases and comparison of results with human effort vs. no human effort (especially on the harder long-range tasks).

A4: (1) **Neither Voyager nor our Optimus-1 requires human annotation**. As stated in A1, we obtain the plans required for the teacher guidance phase through an automated process. This automated process is simple and easy to implement, and it takes approximately two hours.

(2) As mentioned above, we do not require human effort to obtain the plans. So we cannot provide a comparison of results with human effort versus no human effort.

(3) More detailed comparison between Optimus-1 and Voyager: Firstly, Voyager executes sub-goals in the Mineflayer environment by calling APIs (in the form of code), while Optimus-1 uses an action controller to generate low-level actions like a human, which is more challenging. Secondly, Voyager acquires knowledge through environment feedback during both learning and reasoning processes. For instance, when it fails to execute the task 'craft a diamond sword,' it acquires knowledge such as 'two more diamonds are needed' from the environment feedback. In contrast, Optimus-1 learns whether a task is successfully executed through environment feedback only during the learning phase and requires only a small number of plans to complete the learning process.

Tanks for your discussions with us. If you have any further questions, please feel free to contact us.
Thank you very much for taking the time to discuss with us despite your busy schedule. Regarding your concerns, we respond as follows:

> Q1: Example of how a plan is constructed during the teacher guidance phase. Is an LLM used for this? Are these constructed by a human?

A1: **We obtain the plans required for the teacher guidance phase through an automated process**.  For each task group, we randomly select 5 tasks that are not included in the benchmark. Taking the task “craft a wooden sword” as an example, We use a script to automatically obtain the crafting relationships for a wooden sword from the Minecraft Wiki: {1 wooden stick, 2 planks, 1 crafting table → 1 wooden sword}, {1 log → 4 planks}, {2 planks → 4 sticks}, {4 planks → 1 crafting table}. These relationships can be represented as a directed acyclic graph. Then, by performing a topological sort, the graph can be converted into tuples of materials and their quantities: (wooden sword, 1), (crafting table, 1), (wooden stick, 1) (planks, 8), (log, 2). Finally, we prompt GPT-4 to construct a plan in order from basic materials to advanced materials: {1. Get two logs 2. Craft eight planks 3. Craft a crafting table 4. Craft a wooden stick 5. Craft a wooden sword}. The entire process is automated, simple, and easy to implement. We only need to construct 5 (tasks/group) x 7 (groups) = 35 plans in the teacher guidance phase, which takes approximately two hours.

> Q2: Does the free exploration phase happen after the initial expert demonstrations update the HDKG? How does it happen without any initial demonstrations - what is the prompt, or few-shot example used? How is it constructed (human annotated)? 

A2: As described in **Section 2.3 in the manuscript**, we initialize the Hybrid Multimodal Memory as **empty and begin with free exploration to acquire basic knowledge**, such as crafting sticks and mining diamonds. We then proceed to **teacher guidance phase to learn advanced knowledge**, e.g., a diamond sword is obtained by a stick and two diamonds. The entire process does not require additional prompts, few-shot examples, or manual annotations.

**In the free exploration phase**, we randomly initialize the environment, materials, and tasks. For the task “craft a wooden pickaxe”, we provide initial materials (three planks, two sticks), and then Optimus-1 (only the action controller activated) attempts to complete the task. If the environment feedback indicates the task is successful, the knowledge {3 planks, 2 sticks → wooden pickaxe} is added to the HDKG. Note that we randomly initialize materials and their quantities, which means that the task may not always succeed. As a result, each free exploration may not acquire the corresponding knowledge, but it can record the relevant experience (whether successful or fail). In the free exploration phase, **Optimus-1 learns simple atomic operations, such as crafting sticks in the Wooden Group and mining diamonds in the Diamond Group.** This phase is insufficient for Optimus-1 to learn advanced knowledge, such as crafting a diamond sword.  

**In the teacher guidance phase**, Optimus-1 executes each sub-goal sequentially according to the given plan. Once the task is completed, the materials and their corresponding relationships (e.g., {1 wooden stick, 2 diamonds} → {1 diamond sword}) are updated in the HDKG, and the multimodal experience of each sub-goal is stored in the AMEP. **Teacher guidance phase allows Optimus-1 to acquire advanced knowledge and learn multimodal experiences through complete long-horizon tasks.**

> Q3: Do you mean by updating the HDKG, that a fact (i.e. triplet) is added to an actual KG?

A3: **By updating the HDKG, triplets are added to an actual KG**. For example, the current KG contains (2 planks, 'craft', 4 stick), (wooden pickaxe, 'mine', stone). When new knowledge is acquired: (2 stick, 'craft', 1 stone pickaxe), (3 stone, 'craft', 1 stone pickaxe), (1 crafting table,'needed', stone pickaxe). These 3 triples added to current KG, then updated as: (1 planks, 'craft', 4 stick), (wooden pickaxe, 'mine', stone),  (2 stick, 'craft', 1 stone pickaxe), (3 stone, 'craft', 1 stone pickaxe), (1 crafting table,'craft', stone pickaxe).

**The rest of the responses are in the next comment**.
Thanks for authors' rebuttal. I would will change my score to 4. I look forward to seeing some experiments in software manipulation or web navigation based on Optimus-1 framework and also open to further modify the score.
Thank the authors for the rebuttal. Thanks for pointing me to the comparisons in prior benchmarks. The rebuttal has clarified some of my concerns and I will adjust my score accordingly. However, there are still two important concerns remaining:

### The writing of the paper can be improved.
As discussed above, the writing can be significantly improved. Currently the description of the technical section is very convoluted and can be better organized. Some of the terminologies are overly complicated, e.g. Hybrid Multimodal Memory, what's its difference with just Multimodal Memory? Abstracted Multimodal Experience Pool can be simplified to Multimodal Experience Pool. IMHO, such complication in terminology only makes it harder for readers to understand. 

Furthermore, it would be great to ensure the major claims in the paper are scientific as supported by the experiments. In particular, the experiments only show that it is important for the agent to know the rules of Minecraft (e.g. how are materials synthesized) and they are not enough to show GPT4V does not have structured knowledge. 

A significant effort is required for the work to be able to be published at top-tier ML conference.

### The application beyond minecraft remains unclear.
In particular, a potential reason for HDKG to work well in minecraft is that there is a very clear game rule in minecraft (e.g. to make a stone axe we need stone and wooden sticks etc), and it is very unclear to me whether such designs can generalize to more realistic settings like real-world web navigation agents or robotics. I did read the response to reviewer 3HCc but I am not convinced that a better method for minecraft is interesting itself, unless it can be easily generalized to more realistic settings.
Thank you for taking the time and effort to address all questions. Things are clearer at this stage. There are a few follow-up questions.

A1:
> (1) We obtain teacher demonstrations from Minecraft Wiki. For each task group, we randomly select 5 tasks that are not included in the benchmark. We then create corresponding task plans based on content (synthesis relationship between objects) from the Minecraft Wiki

a.) Could you please describe one simple example of how such a plan is constructed. Is an LLM used for this? Are these constructed by a human?

A3:
> (2) During the free exploration phase, Optimus-1 will randomly initialize the environment, materials, and tasks.

a. )Does this happen after the initial expert demonstrations update the HDKG? 
b.) How does it happen without any initial demonstrations - what is the prompt, or few-shot example used? How is it constructed (human annotated)?

---
Additional questions (AQ):
a.) Do you mean by updating the HDKG, that a fact (i.e. triplet) is added to an actual KG?
b.) Could you compare your method in more detail to works such as Voyager, specifically in terms of manual effort needed for the various phases and comparison of results with human effort vs. no human effort (especially on the harder long-range tasks).