MCU: An Evaluation Framework for Open-Ended Game Agents
Authors: Xinyue Zheng
Haowei Lin
Kaichen He
Zihao Wang
QIANG FU
Haobo Fu
Zilong Zheng
Yitao Liang
Venue: ICML 2025 spotlightposter (2025)
Keywords: benchmark
automatic evaluation
agent
Primary area: general_machine_learning->evaluation

Abstract: Developing AI agents capable of interacting with open-world environments to solve diverse tasks is a compelling challenge. However, evaluating such open-ended agents remains difficult, with current benchmarks facing scalability limitations. To address this, we introduce \textit{Minecraft Universe} (MCU), a comprehensive evaluation framework set within the open-world video game Minecraft. MCU incorporates three key components: (1) an expanding collection of 3,452 composable atomic tasks that encompasses 11 major categories and 41 subcategories of challenges; (2) a task composition mechanism capable of generating infinite diverse tasks with varying difficulty; and (3) a general evaluation framework that achieves 91.5\% alignment with human ratings for open-ended task assessment. Empirical results reveal that even state-of-the-art foundation agents struggle with the increasing diversity and complexity of tasks. These findings highlight the necessity of MCU as a robust benchmark to drive progress in AI agent development within open-ended environments. Our evaluation code and scripts are available at https://github.com/CraftJarvis/MCU.

Developing Al agents capable of interacting with dynamic environments- referred to as "open-world" in the lit- erature (Parmar et al., 2023)-t2o to solve diverse tasks remains long-standing challenge in Artificial Intelligence (Kejri- wal et al., 2024). Among the various environments used to study AI agents, games have emerged as a prominent MCU: An Evaluation Framework for Open-Ended Game A Agents
Xinyue Zheng" Haowei Lin
Kaichen He?
Zihao Wang? Qiang Fu Yitao Liang
Haobo Fu Zilong Zheng"
Abstract
choice, they provide real-world challenges within pro- grammable simulators, offering valuable opportunities for real-world simulation (Bruce et al., 2024; Reed et al., 2022; Raad et al., 2024). Compared to other digital environments such as web/apps (Zhou et al., 2023; Qin et al., 2025; Lin et al., 2024), mobile platforms (Pan al., 2024; Wang et al., 2024), and coding IDEs (Xu al., 2022; Jimenez al., 2024; Huang et al., 2023), games present higher degree of control complexity, similar that encountered by physical robotic agents (Nasiriany et al., 2024; Wang et al., 2023b; Zhou et al., 2024b). While compared to robotics, games support long-horizon planning tasks and enable safer and more efficient testing within sandbox environments.
Developing Al agents capable of interacting with open-world environments to solve diverse tasks is compelling challenge. However, evaluating such open-ended agents remains difficult, with current benchmarks facing scalability limitations. address this, we introduce Minecraft Universe (MCU), a comprehensive evaluation framework set within the open-world video game Minecraft. MCU incorporates three key components: (1) an expanding collection of 3,452 composable atomic tasks that encompasses 1l major categories and 41 subcategories of challenges; (2) a task compo- sition mechanism capable of generating infinite diverse tasks with varying difficulty; and (3) a general evaluation framework that achieves 91.5% alignment with human ratings for open-ended task assessment. Empirical results reveal that even state-of-the-art foundation agents struggle with the increasing diversity and complexity of tasks. These findings highlight the necessity of MCU as robust benchmark to drive progress in Al agent development within open-ended environments. Our evaluation code and scripts available https: / /ttp//ggttttttt/e//// com CCr.ftJarvis//........
A crucial aspect of open-ended game agents is their gen- eralizability. The ultimate goal of developing Al agents is to deploy them in real, open- world environments, where they must solve tasks robustly across drastically different situations. With this in mind, early game agents have been studied in procedurally generated Atari-like environments with diverse configurations (Cobbe et al., 2020). Recent efforts shift toward complex environments with greater free- dom (Fan et al., 2022; Hafner, 2021; Team et al., 2021). among which Minecraft stands out. Minecraft is vidco game that provides procedurally generated open-world en- vironments with a state space exceeding the number of atoms the universe (we elaborate on the henefits using Minecraft for open-ended game agents in Section 2.1). This vast environment allows agents to tackle infinite open-ended tasks using human-like actions diverse situations.
1. Introduction
Despite the numerous advantages of Minecraft as perimental environment, we have identified several limita- tions in existing benchmarks that impede comprehensive evaluation of agents' generalizability. These limitations include low task quality (Fan et al., 2022), insufficient di- versity (Hafner, 2021), and the lack of automatic evaluation suites (Cai al., 2024c). We further illustrate these issues in Figure 2. Based these issues, we find that Minecraft agents lack unified benchmark, and each agent is evaluated on a distinct set of tasks (see Section 4 for further discus- sion). To address this, we introduce Minecraft Universe (MCU), an advanced evaluation framework in Minecraft (Figure 1). MCU encompasses thousands of composable
Equal contribution. & Equal contribution as co-coreepnnning authors. Beijing Institute for General Artificial Intelligence (BI- GAI), Beijing, China 'Institute for Artificial Intelligence, Peking University, Beijing, China "Tencent Lab, Shenzhen, China. Cor- respondence to: Zilong Zheng <zlzheng @ ai>, Yitao Liang <yitaol @ pku.edu.cn>>
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MCU: An Evaluation Framework for Open-Ended Game Agents
tasks and provides a scalable automatic evaluation system, designed based the following key principles:
Enduring Challenges To ensure that MCU remains long-term benchmark for agent development, we adopt two key strategies: (I) designing tasks with varying levels of dif- ficulty, where increasing complexity introduces additional challenges such as adverse weather conditions and mislead- ing factors. While state-of-the-art models achieve moderate success on simpler tasks, they struggle with more complex scenarios (Figure 1d). (2) Enabling the composition of atomic tasks into more intricate tasks. This approach expo- nentially increases both the number and complexity of tasks, ensuring that MCU continues to provide lasting challenge.
High Task Diversity Task diversity plays crucial role in: (1) simulating challenges across various scenarios to thoroughly evaluate generalizability, and (2) harnessing the full potential of open-ended game agents capable of solving wide range of tasks. MCU emphasizes two key aspects of task diversity in its implementation: (1) Intra-task diver- sity: We utilize large language model (LLM) to dynami- cally generate task initialization conditions-such as biome, weather, and player states-introducicn variability and ran- domness that closely mirror real gameplay scenarios (Fig- ure Ia). (2) Inter-task diversity: We collect 3,452 atomic tasks within Minecraft, spanning a broad spectrum of chal- lenges, including precise control (e.g,, combat, building), logical reasoning (e.g., crafting, trading), and knowledge application (e.g,, tool use, animal care) (Figure lb). These tasks can further be composed to generate new tasks on a combinatorial scale as detailed in Section 2.3.
2. MCU Benchmark
In this section, we first provide an overview of Minecraft and its game simulator, followed by detailed outline of the construction process for the MCU benchmarking pipeline.
2.1. Minecraft Environment
Minecraft is voxel-based 3D video game that, due to its popularity and wide variety of mechanics, has become prominent domain for RL research (Oh et al., 2016; Tessler et al., 2017). Much of the prior work focuses on small, custom-built Minecraft environments with tasks such as nav- igation (Arumugam et al., 2019; Matiisnn et al., 2019), block placement (Trott et al., 2019; Alaniz, 2018), combat (Uda- gawa et al., 2016), and other similar activities (Shu et al., 2017). More recent efforts have shifted towards studying the full, unmodified human action space, which encompasses tasks like drag-and-drop inventory management and item crafting. In this work, we employ unmodified Minecraft ver- sion 1.16.5 as our testing environment (Guss et al., 201 (b)9,,, utilizing mouse and keyboard inputs as the action space and 640 x 360 RGB image as the observation. The specifics of the action space will be detailed in Table 6.
High Task Quality Minecraft imposes numerous con- straints that render unchecked tasks intractable (Fan al., 2022; Yang al., 2024). For example, the task "mine diamond with a wooden pickaxe" is infeasible because di- amonds cannot be mined with such a tool. Similarly, the task "design and build a transportation system for your city" from the MineDojo benchmark (Fan et al., 2022) presents an extreme challenge even for humans. Other issues in- clude repetitive tasks (Figure 2). To mitigate these issues, MCU filters tasks from a wide range of data sources, en- suring adherence to high-quality standards (Figure lb left). Additionally, in the LLM-based task configuration gener- ator, we incorporate series of soft constraints within the prompt, and implement a refinement mechanism based on self-reflection (Shinn et al., 2023) and the Minecraft simu- lator's feedback by executing the generated configuration. Further implementation details are provided in Section 2.4.
As mentioned in Section 1, Minecraft serves powerful experimental environment due to its unparalleled diversity, complexity, and open-ended nature, which enable creative gameplay and countless possibilities. Below, we outline the key features that make Minecraft particularly suitable for open-ended game agent development:
Automated Evaluation Open-ended tasks (Stanley et al., 2017: Standish, 2003) inherently lack clear success signals and often depend on human evaluation or manually designed metrics (Dubois et al., 2024), which hinders scalability. To address this issue, MCU introduces an automated evaluation system (AutoEval) based on vision-language model (VLM) that fulfills two key objectives: (1) producing evaluation results that closely align with human judgments, and (2) of- fering multi-dimennionn success rates for a comprehensive evaluation for open-ended tasks (Figure Ic), while the designed "task progress" dimen- sion is a process- supervised counterpart of 0-1 success rate. We also show in Figure le that AutoEval cost-efficient.
Vast State Space. Minecraft provides an extraordinar- ily large state space, as illustrated in Table 7 with an intuitive comparison. Its expansive maps, functional blocks, and diverse mobs result in an immense num- ber of possible configurations. This makes Minecraft an ideal platform for studying the generalizability of agents. Additionally, we present illustrative experi- ments in Appendix to demonstrate the importance of vast state spaces for agent generalization.
assessments that go beyond simple
High Complexity Support. Minecraft supports tasks
2

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Atomic Task
LLM-based Task Config Generator
VLM-based Evaluator
MINE
Mine Birt
Mine birch
task description: iron
agent
ore.
Task
redstone
Task descriptons
Overall: Excellent
>
CRAFT
sets
custor custon_init ttttiinntttttattttee coomandss::
Evidence: The agent nine officiently.
Craft cok plants Craft Stone Ase
/give Bs minecraft:stone pickase #randon
Action Control: Excellent
/give és ninecraft: iron_pickase #necessory
Evidence:
fill rinccraft:redstone #nacessary Anecessory /ggive @ss ninecraft:torch 64 frandon drandos
/esecute êp
Material Usage:
Drink potion
Agent Set
Trajoctory video
COMBAT sheep
Commands env iniiaaiiztion
x Refine config
DECORATION
Decorate the Lay carpet
MineStudio
New tasks
BUILD Build house Build
STEVE-1
Streamlined Package for Minecreft Al Agent Development
MINEDOJO
More Agents
the future
b
Data Source
Real-world Challenges
Precise Control:
Right Moos
Craft Tools: precise grid clicks
Potion of
2:681
Enokaet Beti
Host
Beal canna
Bay faros
Combat: accurate
Motion: precise movements
Simulator
Ligitsoo
Strategic Planning:
Leezzatton
Combat: weapon choice & tactical planning
Potanz
Cemiaa
Animal breeding resource cycling
maoos "pdddnn helmat to ingcta".
create "go
Craft Runk
ferwinee
Knowledge Application:
Ferateal don incour mothing
anainals
resource extraction
Orus hoads
Craft Tools: recipe knowledge
MINEDOJO
(Vhereal fat: Jorest Steersteed steep Coonoot mocene
the Hivgatoo
Boer urr
aramal
Thay drinss
Animal Care: feeding & environment pairing
Wootiers Sevccci
Artistic Creation:
SkillForge
Rud Ecapytenn
DNyraet with
Build: structural design & resource allocation
Brainstorm
Spet Creatoee
Band dhocaal
Art: color matching creative expression
candellin
suiid
gaiden.
BRAIN STORM
ttems
Feee mustees
ore Meni
Coa
Reseoraa Mivna
Mosing
Evea Mouna
Hiudd Moung
Fved
Spatial Awareness:
Build: spatial perception & layout
Find & Navigation: path planning
Logical Reasoning:
bouting!!
sittday gitt
Trade: cost calculation & resource exchange
Tool-use: efficiency durability management
Tank Piognes
GROOT
Comparison
1.0
1.0
Medium
Simple
60-
Progess
0.8-
0.75
0.6
Hard
0.89
Coet (U50)
Time (uBus)))
50
0.8
0.65
0.50
0.45
86
0.30
0.2
0.2
15
0.0
0.0
WITRL
Steeping Bed
Mine Grass
Build Paa
Craft
Human
Humaan
Aato
Figure 1: An overview of MCU. Benchmarking pipeline. MCU includes two main components: task generation and trajectory evaluation. The LLM-based task configuration generator instantiates the environment with the necessary prerequisites, random factors, and task descriptions for diverse atomic tasks. These configurations are verified using an environment simulator. The VLM-base evaluator assesses each task trajectory in video form across multiple dimensions, providing comprehensive performance insights. MCU offers a model-agnostic evaluation interface based on Minestudio (Cai et al., 2024a), making it suitahle for various agents. b. Task category distribution. The atomic task set is sourced from the Minecraft wiki, in-game data, existing benchmarks, and brainstorming sessions. spans 11 major categories and 41 subcategories, ensuring high inter-task diversity. For readers unfamiliar with Minecraft. we illustrate the real-world challenges associated with different task categories to provide context. c. Multi-dimensional capabilities. MCU evaluation shows that SOTA agents have made progress in overall task completion and material usage, but still have obvious limitations in creativity and error recognition. d. Inira-task generalizability Varying difficulty levels within the same task lead to performance degradation, which tests the agent's intra-task task generalization capability, Human AutoEval. The automatic evaluation (AutoEval) of MCU 8.1 more effeccife and 4.8x more efticient labeling 30 samples. Best viewed zoomed
3

---

MCU: An Evaluation Framework for Open-Ended Game Agents
MINEDOJO Task Examples
combat combat_sheep____ plains iron_ _ron_armorss goon_ammosdddanood__yoee shield:
creative:1236:
Build life-sized replica of your house in Minecraft
category: combat
collection: youtube
guidance: 'I. Find a sheep.
prompt: build a nether portal
Design and build a transpo- rtation system for your city
Get close to the sheep.
Hit the sheep with your hand or a weapon.
creative:699:
4. The sheep will die and you will get 1-3 pieces of mutton."
collection: youtube
Create a working calculator out of redstone
prompt: hunt sheep in plains with diamond sword, shield...
prompt: make nether portal.
Only executable for programmatic task
Highly repetitive
Not solvable
MCU Task Examples (Ours)
task description: decorate the cave
task description: hunt sheep with
task description: hunt a sheep with diamond_ swood sword custom init commands: -ggive @ss minecraft:diamond sword
custom iiiiiinittti commands:
wooden_swood
!give @ss minceraftttorch 10
custom init commands:
/give @es mine 10
- /give @ss woddera sword
/give @ss minecraft:item_ frame 2
/summon minecraftssheep- -8
/summon minecraftssheep -5
'give @ts minecraft:flowr_ppe Sottoooerrr_________-_
/summon minecraft:ooff-
fgive @ss minecraft:red _dye 5
/give @ss minecraft:golden _appl
Support open-ended tasks
Simulate real game playing
Different difficulty level
A comparison between the "tasks" in MCU and Minedojo (Fan et al., 2022). We investigate the task list provided by Mije and identify several issues. For example. only programmatic tasks that have clear reward signal can be executable in the benchmark; many tasks in their list are repetitive (both No. and No.699 are "build nether portal"); and large amount of tasks in the creative tasks are not solvable even by human. To address this, our MCU benchmark can create executable configurations for open-ended tasks, and ensure intra-task and inter-task diversity to simulate real game playing in different difficulty levels, while preserving solvability tasks.
Figure 2:
requiring advanced problem-solving skills. For in- stance, obtain diamond task (Guss et al., 2019a) in- volves over 20 sub-goals, often takes hours to com- plete, and demands the ability to remember terrain and resource ceaaiions-showcssing the game's s capability to facilitate complex task development and execution.
tialize the environment. To ensure generality, MineStu- dio unifies the agent inference pipeline. Consequently, our evaluation pipeline observes only the generated trajectories, which are consistently formatted by RecordCallbace for compatibility across diverse agents. Additionally, we em- play RewardsCall back to support user-defined metrics, such as task success rates and our AutoEval metries, and enable RL training with MCU evaluation results.
3. Open-Endedness, Minecraft encourages unrestricted exploration, allowing players to engage in wide spec- trum of challenges. These range from defeating the ender dragon, which requires long-horizon decision- making (Jin et al., 2023), to building house with pre- cise control and creativity (Zhang et al., 2020). This open-ended nature fosters the development of agents capable of handling diverse and dynamic objectives.
2.3. Atomic Tasks: Fundamental Testing Units in MCU
We introduce the concept of atomic tasks as the fundamental testing units in MCU. The inspiration for atomic tasks stems from unit testing in software development (Olan, 2003), where a system is decomposed into smaller, independently verifiable units. Similarly, atomic tasks are designed to isolate and evaluate specific agent capabilities.
2.2. Integration with MineStudio
To create robust and user-friendly benchmark, develop MCU based on MineStudio (Cai et al., 2024a), open- source software package designed specifically to facilitate agent development Minecraft. MineStudio provides intu- itive APIS, extensive documentation and tutorials, making it an ideal foundation for our benchmarking framework.
An atomic task defined solely by its goal g, independent of the methods, tools, or specific environmental conditions. During evaluation, atomic task is instantiated, which induces a task-specific initial state distribution P(so (g) (see Section 2.4). For example, the atomic task mine stone is purely goal-driven. Across different evaluation batches, it may manifest as mine stone with a wooden pi ckaxe, mine stone during a rainy day, or other variations. Regardless of these instances, the core goal remains consistent, ensuring reliable test of whether the agent has genuinely acquired the underlying capability. This property makes atomic tasks an effective tool for evaluat- ing an agent's 's ability to achieve goals under diverse and
MineStudio offers users extensive customization options by allowing them to inherit from the CawcraafCCalback class. This facilitates functionalities such as issuing cheat commands, logging episodes, and overriding observations. Leveraging this flexibility, our task configuration pipeline generates the necessary data for class CommandsCallback, SummonMobsCallbeed and FastResetCallback to ini-
4

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Environment Initialization
Verified Initia lization
task, description: diamond sword
Generate the inital config of sheep hunting.
GPT-4o
the sheep
Self verifiiciit Module
hunt sheep
lask, descriptin:
custom int, commands:
/give minecraftdlamonn sword
custom, Init commands:
igive minecraft:diamend swort X
/summon minecraft:cow Ta
isummon minecraftsheep -2
Error code refine
Multi-dimensionll ritteria
Evaluation Results
Generate the rating criterta for hunting sheep with diamond sword.
Overall: completion key steps.
GPT-40
identify and approaches sheep. use dia aamond sword hunt sheep. perform actions precisely.
Overall:
the agent located the sheep, approached it, and hunted with a diamond sword.
evidence:
Actio:
GPT-4v
result: excellent!
Key Frames
evidence: The agent sometimes dift.
strikes the
Sample
evience:
the agent takes out diamond sword from the inventory good!
00:20
Figure 3: Automatic task generation and evaluation pipeline. Top: Task Generation. GPT-4o generates and verifies the environment initialization configuration for each atomic lask, producing executable instructions compatible with Minestudio APIs. Bottom: Automatic Evaluation. A vision-language model (GPT-4o with vision) assesses lask performance by analyzing sampled video frames against criteria generated by GPT-4o, providing detailed evidence and results. The GPT--4o used 20--4--2224008
potentially unseen conditions.
erating tasks like craft item X if item "X" is craftable). (4) Brainstorming innovative tasks with input from domain experts and LLMs. Further details on the collection process for atomic tasks are provided in Appendix B.I.
Atomic tasks can also be combined form more com- plex tasks using logical operators such as "and" (A) and "or" (V), or by introducing constraints such "when," "where," and "how." For instance, agent could be tasked with [mine oak log] or [mine grass] Obare-handeds and {then} [craft sticks), where [] denotes individual atomic tasks. This compositional approach enables the exploration of a vast task space by leveraging the combination of simple goals create more challenging scenarios. We also incorporate this feature in MCU task generation pipeline detailed in Appendix E.2
2.4. Task Configuration Generation
To execute automatic or compositional tasks in the Minecraft environment, tasks must be configured by specifying param- eters such as spawn biomes, player states, inventories, and surrounding mobs. We formalize this configuration process as sampling an initial state so from a task-speciff distri- bution [7))) To simulate realistic gameplay, evaluate agents effectively on given task T, and ensure intra-task diversity (Section 1), we propose a scalable task configura- tion pipeline powered by LLMs. This pipeline incorporates automatic verification mechanism to ensure task validity based on two criteria: (1) the sampled initial state so hibits high diversity, and (2) the environment includes all elements necessary to complete the task (i.e., 80 is compat- ible with T). By leveraging the knowledge and creativity of LLMs, this approach generates diverse and scalable task scenarios, addressing the limitations of manually predefined configurations. The implementation details are follows.
To pursue task diversity, as outlined in Section 1, we have collected 3,452 atomic tasks', each represented as a textual description. These tasks serve as the fundamental building blocks for task synthesis MCU. They may be later com- posed (discussed in the previous paragraph) and instantiated using 4LM-based task configuration generator (see Sec- tion 2.4). The atomic tasks categorized into 11 major categories and 41 sub-categories, covering diverse chal- lenges encountered in Minecraft. These tasks are sourced through the following pathways: (1) Distilling high-quality tasks from existing benchmarks, such as MineDojo (Fan et al., 2022) and SkillForge (Cai et al., 2024c). (2) Extract- ing tasks from the Minecraft Wiki?, (3) Synthesizing tasks using in- game data from the Minecraft simulator (e.g., gen-
Config Generator As detailed in Section 2.2 we utilize three callbacks from MineStudio to initialize the task envi- ronment. A powerful LLM GPT-4o (Achiam et al., 2023) is prompted with a task description and few-shot examples of the parameter format to populate these callbacks, generating
The dataset continuously expanding.
htttp://mineerrafttww wikk/ wiki/
5

---

MCU: An Evaluation Framework for Open-Ended Game Agents
executable configurations and detailed task descriptions for text-conditioned agents and evaluation in Section 2.5. For instance, the atomic task mine diamonds may include the description "mine diamonds with an iron pickaxe", with the initial state 50 providing necessary resources (e.g.. di- amond ores and an iron pickaxe) to eliminate preparatory subtasks. To enhance task diversity, random factors such as ore placement and item arrangements are introduced to pre- vent predictability. The prompt also includes soft constraints to address LLM limitations, such as numerical insensitivity or confusion with game-specific rules. For crafting tasks re- quiring precise materials (c.g., 3 wool blocks and 3 wooden planks), surplus resources are instructed to be generated to account for LLM inaccuracies. Soft constraints also ensure environmental integrity by avoiding the generation of inaccessible structures (e.g., via the /fill command). Complete prompt details are provided in Appendix G.1.
tions. Specifically, we define six key dimensions for evalu- ating agent performance in Minecraft. Prior evaluation, GPT-4o is prompted to generate task-specific criteria for each dimension. The dimensions are as follows, with an example shown in Figure 3 (bottom): (1) Task Progress: Assesses critical steps and factors required for task com- pletion. (2) Action Control: Evaluates the avoidance of unrelated or unnecessary actions. (3) Material Usage: Mea- sures the selection and application of materials. (4) Task Efficiency: Focuses on minimizing repetitions and optimiz- ing strategies. (5) Error Recognition: Assesses the ability to identify and correct errors. (6) Creative Attempts: Rec- ognizes innovative approaches in task execution. Please check Appendix G. for the prompts of criteria generation.
Scoring with Criteria As described in Section 2.2, the agent's rollout trajectories are recorded in video format. To balance resource efficiency and evaluative effectiveness under specific research conditions, we extract one frame every 30 frames from the video. During the evaluation phase, the sampled frames, along with task-specific criteria, are input into the VI... The VLM assesses each dimension by identifying supporting evidence from the video, providing evidence and explanations, and subsequently assigning a score. For each criterion, we define five scoring intervals: very poor, poor, foir,fair, good, and excellent, corresponding to scores of O, 0.25, 0.5, 0.75, and 1, respectively. Please check Appendix G.5 for the prompts of scoring.
Verification To ensure the quality of the generated task configurations, the config generator incorporates self- verification mechanism. Initially, the generated configu- ration is re-validated by the LLM using the Reflxion tech- nique (Shinn et al., 2023). If any errors are detected- such as summoning cows only for the task hunt sheep (Fig- ure 3hethe config generator is prompted to regenerate the configuration. The generated configuration is also validated using the MineStudio Simulator to ensure executability. Er- ror logs from the simulator are also utilized to guide further regeneration of the configuration if necessary.
3. Experiments
2.5. Automatic Evaluation (AutoEval)
In this section, we first demonstrate the effectiveness of AutoEval. Subsequently, we assess the capabilities of state- tetthe-art agents using MCU and provide insights for the development of future open-ended Minecraft agents.
Automatic evaluation is critical for machine learning bench- marks but challenging to design for AI agents. This prob- lem is framed as defining a scoring function f that maps a task description T and an agent's trajectory traj {so, ag. $1, Sr. to a normalized score reflecting the trajectory's 's quality with respect to T. In digital agent bench- marks, success rate based on annotated criteria is commonly used for tasks with clear, objective metrics (e.g., coding. OS operations). However, in open-ended games like Minecraft, defining a single programmatic metric is often infeasible. To address this, we propose a VLM-based multi lintessiimniioall evaluation framework for MCU, comprising two compo- nents (Figure 3): (I) criteria generation: creating clear, task-specific evaluation dimensions; and (2) scoring with criteria: leveraging predefined criteria to infer quality scores from agent performance videos using VLM.
3.1. Validity of Automatic Evaluation
Experimental Setup To validate the effectiveness of au- tomatic evaluation, we compare the results of automatic evaluation methods with human annotations. We also evalu- ate a baseline approach, MineCLIP (Fan et al., 2022), which also focuses on automatic evaluation. Our evaluation is conducted under two distinct settings: (1) comparative eval- uation, where the quality of two trajectories of same task is compared; and (2) absolute rating, where a score is assigned to single trajectory. For AutoEval, the comparative predic- tion for two trajectory is given by comparing the scores.
Criteria Generation Preliminary experiments reveal that directly prompting GPT-4o (as a VLM) to score trajectories without task-specific criteria leads to suboptimal perfor- mance (Table 1). To address this, we introduce a criteria generation pipeline that provides detailed scoring instruc-
Dataset We collected 500 trajectories spanning 60 tasks, derived from both agent rollouts and human gameplay videos. To evaluate the quality of these trajectories, we engaged 20 expert Minecraft players to provide annotations. The players were tasked with performing both compara-
6

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Table 1: Fl scores for predicting the better human annotated tra- jectory across different task categories. The compared methods include MineCLIP (Fan et al., 2022), our AutoEval on both open- access models (MiniCPM (Yao et al., 2024). Jarvis VL. (Li et al., 2025)) and closed API-based models (GPT-4o). The highest score
Table 3: The Pearson correlation and Kendall's r between AutoE. val and human ratings across different dimensions.
Dimension
Pearson
Kendall's
Coefticient
P-value
Coefficient
P-value
for each task category is bolded.
Task Progress
0.78
1,70 x 10-2
0.71
1.94 x 10-19
Action
0.76
6.22 x 10-19
0.67
1.68 x 10-16
Method
Survive
Build
Craft
Mine
Explore
Average
Error Recog.
0.68
1.10 x 10-12
0.62
4.40 x 10-10
MineCLIP
11.0
45.0
44.0
73.0
0.0
34.6
Creativity
0.63
5.90 x 10-9
0.56
1.18 x 10-
AutoEval (MiniCPM)
65.0
43.0
80.0
59.0
53.0
60.0
Efficinccy
0.75
1.10 x 10-16
0.67
1.02 x 10
AutoEval (JrrviVLLA)
73.0
62.0
73.0
84.0
65.0
7144
Material
0.70
2.28 x 10-16
0.63
2.29 x 10 10-15
AutoEval (GPT-40)
100.0
85.0
62.0
73.0
100.0
84.0
Table 2: Fl scores for predicting the better human-annotated jectory across various dimensions (denoted using abbreviations).
correlation for task progress (0.83) and a lower correlation for creativity (0.69).
Progress
Action
Error Rec.
Creative
Efficiency
Material
Average
84.0
96.0
86.0
100.0
92.0
91.0
91.5
3.2. Evaluating Existing Agents with MCU
Agents We evaluate four powerful foundation agents in Minecraft, all supported by MineStudio: (1) VPT (BC), a behavior cloning agent initially pre-trained on YouTube Minecraft videos and further fine-tuned on refined early- game contractor data; (2) VPT (RL), reinforcement leam- ing fine-tuned model based on VPT (BC) that maximizes the reward for obtaining diamonds in Minecraft; (3) STEVE- (Lifshitz et al., 2023), which follows text instructions to complete tasks; and (4) GROOT (Cai et al., 2024c), which solves tasks demonstrated by a reference video. GROOT receives video instructions, STEVE-I receives text instruc- tions, and VPT operates without instructions. We exclude agents that simplify the Minecraft environment by deviating from the native action space. More details these models can be found in Cai et al. (2024c).
tive evaluations and absolute ratings on randomly sampled trajectory pairs for the same task or individual trajectories. Each player contributed hours of work. Details regarding the annotation process are provided in Appendix C.2.
Comparative Evaluation In this setting, participants are asked to vote on the comparative quality of the trajectory videos (denoted as A and B), selecting from the following options: A is better, B is better, tie, or both are bad. We filtered trajectory pairs annotated with first two options for automatic evaluation, resulting in 236 pairs. As shown in Ta- ble our methodology exhibits a significant improvement over MineCLIP, a CLIP model (Radford et al., 2021) fine- tuned on large-scale Minecraft video frames. MineCLIP, however, struggles to capture complex event-level seman- ties with an average Fl score of only 34.6 (compared to 84.0 of AutoEval). For intricate tasks such as craft, which demand precise attention to detail and the recognition of subtle elements, the performance of AutoEval is slightly reduced. We leave further improvements in this for future work. Nevertheless, as demonstrated in Table 2, our auto- mated evaluation metric achieves an average agreement rate of 91.5% with human assessments across all dimensions. This highlights the effectiveness of the criteria-ent eehanced AutoEval for multi-dimensional evaluations.
Experimental Setup While MCU enables scalable task evaluation without extensive human annotation, care- fully select small yet diverse set of 30 atomic and 5 compositional tasks to illustrate our experimental conclu- sions without introducing excessive complexity. A com- prehensive evaluation of a total of 150 tasks, including 90 randomly sampled atomic tasks and 60 compositional tasks, is deferred to Appendix E. The 30 atomic tasks are drawn from six major categories, ensuring inter-task diver- sity while maintaining moderate difficulty level suitable for Minecraft junior players. The difficulty of each task is of simple mode, and each task is evaluated using 30 random seeds. To highlight the importance of intra-task diversity, we select six atomic tasks (1-2 from each category). assess- ing both simple and hard modes of these tasks seperately. Each difficulty level is evaluated with 10 random seeds.
For absolute rating, we collected a total of 227 individual ratings across approximately 50 distinct tasks. The Pearson and Kendall correlation coefficients be- tween AutoEval and human ratings are presented in Table 3, demonstrating a strong overall positive correlation. How- ever, the correlation varies across different dimensions. For instance, objective dimensions such as task progress exhibit high agreement between human evaluators and VLM assess- ments, with a Pearson correlation of 0.78. In contrast, more subjective dimensions, such as creativity, show a lower cor- relation. Additionally, we compute the inter-rater agreement for scoring the same trajectory, revealing a higher Pearson
Absolute Rating
1.2.1. INTER-TASE GENERALIZATION
We present the Auto "task progress" scores for the inter-task generalization experiments in Table 4, while the multi-dimensional performance, averaged over all tasks, is visualized in Figure 1c. Notably, the task progress metric is closely related to the commonly used "task suc-
7

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Table 4: Task progress for 35 tasks. Performance table across all tasks in simple mode, Abbreviations: Exp exploring, FD find diamond, FF- = find forest, FV = find villages, CM=climb mountain; Crv carve, CO = compose obsidian, Drk= drink. F&S- flint&stcel, Slp= = sleep; Ck = cake, Clk = clock, CT = craft table, DS = diamond sword, Ld = ladder; DO = diamond orc, Dt dirt, Gr = grass, 10 = iron orc, Ob = obsidian; End = enderman, Shp = sheep, Ski = skelctons, Spd spiders, Zmb = zombics; BP = build pillar, Cv = cave, NP= = nether portal, SG snow golem: = waterfall; CTS crafting table from scratch, MDS mine diamond from scratch, D&S dye and shear sheep. T&P- till and plant seeds, PAG prepare gift. Best values within the same task are bolded.
Navigation Task
Tool-use Task
Crafting Task
Agent
Exp
FD
FF
FV
CM
Avg
Crv
CO
Drk
F&S
SIp
Avg
Ck
Clk
CT
DS
Ld
Avg
GROOT
0.90
0.56
0.75
0.60
0.60
0.72
0.20
0.10
0.40
0.10
0.85
0.33
0.35
0.60
0.45
0.75
0.25
0.48
Steve-I
0,95
0.50
0.95
0.90
0.35
0.73
0.45
0.00
0.35
0.20
0.10
0.22
0.80
0.70
0.45
0.20
0.70
0.57
VPT (BC)
0.90
0.65
0.87
0.75
0.60
0.75
0.25
0.00
0.40
0.10
0.45
0.24
0.45
0.35
0.30
0.50
0.45
0.41
VPT (RL)
0.70
0.58
0.55
0.50
0.35
0.54
0.15
0.10
0.35
0.15
0.25
0.20
0.70
0.62
0.50
0.30
0.62
0.55
Mining Task
Combating Tisk
Building Task
Compositional Task
Agent
DO
Dt
Gr
10
Ob
Avg
End
Ship
Ski
Spd
Zub
Avg
BP
Cy
NP
SG
WT
Avg
CTS
MDS
D&S
T&P
PAG
Avg
GROOT
0.81
0.70
0.90
0.30
0.50
0.56
0.50
0.75
0.53
0.40
0.20
0.45
0.65
0.15
0.38
0.45
0.71
0.05
0.19
0.23
Steve-1
0.35
0.85
0.95
0.20
0.35
0.54
0.05
0.30
0,40
0.75
0,42
0.54
0.60
0.10
0.30
0.05
0.05
0.13
0.45
0.35
0.30
0.20
0.30
0.13
VPT (BC)
0.30
0.30
0.50
0.15
0.38
0.33
0.25
0.55
0.55
0.35
0.50
0.36
0.00
0.02
0.35
0.00
0.20
0.11
0.30
0.30
0.10
0.00
0.25
0.13
VPT (RL) 0.15
0,35
0.37
0.05
0.35
0.22
0.15
0.35
0.35
0.25
0.30
0.28
0.10
0.02
0.40
0.13
0.25
0.23
0.90
0.50
0.00
0.00
0.30
0.34
Table 5:
Performance changes of GROOT for selected tasks in simple and hard modes. Fach result is averaged over 10 seeds.
gesting important directions for future research aimed at improving these aspects.
Task Progress
Action Control
Task
Simple
Hard
Simple
Hard
A
3.2.2. INTRA-TASK GENERALIZATION
Build Nether Portal
0.45
0.30
-0.15
0.50
0.40
-0.10
Mine Iron Ore
0.56
0.60
0.04
0.44
0.55
0.11
The task progress and action control performance of intra- task generalization experiments for GROOT, the best agent among the compared agents, are presented in Table 5. The results indicate that for the same task, slight changes in the task situation lead to a significant drop in task progress performance. For example, consider the very simple task of sleep in bed, which only requires the agent to identify bed and right-click the mouse. In simple mode, the bed is placed directly in front of the agent on a plain with no surrounding objects. However, in hard mode, both the bed and the agent are inside house, requiring the agent to correctly identify the bed and interact with it. We observed failure cases where the agent mistakenly interacted with a chest in the room or left the house instead. This suggests that GROOT does not learn the skill robustly, a phenomenon not previously reported in the Minecraft agent literature.
Craft to Cake
0.35
0.32
-0.03
0.31
0.25
-0.06
Combat Skeletons
0.56
0.30
-0.26
0.25
0.20
-0.05
Carve Pumpkin
0.20
0.15
-0.05
0.35
0.25
-0.10
Sleep in bed
0.85
0.50
-0.35
0.40
0.30
-0.10
Average
0.50
0.36
-0.14
0.38
0.33
-0.05
cess rate" in agent benchmarks. However, unlike success rate, which is binary (0-1) outcome-based criterion, task progress is a process-super subeevsss metric. This allows for a more fine-gratn -ne-ggriind assessment of an agent's performance beyond simply determining whether task was completed.
We observe that agents often struggle to reliably complete many tasks (e.g., combat and build). However, they still exhibit nonzero task progress, indicating partial progress as determined by the L.M. This suggests that while agents may not always achieve full task success, they are capable of making incremental advancements toward task completion. Among the evaluated agents, we note that VPT (RL), which is specifically RL-tuuned to maximize rewards for obtaining diamonds, performs well on tasks aligned with this objec- tive (e.g., crafting a crafting table from scratch). However, it significantly underperforms on tasks unrelated to its tar- get. This highlights the importance of inter-task diversity in assessing the generalizability of agents. While generalist agents such as GROOT and Steve-I demonstrate better open- endedness, they struggle with specific tasks (c.g., crafting particular items), indicating that future efforts should focus on improving performance in these areas. Additionally, we observe that task progress for compositional tasks is lower than for atomic tasks, underscoring the persistent challenge of MCU, Furthermore, as shown in Figure 1c, current agents fall short in creativity, error recognition, and efficiency, sug-
Nevertheless, some tasks exhibited relatively stable perfor- mance across difficulty levels, such as mine iron ore, craft to cake, and combat skeletons. This may be because certain categories of tasks (c.g.. building, tool-use, and crafting) are more susceptible to difficulty variations due increased environmental complexity, while others are less affected. Overall, enhancing intra-task diversity of tasks is beneficial to test the robust generalzability of agents.
4. Related Work
Benchmarks in Minecraft MineDojo (Fan et al., 2022) introduces a suite of 1,560 creative tasks defined by natu- ral language instructions. However, it suffers from signif- icant redundancy and overly complex tasks, complicating practical evaluation, as shown in Figure 2. BEDD (Milani
8

---

MCU: An Evaluation Framework for Open-Ended Game Agents
et al., 2023) defines five tasks covering various aspects of Minecraft, primarily designed for the MineRL BASALT competition. By decomposing the evaluation framework, BEDD facilitates detailed assessments of agent performance across subgoals and attributes such as human likeness. How- ever, its reliance on human ratings limits scalability. Other works on Minecraft agents (Wang et al., 2023d;e; Cai et al., 2024c; Yuan et al., 2023; Baker al., 2022; Lifshitz al., 2023) lack unified benchmark, with each agent evaluated on its own task set. We argue that establishing a standard- ized benchmark is a high priority for advancing Minecraft agent development.
often cannot be directly defined by rule-based rewards (Guo et al., 2024; Zhang et al., 2023).
LLM-as-a-Juudg The advancement of large language models (LLMs) has demonstrated remarkable performance in instruction following, query understanding, and response generation. This capability has motivated the use of LLMs as judges (Zheng al., 2023a). leveraging their ability to score and rank model outputs. The strong performance of LLMs (Brown et al., 2020), combined with well-designed assessment pipelines (Li et al., 2023; Beigi et al., 2024; Bai et al., 2024), enables fine-grainee and detailed judgments for various evaluation applications, addressing the limita- tions of traditional evaluation methods that require extensive human annotation. Recent efforts have also explored the use of LLMs and vision-language models (VLMs) to evaluate AI agents (Pan et al., 2024; Zhuge et al., 2024). Compared to previous work, MCU is the first to apply this paradigm to open-ended game agent task generation and evaluation. We argue that LLM--ased task generators have the potential to create diverse tasks that are crucial for evaluating open- ended game agents, given the inherent open-endedness of LLMs (Hughes et al., 2024). Furthermore, using the same LLM (or VI.M) as judge ensures more consistent evalua- tion criteria compared to crowdsourcing approaches (Zhou et al., 2023).
Open-ended Agents in Minecraft Many agents have been developed to interact with Minecraft environments. Some methods focus on using Imitation Learning or Rein- forcement Learning to learn various skills in open-world Minecraft to complete short-horizon tasks (Baker et al., 2022; Cai et al., 2024c; Lifshitz al., 2023; Jiang al., 2025; Zhao et al., 2024a; Cai et al.; Yuan et al., 2024; Cai et al., 2024b; Jiang & Lu, 2025). Baker et al. (2022) gener- ates action labels from Minecraft videos on YouTube using IDM and performs imitation learning to obtain an uncondi- tional policy capable of completing various tasks. Lifshitz et al. (2023) uses MineCLIP as a text encoder to obtain a text-conditioned multitask policy based on unconditional VPT (Baker et al., 2022). Minecraft also supports the test- ing of long-horizon programmatic tasks, so some meth- ods accomplish long-horizon tasks by using large language models as planners, combined with skill policies (Wang et al., 2023d; Zhou et al., 2024a; Chen & Gao, 2024; Yuan et al., 2023; Qin et al., 2024; Zheng et al., 2023b). Wang et al. (2023d) has designed an agent pipeline based GPT, which interactively completes tasks from environmental feedback through the self-explanation and zero-shot plan- ning capabilities of LLMs. In order to further enhance the long-horizon capabilities of LLM Agents, some methods have explored efficient explicit memory mechanisms to sup- port agents retrieve and improve from previous interaction trajectories (Wang et al., 2023c; Zhu et al., 2023; Wang et al., 2023a; Park et al., 2024; Li et al., 2024). Unlike designing complex agent pipelines, the rise of VLA (Zitkovich et al., 2023; Kim et al., 2024) has inspired researchers to use end- to-end VLM as a policy to directly fulfill human instructions in Minecraft (?Zhao et al., 2024b). Some methods use a code-as-policy approach (Liang et al., 2022), using Mine- Flayer (PrismarineJS, 2024) as language-condingg conditione pol- icy, combined with a pretrained LLM to accomplish various tasks to avoid agents trapped on short-horizon skills lack- ing when executing long-horizon tasks (Wang et al., 2023a; Yang et al., 2025; Liu et al., 2024b;a). Finally, there are some methods focused on completing open-ended creative tasks in Minecraft, such as building and decoration, which differ from the traditional programmatic object-centric tasks and
5. Conclusion
We introduce Minecraft Universe (MCU), scalable evalua- tion framework for open-ended game agents in Minecraft. MCU features 3,452 diverse and composable atomic tasks, a dynamic task composition mechanism to ensure sustained challenges, and an automated evaluation system with over 90% human alignment. Empirical results indicate that state- of- ofethe-uu art agents struggle with tasks exhibiting high inter- task and intra-task diversity. To support standardized bench- marking, we release MCU-Turbo a curated subset of 100 tasks with structured difficulty settings, as detailed in Ap- pendix F. We hope MCU serve as robust foundations for advancing oprn-world agent research.
A Acknowledgement
This work funded in part by the National Science and Technology Major Project (20227D011 14902), and National Natural Science Foundation of China (62376031). We ac- knowledge other members in CraftJarvis team for their data labeling support and constructive discussions, and thank Dr. Wenzheng Feng for his valuable insights.

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Impact Statement
S., and Rocktäschel, T. Genie: Generative interactive environments, 2024. URL httpt:h/taaaiiii/ org abs/ 2402.15391. 1
This paper presents research aimed at advancing the field of open-ended game agents, contributing to the broader under- standing of adaptive and autonomous artificial intelligence in dynamic environments. While our work has the potential to influence various domains, including Al-driven decision- making and interactive entertainment, we do not identify any specific societal consequences that require immediate attention or emphasis at this time.
Cai, S., Zhang, B., Wang, Z., Ma, X., Liu, A., and Liang, Y. Groot-1.5: Learning to follow multi-mode instruc- tions from weak supervision. In Multi-moda Foundation Model meets Embodied AI Workshop@ ICM1.2024. 9
Cai, S., Mu, Z.,, He, K., Zhang, B., Zheng, X., Liu, A., and Liang, Y. Minestudio: A streamlined package for minecraft ai agent development, 2024a. URL https: //arxiv. org/ hbs//bs///00 abs///2/./22...//// 18293.1.............. 3,
References
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, L,, Aleman, F. I.,, Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., al. Gpt-4 technical report. arXiv preprint arXiv:2303.00 087744 2023. 5
Cai, S., Zhang, B., Wang, Z., Lin, H., Ma, X., Liu, A., and Liang, Y. Groot-2: Weakly supervised multi- modal instruction following agents. arXiv preprint arXiv:2412. 10410, 2024b. 9
Alaniz, S. Deep reinforcement learning with model learning and monte carlo tree search in minecraft. arXiv preprint arXiv: 08456, 2018. 2
Cai, S., Zhang, B., Wang, Z., Ma, X., Liu, A., and Liang, Y. Groot: Learning to follow instructions by watching game- play videos. In The Twelfth International Conference on Learning Representationss. 2024c. 1,5, 7, 9, 18
Arumugam, D., Lee, J. K., Saskin, S., and Littman, M. L.. Deep reinforcement learning from deeeey-d hu- man feedback. arXiv preprini
Chen, J. Y. and Gao, T. Apt: Architectural planning and text- to-blueprint construction using large language models for open-world agents. arXiv preprint 1rXiv:2411.1722) 2024.
2019.
Bai, Y., Ying, J., Cao. Y. Lv, X., He, Y., Wang, X., Yu, J.. Zeng, K., Xiao, Y., Lyu, H., et al. Benchmarking foundation models with In Advances in Neural Information Processing Systems, volume 36, 2024. 9
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning. In International conference on machine learn- ing, pp. 2048-2056. PMLR, 2020.
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching un- labeled online videos. Advances in Neural Information Processing Systems, 2022. 9, 18
Dubois, Y., Li, C. X., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P. S., and Hashimoto, T. B. Alpacafarm: A simulation framework for methods that learn from human feedback. In Advances in Neural Infor- mation Processing Systems, volume 36, 2024. 2
Beigi, A., Tan, Z., Mudiam, N., Chen, C., Shu, K., and Liu, H. Model attribution in ilm-generated disinforma- tion: A domain generalization approach with supervised contrastive learning. In 2024 IEEE 11th International Conference on Data Science and Advanced Analytics, pp. 1-10. IEEE, 2024.
Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Advances in Neural Infor- mation Processing Systems Datasets and Benchmarks, 2022. 1, 4, 5, 6. 7, 8, 18
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pp. 1877-1901, 2020. 9
Guo, Y., Peng. S., Guo, J., Huang, D., Zhang, X., Zhang, R., Hao, Y., Li, L., Tian, Z., Gao, M., et al. Luban: Building open-ended creative agents via autonomous embodied verification. arXiv preprint arXiv:2405. 15414, 2024. 9
Guss, W. H., Codel, C., Hofmann, K., Houghton, B., Kuno, N., Milani, S., Mohanty, S., Liebana, D. P., Salakhutdinov, R., Topin, N., et al. Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors. arXiv preprint XxXiv1904..1009, I (8), 2019a. 4
Bruce, J., Dennis, M., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steigerwald, R., Apps, C., Aytar, Y., Bechtle, S., Behbahani, F., Chan, S., Heess, N., Gonzalez, L., Osindero, S., Ozair, S., Reed, S., Zhang, J., Zolna, K., Clune, J., de Freitas, N., Singh,
10

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C.,, Veloso, M., and Salakhutdinov, R. Minerl: A large- scale dataset of minecraft demonstrations. In Proceedings International Joint Conference on Artificial Intelligence, 2019b. 2
Li, R., Patel, T., and Du, X. Prd: Peer rank and discus- sion improve large language model based evaluations. Transactions on Machine Learning Research, 2023. 9
of the
Li, Z., Xie, Y., Shao, R., Chen, G., Jiang, D., and Nie, Optimus-1: Hybrid multimodal memory empow- ered agents excel in long-horizon tasks. arXiv preprint arXiv:2408 AKIv:2408003615. 2024. 9
Hafner, D. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.00 06780, 2021.
Huang, D., Bu, Q.. Zhang, J. M., Luck, M., and Cui, H. Agentcoder: Multi-agent- code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010 2023. 1
Liang. J., Huang, W. Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753. 2022.
Hughes, E., Dennis, M. D., Parker-Holder, J., Behbahani, F., Mavalankar, A., Shi, Y., Schaul, T., and Rocktäschel, T. Position: Open-endedness is essential for artificial superhuman intelligence. In Proceedings of the 41st In- lernational Conference on Machine Learning, 2024. 9
Lifshitz, S., Paster, K., Chan, H., Ba, J., and Mcllraith, S. Steve- generative model for text-to-behavior in minecraft. In Advances in Neural Information Processing Systems, 2023. 7, 9,9,2 25
Lin, K. Q.. Li, L., Gao, D., Yang. Z., Bai, Z., Lei, W., Wang. I.., and Shou, M. 7. Showui: One vision-language- action model for generalist gui agent. In NeurIPS 2024 Workshop on Open- Open-World Agents, 2024.
Jiang, H. and Lu, Z. Visual grounding for object-level generalization in reinforcement learning. In European Conference Computer Vision, pp. 55-72. Springer, 2025. 9
Liu, S., Li, Y., Zhang, K., Cui, Z., Fang, W., Zheng, Y., Zheng, T., and Song, M. Odyssey: Empowering minecraft agents with open-world skills, arXiv preprint 24Xiv:2407.15325 2024a. 9
Jiang, H., Yue, J., Luo, H., Ding, Z., and Lu, Z. Rein- forcement learning friendly vision- langune model for minecraft. In European Conference on Computer Vision pp. 1-17. Springer, 2025.
Liu, S., Yuan, H., Hu, M., Li, Y., Chen, Y., Liu, S., Lu, Z., and Jia, RI-gpt: Integrating reinforcement learning and code-as-policy. arXiv preprint arXiv:2402.19299, 2024b. 9
Jimenez, C. E., Yang. J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. R. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024.1
Matiisen, T., Oliver, A., Cohen, T., and Schulman, J. Teacher-student curriculum learning. IEEE transactions on neural networks and learning systems, 31(9):3732- 3740, 2019. 2
Jn, E., Hu, J., Huang, Z., Zhang, R., Wu, J., Fei-Fei, L., and Martín-Martin R. Mini-behavion A procedurally generated benchmark for long-horizon decision-making in embodied ai. arXiv preprint arXiv:2310.01824 2023.
Milani, S., Kanervisto, A., Ramanauskas, K., Schulhoff, S., Houghton, B., and Shah, R. Bedd: The minerl basalt evaluation and demonstrations dataset for training and benchmarking agents that solve fuzzy tasks. In Proceed- ings of the Neural Information Processing Systems Track Datasets and Benchmarks 3, 2023. 8, 20
Kejriwal, M., Kildebeck, E.,, Steininger, R., and Shrivastava, A. Challenges, evaluation and opportunities for open- world learning. Nature Machine Intelligence, 6(6):580- 588, 2024.
Nasiriany, S., Maddukuri, A., Zhang, L.., Parikh, A., Lo, A., Joshi, A., Mandlekar, A., and Zhu, Y. Robocasa: Large- scale simulation of everyday tasks for generalist robots. In Robotics:
Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakr- ishna, A., Nair, S., Rafailov, R., Foster, E., Lam, G., San- keti, P., et Openvla: An open-source language- action model. arXiv preprint 092466,,000 2024.
Science and Systems, 2024.
Oh, J., Chockalingam, V., Lee, H., et al. Control of memory, active perception, and action in minecraft. In Interna- tional conference on machine learning, pp. 2790-2799. PMLR, 2016. 2
Li, M., Wang, Z., He, K., Ma, X., and Liang, Y. Jarvis-vla: Post-training large-scale vision language models to play visual games with keyboards and mouse. arXiv preprint arXiv:2503. 16365, 2025. 7
Olan, M. Unit testing: test early, test often. Journal of Computing Sciences in Colleges, 19(2):319-33 9228, 2003.

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Pan, J., Zhang, Y., Tomlin, N., Zhou, Y., Levine, S., and Suhr, A. Autonomous evaluation and refinement of digital agents. arXiv preprint
Stanley, K. O., Lehman, J., and Soros, L. Open-endedness:
The last grand challenge you've never heard of. https: //www. uber
06444,, 2024. 1,
http///oo/reeercchh..ooa 2017. 2
Park, J., Cho, J., and Ahn, S. Mr. steve: Instruction- following agents in minecraft with what-where-wee memory. arXiv preprint arXiv:2411.067:: 2024. 9
Team, O. E. L.,, Stooke, A., Mahajan, A., Barros, C., Deck, C., Bauer, J., Sygnowski, J.. Trebacz, M., Jaderberg, M., Mathieu, M., et al. Open-ended learning leads to gener- ally capable agents. arXiv preprint arXiv:2107.12803 2021.
Parmar, J., Chouhan, S., Raychoudhury, V., and Rathore, S. Open Open-world machine learning: applications, challenges, and opportunities. ACM Computing Surveys, 2023.
Tessler, C., Givony, S., Zahavy, T., Mankowitz, D., and Man- nor, S. A deep hierarchical approach to lifelong learning in minecraft. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017. 2
PrismarineJS. mineflayer:
Create Minecraft bots with a powerful, stable, and high level JavaScript API. https:
//github, cww/Priommiiieecccee
2024. 9
Trott, A., Zheng, S., Xiong, C., and Socher, R. Keeping your distance: balancing shaped rewards. In Advances in Neural Infor- mation Processing Systems, volume 32, 2019. 2
Qin, Y., Zhou, E., Liu, Q. Yin, Z., Sheng, L., Zhang, R., Qiao, Y., and Shao, J. Mp5: A multi-modal open-ended embodied system in minecraft via active perception. In 2024 IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024. 9
Solving sparse reward tasks using self-
Udagawa, H., Narasimhan, T., and Lee, S.-Y. Fighting zombies in minecraft with deep reinforcement learning, Technical report, Technical report, Technical report, Stan- ford University, 2016. 2
Qin, Y.. Ye, Y.. Fang, J., Wang. H., Liang. S., Tian, S., Zhang, J., Li, J., Li, Y., Huang, S., et al. Ui-tars: Pioneer- ing automated gui interaction with native agents. arXiv preprint ar arXiv:2501. (tt.:2501.12222.. 12326, 2025.
Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y.. Fan, L., and Anandkumar, A. Voyager: An open- ended embodied agent with large language models. In Agent Learning in Open- Open-Endedness Workshop, 2023a.
Raad, M. A., Ahuja, A., Barros, C., Besse, F., Bolt, A., Bolton, A., Brownfield, B., Buttimore, G., Cant, M., Chakera, S., et al. Scaling instructable agents across many simulated worlds. arXiv preprint arXiv:2404. 10179, 2024. 1
Wang, J., Xu, H., Ye, J., Yan, M., Shen, W., Zhang, J., Huang, F., and Sang, J. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint 24Xiv:2401.16155 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. 7
Wang. L., Ling, Y., Yuan, Z., Shridhar, M., Bao, C., Qin, Y., Wang, B., Xu, H., and Wang. X. Gensim: Generating robotic simulation tasks via large language models. arXiv preprint 2rXiv:2310.01300 2023b.
Reed, S., Zolna, K. Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al. A generalist agent. arXiv preprint arXiv:2205.06111 2022.
Wang, Z., Cai, S., Liu, A., Jin, Y., Hou, J., Zhang, B., Lin, H., He, Z., Zheng, Z., Yang, Y., Ma, X., and Liang. Y. Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models. ArXiv, abs/2311.05999 2023c. URL https: //api. semanticscholas Parg/CorpussDD 265129059. 9, 18
Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning. In Advances in Neural Information Processing Systems, 2023. 2,
Wang, Z., Cai, S., Liu, A., Ma, X., and Liang, Y. Describe, explain, plan and select: Interactive planning with large language models enables open-workd werld multi-tan -ask agents. In Thirty-seventh Conference on Neural Information Pro- cessing Systems, 2023d. 9, 18
Shu, T., Xiong, C., and Socher, R. Hierarchical and in- terpretable skill acquisition in multi-task reinforcement learning. arXiv preprint arXiv:1712.0029%. 2017. 2
Xu, F. F., Vasilescu, B., and Neubig, G. In-ide code gen- eration from natural language: Promise and challenges. ACM Transactions on Software Engineering and Method- ology, 31(2):1 2022.
Standish, R. K. Open-ended artificial evolution. Interna- tional Journal of Computational Intelligence and Appli- cations, 1775 2003.
12

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Yang, J., Dong, Y., Liu, S., Li, B., Wang, Z., Tan, H., Jiang, C., Kang, J., Zhang. Y., Zhou, K., et al. Octopus: Em- bodied vision-language programmer from environmental feedback. In European Conference on Computer Vision, pp. 20-38. Springer, 2025. 9
Zhou, E., Qin, Y., Yin, Z., Huang, Y., Zhang, R., Sheng, L., Qiao, Y., and Shao, J. Minedreamer: Learning to follow instructions via chain-of- chain-of-imagination for simulated-world control. arXiv preprint 2rXiv:2403.120333 2024a. 9
Zhou, S., Xu, F., Zhu, H., Zhou, X., Lo, R., Sridhar, A.. Cheng, X., Ou, T., Bisk, Y., Fried, D., al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307. 13854, 2023. 1,9
Yang, Y., Sun, F.-Y., Weihs, I.., VanderBilt, E.,, Herrasti, A., Han, W. Wu, J., Haber, N., Krishna, R., Liu, L.,, al. Holodeck: Language guided generation of 3d embod- ied ai environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2
Zhou, Z., Atreya, P., Lee, A., Walke, H., Mees, O., and Levine, S. Autonomous improvement of instruction following skills via foundation models, arXiv preprint arXiv:2407.200
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint 240iv:208.00880 2024.7 7
2024b. 1
Zhu, X., Chen, Y., Tian, H., Tao, C., Su, W., Yang, C., Huang, G., Li, B., Lu, L., Wang, X., et al. Ghost in the minecraft: Generally capable agents for open- world environments via large language models with text-based knowledge and memory. arXiv preprint 2rXiv:2305.17144 2023.
Yuan, H., Zhang, C., Wang, H., Xie, F., Cai, P., Dong, H., and Lu, Z. Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks. arXiv preprint arXiv:2303. 16563, 2023. 9, 18
Zhuge, M., Zhao, C., Ashley, D., Wang, W., Khizbullin, D., Xiong, Y., Liu, Z., Chang, E., Krishnamoorthi, R., Tian, Y., et al. Agent-as-a-judee Evaluate agents with agents. arXiv preprint arXiv:2410. 10934, 2024. 9
Yuan, H., Mu, Z., Xie, F,, and Lu, Z. Pre-training goal- based models for sample-efficient reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. 9
Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, E., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q. Vanhoucke, V., Tran, H., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P. R. Salazar, G., Ryoo, M. S., Reymann, K., Rao, K., Pertsch, K., Mordatch, I., Michalewski, H., Lu, Y., Levine, S., Lee, I.,, Lee, T.- W. E., Leal, I., Kuang, Y., Kalashnikov, D., Julian, R. Joshi, N. J., Irpan, A., Ichter, B., Hsu, J., Herzog, A., Hausman, K., Gopalakrishnan, K., Fu, C., Florence, P., Finn, C., Dubey, K. A., Driess, D., Ding, T., Choro- manski, K. M., Chen, X., Chebotar, Y., Carbajal, J., Brown, N., Brohan, A., Arenas, M. G., and Han, K. Rt-2: essionlanguaage-cttinn models transfer web knowledge robotic control. In Proceedings of The 7th Conference on Robot Learning, 2023. 9
Zhang, C., Cai, P., Fu, Y., Yuan, H., and Lu, Z. Creative agents: Empowering agents with imagination for creative tasks. arXiv preprint vrXiv:2312.025111 2023. 9
Zhang, Z., Kayacan, E., Thompson, B., and Chowdhary, G. High precision control and deep maaming-based corn stand counting algorithms for agricultural robot. Au- tonomous Robots,
2020.
Zhao, G., Lian, K., Lin, H., Fu, H., Fu. Q., Cai, S., Wang, Z., and Liang, Y. Optimizing latent goal by learning from trajectory preference. arXiv preprint arXiv:2412.021155, 2024a. 9
Zhao, Z., Ma, K., Chai, W., Wang, X., Chen, K., Guo, D., Zhang, Y., Wang, H., and Wang, G. Do we really need complex agent system? distill embodied agent into single model. arXiv preprint arXiv:2404 19, 2024b.
Zheng, L., Chiang, W.-L., Sheng. Y., Zhuang. S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li. D., Xing, E., et al. Judg- ing llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, vol- ume 36, pp. 46595-46623, 2023a. 9
Zheng, S., Liu, J., Feng, Y., and Lu, Z. Steve-eye: Equipping 11m- based embodied agents with visual perception in open worlds. arXiv preprint arXiv:23 10. 13255, 2023b.
13

---

MCU: An Evaluation Framework for Open-Ended Game Agents
A. Minecraft Environment Setting
In the regular Minecraft game, the server (or "world") always runs at 20Hz while the client's 's rendering speed can typically reach 10.100Hz. To ensure consistency with the server, the frame rate is fixed at 20 fps for the client. The action and observation spaces in our environment are identical to what human player can operate and observe on their device when playing the game. These details will be further explained in subsequent subsections. Additionally, diagnostic information such as in-game stats, contents of the agent's inventory, and whether any in-game GUI is open is provided by the environment. This information can only be used for tracking, recording, and evaluating purposes but cannot serve as inputs to evaluated agents,
A.1. Minecraft Game World Setting
We have chosen to conduct the test in Minecraft version 1.16.5's survival mode. During this open-world experiment, the agent may encounter situations that result in its death, such as being burned by lava or campfire, getting killed by hostile mobs, or falling from great heights. When this happens, the agent will lose all its items and respawn at a random location near its initial spawn point within the same Minecraft world or the last spot attempted to sleep. Importantly, even after dying, the agent retains knowledge of its previous deaths and can adjust its actions accordingly since there no masking of policy state upon respawn.
Draffing
Invar flon
Figure 4: Minecraft game observation.
A.2. Observation Space
The observation space for human player is limited to the raw pixels visible on the display screen. It does not include any hidden information from the game world, such as hidden blocks or nearby mobs. Additionally, any information contained in the pixels must be perceived by the model rather than directly given, including inventories and health indicators. Human players can access this information by pressing F3, which should considered part of the game screen. There are no restrictions on optional parameters that human players can adjust the display settings, such as field of view, GUI scale (controlling the size of in-game GUT), and brightness. The rendering resolution of Minecraft 640x360; however, is recommended to resize images to lower resolutions for better discernibility and computational efficiency.
A.3. Action Space
The action space is also consistent with human-playing settings, i.c., mouse and keyboard controls. These actions include key presses, mouse movements, and clicks. The specific binary actions that triggered by keypress are shown in Table Table 6. In addition actions triggered by keypresses, the action space also includes mouse movements. Similar to human gameplay, when there are no in-game GUIs open, moving the mouse along the X and Y axes changes the agent's 's yaw and pitch respectively. However, when a GUI is open, camera actions shift the position of the mouse cursor. The mouse movements are relative and adjust their position or camera angle based on their current state. More details can found at fandom. www/wwki /tmm/vii//ooorrole /Controls
14

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Table Binary actions included in the action space.
Action
Human action
Description
forward
key
Move forward.
back
S key
Move backward.
left
A key
Strafe left.
right
D key
Strafe right.
jump
space key
Jump.
inventory
E key
Open or close inventory and the 2x2 crafting grid.
sneak
shift key
Move carefully in the current direction of motion. In the GUT acts as a modifier key: when used with an attack it moves item from/to the inventory to/from the Hotbar, and when used with craft it crafts the maximum number of items possible instead of just 1.
sprint
ctrl key left mouse button
Move fast in the current direction of motion.
attack
Attack; In GUI, pick up the stack of items or place the stack of items in a GUT cell; when used as a double click (attack no attack attack sequence), collect all items of the same kind present in inventory as a single stack.
use
right mouse button
Place the item currently held use the block the player is looking at. In GUI, pick up the stack of items place single item from stack held by the mouse.
drop
Q key
Drop a single item from the stack of items the player is currently holding, If the player presses etrl-Q then it drops the entire stack. In the GUI, the same thing happens except for the item the mouse is hovering over. Switch active item to the one given hotbar cell.
hotbar.[1-9 [1-9]
keys 1
show debug screen
F3 key
See the chunk cache, the memory usage, various parameters, the player's map coordinates, and graph that measures the game's current frame rate.
A.4. Why Minecraft is Suitable for Open-Ended Agent?
1.4.1. STATE SPACE CALCULATION
Minecraft has an enormous state space. Here, we will estimate how many possibilities are contained within the full state space of Minecraft.
Block State The world Minecraft consists of different types blocks, each with potentially multiple states. Let be the number of block types, and W be the world size, which approximately 30M 30M x 384, i.e. the total number of blocks in the world.
The formula for the number of block states is:
Block States = W
Substituting the values B = 500 and W x 3.46 x 18 we get:
Block States = 4
In Proegen environments, the number of block states is estimated based on a pixel space. Block states are calculated based on how many different visual representations (pixels) can be generated. ALE, the levels are artificially fixed and finite.
15

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Table 7: Comparison State Space between Different Benchmarks. Unlike other benchmarks that isolate tasks within separate state spaces, which may simplify the learning process, Minecraft integrates all tasks into a shared state space. This requires the agent to generalize without relying memorizing specific environments. The initial seed reflects the inherent generation capability of procedural generation, while the final state reflects the full range of possibilities contained within the entire game. The final state space of Minecraft (Approx. 10 far exceeds the number of atoms in the universe (Approx. 10 of the combinations are possible.
The constraint is 10 assuming that only percent
Dimension
Minecraft
Procgen/CoinRun
ALE/Pitfall
Initial Sced
2
Fixed
World Size
30M 30M 384 As 3.46 10i5
64 64
Predefined Layout
Block Types
3-5
Block States
5002-46x:10 164
364x64
limited
Entities
Mobs: 30+ Types, Health: 0-20
Obstacles: 3 Classes
Obstacles: Classes
Animals:
30+ Types, Heallhh 0-10
Villagers: 13 Professions, 5 Levels, 100 Trades
Entity Count
107
20
10
Entity States
(30 x 20 30 x 10 + 13 x x 100)10' 2.57
3 Rz 3.49 x 10
Limited
76x
Inventory States
36 Slots, 500+ Item Types, Max Stack 64 (500 x 64)30
N/A
N/A
Final State Space
Approx, 10
Approx. 10
Limited, Predefined Levels
Block Entity x Inventory States Constraint
Entity State Minecraft contains wide variety of entities including animals, mobs, and villagers, each with different properties and states. Mobs have more than 30 types, health ranging from 0 to 20, animals have more than 30 types with health from 0 to 10, and villagers have 13 professions, 5 levels, and 100 trades. The entire map contains approximately 107 entities. Consider all possible combinations, the number of entity states is:
Entity States Ri (30 20 + 30 x 10 + 13 x 5 x 100)10" ~ 2.57 x 75x100
In Proegen environments like CoinRun, the number of entity states is typically smaller due to fewer types of entities and simpler interactions.
Inventory State Minecraft's inventory consists of 36 slots, each capable of holding a variety of items (over 500 types), with a maximum stack size of 64. The formula for the number of inventory states is:
Inventory States (500 x 64)36
Final State Space The final state space is determined by the combination of block states, entity states, inventory states, and possible constraints (such game rules). Assuming all these state spaces are independent, the final state space formula is:
Final State Space = Block States x Entity States x Inventory States x Constraint
For Minecraft's s final state space, assuming a constraint factor of 10-2, we get:
Final State Space v 10 20
A.4.2. FURTHER ATTRIBUTES
Complexity Minecraft presents a highly complex environment composed of diverse elements, including blocks, creatures, terrain, and vegetation. This complexity challenges agents to learn adaptive behaviors across varied tasks, fostering generalization in dynamic setting.
16

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Open-endedness
The open-world nature of Minecraft exposes agents range of environments, requiring exploration and adaptive navigation. The flexibility define tasks of varying difficulty enables targeted evaluation of agent capabilities across diverse challenges.
Dynamism and Unpredictability Unlike static benchmarks, Minecraft features dynamic environmental changes such as day-night cycles, emergent entities, and varied terrain. Agents must develop adaptability and robust decision-making to handle unforeseen events, enhancing their generalization to real-world complexities.
Creativity and Innovation Minecraft supports open-ended tasks like construction and decoration, encouraging agents to explore diverse strategies for goal achievement. This fosters innovation and problem-solving in complex, unstructured settings.
Broad Challenge Coverage Minecraft serves as ideal platform for training and evaluating generalist agents, presenting four key challenges: Long-horizon Decision Making: Tasks decompose into flexible subtask sequences, requiring agents to plan beyond immediate actions. For example, acquiring wool may involve killing sheep, crafting from string, trading with villagers, demanding strategic foresight. Precise Control: Building and crafting require fine-grained movement and accurate object manipulation. Tasks like constructing a Nether portal necessitate precise block placement, challenging agents to handle nigh-dimensional action spaces with stability. Out-of diitiiiiistiitti Generalization: The dynamic environment introduces novel scenarios beyond training data. Agents must generalize to unseen conditions, such as avoiding hazards (e.g., lava) or adapting to ecosystem variations. Compositional Generalization: Agents should infer new task compositions from learned subskills. For instance, if trained craft sticks from planks and ladders from sticks, they should generalize crafting ladders from planks. The vast combinatorial task space in Minecraft makes compositional generalization crucial challenge.
Community and Resources Minecraft's s extensive community provides rich datasets, strategies, and problem-solving techniques. Open-source mods and plugins further enable controlled experimental setups for agent training.
Safe and Controlled Environment Minecraft offers a risk-free virtual world where researchers can precisely manipulate environmental parameters for reinforcement learning, ensuring reproducibility and safety in training agents.
17

---

MCU: An Evaluation Framework for Open-Ended Game Agents
B. Details of Task Generation
B.1. The Source Atomic Tasks
Filtering from Existing Benchmarks We curated tasks from established benchmarks, including Skill-Forge (Cai et al., 2024c), MineDojo (Fan et al., 2022), and prior Minecraft research (Baker et al., 2022; Yuan et al., 2023; Wang et al., 2023d;c). To refine the selection, we performed deduplication to remove redundant tasks, excluded those too difficult for human players (e.g., constructing highly complex architectures automated redstone circuits), and eliminated compositional tasks that could be decomposed into two or more atomic tasks.
Minecraft Wiki Resources Additional tasks were sourced from the official Minecraft Wiki, which categorizes various in-game activities. From these, extracted executable tasks, focusing primarily the Advancement page", which contains a curated list of diverse, engaging, and reasonably challenging tasks designed to enhance gameplay.
Synthesis from In-Game Information The Minecraft simulator defines various item properties, such as "craftable," "mineable," "caatalee"" and "breakable." Leveraging these definitions, we systematically generated atomic tasks, such as craft item X if X is craftable, and similarly for other properties. This method efficiently scales up the task set while ensuring comprehensive coverage of game elements.
LLM and Expert Brainstorming Beyond structured sources, we incorporated tasks generated through brainstorming sessions with expert Minecraft players and L.MMs. This approach was particularly valuable for designing open-ended, creative tasks that pose real-world challenges for agents. Brainstorming was conducted in collaboration with a university Minecraft club.
Since most tasks are derived from the official Minecraft Wiki and in in-game data, they are inherently reliable. Nevertheless, all tasks underwent rigorous validation through human inspection and automated scripts. The finalized atomic task list is provided as supplementary material alongside the code.
B.2. Task Configuration
In this section, we provide an overview of the key considerations for configuring a task, as introduced in Section 2.4. This section aims to offer intuitive understanding of task configuration. For detailed implementation and real-world configurations, please refer to Minestudio Section 2.2 and our integrated codebase.
The initial state of a task encapsulates all the information an agent can utilize based on its intended plan to execute the task. This includes not only the valid input but also any information the agent can derive or perceive, such as the observed 2D pixels of the game scene, inventory items, and coordinates (which can be accessed in-game by pressing F3, particularly the Y-dimension). The inventory I consists of two components: the necessary items for completing the task, denoted as In-without which the agent would not be able to plan and execute the task in a real game-and and additional random items, denoted as I,. Our objective is to manipulate these variables while ensuring that the random elements closely align with the real in-game distribution.
B.2.1. OBSERVATION AND COORDINATE
For fixed version of the Minecraft game, the observation and coordinate elements are determined by the world seed, the coordinate, and the facing direction. The world seed is entirely independent of other variables and can be chosen arbitrarily. The facing direction remains unchanged from its state before teleportation to the task scene, making it inherently random and not subject to manipulation.
Setting coordinate as a valid spawn location for a given task requires satisfying certain preconditions, such as specific biome types or other constraints defined by the game. For example, in the climb the mountain task, the agent must spawn in stony shore biome, while for a task that involves reaching village, the spawn location should be near one.
To facilitate reproducible task execution, we also collect a series of coordinate locations for each selected seed, corresponding to the required preconditions. Each (seed, precondition) pair can be mapped to multiple possible locations, allowing flexibility across different tasks. Minestudio allows for initializing game environment given these predefined random seeds.
hntope://minecraft wwki/w//aeeee Advancemmnt
18

---

MCU: An Evaluation Framework for Open-Ended Game Agents
B.2.2. INVENTORIES
The inventory consists of main components: In, the essential items required to complete the task, and Z,, set of random items acting distractors. Since multiple approaches may exist to accomplish the same goal, In is also treated as random variable. For instance, agent may either iron pickaxe diamond pickaxe to mine diamond ore. To ensure comprehensive testing, we incorporate a variety of possible item sets for In-
Regarding Z, we adjust its presence based on task difficulty. For some tasks, we omit Z, to reduce complexity. In other cases, we introduce random initial inventories by sampling from game snapshots derived from VPT contractor data. This approach ensures that the test environment remains diverse while maintaining a realistic distribution of inventory items.
19

---

MCU: An Evaluation Framework for Open-Ended Game Agents
C. Human Annotation
C.1. Minecraft Quiz
To get an anmotation for multi-dimensional task scores for trajectories used in our experiments (Section 3.2), we designed and distributed a questionnaire to confirm the annotators are familiar with Minecraft. The questionnaire is quiz, containing five multiple-choice questions with 25 options test their familiarity with Minecraft; each correctly answered option is worth 1 point. Then we filtered out the questionnaires with correct rate of less than 75%, and then considered their investigation parts for the remaining questionnaires. The quiz is shown in Table 8. We distributed the questionnaires to the signed up Minecraft annotators, and all of the annotators passed the quiz.
Table 8: The quiz our questionnaire (only questions are presented), is used to judge the respondents' familiarity with Minecraft. The problems are adapted from Milani et al. (2023).
No.
Question
Options
A. speed up the night.
B. change the respawn location.
1
A bed can
C. be crafted from drops of a certain animal in the game.
D. can be crafted by a furnace, but cannot be crafted by a crafting table.
killing hostile mobs.
B. mining trees.
2
You can acquire EXP when
C. jumping on a coal ore block. mining coal.
E. enchanting diamond sword.
A. Skeletons.
B. Zombies.
C. Sheep.
3
What mobs can deal damage to the player?
D. Pigs.
E. Creepers.
F. Enderman.
A. Apples.
B. Dirt.
C. Beef.
4
What items can be eaten?
D. Wheat.
E. Breads.
F. Spider eyes.
A. Wooden logs.
If you mine a block with bare hand, what
B. Wooden planks.
5 kinds of block can drop the corresponding item?
Iron ore.
D. Coal orc.
C.2. Human Videos For Tasks
Human videos serve two purposes: they are used as reference videos for GROOT and for comparison with the trajectory videos generated by the agent models. For each task, we select three world seeds: 19961103, 20010501, and 12345, For each (task, seed) pair, we manipulate the controllable parameters as described above, resulting in three distinct environment configuration files. For each configuration file, we record corresponding human video. Additionally, designate the first configuration file of seed 19961 103 as the reference video for GROOT.
C.3. Human rating system
In our dataset, there total of 60 tasks, each containing 10 rollout trajectories in video format. These videos capture gameplay records from either humans or various agents.
For the absolute rating task, we randomly select one task and present a corresponding video to human raters, who score it across six predefined dimensions.
For the comparative evaluation task, we randomly sample two different rollouts from the same task, referred to as Video
20

---

MCU: An Evaluation Framework for Open-Ended Game Agents
and Video B. These videos may both be from human players, both from agents, from human and the other from agent. Raters then asked to compare Video and Video B each dimension to determine which performs better.
The human rating interfaces are illustrated in Figure 5 and Figure 6. Taking the video comparison website an example, it is designed to evaluate agent performance by displaying two videos side by side, enabling human raters to directly compare their behaviors within the same task. The interface consists of the following modules:
1. Task Description Module: Positioned at the top-right, this module specifies the task to be evaluated (e.g., Survive Shield: Use shield to ward off zombies). It ensures that raters understand the objective before scoring.
2. Video Display Module: Two videos are presented side by side, each replaying agent's gameplay. This layout allows raters observe agent behaviors, mistakes, or innovative strategies in real-time.
3. Scoring Panel:
Located below the videos, this panel enables raters to assess agent performance across six dimensions. For each dimension, raters can indicate which agent performed better, mark a tie, or specify that neither agent took relevant actions.
4. Input and Submission Module: At the top-center, an input box collects rater identifiers to ensure traceability. A Submit button at the bottom sends completed ratings to the database, contributing to the dataset used for benchmarking.
04 Agent Arena X Benchmarking Agents in the Wild
Principle
You millaat the parformance of model from wwwllwwiiggggnenn aececte
Talk pregress Gvoull, which completed th leston??
agent
controff cap los wwwtttitt////oooaaaeaaaa suadar aotiona,
treative: cnytttccive ettempts senblss sed agent during doving tack.
dfficlnn: Talk Complstion Efficency.
snape Coreectly cheese aiver
Vote now!
yopuryyue
Tak
nanke_ddett Use aahedd momb
Clear
Submit
vile
Ferrcce oo Shed's Prandh Danond Natelt
Completoon Efficlecy
dr
suhrrit
berhidtt seating
techddinssttng
+
both sichdiitoohing
bash did nettingg
beeh etd notting
both did nething &
Skip
Figure 5: Video comparison website.
21

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Agent Arena X
Benchmarking Agents the Wild
Principle
the parirmance ofttee
followinn supectt:
progees wnthhagretemmppp.
Action
cocontrsiing
(tte///dudddoonn artious
Eroerecag. Recogriza and reciity it mittiies better.
Creather Any creatice stterphs exiibiediiyy by the agent during duing taok.
Material Correctly utitze mestetia...
Scoring
TASK: Piie grasp
attack: 0
back: 0
forward: 0
jump: 0
left: D
right: 0
sneak: 0
sprint: 0
use: 0
drop: 0
inventory:
hotbar.1.
hotbar.2: 0
hotbar.3: 0
hotbar.4: 0
hotbar.5: 0
hotbar.5: 0
hotbar.74 0
T Teteovedd nCeRL genté to -372.5, 648, $39.5
Kit erve
62
XX000X0
Ooned
Cartrdl
Tmar Comeciiooo
Ewellie Aftrrsts
Tesk Completion lence
Texl odlction application
Mostl
Good
-oodo
Good
Good
-0o03
submit
-Talr
-Tair
Berely
Poor
Poor
Poor
Foor
Hone
Wey Poor
Wery Feor
Wary Aoor
Poor
-Voty Foor
skip
Figure 6: Individual video rating website.
22

---

MCU: An Evaluation Framework for Open-Ended Game Agents
D. Generalization Experiments
In this experiment, we aim to demonstrate that developing open-ended agents requires selecting an environment with a vast state space. We seek show that when the state space is limited, learning within such an environment prone overfitting rather than fostering genuine skill acquisition. Consequently, when tested in open-ended environments (i.c., the test set prepared for this experiment), the agent is likely fail in handling unseen scenarios.
Table 9: Training Hyperparameters
Hyperparameter
Value
Hyperparameter
Value
Steps
25M
GAE Lambda
0.95
Learning Rate
2 x 10
PPO Clip
0.1
Scheduler
Linear
Policy Loss Weight
Optimizer
Adam
Value Loss Weight
0.5
Adam Epsilon
1 x 10-6
KL Loss Weight
0.3
Number of Training GPUs
2
KL Loss Decay 0.995
Batch Size per GPU
1
Reward Discount
0.999
100 Training State Space
1K Training State Space
1M Training State Space
20M Training State Space
10
10
10
10
Train
Test
8
8-
Score
6
4
2
0
0.50 1.00
1.50
2.00
0.50
1.00
1.50
2.00
0.50
1.00
1.50
2.00
0.50
1.00
1.50 2.00
Figure 7: Generalization performance across different training state sizes. The x-axis values should be multiplied by 10 We evaluate the agent on the full distribution of state space. The mean and standard deviation are computed over episodes. number training states increases, the gap between training and testing performance narrows: training curves become lower, while testing performance improves.
D.1. Experimental Setup
To assess the impact of state space size generalization, we conduct experiments on the Hunt Sheep task, using training sets ranging from 100 to 10, 000, 000 states. The state space primarily consists of variables such as equipment, mobs, biome, distance, and the number of sheep. For each setting, subset of the state space is sampled as the training set, while evaluation performed on the full state space.
We train agents using online reinforcement learning with Proximal Policy Optimization (PPO) for 25 million steps, requiring approximately 50 training hours three GPUs. Detailed hyperparameter configurations are listed in Table 9. The implementation is also supported by Minestudio.
D.2. Results
As shown in Figure 7, agents exhibit strong overfitting when trained small datasets. As the training size increases, the generalization gap progressively narrows, with test performance improving as agents become more adept at generalization. To close the generalization gap, agents require exposure to as many as 10 million states.
However, we also observe certain limitations in the agent's 's capacity. For example, at the start of the game, sheep may spawn behind the agent, outside its field of view. The agent fails to develop the behavior of scanning its surroundings before proceeding, often becoming distracted by other objects instead.
D.3. Discussion
These results highlight the necessity of complex environments like Minecraft, where achieving high performance in a limited state space does not necessarily imply strong generalization. Moreover, sufficiently large state spaces challenge
23

---

MCU: An Evaluation Framework for Open-Ended Game Agents
existing reinforcement learning algorithms, offering insights into their capacity limits, At a critical state-space size, further increasing the diversity of states ceases to yield additional improvements in test performance, suggesting upper bound on the agent's generalization ability.
24

---

MCU: An Evaluation Framework for Open-Ended Game Agents
E. Large-Scale Inference and Evaluation of MCU
This section demonstrates MCU's capability for large-scale inference and task evaluation. We conduct an extensive experiment involving 150 tasks, consisting of atomic tasks and 60 composite tasks. The experiments are performed on three agents: VPT (BC), VPT (RL), and STEVE-I. Due to the high of recording reference videos, GROOT is excluded from this evaluation. Specifically, atomic tasks conducted in hard mode, while composite tasks executed in simple mode.
E.1. Atomic Tasks Setup
For atomic tasks, task is randomly sampled from the atomic task list for each experiment. As outlined in Section 2.4, each selected task undergoes task configuration generation and verification ensure the necessary preconditions for execution. This process guarantees that tasks are properly configured and validated before being executed in the experimental environment.
E.2. Composite Tasks Setup
Following the methodology described in Section 2.3, we implement a composite task generation pipeline using logical connectors ("AND" and "OR") to combine multiple atomic tasks. Composite tasks are designed in three distinct formats, illustrated by the following examples (note that users can define additional compositions):
1. Three Atomic Tasks Combined with "AND" "OR"
"Find smooth red sandstone stairs OR mine yellow banner AND sell yellow dye"
"Find melon AND mine lodestone OR craft a wooden pickaxe"
2.
Two Atomic Tasks Combined with "AND" or "OR"
"Find melon AND mine lodestone"
"Craft wooden sword OR find diamond"
3.
Single Atomic Task with No Initial Tools Provided
"Mine red sandstone from scratch"
"Craft stone pickaxe from scratch"
In each case, atomic tasks are randomly sampled from a pool of over 3,000 available tasks. Composite tasks are designed to assess the system's 's ability to handle complex instructions and execute multiple tasks sequentially or in combination.
E.3. Experimental Conclusions
This experiment evaluates agent performance across broad task domain, focusing on atomic tasks in hard mode and compositional tasks. As shown in Figure 9 and Figure 8, agents struggle with these complex scenarios. STEVE-1 exhibits a performance decline compositional tasks, likely due to its reliance on short prompts, as discussed in Lifshitz et (2023). When encountering longer or unseen instructions, experiences distribbuttinn (OOD) issues. The low performance of all agents in creativity and error recognition aligns with the conclusions drawn in Figure lc.
25

---

MCU: An Evaluation Framework for Open-Ended Game Agents
for Progress
Healmap
Control
Healmap Error Recagnition and Correction Viee
with yol
The cater
Hastmap for Creative Attempts
Heatmap for Task Completion Efficiency
Heatmap for Material Selection and Usage
Figure 8: Performance different agents across atomic tasks,
26

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Heatmap for Task Progress
Heatmap for Action Contral E13
Heatmap for Error Recegnition and Sk0 XXX
wote cotceeetii 13
6/1
Healmap for Crealive Altempts
Healmap for Task Completion Efficienty
Healmap for Malerial Selection
Figure 9: Performance of diffcrent agents across 60 compositional tasks.
27

---

MCU: An Evaluation Framework for Open-Ended Game Agents
F. MCU-Turbo: A Standard Benchmark for Evaluating Minecraft Agents
We introduce MCU-Turbo, a canonical benchmark suite for systematically evaluating agents within the Minecraft Universal (MCU) framework. MCU-Turbo is designed standardized evaluation protocol, comprising 80 atomic tasks across 10 categories and 20 compositional tasks. Each task is assessed under two difficulty regimes-Simple and Hard-to rigorously test agent's capabilities in generalization, tool use, long-horizon planning, and robustness environmental variation. The following experiments establish baseline performance using current state-of-the-art agents.
F.1. Baseline Results in the Simple Evaluation Mode
Agents demonstrate varied competencies across different evaluation dimensions under the simple setting. Notably, aspects such creative behavior and error recognition remain challenging across the board, as illustrated in Table 10.
Table 10: Baseline agent performance in simple mode.
Agent Name
Task Progress
Action Control
Error Recognition
Creative Attempts
Task Efficiency
Material Usage
STEVE-I
31.4%
31.9%
13.1%
6.4%
23.2%
35.6%
VPT (BC)
29.1%
29.0%
11.8%
6.2%
21.3%
33.8%
VPT (RL)
25.9%
26.2%
8.8%
5.2%
18.9%
31.3%
JARVIS VLA
25.6%
27.8%
9.3%
5,5%
18.3%
30.5%
F22 Baseline Results in the Hard Evaluation Mode
In the Hard mode, agents are subjected to increased environmental complexity and the presence distractors. As shown in Table 11, performance consistently declines across all dimensions, underscoring the difficulty of generalization under more challenging conditions.
Table 11: Performance degradation in hard mode.
Agent Name
Task Progress
Action Control
Error Recognition
Creative Attempts
Task Efficiency
Material Usage
STEVE-1
23.1%
22.6%
6.9%
6.0%
16.6%
24.5%
VPT (BC)
23.0%
21.7%
6.2%
6.0%
15.6%
25.0%
VPT (RL)
21.0%
20.9%
5.0%
4.6%
14.4%
23.7%
JARVIS-VIL -AA
20.9%
22.3%
5.5%
4.3%
14.5%
22.3%
F3. Agent Performance Creative Programmatic Tasks
Creative tasks present substantially greater challenge compared to programmatic ones. For example, STEVE-1 exhibits 15.8% reduction in task progress when transitioning from programmatic to creative tasks, shown in Table 12. These results highlight the persistent difficulty of generalization in open-ended and less-structured settings.
Table 12: Performance comparison: creative vs, programmatic tasks.
Task Type
Task Progress
Action Control
Error Recognition
Creative Attempts
Task Efficiency
Material Usage
Programmatic 38.4%
36.5%
16.3%
7.7%
27.7%
43.8%
Creative 22.6%
26.2%
9.1%
4.7%
17.7%
25.4%
Drop
-15.8%
-10.3%
-7.2%
-3.0%
-10.0%
-18.4%
Overall, the MCU-Turbo benchmark provides fine-grained insights into agent capabilities across a diverse task spectrum, emphasizes enduring challenges in creativity and adaptive behavior, and aims steer future research toward the development of more general-
general-purpose, purpose, robust Minecraft agents.
28

---

MCU: An Evaluation Framework for Open-Ended Game Agents
G. Prompts
G.I. Prompt for Config Generation of Atomic Tasks
You are an expert of Minecraft, and I new Minecraft player.
2 You should give me all the necessary things need for completing the task.
I will give you the following information:
The taak I want complete:
7. You should perform the following steps to help me:
1. Tell me all valid items, mobs, biomes and all the necessary things to complete task
i
2. Formulate the above information
cheat commands;
10 3. Randomly generate one or two related but not necessarily cheat commands.
4. Only output one simple task description, a thinking process and musto__init_commande
13 e.g. The task I want to complete: Trade for iron helmet.
14 You should respond in the format as described below:
In order to trade for iron helmet, we need at least 5 emerald and armorer nearby. Task description: trade for iron helmet with villager
custom_ custom_init_commade::
/give
:nnerraft:armrr_eee
/give @s sinecraft:emerall 10
/summon villager "2 - (Profession: : raieeraft:armores VillagerData: o wtt.errraaaaamorrrrrreeeo
21
/give sinecraft:diamon 64
22
23 e. g. The task I want to complete: craft crafting table.
You should respond in the format described below:
25 In order to craft crafting table, need least planks.
26 Task description: craft a crafting table
27 custom_ ieeetom_iitt_commadde
/give és minecraft: oak_plakks 64
/give és minecraft: bread 16
/time set night
31
32
The task I want to complete: mine iron_ ore.
You should respond in the format described below: need at least
In order mine iron_ore,
pickaxe better one, and
have iron_ore nearby.
35 Task description: mine iron ore with a stone pickaze custom_inis www__initccommaann
/give és minecraff::rrom.piddaa
/execute as Op at run fill 22 -3 "4 coal_ore 39 /execute as @p at @s run fill --5 - --1 --3 iron_ore
40
/give @s mnoeerafttwwodem.ppee
e.g. The task I
complete: flying trident
rainy day.
43 You should respond in the format as described belox:
44 In order to flying trident on a rainy day, we need a trident enchanted with the riptide enchantment, and set the weather in rainy mode.
Task description: flying trident on rainy day
47 /weather rain
48 /give dp sinecraft:tridess
hranrtttttttriicmcccccccccaattte : "ww.eeiaat ,11 :1}}} 3 /give finecraft:fire.ce Shareeeechhargeeehhhctttttttttt thtc ( mine::"inecrttt 1lv1: 1v]:: 3
/give
52 e.g. The task I vant to complete: combat a zombie.
53 You should respond in the format as described below:
In order to combat zombie, we need weapons, armors and zombie nearby. Firstly, the Diamond Armor Set is the top-tier defensive gear providing exceptional protection. Secondly, the Diamond Sword, can swiftly dispatch zombies.
29

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Additionally, explosive items such Lava and TNT can also effectively deal with zombies. Zombies usually appear at night, so we need night vision. Task description: combat and kill a zombie custom_init_commandee
55
/replaceitem entity @s
head minecraft diamond_helmet
58
/replaceitem entity @s armor chest minecraft hhtpraaatddddmmmmmncneeeeesee
59
/replaceitem entity 6s armor.leg legs minecraft: sauccaat:ddlaoondleee.. ddianond_oogiings
60
/replaceitem entity 8s armor.feer minecraft datcraattdiimood_bboo. /replaceiten entity es weapon maapha.aiinhag minecraft dwarodd_swddd /time set night
62
63
/effect give da night_ 99999 250 true
64
/aummon ninecraft:zomble "3
/give tinecraft:tnt 64
67
e.g. The task want complete: find panda.
68
You should respond in the format as described below:
69
In order to find a panda, we need to make sure there is a panda nearby.
Task description: find panda
71
custom_ing iestom__nit__comandd::
72
/summon minecraft:panda /give @p minecraft:potart
-3
73
e.g. The task I want complete: interact with potion.
76 You should respond in the format as described below:
77
In order to interact with potion, you need at least one potion.
78
Task description: interact with potion
mustom_init_comme /weather rain
s1
/give
imecraft:potiae 2
e.g. The task I want complete: feed sheep.
You should respond in the format described below:
In order to feed a sheep, you may need wheat in inventory and sheep nearby. Task description: feed a sheep with wheat
36
87 custom_initt cattom_init_ccommaa
/summon sinecraft:sheep --2 /summon minecraft: /give minecraft: wheat
90
91
Note:
You should provide accurate information and executable cheat commands of Minecraft. cheat command should be more what is required. For example, the task need at least 10 emerald, provide 20 instead. 95 You should provide all the tools and environments required for completing the task.
94 The quantity items in
For decoration task, you can generate poppy, flower pot, torch, blue bed, red_dye and other similar things.
Do not give me the final target things directly my inventory.
98 Some crafting tasks are not completed using the crafting table, they could be done with tools like the furnace, enchanting table, or brewing stand and so on You need to select the appropriate tool.
Remember provide crafting table, furnace, enchanting table, brewing stand similar items, if the task requires it.
100
When use /fill command, ensure not to generate them in inaccessible locations (such
as high in the sky), and be extremely cautious not to suffocate the agent.
The distance for summoning items should be within blocks.
102
For the "find" task,
is better to use /sunmon /fill, or /execute Attention, there are certain items that cannot be directly summoned, such as trees, sugar cane, bubble_coral, etc. You should use /execute or /give
103
Listing 1: Prompt for Contig Generation Atomic Tasks
G.2. Prompt for Config Generation of Compositional Tasks
You are an expert of Minecraft, am new Minecraft player.
30

---

MCU: An Evaluation Framework for Open-Ended Game Agents
You should give all the necessary things need for completing the task. will give you the following information:
The task want to complete:
You should perform the following stepe to help me:
8 1. Tell me all valid items, mobs, biomes, and all the necessary things to complete the task.
2. Formulate the above information cheat commands.
3.
Only output one simple task description, thinking process, and muston_init_coomandde
e. The task I want complete: Trade for iron helmet mine stone.
You should respond in the format described below:
14
In order to trade for iron helmet, we need least emeralds and an armorer nearby
In order to mine stone, we need a pickaxe, like a diamond pickaxe. Task description: trade for an iron helmet with a villager and mine stone with diamond pickaxe
custom. custom_init_com inttoconnnt..o
-/give @s minecraft:emerale
/summon villager "2 "-2 (Profession: .inerrrff aroorr"" VillagerData: a
profession :
/give és minecraft :wwacraff:diammondd.oo diamond 2
20
21
e.g. The task I want to complete: craft a crafting table and go explore.
22
You should respond in the format as described as belov:
In order to craft crafting table, need at least 4 planks. To explore, you need nothing.
24
Task description: craft a crafting table custom_initi catm__iit__commaan
25
- -/give ds snecrafttoak.plaaat
64
28 e.g. The
I want complete: mine iron_ore and combat a zombie. You should respond in the format as described as below:
29
30
In order to mine iron_ore, we need at least stone pickaxe or a better one, and have iron_ore nearby. In order to combat zombie, zombie nearby. Firstly, the Diamond Armor Set is the top-tier defensive gear, providing exceptional protection. Secondly, the Diamond Sword can swiftly dispatch zombies. Zombies usually appear at night, so we need night vision. Task description: mine iron ore with stone pickaze and kill zombie custom custom_init_com iii.titttmmaaas
need weapons, armors, and
/give @s
34
/execute as @p at @s run fill --5 -22 -- -33 iron_ore
35
/replaceiten entity 0s armor head minecraft: cemecraft:dianee biamond_helme
/replaceites entity 0s armor chest minecraft: diaacafftdddaaodddceeeeltttt /replaceites entity d armor. minecraft :ltaroddoodlmmmd /replaceiten entity es weapon mainhand /time set night
legs minecraft: Bnoecraft:dimoon__1P bnlamond_legging
38
/replaceites entity
39
/effect give Qa night_vision 99999 250 true /auumon ninecraft:zomblc "3
43
44
The task I want to complete: flying trident on a rainy day.
You should respond in the format described as below: rainy day, need trident enchanted with the
In order fly with trident riptide enchantment, and set the weather rainy.
47
Task description: flying trident on a rainy day peeetom_init_comee
48
/veather rain
/give Op minecraft: trident tridentt (Enchantments [id: wwwnemmmannnttinnnnne riptiie" 1v1:1]]]
51
52 e.g The task I want to complete: find bubble_coral and feed a sheep
53 You should respond in the format as described as below:
In order find bubble_coral. order to a sheep, you may need wheat in inventory a sheep nearby.
need to make there bubble_coral nearby.
31

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Task description: find bubble_coral and feed sheep with wheat
56
57
/execute as Op at @s run fill - - --1 --3 bhbbreeftbuublleccoro. /sunmon ineecraftssees
--2
/aummon minecraft: sheep /give ès minecraft:whea. 5
61
g. The task I want to complete: interact with a potion and eat bread.
You should respond in the format described below: In order to interact with potion, you need at least one potion. order to eat bread, you need at least one bread.
64
65
Task description: interact with potion and eat bread custom custom_init_comme
/give Os minecraft: potion 2 /give es minecraft: :bead 2
69
70
Note:
73
You should provide accurate information and executable cheat commands Minecraft. The quantity of items in the cheat command should be more than what is required. For example, if the task needs at least 10 emeralds, provide 20 instead.
74
You should provide all the tools and environments required for completing the task. Attention, there are certain items that cannot directly summoned, such as trees, sugar cane, bubble_coral. etc. You should use /ezecute or /give.
76
77
For decoration tasks, you can generate poppies, flover pots, torches, blue beds, red_dye, and other similar things.
Do not give Some crafting tasks are not completed using the crafting table. They could be done with tools like the furnace, enchanting table, brewing stand, or similar tools. You need to select the appropriate tool.
the final target things directly
my inventory.
79
Remember provide crafting table, furnace, enchanting table, brewing stand, task requires it. Use /give.
similar items
SI When using /fill command, ensure not to generate them in inaccessible locations ( such as high in the sky), and be extremely cautious not to suffocate the agent. The distance for summoning items should be within 4 blocks.
Listing Prompt Config Generation Compositional Tasks
G.3. Prompt for Criteria Generation
You are expert of Minecraft and good at training agents in the AI field. will give you task description in Minecraft, and you need generate the score points for assessing the completion the task.
4. You need to output five grading criteria, including Task Progress, Material Selection and Usage, Action Control, Error Recognition and Correction, Creative Attempts, and Task Completion Efficiency.
You should formulate specific rules under each criterion for different tasks and don't modify the content between the two asterisks (** **)
Building tasks should focus whether the agent has completed the basic shape and structure.
For example, the task name is "build a house and decorate the tree", please generate the score points for it. You should respond in the format described below:
For build house:
12 **Task Progress: the key factors/steps for completing the task**
whether the agent builds four walls
whether the agent builds roof
whether the agent builds door
32

---

MCU: An Evaluation Framework for Open-Ended Game Agents
**Action Control: whether agents have unrelated operations of the task, including useless actions and redundant actions**
** E*ror Recognition and Correction: whether the agent can promptly identify and rectify its mistakes**
20
e.g., whether agents recognize the misaligned walls or incorrect material usage
21
whether the corrected results demonstrate improvement and reduce flaws in the final product.
23
Creative Attempts: any creative attempts exhibited by the agent during the task** e.g., uniquely shaped rooms, distinctive decorative elements like furniture
24
25
** Task Completion Efficiency**
whether the time taken by the agent complete the task falls within reasonable range
28
whether effective construction strategies were employed to minimize unnecssarry repetitions or errors
Material Selection Usage: whether the agent correctly utilizes the given materials*s
31
For decorate tree:
** Task Progress: the key factors/steps for completing the task**
34
there a tree in the image?
35
whether the agent put something on the tree
36
** Acttion Control: whether the agents have unrelated operations of the task, including useless actions and redundant actions**
38
e.g., a purposeless arrangement of blocks, destroying the tree, repeatedly clicking on items in the inventory without using them
E+ror Recognition and Correction: whether the agent promptly identify and rectify its mistakes** whether the corrected results demonstrate improvement and reduce flaws in the final product
41
Creative Attempts: any creative attempts exhibited by the agent during the task** e.g., Evaluate the overall visual effect the decoration, including color coordination, layout rationality, and symmetry e.g., Are the decorations on the tree diverse and abundant?
44
Task Completion Efficiency**
48
whether the time taken by the agent to complete the task falls within reasonable range whether effective construction strategies were employed to minimize unnecessary repetitions
51 Material Selection and Usage: whether the agent correctly utilizes the given materials*.
For example, the task name "dig three holes and fill one", please generate the score points for it. You should respond in the format described below:
57 Progress: the key factors/steps for completing the task**
58 whether the agent is digging the hole
59 whether the agent digs three holes hole
whether the agent fills
62 **Action Control: whether the agents have unrelated operations of the task, including useless actions and redundant actions*
63 e.g., wandering aimlessly, destroying the tree
33

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Error Recognition and Correction: whether agent can promptly identify and rectify its mistakes** whether the corrected results demonstrate improvement and reduce flaws in the final product
66
68
Creative Attempts: any creative attempts exhibited by the agent during the tasks* e.g., using different tools to dig the holes like hands or pickaxes
69
** Completion Efficiency**
72
whether the time taken by the agent complete the task falls within reasonable range
73
whether effective construction strategies were employed to minimize unnecessary repetitions errors
75
Material Selection Usage: whether the agent correctly utilizes the given materials*
76
Note:
For crafting tasks, is important distinguish whether recipe book is opened to identify the final item that needs to crafted. For motion tasks, auch as using an item or eating an item, attention should be paid to the interaction with the item.
79
Listing 3: Prompt for Criteria Generation
G.4. Prompt for Video Comparison
You are an expert in Minecraft and experienced in evaluating agents in the AI field. I will provide the following:
name
Grading criteria for the task
6
Two videos (Video A and Video B) of an agent performing the task.
The grading criteria contain several major categories (surrounded by **) and several evaluation rules under each major category.
You need to carefully compare the agent's 's performance in Videos A and B according to the avaluation rules and output one of the following:
"A better"
12 "B better"
13 "tie"
"both are bad"
The more an agent complies with rules in each criterion, better they perform.
17
Output **"A A is better"** when Video A performed better according to the evaluation 18
rules.
Output **"B is better"** when Video B performed better according to the evaluation rules.
20
Output *** when both videos demonstrate similar capabilities. Output b*th are bad"** when both videos have hardly done anything related to the rules have performed very poorly.
21
23
Before outputting the decision, you should list the relevant evidence from the videos to support your decision (within 80 words). Do not simply copy phrases from the rules.
25
You will
the decision across major criteria:
26
27
1. Task Progress
2. Material Selection and Usage
3. Action Control
4. Error Recognition Correction
34

---

MCU: An Evaluation Framework for Open-Ended Game Agents
5. Creative Attempts
52
6. Task Completion Efficiency
33
You should follow the output format below to organize your reaponse:
36
57
38
Task Progress:
evidence:
40
result: Xxx
41
42
Action Control: evidence: result: xxx
45
46
Error Recognition and Correction:
47
evidence: xxx
result: xxx
Creative Attempts:
51
evidence: xxx
result: xxx
54
Task Completion Efficiency: evidence: xxx
55
56
result: xxx
Material Selection and Usage:
39
evidence: xxx
60
result: XXx
Overall results:
Task Progress: xxx
64
Action Control: xxx
65
Error Recognition and Correction: xxx
Creative Attempta: xxx
Task Completion Efficiency:
Material Selection and Usage: xxx
69
Note:
73
If the evaluation rules include "e.g.", it is only an example, and you should not limited to the listed examples. Consider all phenomena that conform to the major criteria.
Task progress only considers the completion of key ateps of the task and does account for artistic qualities or similar aspects.
75 In categories like task progress, action control, task completion efficiency, and material selection and usage, should ideally choose either A or B as better.
Listing 4: Prompt for Video Comparison
G.5. Prompt for Individual Video Rating
You are an expert of Minecraft and good at evaluating agents the AI field. 2 will give you task name, a grading criteria for this task, and a video of an agent performing the task.
The grading criteria
several major criteria (surrounded by evaluation rules under each major criterion.
and several
You need to score the agent's operations in the video based on the evaluation rules. The more the agent complies with the rules in the criteria, the higher the score it receives.
you think the agent behavior does not relate to the stated rule, score 0.
35

---

For combinations like "a and b c," the average of the should calculated first, and then the higher value between this average the MCU: An Evaluation Framework for Open-Ended Game Agents
If you think the agent's behavior barely relates the stated rule, score 0. 0.1-0.3 8 If the agent's behavior partially relates to the rules, score 0.4-0.6 If the agent's behavior is mostly related to the rules, score 0.7-0.9
If the agent's behavior is completely related to the rules, score 1.
12 If you believe the agent complies with the rule, you should list the relevant evidence from the video (vithin 50 words). Do not simply copy the phrases from the rules. Please assign an appropriate score across five dimensions, including task progress, material selection and usage, action control, creative attempts, and task completion efficiency, based on the final evidence.
recognition and correction,
15 You should follow the following output format to organize outputs. "xxx" is the placeholder. Evidence can be more than one.
If there multiple tasks, such miming and crafting items, please provide comprehensive evaluation, responding with only one overall score.
17
Output format:
20
Task Progress: evidence xxx
21
22
Score: xxx
Action Control:
25
evidence xxx
26
Score: XXX
27
Error Recognition and Correction:
evidence XXx
30
Score: xxx
31
Creative Attempts:
evidence
Score: xхx
35
36
Task Completion Efficiency:
evidence xxI
Score: IXΣ
39
40
Material Selection and Usage:
evidence xxx
Score: xxx
44
Overall Scores:
45
Task Progress: xxx
Action Control: xxx
Error Recognition and Correction: xxx
Creative Attempts: xxx
19 Task Completion Efficiency: xxx
Material Selection and Usage: xxx
Note:
53 If the evaluation rules include "e.g.", it only an example and you should not be limited to the listed "e.g." All phenomena that conform to the major criteria should be considered.
Task progress only considers the completion of key steps of the task and is unrelated to artistic qualities or other such aspects.
55 You should ignore the text shoun on the video.
36 If the video has required materials for the task, they are automatically assigned by
the system and cannot be counted in the task progress.
from tasks and
score of task should be taken as the final result.
Listing Prompt Individual Video Rating
36

---

MCU: An Evaluation Framework for Open-Ended Game Agents
G.6. Pseudo-Code Examples
const doc yaml load (fa. readFileSynccc (task_conf, 'utf8')); // Extract the item name from the task description beask_description.see. " craft_a_")
3 const item_name
// Execute each initialization command to set up the environment custom, mustom_init_comm
forEach (command => bot. chat (command);
7 });
// Find the recipe for crafting the specified item const recipe bot. recipesForr (item_name, craftingTable); // Attempt
craft the item
try {
12
count, craftinggaabeeosstttonn)) console. log ('$(count) $(item_name) crafted successfully 9) } catch(err) { console error c ('aaallddttoccrrattttttn"""" err);
await bot. bot. craft
16
Listing 6: Mineflayer Craft Task Pscudo-Code
from meu, beu_haachmrrk import MinecraftWrapper, VLM [MM_Evluator from utility import load_config, from models import agent_
5 Step 1: Load task configuration for the benchmark config = coad_coofig("buiii ("build_house haa.huuuuuumm))))
Step 2: Initialize the environment with MinecraftWrapper (crrrft
env
wwel=coofig['leeell
Step 3: Initialize the agent (using custom model path and weights) agent agent_creatun (model.path, veight_path).cn cuda agent.evale
# Set the agent to evaluation mode Step 4: Get the initial state for the agent state agent. initial
14 Step 5: Start the environment and reset
obs, info env.reset
terminated, truncated False, False rollout_info
18 Step 6: Agent's rollout while not terminated and not truncated:
20
# Get action from the agent and update state action, state
poant.get_actions (obs, state) Step the environment with the agent's action obs, terminated, truncated, info env env. step stepiaciion) # Save frames (visual feedback from the environment) rollout_info... iallout_info.eppen((nntt append
22
23
34
Check if the agent succeeded in the task programaatiaally 27 success, video_path
srrceaauccsss.an_aaeeeviieee (rolooutttoooooo vin_evaluator = EVM_Eualutoor()
28 Step 7: Evaluate the agent using a Vision -aroonLLanguee Language Model (VLM)
29
vlm_score vin_evaluator evaluate (video_pata) 'build,criteria txt Sfuccess ((successs www.eeewaaaaasseeeeeeoooo
31 print Success
"core)")
Listing 7: MCU Evaluation Process Pseudo-Code
37

---

MCU: An Evaluation Framework for Open-Ended Game Agents
G.7. Case study
The following case clarifies the impact of each metric on evaluating generalization performance. Metries such as task progress and material selection assess basic task alignment, while action control and task efficiency provide insights into optimization strategies. Error correction and creative attempts, in contrast, measure higher-order generalization skills. These are critical for assessing agents in open-ended and complex scenarios, they reveal resilience to failure and capacity for novel strategies.
While Video outperformed Video A across most metrics, the weaknesses in creativity and error correction indicate areas where even high-pert performinffo agents fall short. Incorporating tailored training modules and broader tasks emphasizing these dimensions will enhance the benchmarks utility for developing and evaluating generalist agents.
Task Progress:
Video A: The agent collects dirt blocks and places them vertically but does not reach reasonable height.
Video B; The agent collects dirt blocks, places then vertically, and reaches reasonable height.
result: B is better
Action Control:
Video A: The agent places some blocks horizontally and in unrelated locations.
Video B: The agent places blocks vertically without unnecessary actions
result: B is better
11 Error Recognition and Correction:
12 Video A: The agent does not correct incorrectly placed blocks.
Video B: The agent does not make any noticeable errors that need correction.
reault: B better
16 Creative Attempts:
Video A: The agent does not show any creative attempts.
Video B: The agent does show any creative attempts.
result: tie
20
21 Task Completion Efficiency:
Video A: The agent takes longer time with unnecessary actions.
Video The agent completes the task efficiently without unnecessary actions.
24
result: B better
25
26 Material Selection and Usage:
Video A: The agent uses dirt blocks but places some blocks horizontally and in unrelated locations.
28
Video B: The agent exclusively uses dirt blocks and places them appropriately.
29
result: B is better
Overall reaults:
32
Task Progress: is better
33 Action Control: B is better
34 Error Recognition and Correction: B is better
Creative Attempts: tie
Task Completion Efficiency: B better
Material Selection Usage: better
Listing 8: Video Comparison Evaluation Results
38

---

MCU: An Evaluation Framework for Open-Ended Game Agents
Task Progress:**
Evidence: The agent placed two snow blocks vertically and a carved pumpkin on top, but no Snow Golem was created.
Score: Partially
**Action Control:*: +**
Evidence: The agent placed multiple unnecessary snow blocks around the structure. Score: Barely
Error Recognition and Correction:**
Evidence: The agent did not correct the placement of the carved pumpkin after failing to create Snow Golem.
Score: Barely
12
13 ** Creative Attempts:**
Evidence: No creative attempts decorations observed.
Score: None
17
Task Completion Efficiency:**
Evidence: The agent took excessive time with unnecessary placements and failed to complete the task.
Score: Barely
20
21
Material Selection and Usage:**
22
Evidence: Correct materials (snow blocks and carved pumpkin) were used, but not effectively.
Score: Partially
24
25
** Overall Scores:**
Task Progress: Partially
Action Control: Barely
28
Error Recognition and Correction: Barely
29 Creative Attempts: None
30 Task Completion Efficiency: Barely
Material Selection and Usage: Partially
Listing 9: Individual Video Evaluation Results
39


Reviews:
[{"review_id": "uoKFQBxtUO", "reviewer": "Reviewer_wrgp", "summary": "The paper introduces Minecraft Universe (MCU), a framework that improves evaluation for agents playing Minecraft. MCU includes over 3K composable atomic facts, an LLM-based generator that generates complex tasks by composing the atomic facts, and an automatic evaluation method with a VLM. The paper shows the advantage of MCU tasks vs previous works, and high correlation between human ratings and automatic evaluation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> My main concern regards the quality of the generated data. (How many errors are detected?) quantifying the advantages in Fig.2 (proportion of open-ended tasks, etc.).\n\nThank you for raising this important point.\n\nTo evaluate the quality of the generated data, we randomly sampled 100 atomic tasks and performed configuration generation. We observed a 9.8% error rate, primarily due to issues such as “unknown block_type or item”. In addition, we identified a 3.7% rate of undetected errors (e.g., omission of necessary items). After verification, over 95% of the configurations were confirmed to be accurate.\n\nTo quantify the advantages shown in Fig. 2, we processed the MineDojo creative task list using GPT-4o and filtered out tasks that are not solvable even by humans. From this, 521 tasks remained in MineDojo (note: these tasks are without runnable configurations). We further filtered programmatic tasks according to the definition of atomic tasks, resulting in 268 atomic tasks in MineDojo compared to 2,190 in MCU. Notably, every task in MCU is associated with both simple and hard variants, while MineDojo officially releases difficulty levels for only 64 tasks.\n\n|          | Creative tasks (filtered: solvable & non-repetitive) | Programmatic tasks (filtered: atomic) | Tasks with Difficulty Levels |\n| -------- | ---------------------------------------------------- | ------------------------------------- | ---------------------------- |\n| MineDojo | 521                                                  | 268                                   | 64                           |\n| MCU      | 1,262                                                | 2,190                                 | 345                          |\n\n> Could you please share more details on the brainstorming step (Lines 258–259)? How did this step improve task generation? Does it apply to all atomic tasks?\n\nAs mentioned in Lines 953–957, we incorporate tasks generated through both expert brainstorming and LLM collaboration. This process is useful for generating creative tasks. More specifically:\n\n1. **Expert-LLM collaboration:** Experts iteratively prompt LLMs to produce creative task ideas. For example, they begin with prompts such as *“Let’s brainstorm some creative tasks that intermediate-level Minecraft players could accomplish. {few examples}”* and refine outputs across rounds with feedback like *“Creating a Mona Lisa statue is too complex—can you offer simpler alternatives?”*\n2. **Expert proposal:** We work closely with a university Minecraft club, where experienced players propose engaging and imaginative tasks. This process contributed tasks such as “Prepare a birthday for your neighbor”.\n\nWhile brainstorming significantly enriches task diversity, not all tasks are derived from this process. Appendix C.1 details the four task sources where tasks are collected.\n\n> Doesn’t the few-shot config prompt (Lines 257–261) bias generations to examples similar to those in the prompt? \n\nThank you for this thoughtful observation. To mitigate prompt-induced bias and encourage diversity in LLM-generated configurations, we explicitly design our prompts to promote variability in initialization elements such as biome, weather, and player state (Lines 49–54). \n\nWe validate our current prompt through:\n\n- **Case study:** For the task “craft a crafting table”, we ran 10 generations. The results exhibited wide variation in commands (e.g., /time set day, /give oak_log, /setblock blue_bed), including different wood types like birch, oak, and spruce.\n- **Quantitative analysis**: For 5 randomly selected tasks, we conducted 100 generations each. The item-level overlap with few-shot examples was only 2%, and on average, 89% of the commands in each task were unique.\n\nWe will also keep improving our prompt to achieve better configuration generation diversity.\n\n> For the comparative evaluation (Lines 348–351), why are there “tie” and “both are bad” classes? What about cases where both generations are good?\n\nWhen both generated outputs are of similarly high quality and indistinguishable in effectiveness, we classify the comparison as a **“tie.”** Conversely, we mark them as **“both are bad.”** These cases are recorded solely for annotation clarity and are excluded from final comparison metrics. We only compute metrics based on pairs where a clear winner can be determined (line 351-353).\n\n> While this may be out of scope, the evaluation framework (Sec. 2.5) is only evaluated on a single game, Minecraft. Evaluating the generalizability of the method to additional multimodal tasks (e.g., web browsing, multimodal code generation, other gaming environments), can be helpful for future research.\n\nDue to the constraints of the rebuttal period, extending AutoEval to additional multimodal domains is currently beyond our scope. However, we agree that evaluating AutoEval across different domains would strengthen the contribution. We plan to explore in future work.", "claims_and_evidence": "My main concern regards the quality of the generated data. The verification step (lines 279-289) describes a re-generation process when errors are detected, but I did not see any analysis describing the quality of the generated examples. As atomic facts are composed automatically with an LLM, many issues can arise including hallucinations or low-diversity. I believe the paper could benefit from examining - the diversity of the generated tasks, the verification process in Sec.2.4 (How many errors are detected? How many errors are not detected?), and quantifying the advantages in Fig.2. (proportion of open-ended tasks, distribution of difficulty levels, etc.). \n\nWhile this may be out of scope, the evaluation framework (Sec.2.5) is only evaluated on a single game, Minecraft. Evaluating the generalizability of the method to additional multimodal tasks (e.g., web browsing, multimodal code generation, other gaming environments), can be helpful for future research.\n\nI am happy to consider raising my scores if these issues are addressed.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem. The only issue I see is that the evaluation framework is not shown to generalize to new environments (see Claims and Evidence).", "theoretical_claims": "None", "experimental_designs_or_analyses": "I checked the soundness of the experiments in Sec.3.", "relation_to_broader_scientific_literature": "Developing agents that autonomously plan complex “realistic” games such as Minecraft is a major challenge. The paper improves over prior work (e.g., Minedojo) by introducing new tasks and a new evaluation framework.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "None", "other_comments_or_suggestions": "None", "questions_for_authors": "- Could you please share more details on the brainstorming step (Lines 258-259). How did this step improve task generation? Does it apply to all atomic tasks?\n- Doesn’t the few-shot config prompt (Lines 257-261) bias generations to examples similar to those in the prompt? I believe the paper would benefit from additional discussion regarding the generated tasks.\n- For the comparative evaluation (Lines 348-351), why are there a “tie” and “both are bad” classes? What about cases where both generations are good?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V7upwzFZaK", "reviewer": "Reviewer_Huw3", "summary": "Minecraft Universe (MCU) introduces an advanced evaluation framework for AI agents in Minecraft. The authors build on a history of environments and datasets for Minecraft agents (e.g. MineStudio, MineDojo), to provide a polished evaluation framework with a large diversity of high-quality tasks and a novel automatic evaluation procedure.\n\nTheir experiments include an analysis of their auto-evaluation procedure using human annotation and evaluation of existing agents in MCU.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The correlation metrics on absolute ratings are hard to interpret in isolation (how do I interpret a correlation of 0.71?), unless e.g. you show that the correlation between the auto-eval and human ratings is close to the correlation between humans and other humans.\n\nWe appreciate this point and would like to clarify that we already report inter-rater agreement in Lines 380–384: *“we compute the* ***inter-rater agreement\\*** *for scoring the same trajectory, revealing a higher Pearson correlation for task progress (0.83) and a lower correlation for creativity (0.69).”* This provides a meaningful reference for interpreting the correlation between auto-eval and human ratings in context.\n\n> I am not convinced that composing multiple atomic tasks increases the complexity of the challenge significantly. The ceiling of difficulty for MCU is somewhat limited given the straightforward instruction-following nature of the tasks.\n\nOur core motivation for composing multiple atomic tasks is to introduce **long-horizon dependencies** that go beyond basic instruction following. These composite tasks require capabilities such as high-level **task planning** (e.g., determining the optimal execution order based on task dependencies), **memory management** (given the limited context length of policy models), and **error correction** (recovering from early mistakes that may impact downstream steps).\n\nFor instance, in the composed task *“mine iron, craft an iron pickaxe, and mine diamond”*, the agent must break this into subtasks with strict dependencies. If it fails to mine enough iron, it cannot proceed to craft the pickaxe, and thus cannot complete the final step. Success in such tasks requires more than executing steps sequentially—it demands adaptive reasoning, context-aware decision making, and robustness to cascading errors. These qualities represent a meaningful increase in complexity and present new research challenges.\n\n> How many hours would it take a human to complete the most difficult challenges in Minecraft?\n\nAs a reference, completing the “Ender Dragon” challenge—a commonly recognized long-horizon goal—typically takes a human player between **10 and 30 hours**. This includes time spent exploring, gathering resources, crafting appropriate gear, locating the stronghold, and finally, fighting the dragon (which itself takes 10–30 minutes). The overall process involves multi-stage planning, resource optimization, and consistent execution across diverse subtasks.\n\n> I would recommend establishing a “canonical” setup for users wishing to benchmark agents on MCU: how many tasks, which tasks, and a single final scalar metric that aggregates performance. The current setup is flexible but leaves too many variables up to the users.\n\nThank you for the valuable suggestion. In response, we are working on defining a standardized benchmark configuration for MCU. Specifically, we plan to select **10 primary categories** (excluding the “others” category) and curate **8 representative tasks** from each, resulting in a total of **80 tasks**. Each task will include both simple and hard manually verified configurations. For evaluation, we will report the **average performance across all tasks and dimensions** as a single scalar metric. This canonical setup will be included in the next version of the benchmark to promote consistency and comparability across future work.", "claims_and_evidence": "The main claim of having produced a new and high-quality Minecraft evaluation framework decomposes into two claims:\n- Improvement of agent tasks: Compared to existing task suites (MineDojo) from the authors' observations of issues in existing task datasets and building improvements upon that. While not quantitatively supported, previous issues are highlighted clearly and are improved upon by-design with their new filtered task dataset and configuration framework.\n- Improvement of automated evaluation: The authors design a new automatic evaluation framework that uses VLMs to rate agent trajectories on a predefined rubric of criteria. Section 3.1 provides convincing evidence that their method meaningfully improves upon prior methods (MineCLIP).\n\nSeparately, I want to address the claim of the enduring difficulty of MCU. The authors state \"Enabling the composition of atomic tasks into more intricate tasks. This approach exponentially increases both the number and complexity of tasks\"\n- I am not convinced that composing multiple atomic tasks increase the complexity of the challenge significantly. In general, I would expect that if an agent can perform each of the atomic tasks robustly, composing the tasks together does not make the tasks much harder than simply requiring more time to perform them (complexity stacks ~additively rather than in some more complex way)", "methods_and_evaluation_criteria": "Evaluation of the AutoEval methods:\n- The authors use comparisons of pairwise preferences and absolute ratings against human annotated trajectories to show that their method has a higher agreement with humans.\n- While the evidence does support that their method outperforms competing methods, the setup does have limitations\n  - Pairwise preference datasets are likely to be generally \"easy\" unless the trajectory pairs are carefully chosen to test \"close-calls\" (and I don't think this was done). \n  - On the other hand, the correlation metrics on absolute ratings are hard to interpret in isolation (how do I interpret a correlation of 0.71?), unless e.g. you show that the correlation between the auto-eval and human ratings is close to the correlation between humans and other humans.\n- Overall, I expect that this rubric-based method is a useful and decent judge, but likely to have weaknesses (LLMs tend to be overly optimistic judges) and may be vulnerable to spec gaming / goodhart's law at the edges.\n\nEvaluation of existing agents:\n- The setup for evaluating agents is mostly just MCU's setup, which is sound.\n- The choice of agent baselines and tasks appears suitable.", "theoretical_claims": "No key theoretical claims.", "experimental_designs_or_analyses": "Nothing to add beyond what I've already mentioned.", "relation_to_broader_scientific_literature": "MCU is the latest addition in a long history of Minecraft-based model/agent evaluation/training environments (MineRL, MineDojo, MineStudio). It brings welcome improvements to the space, in particular creating a streamlined and effective evaluation environment which is suitable and practical for evaluating present-day agents.", "essential_references_not_discussed": "None that come to mind.", "other_strengths_and_weaknesses": "Strengths\n- I think MCU is a welcome addition to the agentic benchmarks space, and I will be interested to see how modern agents (e.g. systems like general-purpose Computer-Use agents) perform on it.\n- MCU has a broad diversity of tasks and solid auto-evaluation framework, making it interesting and useful. \n\nWeaknesses\n- I would recommend establishing a \"canonical\" setup for users wishing to benchmark agents on MCU: how many tasks, which tasks, and a single final scalar metric that aggregates performance. The current setup is flexible but leaves too many variables up to the users which can make it difficult to make comparisons.\n- The ceiling of difficulty for MCU is somewhat limited given the straightforward instruction-following nature of the tasks. This is sufficient for the type of agents currently tested in this work, but I expect general agents to quickly become adept at this (in the same way that current LLMs are great at instruction-following), and then this benchmark may be saturated soon.\n  - I am not confident about this claim, so if I am wrong, it would be useful to see some discussion about the difficulty ceiling of the tasks, e.g. how many hours it would take a human to complete the most difficult challenges, and what kinds of reasoning/planning complexities are present.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None. Thank you for your work!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "ZXnCKiLapq", "reviewer": "Reviewer_pB3F", "summary": "MCU proposes a scalable benchmark for open-ended game agents in Minecraft. It introduces 3,452 atomic tasks, spanning 11 categories and 41 subcategories, that can be dynamically composed into complex challenges. Using an LLM-based task configuration generator, the framework creates diverse, realistic scenarios, while a VLM–powered AutoEval system automatically scores agent performance with over 90% alignment to human judgments. Experimental results show that even state-of-the-art agents (e.g., VPT variants, STEVE-I, and GROOT) struggle with task diversity and complexity, highlighting the need for further advances in generalization and creativity in open-world environments.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Due to character limitations, we regret that we can only provide a simplified version of the response below:\n\n> What was the rationale behind selecting these six specific evaluation criteria?\n>\n> Some evaluation criteria may not translate well across all atomic task categories. \n>\n> It is not clear how these criteria should be distinguished. \n\nThe selection of the six evaluation criteria in the MCU framework was guided by the goal of capturing a broad yet practical spectrum of agent competencies needed in open-ended, real-world environments. \n\nRegarding your specific points:\n\n- **Creativity** of simple tasks (e.g., *find_pink_tulip*) can involve creative strategies (e.g., climbing for better visibility).\n- **Error correction** is most informative when agents do make mistakes, but agents that complete tasks without errors are awarded the highest possible score in this category.\n- **Task progress** in open-ended tasks like *decorate_the_cave* can be quantified by tracking the extent and coherence of changes made relative to the environment’s initial state.\n- **Material usage** in \"find\" tasks may involve the use of navigation items such as boats or compasses.\n- On the point of **overlap between criteria**, such as between **task efficiency** and **action control**, we agree that correlations exist. However, they emphasize different aspects: task efficiency focuses on outcomes (e.g., time to completion), while action control emphasizes the process (e.g., avoiding redundant or counterproductive actions).\n\n> Configuration Diversity?\n\nPlease refer to the response to Reviewer TkoE in Question 1.\n\n> How extensive can the soft constraints be? How do you address scalability concerns for complex tasks?\n\n- **Soft Constraints**: Defined by “bad-case analysis”, generally applicable across tasks, ensuring feasibility within Minecraft’s rules. Currently, 8 soft constraints are used.\n- **Scalability**: The task complexity concern is related to current LLM capabilities. As  LLMs improve, more sophisticated tasks will become feasible. Generating task specifications is easier than solving them, so we expect scalability to improve naturally.\n\n> Separate prompts per criterion?\n\n- A test on 20 random samples showed F1 = 91.2% with separate prompts vs. 90.6% with a combined prompt.\n- However, cost rose nearly sixfold since reprocessing was required for each criterion.\n- Thus, a combined approach balances efficiency and accuracy.\n\n> Evaluation Cost & Open-Source Models\n\n- We do not plan to evaluate 3,000 tasks at once; we are selecting a representative subset.\n- For the 35 tasks in the paper, each run (10 trials per task) costs 13.2 USD per agent using GPT-4o.\n- We also tested MiniCPM-V-2_6 (8B, Aug. 2024) and JarvisVLA (7B, Mar. 2025). While open-source VLMs still lag, they are catching up.\n\n| Method          | Survive | Build | Craft | Mine | Explore | Average |\n| --------------- | ------- | ----- | ----- | ---- | ------- | ------- |\n| MineClip        | 11.0    | 45.0  | 44.0  | 73.0 | 0.0     | 34.6    |\n| Ours(MiniCPM)   | 65.0    | 43.0  | 80.0  | 59.0 | 53.0    | 60.0    |\n| Ours(JarvisVLA) | 73.0    | 62.0  | 73.0  | 84.0 | 65.0    | 71.4    |\n| Ours(GPT-4o)    | 100.0   | 85.0  | 62.0  | 71.0 | 100.0   | 84.0    |\n\n> How are hard mode tasks created? \n\nCreated by prompting the LLM to add complexity and constraints (e.g., obstacles, random disturbances). e.g., In “mine_iron_ore,” visually similar ores (gold/coal) are placed nearby to increase ambiguity.\n\n> What criteria or time limits determine when an episode or task is terminated?\n\nFollowing GROOT settings: 600 steps for atomic tasks, 12,000 steps for compositional tasks.\n\n> Can you provide a detailed breakdown of how many tasks originate from each source?\n\n| Minedojo | SkillForge | Minecraft Wiki | In-game data | LLM & expert brainstorming |\n| -------- | ---------- | -------------- | ------------ | -------------------------- |\n| 5.2%     | 0.9%       | 12.0%          | 79.2%        | 2.7%                       |\n\n> Why did you choose not to include tasks with additional constraints or extra criteria?\n\n- Additional constraints would break the “atomic” nature (Section 2.3).\n- Atomic tasks are core test units; overlapping tasks dilute evaluation efficiency.\n\n> Is sampling 1 frame every 30 enough?\n\n| Inteval  | 20   | 25   | 30   | 40   |\n| -------- | ---- | ---- | ---- | ---- |\n| F1 score | 0.75 | 0.87 | 0.90 | 0.68 |\n\n- Too many frames (interval=20) approach GPT-4o’s limit (50 images), risking overload.\n- Too few (interval=40) may miss vital details.\n- Interval=30 strikes the best balance (F1=0.90).\n\n> Evaluating progress in “decorate_the_cave”?\n\nThe criteria generated by LLM: e.g. agents select a suitable cave to decorate, add decorative elements inside the cave, and ensure the cave is well-lit and visually appealing\n\n> Regarding open-sourced datasets, typos, formatting, references etc.\n\nThank you for your meticulous attention to detail. We will fix these issues.", "claims_and_evidence": "1. The authors elaborate on why Minecraft serves as a good evaluation basis with scalable complexity and open-endedness by calculating its vast state space.\n2. They assert strong task diversity by aggregating 3,452 atomic tasks across multiple categories, enabling scalable composite task creation, and including an LLM-based configuration generator mechanism requiring some manual efforts, and they compare MCU to MineDojo to demonstrate improved task solvability and diversity.\n3. The authors prove the effectiveness of AutoEval by crowdsourcing human-evaluated data and reporting a 91.5% alignment between their VLM-based method and the annotated data.\n4. The authors reveal the current limitations of open-ended agents designed for prior Minecraft environments, experimentally showing that SOTA agents (e.g., VPT, STEVE-I, GROOT) struggle as task complexity and diversity increases.\n5. The authors claim that their six evaluation criteria comprehensively capture the challenges of real-world tasks, although this claim remains debatable due to the reasons found in the following sections.", "methods_and_evaluation_criteria": "1. MCU effectively captures the unpredictable nature of open-world gameplay by blending intra-task diversity, through LLM-generated variations in biomes, weather, and player states, with a broad inter-task diversity, as seen in its 3,452 atomic tasks. These tasks span challenges from precise control to complex reasoning and knowledge application, drawing from sources such as the Minecraft Wiki, MineDojo, SkillForge, in-game data, and original contributions from the authors. Moreover, the tasks vary in difficulty, creating a dynamic testing environment to assess agent generalizability in conditions mirroring the complexity of real gameplay.\n2. MCU’s defines atomic tasks solely by their goal, independent of the method, tools, or specific conditions. This approach isolates the fundamental capability the agent must master. For instance, a task like \"mine stone\" can be instantiated under varying initial conditions, ensuring that the agent develops a robust policy rather than overfitting to a single scenario. Moreover, the ability to combine these tasks using logical operators enables the creation of progressively complex challenges that mirror the intricacies of real-world task descriptions. The authors should explicitly quantify how many tasks originate from each source to properly credit them and determine the authors’ original contribution. MineDojo may have repetitive and unsolvable tasks, but it is not clear from the paper to what extent. It would also be beneficial to incorporate task descriptions with constraints.\n3. The authors utilize an LLM-based configuration generator combined with a self-verification loop, leveraging feedback from the Minecraft simulator to ensure task validity. To further enhance this mechanism, they introduce manually defined *soft constraints* in the prompt, guiding the LLM toward generating feasible tasks. However, specifying these soft constraints requires human expertise and detailed prior knowledge of Minecraft, introducing substantial manual effort. This reliance on human-defined constraints limits scalability and task variety.\n4. Prompting GPT to supply surplus resources to ensure solvability can inadvertently lower task difficulty, as agents can exploit the abundance rather than managing resources efficiently. Moreover, surplus resources can mask configuration inaccuracies and complicate evaluation, as agents might complete tasks in unintended ways. For example, if the same task is generated under different biomes or weather conditions but one configuration provides significantly more resources, it may falsely appear easier, leading to erroneous conclusions about the relative difficulty of the biome or weather conditions.\n5. The evaluation pipeline’s reliance on the GPT-4v API may create cost barriers for users, limiting accessibility and scalability. Since other alternatives are not tested, it is not clear whether they would be compatible. While many state-of-the-art LLM APIs, including GPT-4v, incur costs, some open-source alternatives exist, though they may not match GPT-4v’s performance.\n6. Some evaluation criteria may not translate well across all atomic task categories. First, evaluating **creativity** or material usage for a cut-and-dry task like *find_pink_tulip* seems misplaced because the task is inherently straightforward, with little room for creative problem-solving. Second, the **material usage** metric does not seem very relevant for the “Motion” and “Find & Navigation” categories, where materials are rarely required unless the agent needs to craft a pickaxe for mining or a boat for exploring. Third, if an agent performs a task flawlessly with no errors, measuring **error correction** becomes moot since no corrections are necessary. Fourth, it is unclear how to measure **task progress** for an open-ended task such as *decorate_the_cave*. Finally, the presence of trade-offs between evaluation criteria is noteworthy. For instance, higher creativity scores might inherently require the agent to use materials less efficiently or to sacrifice task efficiency for completion. However, certain evaluation metrics exhibit significant overlap. **Avoiding unrelated or unnecessary actions** and adequately using material naturally correlates with **task efficiency**. It is not clear how these criteria should be distinguished.", "theoretical_claims": "There are no formal proofs in this work; the focus is on empirical validation and system design. As such, theoretical claims aren’t a central aspect of the paper.", "experimental_designs_or_analyses": "1. The evaluation pipeline samples every 30th frame from the agent’s trajectory, but this approach may miss important details. For example, an agent could perform unnecessary actions for 29 frames and then behave correctly on the 30th, which could misrepresent its overall performance. It would be useful for the authors to justify the choice of 30 frames and to show that a denser sampling rate does not degrade alignment with human evaluations.\n2. The authors developed a dedicated website for crowdsourcing human evaluations. The annotators’ competency is preemptively checked, and the trajectory comparison is well-designed and intuitive.\n3. AutoEval’s reliability depends heavily on the quality of the underlying VLM. Although I am skeptical about using VLMs to evaluate RL agents in open-ended settings because these models may not fully capture the nuances of gameplay, the authors report a 91.5% average agreement rate with human assessments across the evaluation criteria. Future improvements of VLMs could further improve this alignment.", "relation_to_broader_scientific_literature": "MCU builds directly on earlier evaluation suites like MineDojo and SkillForge, while incorporating recent advances in LLM and VLM technology. It extends prior work by addressing scalability and task diversity, thereby making a significant contribution to the literature on open-ended game agents. The authors discuss the main existing Minecraft-based environments and open-ended agents that have been evaluated in these environments.", "essential_references_not_discussed": "The authors could reference a recent work [1] on Open-World RL on Minecraft.\n\n[1] Li, Jiajian, et al. \"Open-World Reinforcement Learning over Long Short-Term Imagination.\" *arXiv preprint arXiv:2410.03618* (2024).", "other_strengths_and_weaknesses": "I have incorporated all the strengths and weaknesses in the sections above.", "other_comments_or_suggestions": "1. Typo line 134 “based on **a** vision-language model”\n2. Some of the results in Table 1 appear to be incorrectly bolded.\n3. When evaluating composite tasks, it would be informative to examine which subtask the agent chooses in OR compositions, as well as the sequence of subtasks completed in AND compositions.\n4. The titles of the right columns in Figures 8 and 9 don’t fit on the page.", "questions_for_authors": "1. Given that LLMs can produce similar outputs when repeatedly prompted with the same instructions, how do you ensure sufficient diversity in the generated configurations?\n2. Have you considered integrating alternative, possibly open-source, VLMs for automatic evaluation to reduce costs and increase flexibility, and how might these alternatives compare in performance?\n3. MBU uses manually defined soft constraints in the configuration prompts. How extensive can these constraints be, and how do you address scalability concerns for complex, multi-step tasks where capturing all nuances might be difficult and require manual efforts?\n4. Currently, all evaluation criteria are addressed using a single prompt. Have you tested using separate prompts for each criterion, and if so, do these yield improved performance by allowing the VLM to focus on one aspect at a time?\n5. What was the rationale behind selecting these six specific evaluation criteria?\n6. Will the dataset of evaluated trajectories be open-sourced to support further research and the development of alternative evaluation methods?\n7. Section 3.1 mentions 500 trajectories, while Appendix D.3 indicates 600. Could you clarify which is correct?\n8. Why do the annotation website's individual video questions include “which agent” for the “Task Progress” and “Action” principles?\n9. What is the cost of running a full evaluation using MCU, given the reliance on VLMs like GPT-4v? Is cost scalability a concern for potential users?\n10. How are Hard mode tasks created? Do they require task-specific heuristics, or is there a general method to scale task difficulty across different categories and subcategories?\n11. What criteria or time limits determine when an episode or task is terminated?\n12. Can you provide a detailed breakdown of how many tasks originate from each source (e.g., Minecraft Wiki, MineDojo, SkillForge, in-game data, and original designs by the authors)?\n13. Why did you choose not to include tasks with additional constraints or extra criteria, and would their inclusion benefit the benchmark?\n14. How should the evaluation system assess error correction when an agent makes no errors?\n15. Is sampling one frame every 30 frames sufficient to capture all critical details for evaluation? Have you explored how different sampling frequencies might affect the alignment between AutoEval and human judgments?\n16. How can **task progression** be evaluated for an open-ended task such as “*decorate_the_cave*”?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ymm5ibIhtg", "reviewer": "Reviewer_TkoE", "summary": "This paper presents Minecraft Universe (MCU), a novel evaluation framework designed to benchmark open-ended AI agents in Minecraft. The authors develop a system with three main innovations: a large-scale collection of atomic tasks spanning from combining diverse categories and subcategories; an LLM-based task configuration generator that creates diverse task initialization conditions; and a VLM-based automatic evaluation system that rates agent performance across six dimensions. Their experiments with state-of-the-art Minecraft agents (including GROOT, STEVE-I, and VPT variants) reveal significant limitations in current models with the MCU.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> LLM diversity on task generation may be limited\n\nThank you for this thoughtful observation. To mitigate prompt-induced bias and encourage diversity in LLM-generated configurations, we explicitly design our prompts to promote variability in initialization elements such as biome, weather, and player state (Lines 49–54). \n\nWe validate our current prompt through:\n\n- **Case study:** For the task “craft a crafting table”, we ran 10 generations. The results exhibited wide variation in commands (e.g., /time set day, /give oak_log, /setblock blue_bed), including different wood types like birch, oak, and spruce.\n- **Quantitative analysis**: For 5 randomly selected tasks, we conducted 100 generations each. The item-level overlap with few-shot examples was only 2%, and on average, 89% of the commands in each task were unique.\n\nWe will also keep improving our prompt to achieve better configuration generation diversity.\n\n> It is not clear how creativity is measured, e.g. is it more creative if I paint 50% of the house of a different colour vs 20%? Are these steps taken? Do you measure if agents take any kind of new steps in the trajectory of building a house or if they take creative steps that, while might be problematic, could make sense of how to build a house?\n\nCreativity is not simply a matter of percentages. Painting 50% or 20% of the house a different color doesn’t inherently determine creativity. What truly matters is the intention and impact behind the choice. If painting a portion of the house in a different color introduces a unique aesthetic, challenges norms, or enhances the overall design, then it can be considered creative, regardless of the percentage. Even if some steps are risky, they can still be creative if they contribute to the goal and add diversity. \n\nHere are the specific criteria for \"building a house\" generated by the criteria prompt in Appendix G.3:\n\nCreative Attempts: creative attempts exhibited by the agent during doing task\n\n- e.g. using different materials to create a visually appealing design\n- e.g. unique themes or styles applied to the house, use of diverse materials for decoration, innovative lighting techniques\n\nAlso note that, The VLM is guided to consider, but not be limited to, the examples provided. \n\n> Please discuss how MCU enables researchers to assess open-ended capabilities that are not present in other non-Minecraft benchmarks\n\nIn the Introduction and Section 2.1 of our paper, we explain Minecraft’s unique suitability as a benchmark for open-ended capabilities and our motivation for selecting it. In brief:\n\n1.**Vast State Space**\n\nMinecraft’s state space is extraordinarily large—reputedly surpassing the number of atoms in the universe—allowing for virtually unlimited configurations of biomes, blocks, and entities. In contrast, many alternative benchmarks (e.g., Habitat-lab) rely on pre-scanned real world data which makes memorization rather than genuine generalization feasible.\n\n2.**High Task Diversity**\n\nMCU defines 3,452 atomic tasks that can be combined into a staggering number of compositional tasks covering navigation, crafting, combat, and more. This level of intra- and inter-task diversity is uncommon in other platforms, making it difficult to replicate the breadth and depth of challenges found in Minecraft. Other open-ended benchmarks primarily focus on a final goal, such as finding the Oracle in NetHack, are limited in inter-task diversity.\n\n3.**Open-Endedness**\n\nMinecraft naturally accommodates complex, multi-step tasks (e.g., obtaining diamonds) that require agents to plan and coordinate over extended horizons, remember terrain and resource locations, and adapt to dynamic objectives. Such open-ended exploration is central to fostering agents with genuinely flexible and robust capabilities—traits that are difficult to assess using more constrained benchmarks.\n\n> About references to other open-ended benchmarks and RL literature\n\nThank you for your valuable suggestions. We will incorporate these references and discussions in the next version of the paper.", "claims_and_evidence": "I find that the claims in this submission are well-supported, clearly written and easy to follow.", "methods_and_evaluation_criteria": "Yes, they do", "theoretical_claims": "there are not theorethical claims", "experimental_designs_or_analyses": "I examined the experimental designs and analyses in this paper, particularly focusing on their automatic evaluation (AutoEval) system and the agent benchmarking experiments.\n\nThe validation of AutoEval appears methodologically sound. The authors compare their VLM-based evaluation approach against human annotations using appropriate metrics (F1 scores for comparative evaluations and correlation coefficients for absolute ratings). They collect a reasonably sized dataset of 500 trajectories across 60 tasks with human annotations from 20 expert Minecraft players. The correlations only drop in more subject metrics like creativety which is intuitive.\n\nFor the agent evaluation experiments, they test four foundation agents (GROOT, STEVE-I, VPT(BC), and VPT(RL)) on a diverse subset of tasks with multiple random seeds, which is a reasonable approach. They evaluate both inter-task generalization (across different task categories) and intra-task generalization (across difficulty levels), which addresses important dimensions of agent capabilities.\n\nTheir main limitations I noticed were 1) LLM diversity on task generation may be limited 2) It is not clear how creativity is measured, e.g. is it more creative if I paint 50% of the house of a different colour vs 20%?  Is it the steps taken? do you measure if the agents take any kind of new steps in the trajectory of building a house or if they take creative steps that, while might be problematic, could make sense on how to build a house?", "relation_to_broader_scientific_literature": "I missed connecting this work to the FormalMethods + RL literature. Task definition and comoposition in the MCU has strong connections to a this domain of works, where many employed the minecraft-inspired environment from [1]. I would strongly encourage the author to include a paragraph to likn this paper to that body of work since it feels just natural that future FM +RL literature transition to the MCU asa default benchmark.\n\nThe paper does include references to previous works on minecraft as an environment, open-ended agents and llm-as-a-judge work. However, I also missed connections to other open-ended benchmarks like Habitat-lab or Nethack and what aspects of intelligence the MCU measures that those other benchmark don't\n\n[1] Andreas, Jacob, Dan Klein, and Sergey Levine. \"Modular multitask reinforcement learning with policy sketches.\" International conference on machine learning. PMLR, 2017.", "essential_references_not_discussed": "None that I am aware of", "other_strengths_and_weaknesses": "--", "other_comments_or_suggestions": "If possible I would  include a brief summary of the literature overview in the intro. Personally I am not a big fan of relegating the entire related work discussion to the appendix", "questions_for_authors": "Please discuss how MCU enables researches to assess open-ended capabilities that are not present in other non-Minecraft benchmarks", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors introduce Minecraft Universe (MCU), a framework that builds tasks from the Minecraft game to evaluate AI agents' task solving capabilities. There are three key components in MCU: 1) a diverse and large set of atomic tasks that can be composed into more difficult and longer horizon tasks; 2) a task composition mechanism that composes sub-tasks into bigger tasks in a meaningful way; and 3) a evaluation framework that scores an AI agent's task solving video, the authors show the automatically generated scores have high alignment with human ratings. 

This is clearly a strong paper where all reviewers agree that the proposed presented data can be benefit the ICML community, the procedure of collecting the data can be even more useful.  

The reveiwers' concerns were mostly clarification questions on data collection, the properties of the collected tasks, and on how the composition mechanism works. The authors did a great job providing additonal details and clarifications, including some additional experimental results (evaluation on open-sourced models). Reviewers agree the rebuttal has made the work stronger. 

AC's additional comments:
1. Please make sure to include the Related Work section in the main body in your future submissions, this is a very risky move. I wouldn't be surprised if another AC decides to desk reject such a submission, because this is unfair to other submission authors, who have less space to present their work. I decide to let go this time because all reviewers really like the work, and I agree the work can be valuable to the community. 
2. I fail to find if the authors plan to open-source the work, including the data, the data generation pipeline, and the automatic evaluation pipeline. I would encourage the authors to open-source otherwise the work's value will significantly drop. 
3. I feel it's appropriate to cite the "OG" work on Minecraft environment/framework: The Malmo Platform for Artificial Intelligence Experimentation. Johnson et al., 2016. Many existing Minecraft frameworks/benchmarks are built on top of Malmo. 
4. I agree with Reviewer Huw3 that establishing a "canonical" setup for users wishing to benchmark agents on MCU is super important. It may be useful to make a leaderboard website where researchers can submit their scores (with paper supporting their numbers), this could accelerate this line of research a lot. 
5. Minor typo: L328 G.1 -> G.3
Decision: Accept (spotlight poster)