Craftium: Creating Efficient Environments for Open-Ended and Embodied Agents Beyond Gridworlds

Authors: Mikel Malagón, Josu Ceberio, Jose A. Lozano
Primary area: reinforcement learning
Keywords: reinforcement learning, environment, embodied, open-ended, continual learning, meta reinforcement learning

Abstract:
Advancements in open-ended and embodied AI require highly adaptable and computationally efficient environments. Yet, existing platforms often lack the flexibility, efficiency, or richness necessary to drive progress in these areas. Research in fields related to open-endedness, such as unsupervised environment design and continual reinforcement learning, usually defaults to simplistic 2D grid environments, as more complex alternatives are either too rigid or computationally expensive. Conversely, in embodied AI, the field relies on fully featured video games like Minecraft, which are rich in content but computationally inefficient and offer limited customization for creating new tasks. This paper introduces Craftium, a framework based on the open-source Minetest game engine, providing a highly customizable, easy-to-use, and efficient platform for building rich Minecraft-like 3D environments. We showcase environments of different complexity and nature: from simple reinforcement learning tasks to a vast world with many creatures and biomes, along with a customizable procedural task generator. Conducted benchmarks show that Craftium substantially improves the computational cost of Minecraft-based frameworks, achieving +2K steps per second more.

Decision: reject
Reviewer scores: 8, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1729585169006

thread_type:
official_review

<thread_object>
creator: reviewer_tgc5

<content>
summary:
This paper proposes `Craftium`, a framework based on MineTest. It is similar with MineCraft but open source and efficient.

strengths:
1. The simulation is efficient.
2. The writing is good and experiments is detailed.
3. This framework is easy to use.

weaknesses:
1. The difference between `Craftium` and Minecraft is not highlighted in figure 2 and I think it is important. What parts of figure 2 are not supported in MineCraft? 
2. The motivation of experiments are not clear.  In 3.5.2, the line 418 said 
> This example demonstrates how
Craftium environments can be used to analyze and evaluate the ability of large multimodal model based agents to leverage world knowledge to approach complex open-world tasks." 

Is MineCraft also able to do this thing? If it is, I don't think it is your contribution. What's the meaning of rightmost icons of Figure 11?

3. In table 1, one of the advantage of `Craftium`  is "GYMNASIUM".  Can you give a detailed analysis of how GYMNASIUM implementation helps RL training?

4. There is no detailed or systematic analysis or examples on advantages of `Craftium` over MineCraft for RL. During RL, what information can `Craftium` give for a better learning but MineCraft cannot? I think detailed internal state/information should be the advantage.

questions:
- See "Weakness".
- What's the motivation of compare LLava-Agent and PPO+LSTM in Sec 3.5.2?
- Can MineCraft support RL or environment generation? 
- I don't understand what's the disadvantages of MineCraft except the efficiency after reading. Can you compare the provided APIs between MineCraft and `Craftium`?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729627703910

thread_type:
official_review

<thread_object>
creator: reviewer_cdjs

<content>
summary:
The paper introduces Craftium, a framework for creating rich 3D environments for embodied and open-ended AI research. Key contributions:
- A flexible, efficient platform built on the open-source Minetest game engine
- Easy environment creation using Lua API instead of domain-specific languages
- Significant performance improvements over Minecraft-based frameworks (+2K steps/sec)
- Integration with the Gymnasium interface for compatibility with existing RL tools
- Comprehensive examples across different use cases (classic RL, open-ended learning, continual learning)

strengths:
- Built on efficient C++ codebase (Minetest) versus Java (Minecraft)
- Strong compatibility through Gymnasium interface
- Addresses real needs in embodied/open-ended AI research
- Comprehensive benchmarks against VizDoom and MineDojo
- Diverse example environments demonstrating flexibility

weaknesses:
- Currently only supports single-agent scenarios
- Could use more ablation studies on design choices

The paper's claim that "Research in fields related to open-endedness... usually defaults to simplistic 2D grid environments" needs revision, as it overlooks significant recent developments in the field.

The emergence of foundation models has enabled several works to generate rich environments, such as OMNI-EPIC and EnvGen which use large language models for environment generation, but also environments such as Craftax which provides a sophisticated benchmark.

These works demonstrate that the field has already begun moving beyond simple grid-world environments. The authors should acknowledge this recent progress and better position their work within the context of these advances in environment generation for open-ended learning.

[1] OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code\
[2] EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents\
[3] Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learning

questions:
- What are the key limitations in extending to multi-agent scenarios? Do you foresee potential challenges in extending Craftium to multi-agent scenarios, such as synchronization issues, increased computational requirements, or modifications needed to the Minetest engine.
- How does computational performance scale with environment complexity?
- What influenced the choice of Lua for environment creation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688084378

thread_type:
official_review

<thread_object>
creator: reviewer_mcbn

<content>
summary:
The paper presents Craftium, a framework designed to provide computationally efficient and customizable 3D environments for open-ended and embodied AI research, using the Minetest engine. Craftium aims to address limitations in existing environments, which are either computationally demanding or lack flexibility. The authors highlight its compatibility with the Gymnasium API, ease of use in procedural generation, and increased performance over other platforms like Minecraft.

strengths:
- Craftium provides a complex 3D environment similar to Minecraft but offers faster performance, achieving over 2,000 steps per second compared to Minecraft-based frameworks.
- It allows for efficient procedural generation and supports a variety of reinforcement learning (RL) tasks.

weaknesses:
- The paper lacks a detailed comparison with other widely used simulators in robotics, such as IsaacGym, PyBullet, or MuJoCo, which also support 3D environments and open-world scenarios.
- The experimental comparisons are weak, and details on baseline and alternative environment performance are limited. The paper doesn’t clearly outline the real-world application of its contributions or benchmarks for similar environment capabilities in existing platforms.
- Overall, the novelty of the framework is minimal, as the setup resembles that of existing environments, with modifications focused on efficiency rather than new functionality.
- In the procedural environment generation section, experiments demonstrated the generation of environments. However, all the generated environments can be represented using ASCII, meaning they are 2D environments. What is shown is a 3D representation of a 2D environment.
- In table 1, it is difficult to evaluate exactly if a framework is active or not after some time. There is no guarantee that this framework will still be active after a few months. Hence, I think that it is not a comparable characteristic.
- Missing related works that show the possibility of creating 3D environments in an open-ended way [1, 2, 3]. 

[1] Lee, J., Hwangbo, J., Wellhausen, L., Koltun, V., & Hutter, M. (2020). Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47), eabc5986.
[2] Faldor, M., Zhang, J., Cully, A., & Clune, J. (2024). OMNI-EPIC: Open-endedness via Models of human Notions of Interestingness with Environments Programmed in Code. arXiv preprint arXiv:2405.15568.
[3] Earle, S., Kokkinos, F., Nie, Y., Togelius, J., & Raileanu, R. (2024, May). Dreamcraft: Text-guided generation of functional 3D environments in Minecraft. In Proceedings of the 19th International Conference on the Foundations of Digital Games (pp. 1-15).

questions:
- How does Craftium's procedural generation approach compare in diversity and scalability to existing frameworks in continuous and open-ended RL settings?
- Why is “spiders attack” a challenging task? It seems to have denser reward signals than “chop tree”.
- How do training of the tasks compare with that of existing simulators (e.g., MineDojo)? What is the performance against real time?
- In Figure 11, why are there 2 maximum red lines for the LLava-Agent? Why is it in % of steps, and not absolute number of steps?
- In Figure 12, why did the agent not manage to get any reward in the first 2 generated environments? They look like the simplest environments. In any of the generated environments, did the agent successfully complete the task of reaching the diamond?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730712524810

thread_type:
official_review

<thread_object>
creator: reviewer_bq4e

<content>
summary:
The paper introduces Craftium, a new platform leveraging the open-source Minetest game engine to enable the creation of customizable, efficient, and rich 3D environments for research in RL and open-ended agent scenarios. Craftium is designed to bridge the gap between more simple 2D environments and computationally demanding, closed-source 3D platforms like Minecraft. Through various benchmark tests, the authors demonstrate that Craftium achieves significant performance gains, being notable faster than the Minecraft-based frameworks, and present use cases illustrating its flexibility for diverse RL tasks, procedural environment generation, and complex open-world simulations.

Minor comments:
- There is a typo on line 1027 “Then, the agent that the agent, …” 
- This paper seems like relevant work: Grbic, Djordje, et al. "Evocraft: A new challenge for open-endedness." Applications of Evolutionary Computation: 24th International Conference, EvoApplications 2021.

strengths:
- Craftium addresses a critical limitation in AI research, especially in RL and embodied AI, by enabling complex and computationally feasible 3D environments that are both customizable and efficient.

- The paper presents comprehensive benchmark comparisons, e.g. MineDojo and VizDoom, demonstrating its advantage over current platforms in terms of computational performance and versatility.

- The authors provide clear explanations of Craftium’s architecture, including how Minetest’s open-source flexibility allows for modifications crucial for RL, such as reward functions and synchronous client-server interactions.

weaknesses:
The framework currently supports single-agent settings, with multi-agent scenarios identified as future work. Given the growing interest in multi-agent RL, this limitation restricts Craftium’s immediate applicability for research in cooperative and competitive agent settings.

While the paper makes a valuable contribution to the field of environment design for embodied and open-ended AI research, the framework is basically "just" a wrapper around Minetest, thus limiting its novelty.

questions:
Does the current framework support Minecraft Redstone components? These would be useful to allow agents to build more complex structures.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732199407409

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
First, we thank the reviewers for dedicating their time and effort to read our paper and provide thoughtful feedback.

**Summary:** We are pleased to see that reviewers stated that *"Craftium addresses a critical limitation in AI research"* (**R-Bq4e**), *"Addresses real needs in embodied/open-ended AI research"* (**R-CDjS**), and features *"diverse examples demonstrating its flexibility"* (**R-CDjS**). Reviewers also highlight that the paper is well-written (**R-TGC5**). 

At the same time, two main concerns were raised regarding the **(1)** lack of multi-agent support, **R-Bq4e** and **R-CDjS**, and **(2)** the novelty of our contributions, **R-mCbn** and **R-TGC5**. We are happy to announce that **we have added multi-agent support to Craftium**. This new feature directly solves the first concern and significantly enhances the novelty of Craftium, **addressing both issues**. Below we summarize how these concerns were addressed.

**(1) Multi-agent support.** 

We have incorporated multi-agent support into Craftium. This update was facilitated by Minetest's existing multiplayer capabilities and Craftium's initial design, built with a future multi-agent extension in mind.

Previously Craftium executed its environments by running Minetest in single-client mode, where the server and the client were handled by the same process. Now, Craftium can run Minetest in dedicated server mode, allowing multiple clients to connect (see the updated [Figure 2](https://drive.google.com/file/d/1Q6YICabmiXheNm0HwJh0sWcmFseIW9Va/view?usp=sharing)). This feature enables instantiating a Minetest client per agent, that connects to the same server, effectively sharing the same environment. Importantly, this new feature is an addition, that preserves all previous features including single-agent mode, ease of use, procedural generation, versatility, open-world capabilities, computational efficiency, etc.

To illustrate the new capability, we have created an example multi-agent environment (see [screenshot](https://drive.google.com/file/d/1yVrWaJu_5a4u9G9tmmKvMqIFwCsWImuT/view?usp=sharing)). The environment is a one vs one multi-agent combat game, where agents are rewarded (+1) for punching other agents and penalized (-0.1) on damage. We have trained the agents using self-play, a popular approach for this type of scenario [1]. As can be seen in this [plot](https://drive.google.com/file/d/1KCIOGHFhMx0EoIj5_Pk2nPoMcVlmdNAW/view?usp=sharing), the agents can successfully learn the proposed task, showcasing the multi-agent research possibilities of the framework. In the following days, we will upload a revised version of the paper version including the multi-agent feature as well as the new environment and experiments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In its current state, Craftium is an impressive engineering contribution, but the paper, in its current state, does not provide strong evidence supporting its motivation to provide a rich, open-ended environment simulator that provides benefits over preexisting environments. Undoubtedly, Minecraft is a powerful system with the potential to provide this, but the current experiments focus almost entirely on environments whose dynamics are approximately the same as offered in commonly used environments such as NLE/MiniHack, Crafter/Craftax, MuJoCo, and other versions of Minecraft for RL (as pointed out by reviewer mCbn and TGC5). Therefore, the primary contribution demonstrated in this paper is a more efficient wrapper of Minecraft. I encourage the authors to refocus their experiment design to truly demonstrate the claimed benefits of their implementation of Minecraft.