Benchmarking the Spectrum of Agent Capabilities
Authors: Danijar Hafner
Venue: ICLR 2022 Poster (2022)
Keywords: Evaluation
Reinforcement Learning
Environment
Benchmark
Unsupervised Reinforcement Learning
Exploration

Abstract: Evaluating the general abilities of intelligent agents requires complex simulation environments. Existing benchmarks typically evaluate only one narrow task per environment, requiring researchers to perform expensive training runs on many different environments. We introduce Crafter, an open world survival game with visual inputs that evaluates a wide range of general abilities within a single environment. Agents either learn from the provided reward signal or through intrinsic objectives and are evaluated by semantically meaningful achievements that can be unlocked during each episode, such as discovering resources and crafting tools. Consistently unlocking all achievements requires strong generalization, deep exploration, and long-term reasoning. We experimentally verify that Crafter is of appropriate difficulty to drive future research and provide baselines scores of reward agents and unsupervised agents. Furthermore, we observe sophisticated behaviors emerging from maximizing the reward signal, such as building tunnel systems, bridges, houses, and plantations. We hope that Crafter will accelerate research progress by quickly evaluating a wide spectrum of abilities.

Published as a conference paper at ICLR 2022
BENCHMARKING THE SPECTRUM OF AGENT CAPABILITIES
Danijar Hafner
Google Research, Brain Team University of Toronto mai l@danijar.com com
ABSTRACT
Evaluating the general abilities of intelligent agents requires complex simulation environments. Existing benchmarks typically evaluate only one narrow task per environment, requiring researchers to perform expensive training runs many different environments. We introduce Crafter, an open world survival game with visual inputs that evaluates a wide range of general abilities within single environment. Agents either learn from the provided reward signal or through intrinsic objectives and are evaluated by semantically meaningful achievements that can be unlocked during each episode, such as discovering resources and crafting tools. Consistently unlocking all achievements requires strong generalization, deep exploration, and long-tew tengtterm reasoning. We experimentally verify that Crafter is of appropriate difficulty drive future research and provide baselines scores reward agents and unsupervised agents. Furthermore, we observe sophisticated behaviors emerging from maximizing the reward signal, such as building tunnel systems, bridges, houses, and plantations. We hope that Crafter will accelerate research progress by quickly evaluating a wide spectrum of abilities.
INTRODUCTION
Crafter is an open world survival game for reinforcement learning research. Shown in Figure 1, Crafter features randomly generated 2D worlds with forests, lakes, moun- tains, and caves. The player needs to forage for food and water, find shelter to sleep, defend against monsters, col- lect materials, and build tools. The game mechanics are inspired by the popular game Minecraft and were simpli- fied and optimized for research productivity. Crafter aims to be fruitful benchmark for reinforcement learning by focusing on the following design goals:
Research challenges Crafter poses substantial chal- lenges current methods. Procedural generation requires strong generalization, the technology tree evaluates wide and deep exploration, image observations calls for repre- sentation learning, repeated subtasks and sparse rewards evaluate long-term reasoning and credit assignment.
Agent view of procedu- rally generated world Crafter, showing terrain types, resources, and creatures. Agents learn from image inputs and aim unlock range of semantically mean- ingful achievements during each episode. The achievements evaluate strong gen- cralization, wide and deep exploration, and long-term reasoning.
Figure 1:
Meaningful evaluation Agents are evaluated by range of achievements that can be unlocked in each episode. The achievements correspond meaningful milestones in behavior, offering insights into ability spec- trum of both reward agents and unsupervised agents.
Iteration speed Crafter evaluates many agent abilities within a single environment, vastly reducing the compu- tational requirements over benchmarks suites that require training many separate environments from scratch. while making it more likely that the measured perfor- mance is representative of new domains.

---

Published as a conference paper at ICLR 2022
$ python3 -m pip install crafter
Install Crafter
$ python3 -m pip install pygame
Needed for human interface Start the game
$ python3 -m crafter. run run_gui
Figure 2: Play Crafter yourself through the human interface.
RELATED WORK
Benchmarks have been a driving force behind the progress and successes of reinforcement learning as a field (Bellemare al., 2013; Brockman et al., 2016; Kempka et al., 2016; Beattie al., 2016; Tassa et al., 2018; Juliani et al., 2018). Benchmarks often require a large amount of computational resources and yet only test a small fraction of the abilities that a general agent should master (Cobbe et al., 2020). This section directly compares Crafter to four particularly related benchmarks.
Minecraft Crafter is inspired by the successful 3D video game Minecraft, which is available to researchers via Malmo (Johnson et al., 2016) and MineRL (Guss et al., 2019). Minecraft features diverse open worlds with randomly generated and modifiable terrain, as well as many different resources, tools, and monsters. However, Minecraft too complex to be solved by current methods (Milani et al., 2020). it is unclear by what metric agents should be evaluated by, the environment is slow, and can be difficult to use because it requires Java and a window server. In comparison, Crafter captures many principles of Minecraft in simple and fast environment, where results can be obtained in a matter of hours, and where a large number of semantically meaningful evaluation metrics are available for reinforcement learning with or without extrinsic reward. The goal of Crafter is not to replace Minecraft but progress faster towards it.
Atari The Atari Learning Environment (Bellemare et al., 2013) has been the gold standard bench- mark in reinforcement learning. It comprises around 54 individual games, depending on the evaluation protocol (Mnih et al., 2015; Schulman et al., 2017; Badia et al., 2020; Hafner et al., 2020). While the large number of games tests different abilities of agents, they require a large amount of computation. The recommended protocol of training the agent with 5 random seeds on each game for 200M steps requires over 2000 GPU days (Castro et al., 2018; Hessel et al., 2018). This substantially slows down experimentation and makes the complete benchmark infeasible for most academic labs. Moreover, Atari games are nearly deterministic, so agents can approximately memorize their action sequences and are not required to generalize to new situations (Machado et al., 2018).
ProcGen ProcGen (Cobbe et al., 2020) provides benchmark that is similar to Atari but explicitly addresses the determinism present in Atari through the use of procedural generation and randomized textures. It consists of 16 games, where each episode features a randomly generated level layout. Similarly, Crafter relies on procedural generation to provide a different world map with different distribution of resources and monsters for every episode. However, ProcGen still requires training methods on 16 individual games for 200M environment steps, which each focus on a narrow aspect of an agent's s general abilities. In comparison, Crafter evaluates many different abilities of an agent by training only on single environment for 5M steps, substantially accelerating experimentation.
NetHack NetHack (Küttler et al., 2020) is text-based game, where the player traverses a randomly generated system of dungeons with many different items and creatures. Unlike the other discussed environments, NetHack uses symbolic inputs and thus does not evaluate an agent's 's ability to learn representations of high-dimensional inputs. The game is challenging due to the large amount of knowledge required about the many different items and their effects, even for human players. As a result, NetHack requires many environment steps for agents to acquire this domain-specific knowledge; 1B steps were used in the original paper. In contrast, Crafter generates diverse complex worlds from simple underlying rules, focusing more on generalization than memorization of facts.
3 CRAFTER BENCHMARK
We introduce Crafter, a benchmark that evaluates a variety of agent abilities in a single environment. This section describes the game mechanies of the environment, the interface of agent inputs and actions, the evaluation protocol that is based on a range of semantically meaningful achievements, and the open challenges that Crafter poses for future research.
2

---

Published as a conference paper at ICLR 2022
Crafter procedurally generates a unique world for every episode that features several terrain types: grasslands, forests, lakes, mountains, caves. Memorizing action sequences is thus not viable strategy and agents are forced to learn behaviors that generalize to new situations.
Figure 3:
3.1 GAME MECHANICS
This section describes the game mechanics of Crafter, namely its randomly generated world maps, the levels health and other internal quantities that the player has to maintain, the resources can collect and objects and tools can make from them, as well the creatures and how they are influenced by the time of day. The images of all materials and objects are shown in Figure E.I. All randomness in the environment uniquely determined by an integer seed that is derived from the initial seed passed to the environment and the episode number.
Terrain generation unique world is generated for every episode, shown in Figure 3. The world leverages an underlying grid of 64 x 64 cells but the agent only observes the world through pixel images. The terrain features grasslands, lakes, and mountains. Lakes can have shores, grasslands can have forests, and mountains can have caves, ores, and lava. These are determined by OpenSimplex noise (Spencer, 2014), form of locally smooth noise. Within the areas determined by noise, objects appear with equal probability at any location, such as trees in forests and skeletons in caves.
Health and survival The player has levels of health, food, water, and rest that must prevent from reaching zero. The levels for food, water, and rest decrease over time and are restored by drinking from lake, chasing cows or growing fruits to eat, and sleeping in places where monsters cannot attack. Once of the three levels reaches zero. the player starts losing health points. It can also lose health points when attacked by monsters. When the health points reach zero, the player dies. time when the player is not hungry, thirsty, sleepy.
Health points regenerate
Resources and crafting There are many resources, such as saplings, wood, stone, coal, iron, and diamonds, the player can collect in its inventory and use to build tools and place objects in the world. Many of the resources require tools that the place must first build from more basic resources, leading to technology tree with several levels. Standing nearby table enables the player to craft wood pickaxes and swords, as well as stone pickaxes and stone swords. Crafting a furnace from stone enables crafting iron pickaxes and iron swords from both iron, coal, and wood.
Creatures and night Creatures are initialized in random locations and move randomly. Zombies and cows live in grasslands and are automatically spawned and despawned ensure given amount of creatures. At night, the agent's view is restricted and noisy and a larger number of zombies is spawned. This makes it difficult to survive without securing shelter, such cave. Skeletons live in caves and try to keep the player at distance to shoot arrows at the player. The player can interact with creatures decrease their health points. Cows move randomly and offer food source.
3.2 ENVIRONMENT INTERFACE
This section defines the specification of the environment, explains the available actions, agent inputs, episode termination, and additional information provided by the environment. The design goal of these is to make the environment easy to use and inspect. The environment uses the Gym interface (Brockman et al., 2016) with visual agent inputs and flat categorical actions.
3

---

Published as a conference paper at ICLR 2022
Collect Wood
Place Table
Eat Cow
Collect Sapling
Collect Drink
Make Wood Pickaxe
Make Wood Sword
Place Plant
Defeat Zombie
Collect Stone
Place Stone
Eat Plant
Defeat Skeleton
Make Stone Pickaxe
Make Stone Sword
Wake Up
Place Furnace
Collect Coal
Collect Iron
Collect Diamond
Make Iron Pickaxe
Make Iron Sword
Figure 4: The 22 achievements that can unlocked within each episode. The arrows indicate which achievements will be completed along the way of working toward more challenging achievements. Several of the earlier tasks have to be repeated multiple times, such as collecting resources, to progress further. A reward is only given when an achievement is unlocked for the first time during the episode.
Observations Agent receive color images of size 64 x 64 x 3 as their only inputs. The image shows local top-down view of the map, reaching 4 cells and east and 3 cells north and south of the player position. Below this view of the world, the image shows the current inventory state of the player, including its health points, food, water, and rest levels, collected materials, and crafted tools. The agent needs to learn to read its inventory state out of the image.
The action space is flat categorical space with 17 actions, represented by integer indices. The actions allow the player to move in all directions along the grid, interact with the object in front of it, go sleep, place objects, and make tools. Each object and tool has separate action associated with it. Tools are kept in the inventory whereas objects are automatically placed in front of the player. If the agent does not hold the required materials for making object tool, the action has effect.
Actions
Each episode terminates when the player's health points reach O. This happen when the player dies out of hunger, thirst, or tiredness, when attacked by zombie or skeleton, or when falling into lava. Health points automatically regenerate, as long as the agent thirsty, or sleepy. There is no negative reward for dying, as the reward signal already includes penalty for losing health points. Episodes also end when reaching the time limit of 10,000 steps.
Termination
too hungry,
Additional information The environment allows access to privileged information about the world state that the agent is forbidden to observe. This includes numeric inventory counts, achievement counts, the current coordinate of the player on the grid, and semantic grid representation of the map. These can be used for debugging purposes or for other research scenarios, such predicting the underlying environment state to evaluate representation learning or video prediction models.
3.3 EVALUATION PROTOCOL
To evaluate the diverse abilities artificial agents on Crafter, we define two benchmarks. The first benchmark allows agents to access a provided reward signal, while the second benchmark does not and requires agents to purely learn from intrinsic objectives. Besides access to the provided reward signal, the evaluation protocols are identical. An agent is granted budget of 1M environment steps to interact with the environment. The agent performance evaluated through success rates of the individual achievements throughout its training, well as aggregated score.
4

---

Published as a conference paper at ICLR 2022
Achievements To evaluate a wide spectrum of agent abilities, Crafter defines 22 achievements. The achievements are shown in Figure 4 and correspond to semantically meaningful behaviors, such as collecting various resources, building objects and tools, finding food and water, defeating monsters, and waking up safely after sleeping. The achievements cover wide range of difficulties, making them suitable to evaluate both weak and strong players and providing continuous feedback throughout the development process of new methods. Some achievements are independent of each other to test for breadth of exploration, while others depend on each other to test for deep exploration.
Reward Crafter provides a sparse reward signal that is the sum of two components. The main component is a reward of +1 every time the agent unlocks cach achievement for the first time during the current episode. The second component is reward of -0.1 for every health point lost and reward of +0.1 for every health point that is regenerated. Because the maximum number of health points is 9, the second reward component only affects the first decimal of the episode return, and ceiling the episode return yields the number of achievements unlocked during the episode.
Success rates The success rates offer insights into the breadth of abilities learned by an agent. The success rates are computed separately for each of the achievements, as the fraction of training episodes during which the agent has unlocked the achievement at least once. It is computed across all episodes that lead up to the budget of 1M environment steps, requiring agents be data-efficient. Note that the number of environment steps is fixed but the number of episodes can differ between agents. Unlocking an achievement more than once per episode does not affect the success rate.
Score The score summarizes the agent abilities into a single number. It computed by aggregating the success rates for the individual achievements. Unlocking difficult achievements, even if it happens rarely, should contribute more than increasing the success rate of achievements that are already unlocked frequently even further. To account for the range of difficulties of the achievements, we average the success rates log-space, known as the geometric mean. 2 Unlike the reward, the score thus takes the achievement's difficulties into account, without having to know them beforehand.
Discussion
Aggregating across tasks via geometric mean weighs tasks based their difficulty to the agent, resulting in higher scores for agents that explore more broadly. For example, collecting a diamond 1% of the time instead of 0% is meaningful improvement, whereas collecting wood 95% of the time instead of 90% is not. This allows distinguishing how broadly agents have explored their environment even if they achieve similar rewards. The geometric mean also establishes meaningful metric for unsupervised agents, which may get bored of tasks after performing them a few times and then move on new tasks. A caveat of the geometric mean is that agents with rewards are evaluated by something they only indirectly optimize for, which can change their ranking order. Increasing reward and score is generally correlated, but capacity- -imited agents may choose optimize reward by mastering easy tasks and ignoring hard tasks, which only slowly increases the geometric mean.
3.4 RESEARCH CHALLENGES
Crafter aims to evaluate a diverse range of agent abilities within a single enviromment. Thus, if a method performs well on Crafter there should be high chance that it also handles the challenges of other environments. The challenges also make Crafter suitable for evaluating progress on open research questions, such as strong generalization, wide and deep exploration, discovering reusable skills, and long-term memory and reasoning. Crafter is designed to pose the following challenges:
Exploration Independent achievements evaluate wide exploration, without offering a linear path for the agent to follow. Dependent achievements evaluate deep exploration of the technology tree. Collecting a diamond requires an iron pickaxe, which in turn requires a furnace, table, coal, iron, and wood. The furnace requires collecting stone, which requires building a wood pickaxe at a table.
Generalization
Every episode situated a unique world that procedurally generated. More- over, many aspects of the game reoccur in different contexts, such as creatures and resources that can be found in different landscapes and times of day. This forces successful agents to recognize similar situations in different circumstances and be robust to changes in irrelevant details.
'hile allowing large number of environment steps would help agents achieve higher scores more casily, it would result in a comparison of compute resources rather than algorithm quality.
"The Crafter score of an agent is computed as S = exp( x ΣN Iu(1 + s:)) 1, where E [0; 100] is the
agent's 's success rate of achievement i and N 22 is the number of achievements.
5

---

Published as a conference paper at ICLR 2022
12
100
Crafter Score (%6
10
Crafter Score (%)
80
60
6-
40
4-
20
2-
Random
RND (Unsup) Plan2Explore
Rainbow
PPo
DreamerV2
Random
RND (Unsup) PlaneExplore (Unsup)
Rainbow
PPO
DreamerV2
Human Experts
(Unsup)
(a) Crafter scores of various agents
(b) Same scores including human experts
Figure 5: Crafter Benchmark Scores for various agents with and without rewards. Current top methods achieve scores of up to 10% that are far from the 50% of human experts, posing substantial challenge for future research. Crafter scores are computed the geometric mean across achievements of their success rates within the budget of 1M environment steps. Numbers in Table 1.
Reusable skills Advancing in the game requires the agent to repeat several behaviors over long horizons, such as finding food, defending against monsters, and collecting common materials that are needed many times. The behavior of successful agent naturally decomposes into sub-tasks, making Crafter suitable for studying hierarchical reinforcement learning.
Credit assignment Only sparse rewards are given for unlocking an achievement for the first time during each episode. Moreover, several achievements require long-term reasoning, such as collecting the necessary resources for crafting particular tool or planting saplings that can be harvested many hundred time steps later. This makes Crafter a challenge for temporal credit assignment.
Memory The agent inputs only show the player's immediate surroundings, making Crafter partially observed. To survive for a long time, agents need remember where to find lakes to drink and open grasslands to hunt. Moreover, to effectively find rare resources, such as iron and diamonds, the agent needs to remember what parts of the map has already searched.
Representation The agent observes its environment via high-dimensional images, from which has to extract entities that are meaningful for decision making. Similar to applications in the real world, the reward signal is sparse and the amount of environment interaction limited. As a result, successful agents will likely rely on explicit representation learning techniques.
Survival In previous environments, the player can often survive by doing nothing. This allows for degenerate solutions intrinsic objectives, unlike the real world where animals forced adapt to survive and maintain homeostasis and allostasis. In Crafter, the player struggles to survive through the constant pressure of maintaining enough water, food, rest, and defending against zombies.
Table Crafter benchmark scores. The Crafter score computed as the geometric mean of suc- cess rates for all 22 achievements available in the environment. The score prefers general agents that unlock wide range of achievements over those that unlock small number of achieve- ments very frequently. For example, agent that explores many different achievements over the course of training achieves higher score than one that only performs same simple tasks over The score thus establishes meaningful metric both for agents with and without reward.
Method
Score (%)
Return
Human Experts
50.5
14.3
D2eamerV2
1..0+1.2
9.0
PPO
0.3
4.2+1.2
Rainbow
4.2
5.0+1.3
Plan2Explore (Unsup)
2.1
2.1+155
RND (Unsup)
2.0+0.1
0.7+1.3
Random
1.6+0.0
2.1

---

Published as a conference paper at ICLR 2022
DreamerV2
PPO
Rainbow
Success Rate (%)
10
1
0.1
0.01
Collect Coal
Collect Diamond
Collect Drink
Collect Iron
Collect Sapling Collect Stone Collect Wood Defeat Skeleton
Defeat Z Zombie
Eat Cow
Make Iron Pickaxe Make Iron Sword
Eat Plant
Make Stone Pickaxe
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
Place Plant
Place Stone
Place Table
Wake Up
Figure 6: Agent ability spectrum showing the success rates of agents with rewards. These are unlocking percentages for all 22 achievements, computed over all training episodes. Rainbow manages to drink water and forage for food. PPO additionally rarely collects coal and builds stone tools. DreamerV2 achieves these more frequently and additionally sometimes grows and eats fruits. Numbers in Appendix A.
4 EXPERIMENTS
To established baselines for future work, we train various reinforcement learning methods on Crafter either with and without rewards. The two benchmarks follow the evaluation protocol in Section 3.3, which grants each agent a budget of 1M environment frames and computes the success rates of the individual achievements across all training episodes, well as aggregate score for the agent. Furthermore, we analyze the emergent agent behaviors qualitatively and record a dataset of human expert players estimate the difficulty of the environment. The environment, code for the baseline agents and figures in this paper, and the human dataset are available on the project website." 3
4.1 BENCHMARK WITH REWARDS
We provide baselines scores for three reinforcement learning algorithms on Crafter with rewards. DreamerV2 (Hafner et al., 2020) learns world model and optimizes a policy through planning in latent space. We used its default hyper parameters for Atari and increased the model size. PPO (Schulman et al., 2017) is popular method that learns to map input images to actions through policy gradients. We use a convolutional neural network policy with hyper parameters that were tuned for Atari (Hill et al., 2018). Rainbow (Hessel al., 2018) is based on Q-Learning and combines several advances, including for exploration. The defaults for Atari did not work well, so we tuned the hyper parameters for Crafter and found compromise between Atari defaults and the data-efficient version of the method (van Hasselt et al., 2019) to be ideal. All agents trained for 1M environment steps in under 24 hours on single GPU and we repeated the training for 10 random seeds per method. The training reward curves are included in Appendix D.
The scores are listed in Table and visualized in Figure 5. DreamerV2 achieves score of 10.0%, followed by PPO with 4.6% and Rainbow of 4.3%. Despite these being top reinforcement learning methods, they lack behind the score of expert human players of 50.5%, which we describe in further detail in Section 4.3. We conclude that Crafter a challenging benchmark, where current methods make learning progress but future research is needed to achieve high performance. For comparison, we report the episode returns in Table 1, computed over the episodes within the last 10 environment steps of training. We find a trend similar to the scores but notice that the methods are harder to tell apart, because differences on hard tasks that are rarely achieved affect the return less. Moreover, the scores more meaningful for unsupervised agents, which should explore many achievements over time, but not necessarily remain interested in them until the end of training. The success rates for individual achievements visualized in Figure 7, which offer insights into the breadth and depth of agent abilities. Rainbow displays high success rates on easier achievements. PPO learned
Phttps://danijar.com

---

Published as a conference paper at ICLR 2022
Plan2Explore
RND
Random
100
Success Rate (%)
10
1
0.1
0.01
Collect Coal
Collect Diamond Collect Drink
Collect Iron
Collect Sapling Collect Stone Collect Wood Defeat Skeleton
Defeat Z Zombie
Eat Cow
Make Iron Pickaxe Make Iron Sword
Eat Plant
Make Stone Pickaxe Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
Place Plant
Place Stone
Place Table
Wake Up
Figure 7: Agent ability spectrum showing the success rates for Crafter without rewards. Random actions unlock the 6 easiest achievements sometimes, such drinking water and collecting wood. Plan2Explore forages for food and defeats monsters more frequently, to ensure longer survival. RND additionally collects stones and rarely even collects coal and builds furnaces. Numbers in Appendix B.
to additionally make stone tools and furnaces. Dreamer V2 achieved these more frequently and discovered growing and harvesting plants. None of the agents learned to collect and use iron for tools or to collect diamonds, or to achieve high success rates on many of the achievements.
4.2 UNSUPERVISED BENCHMARK
We provide baselines scores for two unsupervised reinforcement learning agents on Crafter without rewards. We also include a baseline that simply chooses random actions. RND (Burda et al., 2018b) a popular exploration method that seeks out novel inputs, estimated as the prediction error of network that aims to predict fixed random embeddings of the input images. We use its default parameters for Atari. Plan2Explore (Sekar et al., 2020) learns a world model to plan for the expected information gain of imagined trajectories, allowing directly seek out imagined states that have not been experienced before. We implement Plan2Explore on top of Dreamer and keep the same hyper parameters. We use a non-episodic value function as RND does. which helps exploration in episodic environments (Burda et al., 8b). All agents trained for IM environment steps in under 24 hours on a single GPU and we repeated the training for 10 random seeds per method.
The scores are listed in Table and in Figure 5. Plan2Explore achieves score of 2.1%, followed by RND at 2.0%, both ahead of the random agent at 1.6%. Despite these being top unsupervised reinforcement learning methods, they lack far behind optimal performance or even the performance of agents that learn with rewards, posing substantial challenge for future research. The results are encouraging. showing that unsupervised objectives by themselves can lead meaningful behaviors (Burda et al., 20) in Crafter. Inspecting the success rates for individual achievements in Figure 6 confirms that Plan2Explore and RND make progress in exploring the different behaviors compared to the random agent, including occasionally collecting coal, placing furnaces, and making stone swords, which are several steps deep into the technology tree.
4.3 EMERGENT BEHAVIORS
To better understand the potential of the environment, we train DreamerV2 for 50M steps and investigate the behaviors qualitatively. In this amount of time, the agent learns to build stone tools and even iron tools on individual occurrences. Interestingly, we observe a range of sophisticated emergent behaviors, such building tunnel systems, building bridges to cross lakes, and outsmarting skeletons by dodging arrows, blocking arrows with stones, and digging through walls to surprise skeletons from the side. Furthermore, DreamerV2 learns to seek shelter protect itself from the zombies at night by hiding in caves and even digging its own caves and closing the entrances with stones. Finally, find that the agent sometimes manages to build plantations of many saplings, defends them against monsters, and eats the growing fruits in order to ensure reliable and steady food supply. video of the emergent behaviors is available the project website.
8

---

Published as a conference paper at ICLR 2022
4.4 HUMAN EXPERTS DATASET
Crafter includes graphical user interface that allows humans to play the game via the keyboard and record the trajectories of the game. The human interface can be installed via the command shown in Figure 2. Through the human interface, total of 100 episodes. The experts were given the instructions of the game and allowed several hours of practice. Out of the 100 episodes, 5 episodes unlock all 22 achievements. The human experts achieved a score of 50.5%, unlocking all achievements as shown in Table C. The achievements most difficult to humans were to collect diamonds and grow and harvest plans, with success rates of 12% and 8%, respectively. While the human dataset is separate from the Crafter benchmark, it provides an estimate human performance and can be used for research on learning from demonstrations and imitation learning. The human dataset is available on the project website.
recorded the games of human experts for a combined
5 DISCUSSION
Future work We selected the difficulty of Crafter to be challenging yet not hopeless for current methods. As research progresses towards solving the challenges that are currently present, it may be- come necessary to extend Crafter by new enemies, resources, items, and achievements. Being written purely in Python, Crafter can easily be extended in this way. Moreover, grouping the 22 achievements into categories, such as memory, generalization, and exploration, would allow us to summarize agent abilities more abstractly (Osband al., 2019). We did not attempt such categorization because it is subjective and will become clearer as more researchers use the environment.
Summary We introduced Crafter, a benchmark with visual inputs that evaluates a variety of general agent abilities in single environment. We described the game mechanies, evaluation protocol, and open challenges posed by the benchmark, and performed experiments with several agents with and without rewards to provide baseline scores. Agents are evaluated based on how frequently they manage to unlock achievements that correspond to semantically meaningful milestones of behavior. We conclude that Crafter is well suited and of appropriate difficulty guide future research on intelligent agents, both for learning from extrinsic rewards and purely from intrinsic objectives.
Acknowledgements We would like to thank Oleh Rybkin, Ben Eysenbach, Sherjil Ozair, Julius Kunze, Feryal Behbahani, Timothy Lillicrap, Jimmy Ba, Nicolas Heess, Kory Mathewson, Moham- mad Norouzi, Hamza Merzic, and Sergey Levine for discussions and feedback.

---

Published as a conference paper at ICLR 2022
REFERENCES
Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pp. 507-517. PMLR, 2020.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Mareus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al. Deepmind lab. arXiv preprini arXiv: 16/2.0380, 2016.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling, The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligncce Research, 47:253-279, 2013.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv: 12894, 2018b.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Mare G
Bellemare. Dopamine: research framework for deep reinforcement learning. arXiv preprint arXiv:1812.06110, 2018.
Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In International conference on machine learning, pp- PMLR, 2020.
William H Guss, Cayden Codel, Katja Hofmann, Brandon Houghton, Noboru Kuno, Stephanie Milani, Sharada Mohanty, Diego Perez Liebana, Ruslan Salakhutdinov, Nicholay Topin, et al. The minerl competition sample efficient reinforcement learning using human priors. arXiv e-prints, pp. arXiv-1904, 2019.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02/99 193, 2020.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. https //g//////i/ withhb...... com/ hill-a/ aas-able-eee-eeeeee base lines, 2018
Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. The malmo platform for artificial intelligence experimentation. In IJCAI, pp. 4246-4247. Citeseer, 2016.
Arthur Juliani, Vincent-Pierne Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al. Unity: A general platform for intelligent agents. arXiv preprint arXiv:1809.0000 026777 2018.
Michat Kempka, Marek Wydmuch, Grzegorz Rune, Jakub Toczek, and Wojeiech Jaskowski. Vizdoom: A doom-based ai research platform for visual reinforcement learning. In 2016 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1-8. IEEE, 2016.
Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. arXiv preprint arXiv:2006.11 13760, 2020.
10

---

Published as a conference paper at ICLR 2022
Marlos C Machado, Marc G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling, Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-562, 2018.
Stephanie Milani, Nicholay Topin, Brandon Houghton, William H Guss, Sharada P Mohanty, Keisuke Nakata, Oriol Vinyals, and Noboru Sean Kuno. Retrospective analysis of the 2019 minerl competition on sample efficient reinforcement learning. In NeurlPS 2019 Competition and Demonstration Track, pp. 203-214. PMLR, 2020.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Human-level control through deep reinforcement learning, Nature, 518(7540):529, 2015.
Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al. Behaviour suite for reinforcement learning. arXiv preprint 1rXiv:1908.03:: 03566, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Ramanan Sekar. Olch Rybkin, Kostas Daniilidis, Picter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. arXiv preprint arXiv:2005.05960 2020.
Kurt Spencer. Noise!, 2014.
URL. https://uniblocks tumb Lumblr.n wwwpeetttobtbooooott.....ee.eee. com/post/
97868843242/noise.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 1001.088000000 2018
Hado P van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 12:14322-1433 2019.

---

Published as a conference paper at ICLR 2022
A
SUCCESS RATES WITH REWARDS
Achievement
Rainbow
PPO
DreamerV2
Collect Coal
0.0%
0.4%
14.7%
Collect Diamond
0.0%
0.0%
0.0%
Collect Drink
24.0%
30.3%
80.0 %
Collect Iron
0.0%
0.0%
0.0%
Collect Sapling
97.4%
66.7%
86.6%
Collect Stone
0.2%
3.0%
42.7% %
Collect Wood
74.9%
83.0%
92.7%
Defeat Skeleton
0.7%
0.2%
2.6%
Defeat Zombie
39.6%
2.0%
53.1 %
Eat Cow
26.1% 1
12.0%
17.1%
Eat Plant
0.0%
0.0%
0.1%
Make Iron Pickaxe
0.0%
0.0%
0.0%
Make Iron Sword
0.0%
0.0%
0.0%
Make Stone Pickaxe
0.0%
0.0%
0.2%
Make Stone Sword
0.0%
0.0%
0.3%
Make Wood Pickaxe
4.8%
21.1%
59.6% %
Make Wood Sword
9.8%
20.1%
40.2%
Place Furnace
0.0%
0.1%
1.8%
Place Plant
94.2%
65.0%
84.4%
Place Stone
0.0%
1.7%
29.0%
Place Table
52.3%
66.1%
85.7%
Wake Up
93.3%
92.5%
92.8 %
Score
4.3%
4.6%
10.0% %
Table A.1: Success rates Crafter with rewards. Success rates are computed the fraction of episodes during which the achievement has been unlocked at least once. is computed across all training episodes within the budget of 1M environment steps. The score is the geometric mean of success rates over all achievements, as described in Section 3.3. Note that the score is computed for each seed separately before averaging over seeds and not the other way around. Numbers within 95% of the best number in each row are highlighted in bold.
12

---

Published as a conference paper at ICLR 2022
B SUCCESS RATES WITHOUT REWARDS
Achievement
Random
RND
Plan2Explore
Collect Coal
0.0%
0.1
0.1%
Collect Diamond
0.0%
0.0%
0.0%
Collect Drink
9.3%
52.1% %
48.7%
Collect Iron
0.0%
0.0%
0.0%
Collect Sapling
50.2%
34.1%
25.5%
Collect Stone
0.0%
0.6%
0.5%
Collect Wood
24.4%
49.6%
46.8%
Defeat Skeleton
0.0%
0.3%
0.2%
Defeat Zombie
0.1%
0.3%
0.2%
Eat Cow
0.4%
0.9%
0.7%
Eat Plant
0.0%
0.0%
0.0%
Make Iron Pickaxe
0.0%
0.0%
0.0%
Make Iron Sword
0.0%
0.0%
0.0%
Make Stone Pickaxe
0.0%
0.0%
0.0%
Make Stone Sword
0.0%
0.0%
0.0%
Make Wood Pickaxe
0.3%
2.5%
3.3%
Make Wood Sword
0.3%
2.6%
3.3%
Place Furnace
0.0%
0.1%
0.0%
Place Plant
44.6%
21.4%
14.0%
Place Stone
0.0%
0.4%
0.3%
Place Table
4.4%
16.7%
16.3%
Wake Up
93.6%
7.8%
47.8%
Score
1.6%
2.0%
2.1 %
Table B.1: Success rates on Crafter without rewards. Success rates are computed as the fraction of episodes during which the achievement has been unlocked at least once. It is computed across all training episodes within the budget of M environment steps. The score is the geometric mean of success rates over all achievements, described in Section 3.3. Note that the score is computed for each seed separately before averaging over seeds and not the other way around. Numbers within 95% of the best number in each are highlighted in bold.
13

---

Published as a conference paper at ICLR 2022
C
SUCCESS RATES OF HUMAN EXPERTS
Achievement
Human Experts
Collect Coal
86.0%
Collect Diamond
12.0%
Collect Drink
92.0%
Collect Iron
53.0%
Collect Sapling
67.0%
Collect Stone
100.0%
Collect Wood
100.0%
Defeat Skeleton
31.0%
Defeat Zombie
84.0%
Eat Cow
89.0%
Eat Plant
8.0%
Make Iron Pickaxe
26.0%
Make Iron Sword
22.0%
Make Stone Pickaxe
78.0%
Make Stone Sword
78.0%
Make Wood Pickaxe
100.0%
Make Wood Sword
45.0%
Place Furnace
32.0%
Place Plant
24.0%
Place Stone
90.0%
Place Table
100.0%
Wake Up
73.0%
Score
50.5%
Table C.1: Success rates of human experts Crafter. The success rates of human experts computed as the fraction of all 100 recorded games during which the achievement has been unlocked least once. To compute the score analogously the artificial agents, we randomly split the 100 games into 5 groups that are treated as the different seeds. We then follow the same procedure as for the artificial agents.
D EPISODE REWARD
Crafter Reward
20
Optimal
DreamerV2
15
Rainbow
10
PPO
5
Random
0
0.0
0.2
0.4
0.6
0.8
1.0
1e6
Figure D.I: Total episode reward with shaded standard deviation. The optimal achievable episode reward is 22. While visualizing rewards can be informative for debugging, final performance Crafter should be reported by computing the score instead. The score takes the different difficulties of the achievements into account and is defined as the geometric mean of the success rates for all achievements, as described in Section 3.3.
14

---

Published as a conference paper at ICLR 2022
E
TEXTURES
Player
Plant
Cow
Zombie
Skeleton
Arrow
Water
Sand
Grass
Tree
Path
Stone
Coal
Iron
Diamond
Lava
Table
Furnace
Figure E.1: Crafter features worlds with several materials, resources, objects, and creatures. The player can interact with these to collect resources, maintain its food and water supplies, craft pickaxes and swords, and defend itself. The textures were specifically created for Crafter.
F ACTION SPACE
Integer
Name
Requirement
Noop
Always applicable. Flat ground left to the agent.
Move Left
2
Move Right
Flat ground right to the agent.
3
Move Up
Flat ground above the agent.
4
Move Down
Flat ground below the agent.
5
Do
Facing creature material: have necessary tool. Energy level is below maximum.
6
Sleep
Place Stone
Stone in inventory. Wood in inventory. Stone in inventory.
Place Table
Place Furnace
Place Plant
Sapling inventory. in inventory.
Make Wood Pickaxe
Nearby table;
Make Stone Pickaxe
Nearby table; wood, stone inventory. Nearby table, furnace; wood, coal, iron an inventory. Nearby table; wood in inventory. Nearby table; wood, stone in inventory. Nearby table, furnace; wood, coal, iron in inventory.
13
Make Iron Pickaxe
14
Make Wood Sword
15
Make Stone Sword
16
Make Iron Sword
Table F.1: The action space is flat categorical space, making Crafter easy to use. The 17 actions enable the agent move, collect materials, place objects, craft objects, and interact with what is in front of the player. Actions whose requirements are not satisfied have no effect.
15

---

Published as a conference paper at ICLR 2022
G
ACHIEVEMENT CURVES OF RAINBOW
Reward
Length
Collect Coal
Collect Diamond
2.0
1.00
10
1.5
0.75
1.0
0.50
200
0.5
0.25
0.0
0.0
0.5
0.0
0.5
0.0
0.5
1.0
1e6
1e6
Collect Drink
Collect Iron
Collect Sapling
Collect Stone
1.00
30
200
0.75
20
0.50
100
0.25
1
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Collect Wood
Defeat Skeleton
Defeat Zombie
Eat Cow
2.0
6
15
1.5
4
4
10
1.0
2
2
0.5
0.0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Eat Plant
Make Iron Pickaxe
Make Iron Sword
Make Stone Pickaxe
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
1.00
6
1.00
0.75
4
4
0.75
0.50
0.50
2
2
0.25
0.25
0.00
0
0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Place Plant
Place Stone
Place Table
Wake Up
3
10.0
8
7.5
4
2.5
1
2
2
0.0
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
0.5
1.0
1e6
1e6
1e6
1e6
Figure G.1: Achievement counts of Rainbow with shaded min and max.
16

---

Published as a conference paper at ICLR 2022
H ACHIEVEMENT CURVES OF PPO
Reward
Length
Collect Coal
Collect Diamond
4
1.00
10
3
0.75
400
2
0.50
200
1
0.25
0.0
0.5
0.0
0.5
0.0
1.0
0.5
1.0
1e6
1e6
Collect Drink
Collect Iron
Collect Sapling
Collect Stone
1.00
100
0.75
20
75
6
50
0.50
4
10
25
0.25
2
0.00
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Collect Wood
Defeat Skeleton
Defeat Zombie
Eat Cow
2.0
2.0
3
15
1.5
1.5
2
10
1.0
1.0
1
0.5
0.5
0.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Eat Plant
Make Iron Pickaxe
Make Iron Sword
Make Stone Pickaxe
1.00
1.00
1.00
2.0
0.75
0.75
0.75
1.5
0.50
0.50
0.50
0.25
0.25
0.25
0.5
0.00
0.00
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
1.00
8
2.0
0.75
6
1.5
0.50
4
1.0
0.25
2
2
0.5
0.00
0
0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Place Plant
Place Stone
Place Table
Wake Up
8
10.0
10
7.5
4
2
5
2
2.5
0
0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
0.5
1.0
1e6
1e6
1e6
1e6
Figure H.1: Achievement counts of PPO with shaded min and max.
17

---

Published as a conference paper at ICLR 2022
I
ACHIEVEMENT CURVES OF DREAMERV2
Reward
Length
Collect Coal
Collect Diamond
2000
1.00
10
1500
0.75
1000
4
0.50
500
2
0.25
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.0
0.5
1.0
1e6
1e6
Collect Drink
Collect Iron
Collect Sapling
Collect Stone
2.0
200
60
1.0
40
0.5
20
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
Collect Wood
Defeat Skeleton
Defeat Zombie
Eat Cow
40
3
6
30
2
20
4
10
1
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Eat Plant
Make Iron Pickaxe
Make Iron Sword
Make Stone Pickaxe
10.0
1.00
1.00
7.5
0.75
0.75
0.50
0.50
2
2.5
0.25
0.25
0.0
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
10
5
1
5
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
le6
le6
1e6
1e6
Place Plant
Place Stone
Place Table
Wake Up
20
60
10
10
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Figure L.I: Achievement counts Dreamer V2 with shaded min and max.
18

---

Published as a conference paper at ICLR 2022
J ACHIEVEMENT CURVES OF RANDOM AGENT
Reward
Length
Collect Coal
Collect Diamond
1.00
1.00
0.75
0.75
300
0.50
0.50
2
200
0
100
0.25
0.25
0.0
0.5
0.0
0.5
0.0
1.0
0.5
1.0
1e6
1e6
Collect Drink
Collect Iron
Collect Sapling
Collect Stone
1.00
6
10.0
7.5
0.75
4
5.0
0.50
2.5
0.25
1
0.00
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Collect Wood
Defeat Skeleton
Defeat Zombie
Eat Cow
1.00
1.00
2.0
0.75
0.75
1.5
0.50
0.50
1.0
0.25
0.25
0.5
0.00
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Eat Plant
Make Iron Pickaxe
Make Iron Sword
Make Stone Pickaxe
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
1.00
3
1.00
0.75
2
2
0.75
0.50
0.50
1
1
0.25
0.25
0.00
0
0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Place Plant
Place Stone
Place Table
Wake Up
2.0
3
1.5
2
1.0
0.5
0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
0.5
1.0
1e6
1e6
1e6
1e6
Figure J.1: Achievement counts of random actions with shaded min and max.
19

---

Published as a conference paper at ICLR 2022
K ACHIEVEMENT CURVES OF UNSUPERVISED RND
Reward
Length
Collect Coal
Collect Diamond
3
1.00
7.5
5.0
400
2
0.75
0.50
200
1
0.25
0.0
0.0
0.5
0.0
0.5
0.0
1.0
0.5
1.0
1e6
1e6
Collect Drink
Collect Iron
Collect Sapling
Collect Stone
1.00
8
100
0.75
15
0.50
4
50
0.25
2
5
0.00
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Collect Wood
Defeat Skeleton
Defeat Zombie
Eat Cow
3
1.00
2.0
10
0.75
1.5
2
0.50
1.0
1
0.25
0.5
0
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Eat Plant
Make Iron Pickaxe
Make Iron Sword
Make Stone Pickaxe
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
1.00
0.75
4
4
0.50
2
2
1
0.25
0.00
0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Place Plant
Place Stone
Place Table
Wake Up
10.0
7.5
3
5.0
2
0.0
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
0.5
1.0
1e6
1e6
1e6
1e6
Figure K.1: Achievement counts of unsupervised RND with shaded min and max.
20

---

Published as a conference paper at ICLR 2022
L ACHIEVEMENT CURVES OF UNSUPERVISED PLANZEXPLORE
Reward
Length
Collect Coal
Collect Diamond
2.0
1.00
10
600
1.5
0.75
400
1.0
0.50
200
0.5
0.25
0.0
0.0
0.5
0.0
0.5
0.0
1.0
0.5
1.0
1e6
1e6
Collect Drink
Collect Iron
Collect Sapling
Collect Stone
1.00
8
60
0.75
40
0.50
4
20
0.25
2
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Collect Wood
Defeat Skeleton
Defeat Zombie
Eat Cow
2.0
1.00
2.0
10
1.5
0.75
1.5
1.0
0.50
1.0
0.5
0.25
0.5
0.0
0.00
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Eat Plant
Make Iron Pickaxe
Make Iron Sword
Make Stone Pickaxe
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
1.00
1.00
0.75
4
4
0.75
0.50
0.50
2
2
0.25
0.25
0.00
0
0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
1e6
1e6
1e6
1e6
Place Plant
Place Stone
Place Table
Wake Up
8
3
2
2
2
0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
1.0
0.5
1.0
1e6
1e6
1e6
1e6
Figure L.1: Achievement counts of unsupervised Plan2Explore with shaded min and max.
21


Reviews:
[{"review_id": "hWPx-oUnaBL", "reviewer": "Reviewer_cMdw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n- New RL environments such as this are always welcome. \n- The paper is well written and well organised.\n- The motivation for the work is clear.\n- The authors have provided code.\n- The paper presents a number of benchmarks against with known RL algorithms.\n- The paper also includes results for human players and against an agent exploring randomly. This is good to see. \n- The procedural generated environment every episode to support learning of general policies is very nice. \n- The gradual unlocking of agent capabilities/achievements as shown in Figure 4 is very good and something different from many RL environments. \n\n# Weaknesses\n- It is not clear what the actual novel contribution of the work is. Yes a new RL environment/domain is good but is this just a cut down Minecraft or is there something more compelling about this environment.\n\n# Questions for the authors\n- It is not clear to me what the goal of the game is. I'll admit not having played Minecraft so perhaps I'm missing something. Is it to unlock as many achievements possible? Or is it to maximise the score? What does a high score tell you about what you have achieved?\n- With regards to the achievements; do these directly translate to the action space? In other words are the achievements the actions available to the agent at any given time? So this means we have a dynamic action space right? As the agent unlocks more achievements and hence the action space gets larger, presumably this will affect the learning curves/times as the exploration space becomes larger. It would be nice to see how the learning changes as the action space increases. ", "summary_of_the_paper": "This paper presents a new RL environment called Crafter that is somewhat inspired by the RL environments based around the Minecraft game. The paper argues that Minecraft is difficult to solve and that Crafter is a simpler 2D Minecraft like game that presents its own challenges for RL research. \n\nSome of the key features are the ability to procedurally generate a new world for each episode (in order to learn general policies) and the ability for the agent to unlock new features/capabilities. \n\nThe authors have provided some benchmarks using known RL algorithms and has also provided results for human players. ", "summary_of_the_review": "The paper is well written and presents a new Minecraft inspired environment for RL research which has some novel features. However, the paper needs to do a better job explaining the research contribution.\nSimilarly any discussion of limitations or future work is lacking.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Hdjx1SpQh8M", "reviewer": "Reviewer_Mm58", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe environment proposed is quite similar to prior work (Mine RL): the game has very similar high level dynamics, with the main differences being that 1) Crafter is 2D rather than 3D, and that 2) Crafter is slightly simpler in terms of amounts of possible tasks (e.g. in Mine RL, the agent can craft many more things than in Crafter).\n\nAdditionally, it is similar to Nethack (which was recently proposed as an AI benchmark) in the 2D nature of the game and it's hierarchical long-horizon dynamics. However – as the authors mention – Nethack is much more challenging given the vast number of items that the agent needs to learn to recognize and use effectively. With Crafter, the authors aim to provide a benchmark that isn't as memorization heavy (even though they still claim that memorization is one of the main research challenges of Crafter in Sec. 3.4).\n\n### Quality / Clarity\n\nOverall, the paper was very well written and clear, and was a pleasure to read!\n\nIn 4.4 there is a typo: \"harvest plans\" -> \"harvest plants\"\n\n\n### Limitations\n\nIn comparing to previous work, the authors state that \"Minecraft is too complex to be solved by current methods (Milani et al., 2020), it is unclear by what metric agents should be evaluated by\". It might be worth elaborating on the first part of the statement, as even in Crafter no agent is able to \"solve\" it (and in fact are very far behind humans, as mentioned in the manuscript). Regarding the second part of the statement, the lack of metrics is part of the appeal of the environment (see https://minerl.io/basalt/), so might be worth rephrasing.\n\nOne of the main selling points of Crafter is the relative sample-efficiency of training relative to other benchmarks. I thought that there was not enough space dedicated to analyzing the relative amount of training required for _convergence_ in Crafter relative to Atari, Mine RL, Nethack, and so on. While all the results for Crafter are capped at 1M timesteps, one could also do the same in these other environments. While I don't expect these agents would be very good at all, that would give a better sense of what order of magnitude of the difference between these environments is. Additionally, it would be nice to see (in the appendix) what the 50M timestep Dreamer-v2 training run looks like (especially, whether it has reached convergence at 50M).\n\nAnother thing that is curious to me is the reason behind evaluating the agents on the average success rate across the whole of training. I understand that this is to take into account sample-efficiency – but that is already being forced by capping the training to 1M timesteps. This additional choice of averaging across epochs seems to greatly penalize algorithms which might be very sample-inefficient at first, but then later catch up, leading to the same final performance as others. It would be nice to at least include the final-epoch success rates as additional information. \n\nAnother thing that is strange is that the random agent seems to achieve the same level of episode length as PPO agents. It would be useful to have some additional clarity about why this is.", "summary_of_the_paper": "This paper introduces a new RL benchmark, Crafter, which is a simplified version of Minecraft. The aim of Crafter is to offer an environment which has many desirable properties: while it is a challenging environment in which complex behaviors are possible and necessary for the agents' survival, it's simultaneously simple enough to make training require much less interactions relative to prior benchmarks.", "summary_of_the_review": "Overall, Crafter – while a straightforward simplification of another existing environment (Mine RL) – could become a useful benchmark task in the field. While it is not fully novel, it positions itself uniquely relative to other benchmarks across various axes relevant for research (pixel vs state based, memory vs not-memory intensive, 2D vs 3D complexity, small vs large required compute). The low amount of compute required for training (while preserving environment and potential behavior complexity) is worth emphasizing, as it could speed up the rate of progress of further research in this space.\n\nIt would however be nice to see addressed the limitations mentioned above.\n\n\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "HDQpJ0BUhX1", "reviewer": "Reviewer_ix7A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This environment has a credible case that it could help accelerate research on exploration and generalization. From the perspective of generalization, its great that each new episode comes with an entirely new, procedurally generated map. Additionally, skeletons and antagonistic enemies introduce additional complexity to the survival task, as does the limited supply of food/water/energy. The existing baselines indicate that the task is neither too hard nor too easy for existing RL agents.\n\nWeaknesses: Let us imagine the world in which a significant portion of the RL research community started focusing on the Crafter benchmark. Would this shift in focus lead to the development of better RL agents along the dimensions of exploration/generalization/reusable-skills/credit-assignment/memory? Would these new agents be generalizable beyond Crafter to other domains and applications? \n\nI see a possible failure case of agents that are generalizable only in the senses that Crafter requires - e.g. able to handle new maps, but perhaps are non-generalizable beyond Crafter. E.g. agents which may understand the specific crafting tree and strategies for acquiring food and shelter that are specific to Crafter. Other environments like ProcGen combat this problem by having entirely separate held-out environments which discourage the agent from overfitting to specifics of the training environments. Crafter seems to lack the ability to change its crafting tree and environment dynamics.\n\nAnother weakness is the fact that this domain is crafted for the purpose of training RL agents - which means that unlike other domains that focus on solving games designed for humans like Atari and NetHack, Crafter may not be inherently interesting or challenging for humans. As a community, would we be impressed by agents that could complete the full set of achievements on this domain? It's unclear, but it's doubtful that anyone would be able to relate to the difficulty of Crafter without having first used it as a research framework. Contrast this to a domain like NetHack / StarCraft2 / DoTA / Chess+Go - in which players are already familiar and can relate to the difficult of solving.\n\nI would also be interested to understand how easy or difficult it is to hand-code a heuristic agent to solve this task. I think it would be somewhat less interesting as a task if it was possible to create a heuristic agent that could solve it using only a few hundred lines of code.\n\nIn summary, it's clear that significant thought and work has gone into creating the Crafter environment. There interesting aspects of the environment that are novel such as the need to find food/water/shelter/sleep. Episodes are long and maps are procedurally generated so there is certainly a possibility that Crafter may be applicable to continuous/lifelong learning agents. However, there are still limitations in terms of the environment being relatively fixed/static at the meta level of the crafting tree and ways of interacting with different types of objects/enemies. This makes it unclear whether agents and algorithms developed for or trained in Crafter will be general beyond this particular environment.\n\nUpdate after author response: I'm still borderline on this paper. I'd love to hear from someone who works on RL in Minecraft/MineRL as to whether this environment would be welcome and useful in the space.", "summary_of_the_paper": "This paper introduces a new environment for development of agent capabilities, called Crafter. The environment is procedurally generated and consists of a 2-D world inhabited by various resources, terrain types, and objects. The agent is rewarded for crafting items and accomplishing achievements from a set of 22 possible achievements. The objective of this environment is to encourage RL research on the topics of exploration, generalization, reusable skills, credit assignment, and memory. Benchmark results are presented for various RL algorithms under a reward and no-reward settings, as well as human experts. In general, the current RL methods fall far short of human performance, although Dreamer-V2 shows emergent behaviors.", "summary_of_the_review": "It's hard to objectively referee this type of a paper because as with many new environments, beauty is in the eye of the beholder. Overall, I'm not particularly excited by another 2D gridworld-style environment - but I will admit that Crafter has several interesting aspects that aren't featured in other environments. Similarly, I'm moderately interested by the emergent behaviors demonstrated by the Dreamer-V2 agent, but not blown away (unlike emergent behaviors in OpenAI Hide and Seek for example). So overall I'm lukewarm on Crafter - I could see it possibly appealing to a subset of the RL community and perhaps helping to drive research in particular subfields of RL, but this is just my guess. Should it be accepted as a conference paper? Perhaps, but it may be better promoted through other means like NeurIPS competitions, blog posts, and similar.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AvE6BaLPdLp", "reviewer": "Reviewer_7FaV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n  - The paper is clearly written, and the evaluation appears sound.\n  - The contribution seems useful. The computational demands of MineRL can be challenging especially for researchers with limited compute resource. Having an environment designed specifically for RL in mind is also useful in terms of permitting modifications in the future, and enabling debugging by directly accessing internal state.\n\nWeaknesses:\n  - The use case for the benchmark is a bit narrow. It's significantly less complex in terms of both visuals and game mechanics than MineRL, so I expect many researchers to continue to prefer MineRL. Minecraft also has benefits for studying human-robot interaction, as there are already many expert Minecraft players, and it is possible to put an AI directly on a Minecraft server -- something not currently supported by this benchmark.\n  - I am concerned that RL algorithms are being trained to maximize the reward but being evaluated on a correlated but distinct metric (aggregate score). It seems conceivable than RL algorithm A could achieve higher reward than algorithm B but a lower score -- how should we interpret such results? In some sense, B produces a stronger policy, but I hesitate to say it is a better RL algorithm (unless you have reason to believe the reward is going to be misspecified in a very particular way).\n\n    While you do report reward in appendix D, I'd suggest either (a) justifying why it's valid to judge RL algorithms based on something other than reward attained; (b) switch to reward as the primary metric.\n  - Section 4.3 could be strengthened by including a quantitative description of the agent behaviour (e.g. score attained) in addition to the (very interesting) qualitative description.  In particular, knowing the score would be helpful to contextualize how close DreamerV2 @ 50M timesteps is to expert performance. In general, it's useful to have a sense of if this benchmark is solveable given a 50x increase in timesteps (making it largely about improving sample efficiency of algorithms) or if there still remain some key challenges there (likely requiring a more radical overhaul of RL algorithms to tackle).\n\nUnfortunately the acknowledgements were left in the manuscript which partially de-anonymizes the submission.\n\nMinor points:\n  - Section 3.1: \"the resources it can collect\" -- \"it\" (for the agent) sounds jarring when you previously refer to agent as \"the player\" (expect he/she). Consider changing \"player\" to \"agent\", or replacing \"it\" with \"the player\" or \"the agent\". This applies to the many subsequent uses of \"it\" to refer to the player as well.\n  - Section 3.1: \"diamonds, the player\" -> \"diamonds, that the player\". Same comment as before about \"it\" and \"player\".\n  - Section 3.1: \"Skeletons live in caves and try to keep the player at a distance to shoot arrows at the player.\" I'm confused by this, are you saying the skeletons shoot players if they move within a certain range? (Perhaps should be \"by shooting\" rather than \"to shoot\".) Or are the skeletons moving, trying to keep the player at a distance where the player can be shot?\n  - Section 3.1: \"The player can interact with creatures to decrease their health points.\" Why would they want to decrease their health points? If they don't, but it's sometimes difficult to avoid, consider using a word other than \"can\" (which implies possibility not necessity).\n  - Section 4.1: \"lack behind the score\" -> \"lag behind ...\"? Similar comment in 4.3: \"lack far behind optimal performance\".\n  - Section 4.1: \"visualized in Figure 7\" -- I think you mean Figure 6? I think 4.2 also should cite Figure 7 instead of Figure 6 (i.e. swap them.)", "summary_of_the_paper": "The work introduces a gridworld-style benchmark inspired by Minecraft. It is similar in style to MineRL, but is significantly more computationally efficient, and easier to use (can be run directly without requiring e.g. an external Java application). Despite this simplification, it proves to be a challenging benchmark, with state-of-the-art RL algorithms achieving no more than 10% average success rate across tasks (e.g. make a pickaxe, collect food, find a diamond).", "summary_of_the_review": "The benchmark developed will be valuable to those seeking to test agents in Minecraft-like environments but without sufficient computational resources to test at scale in MineRL. However, this use case is somewhat narrow, and additionally the choice of evaluation metric (aggregate score) is not well justified (especially in situations where it may diverge from the reward). I would assess the current submission as being borderline, but I lean slightly towards reject. I would consider increasing my score if the evaluation criteria were more clearly justified.\n\nUpdate: I have increased my score to marginal accept after the author's explanation of the choice of GM, and their inclusion of returns in the main paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new RL benchmark that is a simplified 2D version of Minecraft -- it is designed to support complex behaviors but reduce the training complexity. It is very well written and clear, positioned well with respect to other benchmarks, and is likely to improve the speed of development/testing of some RL algorithms. It is likely to appeal to a subset of the community and drive research in some cases, while others may prefer to stick with full 3D Minecraft. As such, there are some mixed reviews on the paper, with open questions as to whether it would be welcomed by people who work on Minecraft-style domains, whether behaviors learned in the simplified 2D environment would generalize to other settings/domains, and the potential for agents to game the environment. The authors are encouraged to take these aspects and perspectives into consideration when revising the paper.
Decision: Accept (Poster)