Benchmarking the Spectrum of Agent Capabilities

1 INTRODUCTION Crafter is an open world survival game for reinforcement learning research . Shown in Figure 1 , Crafter features randomly generated 2D worlds with forests , lakes , mountains , and caves . The player needs to forage for food and water , find shelter to sleep , defend against monsters , collect materials , and build tools . The game mechanics are inspired by the popular game Minecraft and were simplified and optimized for research productivity . Crafter aims to be a fruitful benchmark for reinforcement learning by focusing on the following design goals : Research challenges Crafter poses substantial challenges to current methods . Procedural generation requires strong generalization , the technology tree evaluates wide and deep exploration , image observations calls for representation learning , repeated subtasks and sparse rewards evaluate long-term reasoning and credit assignment . Meaningful evaluation Agents are evaluated by a range of achievements that can be unlocked in each episode . The achievements correspond to meaningful milestones in behavior , offering insights into ability spectrum of both reward agents and unsupervised agents . Iteration speed Crafter evaluates many agent abilities within a single environment , vastly reducing the computational requirements over benchmarks suites that require training on many separate environments from scratch , while making it more likely that the measured performance is representative of new domains . 2 RELATED WORK . Benchmarks have been a driving force behind the progress and successes of reinforcement learning as a field ( Bellemare et al. , 2013 ; Brockman et al. , 2016 ; Kempka et al. , 2016 ; Beattie et al. , 2016 ; Tassa et al. , 2018 ; Juliani et al. , 2018 ) . Benchmarks often require a large amount of computational resources and yet only test a small fraction of the abilities that a general agent should master ( Cobbe et al. , 2020 ) . This section directly compares Crafter to four particularly related benchmarks . Minecraft Crafter is inspired by the successful 3D video game Minecraft , which is available to researchers via Malmo ( Johnson et al. , 2016 ) and MineRL ( Guss et al. , 2019 ) . Minecraft features diverse open worlds with randomly generated and modifiable terrain , as well as many different resources , tools , and monsters . However , Minecraft is too complex to be solved by current methods ( Milani et al. , 2020 ) , it is unclear by what metric agents should be evaluated by , the environment is slow , and can be difficult to use because it requires Java and a window server . In comparison , Crafter captures many principles of Minecraft in a simple and fast environment , where results can be obtained in a matter of hours , and where a large number of semantically meaningful evaluation metrics are available for reinforcement learning with or without extrinsic reward . The goal of Crafter is not to replace Minecraft but progress faster towards it . Atari The Atari Learning Environment ( Bellemare et al. , 2013 ) has been the gold standard benchmark in reinforcement learning . It comprises around 54 individual games , depending on the evaluation protocol ( Mnih et al. , 2015 ; Schulman et al. , 2017 ; Badia et al. , 2020 ; Hafner et al. , 2020 ) . While the large number of games tests different abilities of agents , they require a large amount of computation . The recommended protocol of training the agent with 5 random seeds on each game for 200M steps requires over 2000 GPU days ( Castro et al. , 2018 ; Hessel et al. , 2018 ) . This substantially slows down experimentation and makes the complete benchmark infeasible for most academic labs . Moreover , Atari games are nearly deterministic , so agents can approximately memorize their action sequences and are not required to generalize to new situations ( Machado et al. , 2018 ) . ProcGen ProcGen ( Cobbe et al. , 2020 ) provides a benchmark that is similar to Atari but explicitly addresses the determinism present in Atari through the use of procedural generation and randomized textures . It consists of 16 games , where each episode features a randomly generated level layout . Similarly , Crafter relies on procedural generation to provide a different world map with different distribution of resources and monsters for every episode . However , ProcGen still requires training methods on 16 individual games for 200M environment steps , which each focus on a narrow aspect of an agent ’ s general abilities . In comparison , Crafter evaluates many different abilities of an agent by training only on a single environment for 5M steps , substantially accelerating experimentation . NetHack NetHack ( Küttler et al. , 2020 ) is a text-based game , where the player traverses a randomly generated system of dungeons with many different items and creatures . Unlike the other discussed environments , NetHack uses symbolic inputs and thus does not evaluate an agent ’ s ability to learn representations of high-dimensional inputs . The game is challenging due to the large amount of knowledge required about the many different items and their effects , even for human players . As a result , NetHack requires many environment steps for agents to acquire this domain-specific knowledge ; 1B steps were used in the original paper . In contrast , Crafter generates diverse complex worlds from simple underlying rules , focusing more on generalization than memorization of facts . 3 CRAFTER BENCHMARK . We introduce Crafter , a benchmark that evaluates a variety of agent abilities in a single environment . This section describes the game mechanics of the environment , the interface of agent inputs and actions , the evaluation protocol that is based on a range of semantically meaningful achievements , and the open challenges that Crafter poses for future research . 3.1 GAME MECHANICS . This section describes the game mechanics of Crafter , namely its randomly generated world maps , the levels of health and other internal quantities that the player has to maintain , the resources it can collect and objects and tools it can make from them , as well as the creatures and how they are influenced by the time of day . The images of all materials and objects are shown in Figure E.1 . All randomness in the environment is uniquely determined by an integer seed that is derived from the initial seed passed to the environment and the episode number . Terrain generation A unique world is generated for every episode , shown in Figure 3 . The world leverages an underlying grid of 64 × 64 cells but the agent only observes the world through pixel images . The terrain features grasslands , lakes , and mountains . Lakes can have shores , grasslands can have forests , and mountains can have caves , ores , and lava . These are determined by OpenSimplex noise ( Spencer , 2014 ) , a form of locally smooth noise . Within the areas determined by noise , objects appear with equal probability at any location , such as trees in forests and skeletons in caves . Health and survival The player has levels of health , food , water , and rest that it must prevent from reaching zero . The levels for food , water , and rest decrease over time and are restored by drinking from a lake , chasing cows or growing fruits to eat , and sleeping in places where monsters can not attack . Once one of the three levels reaches zero , the player starts losing health points . It can also lose health points when attacked by monsters . When the health points reach zero , the player dies . Health points regenerate over time when the player is not hungry , thirsty , or sleepy . Resources and crafting There are many resources , such as saplings , wood , stone , coal , iron , and diamonds , the player can collect in its inventory and use to build tools and place objects in the world . Many of the resources require tools that the place must first build from more basic resources , leading to a technology tree with several levels . Standing nearby a table enables the player to craft wood pickaxes and swords , as well as stone pickaxes and stone swords . Crafting a furnace from stone enables crafting iron pickaxes and iron swords from both iron , coal , and wood . Creatures and night Creatures are initialized in random locations and move randomly . Zombies and cows live in grasslands and are automatically spawned and despawned to ensure a given amount of creatures . At night , the agent ’ s view is restricted and noisy and a larger number of zombies is spawned . This makes it difficult to survive without securing a shelter , such as a cave . Skeletons live in caves and try to keep the player at a distance to shoot arrows at the player . The player can interact with creatures to decrease their health points . Cows move randomly and offer a food source . 3.2 ENVIRONMENT INTERFACE . This section defines the specification of the environment , explains the available actions , agent inputs , episode termination , and additional information provided by the environment . The design goal of these is to make the environment easy to use and inspect . The environment uses the Gym interface ( Brockman et al. , 2016 ) with visual agent inputs and flat categorical actions . Observations Agent receive color images of size 64 × 64 × 3 as their only inputs . The image shows a local top-down view of the map , reaching 4 cells west and east and 3 cells north and south of the player position . Below this view of the world , the image shows the current inventory state of the player , including its health points , food , water , and rest levels , collected materials , and crafted tools . The agent needs to learn to read its inventory state out of the image . Actions The action space is a flat categorical space with 17 actions , represented by integer indices . The actions allow the player to move in all 4 directions along the grid , interact with the object in front of it , go to sleep , place objects , and make tools . Each object and tool has a separate action associated with it . Tools are kept in the inventory whereas objects are automatically placed in front of the player . If the agent does not hold the required materials for making an object or tool , the action has no effect . Termination Each episode terminates when the player ’ s health points reach 0 . This can happen when the player dies out of hunger , thirst , or tiredness , when attacked by a zombie or skeleton , or when falling into lava . Health points automatically regenerate , as long as the agent is not too hungry , thirsty , or sleepy . There is no negative reward for dying , as the reward signal already includes a penalty for losing health points . Episodes also end when reaching the time limit of 10,000 steps . Additional information The environment allows access to privileged information about the world state that the agent is forbidden to observe . This includes numeric inventory counts , achievement counts , the current coordinate of the player on the grid , and a semantic grid representation of the map . These can be used for debugging purposes or for other research scenarios , such as predicting the underlying environment state to evaluate representation learning or video prediction models .

Summary: This paper introduces a new environment for development of agent capabilities, called Crafter. The environment is procedurally generated and consists of a 2-D world inhabited by various resources, terrain types, and objects. The agent is rewarded for crafting items and accomplishing achievements from a set of 22 possible achievements. The objective of this environment is to encourage RL research on the topics of exploration, generalization, reusable skills, credit assignment, and memory. Benchmark results are presented for various RL algorithms under a reward and no-reward settings, as well as human experts. In general, the current RL methods fall far short of human performance, although Dreamer-V2 shows emergent behaviors.