Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new RL benchmark, Crafter, which is a simplified version of Minecraft. The aim of Crafter is to offer an environment which has many desirable properties: while it is a challenging environment in which complex behaviors are possible and necessary for the agents' survival, it's simultaneously simple enough to make training require much less interactions relative to prior benchmarks.
main_review: ### Originality

The environment proposed is quite similar to prior work (Mine RL): the game has very similar high level dynamics, with the main differences being that 1) Crafter is 2D rather than 3D, and that 2) Crafter is slightly simpler in terms of amounts of possible tasks (e.g. in Mine RL, the agent can craft many more things than in Crafter).

Additionally, it is similar to Nethack (which was recently proposed as an AI benchmark) in the 2D nature of the game and it's hierarchical long-horizon dynamics. However – as the authors mention – Nethack is much more challenging given the vast number of items that the agent needs to learn to recognize and use effectively. With Crafter, the authors aim to provide a benchmark that isn't as memorization heavy (even though they still claim that memorization is one of the main research challenges of Crafter in Sec. 3.4).

### Quality / Clarity

Overall, the paper was very well written and clear, and was a pleasure to read!

In 4.4 there is a typo: "harvest plans" -> "harvest plants"


### Limitations

In comparing to previous work, the authors state that "Minecraft is too complex to be solved by current methods (Milani et al., 2020), it is unclear by what metric agents should be evaluated by". It might be worth elaborating on the first part of the statement, as even in Crafter no agent is able to "solve" it (and in fact are very far behind humans, as mentioned in the manuscript). Regarding the second part of the statement, the lack of metrics is part of the appeal of the environment (see https://minerl.io/basalt/), so might be worth rephrasing.

One of the main selling points of Crafter is the relative sample-efficiency of training relative to other benchmarks. I thought that there was not enough space dedicated to analyzing the relative amount of training required for _convergence_ in Crafter relative to Atari, Mine RL, Nethack, and so on. While all the results for Crafter are capped at 1M timesteps, one could also do the same in these other environments. While I don't expect these agents would be very good at all, that would give a better sense of what order of magnitude of the difference between these environments is. Additionally, it would be nice to see (in the appendix) what the 50M timestep Dreamer-v2 training run looks like (especially, whether it has reached convergence at 50M).

Another thing that is curious to me is the reason behind evaluating the agents on the average success rate across the whole of training. I understand that this is to take into account sample-efficiency – but that is already being forced by capping the training to 1M timesteps. This additional choice of averaging across epochs seems to greatly penalize algorithms which might be very sample-inefficient at first, but then later catch up, leading to the same final performance as others. It would be nice to at least include the final-epoch success rates as additional information. 

Another thing that is strange is that the random agent seems to achieve the same level of episode length as PPO agents. It would be useful to have some additional clarity about why this is.
summary_of_the_review: Overall, Crafter – while a straightforward simplification of another existing environment (Mine RL) – could become a useful benchmark task in the field. While it is not fully novel, it positions itself uniquely relative to other benchmarks across various axes relevant for research (pixel vs state based, memory vs not-memory intensive, 2D vs 3D complexity, small vs large required compute). The low amount of compute required for training (while preserving environment and potential behavior complexity) is worth emphasizing, as it could speed up the rate of progress of further research in this space.

It would however be nice to see addressed the limitations mentioned above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new RL environment called Crafter that is somewhat inspired by the RL environments based around the Minecraft game. The paper argues that Minecraft is difficult to solve and that Crafter is a simpler 2D Minecraft like game that presents its own challenges for RL research. 

Some of the key features are the ability to procedurally generate a new world for each episode (in order to learn general policies) and the ability for the agent to unlock new features/capabilities. 

The authors have provided some benchmarks using known RL algorithms and has also provided results for human players. 
main_review: # Strengths
- New RL environments such as this are always welcome. 
- The paper is well written and well organised.
- The motivation for the work is clear.
- The authors have provided code.
- The paper presents a number of benchmarks against with known RL algorithms.
- The paper also includes results for human players and against an agent exploring randomly. This is good to see. 
- The procedural generated environment every episode to support learning of general policies is very nice. 
- The gradual unlocking of agent capabilities/achievements as shown in Figure 4 is very good and something different from many RL environments. 

# Weaknesses
- It is not clear what the actual novel contribution of the work is. Yes a new RL environment/domain is good but is this just a cut down Minecraft or is there something more compelling about this environment.

# Questions for the authors
- It is not clear to me what the goal of the game is. I'll admit not having played Minecraft so perhaps I'm missing something. Is it to unlock as many achievements possible? Or is it to maximise the score? What does a high score tell you about what you have achieved?
- With regards to the achievements; do these directly translate to the action space? In other words are the achievements the actions available to the agent at any given time? So this means we have a dynamic action space right? As the agent unlocks more achievements and hence the action space gets larger, presumably this will affect the learning curves/times as the exploration space becomes larger. It would be nice to see how the learning changes as the action space increases. 
summary_of_the_review: The paper is well written and presents a new Minecraft inspired environment for RL research which has some novel features. However, the paper needs to do a better job explaining the research contribution.
Similarly any discussion of limitations or future work is lacking.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new environment for development of agent capabilities, called Crafter. The environment is procedurally generated and consists of a 2-D world inhabited by various resources, terrain types, and objects. The agent is rewarded for crafting items and accomplishing achievements from a set of 22 possible achievements. The objective of this environment is to encourage RL research on the topics of exploration, generalization, reusable skills, credit assignment, and memory. Benchmark results are presented for various RL algorithms under a reward and no-reward settings, as well as human experts. In general, the current RL methods fall far short of human performance, although Dreamer-V2 shows emergent behaviors.
main_review: Strengths: This environment has a credible case that it could help accelerate research on exploration and generalization. From the perspective of generalization, its great that each new episode comes with an entirely new, procedurally generated map. Additionally, skeletons and antagonistic enemies introduce additional complexity to the survival task, as does the limited supply of food/water/energy. The existing baselines indicate that the task is neither too hard nor too easy for existing RL agents.

Weaknesses: Let us imagine the world in which a significant portion of the RL research community started focusing on the Crafter benchmark. Would this shift in focus lead to the development of better RL agents along the dimensions of exploration/generalization/reusable-skills/credit-assignment/memory? Would these new agents be generalizable beyond Crafter to other domains and applications? 

I see a possible failure case of agents that are generalizable only in the senses that Crafter requires - e.g. able to handle new maps, but perhaps are non-generalizable beyond Crafter. E.g. agents which may understand the specific crafting tree and strategies for acquiring food and shelter that are specific to Crafter. Other environments like ProcGen combat this problem by having entirely separate held-out environments which discourage the agent from overfitting to specifics of the training environments. Crafter seems to lack the ability to change its crafting tree and environment dynamics.

Another weakness is the fact that this domain is crafted for the purpose of training RL agents - which means that unlike other domains that focus on solving games designed for humans like Atari and NetHack, Crafter may not be inherently interesting or challenging for humans. As a community, would we be impressed by agents that could complete the full set of achievements on this domain? It's unclear, but it's doubtful that anyone would be able to relate to the difficulty of Crafter without having first used it as a research framework. Contrast this to a domain like NetHack / StarCraft2 / DoTA / Chess+Go - in which players are already familiar and can relate to the difficult of solving.

I would also be interested to understand how easy or difficult it is to hand-code a heuristic agent to solve this task. I think it would be somewhat less interesting as a task if it was possible to create a heuristic agent that could solve it using only a few hundred lines of code.

In summary, it's clear that significant thought and work has gone into creating the Crafter environment. There interesting aspects of the environment that are novel such as the need to find food/water/shelter/sleep. Episodes are long and maps are procedurally generated so there is certainly a possibility that Crafter may be applicable to continuous/lifelong learning agents. However, there are still limitations in terms of the environment being relatively fixed/static at the meta level of the crafting tree and ways of interacting with different types of objects/enemies. This makes it unclear whether agents and algorithms developed for or trained in Crafter will be general beyond this particular environment.

Update after author response: I'm still borderline on this paper. I'd love to hear from someone who works on RL in Minecraft/MineRL as to whether this environment would be welcome and useful in the space.
summary_of_the_review: It's hard to objectively referee this type of a paper because as with many new environments, beauty is in the eye of the beholder. Overall, I'm not particularly excited by another 2D gridworld-style environment - but I will admit that Crafter has several interesting aspects that aren't featured in other environments. Similarly, I'm moderately interested by the emergent behaviors demonstrated by the Dreamer-V2 agent, but not blown away (unlike emergent behaviors in OpenAI Hide and Seek for example). So overall I'm lukewarm on Crafter - I could see it possibly appealing to a subset of the RL community and perhaps helping to drive research in particular subfields of RL, but this is just my guess. Should it be accepted as a conference paper? Perhaps, but it may be better promoted through other means like NeurIPS competitions, blog posts, and similar.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The work introduces a gridworld-style benchmark inspired by Minecraft. It is similar in style to MineRL, but is significantly more computationally efficient, and easier to use (can be run directly without requiring e.g. an external Java application). Despite this simplification, it proves to be a challenging benchmark, with state-of-the-art RL algorithms achieving no more than 10% average success rate across tasks (e.g. make a pickaxe, collect food, find a diamond).
main_review: Strengths:
  - The paper is clearly written, and the evaluation appears sound.
  - The contribution seems useful. The computational demands of MineRL can be challenging especially for researchers with limited compute resource. Having an environment designed specifically for RL in mind is also useful in terms of permitting modifications in the future, and enabling debugging by directly accessing internal state.

Weaknesses:
  - The use case for the benchmark is a bit narrow. It's significantly less complex in terms of both visuals and game mechanics than MineRL, so I expect many researchers to continue to prefer MineRL. Minecraft also has benefits for studying human-robot interaction, as there are already many expert Minecraft players, and it is possible to put an AI directly on a Minecraft server -- something not currently supported by this benchmark.
  - I am concerned that RL algorithms are being trained to maximize the reward but being evaluated on a correlated but distinct metric (aggregate score). It seems conceivable than RL algorithm A could achieve higher reward than algorithm B but a lower score -- how should we interpret such results? In some sense, B produces a stronger policy, but I hesitate to say it is a better RL algorithm (unless you have reason to believe the reward is going to be misspecified in a very particular way).

    While you do report reward in appendix D, I'd suggest either (a) justifying why it's valid to judge RL algorithms based on something other than reward attained; (b) switch to reward as the primary metric.
  - Section 4.3 could be strengthened by including a quantitative description of the agent behaviour (e.g. score attained) in addition to the (very interesting) qualitative description.  In particular, knowing the score would be helpful to contextualize how close DreamerV2 @ 50M timesteps is to expert performance. In general, it's useful to have a sense of if this benchmark is solveable given a 50x increase in timesteps (making it largely about improving sample efficiency of algorithms) or if there still remain some key challenges there (likely requiring a more radical overhaul of RL algorithms to tackle).

Unfortunately the acknowledgements were left in the manuscript which partially de-anonymizes the submission.

Minor points:
  - Section 3.1: "the resources it can collect" -- "it" (for the agent) sounds jarring when you previously refer to agent as "the player" (expect he/she). Consider changing "player" to "agent", or replacing "it" with "the player" or "the agent". This applies to the many subsequent uses of "it" to refer to the player as well.
  - Section 3.1: "diamonds, the player" -> "diamonds, that the player". Same comment as before about "it" and "player".
  - Section 3.1: "Skeletons live in caves and try to keep the player at a distance to shoot arrows at the player." I'm confused by this, are you saying the skeletons shoot players if they move within a certain range? (Perhaps should be "by shooting" rather than "to shoot".) Or are the skeletons moving, trying to keep the player at a distance where the player can be shot?
  - Section 3.1: "The player can interact with creatures to decrease their health points." Why would they want to decrease their health points? If they don't, but it's sometimes difficult to avoid, consider using a word other than "can" (which implies possibility not necessity).
  - Section 4.1: "lack behind the score" -> "lag behind ..."? Similar comment in 4.3: "lack far behind optimal performance".
  - Section 4.1: "visualized in Figure 7" -- I think you mean Figure 6? I think 4.2 also should cite Figure 7 instead of Figure 6 (i.e. swap them.)
summary_of_the_review: The benchmark developed will be valuable to those seeking to test agents in Minecraft-like environments but without sufficient computational resources to test at scale in MineRL. However, this use case is somewhat narrow, and additionally the choice of evaluation metric (aggregate score) is not well justified (especially in situations where it may diverge from the reward). I would assess the current submission as being borderline, but I lean slightly towards reject. I would consider increasing my score if the evaluation criteria were more clearly justified.

Update: I have increased my score to marginal accept after the author's explanation of the choice of GM, and their inclusion of returns in the main paper.
review_title: null

Meta-review: This paper introduces a new RL benchmark that is a simplified 2D version of Minecraft -- it is designed to support complex behaviors but reduce the training complexity. It is very well written and clear, positioned well with respect to other benchmarks, and is likely to improve the speed of development/testing of some RL algorithms. It is likely to appeal to a subset of the community and drive research in some cases, while others may prefer to stick with full 3D Minecraft. As such, there are some mixed reviews on the paper, with open questions as to whether it would be welcomed by people who work on Minecraft-style domains, whether behaviors learned in the simplified 2D environment would generalize to other settings/domains, and the potential for agents to game the environment. The authors are encouraged to take these aspects and perspectives into consideration when revising the paper.
Decision: Accept (Poster)