Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: In this paper, the authors propose a new benchmark for reinforcement learning based on a popular game — Minecraft. It includes thousands of tasks with unified observation and action spaces, and a large knowledge base with multimodal data collected from the Internet. Moreover, a novel learning algorithm is developed to learn reward functions automatically, successfully guiding the learning agent and reducing the burden of designing reward functions manually. In general, it is a good demonstration to show how reinforcement learning could benefit from recent success (e.g., large pre-trained models) in the natural language processes and computer vision.
strengths: Overall, it is a good paper relevant to the broader research community with significant contributions. I especially like the idea of applying a video-test contrastive model to generate reward signals automatically. Potentially, this method could greatly lessen the difficulty of manually designing reward functions for tasks with large knowledge bases.


weaknesses: I find no major weaknesses in this work.

At the same time, I found the website the authors have built is much more appealing than the paper writing. which the authors could consider to improve their paper presentation.
correctness: Yes.


clarity: Yes.
relation to prior work: The authors discuss related work on four topics: open-ended environments for decision-making agents, Minecraft for AI research, Internet-scale multimodal knowledge bases, and embodied agents with large-scale pre-training. It will be great if the authors can add some comments about a concurrent work(https://openai.com/blog/vpt/) in an updated version.
additional feedback: In Table 2, it is shown that MINECLIP consistently agrees well with the ground-truth human judgment, even for MINECLIP(avg). However, there is a large performance gap between using rewards generated with MINECLIP(avg) and using manual rewards, as shown in the first 4 rows in Table 1. Why?
In section 3.1, it is mentioned that the GPT-3-davinci model is used to generate detailed guidance for a subset of the taks. From the presented example, it seems that the generated guidance contains step-by-step information. Is the full guidance used to compute the reward signals or only a part of it is used at each time-step? How useful is the guidance overall? How does a learning agent perform without detailed guidance?
review_title: A good environment for open-ended RL research 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: 
MineDojo includes an extended API for Minecraft, multiple datasets, and a demonstration of one successful path to leverage these components. It provides multiple significant contributions towards the goal of training an agent which can solve a general set of goals in an open-world environment, including (1) a broad set of diverse tasks, both clearly defined and open ended, (2) methods for evaluating the clearly defined tasks along with a proposed, and validated, method for evaluating the open ended tasks (3) an API for extending the task and environment set (4) proof of concept in the form of the training procedure for an agent that can generalize across the set of tasks and (5) a vast environment-specific multimodal knowledge base.
strengths: The scale of the dataset provided particularly distinguishes this submission, in addition to the creative leveraging of this dataset to both facilitate training and provide a proposed evaluation method for more open ended tasks. These contributions not only enable research on the specific question of training an versatile language guided agent in complex open-ended environment, but are also relevant for investigation into a variety of topics including pre training for RL and offline learning methods, multitask or meta-learning, etc.  The dataset is broadly accessible and well documented and clear efforts have been made to address the (limited) ethical concerns inherent in collecting a dataset from YouTube and Reddit.
weaknesses: I see no major weaknesses. To make the proposed automatic evaluation method more compelling, it would be useful to see its performance on the programmatically evaluated tasks in addition to its match to the provided human evaluations on the  4 creative tasks investigated.
correctness: The dataset is constructed in a sound way, with the details clearly documented making for easy reproducibility or modification if desired.  The experiments provided support the claim of potential benefit from both the diversity of available tasks (as seen in the ability to easily craft a set of tasks to evaluate which are diverse but can also be grouped into similar subsets) and the benefit of the provided dataset (as demonstrated by the strong human correlation of the trained reward model and the strong performance of the agent trained using this reward model).
clarity: Yes
relation to prior work: The related work section is thorough and clear.
additional feedback: NA
review_title: Strong contribution towards goal of training general agents

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This work introduces a multi-task open-ended environment in Minecraft accompanied by a large-scale knowledge base to facilitate building generalizable agents. 
The task suite goes beyond previous Minecraft benchmarks both for its massive number of tasks and for the language-conditioned creative tasks that are more complex than programmatic tasks.
The authors also demonstrate an example research project built on MineDojo, which first learns a reward proxy from videos in the dataset and then trains a language-conditioned RL agent with this reward to solve both programmatic and creative tasks.

strengths: 1. MineDojo provides diverse and rich benchmark environments. Both manual and automatic approaches are adopted to create orders of magnitude larger number of provided environments than previous benchmarks. 
2. The benchmark contains interesting creative tasks that rely more on semantic understanding of languages and typically require learning more complicated behaviors. 
3. The environments and the processed knowledge base are open-sourced, which can significantly benefit the community on language-conditioned RL, multi-task learning, and open-ended learning. 

weaknesses: The example research project that trains a language-prompted agent to solve a subset of the task suite is relatively weak compared to the great capability of MineDojo in the following aspects:

1. The Wiki pages and Reddit posts are included in the knowledge base, but the conducted experiments have not utilized them.
It could be better to provide some evidence that these two sources of data can benefit learning interesting and complex skills as mentioned in the Introduction.
2. Experiment results are only presented over 12 tasks while MineDojo supports thousands of tasks. It might be better to explain how and why the 12 tasks are selected.
3. The current version is more of a language-conditioned multi-task RL agent, rather than an open-ended agent. 
Some potential enhancements could be testing whether the agent can generalize to open-vocabulary commands and trying to train one agent capable of different task groups.

For future projects that develop agents using the creative task suite, it is unclear how to benchmark their performances since the creative task suite does not have any manual reward or code-defined success criterion.
This work shows that the learned MineCLIP reward is a good evaluation metric for the 4 creative tasks, but it remains unclear whether MineCLIP can scale up / generalize to all 1553 tasks.
Also, it will be of great value to suggest evaluation protocols for future research that play with new tasks (either manually constructed or proposed by an open-ended learning agent) outside the current task suites with MineDojo API.

correctness: Most claims are well supported with details and experimental results. 
There might be a bit of overclaiming in the learned evaluation protocol since its alignment with human judgment is only reported on 4 different tasks.
clarity: The paper is generally well organized and easy to follow. Some details deserve more clarification: 
The evaluation metric in Table 1 is not explicitly described in the caption or the main paper. 
I guess the first 8 tasks are evaluated with the percentage of success and the last 4 creative tasks are evaluated with the predicted success signal thresholded from MineCLIP reward.

relation to prior work: This work has thoroughly discussed previous attempts for open-ended learning and also particularly benchmarks built with Minecraft.

additional feedback: Please refer to the previous fields.
review_title: Nice platform for building open-ended agents

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This work introduces a new Minecraft benchmark for developing general-purpose agents, equipped with large-scale multimodal knowledge from Youtube, Reddit, and Wiki. The benchmark suite includes more than 2000 tasks with unified observation and action spaces. Tasks can be categorized into kinds, programmatic tasks that have well-defined rewards and creative tasks that are more open-ended and thus are not associated with well-defined metrics of success. The paper also introduces 1) MINECLIP, a contrastive reward model that aligns video segments with text instructions (subtitles), and is used as a reward function for training RL agents, and 2) tips and tricks to make MINECLIP more efficient to query.


strengths: - Extensive evaluation on 12 tasks. 
- The dataset contains rich multimodal collections of crowd knowledge.
- The proposed MINECLIP tips to make language-conditioned reward models efficient can be useful for future embodied AI benchmarks.
- GitHub documentation and the gym-style API make this an excellent resource.
weaknesses: - To my understanding, neither MINECLIP nor the multitask RL agent are open-sourced. This seems to be a necessary component for future works to compare with. 
- Would be nice to include GATO and Unified-IO in related work which does not require unified observation and action spaces across tasks. Motivating the need for such unification would also be good to include as reasoning for this new benchmark.
- Please include more details on line 215 (" to preserve the layout information, we additionally save the screenshots of entire pages and extract 5.8M bounding boxes of the visual elements"). A visual example of the extracted elements would be sufficient. 


correctness: The evaluation and experimental design seem carefully crafted. The benchmark is also carefully designed and already open-sourced on GitHub with extensive documentation and a clear API. Wondering if the authors are considering open-sourcing the construction process for users to extend this work to other platforms and tasks, and how easy it is to incorporate new tasks.
clarity: The paper is generally well-written. The supplementary material contains information for several key details, from descriptions of the observation and action spaces to limitations and societal impact.
relation to prior work: Clear motivation and contribution over prior benchmarks.
additional feedback: - Any plans for open-sourcing the learned models?
- How easy is it for users to extend the framework with more tasks?
review_title: Great resource for building general-purpose agents with extensive documentation 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This work provides an excellent open-world platform for embodied AI agents to learn continually in solving multi-tasks. The work contributed in three main ways: 
1) Provide an environment API on the Minecraft game to help support the learning of a multitude of tasks and goals in an open world. 
2) The paper also provided a large-scale database of multimodal knowledge extracted from public domains to support learning in an open world.
3) The paper also introduces a novel agent learning algorithm that leverages a large pre-trained video-language model to tackle multi-tasks learning as a learned reward function. 


strengths: This work helps to address a significant problem in existing research in embodied AI research which is the constraint of training in isolated worlds/environments/rooms with limited complexity and diversity. The work aims to build a generalist embodied agent by providing 1) an open-world environment that supports an unlimited variety of open-ended goals and tasks, 2) curating a large-scale database of prior knowledge of this open-world environment that is useful for learning and 3) having a flexible and scalable agent's architecture that allows the conversion of large-scale knowledge into actionable insights. Furthermore, I like the idea of introducing a list of creative tasks that the reward function cannot be manually designed. The proposed novel task evaluation metric for the proposed tasks further shows that these proposed metrics align with human evaluations. 
weaknesses: 1. According to line 171, the author describes some of the creative tasks that would be brainstormed. I agree that some of the tasks lack straightforward success criteria, and humans cannot manually define the reward function. However, the tasks under creative tasks in Table 1 use tasks such as "find ocean, find the nether portal, etc" I am not sure if this can be classified as a form of navigation task in which the reward can be defined as long as the state space is made known. So my question is, how does the author truly quantify creative tasks apart from programmatic tasks? 
2. How did the paper utilise the unstructured Minecraft wiki information for learning actionable knowledge? This part wasn't mentioned clearly.
3. Just a suggestion, the author could provide a table to show a more straightforward comparison between the different platforms supporting Minecraft for AI research. The paper could also give some motivation and applications for the embodied AI research (citing a few survey papers from the field would be good).


correctness: The dataset and simulator are constructed in a sound way. 
clarity: The paper is well written with clarity. 
relation to prior work: To my knowledge, the authors provide a good overview of existing work from embodied AI simulator, Minecraft for AI research and internet-scale multimodal knowledge bases. However, for the related work on embodied AI, the author could have given some motivation and applications for the embodied AI research (citing a few survey papers from the field would be good).
additional feedback: No additional feedback.
review_title: MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper proposes the MineDOJO benchmark, which comes with an accompanying dataset. The benchmark is a selection of around 3100 tasks in the Minecraft simulator, each of which is described a natural-language prompt. The tasks are split into two categories: programmatic tasks, which have a ground truth reward function based on the simulator state; and creative tasks, that don't have a ground truth reward function and are more open-ended. The accompanying dataset has a large selection of youtube videos about minecraft, with time-aligned text transcripts, as well as around 7000 freeform minecraft wiki pages and around 350k reddit posts from r/Minecraft. The work propose using the video-text data to learn a CLIP-style model that learns to match open-vocabulary natural text descriptions (from the transcripts) to 16-frame video segments. This trained model is then used as a reward function for reinforcement learning, and also as an evaluation mechanism, especially for the creative tasks where there is no alternative. They show that this evaluation mechanism is fairly well-correlated with human evaluations of task completion, and show that using this reward function combined with PPO with self imitation learning enables the learning of language-conditioned policies which get some reward, and that their method performs better than just using plain CLIP (from OpenAI) as the reward function.
strengths: The large amount of diversity in the task space, the open-endedness of the underlying environment, the large amount of easy-to-use text, image and video data are all major strengths of the contribution. The paper is well-written (although many details are left to the appendix, which is understandable given the amount of information presented). Demonstrating an initial approach in MineCLIP is beneficial to enable future research, as there is an existing solid baseline.
weaknesses: There isn't much time spend on recommended evaluation protocols for future researchers. It would be beneficial to define (possibly several) fixed task selections, so that future researchers will have comparable results. This will encourage more robust research, and disincentivise cherry-picking tasks that future methods perform well on.

While there is a focus in the paper and dataset on open-ended learning, a key aspect of open-ended embodied AI is generalisation, and this doesn't seem to be discussed much or tackled by the benchmark as-is. I think Minecraft is a great domain to test generalisation due to the underlying procedural generation, and it would be great to leverage that to enable evaluating how general the learned policies are. For example, for simple tasks that aren't too dependent on the exact features of the terrain, randomising over procedurally generated environments (with fixed seeds) during training, and then evaluating on held-out environments (as in OpenAI Procgen for example) would enable this. A stronger form of task generalisation would tie in with the above recommendation: a subset of tasks could be chosen for training, and then models could be evaluated on their generalisation to new tasks zero-shot (or after a small amount of fine-tuning, if zero-shot performance isn't good).

I think if these recommendations and weaknesses are addressed (which seems feasible within the rebutal period) then I'd be happy to raise my score to an 8.

correctness: Generally the work is correct in its claims, given the experimental results. However, there are some misleading claims that I think aren't sufficiently cautious. For example, it's repeatedly claimed that the MineCLIP method is "competitive" with the hand-crafted reward functions, but there's not a statistical comparison to back up this fact. Secondly, in the intro (and in other places) it's claimed that "Our learned evaluation metric agrees with human judgement in > 97% of test cases and is consistently high-quality across different tasks", but this is only validated on 4 of the creative tasks, all of which seem on the relatively simple side. While it's not surprising that the MineCLIP approach works for these simple tasks (that can effectively checked in only a few frames), the more complex creative tasks such as building a haunted house or a mansion seem much harder to evaluate. It would be beneficial to investigate whether the MineCLIP reward function generalises well to these cases.
clarity: The paper is clear and well-written, apart from the concerns raised in the correctness section above.
relation to prior work: Prior work is generally well-cited. However, I think it would be beneficial to cite work on open-ended algorithms, such as POET, PAIRED and other works in this space, and contrasting their approach to open-ended agents with the approach presented in this paper.
additional feedback: How are tasks chosen for evaluation? Did you try multi-task training with more tasks?

How should we understand the reward numbers in Table 1? Are they normalised? If not, could they be? Could you present an average (normalised) reward across all programmatic tasks, so that we can validate whether your approach is "competitive" with the hand-designed reward functions in a statistical way. This could include using the tools from RLiable for more robust evaluation, given you train for multiple seeds over multiple tasks.

Could you describe Table 2 in it's caption, in addition to drawing conclusions from it?
review_title: MineDOJO Review

Meta-review: metareview: This paper proposes a new RL benchmark called MineDojo, which consists of thousands of diverse tasks on Minecraft game. The main challenge in designing such a benchmark is to define a good reward function to specify the desired task. This is very tricky since Minecraft is open-ended environment and there are various tasks, where human can't easily define the reward. To address this issue, the authors propose a very interesting idea: utilizing multi-modal (text-video) encoder as a reward. Specifically, the authors pre-trained the multi-modal encoder using contrastive learning (similar to CLIP) using multimodal data collected from the Internet and utilized the similarity between agent's behavior (video) and text as a reward function. Throughout human evaluation, the authors showed that their reward function can induce desired behavior (described by text). Since the authors open-sourced simulator, pre-trained reward model and agents, Minedojo can be a good starting point for making a progress on developing open-ended, multi-task RL agents.

Overall, all reviewers agreed that this is very solid submission and authors also handled concerns from reviewers during discussion period. I recommend acceptance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept