BEDD: The MineRL BASALT Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks
Authors: Stephanie Milani
Anssi Kanervisto
Karolis Ramanauskas
Sander V Schulhoff
Brandon Houghton
Rohin Shah
Venue: NeurIPS 2023 Datasets and Benchmarks Oral (2023)
Keywords: learning from human feedback
minecraft
human evaluation
embodied agents
rlhf
demonstrations
benchmark
evaluations
TL;DR: To facilitate algorithm development for the BASALT benchmark, we provide a large-scale dataset of human demonstrations and evaluations, along with a streamlined codebase for training, evaluating, and analyzing algorithms.

Abstract: The MineRL BASALT competition has served to catalyze advances in learning from human feedback through four hard-to-specify tasks in Minecraft, such as create and photograph a waterfall. Given the completion of two years of BASALT competitions, we offer to the community a formalized benchmark through the BASALT Evaluation and Demonstrations Dataset (BEDD), which serves as a resource for algorithm development and performance assessment. BEDD consists of a collection of 26 million image-action pairs from nearly 14,000 videos of human players completing the BASALT tasks in Minecraft. It also includes over 3,000 dense pairwise human evaluations of human and algorithmic agents. These comparisons serve as a fixed, preliminary leaderboard for evaluating newly-developed algorithms.  To enable this comparison, we present a streamlined codebase for benchmarking new algorithms against the leaderboard. In addition to presenting these datasets, we conduct a detailed analysis of the data from both datasets to guide algorithm development and evaluation. The released code and data are available at https://github.com/minerllabs/basalt-benchmark.

BEDD: The MineRL BASALT
Evaluation and Demonstrations Dataset for Training and Benchmarking Agents that Solve Fuzzy Tasks
Stephanie Milani
Anssi Kanervisto
Carnegie Mellon University smi laniianiii 0cs cmu. edu
Microsoft Research
anssi www.kkkaaeiittiiiiirrroooooooooo wtw.:/tttttmiicooootocoooooo com
Karolis Ramanauskas University of Bath kr711 7711@bathh.. ac uk
Sander Schulhoff University of Maryland sschulho@und. edu
Brandon Houghton
OpenAl
brandon@opemai.c. com
Rohin Shah rohinmshah@gmaill. com
Abstract
The MineRL BASALT competition has served to catalyze advances in learning from human feedback through four sare-fo--pecify tasks in Minecraft, such as create and photograph waterfall. Given the completion of two years of BASALT competitions, we offer to the community a formalized benchmark through the BASALT Evaluation and Demonstrations Dataset (BEDD), which serves source for algorithm development and performance assessment. BEDD consists collection of 26 million image-action pairs from nearly 14,000 videos of human players completing the BASALT tasks in Minecraft. It also includes over 3,000 dense pairwise human evaluations of human and algorithmic agents. These com- parisons serve as fixed, preliminary leaderboard for evaluating newly-developed algorithms. To enable this comparison, we present a streamlined codebase for benchmarking new algorithms against the leaderboard. In addition to presenting these datasets, we conduct detailed analysis the data from both datasets to guide algorithm development and evaluation. The released code and data are available at https: //gi thub. com/minemeeeeeeeeeee labs basaltaaattt.. benchmark.
Introduction
In traditional reinforcement learning, an agent learns how to act using reward based on an explicitly- defined reward signal [42]. This reward signal is often carefully designed by domain experts to communicate the intended goal for the agent to accomplish. Precisely specifying this form of feedback programmatically requires designers to a priori enumerate all potential outcomes or constraints on how they would like the task to be completed. This enumeration is difficult to achieve in practice, and the resulting reward signals often fall short correctly specifying the designer's intent [29]. To address this challenge, researchers have explored the idea of incorporating alternative channels for communicating information about the desired behavior of the agent. This class techniques generally called learning from human feedbuck (LfHF) [8, 23]. The goal of LfHF is to utilize the feedback modalities most likely to result in an agent acting according to human-desired specifications.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Training Rollouts
Evaluation Against Bacelines
Left
Right
Denoesstations
Trueball eraluation platform
Left than rUPrcal Agrrataeeing
New methad ranking
Baseline solutions
Bundlnn raniking
Figure 1: The BASALT benchmark. Green parts show the components of the included datasets. The agents compared using video recordings of them solving the task. We contribute data for training agents to perform fuzzy tasks and for evaluating agents with human evaluators. We contribute code for further benchmarking.
The complexity of this approach has been exemplified in the MineRL BASALT competitions [39], which leverage the open-ended and flexible Minecraft environment to promote the development of LfHF algorithms. This competition series has provided a platform for developing agents capable of solving fuzzy tasks that lack well-defined reward signals. However, despite two years of competition, no agents have yet matched human performance levels [33], emphasizing the need to establish the BASALT tasks as a consistent, standardized benchmark. In this paper, clarify the BASALT benchmark and present concrete evaluation recommendations toward the goal of consistency.
To support this benchmark we introduce the BASALT Evaluation and Demonstrations Dataset (BEDD), an open and accessible dataset for learning to solve fuzzy tasks from human feedback. As shown in Figure 1, this dataset consists of three main ingredients: the Demonstrations Dataset, the Evaluation Dataset, and supporting code for utilizing and analyzing the data, The Demonstrations Dataset consists of over 26 million actien pairs from 14,000 videos of labeled Minecraft gameplay of human players completing the BASALT tasks. To facilitate evalu- ating agents using real human judgments, we present the Evaluation Dataset, derived from the most recent BASALT competition [25]. The Evaluation Dataset consists of over 3,000 dense pairwise human evaluations of videos of various agents performing the BASALT tasks. By dense, we mean that each evaluation includes a comparison of the relative complcomnnntion performance of the agents and at least four additional questions, such as which agent results in 27,905 comparison points between 17 diffcrent agents. This dataset also includes a natural language response justifying why agent was selected as being the better of the two. The provided data functions as a leaderboard, offering researchers the ability to compare their newly-developed algorithm against various agents without redoing all costly human evaluations from scratch.
human-like. This
To facilitate the use of the Demonstrations Dataset and the Evaluation Dataset, we present streamlined codebase. With this codebase, one can train a new model from the demonstration dataset and evaluate against the provided leaderboard. Alongside the presentation these datasets, we conduct detailed analysis of the data to guide algorithm development and evaluation. We release the code and detailed documentation for others to perform such analyses. We hope that this codebase will assist others with quickly developing and evaluating LiHF algorithms to spur further progress toward agents that are better aligned with human intent.
2 The MineRL BASALT Benchmark
We provide an overview of the MineRL BASALT benchmark. consisting of a task suite and evaluation framework for learning from human feedback, illustrated in (Figure 1). The benchmark uses Minecraft, videogame that provides rich and complex environment in which to define different tasks. The states are pixel observations: actions are regular keyboard and mouse actions, closely following how humans play the game. This includes navigating the crafting menus using mouse (Figure 2).

---

Figure 2: Two example images of the pixel observations provided by the MineRL. environment used by BASALT. The agent receives pixels as observations and must use regular keyboard and mouse actions, including navigating the crafting menu with mouse to make items.
2.1 Tasks
All tasks are accompanied by a Gym environment [4] and either a simple English-language task description or a reward function to indicate the desired agent behavior. The reward-free tasks have the language task specification; the rewarded tasks are accompanied by a reward function.
We use the same four reward-free tasks from the BASALT competitions, since these tasks were empirically validated be challenging [33, 38]. The goal of FindCave for the agent discover a ceturally-generatte cave. To complete takeWaterfall, the agent must create a waterfall and subsequently take picture of it. Because taking a photograph is not supported in Minecraft, we simulate it by providing the agent with an option to end the episode: the state at the moment that the agent ends the episode is its photograph. animal pen next to an existing house, then corral matching pair of farm animals into the pen. In BuildVi 1 lageHHusse the agent spawns in a village and must build a new house in the same style as the other surrounding houses without damaging the village. For more task details, see Appendix A.
CreateVillageAniie
the agent must build an
Because reward-free tasks challenging evaluate, we include ObtainDi ammndhhveee [33], task with a concrete reward function, to enable quick iteration. This task is more challenging than ObtainDi amond [17], a well-established benchmark in reinforcement learning, since requires an additional crafting step after obtaining the diamond. An agent receives reward each time it obtains the next required resource in the crafting tree towards a diamond shovel.
2.2 Evaluation
To fairly compare algorithms on a benchmark, the parameters of that benchmark must be concretely defined. Previously, many techniques claimed to solve MineRL ObtainDi amond. Often, these successes involved giving the agent access to the full game state instead of pixel observations, building in human priors through extensive action shaping, or simplifying the underlying environment dynamics, Changes like these should be clearly stated and applied equally to compared methods. To avoid such discrepancies in the future, we specify some methodological best practices when using the BASALT benchmark: i) only pixel observations provided by the four environments should be used, ii) the environments should not be modified in any way, iii) action shaping [26] is permitted, as long it applied to all methods, iv) algorithms should be evaluated with TrueSkill [20] using the specific hold-out test seeds provided in Appendix A, and v) the final evaluations must be conducted with human evaluators. For further guidance on how to fairly compare algorithms, we refer the reader to recent work [35].
BASALT agents are evaluated on hold-out test environments to produce multiple videos of agents attempting to complete that task. A single trial consists of two agents pitted against one another, which is shown to human judge. This judge determines which agent completed the task better. The resulting dataset of human evaluations assessed using the TrueSkill system, which dynamically assesses the skills of particular player (in this case, an LfHF algorithm). In addition to estimating the relative skill of an algorithm, provides an uncertainty estimation. For more extensive discussion of this procedure and a comparison of TrueSkill to other rating systems, please see Appendix A. We hope that our demonstrations of successful task completions and the benchmarked algorithms from the competition serve starting point for evaluating the performance of other algorithms.
3

---

Developing algorithms while leveraging real human feedback is both expensive for machine learning practitioners and tinecconunnigg for human evaluators. Automated human evaluations have emerged as a valuable part of the pipeline for assessing various aspects of machine learning models [7, 10]. The goal not to replace human evaluations: instead, it complements the process by providing quick, iterative feedback for algorithm development and initial assessment. As result, we propose automating evaluations [12] as an additional component of the BASALT benchmark. The inclusion of ObtainnDamonnSSooee may help develop reward modeling techniques [22] due to its associ- ated concrete reward function. More generally, we hope that with the release of the Evaluati in Dataset, others can begin developing approaches toward this goal.
2.3 Benchmarking Algorithms on BASALT
To assist with the development of LiHF algorithms and automated evaluations, we implement and share a codebase with two major contributions. First, the code contains an example of training a LIHF algorithm with the shared data in the Demonstrations Dataset. Second, we include the tools for performing the evaluations presented in this work. The code is a Python-instalee installable library, which allows the functionality to be imported into other codebases for use research.
The training example provides tools to train a behavior cloning model on top of the Video Pre PraTniiigg (VPT) [2] model using the imitation [14] library. VPT is large foundation model that can complete various tasks in Minecraft, but it is difficult to fine-tune on new tasks due to its size. Inspired by the success of an imitation-learning approach' in the BASALT 2022 competition, we use behavior cloning as the base algorithm with the rich embeddings from the VPT model as input. Motivated by the original VPT results, we remove the no-op actions from the demonstration dataset. We provide code to use different variants of the VPT model for benchmarking. The final output is a set of videos use in evaluations against the shared recordings of other agents in the Evaluation Dataset.
Because setting up human evaluations is coneucins,,,ng, we share our platform for conducting human evaluations. The platform is served a webpage backed by simple Python- Python-based webserver. The collected data is either stored locally in a SQLite database or remotely in a more scalable form. The platform includes flexible API to add remove agents from the of comparisons. After a simple setup, researchers can point human evaluators to a specific URL to provide answers. We provide examples of the form that the human evaluators will see Appendix D. We also share code to create figures of the TrulSkill ratings (and all analyses in Section 4), given the answers contained in the resulting database. This whole process serves an end-to-end example of creating new method with Demonstrations Dataset, then evaluating it with data from Evaluation Dataset.
3 BASALT Evaluation and Demonstrations Dataset
We now introduce BEDD, our extensive dataset of human demonstrations and algorithm evaluations. This dataset consists of the following components:
The Demonstrations Dataset, set of 13,928 videos (state-action pairs) demonstrating largely successful task completion attempts of the reward-free tasks,
The Evaluation Dataset, set of 3,049 dense pairwise comparisons of algorithmic and human agents attempting to complete the BASALT tasks, and
The code for utilizing and analyzing these datasets for developing LfHF algorithms (some details in Section 2.3).
For full datasheet [13), please see Appendix B.
3.1 Demonstrations Dataset: Completing Reward-Free Tasks in Minecraft
The Demonstrations Dataset for developing new methods consists of 361 hours (26 million image-action pairs) of human demonstrations of the reward-free BASALT tasks. This data consists of labeled trajectories, both with high-resolution image observations and keyboard and mouse actions for each frame. In total, this is 651 GB of data. Table decomposes the high-level data statistics by task. More details about this dataset in Appendix C.
'https: http//gtttttutttteeeeeeeeeee.het www.///www.ss.oocoooot http:///hhhishhhssmmmmem.......

---

Task
Videos
Episodes
Hours
Size
Ep. len,
Success %
FindCave
5,466
5,466
165GB
60
93%
MakeWaterfall
4,230
4,176
97
175GB
84
98%
CreateVilllagenniialPeer
2,833
2,708
89
165GB
119
95%
BuildVillageHouse
1,399
778
85
146GB
391
92%
Total
13,928
13,128
361
651GB
99
95%
Table 1: High-level demonstration data statistics decomposed by task. Episode length is the average episode length in seconds. A demonstration is counted success if the player manually ended the episode instead of dying or timing-out.
Words in
Response Sentiment
Task
Comparisons
Hours
Response
O
4
C>
FindCave
722
60
27,948
79%
14%
7%
MakeWaterfall
682
56
26,437
76%
7%
17%
CreateViillageknimalleen
914
81
32,768
57%
11%
32%
BuildVillageHousee
731
76
26,917
63%
9%
28%
Total
3,049
273
114,070
Table 2: High-level evaluation data statistics decomposed by task. We report the total number of agent-agent comparisons, human labor hours, and words used in the natural- -antuaal justifications of selecting specific agent as the best one. We also report the percent positive, neutral, and negative sentiments in these justifications.
Each demonstration consists of trajectory [sp, dns ........5 , aNJ, sequence of state-action pairs, where N is the trajectory length. These pairs are contiguously sampled at every Minecraft game tick (20Hz). Each state consists of the 640x360 RGB frame from the perspective of the player (see Figure 2). Each action consists of two parts M is all "mouse" interactions (change in view, pitch, and yaw), mimicking the native human control interface of Minecraft. This dataset serves as a starting point for using demonstrations as a form of feedback to train agents.
[K, MM. where K is all keyboard interactions,
3.2 Evaluation Dataset: Evaluating BASALT Agents
The Evaluation Dataset contains 3,049 pairwise comparisons of different algorithms, produced from 273 hours human labeling effort by 65 unique MTurk workers. All responses are contained in single JSON file. Table 2 decomposes this dataset by task. Each evaluation in the dataset consists of the following: (i) the names of the two agents used in the comparison, (ii) the corresponding videos shown to the human judge, (iii) answer to the question of which player is better overall (Left, Right, Draw), (iv) a natural-language justification of this choice, (v) answers at least one direct question about concrete achievements by the players (Left, Right, Both), and answers to 4-7 comparative questions, such as which agent was more human-like (Left, Right, Draw, N/A). The direct and comparative questions are task-specific. Including the choice of which algorithm performed best, this dataset consists of a total of 27, 97,955 comparisons along various factors.
We provide all responses in our public release of the data. We also include a list of anonymized MTurk workers whose responses we found not to suit our standards (e.g., providing the same answer to every task). Before performing the analyses reported in this paper, we filtered the data to exclude these responses. We provide all details needed understand and reproduce these evaluations in Appendix D. With this dataset, one may compare develodeee algorithm with 17 possible agents: the top 13 teams the 2022 BASALT competition, a behavioral cloning baseline, a random agent, and two human experts (two of the authors of this paper). Researchers can use the provided human evaluations to kickstart the human evaluation, without needing to dedicate costly human labor to evaluating all algorithms from scratch.
5

---

500
Number of Stepi with Move Kiy Pressed
Number af Right Mouse Clicks
7000
400
60000
5000
300
4000
200
2000
2000
100
Waterfall
(a) Right mouse button clicks
Movement key presses
The distributions of the number of right mouse button clicks and movement key presses across tasks in the Demonstrations Dataset, which proxies for the number of blocks placed and distance traveled per episode, respectively. Because BuildVillageiinn complete, reasonable that it requires more right mouse button clicks and movement key presses.
Figure 3:
takes the longest to
4 Analysis: Demonstrations Dataset
We now analyze the Demonstrations Dataset. Because the BASALT tasks lack concrete reward functions, evaluating the progress agents on these tasks is challenging. This difficulty necessitates the use of informative proxy measures. As result, when analyzing the Demonstrations Dataset, we focus on defining proxy measures that may be useful for understanding the data or tracking training progress. In this section, describe these proxies and present the results of analysis.
We first seek to understand the relative difficulty of the four reward- free tasks. Given the experience of the contracted data collectors with Minecraft, believe that the length of the demonstration is sonable proxy for task difficulty. Using demonstration length as a proxy for task difficulty, we note that BuildVillageHoous
is likely the most challenging task, for humans: each video lasts around 6.52 minutes on average, while the next most cone-conuuning task, Create L takes average of around 1.98 minutes (Table I). By this metric, the easiest task is FindCave (1 minute). In practice, one could use this metric to assess training progress or likelihood of task completion. If an agent completes task worse quality behavior. In contrast, if an agent takes a similar amount of time to the average to complete task, then that could signal better behavior (but is clearly not definitive).
malPen,
that far away from the average, that may signal
We also want to understand when an agent human or AI - may be underperforming on the task. Because FindCave requires navigation to find a cave, agent that remains stationary is likely unsuccessful. Similarly, because CreateVill weatteeiaaggnnimmaaP... construction of objects, agents that fail place any blocks likely do succeed at the task. As a result, we employ the number of steps with an active movement key as a proxy for the distance traveled within the tasks. We also the count of right mouse button clicks proxy for the number of blocks placed. The only other actions performed by clicking the right mouse button are using crafting table chest, of which there relatively few per episode.
and Build BuildVillageHous require the
These proxies are depicted in the per-episode distributions within the Demonstrations Dataset in Figure 3. The number of right mouse button clicks is highest around 180 in BuildVillageefouse These align with expec-
and between 20-40 in MakeWaterfall and CreateVill SeraateiillageAnimalle... tations, given the amount of building required to complete the tasks. The number of movement actions has the highest variability in FindCave. This task also has the shortest time-out at 3,600 steps. Perhaps better proxy would be the average number of movement actions per step: FindCave should have the highest number, as the task mainly consists of moving around.
We include these results both as way understand this dataset and as values to monitor during agent training estimate agent performance. However, is crucial to avoid using these metries as direct optimization targets, as they can be casily exploited.

---

Normalized TrueSkill
Sentiment
Agent
Human2
2.43
92%
5%
3%
Human
2.17
92%
6%
2%
GoUp
0.73
74%
19%
7%
UniTeam
0.13
66%
25%
9%
BC-Baseline
0..32
65%
26%
9%
Random
- -1.35
63%
29%
8%
Table Normalized TrueSkill score and percent of positive, neutral, and negative sentiments of the natural language justifications for selecting an agent as the best one.
5 Analysis: Evaluation Dataset
We first analyze the overall dataset, then focus on a few main comparisons: the top two algorithms from the BASALT 2022 competition (GoUp and UniTeam), the behavioral cloning baseline (BC- Baseline), the two human experts (Human1 and Human2), and a random agent (Random). This subset of data corresponds to 394 of 3,049 total comparisons or nearly 34 hours of human labeling effort. For additional details about the subsequent analyses, please see Appendix E.
5.1 Task-Based Analysis
Table provides a high-level overview of the 3,049 total evaluations, decomposed by task. The human judges generally responded using similar number of words across the different tasks. Only the responses to FindCave and MakeWaterfall contained significantly more words on average than CreateVVillageAmimalll
(F = 3.72, p 0.01; 95% CI: 0.34 5.37, p 0.02; 95% CI: 0.36 to 5.47, p = 0.02, respectively). All other tasks exhibited no significant differences between means. This result suggests that expressing the specific rationale for these simpler tasks may be easier.
To understand the general perceptions of the human judges of the different tasks, we categorized each response into positive, neutral, negative sentiment [31]. We then analyzed the differences in the distribution across tasks. The human judges displayed different levels of sentiment in their re- sponses, depending the task.' They responded most positively to FindCave and least positively to CreateVillageAnimaan Pen. The only sentiment distributions that were not significant were between AnimalPen and Bui welilliilllleee lavillageooeus (X²(1, N 1645) 6.35, p 0.042), and FindCave and NakeWaterfall (X*(1, N = 1404) = 2.694, p = 0.260). All other pairs of tasks exhibited sig- nificant differences in sentiment." Due to the large number of subtask dependencies required for completing peeateeillagenniiii
and BuildVil hivilllae ageHouse (as well as the greater amount of time required on average for human demonstrators to complete these two tasks, detailed in Table I and Section 4), they are considered to be more challenging than FindCave and MakeWaterfall. The more positive sentiment toward FindCave and MakeWaterfall may be due to an increased focus on successes due to their relative ease of completion.
5.2 Agent-Based Analysis
We now turn our attention to agent- based analysis. We first provide a brief overview of the included agents. GoUp uses human knowledge to decompose the tasks into the same high-level sequence, then uses computer vision techniques to identify the goal for each task (e.g., the cave). Uni combines behavioral cloning with search by embedding the current of images with pre-trained VPT network and then searching for the nearest embedding point in the VPT latent space to find the situation use as reference and copying the corresponding expert actions [32]. More details about the other assessed agents can be found in previous work [33].
We conducted a one-way ANOVA with a post-hoc Tukey's HSD test to obtain these results. Results presented (ANOVA F' value, value; CI for Tukey's HSD. value). 'We conducted a chi-square
independence examine the relationship between task and sentiment classification. The relation between these variables was significant, X (6, N = 3049) = 132.21, p < 0011.
"These results are from Bonferroni-corredd pairwise Chi-square tests.
7

---

6.5
6,0
Score
0.0
found
1.5
cuick and ttheret Movemant
Found Care Pastar
Better Covn Seatch in uniowen Aread
Butter Naviganen to Cart sread
tave Farcaptonn
better Cave Desection
fumon
Greated Rattrrai
0+
More Quice Efficenn Morenent
Looked Nattrial
Botter Location Waterial
better fitton of Wateraai
More Homan Like
FindCave
(b) MakeWaterfall
10
3.0
0.5
300r
Score
0.0
-0
0,5
-1.0
vilage Harm
Getoer Hesp Locaton Hore Honsee Lon strucior
Interttioall Bock Maacnng
Appropriate Bluck riacing
Botter Styte Mascoind More astrecive Houde Hore Hurton
Better Endtaee Space Conner Sumter fesimats Cornec hype of Anomata
vilsug
Prpper fot to House Location More sfrectie Search Efreciive Pessing
More thrraan Like
(c) BuildvillaeeHouse
(d) An Pen
Figure 4: Comparison of baseline solutions, top BASALT 2022 competition solutions, and humans on additional questions. Bars represent average score, and error bars represent standard error. Higher is better: it means that agent exhibited more of that factor according to human judges.
We present the agent-based overview of the evaluations in Table 3. Note that the reported TrueSkill scores are computed over the 3,049 total comparisons, whereas the sentiment analysis is performed only the 394 agent-specific entries. The human agents performed the best using TrueSkill ranking, with large performance gap between them and the best- rest-performing algorithmic agent. This result emphasizes the difficulty of BASALT for algorithmic agents.
The only sentiment distributions that were significant were between either of the two human experts and the other algorithms. In particular, the sentiment toward comparisons that included human players were significantly more positive." 4 All other pairs algorithms did not exhibit significant differences in sentiment. In general, human judges were generally positive when evaluating the agents. The judges may have been more positive due to social influence: they knew that the videos were produced by different players for ranking in the competition.
We now investigate the responses to the direct and comparative questions. For succinetness, we summarize each question into a single factor. For example, we use "More Human Like" to capture the question, "Which player seemed more human-like (rather than bot or computer player)?". Figure presents these results. Importantly, no machine learning algorithms outperform humans on any of the factors, indicating that there is still plenty of improvement be made. When analyzing future algorithms, suggest preserving this decomposition to both validate that the responses are sensible and obtain a fine-grained understanding agent capabilities. To validate sensible responses, we suggest checking the scores for the random agent: except for causing the least harm, this agent should score poorly compared to the other agents.
Finally, we highlight few interesting findings that are enabled by our decomposed and extensive set of evaluation criteria. Figure 4b (MakeWaterfal all) in the paper reveals that, although Team GoUp's algorithm can create waterfalls at a rate more similar to the human players, it struggles along all other criteria, including choosing good location and taking a high-qualit photograph. Only looking at the binary success/failure condition for creating waterfall would ignore these important nuances in the algorithm's behavior. As another example, Figure 4a (FindCave) suggests that, while Team GoUp's is algorithm still struggles to find caves, it can reasonably search for and navigate to areas that are likely to have caves. This finding suggests that the performance bottleneck may the cave detection system employed by this approach.

---

Related Work
Learning from Human Feedback Learning from human feedback has become a crucial research direction in machine learning [1, 8, 391, aiming to leverage human expertise to improve algorithm performance and generalization. Various algorithms have been proposed to integrate human feedback into the learning process, often using techniques such as imitation learning [21], inverse reinforcement learning [5. 45], and reward modeling [41]. Most commonly, these algorithms are evaluated in standard reinforcement learning benchmarks [3. 9, 11, 43]. However, most of these benchmarks do not have the property that human feedback is critical for task identification, as in BASALT. some Atari games, if an agent does anything other than the intended gameplay, it dies and resets to the initial state, so pure curiosity-ban -hassd agents perform well [6]. In contrast, BASALT tasks require human feedback or data to identify and complete tasks.
Minecraft for Machine Learning Minecraft is as a valuable platform for machine learning research [19, 30, 40] due to its complex. dynamic environment and customizable game engine. As a result, numerous competitions and benchmarks [16, 18, 24, 37] have been developed to improve Al capabilities. These often include tasks with concrete or programmatic reward functions, such as learning to collect diamonds in a sample-efficent way [17], multi-agent learning of cooperative and competitive tasks [36], and more [12]. In contrast, emphasize tasks that are designed to be hard to specify through a reward signal. Other work focuses on natural language as a specific modality for communicating intent [15, 27, 28, 34]; instead, we aim to promote the development of algorithms that generally learn from human feedback, including natural language. Some techniques require complete game state information (knowledge of blocks and items around them) [44, 46]; however, our benchmark only permits agents to access observations, not game state, to promote learning from pixels, which is more generalizable to other tasks.
Comparison to MojeDojo Perhaps the most similar benchmark to ours is MineDojo [12], which emerged during the BASALT competitions and contributed large dataset of Minecraft gameplay with the aim of developing cenerally-capable embodied agents. The focus of MineDojo was to provide massive dataset seraped from the internet; in contrast, we focus curating smaller set of high-quality demonstrations and evaluations. Our demonstration data was produced by experienced Minecraft data collectors using a consistent Minecraft version and settings. In contrast, the MineDojo data, although plentiful, contains many videos with streamer overlays and non-Minecraft parts, different texture packs, mod packs, and more. Although this diversity may be beneficial in some settings, previous work had to filter out such data before it could be useful for training [2]. Another critical difference is evaluation: MineDojo provides only binary success or failure criterion; instead, our recommended evaluation and resulting Evaluation Dataset involves human judgments across a range of quantitative (e.g., Found Cave) and qualitative (c.g.. Style Matching) criteria.
7 Conclusion
We proposed BEDD, a large and accessible dataset to facilitate algorithm development for the BASALT benchmark on learning from human feedback. The benchmark consists of five tasks in Minecraft, four of which lack reward function. We clarified the benchmark, providing concrete evaluation recommendations and introducing another avenue for benchmarking: automated evaluations. Our dataset, BEDD, consists of two key parts: Demonstrations Dataset and Evaluation Dataset. Demonstrations Dataset contains almost 14,000 videos showing successful task completions of the four reward-free tasks. The Evaluation Dataset consists of over 3,000 human evaluations to support the development of automated evaluations of sard-to-specify tasks. To our knowledge, this dataset is the largest of its type: one that supports both training and evaluating LiHF agents tasks with hard-to- tard-to--pe reward functions.
We demonstrated the utility of our dataset and benchmark by presenting an analysis of the demon- strations and the several algorithms in the Evaluation Dataset. We showed that the algorithms exhibit varying degrees of performance on our tasks, as evaluated by human judges on a variety of factors. Our results suggest that there ample room for improvement learning from human feedback. With the inclusion of our accessible code for benchmarking and evaluating agents, we hope that our contributions will encourage the development of more effective approaches for both learning from human feedback and evaluating these techniques in the future.

---

Acknowledgments and Disclosure of Funding
Creating these datasets and this benchmark was only possible with the help of many people and organizations. FTX Future Fund Regranting Program, Microsoft, Encultured AI, and AI Journal provided financial support for the competition and resulting data. We thank Berkeley Existential Risk Initiative (BERI) for the support in organizing the BASALT competition. Karolis Ramanauskas was supported by the UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent Al (ART-AI) [EP/S023437/1] and the University of Bath. We thank our amazing advisory board of the 2022 competition - Fei Fang. Kianté Brantley, Andrew Critch, Sam Devlin, and Oriol Vinyals -for for their advice and guidance. We also thank any previous organizers or advisors of the previous competitions for their contributions. Finally, we thank Alcrowd for their help hosting the competion and the MTurk workers for their work evaluating the submissions.
References
[1] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint 1X02:1992.042577 2019.
[2] B. Baker, Akkaya, P. Zhokov, Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-2444 24654,, 2022.
[3] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[4] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: 1606.0114 2016.
[5] D. S. Brown, Y. Cui, and S. Nickum. Risk-aware active inverse reinforcement learning. In Conference on Robot Learning, pages 362-372. PMLR, 2018.
[6] Y. Burda, H Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv: 1808. 04355, 2018.
[7] V. Chen, N. Johnson, N. Topin, G. Plumb, and A. Talwalkar. Use-case Use-case-grounded simulations for explanation evaluation. Advances in neural information processing systems, 2022.
[8] F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[9] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to benchmark reinforcement learning, 2019.
[10] S. Devlin, R. Georgeseu, I. Momennejad, J. Rzepecki, E. Zuniga, G. Costello, Guy Leroy, A. Shaw, and K. Hofmann. Navigation turing test (ntt): Learning to evaluate human-like navigation, In International Conference on Machine Learning, pages
PMLR, 2021.
[11] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329- 1338. PMLR, 2016.
[12] L. Fan, G. Wang, Y. Jiang, Mandlekar, Y. Yang, H. Zhu, A. Tang, D. A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.08855 2022.
[13] Gebru, Morgenstern, B. Vecchione, J. Wortman Vaughan, H. Wallach, H. Daumé III, and K. Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-92 2021.
[14] A. Gleave, M. Taufeeque, J. Rocamonde, E. Jenner, S. H. Wang, S. Toyer, M. Ernestus, N. Belrose, S. Emmons, and Russell. imitation: Clean imitation learning implementations.
2022.
10

---

[15] J. Gray, K. Srinet, Y. Jernite, H. Yu, Z. Chen, D. Guo, S. Goyal, C. L. Zitnick, and A. Szlam. Craftassist: A framework for dialogue-enabled interactive agents. arXiv preprint arXiv: 1907.08584, 2019.
[16] D. Grbic, B. Palm, E. Najarro, C. Glanois, and S. Risi. Evocraft: A new challenge for open-endedness. In Applications of Evolutionary Computation: 24th International Conference, 2021.
[17] W. H. Guss, C. Codel. K. Hofmann, B. Houghton, N. Kuno, S. Milani, S. Mohanty, D. P. Liebana, R. Salakhutdinov, N. Topin, et al. Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors. The 33rd Conference on Neural Information Processing Systems (NeurIPS) Competition Track, 2019.
[18] D. Hafner. Benchmarking the spectrum of agent capabilities. arXiv preprint arXiv:2109.0678 2021.
[19] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04/00 2023.
[20] R. Herbrich, I. Minka, and T. Graepel. TrueskillIM; bayesian skill rating system. Advances in neural information processing systems, 19, 2006.
[21] R. Hoque, A. Balakrishna, E. Novoseller, Wilcox, D. Brown, and Goldberg. Thriftydag- ger: Budget-aware novelty and risk gating for interactive imitation learning. 2021.
[22] B. Ibarz, J. Leike, T. Pohlen, G. Irving. S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
[23] H. Jeon, S. Milli, and A. Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415 4426, 2020.
[24] M. Johnson, K. Hofmann, T. Hutton, and D. Bignell. The malmo platform for artificial intelligence experimentation. IJCAI, pages 4246-4247, 2016.
[25] A. Kanervisto. S. Milani, K. Ramanauskas, B. V. Galbraith, S. H. Wang. B. Houghton, S. Mo- hanty, and R. Shah. The MineRL BASALT competition on learning from human feedback. In The 36th Conference Neural Information Processing Systems (NeurIPS) Competition Track, 2022.
[26] A. Kanervisto, C. Scheller, and V. Hautamäki. Action space shaping in deep reinforcement learning. In 2020 IEEE Conference on Games (CoG), pages 479 486. IEEE, 2020.
[27] J. Kiseleva, Z. Li, M. Aliannejadi, S. Mohanty, M. ter Hoeve, M. Burtsev, A. Skrynnik, A. Zholus, A. Panov, K. Srinet, et al. Neurips 2021 competition iglu: Interactive grounded language understanding in collaborative environment. arXiv preprint arXiv: 10. 10.06366. 2021.
[28] J. Kiseleva, A. Skrynnik, A. Zholus, S. Mohanty, N. Arabzadeh, M.-A. Côté, M. Aliannejadi, M. Teruel, Z. Li, M. Burtsev, M. ter Hoeve, Z. Volovikova, A. Panov, Y. Sun, K. Srinet, A. Szlam, and A. Awadallah. Iglu 2022: Interactive grounded language understanding in a collaborative environment neurips 2022. arXiv preprint arXiv:2205.1377. 2022.
[29] V. Krakovna, J. Uesato, V. Mikulik, M. Rahtz, T. Everitt, R. Kumar, Z. Kenton, J. Leike, and Legg. Specification gaming: the flip side of ai ingenuity. DeepMind Blog. 2020.
[30] S. Lifshitz, K. Paster, H. Chan, J. Ba, and S. Mcllraith. Steve-1: A generative model for text-to-behavior in minecraft. arXiv preprint MXXiv233060009377 2023.
[31] S. Loria et al. textblob documentation. Release 0. 2(8), 2018.
[32] Malato, F. Leopold, A. Raut, V. Hautamäki, and A. Melnik. Behavioral cloning via search in video pretraining latent space. arXiv preprint 22X 2022.
11

---

[33] S. Milinii, A. Kanervisto, K. Ramanauskas, S. Schulhoff, B. Houghton, S. Mohanty, B. Galbraith, K. Chen, Y. Song. T. Zhou, al. Towards solving fuzzy tasks with human feedback: A retrospective of the minerl basalt 2022 competition. arXiv preprint arXiv:2303. 13512, 2023.
[34] A. Narayan-Chen, P. Jayannavar, and J. Hockenmaier. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1400-5515,,1 July 2019.
[35] A. Patterson, S. Neumann, M. White, and A. White. Empirical design in reinforcement learning. arXiv preprini arXiv:2304.01315 2023.
[36] D. Perez-Liebana, K. Hofmann, S. P. Mohanty, N. Kuno, A. Kramer, S. Devlin, R. D. Gaina, and D. Ionita. The multi-agent reinforcement learning in Malmö (MARLÖ) competition. arXiv preprint 1rXiv:1901.08/22 022 2019.
[37] C. Salge, M. C. Green, R. Canaan, and J. Togelius. Generative design in minecraft (gdme) settlement generation competition. In Proceedings of the 13th International Conference on the Foundations of Digital Games, pages 1-10, 2018.
[38] R. Shah, S. H. Wang, C. Wild, S. Milani, A. Kanervisto, V. G. Goecks, N. Waytowich, D. Watkins- Valls, B. Prakash, E. Mills, et al. Retrospective on the 2021 minerl basalt competi- tion on learning from human feedback. In NeurlPS 202/ Competitions and Demonstrations Track, pages 259-272. PMLR, 2022.
[39] R. Shah, C. Wild, S. H. Wang, N. Alex, B. Houghton, W. Guss, S. Mohanty, A. Kanervisto, Milani, N. Topin, et al. The minerl basalt competition on learning from human feedback. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Competition Track, 2021.
[40] R. Smit and H. Smuts. Game-based learning-teachinnn artificial intelligence to play minecraft: systematic. Proceedings of Society, 93:1 00:1888200 202, 2023.
[41] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing S 33:3008-300 10018 2020.
[42] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018
[43] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki,
J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint 1rXiv:1801.00699 2018.
[44] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L.. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. 2023.
[45] H.-N. Wu and M. Wang. Human in tn-the-lopp behavior modeling via an integral concurrent adaptive inverse reinforcement learning. IEEE Transactions Neural Networks and Learning Systems, 2023.
[46] X. Zhu, Y. Chen, H. Tian, C. Tao, W. Su, C. Yang, G. Huang, Li, X. Wang, Y. Qiao, Z. Zhang, and J. Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. 2023.
12


Reviews:
[{"review_id": "m50DzhLKes", "reviewer": "Reviewer_328S", "summary": "", "questions": "", "limitations": "Yes,  the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. A very strong data set with detailed and carefully design. \n2.  A codebase for further study.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "EPLVuWLS4O", "reviewer": "Reviewer_Vxj5", "summary": "", "questions": "", "limitations": "It is not clear whether the scale of the collected data (26M image-text pairs from 14k videos) is sufficient to test larger scale models. Authors should elaborate whether this is the case and how does it compare to data amounts employed in MineDojo.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Remarkable data collection effort: 14k videos of human play, with annotations that also score human behavior aspects (eg human-like score, over 3000 annotations of various human and algorithmic agents provided by human evaluators), in addition to conventional reward based scores\n- Relevant topic: providing benchmark for algorithms that learn from human feedback to cope with complex open world environment in Minecraft\n- A streamlined codebase to make it easy for the potential users to evaluate a new model and to compare it to the existing leaderboard\n- Aiming at open sourcing whole dataset and workflows around it", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "geDhhjj8V4", "reviewer": "Reviewer_PcC9", "summary": "", "questions": "", "limitations": "The authors did provide a limitations section in the Appendix. Along with the suggestion above this limitations section seems reasonably thorough.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Overall on all these axes I believe this is a strong submission which will be of use to the community. One caveat to my review is that I do not directly work on Minecraft RL tasks or current RL interaction datasets.\n\n\n1. The main data contributions (video data for behavioral cloning purposes) are highly relevant to the community. \n2. The quality of the analysis of these successful examples goes beyond simple data scraping, with useful human annotations for success and failure cases. The authors appear to have put significant effort into data cleanliness and verification.\n3. The evaluation benchmark and automated evaluation suggestions also seem very useful for quick iteration, as human eval is hard to set up. I did not have a chance to directly evaluate the code for this however.\n4. The standardization of BASALT benchmarks through explicit evaluation criterion is likely useful for the community. Especially for ensuring that certain benchmarks stay rigorous.\n5. Ambiguous tasks with unclear termination criteria are also useful to demonstrate complex behavior. This paper suggested an interesting way to evaluate these methods using direct comparison (TrueSkill, ELO, etc.).", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper has received a consistent set of review comments, including two strong acceptance reviews and one weak acceptance review. All reviewers have praised the substantial contributions made in terms of data collection, baseline construction, and codebase. In light of these positive assessments, I believe the paper merits a clear 'accept' recommendation.
Decision: Accept (Oral)