SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We now turn our attention to agent-based analysis. We first provide a brief overview of the included agents. GoUp uses human knowledge to decompose the tasks into the same high-level sequence, then uses computer vision techniques to identify the goal for each task (e.g., the cave). UniTeam combines behavioral cloning with search by embedding the current set of images with a pre-trained VPT network and then searching for the nearest embedding point in the VPT latent space to find the situation to use as reference and copying the corresponding expert actions [32]. More details about the other assessed agents can be found in previous work [33].
2We conducted a one-way ANOVA with a post-hoc Tukey's HSD test to obtain these results. Results presented as (ANOVA  $F$  value,  $p$  value; CI for Tukey's HSD,  $p$  value).
<sup>3</sup>We conducted a chi-square test of independence to examine the relationship between task and sentiment classification. The relation between these variables was significant,  $X^2 (6,N = 3049) = 132.21,p < .001$ .
4These results are from Bonferroni-corrected pairwise Chi-square tests.
(a) FindCave
(b) MakeWaterfall
(c) BuildVillageHouse
(d) AnimalPen
Figure 4: Comparison of baseline solutions, top BASALT 2022 competition solutions, and humans on additional questions. Bars represent average score, and error bars represent standard error. Higher is better: it means that agent exhibited more of that factor according to human judges.
We present the agent-based overview of the evaluations in Table 3. Note that the reported TrueSkill scores are computed over the 3,049 total comparisons, whereas the sentiment analysis is performed only on the 394 agent-specific entries. The human agents performed the best using TrueSkill ranking, with a large performance gap between them and the best-performing algorithmic agent. This result emphasizes the difficulty of BASALT for algorithmic agents.
The only sentiment distributions that were significant were between either of the two human experts and the other algorithms. In particular, the sentiment toward comparisons that included human players were significantly more positive.<sup>4</sup> All other pairs of algorithms did not exhibit significant differences in sentiment. In general, human judges were generally positive when evaluating the agents. The judges may have been more positive due to social influence: they knew that the videos were produced by different players for ranking in the competition.
We now investigate the responses to the direct and comparative questions. For succinctness, we summarize each question into a single factor. For example, we use "More Human Like" to capture the question, "Which player seemed more human-like (rather than a bot or computer player)?". Figure 4 presents these results. Importantly, no machine learning algorithms outperform humans on any of the factors, indicating that there is still plenty of improvement to be made. When analyzing future algorithms, we suggest preserving this decomposition to both validate that the responses are sensible and obtain a fine-grained understanding of agent capabilities. To validate sensible responses, we suggest checking the scores for the random agent: except for causing the least harm, this agent should score poorly compared to the other agents.
Finally, we highlight a few interesting findings that are enabled by our decomposed and extensive set of evaluation criteria. Figure 4b (MakeWaterfall) in the paper reveals that, although Team GoUp's algorithm can create waterfalls at a rate more similar to the human players, it struggles along all other criteria, including choosing a good location and taking a high-quality photograph. Only looking at the binary success/failure condition for creating a waterfall would ignore these important nuances in the algorithm's behavior. As another example, Figure 4a (FindCave) suggests that, while Team GoUp's algorithm still struggles to find caves, it can reasonably search for and navigate to areas that are likely to have caves. This finding suggests that the performance bottleneck may be the cave detection system employed by this approach.
Learning from Human Feedback Learning from human feedback has become a crucial research direction in machine learning [1, 8, 39], aiming to leverage human expertise to improve algorithm performance and generalization. Various algorithms have been proposed to integrate human feedback into the learning process, often using techniques such as imitation learning [21], inverse reinforcement learning [5, 45], and reward modeling [41]. Most commonly, these algorithms are evaluated in standard reinforcement learning benchmarks [3, 9, 11, 43]. However, most of these benchmarks do not have the property that human feedback is critical for task identification, as in BASALT. In some Atari games, if an agent does anything other than the intended gameplay, it dies and resets to the initial state, so pure curiosity-based agents perform well [6]. In contrast, BASALT tasks require human feedback or data to identify and complete tasks.
Minecraft for Machine Learning Minecraft is as a valuable platform for machine learning research [19, 30, 40] due to its complex, dynamic environment and customizable game engine. As a result, numerous competitions and benchmarks [16, 18, 24, 37] have been developed to improve AI capabilities. These often include tasks with concrete or programmatic reward functions, such as learning to collect diamonds in a sample-efficient way [17], multi-agent learning of cooperative and competitive tasks [36], and more [12]. In contrast, we emphasize tasks that are designed to be hard to specify through a reward signal. Other work focuses on natural language as a specific modality for communicating intent [15, 27, 28, 34]; instead, we aim to promote the development of algorithms that generally learn from human feedback, including natural language. Some techniques require complete game state information (knowledge of blocks and items around them) [44, 46]; however, our benchmark only permits agents to access observations, not game state, to promote learning from pixels, which is more generalizable to other tasks.
Comparison to MineDojo Perhaps the most similar benchmark to ours is MineDojo [12], which emerged during the BASALT competitions and contributed a large dataset of Minecraft gameplay with the aim of developing generally-capable embodied agents. The focus of MineDojo was to provide a massive dataset scraped from the internet; in contrast, we focus on curating a smaller set of high-quality demonstrations and evaluations. Our demonstration data was produced by experienced Minecraft data collectors using a consistent Minecraft version and settings. In contrast, the MineDojo data, although plentiful, contains many videos with streamer overlays and non-Minecraft parts, different texture packs, mod packs, and more. Although this diversity may be beneficial in some settings, previous work had to filter out such data before it could be useful for training [2]. Another critical difference is evaluation: MineDojo provides only binary success or failure criterion; instead, our recommended evaluation and resulting Evaluation Dataset involves human judgments across a range of quantitative (e.g., Found Cave) and qualitative (e.g., Style Matching) criteria.
We proposed BEDD, a large and accessible dataset to facilitate algorithm development for the BASALT benchmark on learning from human feedback. The benchmark consists of five tasks in Minecraft, four of which lack a reward function. We clarified the benchmark, providing concrete evaluation recommendations and introducing another avenue for benchmarking: automated evaluations. Our dataset, BEDD, consists of two key parts: Demonstrations Dataset and Evaluation Dataset. Demonstrations Dataset contains almost 14,000 videos showing successful task completions of the four reward-free tasks. The Evaluation Dataset consists of over 3,000 human evaluations to support the development of automated evaluations of hard-to-specify tasks. To our knowledge, this dataset is the largest of its type: one that supports both training and evaluating LfHF agents on tasks with hard-to-specify reward functions.
We demonstrated the utility of our dataset and benchmark by presenting an analysis of the demonstrations and the several algorithms in the Evaluation Dataset. We showed that the algorithms exhibit varying degrees of performance on our tasks, as evaluated by human judges on a variety of factors. Our results suggest that there is ample room for improvement in learning from human feedback. With the inclusion of our accessible code for benchmarking and evaluating agents, we hope that our contributions will encourage the development of more effective approaches for both learning from human feedback and evaluating these techniques in the future.
Creating these datasets and this benchmark was only possible with the help of many people and organizations. FTX Future Fund Regranting Program, Microsoft, Encultured AI, and AI Journal provided financial support for the competition and resulting data. We thank Berkeley Existential Risk Initiative (BERI) for the support in organizing the BASALT competition. Karolis Ramanauskas was supported by the UKRI Centre for Doctoral Training in Accountable, Responsible and Transparent AI (ART-AI) [EP/S023437/1] and the University of Bath. We thank our amazing advisory board of the 2022 competition — Fei Fang, Kianté Brantley, Andrew Critch, Sam Devlin, and Oriol Vinyals — for their advice and guidance. We also thank any previous organizers or advisors of the previous competitions for their contributions. Finally, we thank AICrowd for their help hosting the competition and the MTurk workers for their work evaluating the submissions.
[1] D. Arumugam, J. K. Lee, S. Saskin, and M. L. Littman. Deep reinforcement learning from policy-dependent human feedback. arXiv preprint arXiv:1902.04257, 2019.
[2] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639-24654, 2022.
[3] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[4] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv:1606.01540, 2016.
[5] D. S. Brown, Y. Cui, and S. Niekum. Risk-aware active inverse reinforcement learning. In Conference on Robot Learning, pages 362-372. PMLR, 2018.
[6] Y. Burda, H Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv:1808.04355, 2018.
[7] V. Chen, N. Johnson, N. Topin, G. Plumb, and A. Talwalkar. Use-case-grounded simulations for explanation evaluation. Advances in neural information processing systems, 2022.
[8] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[9] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman. Leveraging procedural generation to benchmark reinforcement learning, 2019.
[10] S. Devlin, R. Georgescu, I. Momennejad, J. Rzepecki, E. Zuniga, G. Costello, Guy Leroy, A. Shaw, and K. Hofmann. Navigation turing test (ntt): Learning to evaluate human-like navigation. In International Conference on Machine Learning, pages 2644–2653. PMLR, 2021.
[11] Y. Duan, X. Chen, R. Houthooft, J. Schulman, and P. Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329-1338. PMLR, 2016.
[12] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D. A. Huang, Y. Zhu, and A. Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowledge. arXiv preprint arXiv:2206.08853, 2022.
[13] T. Gebru, J. Morgenstern, B. Vecchione, J. Wortman Vaughan, H. Wallach, H. Daumé III, and K. Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-92, 2021.
[14] A. Gleave, M. Taufeeque, J. Rocamonde, E. Jenner, S. H. Wang, S. Toyer, M. Ernestus, N. Belrose, S. Emmons, and S. Russell. imitation: Clean imitation learning implementations. 2022.
[33] S. Milani, A. Kanervisto, K. Ramanauskas, S. Schulhoff, B. Houghton, S. Mohanty, B. Galbraith, K. Chen, Y. Song, T. Zhou, et al. Towards solving fuzzy tasks with human feedback: A retrospective of the minerl basalt 2022 competition. arXiv preprint arXiv:2303.13512, 2023.
[34] A. Narayan-Chen, P. Jayannavar, and J. Hockenmaier. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405-5415, July 2019.
[35] A. Patterson, S. Neumann, M. White, and A. White. Empirical design in reinforcement learning. arXiv preprint arXiv:2304.01315, 2023.
[36] D. Perez-Liebana, K. Hofmann, S. P. Mohanty, N. Kuno, A. Kramer, S. Devlin, R. D. Gaina, and D. Ionita. The multi-agent reinforcement learning in Malmö (MARLO) competition. arXiv preprint arXiv:1901.08129, 2019.
[37] C. Salge, M. C. Green, R. Canaan, and J. Togelius. Generative design in mycrafth (gdmc) settlement generation competition. In Proceedings of the 13th International Conference on the Foundations of Digital Games, pages 1-10, 2018.
[38] R. Shah, S. H. Wang, C. Wild, S. Milani, A. Kanervisto, V. G. Goecks, N. Waytowich, D. Watkins-Valls, B. Prakash, E. Mills, et al. Retrospective on the 2021 minerl basalt competition on learning from human feedback. In NeurIPS 2021 Competitions and Demonstrations Track, pages 259-272. PMLR, 2022.
[39] R. Shah, C. Wild, S. H. Wang, N. Alex, B. Houghton, W. Guss, S. Mohanty, A. Kanervisto, S. Milani, N. Topin, et al. The minerl basalt competition on learning from human feedback. In The 35th Conference on Neural Information Processing Systems (NeurIPS) Competition Track, 2021.
[40] R. Smit and H. Smuts. Game-based learning-teaching artificial intelligence to play mimecraft: a systematic. Proceedings of Society, 93:188-202, 2023.
[41] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[42] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[43] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00690, 2018.
[44] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models. 2023.
[45] H.-N. Wu and M. Wang. Human-in-the-loop behavior modeling via an integral concurrent adaptive inverse reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[46] X. Zhu, Y. Chen, H. Tian, C. Tao, W. Su, C. Yang, G. Huang, B. Li, L. Lu, X. Wang, Y. Qiao, Z. Zhang, and J. Dai. Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. 2023.