CrowdPlay: Crowdsourcing human demonstration data for offline learning in Atari games

Crowdsourcing has been instrumental for driving AI advances that rely on largescale data . At the same time reinforcement learning has seen rapid progress through benchmark environments that strike a balance between tractability and real-world complexity , such as ALE and OpenAI Gym . In this paper we aim to fill a gap at the intersection of these two : The use of crowdsourcing to generate large-scale human demonstration data in the support of advancing research into imitation learning and offline learning . To this end , we present CrowdPlay , a complete crowdsourcing pipeline for any standard RL environment including OpenAI Gym ( made available under an open-source license ) ; a large-scale publicly available crowdsourced dataset of human gameplay demonstrations in Atari 2600 games , including multimodal behavior and human-human and human-AI multiagent data ; offline learning benchmarks with extensive human data evaluation ; and a detailed study of incentives , including real-time feedback to drive high quality data . We hope that this will drive the improvement in design of algorithms that account for the complexity of human , behavioral data and thereby enable a step forward in direction of effective learning for real-world settings . 1 INTRODUCTION . Crowdsourcing has been instrumental in many AI advances , especially recent rapid progress in deep neural network models , which often rely on large training sets . For instance ImageNet ( Deng et al. , 2009 ) , a large database of annotated images , has enabled a number of breakthroughs in image classification ( Krizhevsky et al. , 2012 ) . At the same time , reinforcement learning ( RL ) has seen rapid progress in the last few years , fueled in part by the development of standard , easily accessible , benchmark environments like the Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ; Machado et al. , 2018 ) and OpenAI Gym ( Brockman et al. , 2016 ) . What has been underexplored is the intersection of the two : Using large-scale crowdsourced human data for offline learning , including imitation learning and offline RL . We present CrowdPlay , a framework , methodology , and dataset that we hope will do for offline learning what ALE and OpenAI Gym did for online learning . CrowdPlay supports flexible and scalable crowdsourcing that is geared towards multi-channel recruitment , and is able to interface with any OpenAI Gym or Gym-like Markov decision process ( MDP ) environment . It supports realtime feedback to participants that can be used to boost data quality , as well as both purely human and mixed human-AI multiagent environments . CrowdPlay is also the first dataset based on Atari 2600 games that features multimodal and multiagent behavior . It includes both data from normal gameplay as well as explicitly multimodal behavioral data , where players are given instructions to follow a specific behavior . In addition to single-agent data , the dataset includes data from two-player , human-AI and human-human games , and with both competitive and cooperative rewards . Participants were recruited through multiple channels ( under IRB , details withheld for blind submission ) including Amazon Mechanical Turk , Lab in the Wild , undergraduate students , and multiple social media channels . For some platforms we also include data with a range of different incentive structures for participants . The Atari games were run using ALE and a multiagent version of OpenAI Gym , guaranteeing that transitions are identical to what would be seen in standard Atari RL environments . In this paper we focus on the use of CrowdPlay for Atari 2600 games , but a major advantage of the approach is that it works for any Gym-like MDP environment . We believe that Atari 2600 games are interesting for imitation learning ( IL ) and offline RL for the same reasons that they were seminal as a challenge problem in the development of RL : they offer a balance between achievable short-term research advances and sufficient complexity . More recent work in psychology has also shown them to be of sufficient richness to support the study of human learning behavior ( Tsividis et al. , 2017 ) . Further , and despite the richness of the data , it is easy to collect at scale through the use of crowdsourcing and a web browser , and moreover , it can be used together with an established simulator for evaluation purposes . The CrowdPlay pipeline directly interfaces with standard RL environments . In particular this means that trajectories and transitions are guaranteed to be the same for human data as they are for RL environments ; that offline learning methods automatically have access to a simulator ; and that crowdsourcing with human players need not develop environments and tasks from scratch , but can make use of AI agents that can interact with human agents in a benchmark environment . 1.1 RELATED WORK . Crowdsourcing Previous platforms for crowdsourcing such as TurkServer ( Mao et al. , 2012 ) have focused on supporting synchronous participation through Amazon Mechanical Turk participants and used to study economic behavior in simple experimental environments ( and not for the generation of human behavioral data ) . Tylkin et al . ( 2021 ) use a combination of ALE and Javatari for crowdsourcing Atari data in the context of evaluating the performance of an AI agent in human-AI collaboration . They propose modifying two-player Space Invaders to make it cooperative , and to train AI agents using randomized starting position , both of which we adopt in ( some of ) our multiagent environments . Their crowdsourcing approach is Atari-specific and not publicly available . Much work has been done on the study of incentives for participants in paid crowdsourcing studies , and this is also part of the focus of our work . Prior work ( Mao et al. , 2013 ; Mason & Watts , 2009 ; Yin et al. , 2013 ; Harris , 2011 ; Shaw et al. , 2011 ) has largely found that quality-dependent payments may increase quantity of work more than quality of work , and has not looked at real-time feedback on work quality . Offline Learning . Much work has been done on offline learning ( Rashidinejad et al. , 2021 ) , including Behavior Cloning ( BC ) ( Pomerleau , 1989 ) , Batch Constrained Q-Learning ( BCQ ) ( Fujimoto et al. , 2019 ) , Conservative Q-Learning ( CQL ) ( Kumar et al. , 2020 ) , Implicit Quantile Network ( IQN ) ( Dabney et al. , 2018 ) , DQN ( Mnih , 2015 ) and Off-policy version of Soft ActorCritic ( Haarnoja et al. , 2018 ) . Aytar et al . ( 2018 ) demonstrate learning hard exploration games from unaligned human demonstration videos . Recent work ( Schrittwieser et al. , 2021 ) shows a sample-efficient model-based online and offline learning algorithm . Specific to Atari , Kanervisto et al . ( 2020 ) benchmark behavioral cloning algorithms on existing data from several video games , including Atari 2600 games . Laurens & Kazmi ( 2021 ) clone River Raid agents using existing datasets , and develop evaluation metrics based on action distribution and playstyle . Datasets Atari Grand Challenge ( AGC ) ( Kurin et al. , 2017 ) is a dataset consisting of 45 hours of standard gameplay from five Atari 2600 games . The authors also make their Atari-specific data collection software available . They use a browser app running the Atari emulator in the browser based on Javatari . It is unclear to us if this can be guaranteed to always have identical execution to the ALE emulator . Atari-Head ( Zhang et al. , 2020 ) features 117 hours of gameplay data and includes eye-tracking data . This data was collected using ALE . However , the emulator was run in a semi-frame-by-frame mode , advancing the emulator state only when a key was pressed , and at a maximum of 20 frames per second . The focus of the study was on attention tracking , and is not intended to be representative of natural human gameplay behavior . D4RL ( Fu et al. , 2020 ) and RL Unplugged ( Gulcehre et al. , 2020 ) both also provide datasets for offline learning , but both focus on synthetic data . 2 CROWDPLAY : THE PIPELINE . 2.1 OVERVIEW . The heart of our pipeline are the CrowdPlay backend and frontend , which is a client-server architecture that streams OpenAI Gym environments and similar MDP environments to web browser clients . It is highly extensible , scalable to hundreds or thousands of concurrent users , and allows the real-time capture of both trajectories as well as related statistics . It is geared toward multi-channel recruitment of participants and strong incentives . As its most important feature , it interfaces directly with OpenAI Gym and similar environments , thus opening up the entire array of standard RL environments to rapid crowdsourcing of behavioral data into the support of research into IL and offline RL . It also supports multi-agent environments , including mixed human-AI environments . Complementing this is an engine to support the local download of the generated dataset , including storage of metadata in a relational database for fast access , and compressed trajectories for storage efficiency . The data download can be real-time and is incremental . We give a short overview of the CrowdPlay architecture , and refer the reader to Appendix A.1 for more information . 2.2 SOFTWARE ARCHITECTURE . CrowdPlay provides a highly extensible , high performance client-server architecture for streaming MDP environments to remote web browser clients . The backend interfaces with OpenAI Gym and similar MDP environments . Actions are collected as keypresses in the browser client , and sent to the backend where they are fed into the MDP ’ s “ step ” function . The returned observation is sent back to the browser client for display . This is repeated to generate an episode trajectory . The remainder of the CrowdPlay software infrastructure is built to make this basic loop into a structure that is robust , performant , extensible , user-friendly and scalable . Figure 1 shows the key parts of the CrowdPlay architecture . Communication between the browser client and backend is through high-performance socket connections . The backend is built to be scalable both within-instance , using multiple processes , as well as across-instance using a load balancer and autoscaling instance groups . Trajectories are stored di- rectly as compressed , serialized Python objects , allowing both very easy modification of data capture as well as immediate decoding for use in existing Python-based learning pipelines . CrowdPlay also supports multi-agent environments . It allows multiple human participants by routing multiple browser clients to a single MDP environment . Mixed human-AI environments are supported through pre-trained neural network policies . For robustness , AI agents can also take over control of a human agent on the fly , in the case that a human player disconnects from a multiagent environment , allowing uninterrupted gameplay for the remaining human players . A major focus in the design of CrowdPlay is providing the ease of access of the generated data in downstream ML pipelines . We believe it is crucial for these pipelines to have access not only to the same simulator as the crowdsourcing pipeline , but also to the same observation pre-processing tools that are used in state-of-the-art RL methods on these simulators . Addressing these design goals , CrowdPlay includes a local metadata search engine and a custom , Deepmind-style ( Mnih et al. , 2015 ) observation processing function for offline data . We give more details in Appendix A.1 . CrowdPlay provides an extensible and easy-to-use framework for collecting structured metadata and real-time statistics per user , session , episode , and individual steps . This is used for capturing data quality information , as well as for driving real-time incentives for participants , depending on recruitment platform . We discuss the platforms that we target in more detail in Appendix A.3 , and the various incentive designs and their effect on data in Section 4 .

Summary: This paper presents CrowdPlay, a crowdsourcing platform to collect human demonstrations for any MDP. It also accompanies a dataset of human gameplay on Atari games with multi-agent and some multi-behavior aspects. The paper benchmarks existing offline RL algorithms on this dataset, and details incentive design mechanisms for future crowdsourcing jobs.