CrowdPlay: Crowdsourcing human demonstration data for offline learning in Atari games

Crowdsourcing has been instrumental for driving AI advances that rely on largescale data . At the same time reinforcement learning has seen rapid progress through benchmark environments that strike a balance between tractability and real-world complexity , such as ALE and OpenAI Gym . In this paper we aim to fill a gap at the intersection of these two : The use of crowdsourcing to generate large-scale human demonstration data in the support of advancing research into imitation learning and offline learning . To this end , we present CrowdPlay , a complete crowdsourcing pipeline for any standard RL environment including OpenAI Gym ( made available under an open-source license ) ; a large-scale publicly available crowdsourced dataset of human gameplay demonstrations in Atari 2600 games , including multimodal behavior and human-human and human-AI multiagent data ; offline learning benchmarks with extensive human data evaluation ; and a detailed study of incentives , including real-time feedback to drive high quality data . We hope that this will drive the improvement in design of algorithms that account for the complexity of human , behavioral data and thereby enable a step forward in direction of effective learning for real-world settings . 1 INTRODUCTION . Crowdsourcing has been instrumental in many AI advances , especially recent rapid progress in deep neural network models , which often rely on large training sets . For instance ImageNet ( Deng et al. , 2009 ) , a large database of annotated images , has enabled a number of breakthroughs in image classification ( Krizhevsky et al. , 2012 ) . At the same time , reinforcement learning ( RL ) has seen rapid progress in the last few years , fueled in part by the development of standard , easily accessible , benchmark environments like the Arcade Learning Environment ( ALE ) ( Bellemare et al. , 2013 ; Machado et al. , 2018 ) and OpenAI Gym ( Brockman et al. , 2016 ) . What has been underexplored is the intersection of the two : Using large-scale crowdsourced human data for offline learning , including imitation learning and offline RL . We present CrowdPlay , a framework , methodology , and dataset that we hope will do for offline learning what ALE and OpenAI Gym did for online learning . CrowdPlay supports flexible and scalable crowdsourcing that is geared towards multi-channel recruitment , and is able to interface with any OpenAI Gym or Gym-like Markov decision process ( MDP ) environment . It supports realtime feedback to participants that can be used to boost data quality , as well as both purely human and mixed human-AI multiagent environments . CrowdPlay is also the first dataset based on Atari 2600 games that features multimodal and multiagent behavior . It includes both data from normal gameplay as well as explicitly multimodal behavioral data , where players are given instructions to follow a specific behavior . In addition to single-agent data , the dataset includes data from two-player , human-AI and human-human games , and with both competitive and cooperative rewards . Participants were recruited through multiple channels ( under IRB , details withheld for blind submission ) including Amazon Mechanical Turk , Lab in the Wild , undergraduate students , and multiple social media channels . For some platforms we also include data with a range of different incentive structures for participants . The Atari games were run using ALE and a multiagent version of OpenAI Gym , guaranteeing that transitions are identical to what would be seen in standard Atari RL environments . In this paper we focus on the use of CrowdPlay for Atari 2600 games , but a major advantage of the approach is that it works for any Gym-like MDP environment . We believe that Atari 2600 games are interesting for imitation learning ( IL ) and offline RL for the same reasons that they were seminal as a challenge problem in the development of RL : they offer a balance between achievable short-term research advances and sufficient complexity . More recent work in psychology has also shown them to be of sufficient richness to support the study of human learning behavior ( Tsividis et al. , 2017 ) . Further , and despite the richness of the data , it is easy to collect at scale through the use of crowdsourcing and a web browser , and moreover , it can be used together with an established simulator for evaluation purposes . The CrowdPlay pipeline directly interfaces with standard RL environments . In particular this means that trajectories and transitions are guaranteed to be the same for human data as they are for RL environments ; that offline learning methods automatically have access to a simulator ; and that crowdsourcing with human players need not develop environments and tasks from scratch , but can make use of AI agents that can interact with human agents in a benchmark environment . 1.1 RELATED WORK . Crowdsourcing Previous platforms for crowdsourcing such as TurkServer ( Mao et al. , 2012 ) have focused on supporting synchronous participation through Amazon Mechanical Turk participants and used to study economic behavior in simple experimental environments ( and not for the generation of human behavioral data ) . Tylkin et al . ( 2021 ) use a combination of ALE and Javatari for crowdsourcing Atari data in the context of evaluating the performance of an AI agent in human-AI collaboration . They propose modifying two-player Space Invaders to make it cooperative , and to train AI agents using randomized starting position , both of which we adopt in ( some of ) our multiagent environments . Their crowdsourcing approach is Atari-specific and not publicly available . Much work has been done on the study of incentives for participants in paid crowdsourcing studies , and this is also part of the focus of our work . Prior work ( Mao et al. , 2013 ; Mason & Watts , 2009 ; Yin et al. , 2013 ; Harris , 2011 ; Shaw et al. , 2011 ) has largely found that quality-dependent payments may increase quantity of work more than quality of work , and has not looked at real-time feedback on work quality . Offline Learning . Much work has been done on offline learning ( Rashidinejad et al. , 2021 ) , including Behavior Cloning ( BC ) ( Pomerleau , 1989 ) , Batch Constrained Q-Learning ( BCQ ) ( Fujimoto et al. , 2019 ) , Conservative Q-Learning ( CQL ) ( Kumar et al. , 2020 ) , Implicit Quantile Network ( IQN ) ( Dabney et al. , 2018 ) , DQN ( Mnih , 2015 ) and Off-policy version of Soft ActorCritic ( Haarnoja et al. , 2018 ) . Aytar et al . ( 2018 ) demonstrate learning hard exploration games from unaligned human demonstration videos . Recent work ( Schrittwieser et al. , 2021 ) shows a sample-efficient model-based online and offline learning algorithm . Specific to Atari , Kanervisto et al . ( 2020 ) benchmark behavioral cloning algorithms on existing data from several video games , including Atari 2600 games . Laurens & Kazmi ( 2021 ) clone River Raid agents using existing datasets , and develop evaluation metrics based on action distribution and playstyle . Datasets Atari Grand Challenge ( AGC ) ( Kurin et al. , 2017 ) is a dataset consisting of 45 hours of standard gameplay from five Atari 2600 games . The authors also make their Atari-specific data collection software available . They use a browser app running the Atari emulator in the browser based on Javatari . It is unclear to us if this can be guaranteed to always have identical execution to the ALE emulator . Atari-Head ( Zhang et al. , 2020 ) features 117 hours of gameplay data and includes eye-tracking data . This data was collected using ALE . However , the emulator was run in a semi-frame-by-frame mode , advancing the emulator state only when a key was pressed , and at a maximum of 20 frames per second . The focus of the study was on attention tracking , and is not intended to be representative of natural human gameplay behavior . D4RL ( Fu et al. , 2020 ) and RL Unplugged ( Gulcehre et al. , 2020 ) both also provide datasets for offline learning , but both focus on synthetic data . 2 CROWDPLAY : THE PIPELINE . 2.1 OVERVIEW . The heart of our pipeline are the CrowdPlay backend and frontend , which is a client-server architecture that streams OpenAI Gym environments and similar MDP environments to web browser clients . It is highly extensible , scalable to hundreds or thousands of concurrent users , and allows the real-time capture of both trajectories as well as related statistics . It is geared toward multi-channel recruitment of participants and strong incentives . As its most important feature , it interfaces directly with OpenAI Gym and similar environments , thus opening up the entire array of standard RL environments to rapid crowdsourcing of behavioral data into the support of research into IL and offline RL . It also supports multi-agent environments , including mixed human-AI environments . Complementing this is an engine to support the local download of the generated dataset , including storage of metadata in a relational database for fast access , and compressed trajectories for storage efficiency . The data download can be real-time and is incremental . We give a short overview of the CrowdPlay architecture , and refer the reader to Appendix A.1 for more information . 2.2 SOFTWARE ARCHITECTURE . CrowdPlay provides a highly extensible , high performance client-server architecture for streaming MDP environments to remote web browser clients . The backend interfaces with OpenAI Gym and similar MDP environments . Actions are collected as keypresses in the browser client , and sent to the backend where they are fed into the MDP ’ s “ step ” function . The returned observation is sent back to the browser client for display . This is repeated to generate an episode trajectory . The remainder of the CrowdPlay software infrastructure is built to make this basic loop into a structure that is robust , performant , extensible , user-friendly and scalable . Figure 1 shows the key parts of the CrowdPlay architecture . Communication between the browser client and backend is through high-performance socket connections . The backend is built to be scalable both within-instance , using multiple processes , as well as across-instance using a load balancer and autoscaling instance groups . Trajectories are stored di- rectly as compressed , serialized Python objects , allowing both very easy modification of data capture as well as immediate decoding for use in existing Python-based learning pipelines . CrowdPlay also supports multi-agent environments . It allows multiple human participants by routing multiple browser clients to a single MDP environment . Mixed human-AI environments are supported through pre-trained neural network policies . For robustness , AI agents can also take over control of a human agent on the fly , in the case that a human player disconnects from a multiagent environment , allowing uninterrupted gameplay for the remaining human players . A major focus in the design of CrowdPlay is providing the ease of access of the generated data in downstream ML pipelines . We believe it is crucial for these pipelines to have access not only to the same simulator as the crowdsourcing pipeline , but also to the same observation pre-processing tools that are used in state-of-the-art RL methods on these simulators . Addressing these design goals , CrowdPlay includes a local metadata search engine and a custom , Deepmind-style ( Mnih et al. , 2015 ) observation processing function for offline data . We give more details in Appendix A.1 . CrowdPlay provides an extensible and easy-to-use framework for collecting structured metadata and real-time statistics per user , session , episode , and individual steps . This is used for capturing data quality information , as well as for driving real-time incentives for participants , depending on recruitment platform . We discuss the platforms that we target in more detail in Appendix A.3 , and the various incentive designs and their effect on data in Section 4 .

Summary: This paper proposes a novel framework CrowdPlay for crowdsourcing human data based on standard RL environments. This CrowdPlay pipeline not only supports recruiting different users from different channels to collect multimodal behaviors and data but also designs diverse and real-time incentive mechanisms to guarantee and improve the quality of data. Furthermore, the authors present a dataset, which is publicly available, along with benchmarks on Atari 2600 games, to enable further research on Imitation Learning and Offline Learning.