Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents CrowdPlay, a crowdsourcing platform to collect human demonstrations for any MDP. It also accompanies a dataset of human gameplay on Atari games with multi-agent and some multi-behavior aspects. The paper benchmarks existing offline RL algorithms on this dataset, and details incentive design mechanisms for future crowdsourcing jobs.
main_review: Learning from human labelled datasets is a tried and tested paradigm to make progress on hard learning problems. We’ve seen this play out in CV and NLP for long, and although some RL work does rely on human demonstration data, collecting these datasets is often an ad-hoc process relying on designing custom, specific pipelines for each problem. This paper introduces a platform that makes it seamless to crowdsource human demonstrations for any underlying simulator. The platform also provides an easy way to post process the collected data for learning experiments. 

This is an important and refreshing piece of work: better research tools and infrastructure are leverage multipliers on productivity of downstream research work. The specific area: a pipeline for collecting labelled offline RL datasets also seems under-served. 

I like the design choices the paper makes in building this platform and they seem to have been well thought. In particular, not requiring the simulator / emulator to be run in the browser is a big plus, which means the platform can serve a wide range of simulators via the client / server architecture. I also appreciate the authors providing an anonymous staging for the platform, and that the entire platform will be open-source. 

I do have some questions for clarification: 
  * Does CrowdPlay enable researchers to recruit participants directly from the platform? It doesn’t seem like it (since the data collected itself was from several channels), but if it does offer an option it would be nice to elaborate on what steps CrowdPlay takes to ensure players are fairly compensated. 
  * The extend of the multi-modal dataset seems quite small, limited to 2 games. Are there plans to expand the dataset? 
  * Have you bencharked the latency of the architecture (time taken to do one keypress->send-to-server->env.step()->send obs to client loop)? I would imagine high latency would disincentive participants from completing the demos. 

Finally, I believe you should these two references which are related to your work: [1] shows that even unaligned YouTube videos can be leveraged for playing hard exploration games in Atari. [2] is the state-of-the-art offline RL algorithm on Atari games. 

[1] Aytar, Y., Pfaff, T., Budden, D., Paine, T. L., Wang, Z., & de Freitas, N. (2018). Playing hard exploration games by watching YouTube. NeurIPS 2018.

[2]  Schrittwieser, J., Hubert, T., Mandhane, A., Barekatain, M., Antonoglou, I., & Silver, D. (2021). Online and offline reinforcement learning by planning with a learned model. NeurIPS 2021
summary_of_the_review: This paper presents an important and under-looked infrastructure contribution: a platform to make collecting human data easy for any existing simulators. The platform makes use of good decision decisions. The accompanying Atari dataset could be useful for useful for muli-agent and multi-modal RL research as well. Therefore I recommend a strong acceptance for this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a tool to crowdsource data collection using a client-server architecture running OpenAI gym on the backend and providing a web interface for human players to use. Along with that, the authors release a large dataset of six Atari games, including multimodal and multiplayer data.
main_review: ### Pros

- The paper introduces a useful tool for the community that can impact the development of the field.
- The dataset is pretty big.
- The paper provides a lot of useful information about the data collection process itself.

### Cons

The paper focuses too much on the details of data collection and the tool itself rather than providing insights about the data itself and justifying why this particular dataset can be useful to the community. In this form, this is the paper about how to effectively collect a dataset rather than a dataset itself. To make the paper impactful, I encourage the authors to focus on two points:

- What is so special about the data? What properties does it have? Can you provide any insights into the data?
- What are the challenges that your dataset poses to the community? Why should we focus on Atari? Why are the existing datasets not enough to develop strong offline RL algorithms? Section 5 of the paper should be expanded to address these questions.

Other comments/questions:

- "CrowdPlay" is also the first dataset based on Atari 2600 games that features multimodal and multiagent behaviour." Please, define multimodality a bit more formally. Atari Grand Challenge dataset you cite also has multimodal behaviour because different people might play in a different way.
- While your tool is more general, Atari Grand Challenge you present in related work also released a tool for collecting the data for the Atari games, not the dataset alone. Please, reflect this in the related work section.
- page 6: "including data such as playtime, score, and various in-game behaviour characteristics". Can you, please, specify what these various characteristics are?
- "Deepmind-style" observation processing function has to be defined more formally. Which paper it was first used in? Please, add a reference.
- A lot of software engineering/data processing details can safely go to the appendix, whereas dataset information that is currently in the appendix should be in the main body of the paper.
- Figure 3 uses the term "good data" that uses the term "task adherence". Can you, please, explain what task adherence is?
- "Our framework provides a variety of data compositions that exhibit real-world intricacies and diversities". What do you precisely mean by this? What real-world characteristics can we find in a dataset of Atari screenshots?
- "allows easily accessible crowdsourcing for any existing RL environments". Not all RL environments are gym environments.
- Can you explain the reasons behind multiplayer data? Why do we need such a dataset?
- The paper has to have hyperparameters for the experiments in Section 5. Saying 'default parameter values from d3rlpy' is not enough since the values might change over time (providing commit hash will be much better), or the repo might get removed.
summary_of_the_review: The paper introduces a useful tool that can be impactful for the community (Imitation Learning, Offline RL), however, I believe that it is not ready for publication due to the presentation and lack of dataset analysis. I encourage the authors to focus on the following two points:

- Describing the dataset in more detail providing the insights about the data and their properties.
- Explaining why the dataset is challenging to the existing methods, and how it can drive the further development of the fields of imitation learning and offline RL.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel framework CrowdPlay for crowdsourcing human data based on standard RL environments. This CrowdPlay pipeline not only supports recruiting different users from different channels to collect multimodal behaviors and data but also designs diverse and real-time incentive mechanisms to guarantee and improve the quality of data. Furthermore, the authors present a dataset, which is publicly available, along with benchmarks on Atari 2600 games, to enable further research on Imitation Learning and Offline Learning.
main_review: This paper proposes a novel framework CrowdPlay for crowdsourcing human data based on standard RL environments. The presentation of this paper is easy to follow and its content is technically sound. The design and technical details of the software structure are well disclosed and reproducible. Furthermore, the dataset provided by this paper is conducive to the research on Reinforcement Learning.

There are several issues that could be further addressed:
- Since some of the datasets provided in this paper are imbalanced, will the performance of the AI agents trained by these data be affected by the imbalance? Furthermore, more details of the AI agent training procedure should be demonstrated. E.g., how are the agents trained, and what are the differences between the standard and cooperative variants?
- As shown in Figure 3, the authors also compared the data from social media users and email raffle. What’s the incentive mechanism used on social media users?
- There are some minor issues on misspelling, e.g. "what ALE and Gym what ALE and AI Gym" (in Page. 1), “(in both Space Invaders and Riverraid,” (missing a character, in Page. 5).

summary_of_the_review: Overall, this is a valuable piece of work. The proposed framework CrowdPlay, including the software and the dataset, is interesting and attractive for researchers in the areas of Reinforcement Learning. It would be better if the above issues could be further addressed to make it smoother.
review_title: null

Meta-review: This paper studies the problem of how to collect demonstrations via crowd sourcing for imitation and offline learning. The paper received mixed reviews initially. The reviewers had difficulty understanding empirical results, asked for some more ablations, and were little unconvinced by the proposed usefulness of the collected data. The authors provided a strong thoughtful rebuttal that addressed many of those concerns. The paper was discussed extensively with one of the reviews who increased their score from 3 to 5. Reviewers generally agree that the paper is good but not all reviewers are on-board with acceptance. AC recommends accept but agrees with the reviewers and the authors are urged to look at reviewers' feedback and incorporate their comments in the camera-ready.
Decision: Accept (Poster)