Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: (EDIT: Score raised from 4 to 5 after author rebuttal)

The authors propose and develop the MDP playground: a codebase which can generate random graphical POMDPs. The properties of the POMDPs can be controlled using tens of knobs that have been specifically selected to highlight particular challenges of reinforcement learning. They then demonstrate how the MDP playground can be used to generate insights about the performance of RL algorithms that can then be transferred to more complex settings.
strengths: The biggest strength of the MDP playground is that it enables quick iteration and feedback when designing RL algorithms: this is a niche that to my knowledge had not been filled before (although arguably bsuite may count, though it does still take longer to run than the MDP playground). The framework appears to be very flexible, while still being easy to use, and covers a wide range of RL use cases, which should give it broad appeal.
weaknesses: In my view the paper suffers from two major limitations that drive my low score. First, the paper is quite dense and unclear. I expand on this in the section on clarity, including some recommendations for better exposition which would allow the reader to better grasp the ideas in the paper.

Second, the utility of the MDP playground depends on the key claim that algorithm performance on the MDP playground is predictive of performance in more complex environments. However, there does not seem to be much evidence presented for this key claim: the evidence seems to be a single experiment which shows that the time unit is an important parameter in the MDP playground as well as in the HalfCheetah. More evidence needs to be presented to establish this key claim. Currently, I would not feel enthusiastic about using the MDP playground myself, because I don’t feel confident that the performance of an algorithm on the playground has much bearing on the performance of the algorithm in the setting I actually care about.

(In contrast, the other experiments presented on irrelevant features and multiple dimensions don’t seem very important to me, since the authors do not test whether the proposed “insight” generalizes to more complex environments.)

A more compelling experiment could look as follows:
1. Choose some complex environment (such as the HalfCheetah).
2. Determine which of the dimensions in the MDP playground could apply to that environment (there should be at least 3, and preferably 5+, for the experiment to be compelling).
3. Choose 2-3 RL algorithms.
4. Run the RL algorithm on the MDP playground to find how performance varies across the various dimensions. Do the same for the more complex environment (note this means you’ll have to modify the complex environment to allow the dimension to be varied). Check whether the results in the MDP playground are predictive of those in the more complex environment.
correctness: To my knowledge, the claims made are correct, apart from some minor nits mentioned below. However, as mentioned in the weaknesses, I am not convinced that the MDP playground is useful, though this may be more an issue of clarity.

Nits:

> An implicit assumption for many agents is that rewards are immediate depending on only the current information state and action.

This does not seem correct to me. It is well known that one of the main challenges in RL is credit assignment, and algorithm design takes this challenge into account. The entire point of credit assignment is that rewards are not immediate.

> This was done exhaustively to allow as many dimensions as possible

To claim that this was done exhaustively you should have some argument that there are no other dimensions that you have failed to find. This seems very hard to argue (and probably not true).

It is of course true that the MDP playground has identified a very large number of dimensions, and you should emphasize that -- it is only the word “exhaustively” that I take issue with.
clarity: I found the paper dense and hard to understand; the authors have crammed in a lot of content that is explained very briefly without clear examples and figures. For example, it took me until halfway through the paper to understand that these are what we might call “graphical” (PO)MDPs, rather than (say) gridworlds.

Some recommendations to improve clarity:
1. Choose half of the content from the paper, explain that half very well, and move the remaining half into appendices (or cut it entirely). See subsequent sections for how to do so.
2. Rather than explaining all ten dimensions with a paragraph each, start with a list of the ten dimensions with just one sentence explaining what that dimension is. Then choose 3-5 dimensions to explain in more depth. Include any dimensions that you intend to reference in your experiments (e.g. the time unit dimension).
3. Include a figure of a graphical (PO)MDP that might be generated by the playground. For each dimension that you explain in depth, explain with reference to specific states, actions and transitions in the figure, what the dimension corresponds to and how it can be controlled in your framework. An example of this type of exposition can be found in [1].
4. Merge Sections 2 and 3, so that you only explain the details of each dimension once. Once I got to Section 3, I had already forgotten what had been said about that dimension in Section 2 (because I had read about ten other dimensions in between!)
5. This is less related to the other points, but please show learning curves for agents rather than just AUC numbers (for example in Figures 1 and 2). These should take roughly the same amount of space, as you can plot multiple learning curves on the same graph.

[1] Turner, Alexander Matt, et al. "Optimal Policies Tend to Seek Power." arXiv preprint arXiv:1912.01683 (2019).
relation to prior work: Yes. Bsuite is the obvious benchmark to compare to, and the authors discuss it in detail. I might recommend that the authors emphasize insights that couldn’t have been gained from bsuite (for example, I believe the insight about the time unit being important could not have come from bsuite).
additional feedback: N/A
review_title: Interesting benchmark for an important niche, but not clear that insights generalize to complex environments

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a set of tasks for reinforcement learning with minimal complexity for quickly designing and debugging new algorithms. Different variations of the tasks are introduced to disentangle factors that affect an agent's learning performance. The paper shows a few examples of the transfer of the high-level insights drawn from the proposed tasks to the more complex tasks.
strengths: The paper proposes a rich set of variations of the RL tasks that can significantly affect the performance of an RL system but are often overlooked since they have been tuned by experts in standard RL benchmarks. This will bring more attention to research in this area to make RL systems more robust to such hyperparameters. Additionally, I find the explanation of the design decisions in Appendix H inspiring.
weaknesses: My major concern is, to what degree can the insights from the simple environments transfer to the more complex environments? The examples in section 4 are not convincing to me. For the image transformation example, what is the message of Figure 1a-c? Is the same observation that image transformation has a different effect on DQN and Rainbow observed in more complex environments? The example of varying time units seems obvious as a larger time unit means less control over the agent which will eventually hurt performance. An important question is how are the default values set for different dimensions of the tasks? For the high-level insights drawn from the experiments, are they consistent across other dimensions? For example, when evaluating the effect of the time unit, how much does the value of the diameter of the MDP affect the conclusion? If the high-level insights are consistent across different sets of default values, it becomes more convincing that they will also transfer to more complex environments.
correctness: See the weakness seciton.
clarity: Discussion of the related work is clearly explained. A few parts of the experiments could be explained more clearly: Apart from the figure 1a-c mentioned above, it is also not clear to me what is the message of figure 2a-b.
relation to prior work: Yes.
additional feedback: No additional feedback.
review_title: A good testbed for RL but more justification is needed

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: - Presents MDP playground -- a tool-kit/ platform  to study various design details of RL agents  
- Outlines dimensions along with the RL agents can be investigated. Provides clean and easy to instantiate abstractions.
- Provides exhaustive studies investigating these dimensions, drawing insights and highlighting take-aways.
strengths: - An exhaustive study 
- Clearly outlines and motivates each dimensions along with RL agents should be investigates
- Detailed set of experiments in the Appendix 
- Includes both continuous and discrete domains
- Clearly positions itself wrt to prior works
- I appreciate authors being clear about their computational and carbon footprint
weaknesses: - Claims at certain places are over-read and over-generalized -- for example its quite unlikely that the intuitions derived in Section4 using the toy envs will generalize more broadly. Intuitions derived here can be used as guiding principles (and I believe this is what authors wanted to convey). I'd recommend authors to revise the language and presentation to avoid mis-communication 
- There is a growing concern in the RL community that we have overgrown past environments in OpenAIGym and dmControl. They are too simple and narrowly defined. Authors here however refer to 'HalfCheetah' as complex environment. Some recalibration of expectation is needed. This is especially critical since this work is posed as a "benchmark", if successful, will be used study and facilitate future developments. An uncalibrated benchmark has the potential to steer the field in the wrong direction.
correctness: - Paper is easy to read, well motivated in its design choices and thorough in its experiments. 
clarity: - easy to read and follow 
- Appendix is too exhaustive with 
relation to prior work: - Section 5 clearly positions itself wrt prior work in terms of scope and problem space
additional feedback: - Extending the analysis to a few complex domains [1] can be really valuable. 

[1] @INPROCEEDINGS{Rajeswaran-RSS-18,
    AUTHOR    = {Aravind Rajeswaran AND Vikash Kumar AND Abhishek Gupta AND
                 Giulia Vezzani AND John Schulman AND Emanuel Todorov AND Sergey Levine},
    TITLE     = "{Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations}",
    BOOKTITLE = {Proceedings of Robotics: Science and Systems (RSS)},
    YEAR      = {2018},
}
[2]
review_title: A comprehensive work providing tools for scrutinize Reinforcement Learning

Meta-review: In this paper, the authors propose a new framework for testing RL algorithms. In particular, the framework offers knobs for independently varying many aspects that make RL tasks difficult (e.g. density of rewards or stochasticity).  Reviewer feedback centered on (a) clarity and (b) significance.  The authors are of course strongly encouraged to continue to improve the clarity of their paper, and to make sure claims are appropriately precise and contextualized.  With respect to significance, a consistent concern was whether progress on the synthetic tasks would generalize to progress on more realistic tasks.  The authors are encouraged to either add more clear evidence that the tasks are good benchmarks of progress and/or to make clearer that the dominant purpose of the playground is for understanding RL algorithms (e.g. debugging and controlled experiments) rather than benchmarking.  I'd encourage the authors to revise and resubmit.
Decision: Reject