Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper presents the « Behavior Suite for Reinforcement Learning » (bsuite), which is a set of RL tasks (called « experiments ») meant to evaluate an algorithm’s ability to solve various key challenges in RL. Importantly, these experiments are designed to run fast enough that one can benchmark a new algorithm within a reasonable amount of time (and money). They can thus be seen as a « test suite » for RL, limited to small toy problems but very useful to efficiently debug RL algorithms and get an overview of some of their key properties. The paper describes the motivation behind bsuite, shows detailed results from some classical RL algorithms on a couple of experiments, and gives a high-level overview of how the code is structured.

I really believe such a suite of RL tasks can indeed be extremely useful to RL researchers developing new algorithms, and as a result I would like to encourage this initiative and see it published at ICLR to help it gain additional traction within the RL community.

The paper is easy to read, motivates well the reasons behind bsuite, and shows some convincing examples. However, in my opinion there remain a few important issues with this submission:

1.	There is no « related work » section to position bsuite within the landscape of RL benchmarks (ex: DMLab, ALE / MinAtar, MuJoCo tasks, etc.). I believe it is important to add one.

2.	The current collection of experiments appears to be quite limited. The authors acknowledge the lack of hierarchical RL, but what about other aspects like continuous control, parameterized actions, multi-agent, state representation learning, continual learning, transfer learning, imitation learning / inverse RL, self-play, etc? It is unclear to me whether the goal is to grow bsuite in all these directions (and more) over time, or if there is some kind of « boundary » the authors have in mind regarding the scope of bsuite. Regardless, the fact is that in its current form, bsuite appears to be suited only to a limited subset of current RL research.

3.	I wish an anonymized version of the code had been provided, so that reviewers could test it. In particular I wonder (a) if it is easy to setup and run under Windows, and (b) if it is straighforward to plug a bsuite experiment within an algorithm based on the popular OpenAI gym API (I think the latter is true from what is said at the end of Section 4, but I would have appreciated being able to try it out myself).

Additional minor remarks:
•	I noticed two anoymity-related issues with the provided links: (1) the Google Colab notebook revealed to me the name of its author when clicking the « Open in Playground » link to be able to run it, and (2) the bsuite-tutorial link asks for permission, which might let the authors access reviewer info. I would not hold it against the authors though as I believe these are genuine mistakes and they did their best to preserve anonymity.
•	t > 2 in Section 2.1 should probably be t >= 2
•	In FIg. 2b the label for the y axis seems incorrect since good results are near 0
•	Please explain what is the dashed grey line in Fig. 4b
•	I was unable to understand the last 2 sentences of Section 4
•	Sections C.2, D.2 and E.2 all have the same plots
•	A few typos: incomplete sentence near bottom of p.3 (« the internal workings… »), « These assessment », « expeirments », « recurrant », « length ? 1 », « together with an analysis parses this data », « anonimize », « bsuite environments by implementing », « even if require »

Review update: the authors have addressed my concerns, and I look forward to using bsuite in my research => review score increased to "Accept"
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Behaviour Suite for Reinforcement Learning

In this paper the authors provide a set of light-weighted but dedicated designed environments, so that researchers can use the environments as a quick indication of the ability of the proposed (or existing) algorithms.
I think the paper is well-written, with the intuition clearly demonstrated.

I tend to vote for rejection though, given that the novelty in the project is relatively limited.
But I believe in general it is a very valuable project that will be beneficial to future research and I would like to recommend for a workshop publication.

Pros:
- The paper is well written, easy to understand. 
- Provide an industry level code base that can be used efficiently and easily.
The project will be of great value to the research community in the near future.

Cons:
- The novelty of the project is relatively limited. 
The proposed and implemented environments have been studied before.
- No explicit conclusion from the evaluation.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors propose a set of benchmarks for evaluating different aspects of reinforcement learning algorithms such as generalisation, exploration, and memory. The aim is to provide a set of simple environments to better understand the RL algorithms and also to provide a set of scores that summarise the performance in each respect. The code of the benchmark is also released.
 
The paper is well written and clear, and generally can provide a useful contribution. In particular, I like the idea of having a set of benchmarks which can be used for the diagnosis of RL algorithms. Having said this, I have the following concerns which are mostly related to the presentation of the paper. Given clarifications in an author response, I would be willing to increase the score. 

- Based on section 1.1 and elsewhere, it seems that the main driver for developing this benchmark has been connecting theory to practical algorithms (which in my opinion is an important step). However, how this can be achieved using the proposed benchmark is not shown in the paper. This can be for example showing how the generalisation score proposed here is linked to theoretical accounts. Or for example in section 2.1, by showing that the memory length 30 for RNN is related to the theoretical expectations. Alternatively, if linking theory and experiments is not the main driver of this work, then it seems a bit unclear what the point of presenting section 1.1 (and other related discussions) is within the context of the paper.

- In terms of novelty, currently the differences between the current work and previous attempts to develop benchmarks is unclear (some examples are mentioned below). In general, a related work section is vital here, but missing in the paper. It should clearly state what the previous attempts in developing benchmarks are, their shortcomings, and how the current work addresses them. 

- Some statements in the paper sound more like opinions (which I happen to agree with) rather than something being based on the results of the paper. For example, "We should not turn away from deep RL just because our current theory is not yet developed". It is unclear how this statement is related to the results obtained in this work.

- In section 3, I would like to see some real examples in which bsuite can be used for diagnosis. I find this application of bsuite (diagnosis) very interesting, but as it stands section 3 is more like a tutorial rather than providing a concrete example.

- There are some aspects of RL which are specific to certain classes of RL. For example, in model-based RL, aspects such as the dynamics bottleneck and the planning horizon dilemma have been previously looked at, but are not presented in bsuite. How do the authors envision incorporating such aspects into their framework?

Minor:
- "anything for length ¿ 1" -> replace ¿

- what is the dashed grey line in Fig 4b?

References:
Duan, Yan, et al. "Benchmarking deep reinforcement learning for continuous control." International Conference on Machine Learning. 2016.

Benchmarking Model-Based Reinforcement Learning, Wang et al, 2019.

review_title: Official Blind Review #3

Meta-review: This paper proposes a platform for benchmarking and evaluating reinforcement learning algorithms.  While reviewers had some concerns about whether such a tool was necessary given existing tools, reviewers who interacted with the tool found it easy to use and useful. Making such tools is often an engineering task and rarely aligned with typical research value systems, despite potentially acting as a public good. The success or failure of similar tools rely on community acceptance and it is my belief that this tool surpasses the bar to be promoted to the community at a top tier venue.  
Decision: Accept (Spotlight)