Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper provides a description of a new framework for reproducible and efficient RL experiments, as well as benchmarks of many algorithms on popular environments, such as `Atari and Roboschool. 

Pros:
- I agree that reproducibility is an extremely important question for the RL research, and thus such a code library is very beneficial for the community.
- The library is well designed, and allows for creating extensions rather easily in the future.
- Benchmarks are quite extensive and instructive.

Cons:
-  Comparison with the library [1] is missing (see also [2] for description and benchmarks). As both libraries are focused on reproducibility and flexible implementations of algorithms, such a comparison would support authors claims.
- I am not sure that ICLR is the right venue for such paper. Perhaps a more specialized conference of a workshop would be better.
- Anonymity violation

Questions:
- How difficult it is to implement distributional algorithms in your framework?
- What about different exploration strategies? (Boltzmann, epsilon-greedy, parameter noise etc.). I guess it should be quite easy to make it configurable as well


[1] https://github.com/catalyst-team/catalyst
[2] https://arxiv.org/pdf/1903.00027.pdf

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper presents a new RL library called « SLM Lab ». Its most relevant features for RL research are: (1) modularity to help re-use existing components (thus reducing the risk of subtle implementation differences when comparing algorithms), (2) implementations of most popular algorithms like DQN & variants, A3C, PPO, SAC, (3) ability to parallelize both actors (through vectorized environments) and the learner (through distributed gradient descent), and (4) utilities for hyper-parameter optimization, reproducible experiments and reporting. The paper also reports performance over Atari games, Roboschool environments as well as some Unity ML-Agents tasks. Finally, it provides a high-level overview of SLM Lab’s capabilities compared to 23 other RL open source libraries, showing that it is the only one that combines: reporting of the performance of the implemented algorithms, ability to specify hyper-parameters in the config file, parallelization, hyper-parameter optimization, and visualization of the results.

Overall this looks like a solid RL library, but I am not convinced that it brings enough novelty to the RL software landscape for a published ICLR paper — it would better fit in a workhop dedicated to ML libraries for instance, thus the weak reject.

Table 3 shows that SLM Lab is the only RL library with such a broad offering of features, and this is definitely impressive, but I would argue that many of these features can typically be added to other libraries by plugging in other open source software. For instance there are several tools for experiment management and hyper-parameter optimization (and for RLLib in particular, hyper-parameter optimization is not checked but is straightforward with Ray Tune). TensorBoard can also often be easily used for visualization.

The parallelization capabilities of SLM Lab also seem limited: if I understand correctly, actor parallelization can only occur on a single machine, and thus an algorithm like Ape-X or R2D2 could not be implemented. If this is correct then it is a major limitation of the framework, since such parallelization across multiple computers can be extremely useful when environments are slow and costly to run (in CPU / RAM).

It is not clear to me to which extent multi-agent is supported. It seems like it is possible to have multiple agents in one environment, but is that enough for general multi-agent RL? (ex: how to specify individual / team rewards? share information between agents? deal with agents not acting all at the same timestep? centralize part of the training / execution?…)

I appreciate that the paper presents a discrete version of SAC, mentioning how easy it was to implement thanks to the modular design of SLM Lab, but results from Table 1 do not look very good (especially since it did not work on some of the environments). Relying on the Gumbel-softmax might not be the most robust & stable way to train a discrete SAC — see e.g. the recent « Soft Actor-Critic for Discrete Action Settings » for a different approach.

Finally, it is also great to have some benchmarks of the algorithms being implemented, but at least for Atari, I am not aware of previous work using the exact same evaluation setting, so it is hard to tell how they compare to other implementations.

In spite of the above, I do not mean to criticize SLM Lab too heavily as from what I can tell it seems to be a a solid library with many useful features, and I am sure many researchers will find it useful in their day-to-day work.

Minor points:
- Anonymity was clearly violated with the two github links
- Having some synthetic result on Atari (like the typically reported median human-normalized score) would be good
- I am personally not a fan of large config JSON files due to the lack of comments in JSON
- A.5 (« Key Implementation Lessons ») is great!

Review update after author feedback: I am on the fence for this paper, but still leaning towards rejection due to the fact I am still not convinced that this library brings that much novelty compared to existing other libraries (although it seems like a nice RL library, I am not sure ICLR is the right venue for talking about it). The authors argue that their benchmark results are a key contribution of the paper, but I do not find these results particularly insightful, especially the Atari ones that are not comparable to previous results (due to using a different evaluation method) and the lack of state-of-the-art algorithms like Rainbow or IQN.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: SLM Lab is a software framework for reinforcement learning, which includes many different algorithms, networks, and memory types. The framework is well structured and modular. Thus, it is easily extendable for anyone and can be a pinnacle for future RL research. 

The really like the paper. It is well written, easy to read, and provide a valuable platform / framework to the community, both the scientific community as well as practitioners. Although the scientific contribution may be low in the paper, I think the significance and potential impact of the paper outweigh that. 

The paper also include many results from running the framework in various configurations, showing the flexibility and usefulness of it.

The code for SLM Lab is released open source, which is very valuable and enables future research in RL. 

review_title: Official Blind Review #2

Meta-review: A new software framework fo Deep RL is introduced. This is a useful work for the community, but it is not a research work. I agree with Reviewer4 that somehow it is not a right venue: other papers need to have technical contributions, SOTA, and here - it is difficult but it is another type of work - accurate technical implementation and commenting. I do not feel right to have as it a paper on ICLR. 
Decision: Reject