Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper presents a scalable reinforcement learning training architecture which combines a number of modern engineering advances to address the inefficiencies of prior methods. The proposed architecture shows good performance on a wide variety of benchmarks from ALE to DeepMind Lab and Google Research Football. Important to the community, authors also open source their code and provide an estimate which shows that the proposed framework is cheaper to run on cloud platforms.

Pros:
1. This work is solid from the engineering perspective. It effectively addresses the problems with prior architectures and the accompanying source code is clear and well structured. It is also extensively tested on several RL benchmarks.

2. The proposed framework is especially suited for training large models as the model parameters are not transferred between actors and learners.

3. The paper is well written and organized.

Cons:

1. The gain of the main algorithmic improvement (SEED architecture) over the baseline (IMPALA architecture) is obscured by the usage of different hardware. TPUv3 has different characteristics than Nvidia P100/V100 GPU chips which also might contribute to the speed up.

Questions:

1. Is it possible to provide more “apple-to-apple” comparison by running SEED and IMPALA on the same hardware (TPUv3 or Nvidia P100/V100 GPU)? 
review_title: Official Blind Review #5

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new reinforcement learning agent architecture which is significantly faster and way less costly than previously distributed architectures. To this end, the paper proposes a new architecture that utilizes modern accelerators more efficiently.  The paper reads very well and the experimental results indeed demonstrate improvement. Nevertheless, even though working in deep learning for years and have also some experience with Reinforcement learning I am not in the position to provide an expert judgment on the novelty of the work. I do not know if ICLR is the right place of the paper (I would probably suggest a system architectures conference for better assessment of the work).
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper presents SEED RL, which is a scalable reinforcement learning agent. The approach restructure the interface / division of functionality between the actors (environments) and the learner as compared to the distributed approach in IMPALA (a state-of-the-art distributed RL framework). Most importantly, the model is only in the learner in SEED while it is distributed in IMPALA. 

The architectural change from to IMPALA to SEED feels reasonable, and the results support the choices in a positive way.

SEED is evaluated using a large number of benchmarks using three environments, and the performance is compared to IMPALA. The results are very good, shows good scalability, and significantly reduced training times.  

The paper is well written, easy to read, and I enjoyed it. 

The code for SEED is released open source, which enables future research to build upon SEED. 

review_title: Official Blind Review #2

Meta-review: The paper presents a framework for scalable Deep-RL on really large-scale architecture, which addresses several problems on multi-machine training of such systems with many actors and learners running.  Large-scale experiments and impovements over IMPALA are presented, leading to new SOTA results. The reviewers are very positive over this work, and I think this is an important contribution to the overall learning / RL community.
Decision: Accept (Talk)