Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Reinforcement learning (RL) training speed is broadly evaluated on two dimensions:  sample efficiency (the number of environment interactions required) and wall-clock time.  Improved wall-clock training time has been achieved through distributed actors and learners, but often at the expense of sample efficiency.  IMPACT repurposes successful concepts from deep RL - the target network, importance sampling and a replay buffer to demonstrate improvements on both axes in on three continuous environments and three games from the Atari Learning Environment.

Positives
This was a well-written paper proposing to address the sample efficiency of distributed RL algorithms.  The diagrams of the algorithm were also well-done.  Improving the sample efficiency of algorithms is an important objective and the approaches followed here are sensible.

Additionally, the ablations and examination of the sensitive hyperparameters of the algorithm are useful analyses.  These indicate relative insensitivity to the target network update frequency, but both the importance sampling equation and the circular buffer hyperparameters are described.


Negatives
IMPACT introduces additional hyperparameters which are tuned for each continuous control task and discrete control task.  However, there is no description of the hyperparameter tuning budget allocated to IMPACT, PPO, IMPALA.  

Regarding the discrete environment, the game selection should be elaborated upon and if sufficient compute is available, the algorithm should be tested elsewhere.  Specifically, it is atypical (though not necessarily incorrect) to tune specific hyperparameters for each game in the Atari Learning Environment.  Traditionally, algorithms have been justified as robust and useful by the lack of need to tune per game.  Table 4 demonstrates a high degree of tuning for IMPACT due to game-specific changes for clip param, grad clip, lambda, num sgd iter, train_batch_size, value function loss coeff, kl coeff.  However, Table 5 (IMPALA) and Table 6 (PPO) have fewer noted changes.

Small nits: 
- Define advantage in the policy gradient equation
- Figure 4 is ahead of Figure 3 in the compiled LaTeX


Questions
- How were the discrete control games selected?
- What was the hyperparameter tuning budget for IMPACT versus PPO or IMPALA?
- If a fixed hyperparameter budget is allocated in advance and new environments are randomly selected, does IMPACT favorably compare to IMPALA and PPO?
- IMPALA performs remarkably badly in the three continuous control tasks, even on wall-clock time.  What validations have been done here to ensure the algorithm is operating as intended?

I will increase my rating if the robustness and improvements of this algorithm can be validated in randomly chosen games/continuous control environments for a fixed hyperparameter budget for IMPACT and both baselines.  Also, the IMPALA baseline should be validated for the continuous control tasks - it's surprising that this once SOTA-algorithm flounders in even simple tasks like Hopper-v2 or HalfCheetah-v2.


-----------------
Update:
The authors have addressed my initial concerns carefully through extra experiments and details in the Appendix and I have updated my rating accordingly.  Thanks!

review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new distributed algorithm for reinforcement learning. The paper lists three main contributions: a target network for stabilizing the surrogate objective, a circular buffer, and truncated importance sampling. 

I'm not that familiar with RL, however I'm very familiar with distributed training in other contexts. Therefore, the significance of the contributions in the RL domain is a bit unclear to me. However, the contributions in the area of distributed training is relatively fair. The introduction of a circular buffer is not very novel. Further, the trade-offs / adaption of update frequency etc. are standard ways to improve performance in distributed training. 

The evaluation of the proposed algorithm is reasonably well done (considering the page limits), with a suitable set of benchmarks (although relatively few). The results are promising and could have a significance for practitioners.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper introduces IMPACT which is a distributed RL algorithm that shortens training time of RL systems while maintaining/ improving the sample efficiency. It is built on top of the famous PPO algorithm (https://arxiv.org/abs/1707.06347). The authors break down the novel component of their model into three categories: target network, circular buffer, and importance sampling. They evaluate the effectiveness of each component through different experiments. 

Overall the paper is well-written and the ideas are communicated clearly. I like how the evaluation is done in different environments (discrete and continuous action-space) and improves the results independent of the task settings.  

One question: you mentioned in section 4.3: "For fairness, same network hyperparameters were used across PPO, IMPALA, and IMPACT." I suppose it would be a fair comparison if you choose the hyperparameters for each algorithm separately ( according to the highest value they achieve on the measured metric.) How did you end up choosing the hyperparameters for your own experiments? Are they fine-tuned for IMPACT?

It seems like IMPACT is not always doing better than PPO in the discrete control domain as shown in Figure 6. Specifically, in part (a) it looks like PPO is beating both IMPALA and IMPACT for BreakoutNoFrameskip and PongNoFrameskip. It would be nice if authors could do an analysis of these cases and add a discussion as to why this is happening.

Overall I think this is an interesting paper which can motivate more work in this area.


------------------------------------------------------------------------------------------------------------------------------------------------
Updates:
I would like to thank the authors for their response. I have read the revised version and it looks good to me.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper studies a novel way for distributed RL training which combines the data reuse of PPO with the asynchronous updates of IMPALA. The main contribution is the observation that using a target network is necessary for achieving stable learning. I think this is an important result which seems to be validated by another ICLR submission (https://openreview.net/forum?id=SylOlp4FvH). The experimental section could definitely be improved--I was hoping to see more results on Atari or DMLab.

Two comments:
The V-trace equations (page 3 and 5) don't mention any clipping of the importance weights c and rho--can you clarify if this is a typo or if you don't use clipping?
It would be great to see experiments showing how learning curves scale with the number of workers.

-----------------------------------------------------------------------------------
Thanks for clarifying and for the extra experiments. I'm keeping my score as I still think it's appropriate.
review_title: Official Blind Review #3

Meta-review: The authors propose a novel distributed reinforcement learning algorithm that includes 3 new components: a target network for the policy for stability, a circular buffer, and truncated importance sampling. The authors demonstrate that this improves performance while decreasing wall clock training time.

Initially, reviewers were concerned about the fairness of hyper parameter tuning, the baseline implementation of algorithms, and the limited set of experiments done on the Atari games. After the author response, reviewers were satisfied with all 3 of those issues.

I may have missed it, but I did not see that code was being released with this paper. I think it would greatly increase the impact of the paper at the authors release source code, so I strongly encourage them to do so.

Generally, all the reviewers were in consensus that this is an interesting paper and I recommend acceptance.
Decision: Accept (Poster)