Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary: This paper proposes a Decentralized Distributed architecture for PPO. The idea was demonstrated on Habitat-Sim, the Habitat Autonomous Navigation Challenge 2019. The experiment shows how the implementation can achieve a speedup in training and near-linear scalability, over non-distributed and centralized architecture. 

Overall, the paper is well written and easy to follow. The idea is somewhat technical, e.g. distributed implementation of PPO and preemption for the rollouts of stragglers. The experimental results look promising but lack extensive evaluations given such a technical contribution. Therefore the paper has limited contributions. I have some following major comments.

1. As it is a technical paper, it would be nice if the authors could describe more on the implementation of challenges and the hacks.

2. The experiments could also be compared to one or two other distributed RL frameworks. It would also be helpful if detailed experiment settings are detailed, e.g. GPU characteristics, DDPPO's hyperparameters, etc.

3. The Transfer Learning tasks is a general setting for for any methods, which are not limited to only Distributed Approach. The results and setting there do not have links such as why and how distributed approaches help such transfer learning.


* minor comments:
- Eq.1: expectation should also be w.r.t transition's stochasticity.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper presents a novel scheme of distributing PPO reinforcement learning algorithm for hundreds of GPUs. Proposed technique was validated for pointgoal visual navigation task on recently introduced Habitat challenge and sim. 

Besides the technical contribution, paper shows that when have enough computational power of billions simulation runs, it is possible to learn nearly perfect visual navigation (given RGBD + GPS inputs) via reinforcement learning. 
Authors also study the task itself and show that it is yet not possible to achieve a good results without dense (each step) GPS signal, while the "Blind" agent, which has only GPS+compass error achieves quite high results given the billion-scale training time. 
This suggests that PointGoal navigation with dense GPS signal is might be a poor choice to benchmark RL algorithms and we should proceed to harder tasks.


Overall I like the paper a lot and think that it should be accepted.

*** 
I haven`t changed my mind after the rebuttal: the paper is good and should be accepted.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The contribution of this paper is twofold. First the authors propose and implement and new scalable RL training procedure they call 'Decentralized Distributed Proximal Policy Optimization' (DD-PPO). This method extends PPO in a way that is 'synchronous' and 'distributed': no parameter server exists to asynchronously collect experience from worker agents and instead an 'explicit communication state' exists during which all workers communicate gradient updates between one another to update parameters based on their experiences. However, naively implementing this approach is limited by 'stragglers', the slowest of the workers that all other agents need to wait for. To overcome this limitation, the authors add a 'preemption threshold' which halts worker rollouts after a certain percentage has completed. The authors use this procedure to tackle the task of PointGoal navigation, in which an agent tries to reach a point in space specified by its relative location to the agent. They achieve state-of-the-art performance (arguably 'mastery') on this task. Finally, the authors show that their learned policy transfers reasonably well to two other navigation tasks, 'flee' and 'exploration'. Code is also provided.

This is an impressive paper. The algorithmic contribution is useful and the results are state-of-the-art. The paper is written very clearly, and it was a pleasure to read. The investigation of the task of PointGoal Navigation, and the impact of various changes on the overall performance of the system . The transfer tasks, to the 'flee' and 'exploration' tasks, were interesting, and further reinforced the results in the paper.

My only high-level 'complaint' is that the title might be tempered: 'Mastering PointGoal Navigation' is a very strong claim. The results in the paper are very impressive, but 'Mastering', in my mind, implies (among other things) that the agent is guaranteed to reach its goal, though I understand that it is perhaps a minor point, considering the failure rate is systematically <3%. I also make a comment later about how the Matterport 3D environments may be a biased set, and that there are almost certainly environments that *would* make this system fail, for one reason or another. I am willing to accept the title as-is (and could be overwritten by other reviewers), but I would like some discussion on a more appropriate tile, if possible.

I recommend this paper for acceptance.

Minor comments:
- Introduction: "Thus, there is a need to develop a different distributed architecture." It is unclear how this sentence logically follows from the previous sentence. I am on board with the vision in general, and it is quite clear to me: having a single parameter server is limiting. However, that the move from CPU- to GPU-based agents is what reveals this limitation is not quite so clear. Reworking this paragraph (or perhaps adding a sentence) to make this clearer would be helpful.
- Introduction: "This means there is no scope for mistakes of any kind --- no wrong turn at a crossroad, ..." This is a fascinating point, though I feel reflects a bias in the types of environments that appear in the training data set. It is a defining behavior of intelligent embodied agents that they are capable of recovering from their mistakes; in general, real-world environments and navigation tasks have inherent ambiguity that cannot be resolved without exploration (a 'maze' is an extreme example of this). One question comes to mind: if the environments were ambiguous in this way, *could* the agent recover from its mistakes. There is not much evidence in the paper one way or the other. Since there are few examples in the data that seem to bring about such scenarios, it is not obvious how well this agent would perform in the face of this uncertainty. However, the results later in the paper about how "No GPS+Compass remains unsolved" is related to this point. A comment (or a figure in the Appendix) from the authors describing some of the failure cases would be instructive: do these failures occur in environments with such an ambiguity or, conversely, are there some examples in which the agent succeeds in reaching the goal but must overcome failure during its travel?
- The acronym 'SL' (for Supervised Learning) is not defined and is used only once. Prefer using 'supervised learning'.
review_title: Official Blind Review #2

Meta-review: The authors present and implement a synchronous, distributed RL called Decentralized Distributed Proximal Policy Optimization. The proposed technique was validated for pointgoal visual navigation task on recently introduced Habitat challenge 2019 and got the state of art performance.

Two reviews recommend this paper for acceptance with only some minor comments, such as revising the title. The Blind Review #2 has several major concerns about the implementation details. In the rebuttal, the authors provided the source code to make the results reproducible.

Overall, the paper is well written with promising experimental results. I also recommend it for acceptance.

Decision: Accept (Poster)