DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames
Authors: Erik Wijmans
Abhishek Kadian
Ari Morcos
Stefan Lee
Irfan Essa
Devi Parikh
Manolis Savva
Dhruv Batra
Venue:  (2020)
Keywords: autonomous navigation
habitat
embodied AI
pointgoal navigation
reinforcement learning

Abstract: We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in resource-intensive simulated environments. DD-PPO is distributed (uses multiple machines), decentralized (lacks a centralized server), and synchronous (no computation is ever "stale"), making it conceptually simple and easy to implement. In our experiments on training virtual robots to navigate in Habitat-Sim, DD-PPO exhibits near-linear scaling -- achieving a speedup of 107x on 128 GPUs over a serial implementation. We leverage this scaling to train an agent for 2.5 Billion steps of experience (the equivalent of 80 years of human experience) -- over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs. 

This massive-scale training not only sets the state of art on Habitat Autonomous Navigation Challenge 2019, but essentially "solves" the task -- near-perfect autonomous navigation in an unseen environment without access to a map, directly from an RGB-D camera and a GPS+Compass sensor.  Fortuitously, error vs computation exhibits a power-law-like distribution; thus, 90% of peak performance is obtained relatively early (at 100 million steps) and relatively cheaply (under 1 day with 8 GPUs). Finally, we show that the scene understanding and navigation policies learned can be transferred to other navigation tasks -- the analog of "ImageNet pre-training + task-specific fine-tuning" for embodied AI. Our model outperforms ImageNet pre-trained CNNs on these transfer tasks and can serve as a universal resource (all models and code are publicly available). 

Published as a conference paper at ICLR 2020
DD-PPO: LEARNING NEAR-PERFECT POINTGOAL NAVIGATORS FROM 2.5 BILLION FRAMES
Erik Wijmans Abhishek Kadian² Ari Morcos Stefan Leet.3 Irfan Essa
Devi Parikh :2 Manolis Savva Dhruv Batra' 1.2
Georrga Institute of Technology FFacebook AI Research Orreonn State University "Simon Fraser University
ABSTRACT
We present Decentralized Distributed Proximal Policy Optimization (DD-PPO), a method for distributed reinforcement learning in essource-intense simulated en- vironments. DD-PPO is distributed (uses multiple machines), decentralized (lacks a centralized server), and synchronous (no computation is ever 'stale"), making it conceptually simple and easy to implement. In our experiments on training virtual robots to navigate in Habitat-Sim (Savva et al., 2019), DD-PPO exhibits neur-linear scaling achieving a speedup of 107x on 128 GPUs over a serial im- plementation. We leverage this scaling to train an agent for 2.5 Billion steps experience (the equivalent of 80 years of human experience) over 6 months of GPU-time training in under 3 days of wall-clock time with 64 GPUs.
This massive-scale training not only sets the state of art on Habitat Autonomous Navigation Challenge 2019, but essentially 'solves' the task near-perfect au- tonomous navigation in an wnseen environment without access to a map, directly from an RGB-D camera and a GPS+Compass sensor. Fortuitously, error vs com- putation exhibits a power-law-lie like distribution; thus, 90% of peak performance is obtained relatively early (at 100 million steps) and relatively cheaply (under day with 8 GPUs). Finally, we show that the scene understanding and navigation poli- cies learned can be transferred to other navigation tasks - the analog of 'ImageNet pre-training + task-specific fine-tuning' for embodied AI Our model outperforms ImageNet pre-trained CNNs on these transfer tasks and can serve as a universal resource (all models and code are publicly available).
Code: ht https://gi thub. com/ f Facebookreseea wwwwereeeaaacaaaatteeeeeeeeeeeee tat-api youtube. wwtp:ootwtwwowwoeeoooooooooeeeoe weto///atch/veeee 5PB V5i1v4
Video:
INTRODUCTION
Recent advances deep reinforcement learning (RL) have given rise systems that can outperform human experts at variety of games (Silver et al., 2017: Tian et al., 2019; OpenAI, 2018). These advances, even more-so than those from supervised learning, rely on significant numbers of training samples, making them impractical without large-scale, distributed parallelization. Thus, scaling RI. via multi-node distribution is of importance to AI that is the focus of this work.
Several works have proposed systems for distributed RL (Heess et al., 2017; Liang et al., 2018 8a; Tian et al., 2019; Silver et al., 2016; OpenAI, 2018: Espeholt al., 2018). These works utilize two core components: 1) workers that collect experience ('rollout workers') () and 2) a parameter server that optimizes the model. The rollout workers are then distributed across, potentially, thousands of CPUs' However, synchronizing thousands of workers introduces significant overhead (the parame- ter server must wait for the slowest worker, which can be costly as the number of workers grows). To combat this, they wait for only few rollout workers, and then asynchronously optimize the model.
However, this paradigm of single parameter server and thousands of (typically CPU) workers appears to be fundamentally incompatible with the needs of modern computer vision and robotics communities. Over the last few years, large number of works have proposed training virtual robots (or 'embodied agents') )) in rich 3D simulators before transferring the learned skills to reality (Beattic
Work done while an intern at Facebook Al Research, Correspondence to e etw@gatech. edu. Environments in OpenAl Gym (Brockman al., 2016) and Atari games be simulated on solely CPUs.

---

Published as a conference paper at ICLR 2020
Performance on PointGoal Navigation
1.0
GPI (higher is better)
0.9
0.8
0.7
0.6
Train
0.5
Val
0.4
10'
10"
10" 2.5 10"
Steps (experience; log-scale)
Depth (D)
GPSHCOMDOSS
10
100 160
GPU (days; log-scale)
3.0
Wall clock time (days: log-scale)
Figure I: Left: In PointGoal Navigation, an agent must navigate from a random starting location (blue) to a target location (red) specified relative the agent ("Go 5m north, 10m east of you") in previously unseen environment withoul access to a map. Right: Performance (SPL; higher is better) of an agent equipped with RGB-D and Compass sensors on the Habitat Challenge 2019 (Savva et al., 2019) train & val sets. Using DD-PPO, train agents for over 180 days of GPU-time in under days of wall-clock time with 64 GPUs, achieving statc-of-art results and 'solving' the task.
et al., 2016; Chaplot et al., 2017: Das al., 2018; Gordon et al., 2018; Anderson et al., 2018b; Wij- mans et al., 2019; Savva et al., 2019). Unlike Gym or Atari, 3D simulators require GPU acceleration, and, consequently, the number of workers is greatly limited (23 w vs. 2 15). The desired agents operate from high dimensional inputs (pixels) and, consequentially. use deep networks (ResNet50) that strain the parameter server. Thus, there is need develop new distributed architecture
Contributions. We propose a simple, synchronous, distributed RL method that scales well. We call this method Decentralized Distributed Proximal Policy Optimization (DD-PPO) as it is decentralized (has no parameter server), distributed (runs across many different machines), and we use it to scale Proximal Policy Optimization (Schulman et al., 2017).
In DD-PPO, each worker alternates between collecting experience in resource-intensive and GPU accelerated simulated environment and optimizing the model. This distribution is synchronous there is an explicit communication stage where workers synchronize their updates the model (the gradients). To avoid delays due to stragglers, we propose a preemption threshold where the experience collection of stragglers is forced to end early once pre-specified percentage of the other workers finish collecting experience. All workers then begin optimizing the model.
We characterize the scaling of DD-PPO by the steps of experience per second with N workers rela- tive to worker. We consider two different workloads, 1) simulation time is roughly equivalent for all environments, and 2) simulation time can vary dramatically due large differences in environ- ment complexity. Under both workloads, we find that DD-PPÓ scales near-linearly. While we only examined our method with PPO, other on-policy RL algorithms can easily be used and believe the method is general enough to be adapted to off--pliiy RL algorithms.
We leverage these large-scale engineering contributions answer a key scientific question aris- ing in embodied navigation. Mishkin et al. (2019) benchmarked classical (mapping + planning) and learning- bearning-based methods for agents with RGB-D and GPS+Compass sensors PointGoal Nay- igation (Anderson et al., 2018a) (PointGoalNav), see Fig. 1, and showed that classical methods outperform learning-based However, they trained for 'only" 5 million steps of experience. Savva et al. (2019) then scaled this training to 75 million steps and found that this trend reverses - learning- based outperforms classical, even in unseen environments! However, even with an order of magni- tude more experience (75M vs 5M), they found that learning had not yet saturated. This begs the question what are the fundamental limits of learnability in PointGoalNav? Is this task entirely leamaale?? We answer this question affirmatively via an "existence proof".
Utilizing DD-PPO, find that agents continue improve for long time (Fig. 1) not only setting the state of art in Habitat Autonomous Navigation Challenge 2019 (Savva et al., 2019), but essen- tially 'solving' PointGoalNav (for agents with GPS+Compass). Specifically, these agents 1) almost
2

---

Published as a conference paper at ICLR 2020
Asynchronous
Synchronous via Distributed Data Parallel
Rollout worker
Rollout
worker
Exp. Collection Optimization
Exp. Collection Optimization
Gradient
Rollout
Rollout
Gradient
Gradient
Parameters
Parameters
Exp. Collection Optimization
Gradient
Exp. Collection Optimiization
Parameter Server
GPU
CPU
Asynchronous Communcation
Synchronous Communcation
Process
Process
Comparison of asynchronous distribution (left) and synchronous distribution via dis- tributed data parallelism (right) for RL. Left: rollout workers collect experience and asynchronously send it to the parameter-server. Right: worker alternates between collecting experience, synchro- nizing gradients, and optimization. We find this highly effective in resource-intensive environments.
Figure 2:
always reach the goal (failing on 1/1000 val episodes on average), and 2) reach it nearly as effi- ciently possible nearly matching (within 3% of) the performance of shortest-path oracle! It worth stressing how uncompromising that comparison is in a new environment, an agent nav- igating without a map traverses path nearly matching the shortest path on the map. This means there is no scope for mistakes of any kind no wrong turn at crossroad, no back-tracking from a dead-end, exploration or deviation of any kind from the shortest-path. Our hypothesis is that the model learns to exploit the statistical regularities in the floor-plans of indoor environments (apart- ments, offices) in datasets. The more challenging task of navigating purely from an RGB camera without GPS+Compass demonstrates progress but remains an open frontier.
Finally, we show that the scene understanding and navigation policies learned on PointGoalNav can be transferred to other tasks (Flee and Explore (Gordon et al., 2019)) - the analog of 'ImageNet pre-training task-specific fine-tuning' for Embodied AL. Our models are able to rapidly learn these new tasks (outperforming ImageNet pre- pre-trained CNNs) and can be utilized as near-perfect neural PointGoal controllers, universal resource for other high-level navigation tasks (Anderson et al., 2018;; Das et al., 2018). We make code and trained models publicly available.
2 PRELIMINARIES: RL AND PPO
Reinforcement learning (RL) is concerned with decision making in Markov decision processes. In partially observable MDP (POMDP), the agent receives observation that does not fully specify the state (st) of the environment, og (e.g. an egocentric RGB image), takes an action a, and given reward rt- The objective is to maximize cumulative reward over an episode, Formally, let be sequence of (or, (t, rt) where a₂ ~ .(. or), and 8 ~ T(s,, at). For a discount factor y, which balances the trade-off between exploration and exploitation, the optimal policy, n², is specified by
T
π²" argmax [Rr], where, R =
rt.
(1)
1=1
One technique to find n" Proximal Policy Optimization (PPO) (Schulman et al., 2017), an on-policy algorithm in the policy-gradic family. Given a 6-parameterized policy нæ and set of trajectories collected with it (commonly referred to as a 'rollout'), PPO updates Ty as follows. Let
A, = R₁ - V,, be the estimate of the advantage, where R2 7 and V. is the expected value of RG, and r0) mela, loz) be the ratio of the probability of the action at under the cur- тo, (at loe)
rent policy and the policy used to collect the rollout. The parameters are then updated by maximizing
clip(r.(0), - c)A, ]
JPPO = Fr min
re (
iuportance-weighee advantage
proximity clipping
This clipped objective keeps this ratio within e and functions as trust-region optimization method; allowing for the multiple gradient updates using the rollout, thereby improving sample efficiency.
3

---

Published as a conference paper at ICLR 2020
CNN
CNN
CNN
CNN
CNN
Pollcy
>
h
Policy
Policy
D
Pollcy
V
Polley +
<forward>
<left>
<stop>
Figure 3: Our agent for PointGoalNav. At very time-step, the agent receives an egocentric Depth or RGB (shown here) observation, utilizes its GPS+Compass sensor to update the target position to be relative to its current position, and outputs the next action and an estimate of the value function.
3 DECENTRALIZED DISTRIBUTED PROXIMAL POLICY OPTIMIZATION
In reinforcement learning, the dominant paradigm for distribution is asynchronous (see Fig. 2)- Asynchronous distribution is notoriously difficult even minor errors can result in opaque crashes and the parameter server and rollout workers necessitate separate programs.
In supervised learning, however, synchronous distributed training via data parallelism (Hillis & Steele Jr, 1986) dominates. As general abstraction, this method implements the following: at step k, worker n has a copy of the parameters, 0%,- calculates the gradient, 00% and updates via 12
0 ++1 ParamUpdate (0t: AalRReduce(00 00%) Par (ot
1 N
(3)
where Par aapamate is any first-order optimization technique (e.g. gradient descent) and AllReduce performs a reduction (e.g. mean) over all copies of variable and returns the result to all work- ers. Distributed DataParallel scales very well (near-linear scaling up to 32,000 GPUs (Kurth et al., 2018)), and is reasonably simple to implement (all workers synchronously running identical code).
We adapt this to on-policy RI. as follows: At step k; a worker n has a copy of the parameters 0% gathers experience (rollout) using Toy, calculates the parameter-gradient Vo via any policy-gradient method (e.g. PPO), synchronizes these gradients with other workers, and updates the model:
g Par amupdaaee ( 0tt, AllReduce [VOTPPO ..., (P)Po VeJ OKk))
A key challenge to using this method in RI. is variability in experience collection run-time. In super- vised learning, all gradient computations take approximately the same time. In RL, some resource- intensive environments can take significantly longer to simulate. This introduces significant syn- chronization overhead as every worker must wait for the slowest to finish collecting experience. To combat this, we introduce preemption threshold where the rollout collection stage of these strag- glers is preempted (forced to end early) once some percentage, p%. (we find 60% to work well) of the other workers are finished collecting their rollout; thereby dramatically improving scaling, We weigh all worker's contributions to the loss equally and limit the minimum number of steps before preemption to one-fourth the maximum to ensure all environments contribute to learning.
While we only examined our method with PPO, other on-policy RI algorithms can easily be used and we believe the method can be adapted to off -ff-cy RL algorithms. Off-policy RL algorithms also alternate between experience collection and optimization, but differ in how experience is col- lected/used and the parameter update rule. Our adaptations simply add synchronization the opti- mization stage and a preemption to the experience collection stage.
Implementation. We leverage PyTorch's (Paszke al., 2017) iistribuuteDaatPPaalllee to syn- chronize gradients, and TCPStore a simple distributed key-value storage to track how many workers have finished collecting experience. See Apx. E for detailed description with code.
EXPERIMENTAL SETUP: POINTGOAL NAVIGATION, AGENTS, SIMULATOR
PointGoal Navigation (Nain An agent is initialized at a random starting position and orientation in new environment and asked navigate to target coordinates specified relative the
4

---

Published as a conference paper at ICLR 2020
agents position; no map is available and the agent must navigate using only its sensors in our case RGB-D (or RGB) and GPS+Compass (providing current position and orientation relative to start).
The evaluation criteria for an episode is as follows (Anderson et al., 8a): Let S indicate "success' (did the agent stop within 0.2 meters of the target?), be the length of the shortest path between start and target, and be the length of the agent's 's path, then Success weighted by (normalized inverse) Path Length SPL. S max(1.p) It worth stressing that SPL is highly punitive metric- - to achieve SPL = 1, the agent (navigating without the map) must match the performance of the shortest-path oracle that has access to the map! There is scope for any mistake no back-tracking from a dead-end, no exploration or deviation from the shortest path. In general, this may not even be possible in new environment (certainly not if an adversary designs the map).
wrong turn at crossroad,
Agent. As in Savva al. (2019), the agent has 4 actions, stop, which indicates the agent has reached the goal, move_forward (0.25m), turn_left (10"), and turn_right (109). It receives 256x256 sized images and uses the GPS+Compass to compute target coordinates relative to its current state. The RGB-D agent is limited to only Depth as Savva et al. (2019) found this to perform best.
Our agent architecture (Fig. 3) has two main components a visual encoder and a policy network.
The visual encoder is based on either ResNet (He et al., 2016) or SE (Hu et al., (018))RRsNeel (Xie et al., 2017) with the number of output channels at every layer reduced by half. We use first layer of 2x2-AvgPool to reduce resolution (cssentially performing low-pass filtering + down-sampling) we find this to have no impact on performance while allowing faster training. From our initial experiments, we found necessary to replace every BatchNorm layer (loffe & Szegedy, 2015) with GroupNorm (Wu & He, 2018) account for highly correlated inputs seen in on-policy RL.
The policy is parameterized by a 2-layer LSTM with a 112-dimensional hidden state. It takes three inputs: the previous action, the target relative to the current state, and the output of the visual encoder. The LSTM's output is used to produce a softmax distribution over the action space and an estimate of the value function. See Appendix C for full details.
Training. We use PPO with Generalized Advantage Estimation (Schulman et al., 2015). We set the discount factor ry to 0.99 and the GAF parameter 7 to 0.95. Each worker collects (up to) 128 frames of experience from 4 agents running in parallel (all in different environments) and then performs epochs of PPO with 2 mini-batches per epoch. We use Adam (Kingma & Ba, 2014) with a learning rate of 2.5 10-4. Unlike popular implementations of PPO, we do not normalize advantages as we find this leads to instabilities. We use DD-PPO to train with 64 workers on 64 GPUs.
The agent receives terminal reward FT 2.5 SPL, and shaped reward (a st) = A geoodist 0.01, where Agoo dist is the change in geodesic distance to the goal by performing action a, in state 8t-
Simulator+Datasees: Our experiments are conducted using Habitat, 3D simulation platform for embodied Al research (Savva et al., 2019). Habitat is modular framework with a highly performant and stable simulator, making it an ideal framework for simulating billions of steps of experience.
We experiment with several different sources of data. First, we utilize the training data released as part of the Habitat Challenge 2019, consisting of 72 scenes from the Gibson dataset (Xia et al., 2018). We then augment this with all 90 scenes in the Matterport3D dataset (Chang et al., 2017) to create a larger training set (note that Matterport3D meshes tend to be larger and of better quality). 2 Furthermore, Savva et al. (2019) curated the Gibson dataset by rating every mesh reconstruction on a quality scale of 0 to 5 and then filtered all splits such that each only contains scenes with a rating of 4 or above (Gibson-4+) leaving all scenes with lower rating previously unexplored. We examine training on the 332 scenes from the original train split with a rating of 2 or above (Gibsson2++).
BENCHMARKING: HOW DOES DD-PPO SCALE?
In this section, we examine how DD-PPO scales under two different workload regimes - homoge- neous (every environment takes approximately the same amount of time to simulate) and heteroge- neous (different environments can take orders of magnitude more/less time to simulate). We examine the number of steps of experience per second with N workers relative to worker. We compare dif- ferent values of the preemption threshold p%. We benchmark training our ResNet50 PointGoalNar agent with Depth on a cluster with Nvidia V100 GPUs and NCCL.2.4.7 7 with Infiniband interconnect.
We use all Matterporr3D scenes (including test and val) as only evaluate Gibson validation and test.
5

---

Published as a conference paper at ICLR 2020
Homogeneous Scaling
Heterogeneous Scaling
250
Relative Scaling
200
40%
150
60%
80%
100
100%
50
Ideal
0
50
100
150
200
250
50
100
150
200
Number of GPUs
Number of GPUs
Figure 4: Scaling performance (in steps of experience per second relative I GPU) of DD-PPO for various preemption threshold, p%, values. Shading represents 95% confidence interval.
Homogeneous. To create a homogeneous workload, train on scenes from the Gibson dataset, which require very similar times to simulate agent steps. As shown in Fig. 4 (left), DD-PPO exhibits near-linear scaling (linear = idcal) for preemption thresholds larger than 50%, achieving a 196x speed up with 256 GPUs relative to GPU and an 7.3x speed up with 8 GPUs relative to 1.
Heterogeneous, To create heterogeneous workload, train scenes from both Gibson and Matterport3D. Unlike Gibson, MP3D scenes vary significantly in complexity and time to simulate the largest contains 8GB of data while the smallest is only 135MB. DD-PPO scales poorly preemption threshold of 100% (no preemption) due to the substantial straggler effect (one rollout taking substantially longer than the others); see Fig. 4 (right). However, with preemption threshold of 80% or 60%, we achieve near-identinn degradation performance of models trained with any of these values for the preemption threshold despite learning in large scenes occurring at lower frequency.
scaling to the homogeneous workload! We found no
6 MASTERING POINTGOAL NAVIGATION WITH GPS+COMPASS
In this section, we answer the following questions: 1) What are the fundamental limits of learnability in PointGoalNav navigation? 2) Do more training scenes improve performance? 3) Do better visual encoders improve performance? 4) Is PointGoa GoalNav 'solvable' when navigating from RGB instead of Depth? 5) What are the open/unsolved problems specifically, how does navigation without GPS+Compass perform? 6) Can agents trained for PointGoalNav be transferred to new tasks?
Agents continue to improve for long time. Using DD-PPO, train agents for 2.5 Billion steps of experience with 64 Tesla V100 GPUs in 2.75 days 180 GPU-days of training, the equivalent of 80 years of human experience (assuming human second per step). As comparison, Savva et al. (2019) reached 75 million steps (an order of magnitude more than prior work) in 2.5 days using 2 GPUs at that rate, it would take them over a month (wall-clock time) achieve the scale of study, Fig. shows the performance of an agent with RGB-D and GPS+Compass sensors, utilizing an Se-5esNeXt50 visual encoder, trained on Gibson-22 2+ it does not saturate before 1 billion steps", suggesting that previous studies were incomplete by 1-2 orders of magnitude. Fortuitously, error vs computation exhibits a power-law-like distribution; 90% of peak performance obtained relatively early (100M steps) and relatively cheaply (in 0.1 day with 64 GPUs and in 1 day with 8 GPUs*). Also the strong generalization (train to val) and corresponding lack of overfitting.
noteworthy in Fig.
Increasing training data helps. Tab. presents results with different training datasets and visual encoders for agent with RGB-D and GPS+Compass. Our most basic setting (ResNet50, Gibson-4+ training) already achieves SPL of 0.922 (val), 0.917 (test), which nearly misses (by 0.003) the top of the leaderboard for the Habitat Challenge 2019 RGB-D track". Next, we increase the size of the training data by adding in all Matterport3D scenes and see an improvement of ~0003 SPL to 0.956 (val), 0.941 (test). Next, we compare training on Gibson-4+ and Gibson-2+: Recall that Gibson 3} corresponds to poorly reconstructed scenes (see Fig. 11). A priori, it is unclear whether the net effect of this addition would be positive or negative; adding them provides diverse experience to the
sensors (RGB), training datasets (Gibson-44). and visual encoders. "The current on-demand price of an 8-GPU AWS instance (p2.8xlarge) is $7. 2/rr, or $172.8 for 1 day. chttps://eevllle cloudev. www.ddc.orgwwb//cceeeo /aggdd/.orrg/eeb/baaaeeee//////e chal paaeeeaaaeee255 /eaa////aaeeeer///33333
'These trends consistent

---

Published as a conference paper at ICLR 2020
Validation
Test Standard
Training Datasct
Agent Visual Encoder
SPL
Success
SPL
Success
Gibson-4+
ResNet50
0.922 + 0.004
0.967 + 0.003
0.917
0,970
Gibson-4+ and MP3D
RosNet50
0.956 + 0.002
0.996 # 0.002
0.941
0,996
Gibson-2+
ResNet50
0.956 + 0.003
0,994 + 0.002
0.944
0,982
SE-ResNeXi50
0.959 0.002
0,999 + 0.001
0,943
0,988
SE-ResNcXcI01 1024-d LSTM
0.969 0.002
0.997 + 0.001
0.948
0.980
Table I: Performance (higher is better) of different architectures for agents with RGB-D and GPS+Compass sensors on the Habitat Challenge 2019 (Savva et al., 2019) validation and test-std splits (checkpoint selected on val). 10 samples taken for each episode on val. Gibson-4+ (2+) refers to the subset of Gibson train scenes (Xia et al., 2018) with a quality rating of 4 (2) or higher. See Tab. for results of the best DD-PPO agent for Blind, RGB, and RGB-D and other baselines.
agent, however, it is poor quality data. We find a potentially counter-intuitive result - adding poor 3D reconstructions to the train set improves performance on good reconstructions in val/test by ~00.33 SPL from 0.922 (val), 0.917 (test) to 0.956 (val), 0.944 (test). Our conjecture is that training on poor (Gibson-(2.3)) )) and good (4+) reconstructions leads to robustness representations learned.
Better visual encoders and more parameters help. Using a better visual encoder, SE (Hu et al., 2018)-ResNeXt50 (Xie al., 2017) instead of ResNet50, improves performance by 0.003 SPL (Tab. 1). Adding capacity to the visual encoder (SE-ResNeX:101 vs SE-ResNeX(50) and navigation policy (1024-d vs 512-d LSTM) further improves performance by 0.010 SPL.
PointGoalNav 'solved' with RGB-D and GPS+Compass. Our best agent - SE-ResNeXt101 + 1024-d LSTM trained on Gibson-2+ achieves SPL of 0.969 (val), 0.948 (test), which not only sets the state of art on the Habitat Challenge 2019 RGB-D track but is also within 3-5% of the shortest-path oracle Given the challenges with achieving near-perfect SPL in new environments, it is important to dig deeper. Fig. 13 shows (a) distribution of episode lengths in val and (b) SPL vs episode length. We see that while the dataset is dominated by short episodes (2-12m), the performance of the agent is remarkably stable over long distances and average SPL is not necessarily inflated. Our hypothesis the agent has learned exploit the structural regularities in layouts of real indoor environments. One (admittedly imperfect) way to test this is by training Blind agent with only a GPS+Compass sensor. Fig. 13 shows that this agent is able to handle short-range navigation (which primarily involve turning to face the target and walking straight) but performs very poorly on longer trajectories SPL of 0.3 (Blind) vs 0.95 (RGB-D) at 20-25m navigation. Thus, structural regularities, in part, explain performance for short-range navigation. For long-range navigation, the RGB-D agent is extracting overwhelming signal from its Depth sensor. We repeat this analysis on two additional navigation datasets proposed by Chaplot et al. (2019)- longer episodes and 'harder' episodes (more navigation around obstacles) and find similar trends (Fig. 14). This discussion continues in Apx. A.
Performance with RGB is also improved. So far we studied RGB-D as this performed best in Savva et al. (2019). We now study RGB (with SE-ResNeX:50 encoder). We found it crucial to train on 2 and all of Matterport33, ensuring diversity in both layouts (Gibson-2+) and appearance (Matterport3D)) and to channel-wis normalize RGB (subtract by mean and divide by standard devi- ation) as our networks lack BatchNorm. Performance improves dramatically from 0.57 (val), 0.47 (test) SPL in Savva et al. (2019) to near-perfect success 0.991 (val), 0.977 (test) and high SPL 0.929 (val), 0.920 (test). While SPL is considerably lower than the Depth agent, (0.929 vs 0.959), inter- estingly, the RGB agent still reaches the goal similar percentage of the time (99.1%% 199.1% vs 99.9%) This agent achieves state-of-art the Habitat Challenge 2019 RGB track (rank 2 entry has 0.89 SPL).
No GPS+Compass remains unsolved. Finally, we examine if we also achieve better performance on the significantly more challenging task of navigation from RGB without GPS+Compass. At 100 million steps (an amount equivalent to Savva et al. (2019)), the agent achieves 0 SPI. By train- ing to 2.5 billion steps, we make some progress and achieve 0.15 SPL. While this is a substantial improvement, the task continues to remain an open frontier for research in embodied Al.
Transfer Learning, We examine transferring our agents to the following tasks (Gordon et al., 2019)
Videoos http:://www.yttuubee.c//////eees
7

---

Published as a conference paper at ICLR 2020
Fiee
Expl oration
Pertormancs (ligher ta bether)
16
Scratch
0.4
ImageliefEncdee SorachPulkyy
PoinGGofhaa Scrtder-Stttttttlttt
0.2
PointDodNavvroode Politun
Naural Controller
0.0
10
15
20
25
10
15
20
25
30
Steps (exper millions
(experience)) millons
Figure 5: Performance (higher is better) on Flee (left) and Exploration (right) under five settings.
Flee The agent maximizes its geodesic distance from its starting location. Let s, be the agent's position at time t, and Mor(s)) denote the maximum distance over all reachable points, then the agent maximizes Dr = Geo( 8T, 50) Marr(s)) The reward rr = 5 Dr - Dz-1)-
Exploration The agent maximizes the number of locations (specified by Im cubes) visited. Let |Visited;| denote the number of location visited at time t, then the agent maximizes |Visitedr]- Visited;-1l).
The reward is T 0.25([Visited,,
We use a NointGoalNav-traine agent with RGB and GPS+Compass, remove the GPS+Compass, and transfer to these tasks under five different settings:
Scratch. All parameters (visual encoder policy) are trained from scratch for each new task. Improvements over this baseline demonstrate benefits of transfer learning.
The visual encoder is initialized with ImageNet pre-trained weights and frozen; the navigation policy is trained from scratch.
The visual encoder is initialized from PointGoalNav and frozen; the navigation policy is trained from scratch. wnweaaaawwwnnnddeee-eeeeuue
Both visual encoder and policy parameters are initial- ized from PointGoalNav (critic layers are reinitialized). Encoder is frozen, policy is fine-tuned." 7 V Neural Controller We treat our agent as differentiable neural controller, a closed-loop low- level controller than can navigate to specified coordinate. We utilize this controller in new task by training a hight-weight high-level planner that predicts a goal-coordiio coordiinr (at each time-step) for the controller to navigate to. Since the controller is fully differentiable, we can backprop through it. We freeze the controller, train the plannertcontroller system with PPO for the new task. The planner is 2-layer LSTM and shares the (frozen) visual encoder with the controller.
Fig. 5 shows performance vs. experience results (higher is better). Nearly all methods outperform leamning from scratch, establishing the value of transfer learning. Poant pre-trained visual encoders dramatically outperforms ImageNet pre- pre-trained ones, indicating that the agent has learned generally useful scene understanding. For both tasks, fine-tuning an existing policy allows it to rapidly learn the new task, indicating that the agent has learned general navigation skills. Neurrl Controller outperforms ration, indicating that the agent can indeed be 'controlled' or directed to target locations by a planner. Overall, these results demonstrate that our trained model useful for more than just PointGoalNav,
on Flee and is competitive on Explo-
7 RELATED WORK
Visual Navigation. Visual navigation in indoor environments has been the subject of many recent works (Gupta et al., 2017; Das et al., 2018; Anderson et al., 201 8b; Savva et al., 2019; Mishkin et al., 2019). Our primary contribution is DD-PPO, thus we discuss other distributed works.
In the general case, computation in reinforcement learning (RL) in simulators can be broken down into 4 roles: 1) Simulation: Takes actions performed by the agent as input, simulates the new state, returns observations, reward, etc. 2) Inference: Takes observations as input and utilizes the agent policy to return actions, value estimate, etc. 3) Learner: Takes rollouts as input and computes gradients to update the policy's parameters. 4) Parameter server/master: Holds the source of truth for the policy's parameters and coordinates workers.
Since PointGoalNav policy expects goal-coordinate, we input a 'dummy' arbitrarily-chosen vector for the transfer tasks, which the agent quickly learns ignore.
8

---

Published as a conference paper at ICLR 2020
Synchronous RL. Synchronous RL systems utilize single processes to perform all four roles; this design is found in RL libraries like OpenAI Baselines (Dhariwal et al., 2017) and Py- torchRL (Kostrikov, 2018). This method is limited to single nodes worth of GPUs.
Synchronous Distributed RL. The works most closely related to DD-PPO also propose to scale synchronous RL by replicating this ainulationfiffernneelleene process across multiple GPUs and then synchronize gradients with AlIReduce. Stooke Abbeel (2018) experiment with Atari and find it not effective however. We hypothesize that this is due to a subtle difference - this distribu- tion design relies on a single worker collecting experience from multiple environments, stepping through them in lock step. This introduces significant synchronization and communication costs as every step in the rollout must be synchronized across as many as 64 processes (possible because each environment is light, e.g. Atari). For instance, taking 1 step in 8 parallel pong environments takes approximately the same wall-cloek time as pong environment, but it takes 10 times longer to take 64 steps in lock-step; thus gains from parallelization are washed out due to the lock-step synchronization. In contrast, we study resource-intensive environments, where only 2 or 4 environ- ments per worker possible, and find this technique be effective. Liang al. (2018b) mirror our findings (this distribution method can be effective for resource intensive simulation) in GPU- accelerated physics simulation, specifically MuJoCo (Todorov et al., 2012) with NVIDIA Flex. In contrast to our work, they examine scaling up to only 32 GPUs and only for homogeneous work- loads. In contrast to both, we propose an adaption to mitigate the straggler effect- - preempting the experience collection (rollout) of stragglers and then beginning optimization. This improves scaling for homogeneous workloads and dramatically improves scaling for heterogeneous workloads.
Asynchronous Distributed RI. Existing public frameworks for asynchronous distributed reinforce- ment learning (Heess al., 2017; Liang et al., 2018a; Espeholt et al., 2018) use single (CPU-only) process to perform the simulation and inference roles (and then replicate this process to scale). A separate process asynchronously performs the learner and parameter server roles (note its not clear how to use more than one these processes as it holds the source of truth for the parameters). Adapt- ing these methods to the resource-intensio environments studied in this work (e.g. Habtiat (Savva et al., 2019)) encounters the following issues: 1) Limiting the inference/simulation processes to CPU-only is untenable (deep networks and need for GPU-accelerated simulation). While the infer- ence/simulation processes could be moved to the GPU, this would be ineffective for the following: GPUs operate most efficiently with large batch sizes (cach inference/simulltion process would have batch size of 1), CUDA runtime requires ~600MB of GPU memory per process, and only one CUDA kernel (function that runs on the GPU) can executed by the GPU at a time. These issue contribute and lead low GPU utilization. In contrast, DD-PPO utilizes a single process per GPU and batches observations from multiple environments for inference. 2) The single process learner/- parameter server is limited a single node's worth of GPUs. While this not a limitation for small networks and low dimensional inputs, our agents take high dimensional inputs (e.g. a Depth sensor) and utilize large neural networks (ResNet50), thereby requiring considerable computation com- pute gradients. In contrast, DD-PPO has no parameter server and every GPU computes gradients, supporting even very large networks (SE-ResNeX:101).
Straggler Effect Mitigation. In supervised learning, the straggler effect is commonly caused by heterogeneous hardware or hardware failures. Chen et al. (2016) propose a pool of "back-up" workers (there are N + b workers total) and perform the parameter update once N workers finish. In comparison, their method a) requires parameter server, and b) discards all work done by the stragglers. Chen et al. (2018) propose to dynamically adjust the batch size of each worker such that all workers perform their forward and backward pass in the same amount of time. Our method aims to reduce variance in experience collection times. DD-PPO dynamically adjusts worker's batch size as necessary side-effect of preempting experience collection in on-policy RL.
Distributed Synchronous SGD. Data parallelism is a common paradigm in high performance com- puting (Hillis & Steele Jr, 1986). In this paradigm, parallelism is achieved by workers performing the same work on different data. This paradigm can be naturally adapted to supervised deep learn- ing (Chen et al., 2016). Works have used this to achieve state-of-the-ar results in tasks ranging from computer vision (Goyal et al., 2017; He et al., 2017) to natural language processing (Peters et al., 2018: Devlin et al., 2018; Ott et al., 2019). Furthermore, multiple deep learning frameworks provide simple-to-use wrappers supporting this parallelism model (Paszke et al., 2017; Abadi et al., 2015; Sergeev & Balso, 2018). We adapt this framework to reinforcement learning.

---

Published as a conference paper at ICLR 2020
8 ACKNOWLEDGEMENTS
The Georgia Tech effort was supported in part by NSF, AFRL, DARPA, ONR YIPS, ARO PECASE. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Martin Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawar, lan Goodfellow, Andrew Harp. Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenherg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL httphhtp/////e/ee flow. org/. Software available from tensorflow.org.
Peter Anderson, Angel Chang. Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozheh Mottaghi, Manolis Savva, et al. On evaluation of embodied naviga- tion agents. arXiv preprint 1rXiv:180..06757, 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, lan Reid, Stephen Gould, and Anton den Hengel. Vision-and lan--aaaa- navigation: Interpreting Uisuall-grrunnde navigation instructions in real environments. In CVPR, 2018b.
Charles Beattie, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, Julian Schrittwieser, Keith Anderson, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Decpmind lab. arXiv, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3D: Learning from RGB-D data in indoor environments. International Conference on 3D Vision (3DV), 2017.
Devendra Singh Chaplot, Kanthashree Mysore Sathyendra, Rama Kumar Pasumarthi, Dheeraj Rajagopal, and Ruslan Salakhutdinov. Gated-attention architectures for task-oriented language grounding. arXiv preprint arXiv: 170.007230 2017.
Devendra Singh Chaplot, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhutdinov. Modular visual nav- igation using active neural mapping. http://www. cs. www.cmu.eeeeooa http.eeluu/cchhpppoo/ppppsssssee mapping. 2019.
Chen Chen, Qizhen Weng, Wei Wang, Baochun Li, and Bo Li. Fast distributed deep learning via worker- adaptive batch sizing, arXiv preprint urXiv:/806.0220 82580000050 2018.
Jianmin Chen, Xinghao Pan, Rajat Monga, Samy Bengio, and Rafal Jozefowicz. Revisiting distributed syn- chronous sgd. arXiv preprint arXiv: /604:0000400098 2016.
Abhishck Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Ques- tion Answering. In CVPR, 2018.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint 1rXiv:1810.04805, 2018.
Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schul- man, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov. Openai baselines. https://ge thub. com/openai/ baselines, 2017.
Lasse Espeholt, Huhert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, ct al. Impala: Scalable distributed deep-rl with importance weighted actor-lcamer architcctures. arXiv preprint arXiv: 18Xx:1800..000 1802.0/ 2018.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. IQA: Visual question answering in interactive environments. In CVPR, 2018.
Daniel Gordon, Abhishek Kadian, Devi Parikh, Judy Hoffiman, and Dhruv Batra. Splitnet: Sim2sim and (ask2task transfer for embodied visual navigation. ICCV, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 010.77006000777 2017.
11

---

Published as a conference paper at ICLR 2020
Saurabh Gupta, James Davidson, Sergey Levine. Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition, pp. 2616,2625, 2017.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IPFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE 2017.
international conference on computer vision, pp.
Nicolas Heess, Srinivasan Sriram. Jay Lemmon, Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: (70.00270,,2286, 2017.
W Daniel Hillis and Guy L Steele Jr. Data parallel algorithms. Communication of the ACM, 19(12)11170 1183, 1986.
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-eecitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 7132-7141, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 148-456, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6988 2014.
Ilya Kostrikov. Pytorch implementations of reinforcement learning algorithms. httpp://gitthhh com/ 2018.
Thorsten Kurth, Sean Treichler, Joshua Romero, Mayur Mudigonda, Nathan Luchr, Everett Phillips, Ankur Mahesh, Michael Matheson, Jack Deslippe, Massimiliano Fatica, et al. Exascale deep learning for climate analytics. In Proceedings of the International Conference for High Performance Computing. Networking, Storage, and Analysis, pp. 51. IEEE Press, 2018.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg. Joseph E. Gonzalez, Michael Jordan, and Ion Stoica. RLlib: Abstractions for distributed reinforcement learning. In Interna- tional Conference on Machine Learning (ICML), 2018a.
Jacky Liang, Viktor Makoviychuk, Ankur Handa, Nuttapong Chentanez, Miles Macklin, Dieter Fox. Gpu- accelerated robotic simulation for distributed reinforcement learning. In Conference on Robot Learning, 270-20 280, 2018b.
Dmytro Mishkin, Alexey Dosovitskiy, and Vladlen Koltun. Benchmarking classic and learned navigation in complex 3d environments. arXiv preprint arXiv: 10915, 2019.
OpenAL. Openai five. https://blog..ppenn http://ccoo/pppaaiii///aa 2018.
Myle Ott, Sergey Edunov, Alexci Buevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. arXiv preprint arXiv:1904.01038, 2019.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop, 2017.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint 1rXiv:1802.05366 2018.
Manolis Savva, Abhishck Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. ICCV, 2019.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional contin- control using generalized advantage estimation. arXiv preprint arXiv: 1506.02438. 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint 1rXiv:1707.06347 2017.
2017.
Alexander Sergeev and Mike Del Balso. Horovod: fast and easy distributed deep learning in TensorFlow. arXiv preprint 1802:00802 2018.
12

---

Published as a conference paper at ICLR 2020
David Silver, Aja Huang, Christopher J. Maddison, Arthur Guez, Laurent Sifre, George den Driessche, Julian Schrittwieser, Joamnis Antonoglou, Veda Panneershelvam, Mare Lanctot, Sander Dieleman, Do- minik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madelcine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowl- edge. Nature, 350(767)):354, 2017.
Adam Stooke and Pieter Abheel. Accelerated methods for deep reinforcement learning. arXiv preprint arXiv: 1oX3vv8803002811 2018.
Yuandong Tian, Jerry Ma, Qucheng Gong, Shubho Sengupta, Zhuoyuan Chen, James Pinkerton, and C Lawrence Zitnick. Elf opengo: An analysis and open reimplementation of alphazero. arXiv preprint arXiv:1902.04522 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for madel-based control. 2012 JEEE/RS/ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering photorealistic environments with point cloud perception. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 6659-6668, 2019.
Yuxin Wu and Kaiming He. Group normalization. arXiv preprint arXiv:1803.084999 2018.
Xia, Amir R. 7amir, Zhiyang Hc, Alexander Sax, Jitendra Malik, and Silvio Savarese. Gibson env: real- world perception for embodied agents. In CVPR, 2018.
Saining Xie, Ross Girshick. Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500 1500, 2017.
13

---

Published as a conference paper at ICLR 2020
Geudesic Distance (rows) SPL (cols)
0.00-0.20
0.20-0.50
0.50-0.90
0.90-0.95
0.95-1.00
10.0-11.7
11.7-13.4
13.4-15.0
15.0-16.7
16.7-26.8
Figure 6: Example episodes broken down by geodesic distance between agent's 's spawn location and target (on rows) SPL achieved by the agent (on cols). Gray represents navigable regions on the map while white is nov-navia. The agent begins at the blue square and navigates to the red square. The green line shows the shortest path the map (or oracle navigation). The blue line shows the agent's trajectory. The color of the agent's trajectory changes changes from dark to light over time. Navigation dataset from the longer validation episodes proposed in Chaplot et al. (2019).
A ADDITIONAL ANALYSIS AND DISCUSSION
In this section, we continue the analysis of our agent and examine differences in its behavior from classical, hand-designed agent the map-and-plan baseline agent proposed in Gupta et (2017).
Intricacies of SPI. Given an agent that always reaches the goal (000% success), SPL can be seen measuring the effi ciency of an agent vs. an oracle - i.e. an SPL of 0.95 means the agent is 5% less efficient than an oracle. Given the challenges of near-perfect autonomous navigation without a map in novel environments we outlined, being 5% less efficient than an or- acle seems impossibese However, this comparison/vied potentially miss-leadin leading. Percentage errors are potentially miss-leading for long paths. Over 10 meter episode, the agent can deviate from the oracle path by up-to a meter and still be within 10%. As consequence, significant qualitative errors can result in an insignificant quantitative error (see Fig. 6).
SPL 0.69, Chapiot et al. Longer Validation Episades
10.0
12.5
10.0
7.5
0.0
0.8
0.8
Figure 7: Histogram of SPL for
non-perfect (SPI.<0.99) episodes.
Error recovery. Given the near-perfect performance of our agent (on average). we explicitly examine if is able to recover from its own navigation errors. Fig. 6 column shows several examples of error recovery, including several well executed backtracks (video: ht https: / /www. youtube. com/ wwtchh2==BAugWLLSTT
dicating that the agent is effective at recovering from its own navigation errors. Next, we look at the statistics of non-perfect (SPI.<0.99) episodes on the longer validation episodes proposed in Chaplot et al. (2019). Non-perfect episodes make up the majority of episodes (54%, see Fig. 7) with an average SPL of 0.85 (99.0% success) compared to 0.92 SPL (99.5% success) over all episodes. Thus there many episodes where the agent makes significant deviation from the shortest path and reaches the goal (a 15% deviation on long trajectories (> 10m) is significant).
14

---

Published as a conference paper at ICLR 2020
When does the agent fail? Column 2 in Fig. 6 shows that the agent performs poorly when the ratio of the geodesic distance to goal and euclidean distance to goal. However, the agent is able to eventually overcome this failure mode and reach the goal in most cases.
Row 1 column 1 in Fig. 6 shows that the agent fails or performs poorly when needs to go slightly up/down stairs. The data-set generation process used in Savva et al. (2019) only guarantees a start and goal pair won't 't be on different floors, but there remains a possibility that the agent will need to traverse the stairs slightly. However, these situations are rare, and, in general, the stairs should be avoided. Furthermore, the GPS sensor provides location in 2D, not 3D.
The remaining failure cases of column I in Fig. 6 show that a singular location in one environment acts as sink for the agent (once it enters this location, is almost never able leave it). At this location, there is large hole in the mesh (an entire wall is missing). Utilizing visual encoders that explicitly handle missing values may allow the agent to overcome this failure mode.
Differences from a classical agent. We compare the behavior of our agent with the classical map- and-plan baseline agent proposed in Gupta et al. (2017). This agent achieves 0.92 val (0.89 test) SPL with 0.976 success. & By comparing and contrasting qualitative behaviors, we can determine what behaviors learning-based methods enable. We make the following observation.
The learned agent able to recover from unexpected collisions without hurting SPL. The map- and-plan baseline agent incorporates specific collision recovery behavior where, after repeated collisions, the agent turns around and backs up 1.25m. This behavior brings the obstacle into view, maps it, and then allows the agent to create a plan to avoid it. In contrast, our agent is able to navigate around unseen obstacles without such a large impact on SPL. Determining the set of action sequences and heuristics necessary to do this is what learning enables.
B RELATED WORK CONTINUED
Straggler Effect Mitigation. In supervised learning, the straggler effect is commonly caused by heterogeneous hardware or hardware failures. Chen et al. (2016) propose a pool of "back-up" workers (there are N + b workers total) and perform the parameter update once N workers finish. In comparison, their method a) requires parameter server, and b) discards all work done by the stragglers. Chen et al. (2018) propose to dynamically adjust the batch size of each worker such that all workers perform their forward and backward pass in the same amount of time. Our method aims to reduce variance in experience collection times. DD-PPO dynamically adjusts worker's batch size as a necessary side-effect of preempting experience collection in on-policy RL.
Distributed Synchronous SGD. Data parallelism is a common paradigm in high performance com- puting (Hillis & Steele Jr, 1986). In this paradigm, parallelism is achieved by workers performing the same work on different data. This paradigm can be naturally adapted to supervised deep learn- ing (Chen et al., 2016). Works have used this to achieve state-of-the- art results in tasks ranging from computer vision (Goyal et al., 2017; He et al., 2017) to natural language processing (Peters al., 2018; Devlin et al., 2018; Ott et al., 2019). Furthermore, multiple deep learning frameworks provide simple-to-use wrappers supporting this parallelism model (Paszke al., 2017; 7; Abadi et al., 2015; Sergeev & Balso, 2018). We adapt this framework to reinforcement learning.
C AGENT DESIGN
In this section, we outline the exact agent design we use. We break the agent into three components: visual encoder, goal encoder, and navigation policy.
Visual Encoder. Out visual encoder uses one of three different backbones, ResNet50 (He et al., 2016). Squeeze-Excite((SS (Hu et al., ResNeeet5 (Xie et al., 2017), and SE- For all backhones, we reduce the number of output channels at each layer by half. We also add a 2x2-AvgPool hefore each backbone so that the effective resolution is 128x128 128. Given these modifi- cations, each backbone produces a 1024x4x4 feature map. We then convert this a 128x4x4 feature map with a 3x3-Conv.
We replace every BatchNorm layer with GroupNorm (Wu & He, 2018) to account for the highly correlated trajectories seen in on-policy RL and massively distributed training.
"nttps://github http::///tttiee.ccccs/ss/sappppp
15

---

Published as a conference paper at ICLR 2020
Hamageneous Scaling
Multi-Node Hamogeneous Scaling
250
15.0
40%
60%
60%
Relatvo Scaling
200
80%
Relative Scaling
12.5
80%
150
100%
10.0
100%
Ideal
Ideal
100
2.5
0
50
100
150
200
250
50
100
150
200
250
Number GPUs
Number GPUs
(a)
(b)
Heterogeneous Scaling
Multi-Node Heterogeneous Scaling
250
40%
40%
60%
60%
Relative Scaling
200
80%
Relative Scaling
12.5
80%
150
100%
10.0
100%
Ideal
Ideal
100
0
50
100
150
200
250
50
100
150
200
250
Number GPUs
Number GPUs
(c)
(d)
Figure 8: Scaling of DD-PPO under homogeneous and hetcrogeneous workloads for various differ- ent values of the percentage of rollouts that are fully completed by optimizing the model. Shading represents a bootstrapped 95% confidence interval.
Goal encoder. Habitat (Savva al., 2019) provides the vector pointing to the goal in ego-centric polar coordinates. We convert this to magnitude and a unit vector, i.e. [d, #] to [d, cos(#), (fn[ to account for the discontinuity at the z-axis in polar coordinates. We pass the goal vector to fully connected layer, resulting in a 12-dimensional representation.
Navigation Policy. Our navigation policy takes the 64x4x4 feature map from the visual encoder, flat- tens it, and then converts the 2048-d vector to the same size as the hidden size via fully-connected layer. It then concatenates this vector with output of the goal encoder, and 32-dimensional embed- ding the previous action taken (or the start-token in the case of the first action) and then passes this to 2-layer LSTM with either 51 512-dimensional or 1024-dimensional hidden dimension. The output of the LSTM used as input to fully connected layer, resulting soft-max distribution of the action space and estimate of the value function.
D ADDITIONAL SCALING DETAILS
We the following procedure for benchmarking the throughput of our proposed DD-PPO: Each optimizer selects 4 scenes at random and then performs the process collecting experience and optimizing the model based that experience 10 times. We calculate throughput the total number of steps of experience collected over the last 5 rollout/optimizing steps divided by the amount of time taken. We repeat this procedure over 10 different random seeds (we use the same random seeds for all variations of number of GPUs and sync-fraction fraction values).
E DD-PPO IMPLEMENTATION
Utilizing Distributed Data Parallel supervised learning is straightforward as frameworks such as PyTorch (Paszke et al., 2017) provide simple wrapper. The recommended way to use these wrappers is to first write training code that runs on single GPU and then enable dis- tributed training via the wrapper. We follow similar approach. Given implementation of
16

---

Published as a conference paper at ICLR 2020
Validation
Test Standard
Perception
Method
SPL
Success
SPL
Success
Random
0.02
0.03
0.02
Forward-only
0.00
0.00
0.00
Blind
Goal-follower
0.23
0.23
0.23
DD-PPO (RL)
0.729 0.005
0.973
0.003
0.676
0.947
RGB
DD-PPO (RL)
0.929 + 0.003
0.991
0.002
0,920
0.977
RGB-D (Depth)
DD-PPO (RL)
0.969 + 0.002
0.997
0.001
0.948
0.980
Table 2: Performance (higher is better) of various sensors and agent methods the Habitat Chal- lenge 2019 (Savva et al., 2019) validation and test splits (checkpoint selected on val). Random, Forward-only, and Goal-follower taken from Savva al. (2019). Best visual encoder reported for DD-PPO.
PPO that runs one GPU create a decentralized distributed variant by adding gradient syn- chronization, leveraging highly performant code written for this purpose in popular deep-learning frameworks, e.g. tf. distiriibbtee MirroredStrategy in TensorFlow (Abadi et al., 2015) and torch.nn. waw.nnnllpaalllll.. www...tiibuueeeataeee... synchronize any training or rollout statisties between workers most cases these can also be synchronized via AllReduce.
in PyTorch. Note that care must be taken to
We track how many workers have finished the experience collection stage with distributed key- value storage we use PyTorch's torch. distributed, TCPStore, however almost any distributed key-value storage would be sufficient.
See Fig. for an example implementation which adds 1) gradient synchronization torch. nn. ber of workers have finished the experience collection stage with a torch. www.ributed..... TCPStore.
and 2) preempts stragglers by tracking the num-
See Fig. 10 for a visual depiction of DD-PPO.
F TRANSFER EXPERIMENTS ADDITIONAL DETAILS
For the transfer learning experiments, we utilize the same PPO hyper-parameters as the PointGoalNav experiments. We DD-PPO to train with 8 workers on GPUs. We train agents on Gibson-4+ and evaluate on the Habitat Challenge 2019 Validation scene and starting lo- cations (the goal location is simply discarded).
The ImageNet encoder trained using the same hyper-parameters and training procedure as Xie et al. (2017) with no data-augmentation.
G NEURAL CONTROLLER ADDITIONAL DETAILS
The planner for neural controller used in Sec. shares the same architecture as our agent's policy, but utilizes 512-d hidden state. takes as input the previous action of the controller (or the start token), and the output of the visual encoder (which is shared with the controller). The output of the LSTM is then used to produced an estimate of the value function and a 3-dimensional vector specifying the PointGoal in magnitude and unit direction vector format. The magnitude competent passed through an ELU activation and offset by 0.75. Each component the unit direction vector passed through tanh activation note that do not re-normalize this vector have a length of as we find doing so both unnecessary and harder to optimize.
17

---

" If this wasn't being used, here is where the gradients would Published as a conference paper at ICLR 2020
master_adde <hostname world rank machine>
master_port <free TCP port on world rank 0ss machine> world_rank # <worker's unique ID> world_size f <number of workers> local_rank # <the ID of the GPU use>
Setup the group of workers store torch. distributed. TCPStore master_addr,
master_port,
world_size, world_rank == 0,
torch. distributedd htw.:bbueeed.iittpoeeeeeeeeeeeee init_prroces_ggruu
backend="NcCL-
world_size=world_szz
rank=world_rank,
store=store. )
# Tracks how many workers have finished their rollout num_workers_done torch. diwrrihdditrbbuee... PrefixStore(
"num_workers_done" store
)
device torch... device ("cuda", local_rank)
model PolicyNetwork (...)
model... to (device)
Add gradient synchronization to the model
model torch. nn. bep.rrrll....issrbbbeebttteeeee .isrrbbuueedatteaaalllll model, [device], device
while not n)t_converged()
num_workers_done. set ("done", "ø") for step in range expxeereeeieee_sseps)) sollect_step (model) # Preempt stragglers
if
int @num_workers_donr get ("done greemption_thresheeee
world_size
and step >= max_ eex_experience_steps
):
break
Mark that worker is done collecting experience wun_workers_done and ("done", 1)
# Update the model using PPO
for in range((__po__poohs for in range range (nnpoobn_____aatcc)) batch get_batch()
loss = evaluate (model, batch) loss backward
DistributedbbaParalle automatically performs an AllReduce on all gradients
# during the backward call.
# calls to AllReduce made.
step_optimizer (model)
Figure 9; Implementation of DD-PPO using PyTorch (Paszke al., 2017) v1.1 and the NCCL. backend. We use SLURM to populate the world rank, world size, and local rank fields.
18

---

Published as a conference paper at ICLR 2020
Percent processes finished collecting experience)
Monitor
Preemplion
threshold
Process #1
Stop
Stop
Stop
Stop
Done
Optimizze Model
Process #2
Stop
Step
Stop
Step
Done
Cradient Synchronizetion
Process #N
Step
Step
Preempted
Optimize Model
Figure 10: Illustration of DD-PPO. Processes collecting experience environments that are more costly simulate (stragglers) have their experience collection stage preempted such that other pro- cesses do not have to wait for them. Note that we implement the monitor with a simple key-value storage and have processes preempt themselves. Note that the order of processes irrelevant and done solely for aesthetic purposes.
2: big holes significant texture issues, but good reconstruction
small holes, some texture issues, good reconstruction
4: holes, some texture issues, good reconstruction
5: no holes, uniform textures, good reconstruction
Figure 11: Examples of Gibson meshes for given quality rating from Savva et al. (2019)
19

---

Published as a conference paper at ICLR 2020
2.875
0,975
0.775
2.905
3.925
0,000
0.000
0.000
Tran
Tran
Tran
Vai
Vil
0.800
0.590
0.890
200
1500
2000
2000
1900
1500
2002
2500
1980
2003
2500
Stops (cncerionc)) milllooss
Sxps (oxpericne]] inm llons
Stopo (oxperinnc) in lons
0.375
0.975
0.515
Humu!
3.800
0.800
0.000
0.025
8.000
0.90
0.000
200 1/90
1500
2000
2500
0.000
1900 Stapse (espertenco)
200
10.00
2035
2003
500
1000
1580
2000
2500
Stepe pmspereence
(scpertencn) million
Figure 12: Training and validation performance (in SPL; higher is better) of different architectures for Depth agents with GPS+Compass on the Habitat Challenge 2019 (Savva al., 2019). Gib- son (Xia et al., 2018)-4+ refers to the subset of Gibson train scenes with quality rating of 4 or better. Gibson-4+ and MP3D refers to training on both Gibson-4+ and all of Matterport3D. Gibson- 2+ refers to training the subset of Gibson train scenes with quality rating of 2 or better.
Habitat Challenge 19 Validation Episodes
Success
Perfomance (higheris better)
0.8
Blind
0.6
RGB
RGB-D (Depth)
0.4
0.0
10
15
20
25
GeodeesiDiisaanceaStatt,,, Goal)
GacdessicDisannceSSaatt Goal)
Figure 13: Performance Geodesic Distance from start goal for Blind, RGB, and RGB-D (us- ing Depth only) models trained with DD-PPO on the Habitat Challenge 2019 (Savva et al., 2019) validation split. Bars the bottom represent the fraction of episodes within each geodesic distance bin.
20

---

Published as a conference paper at ICLR 2020
Chaplot et al. (2019) Longer Validation Episodes
SPL
Success
1.0
Performance (higher beiter)
0.8
0.6
RGB
RGB-D (Depth)
0.4
0.2
20
GeodesiiDissaaneeSSaa Goal)
GeodesicDistanceeSShrt Goal)
Chaplot et al. (2019) Harder Validation Episodes
SPL
Success
1.0
Performance (higher better)
Bind
0.8
RGB
RGB-D (Depth)
0.2
15
GeodessieDistannee(Sarrr Goal)
GredeesicDistanceeSSatt, Goal)
Figure 14: Performance vs. Geodesic Distance from start to goal for Blind, RGB, and RGB-D (using Depth only) models trained with DD-PPO on the longer and harder validation episodes proposed in Chaplot et al. (2019). Bars at the bottom represent the fraction of episodes within each geodesic distance bin.
21


Reviews:
[{"review_id": "r1xyq2tnKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a novel scheme of distributing PPO reinforcement learning algorithm for hundreds of GPUs. Proposed technique was validated for pointgoal visual navigation task on recently introduced Habitat challenge and sim. \n\nBesides the technical contribution, paper shows that when have enough computational power of billions simulation runs, it is possible to learn nearly perfect visual navigation (given RGBD + GPS inputs) via reinforcement learning. \nAuthors also study the task itself and show that it is yet not possible to achieve a good results without dense (each step) GPS signal, while the \"Blind\" agent, which has only GPS+compass error achieves quite high results given the billion-scale training time. \nThis suggests that PointGoal navigation with dense GPS signal is might be a poor choice to benchmark RL algorithms and we should proceed to harder tasks.\n\n\nOverall I like the paper a lot and think that it should be accepted.\n\n*** \nI haven`t changed my mind after the rebuttal: the paper is good and should be accepted.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "r1lmtrSe9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper proposes a Decentralized Distributed architecture for PPO. The idea was demonstrated on Habitat-Sim, the Habitat Autonomous Navigation Challenge 2019. The experiment shows how the implementation can achieve a speedup in training and near-linear scalability, over non-distributed and centralized architecture. \n\nOverall, the paper is well written and easy to follow. The idea is somewhat technical, e.g. distributed implementation of PPO and preemption for the rollouts of stragglers. The experimental results look promising but lack extensive evaluations given such a technical contribution. Therefore the paper has limited contributions. I have some following major comments.\n\n1. As it is a technical paper, it would be nice if the authors could describe more on the implementation of challenges and the hacks.\n\n2. The experiments could also be compared to one or two other distributed RL frameworks. It would also be helpful if detailed experiment settings are detailed, e.g. GPU characteristics, DDPPO's hyperparameters, etc.\n\n3. The Transfer Learning tasks is a general setting for for any methods, which are not limited to only Distributed Approach. The results and setting there do not have links such as why and how distributed approaches help such transfer learning.\n\n\n* minor comments:\n- Eq.1: expectation should also be w.r.t transition's stochasticity.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HyghBWDjYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The contribution of this paper is twofold. First the authors propose and implement and new scalable RL training procedure they call 'Decentralized Distributed Proximal Policy Optimization' (DD-PPO). This method extends PPO in a way that is 'synchronous' and 'distributed': no parameter server exists to asynchronously collect experience from worker agents and instead an 'explicit communication state' exists during which all workers communicate gradient updates between one another to update parameters based on their experiences. However, naively implementing this approach is limited by 'stragglers', the slowest of the workers that all other agents need to wait for. To overcome this limitation, the authors add a 'preemption threshold' which halts worker rollouts after a certain percentage has completed. The authors use this procedure to tackle the task of PointGoal navigation, in which an agent tries to reach a point in space specified by its relative location to the agent. They achieve state-of-the-art performance (arguably 'mastery') on this task. Finally, the authors show that their learned policy transfers reasonably well to two other navigation tasks, 'flee' and 'exploration'. Code is also provided.\n\nThis is an impressive paper. The algorithmic contribution is useful and the results are state-of-the-art. The paper is written very clearly, and it was a pleasure to read. The investigation of the task of PointGoal Navigation, and the impact of various changes on the overall performance of the system . The transfer tasks, to the 'flee' and 'exploration' tasks, were interesting, and further reinforced the results in the paper.\n\nMy only high-level 'complaint' is that the title might be tempered: 'Mastering PointGoal Navigation' is a very strong claim. The results in the paper are very impressive, but 'Mastering', in my mind, implies (among other things) that the agent is guaranteed to reach its goal, though I understand that it is perhaps a minor point, considering the failure rate is systematically <3%. I also make a comment later about how the Matterport 3D environments may be a biased set, and that there are almost certainly environments that *would* make this system fail, for one reason or another. I am willing to accept the title as-is (and could be overwritten by other reviewers), but I would like some discussion on a more appropriate tile, if possible.\n\nI recommend this paper for acceptance.\n\nMinor comments:\n- Introduction: \"Thus, there is a need to develop a different distributed architecture.\" It is unclear how this sentence logically follows from the previous sentence. I am on board with the vision in general, and it is quite clear to me: having a single parameter server is limiting. However, that the move from CPU- to GPU-based agents is what reveals this limitation is not quite so clear. Reworking this paragraph (or perhaps adding a sentence) to make this clearer would be helpful.\n- Introduction: \"This means there is no scope for mistakes of any kind --- no wrong turn at a crossroad, ...\" This is a fascinating point, though I feel reflects a bias in the types of environments that appear in the training data set. It is a defining behavior of intelligent embodied agents that they are capable of recovering from their mistakes; in general, real-world environments and navigation tasks have inherent ambiguity that cannot be resolved without exploration (a 'maze' is an extreme example of this). One question comes to mind: if the environments were ambiguous in this way, *could* the agent recover from its mistakes. There is not much evidence in the paper one way or the other. Since there are few examples in the data that seem to bring about such scenarios, it is not obvious how well this agent would perform in the face of this uncertainty. However, the results later in the paper about how \"No GPS+Compass remains unsolved\" is related to this point. A comment (or a figure in the Appendix) from the authors describing some of the failure cases would be instructive: do these failures occur in environments with such an ambiguity or, conversely, are there some examples in which the agent succeeds in reaching the goal but must overcome failure during its travel?\n- The acronym 'SL' (for Supervised Learning) is not defined and is used only once. Prefer using 'supervised learning'.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present and implement a synchronous, distributed RL called Decentralized Distributed Proximal Policy Optimization. The proposed technique was validated for pointgoal visual navigation task on recently introduced Habitat challenge 2019 and got the state of art performance.

Two reviews recommend this paper for acceptance with only some minor comments, such as revising the title. The Blind Review #2 has several major concerns about the implementation details. In the rebuttal, the authors provided the source code to make the results reproducible.

Overall, the paper is well written with promising experimental results. I also recommend it for acceptance.

Decision: Accept (Poster)