DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames

1 INTRODUCTION . Recent advances in deep reinforcement learning ( RL ) have given rise to systems that can outperform human experts at variety of games ( Silver et al. , 2017 ; Tian et al. , 2019 ; OpenAI , 2018 ) . These advances , even more-so than those from supervised learning , rely on significant numbers of training samples , making them impractical without large-scale , distributed parallelization . Thus , scaling RL via multi-node distribution is of importance to AI – that is the focus of this work . Several works have proposed systems for distributed RL ( Heess et al. , 2017 ; Liang et al. , 2018a ; Tian et al. , 2019 ; Silver et al. , 2016 ; OpenAI , 2018 ; Espeholt et al. , 2018 ) . These works utilize two core components : 1 ) workers that collect experience ( ‘ rollout workers ’ ) , and 2 ) a parameter server that optimizes the model . The rollout workers are then distributed across , potentially , thousands of CPUs1 . However , synchronizing thousands of workers introduces significant overhead ( the parameter server must wait for the slowest worker , which can be costly as the number of workers grows ) . To combat this , they wait for only a few rollout workers , and then asynchronously optimize the model . However , this paradigm – of a single parameter server and thousands of ( typically CPU ) workers – appears to be fundamentally incompatible with the needs of modern computer vision and robotics communities . Over the last few years , a large number of works have proposed training virtual robots ( or ‘ embodied agents ’ ) in rich 3D simulators before transferring the learned skills to reality ( Beattie ∗Work done while an intern at Facebook AI Research . Correspondence to etw @ gatech.edu . 1Environments in OpenAI Gym ( Brockman et al. , 2016 ) and Atari games can be simulated on solely CPUs . et al. , 2016 ; Chaplot et al. , 2017 ; Das et al. , 2018 ; Gordon et al. , 2018 ; Anderson et al. , 2018b ; Wijmans et al. , 2019 ; Savva et al. , 2019 ) . Unlike Gym or Atari , 3D simulators require GPU acceleration , and , consequently , the number of workers is greatly limited ( 25 to 8 vs. 212 to 15 ) . The desired agents operate from high dimensional inputs ( pixels ) and , consequentially , use deep networks ( ResNet50 ) that strain the parameter server . Thus , there is a need to develop a new distributed architecture . Contributions . We propose a simple , synchronous , distributed RL method that scales well . We call this method Decentralized Distributed Proximal Policy Optimization ( DD-PPO ) as it is decentralized ( has no parameter server ) , distributed ( runs across many different machines ) , and we use it to scale Proximal Policy Optimization ( Schulman et al. , 2017 ) . In DD-PPO , each worker alternates between collecting experience in a resource-intensive and GPU accelerated simulated environment and optimizing the model . This distribution is synchronous – there is an explicit communication stage where workers synchronize their updates to the model ( the gradients ) . To avoid delays due to stragglers , we propose a preemption threshold where the experience collection of stragglers is forced to end early once a pre-specified percentage of the other workers finish collecting experience . All workers then begin optimizing the model . We characterize the scaling of DD-PPO by the steps of experience per second with N workers relative to 1 worker . We consider two different workloads , 1 ) simulation time is roughly equivalent for all environments , and 2 ) simulation time can vary dramatically due to large differences in environment complexity . Under both workloads , we find that DD-PPO scales near-linearly . While we only examined our method with PPO , other on-policy RL algorithms can easily be used and we believe the method is general enough to be adapted to off -policy RL algorithms . We leverage these large-scale engineering contributions to answer a key scientific question arising in embodied navigation . Mishkin et al . ( 2019 ) benchmarked classical ( mapping + planning ) and learning-based methods for agents with RGB-D and GPS+Compass sensors on PointGoal Navigation ( Anderson et al. , 2018a ) ( PointGoalNav ) , see Fig . 1 , and showed that classical methods outperform learning-based . However , they trained for ‘ only ’ 5 million steps of experience . Savva et al . ( 2019 ) then scaled this training to 75 million steps and found that this trend reverses – learningbased outperforms classical , even in unseen environments ! However , even with an order of magnitude more experience ( 75M vs 5M ) , they found that learning had not yet saturated . This begs the question – what are the fundamental limits of learnability in PointGoalNav ? Is this task entirely learnable ? We answer this question affirmatively via an ‘ existence proof ’ . Utilizing DD-PPO , we find that agents continue to improve for a long time ( Fig . 1 ) – not only setting the state of art in Habitat Autonomous Navigation Challenge 2019 ( Savva et al. , 2019 ) , but essentially ‘ solving ’ PointGoalNav ( for agents with GPS+Compass ) . Specifically , these agents 1 ) almost always reach the goal ( failing on 1/1000 val episodes on average ) , and 2 ) reach it nearly as efficiently as possible – nearly matching ( within 3 % of ) the performance of a shortest-path oracle ! It is worth stressing how uncompromising that comparison is – in a new environment , an agent navigating without a map traverses a path nearly matching the shortest path on the map . This means there is no scope for mistakes of any kind – no wrong turn at a crossroad , no back-tracking from a dead-end , no exploration or deviation of any kind from the shortest-path . Our hypothesis is that the model learns to exploit the statistical regularities in the floor-plans of indoor environments ( apartments , offices ) in our datasets . The more challenging task of navigating purely from an RGB camera without GPS+Compass demonstrates progress but remains an open frontier . Finally , we show that the scene understanding and navigation policies learned on PointGoalNav can be transferred to other tasks ( Flee and Explore ( Gordon et al. , 2019 ) ) – the analog of ‘ ImageNet pre-training + task-specific fine-tuning ’ for Embodied AI . Our models are able to rapidly learn these new tasks ( outperforming ImageNet pre-trained CNNs ) and can be utilized as near-perfect neural PointGoal controllers , a universal resource for other high-level navigation tasks ( Anderson et al. , 2018b ; Das et al. , 2018 ) . We make code and trained models publicly available . 2 PRELIMINARIES : RL AND PPO . Reinforcement learning ( RL ) is concerned with decision making in Markov decision processes . In a partially observable MDP ( POMDP ) , the agent receives an observation that does not fully specify the state ( st ) of the environment , ot ( e.g . an egocentric RGB image ) , takes an action at , and is given a reward rt . The objective is to maximize cumulative reward over an episode , Formally , let τ be a sequence of ( ot , at , rt ) where at ∼ π ( · | ot ) , and st+1 ∼ T ( st , at ) . For a discount factor γ , which balances the trade-off between exploration and exploitation , the optimal policy , π∗ , is specified by π∗ = argmax π Eτ∼π [ RT ] , where , RT = T∑ t=1 γt−1rt . ( 1 ) One technique to find π∗ is Proximal Policy Optimization ( PPO ) ( Schulman et al. , 2017 ) , an on-policy algorithm in the policy-gradient family . Given a θ-parameterized policy πθ and a set of trajectories collected with it ( commonly referred to as a ‘ rollout ’ ) , PPO updates πθ as follows . Let Ât = Rt − V̂t , be the estimate of the advantage , where Rt = ∑T i=t γ i−tri , and V̂t is the expected value of Rt , and rt ( θ ) = πθ ( at|ot ) πθt ( at|ot ) be the ratio of the probability of the action at under the current policy and the policy used to collect the rollout . The parameters are then updated by maximizing J PPO ( θ ) = Et [ min ( rt ( θ ) Ât︸ ︷︷ ︸ importance-weighted advantage , clip ( rt ( θ ) , 1− , 1 + ) Ât︸ ︷︷ ︸ proximity clipping term ) ] ( 2 ) This clipped objective keeps this ratio within and functions as a trust-region optimization method ; allowing for the multiple gradient updates using the rollout , thereby improving sample efficiency . 3 DECENTRALIZED DISTRIBUTED PROXIMAL POLICY OPTIMIZATION . In reinforcement learning , the dominant paradigm for distribution is asynchronous ( see Fig . 2 ) . Asynchronous distribution is notoriously difficult – even minor errors can result in opaque crashes – and the parameter server and rollout workers necessitate separate programs . In supervised learning , however , synchronous distributed training via data parallelism ( Hillis & Steele Jr , 1986 ) dominates . As a general abstraction , this method implements the following : at step k , worker n has a copy of the parameters , θkn , calculates the gradient , ∂θ k n , and updates θ via θk+1n = ParamUpdate ( θkn , AllReduce ( ∂θk1 , . . . , ∂θ k N ) ) = ParamUpdate ( θkn , 1 N N∑ i=1 ∂θki ) , ( 3 ) where ParamUpdate is any first-order optimization technique ( e.g . gradient descent ) and AllReduce performs a reduction ( e.g . mean ) over all copies of a variable and returns the result to all workers . Distributed DataParallel scales very well ( near-linear scaling up to 32,000 GPUs ( Kurth et al. , 2018 ) ) , and is reasonably simple to implement ( all workers synchronously running identical code ) . We adapt this to on-policy RL as follows : At step k , a worker n has a copy of the parameters θkn ; it gathers experience ( rollout ) using πθkn , calculates the parameter-gradients∇θ via any policy-gradient method ( e.g . PPO ) , synchronizes these gradients with other workers , and updates the model : θk+1n = ParamUpdate ( θkn , AllReduce ( ∇θJ PPO ( θk1 ) , . . . , ∇θJ PPO ( θkN ) ) ) . ( 4 ) A key challenge to using this method in RL is variability in experience collection run-time . In supervised learning , all gradient computations take approximately the same time . In RL , some resourceintensive environments can take significantly longer to simulate . This introduces significant synchronization overhead as every worker must wait for the slowest to finish collecting experience . To combat this , we introduce a preemption threshold where the rollout collection stage of these stragglers is preempted ( forced to end early ) once some percentage , p % , ( we find 60 % to work well ) of the other workers are finished collecting their rollout ; thereby dramatically improving scaling . We weigh all worker ’ s contributions to the loss equally and limit the minimum number of steps before preemption to one-fourth the maximum to ensure all environments contribute to learning . While we only examined our method with PPO , other on-policy RL algorithms can easily be used and we believe the method can be adapted to off -policy RL algorithms . Off-policy RL algorithms also alternate between experience collection and optimization , but differ in how experience is collected/used and the parameter update rule . Our adaptations simply add synchronization to the optimization stage and a preemption to the experience collection stage . Implementation . We leverage PyTorch ’ s ( Paszke et al. , 2017 ) DistributedDataParallel to synchronize gradients , and TCPStore – a simple distributed key-value storage – to track how many workers have finished collecting experience . See Apx . E for a detailed description with code .

Summary: The contribution of this paper is twofold. First the authors propose and implement and new scalable RL training procedure they call 'Decentralized Distributed Proximal Policy Optimization' (DD-PPO). This method extends PPO in a way that is 'synchronous' and 'distributed': no parameter server exists to asynchronously collect experience from worker agents and instead an 'explicit communication state' exists during which all workers communicate gradient updates between one another to update parameters based on their experiences. However, naively implementing this approach is limited by 'stragglers', the slowest of the workers that all other agents need to wait for. To overcome this limitation, the authors add a 'preemption threshold' which halts worker rollouts after a certain percentage has completed. The authors use this procedure to tackle the task of PointGoal navigation, in which an agent tries to reach a point in space specified by its relative location to the agent. They achieve state-of-the-art performance (arguably 'mastery') on this task. Finally, the authors show that their learned policy transfers reasonably well to two other navigation tasks, 'flee' and 'exploration'. Code is also provided.