Large Batch Simulation for Deep Reinforcement Learning

1 INTRODUCTION . Speed matters . It is now common for modern reinforcement learning ( RL ) algorithms leveraging deep neural networks ( DNNs ) to require billions of samples of experience from simulated environments ( Wijmans et al. , 2020 ; Petrenko et al. , 2020 ; OpenAI et al. , 2019 ; Silver et al. , 2017 ; Vinyals et al. , 2019 ) . For embodied AI tasks such as visual navigation , where the ultimate goal for learned policies is deployment in the real world , learning from realistic simulations is important for successful transfer of learned policies to physical robots . In these cases simulators must render detailed 3D scenes and simulate agent interaction with complex environments ( Kolve et al. , 2017 ; Dosovitskiy et al. , 2017 ; Savva et al. , 2019 ; Xia et al. , 2020 ; Gan et al. , 2020 ) . Evaluating and training a DNN on billions of simulated samples is computationally expensive . For instance , the DD-PPO system ( Wijmans et al. , 2020 ) used 64 GPUs over three days to learn from 2.5 billion frames of experience and achieve near-perfect PointGoal navigation in 3D scanned environments of indoor spaces . At an even larger distributed training scale , OpenAI Five used over 50,000 CPUs and 1000 GPUs to train Dota 2 agents ( OpenAI et al. , 2019 ) . Unfortunately , experiments at this scale are out of reach for most researchers . This problem will only grow worse as the field explores more complex tasks in more detailed environments . Many efforts to accelerate deep RL focus on improving the efficiency of DNN evaluation and training – e.g. , by “ centralizing ” computations to facilitate efficient batch execution on GPUs or TPUs ( Espeholt et al. , 2020 ; Petrenko et al. , 2020 ) or by parallelizing across GPUs ( Wijmans et al. , 2020 ) . However , most RL platforms still accelerate environment simulation by running many copies of off-the-shelf , unmodified simulators , such as simulators designed for video game engines ( Bellemare et al. , 2013 ; Kempka et al. , 2016 ; Beattie et al. , 2016 ; Weihs et al. , 2020 ) , on large numbers ∗Correspondence to bps @ cs.stanford.edu of CPUs or GPUs . This approach is a simple and productive way to improve simulation throughput , but it makes inefficient use of computation resources . For example , when rendering complex environments ( Kolve et al. , 2017 ; Savva et al. , 2019 ; Xia et al. , 2018 ) , a single simulator instance might consume gigabytes of GPU memory , limiting the total number of instances to far below the parallelism afforded by the machine . Further , running many simulator instances ( in particular when they are distributed across machines ) can introduce overhead in synchronization and communication with other components of the RL system . Inefficient environment simulation is a major reason RL platforms typically require scale-out parallelism to achieve high end-to-end system throughput . In this paper , we crack open the simulation black box and take a holistic approach to co-designing a 3D renderer , simulator , and RL training system . Our key contribution is batch simulation for RL : designing high-throughput simulators that accept large batches of requests as input ( aggregated across different environments , potentially with different assets ) and efficiently execute the entire batch at once . Exposing work en masse facilitates a number of optimizations : we reduce memory footprint by sharing scene assets ( geometry and textures ) across rendering requests ( enabling orders of magnitude more environments to be rendered simultaneously on a single GPU ) , amortize rendering work using GPU commands that draw triangles from multiple scenes at once , hide latency of scene I/O , and exploit batch transfer to reduce data communication and synchronization costs between the simulator , DNN inference , and training . To further improve end-to-end RL speedups , the DNN workload must be optimized to match high simulation throughput , so we design a computationally efficient policy DNN that still achieves high task performance in our experiments . Large-batch simulation increases the number of samples collected per training iteration , so we also employ techniques from large-batch supervised learning to maintain sample efficiency in this regime . We evaluate batch simulation on the task of PointGoal navigation ( Anderson et al. , 2018 ) in 3D scanned Gibson and Matterport3D environments , and show that end-to-end optimization of batched rendering , simulation , inference , and training yields a 110× speedup over state-of-the-art prior systems , while achieving 97 % of the task performance for depth-sensor-driven agents and 91 % for RGB-camera-driven agents . Concretely , we demonstrate sample generation and training at over 19,000 frames of experience per second on a single GPU.1 In real-world terms , a single GPU is capable of training a virtual agent on 26 years of experience in a single day.2 This new performance regime significantly improves the accessibility and efficiency of RL research in realistic 3D environments , and opens new possibilities for more complex embodied tasks in the future . 2 RELATED WORK . Systems for high-performance RL . Existing systems for high-performance RL have primarily focused on improving the efficiency of DNN components of the workload ( policy inference and optimization ) and use a simulator designed for efficient single agent simulation as a black box . For example , Impala and Ape-X used multiple worker processes to asynchronously collect experience for a centralized learner ( Espeholt et al. , 2018 ; Horgan et al. , 2018 ) . SEED RL and Sample Factory built upon this idea and introduced inference workers that centralize network inference , thereby allowing it to be accelerated by GPUs or TPUs ( Espeholt et al. , 2020 ; Petrenko et al. , 2020 ) . DD-PPO proposed a synchronous distributed system for similar purposes ( Wijmans et al. , 2020 ) . A number 1Samples of experience used for learning , not ‘ frameskipped ’ metrics typically used in Atari/DMLab . 2Calculated on rate a physical robot ( LoCoBot ( Carnegie Mellon University , 2019 ) ) collects observations when operating constantly at maximum speed ( 0.5 m/s ) and capturing 1 frame every 0.25m . of efficient implementations of these ideas have been proposed as part of RL frameworks or in other deep learning libraries ( Liang et al. , 2018 ; Stooke & Abbeel , 2019 ; Küttler et al. , 2019 ) . We extend the idea of centralizing inference and learning to simulation by cracking open the simulator black box and designing a new simulation architecture for RL workloads . Our large-batch simulator is a drop-in replacement for large numbers of ( non-batched ) simulation workers , making it synergistic with existing asynchronous and synchronous distributed training schemes . It reduces the number of processes and communication overhead needed for asynchronous methods and eliminates separate simulation worker processes altogether for synchronous methods . We demonstrate this by combining our system with DD-PPO ( Wijmans et al. , 2020 ) . Concurrently with our work , CuLE , a GPU-accelerated reimplementation of the Atari Learning Environment ( ALE ) , demonstrates the benefits of centralized batch simulation ( Dalton et al. , 2020 ) . While both our work and CuLE enable wide-batch execution of their respective simulation workloads , our focus is on high-performance batch rendering of complex 3D environments . This involves optimizations ( GPU-driven pipelined geometry culling , 3D asset sharing , and asynchronous data transfer ) not addressed by CuLE due to the simplicity of rendering Atari-like environments . Additionally , like CuLE , we observe that the large training batches produced by batch simulation reduce RL sample efficiency . Our work goes further and leverages large-batch optimization techniques from the supervised learning literature to mitigate the loss of sample efficiency without shrinking batch size . Large mini-batch optimization . A consequence of large batch simulation is that more experience is collected between gradient updates . This provides the opportunity to accelerate learning via large mini-batch optimization . In supervised learning , using large mini-batches during optimization typically decreases the generalization performance of models ( Keskar et al. , 2017 ) . Goyal et al . ( 2017 ) demonstrated that model performance can be improved by scaling the learning rate proportionally with the batch size and “ warming-up ” the learning rate at the start of training . You et al . ( 2017 ) proposed an optimizer modification , LARS , that adaptively scales the learning rate at each layer , and applied it to SGD to improve generalization further . In reinforcement learning and natural language processing , the Adam optimizer ( Kingma & Ba , 2015 ) is often used instead of SGD . Lamb ( You et al. , 2020 ) combines LARS ( You et al. , 2017 ) with Adam ( Kingma & Ba , 2015 ) . We do not find that large mini-batch optimization harms generalization in reinforcement learning , but we do find it decreases sample efficiency . We adapt the techniques proposed above – learning rate scaling ( You et al. , 2017 ) and the Lamb optimizer ( You et al. , 2020 ) – to improve sample efficiency . Simulators for machine learning . Platforms for simulating realistic environments for model training fall into two broad categories : those built on top of pre-existing game engines ( Kolve et al. , 2017 ; Dosovitskiy et al. , 2017 ; Lee et al. , 2019 ; Gan et al. , 2020 ; James et al. , 2020 ) , and those built from scratch using open-source 3D graphics and physics libraries ( Savva et al. , 2017 ; 2019 ; Xia et al. , 2018 ; 2020 ; Xiang et al. , 2020 ; Zeng et al. , 2020 ) . While improving simulator performance has been a focus of this line of work , it has been evaluated in a narrow sense ( i.e . frame rate benchmarks for predetermined agent trajectories ) , not accounting for the overall performance of end-to-end RL training . We instead take a holistic approach to co-design rendering and simulation modules and their interfaces to the RL training system , obtaining significant gains in end-to-end throughput over the state of the art . 3 SYSTEM DESIGN & IMPLEMENTATION . Batch simulation accelerates rollout generation during RL training by processing many simulated environments simultaneously in large batches . Fig . 2 illustrates how batch simulation interacts with policy inference to generate rollouts . Simulation for sensorymotor agents , such as the PointGoal navigation task targeted by our implementation , can be separated into two tasks : determining the next environment state given an agent ’ s actions and rendering its sensory observations . Therefore , our design utilizes two components : a batch simulator that performs geodesic distance and navigation mesh ( Snook , 2000 ) computations on the CPU , and a batch renderer that renders complex 3D environments on the GPU . During rollout generation , batches of requests are passed between these components – given N agents , the simulator produces a batch of N environment states . Next , the renderer processes the batch of environment states by simultaneously rendering N frames and exposing the result directly in GPU memory . Agent observations ( from both the simulator and the renderer ) are then provided as a batch to policy inference to determine the next actions for the N agents . The key idea is that the batch simulator and renderer implementations ( in addition to the DNN workload ) take responsibility for their own parallelization . Large batch sizes ( values of N on the order of hundreds to thousands of environments ) provide opportunities for implementations to efficiently utilize parallel execution resources ( e.g. , GPUs ) as well as amortize processing , synchronization , and data communication costs across many environments . The remainder of this section describes the design and key implementation details of our system ’ s batch simulator and batch renderer , as well as contributions that improve the efficiency of policy inference and optimization in this regime .

Summary: This paper shows that batch simulation can accelerate reinforcement learning in 3D environments. Batch simulation accepts and executes large batches of simulation requests at the same time on one accelerator. The authors demonstrate that this technique can substantially speed up the processing and achieve ~100x speed up in convergence. They also propose minor-changes to DD-PPO to speed up the convergence even further. The authors also included the code which is always appreciated.