VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement
Authors: Erik Wijmans
Irfan Essa
Dhruv Batra
Venue: NeurIPS 2022 Accept (2022)

Abstract: We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning in heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). Specifically, it learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL) enabling high throughput.

We find that VER leads to significant and consistent speed-ups across a broad range of embodied navigation and mobile manipulation tasks in photorealistic 3D simulation environments. Specifically, for PointGoal navigation and ObjectGoal navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO, the current state of art for distributed SyncOnRL, with similar sample efficiency. For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup) on 1 GPU and 170% faster (2.7x speedup) on 8 GPUs than DD-PPO. Compared to SampleFactory (the current state-of-the-art AsyncOnRL), VER matches its speed on 1 GPU, and is 70% faster (1.7x speedup) on 8 GPUs with better sample efficiency.

We leverage these speed-ups to train chained skills for GeometricGoal rearrangement tasks in the Home Assistant Benchmark (HAB). We find a surprising emergence of navigation in skills that do not ostensible require any navigation. Specifically, the Pick skill involves a robot picking an object from a table. During training the robot was always spawned close to the table and never needed to navigate. However, we find that if base movement is part of the action space, the robot learns to navigate then pick an object in new environments with 50% success, demonstrating surprisingly high out-of-distribution generalization.

VER: Scaling On-Policy RL Leads to the Emergence of Navigation in Embodied Rearrangement
Erik Wijmans 1.2
Irfan Essa 1.3
Dhruy Batra
Georgia Institute of Technology Meta Al (etw, irfan, dbatra)@gatech.cemm
3 Googge Atlanta
Abstract
We present Variable Experience Rollout (VER), a technique for efficiently scaling batched on-policy reinforcement learning heterogenous environments (where different environments take vastly different times to generate rollouts) to many GPUs residing on, potentially, many machines. VER combines the strengths of and blurs the line between synchronous and asynchronous on-policy RL methods (SyncOnRL and AsyncOnRL, respectively). Specifically, it learns from on-policy experience (like SyncOnRL) and has no synchronization points (like AsyncOnRL) enabling high throughput.
We find that VER leads to significant and consistent speed-ups across a broad range of embodied navigation and mobile manipulation tasks in photorealistic 3D simulation environments. Specifically, for PointGoal navigation and ObjectGoal navigation in Habitat 1.0, VER is 60-100% faster (1.6-2x speedup) than DD-PPO, the current state of art for distributed SyncOnRL. with similar sample efficiency. For mobile manipulation tasks (open fridge/cabinet, pick/place objects) in Habitat 2.0 VER is 150% faster (2.5x speedup) on I GPU and 170% faster (2.7x speedup) 8 GPUs than DD-PPO. Compared to SampleFactory (the current state-of-the-ard AsyncOnRL), VER matches its speed on GPU, and is 70% faster (1.7x speedup) 8 GPUs with better sample efficiency.
We leverage these speed-ups to train chained skills for GeometricGoal rear- rangement tasks in the Home Assistant Benchmark (HAB). We find surprising emergence of navigation in skills that do not ostensible require any navigation. Specifically, the Pick skill involves robot picking an object from a table. During training the robot was always spawned close to the table and never needed to navigate. However, we find that if base movement is part of the action space, the robot learns to navigate then pick an object in new environments with 50% success, demonstrating surprisingly high disttribbitiitoo generalization. Code: gittub.com/ffcebookkccccccoc..ce htt..///cccooooeeer///abiitataat
Introduction
Scaling matters. Progress towards building embodied intelligent agents that capable of performing goal driven tasks has been driven, in part, by training large neural networks in photo-realistic 3D envi- ronments with deep reinforcement learning (RL) for (up to) billions of steps of experience [Wijmans et al., 2020, Maksymets et al., 2021, Mezghani et al., 2021, Ramakrishnan et al., 2021, Miki et al., 2022]. To enable this scale, RL systems must be able to efficiently utilize the available resources (e.g. GPUs), and scale to multiple machines all while maintaining sample-efficient learning.
One promising class of techniques to achieve this scale is batched on-policy RL. These methods collect experience from many (N) environments simultancously using the policy and update it with this cumulative experience. They are broadly divided into two classes: synchronous (SyncOnRL)
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Tily Hode
SynoOnRL
VER (Ours)
AsyncOnRL
300%
5
B
Learning
Learning
E
Learning
Policy Inference
Simulation
Time
Figure I: (Left) RL Training Systems. In SyncOnRL, actions are computed for all environments, then all environments are stepped. Experience collection is paused during learning. In AsyncOnRL, computing actions, stepping environments, and learning all occur without synchronization. In VER, variable amount of experience is collected from each environment, enabling synchronous learning without the straggler effect. (Right) skill policies with navigation are more robust handoff errors.
and asynchronous (AsyncOnRL). SyncOnRL contains two synchronization points: first the policy is executed for the entire batch (or + (4) ue)b (Fig. 1 A), then actions are executed in all environments, (82, 4z - St+1, 1= (Fig. B), until T steps have been collected from all N environments. This (T, -shhapedd batch of experience used to update the policy (Fig. 1 C). Synchronization reduces throughput as the system spends significant (sometimes the most) time waiting for the slowest environment to finish. This is the straggler effect [Petrini et al., 2003, Dean and Ghemawat, 2004].
AsyncOnRL removes these synchronization points, thereby mitigating the straggler effect and improv- ing throughput. Actions taken as soon as they are computed, (₁ -> Ot+1 (Fig. 1 D), the next action is computed as soon as the observation is ready, - ae (Fig. E), and the policy is updated as soon as enough experience collected. However, AsyncOnRL systems are not able to ensure that all experience has been collected by only the current policy and thus must consume near-policy data. This reduces sample efficiency [Liu et al., 2020]. Thus, status quo leaves us with an unpleasant tradeoff - high sample-efficieng with low throughput or high throughput with low sample-efficiency.
In this work, we propose Variable Experience Rollout (VER). VER combines the strengths of and blurs the line between SyncOnRL and AsyncOnRL. Like SyncOnRL, VER collects experience with the current policy and then updates it. Like AsyncOnRL, VER does not have synchronization points it computes next actions, steps environments, and updates the policy as soon as possible. The inspiration for VER comes from two key observations:
1) AsyncOnRL mitigates the straggler effect by implicitly collecting a variable amount of experience from each environment more from fast-to- fast-to-simulate environments and less from slow ones.
2) Both SyncOnRL and AsyncOnRL use a fixed rollout length, steps of experience. Our key insight is that while fixed rollout length may simplify an implementation, it is not requirement for RL.
These two key observations naturally lead rollouts with a variable number of steps. VER adjusts the rollout length for each environment based on its simulation speed. It explicitly collects more experience from fast-to-simulate environments and less from slow ones (Fig. 1). The result is an RL system that overcomes the straggler effect and maintains sample-efficiency by learning from on-policy data.
variable experience rollout (VER), i.e. collecting
VER focuses on efficiently utilizing a single GPU. To enable efficient scaling to multiple GPUs, we combine VER with the decentralized distributed method proposed in [Wijmans et al., 2020].
First, we evaluate VER on establishl embodied navigation tasks using Habitat 1.0 [Savva et al., 2019] on 8 GPUs VER trains PointGoal navigation [Anderson et al., 2018] 60% faster than Decen- tralized Distributed PPO (DD-PPO) [Wijmans et al., 2020], the current state-of-the-art for distributed on-policy RL, with the same sample efficiency. For ObjectGool navigation [Batra et al., 2020b), an active area of research, VER is 100% faster than DD-PPO with (slightly) better sample efficiency.
Next, we evaluate VER on the recently introduced (and significantly more challenging) GeometricGoal rearrangement tasks [Batra al., 2020a] in Habitat 2.0 [Szot et al., 2021]. In GeoRearrange, virtual robot is spawned in new environment and asked to rearrange set of objects from their initial to
Following standard notation, st is (PO)MDP state, ar is the action taken, and or is the agent observation.

---

EnvStep
Shared CPU Memory
Batch(EnvStep)
Environment Worker
Action
N
Batch(Actions)
Inference Worker
Policy Weights
Policy Weights
Learning Worker
Shared GPU Memory
Batch(EnvStep)
Mini-batch
Figure 2: VER system architecture. Environment workers receive actions to simulate and return the result of that environment step (EnvStep). Inference workers receive batches experience from environment workers. They return the new action to take to environment workers and write the experience into GPU shared memory for learning.
desired coordinates. These environments have highly variable simulation time (physics simulation time increases if the robot bumps into something) and require GPU-acceleration (for photo-realistic rendering), limiting the number of environments that can be run in parallel.
On 1 GPU, VER is 150% faster (2.5x speedup) than DD-PPO with the same sample efficiency. VER as fast as SampleFactory [Petrenko et al., 2020], the state-of-the-art AsyncOnRL, with the same sample efficiency. VER is as fast AsyncOnRL in pure throughput; this is surprisingly strong result. AsyncOnRL never stops collecting experience and should, in theory, be a strict upper bound on performance. VER is able to match AsyncOnRL for environments that heavily utilize the GPU for rendering, like Habitat. In AsyncOnRL, learning, inference, and rendering contend the GPU which reduces throughput. In VER, inference and rendering contend for the GPU while learning does not.
On 8 GPUs, VER achieves better scaling than DD-PPO, achieving a 6x speed-up (vs. 6x for DD-PPO) due to lower variance in experience collection time between GPU-workers. Due to this efficient multi-GPU scaling, VER is 70% faster (1.7x speedup) than SampleFactory on GPUs and has better sample efficiency as it learns from on-policy data.
Finally, leverage these SysML contributions study open research questions posed in prior work. Specifically, we train RL policies for mobile manipulation skills (Navigate, Pick, Place, etc.) and chain them via task planner. Szot al. [2021] called this approach TP-SRL and identified a critical "handoff problem' - downstream skills are set up for failure by small errors made by upstream skills (e.g. the Pick skill failing because the navigation skill stopped the robot bit too far from the object )
We demonstrate a number of surprising findings when TP-SRL is scaled via VER. Most importantly, find the emergence of navigation when skills that do not ostensibly require navigation (e.g. pick) are trained with navigation actions enabled. In principle, Pick and Place policies do not need to navigate during training since the objects always in arm's reach, but in practice they learn to navigate to recover from their mistakes and this results in strong out-of-distribution test-time generalization. Specifically, TP-SRL without a navigation skill achieves 50% success NavPick and 20% success on a Nav www.PPiiaylaaac. task simply because the Pick and Place skills have learned navigate (sometimes across the room!). TP-SRL with Navigate skill performs even stronger: 90% on NavPickNavPlacc and 32% on successive CiPPickNavPlacee (called Tidy House in Szot al. [2021]), which are +32% and +30% absolute improvements Szot al. [2021], respectively. Prepare Groceries and Set Table, which both require interaction with articulated receptacles (fridge, drawer), remain as open problems (5% and 0% Success. respectively) and are the next frontiers.
2 VER: Variable Experience Rollout
The key challenge that any batched on-policy RL technique needs to address is variability of sim- ulation time for the environments in a batch. There two primary sources of this variability: action-level and episode-level. The amount of time needed to simulate an action within an envi- ronment varies depending the the specific action, the state of the robot, and the environment (e.g. simulating the robot navigating on a clear floor is much faster than simulating the robot's 's arm colliding with objects). The amount of time needed to simulate an entire episode also varies environ- ment to environment irrespective of action-level variability (e.g. rendering images takes longer for visually-complex scenes, simulating physics takes longer for scenes with large number of objects).
3

---

Vari Variable Experience Rollout
Datchiing
Packed Sequence Format
IAQ
0
0
MxX
And
T12
Figure 3: (A) VER collects variable amount of experience from each environment. The length of each step represents the time taken to collect it. (B) VER mini-batch. The solid bars denote episode boundaries. The steps selected for the first mini-batch have a dashed border. (C) The PackedSequence data format represents a set of sequences with variable length in a linear buffer such that all elements from each timestep area next one-another in memory.
2.1 Action-Level Straggler Mitigation
We mitigate the action-level straggler effect by applying the experience collection method of AsyncOnRL to SyncOnRL. We represent this visually in Fig. and describe it in text bellow.
Environment workers receive the next action and step the environment, EnvStep, e.g, St, Ot. 8141, 84+1,0++1,7 They write the outputs of the environment (observations, reward, etc.) into pre- allocated CPU shared memory for consumption by inference workers.
Inference workers receive batches of steps of experience from environment workers. They perform inference with the current policy to select the next action and send it to the environment worker using pre-allocated CPU shared memory. After inference they store experience for learning in shared GPU memory. Inference workers requests instead of waiting for a fixed number of requests to arrive?. This allows us to leverage the benefits of batching without introducing synchronization points between environment workers.
dynamic batching and perform inference all outstanding inference
This experience collection technique similar to that of HTS-RL [Liu et al., 2020] (SyncOnRL) and SampleFactory [Petrenko et al., 2020] (AsyncOnRL). Unlike both, we do not overlap experience collection with learning. This has various system benefits, including reducing GPU memory usage and reducing GPU driver contention. More details are available in Appendix A.
2.2 Environment-Leevel Straggler Mitigation
In both SyncOnRL and AsyncOnRL. the data used for learning consists of N rollouts of equal-T steps of experience, an (T, N)) -shappe batch. In SyncOnRL these sets are all collected with the current policy, this leads to the environment-leww constraint that experience must be strictly on-policy, and thereby implicitly changes the experience collection rate for each environment.
straggler effect. AsyncOnRL mitigates this by relaxing the
Variable Experience Rollout (VER). We instead relax the constraint that we must use N rollouts of equal-T steps. Specifically, VER collects T' x N steps of experience from N environments without a constraint on how many steps of experience are collected from each environment. This explicitly varies the experience collection rate for each environment in effect, collecting more experience from environments that are fast to simulate. Consider the 4 environments shows in Fig. 3A. The length of the each step representation the wall-clock time taken to collect it, some steps fast, some are slow. VER collects more experience from environment as is fastest to step and less from 1, the slowest.
Learning mini-batel creation. VER is designed with recurrent policies in mind because memory key in long-range and partially observable tasks like HAB. When training recurrent policies, we must create mini-batches of experience with sequences for Back-propagationnttrooug B bathhe are constructed by spitting the N environments' experience into B (T, N/BB)-size mini-batches. A similar procedure would result in mini-batches of different sizes with VER, This would harm optimization because learning rate and optimization mini-batch size are intertwined and automatically adjusting the learning rate is open question [Goyal al., 2017, You et al., 2020].
Normally
To under VER's mini-batching batching, first note that there are two reasons for the start of new sequence of experience: rollout starts (Fig. 3B, step 0) and episode starts (Fig. 3B, a step after a bar). These
In practice we introduce both a minimum and maximum number of requests to prevent under atdlizztilization of compute and over- utillizzzi of memory.

---

two boundaries types are independent episodes can end at any arbitrary step within the rollout and then that environment will reset and start a new episode. Thus when we collect experience from N environments, we will have K 2 N sequences to divide between the mini-batches. We distributed these K sequences between the mini-batches. We randomly order the sequences, then the first T N/B steps are the first mini-batch, the next T to the second, etc. See Fig. 3B for an example.
Batching computation for learning. The mini-batches constructed from the algorithm above have sequences with variable length. To batch the computation of these sequences we use CuDNN's PackedSequence data model. This data model represents a set of variable-length sequences (Fig. 3C left) such that all eereenceeeemeets at a time-step are contiguous in memory (Fig. 3C right) enabling batched computation on each time-step for components with temporal dependence, e.g. the RNN and that all elements across all time-steps are also contiguous enabling batched computation across all time-steps for network components that don't have a temporal dependence, e.g. the visual encoder.
During experience collection we write experience into a linear buffer in GPU memory and then arrange each mini-batch as a PackedSequence. This takes less than 10 milliseconds (per learning phase); orders of magnitude less than experience collection (~3s) or learning (~11.5)) in our experiments.
Learning method. The experience and mini-batches generated from VER are well-suited for use with RL methods that use on-policy data [Sutton and Barto, 1992]. We use Proximal Policy Optimization (PPO) [Schulman et al., 2017] as it is known to work well for embodied AI tasks and recurrent policies.
Inflight actions One subtle design choice is the following - when VER finishes a T x N experience collection, there will be (slow) environments that haven't completed simulation yet. Instead of discarding that data, we choose to collect this experience in the next rollout. We find this choice leads to speed gains without any sample-effiiency loss.
2.3 Multiple GPUs
We leverage the decentralized distributed training architecture from Wijmans et al. [2020] to scale VER to multiple GPUs (residing on a single or multiple nodes). In this architecture, each GPU both collects experience and learns from that experience. During learning, gradients from each GPU-worker are averaged with an AllReduce operation. This is synchronous operation and thus introduces a GPU-worker preempting stagglers stopping collecting experience early and proceeding to learning after fixed number of GPU-workers have finished experience collection.
straggler effect. Wijmans et al. [2020] mitigate this effect by
We instead approximate the optimal preemption for each experience collection phase. Given the learning time, LT, and function Time(S) that returns the time needed collect S steps of experience. the optimal number of steps S to collect before preempting stragglers is maxş (/(Tine(S)+LL), s.t. S < (T - #GPUs). For LT, we record the time from the last iteration as this doesn't 't change between iterations. We estimate Time(S) (S) using the average step time to receive step of experience from each environment (this includes both inference time and simulation time) from the previous rollout. This will be poor estimation in some circumstances but we find it to work well in practice.
We improve upon sample efficiency of DD-PPO by filling the preempted rollouts with experience from the previous rollout. We perform extra epochs of PPO on this now "stale' data instead of correcting for the off-policy return as we find this simpler and effective. This comes with no effective computation cost since the maximum batch size per GPU is unchanged.
3 Embodied Navigation: Benchmarking
First, we benchmark VER on the embodied navigation [Anderson al., 2018] tasks in Habitat 1.0 [Savva et al., goal here is simply to show training speed-ups in well-studied tasks (and in the case of PointNav, well-saturated task with no room left for accuracy improvements). We present accuracy improvements and in-depth analysis on challenging rearrangement tasks in Section 6.
PointNav [Anderson et al., 2018] and ObjectNav [Batra et al., 2020b]. Our
For both tasks, we use standard architectures from Habitat Baselines [Savva et al., 2019, Wijmans et al., 2020] the ResNet18 encoder and 2 layer LSTM. Following Ye et al. [2020, 2021], we add Action Conditional Contrastive Coding [Guo et al., 2018].
PointNav. We train PointNav agents with one RGB camera the HM3D dataset [Ramakrishnan et al., 2021] for 1.85 billion steps of experience on GPUs. We study the RGB setting (and not Depth) because this is the more challenging version of the task and thus we expect it to be more sensitive to
5

---

(A)
(B)
(C)
103-
PointCioal Navigution
Objecttiooa Navigation
Pointscal Navigation
ObjectGoal Navigation 4.3x tyste
fast
Succets
Taok
DD-PPO
VER
Success
Surcess
47
PointGoolNa
1065
5823
-0
96
ObjectNav
1015
2019
Method
05
3D-FPO
VIB (Oun)
Time (Days)
Time (Days)
83
Experience (Steps, Milloon))
0 500
1050
15:00
Experience (Stops, Millonss
2:00
400
600
CPU Time (Days)
20
20 GPU Time (Days)
Figure 4: (A) Navigation Tasks training steps per second on 8 GPUs. VER is 60%- faster than DD-PPO. (B) Sample efficiency on ObjectNav and PointNav performance (validation success). VER has similar or slightly better sample efficiency than DD-PPO, indicating that performance is not negatively impacted by the non-uniform sampling of experience from environments. Shading is a 95% confidence interval over 3 seeds. (C) Compute efficiency on ObjectNav and PointNav performance (validation success). To reach the maximum success achieved by DD-PPO, 97.4% on PointNav and 13.0% on ObjectNav, VER uses 1 1.6x less compute on PointNav (saving 16 GPU-days) and 4.3x less compute on ObjectNav (saving 33.4 4 GPU-days).
possible differences in the training system. We examine VER along two axes: 1) training throughput the number of samples of experience per second (SPS) the system collects and learns from, 2) sample efficiency. On 8 GPUs, VER trains agents 60% faster than DD-PPO, from 3065 SPS to 5325 SPS (Fig. 4A), with similar sample efficiency (Fig. 4B).
ObjectNav. We train ObjectNav agents with one RGB and one Depth camera on the MP3D [Chang et al., 2017] dataset for 600 million steps of experience on 8 GPUs. VER trains agents 100% faster than DD-PPO, 1021 to 2019 (Fig. 4A), with slightly better sample efficiency (Fig. 4B). Due to improved throughput and sample efficiencyy VER uses 4.3x less compute than DD-PPO to reach the same success (Fig. 4C). There are two effects that enable better sample efficiency with VER. First, we perform additional epochs of PPO on experience from the last rollout when GPU-worker is preempted. Second, the variable experience rollout mechanism results in a natural curriculum. Environments often faster to simulate when they are casier (i.e. smaller home), more experience will be collected in these easier cases.
4 Embodied Rearrangement: Task, Agent, and Training
Next, we use VER to study the recently introduced (and more challenging) GeometricGoal rearrange- ment rearrangement tasks [Batra et al., 2020a] in Habitat 2.0 [Szot et al., 2021].
Task. In GeoRearrange, an agent initialized in an unknown environment and tasked with rearranging objects in its environment. The task is specified as a set of coordinate pairs { [(Poeemiiit, Posérinal) J= The agent must bring each object at Posemitial to Posefinal where Pose is the initial or desired center- of-mass location for the object(s). We use the Home Assistant Benchmark (HAB) which consists of 3 scenarios of increasing difficulty: Tidy House, Prepare Groceries, and Set Table.
In Tidy House, the agent is tasked with moving 5 objects from their initial locations to their final locations. The objects are rarely in containers (i.e. fridge or cabinet drawer) and when they are, the containers are already opened. In Prepare Groceries, the agent must move 3 objects from the kitchen counter into the open fridge (or open fridge counter). This stresses picking and placing in the fridge, which is challenging. In Set Table, the agent must move object from the closed kitchen cabinet drawer to the table and 1 object from the closed fridge the table. This requires opening the cabinet and fridge, and picking from these challenging receptacles.
Simulation. We use the Habitat simulator with the ReplicaCAD Dataset [Savva al., 2019, Szot et al., 2021]. The robot policy operates at 30 Hz and physics is simulated at 120 Hz.
Agent. The agent is embodied as Fetch robot with a 7-DOF arm. The arm is controlled via joint velocities. At every time step the policy predicts delta in motor position for joint in the arm. We find joint velocity control equally easy to learn but faster to simulate than the end-effector control used in Szot et al. [2021]. The arm is equipped with a suction gripper. The agent must control the arm such that the gripper is in contact with the object to grasp and then activate the gripper. The object is dropped once the gripper is deactivated. This more realistic than the 'magic' grasp action used in Szot et al. 1]. The robot base (navigation) is controlled by the policy commanding a desired linear speed and

---

DD-PPO
NoVER
VER (Ours)
SampleFactory
GPUs
Mean
Max
Mean
Max
Mean
Max
Mean
Max
174_
442.19
327
428 1:1
428
534
427
517.3
283
696
592-:
786:11
716::32
945
804=15
1022+t
468
1337:134
1097:430
1601 1:50
1432
1915
1286
1568
1066
1754:1156
2216::60
3438
2861
3829
1662
1842.0
Table I: SyncOnRL, VER, and AsyncOnRL benchmarking, Mean/max system throughput (SPS) over 20 million training steps. VER is 150% faster than DD-PPO on I GPU and 170% faster on GPUs. Hardware: Tesla V100(s) with 10 CPUs per GPU.
angular velocity. The robot is equipped with one Depth camera attached to its head, proprioceptive sensors that provide the joint positions of its arm, and a GPS+Compass sensor that provides its heading and location relative to its initial location. The policy models a ~ T(- 3;-1) instead of a ~ T(: s.). This is more realistic and enables physics and rendering to be overlapped [Szot et al., 2021].
We build upon the YaskPllnning-Skiii SkillIRL (TP-SRL) method proposed in Szot et al. [2021]. TP-SRL is a hierarchical method for GeoRearrange that decomposes the task into a series of skills Navigate, Pick, Place, and [Open, Close } x (Cabinet, Fridge). Skills are controlled via a skill-policy (learned with RL) and chained together via task planner. One of the key challenges is the 'handoff problem' downstream skills are setup for failure due to slight errors made by the upstream skill. We give all skill policies access to navigation actions to allow them to correct for these errors.
Architecture. All skill policies share the same architecture. We use ResNet18 [He et al., 2016] to process the 128x 128 visual input. Following Wijmans et al. [2020], we reduce with width of the network by half and use GroupNNrrm [Wu and He, 2018]. We also apply some of the recent advancements from ConvNeXt [Liu et al., 2022]. We use a patch-ify stem, dedicated down-sample stages, layer scale [Touvron et al., 2021], and dilated convolutions [Yu and Koltun, 2015] (this mimics larger kernel convolutions without increasing computation). The visual embedding is then combined with the proprioceptive observations and previous action, and then processed with a 2-layer LSTM [Hochreiter and Schmidhuber, 1997]. The output of the LSTM is used to predict the action distribution and value function. Actions are sampled from independent Gaussian distributions.
Training, We train agents using VER and PPO [Schulman et al., 2017] with Generalized Advan- tage Estimation [Schulman et al., 2016]. We use minimum entropy constraint with a learned coefficient [Haarnoja et al., 2018] as we find this to be more stable given our diverse set of skills than fixed coefficient. Formally, let H(T)) be the entropy of the policy, we then minimize (([((=)] - A) - ((()) where [f-1lag is the stop gradient operator. We set the target entropy, X, to zero for all tasks. We use the Adam optimizer [Kingma and Ba, 2015] with an initial learning rate of 2.5 x 10-4 and decay it to zero with cosine schedule. To correct for biased sampling in VER, we use truncated importance sampling weighting [Espeholt et al., 2018] with a maximum of 1.0.
5 Embodied Rearrangement: Benchmarking
In this section, we benchmark VER for training open-fridge policies because this task involves interaction of the robot with an articulated object (the fridge) and represents a challenging case for the training system due to large variability in physics time. In Section 6, we analyze the task performance, which requires all skills. For all systems, we set the number of environments, N, to 16 per GPU.
5.1 System throughout
VER is 150 % faster than DD-PPO (Table 1); an even larger difference than in the simpler navigation tasks we studied before. DD-PPO has no mechanism mitigate the action-level or episode-level straggler effects. In their absence, DD-PPO has similar throughput as VER (Table 1, Max). Under these effects, DD-PPO's throughput reduces 150% compared to 20% for VER (Table 1, Mean vs. Max).
Variable experience rollouts are effective. We compare VER with VER minus variable experience rollouts (NoVER). NoVER is a 'steel-manned" baseline for VER and benefits from all our micro- optimizations. VER is 30% faster than NOVER (Table 1).
VER closes the gap to AsyncOnRL. On 1 GPU, VER is as fast as SampleFactory [Petrenko et al., 2020], the fastest single machine AsyncOnRL. Intuitively AsyncOnRL should be a strict upper-bound on performance never stops collecting experience while VER does. However this doesn't 't take into
7

---

Success (Higher is Better)
100
1 CIPU
GPUS
GPUS
GPUS
90
DD-PP
(Curs)
80
Samplefactory
10 Experience (Steps Millions)
10 Experience (Steps Millions)
Experience (Steps Millions)
Experience (Steps Millions)
Success (Higher is Better)
GPUs
2,3x faster
2.7x Taster
2.7x faster
90
1.9k faster
VER (Durs)
80
SamplcFactory
70
20 Time (Hours)
10
30
5 10
10
2
Time (Hourrs
Time (Hours)
Time (Hours)
Figure 5: Training efficiency on Open Fridge. VER has similar sample efficiency as DD-PPO (SyncOnRL). SampleFactory (AsyncOnRL) has similar sample efficiency with GPU but this reduces as policy lag increases with more GPUs. On 8 GPUs, VER uses 2.7x less compute than DD-PPO to reach the same performance. This saves 32-days of GPU-time, per skill, for the full training schedule of 500 million frames. The shaded region is a 95% bootstrapped confidence interval over 5 seeds. We use interquartile mean (IQM) as our summary statistic [Agarwal et al., 2021].
account the realities of hardware. Recall that we are training an agent with a large visual encoder. This means that updating the parameters of the agent takes large amount of time (~150ms per mini-batch of size 1024 on V100). Further, Habitat uses the GPU for rendering. The use of the GPU for both rendering and learning simultaneously results in GPU driver contention. In SampleFactory, learning time and experience collection time are roughly double that of VER.
Multi-GPU scaling. VER has better multi-GPP scaling than DD-PPO, achieving a 6. 6.7x speed-up on 8 GPUs compared 6x. The rollout collection time in VER lower variance, which improves scaling. On 2 GPUs, SampleFactory is 12% faster than VER. Here one GPU is used for learning+inferent and the other is used for rendering. This creates a nice division of work and doesn't 't result in costly GPU contention. On 4 and 8 GPUs however, the single GPU used for learning in SampleFactory is the bottleneck and VER has higher throughput (nearly 100% faster on 8 GPUs). While it is possible to implement multi-GPU learning for AsyncOnRL, it is left to the user to balance the number of GPUs used for experience collection and learning. VER balances GPU-time between these automatically.
5.2 Sample and Compute Efficiency
Next we examine sample efficiency of the training systems. On 1 GPU, VER has slightly worse sample efficiency than DD-PPO (Fig. 5). We suspect that this could be fixed by hyper-parameter choice. On 2 and 4 GPUs, VER has identical sample efficiency and better sample efficiency on 8 GPUs, Fig. Interestingly, SampleFactory has similar sample efficiency with I GPU (Fig. 5) and is more stable. This is because reducing the number of GPUs also reduces the batch-size and PPO is known to not be batch-size invariant [Hilton et al., 2021] I]. We believe the stale data serves a similar function to PPO-EWMA [Hilton et al., 2021], which uses an exponential moving average of the policy weights for the trust region. On 2, 4, and GPUs, VER has better sample efficiency than SampleFactory. Combing the findings of throughput and sample efficiency, we find that VER always achieves given performance threshold in the least amount of training time (outright or a tie) (Fig. 5 Lower).
6 Embodied Rearrangement: Analysis of Learned Skills
We examine the performance of TP-SRL on the Home Assistant Benchmark (HAB) [Szot al., 2021]. We examine both skill policies trained with the full action space and the limited, per-skill specific action space used in Szot et al. [2021], Each skill is trained with VER for 500 million steps of experience on 8 GPUs. This takes less than 2 days per skill.
6.1 Performance on HAB
We find that skills with navigation actions improve performance on the full task but does not change performance the skill's train task (e.g. Pick achieves 90% success with and without during training).
8

---

Tidy House
Prepare Groceries
Set Table
icc paze Rit Pace For Race Poc Flace HoS Haae
Pace
Pd
PCC
Cpee CAE-ARE
Pace
open Irepertct
Rice
Success (Higher is Better)
+30%
+7.5%
5%
Interaction #
nttrrcttoon #
Interaction
Chained Skil
Chained Skill+Skil Nevigation
Chained SkillNo Nev)+Skill Nevigation
Figure 6: HAB Performance the Tidy House, Prepare Groceries, and Set Table scenarios. Skill policies with navigation (TP-SRL+SKil) Navigation) outperform skill policies without navigation (TP-SRL) despite not strictly needing this ability. Further, find that these skill policies have learned emergent navigation. SR..LNNNaa Navigation achieves 50% success on NavPick (Tidy House) and 20% success NayPickNavPlace (Tidy House) despite all navigation being performed by skills policies that ostensibly don't 't need to navigate.
Fig. 6 shows smaller drops in performance between every interaction (there a navigation between each interaction) demonstrating that skills with navigation effectively correct for handoff errors. This impactful after place as the navigation policy tends to make more errors when navigating to the next location after place. On Tidy House, full task performance improves from 2% Success to 32%.
On Prepare Groceries (which requires picking/placing from/into the fridge) and Set Table (which requires opening the fridge/cabinet and then picking from it) performance improves slightly. 0% to 5% Prepare Groceries and 0% 7.5% on the first pick+place on Set Table. Both these tasks remain open problems and the next frontier.
6.2 Emergent Navigation
Next we examine the skill policies are able navigate to correct for highly out-of-distribution initial location. We examine (T-SRL(NNNav), which that omits the navigation skill. In this agent all navigation is done by skill policies that ostensibly never needed navigate during training.
We find emergent navigation in both the Pick and Place policies. TN-NRL(NoNvv) achieves 50% success NavPick (Fig. 6, Tidy House interaction 1), and 20% success NavPickNavPlace (Fig. 6, Tidy House interaction 2). The latter is on-par with the TaskPPlaming=-SenseP Plans (TP-SPA) classical baseline and significantly better than the MonolithieRL baseline in Szot al. [2021].
The Pick and Place policies were trained on tasks that requires no navigation but both are capable of navigation. We provided examples of both the training task for Pick and Place, and TP-SRI on Tidy House the supplementary materials.
On Prepare Groceries and Set Table, the navigation performance of these policies is worse (-34% Success and -45% Success on the first interaction, respectively). Prepare Groceries requires picking from the fridge, which is challenging and requires navigation that doesn't 't accidentally close the door. Set Table requires opening the cabinet and then picking, which introduces an additional OpenCab skill and requires more precise navigation and picking from Pick. Performance is non-zero (15% and 4% on the first interaction, respectively) in both scenarios; indicting that the skill policies are capable of navigating even in these scenarios, albeit less successfully than in Tidy House.
We hypothesize that the Pick and Place polices learned navigation because this was useful early in training. Early in training the policies have yet to learn that only minimal navigation is needed to complete the task. Therefore the policy will sometimes cause itself move away from the pick object/place location and will navigate back. Navigation is then not forgotten as the policy converges.
We examined the behavior of Pick policy early in training and found that does tend to move away from the object it needs to pick up and sometimes moves back. Although the magnitude of navigation small and quite infrequent, so the degree of generalization is high.
This result, and higher performance on HAB, highlights that may not always be beneficial to remove 'unneeded' actions. Szot et al. [2021] removed navigation where possible to improve sample efficiency

---

and training throughput'. Our experiments corroborate this; training without navigation improves both sample efficiency and training speed. However, by enabling navigation and allowing the agent to learn how to (not) use it, we arrived upon emergent navigation and improved HAB performance.
7 Related Work
AsyncOnRL methods provide high-throughput on-policy reinforcement learning [Espeholt et al., 2018, Petrenko et al., 2020]. However, they have reduced sample efficiency as they must correct for near- policy, or 'stale", data. Few support multi-GPU learning and, when they do, the user must manually balance compute between learning and experience collection [Espeholt et al., 2020]. VER achieves the same throughput on 1-GPU while learning with on-policy data, has better sample efficiency, supports multi-GPU learning, and automatically balances compute between learning and simulation.
SyncOnRL. HTS-RL [Liu et al., 2020] also use the experience collection techniques as AsyncOnRL to mitigate the action-level straggler effect. Unfortunately inefficiencies in the provided implementation prevent a meaningful direct comparison and hide the full effectiveness of this technique. In Appendix E we show that our re-implementation is 110% faster (2. 1x speedup) and thus instead compare to the stronger baseline of NoVER in the main text. We propose a novel mechanism, variable experience rollouts, to mitigate the episode-level straggler effect and thereby close the gap to AsyncOnRL. We use and build upon Decentralized Distributed PPO (DD-PPO) [Wijmans et al., 2020], which proposed a distributed multi-GPU method based on data parallelism [Hillis and Steele Jr, 1986].
Batched simulators simulate multiple agents (in multiple environments) simultancously and are responsible for their own parallelization [Shacklett et al., 2021, Petrenko et al., 2021, Freeman et al., 2021, Makoviychuk et al., J. While these systems offer impressive performance, none currently support benchmark like HAB (which combines physies and photo-realism nor the flexibility of Habitat, A12Thor [Kolve et al., 2017], or ThreeDWorld [Gan et al., 2020]. VER enables researchers to first explore promising directions using existing simulators and then build batched simulators with the knowledged gained from their findings.
8 Societal Impact, Limitations, and Conclusion
Our main application result is trained using the ReplicaCAD dataset [Szot et al., 2021], which is limited to only US apartments, and this may have negative societal impacts for deployed assistants. VER was designed and evaluated for tasks with both GPU simulation and large neural networks. For tasks with CPU simulation and smaller networks, may have less throughput than AsyncOnRL and overlapping experience collection and learning would likely be beneficial. Our implementation supports overlapping learning and collecting rollout, but more overlap may be beneficial. The TP-SRL agent we build upon requires oracle knowledge, e.g. that the cabinet must be opened before picking.
expect it to improve upon SyncOnRL but it
We have presented Variable Experience Rollout (VER). VER combines the strengths and blurs the line between SyncOnRL and AsyncOnRL. Its trains agents for embodied navigation tasks in Habitat 1.0 600% faster (1.6x to 2x speedup) than DD-PPO with similar sample efficiency saving 19.2 GPU-days on PointNav and 28 GPU-day for ObjectNav per seed in our experiments. On the recently introduced (and more challenging) embodied rearrangement tasks in Habitat 2.0, VER trains agents 150% faster than DD-PPO and is fast as SampleFactory (AsyncOnRL)
1 GPU. On 8 GPUs, VER is 180% faster than DD-PPO and 70% faster than SampleFactory with better sample efficiency saving 32 GPU-days vs. DD-PPO and 11.2 GPU-days
SampleFactory per skill in experiments. We use VER to study rearrangement, We find the emergence of navigation in policies that ostensibly require no navigation when given access to navigation actions. This results in strong progress Tidy House (+30% success) and shows that it may not always be advantageous to remove 'unneeded actions'.
Acknowledgements. The Georgia Tech effort was supported in part by NSF, ONR YIP, and ARO PECASE. EW is supported in part by an ARCS fellowship. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government, or any sponsor.
Personal correspondence with authors.
10

---

References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, et al. On evaluation of embodied navigation agents. arXiv preprint arXiv: 1rXiv:1807.06757 2018. 2,5
Dhruv Batra, Angel X Chang, Sonia Chernova, Andrew Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. Rearrangement: challenge for embodied ai. In arXiv preprint 20Xiv:2011.019755 2020a. 2,
Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects, 2020b. 5,5
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. Matterport3d: Learning from rgb-d data in indoor environments. In International Conference on 3D Vision (3DV), 2017. License: http://kaldir.
vc. in. tum. de/maee/maaaaa de/matterponne wewattterrottii S/mrpporr/MP_SSSS pdf. 6
Jeffrey Dean and Sanjay Ghemawat. Mapreduce: Simplified data processing on large clusters. USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2004.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-ri with importance weighted actor-learner architectures. arXiv preprint 1802.0::02.0.11
2018. 7, 10
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable the International Conference
and efficient deep-rl with accelerated central inference. Proceedings the
on Learning Representations (ICLR), 2020. 10
C Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. arXiv preprint arXiv:2106.132 10261.3221... 2021. 10
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04995 2020. 10
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 17x.v:/70.002777. 2017.
Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Bernardo A Pires, and Rémi Munos. Neural predictive belief representations. arXiv preprint 1rXiv: 2018. 5
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 1rxiv: 2018.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
W Daniel Hillis and Guy L Steele Jr. Data parallel algorithms. Communications of the ACM, 29(12): 1183, 1986. 10
Jacob Hilton, Karl Cobbe, and John Schulman. Batch size-invariance for policy optimization. arXiv preprint arXiv:21 :tti::211000000 10.00641, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9()) 1785-1780, 1997.
11

---

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR), 2015.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. AI2-THOR: An Interactive 3D Enviromment for Visual Al. arXiv, 2017. 10
lou-Jen Liu, Raymond Yeh, and Alexander Schwing. High-throughput synchronous deep rl. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020. 2, 4, 10
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.
A convnet for the 2020s. arXiv preprint arXiv:2201.03544, 2022. 7
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-based physics simulation for robot learning. arXiv preprint arXiv:2 10470, 2021. 10
Oleksandr Maksymets, Vincent Cartillier, Aaron Gokaslan, Erik Wijmans, Wojciech Galuba, Stefan Lee, and Dhruv Batra. Thda: Treasure hunt data augmentation for semantic navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15374-15383, 2021. 1
Lina Mezghani, Sainbayar Sukhbaatar, Thibaut Lavril, Oleksandr Maksymets, Dhruv Batra, Piotr Bojanowski, and Karteek Alahari. Memory-augmented reinforcement learning for image-goal navigation. arXiv preprint arXiv:2101.0518/ 2021.
Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7 (62):eabk2822, 2022.
Aleksei Petrenko, Zhehui Huang. Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In Proceedings of the International Conference on Machine Learning (ICML), pages 7652-7662.
PMLR, 2020. 3, 4, 7, 10
Aleksei Petrenko, Erik Wijmans, Brennan Shacklett, and Vladlen Koltun. Megaverse: Simulating embodied agents one million experiences per second. In Proceedings of the International Conference on Machine Learning (ICML), pages 8556 8566..56.777 PMLR, 2021. 10
Fabrizio Petrini, Darren J Kerbyson, and Scott Pakin. The case of the missing supercomputer performance: Achieving optimal performance on the 8,192 processors of asci q. In SC'03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing, pages 55-55. IEEE, 2003.
Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets. Alex Clegg, John Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, et al. mabitot- 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. Neural Information Processing Systems Benchmarks and Datasets, 2021. 1,5
Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra. Habitat: A Platform for Embodied AI Research. In Proceedings of IEEE International Conference Computer Vision (ICCV), 2019. 5,5,
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2016. 7
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017. 5,7
Brennan Shacklett, Erik Wijmans, Aleksei Petrenko, Manolis Savva, Dhruv Batra, Vladien Koltun, and Kayvon Fatahalian. Large batch simulation for deep reinforcement learning. In Proceedings of the International Conference Learning Representations (ICLR), 2021. 10
12

---

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1992.
Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat. Advances in Neural Information Processing Systems (NeurIPS), 2021. 2, 3, 6, 7, 8, 10
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 32-42, 2021." 7
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning near-pen carr-perfect pointgoal navigators from 2.5 billion frames. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. 1, 2, 7, 10
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of European Conference on Computer Vision (ECCV), 2018. 7
Joel Ye, Dhruv Batra, Erik Wijmans", and Abhishek Dast, Auxiliary tasks speed up learning pointgoal navigation. Conference on Robot Learning (CoRL) (In submission), 2020. 5
Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable objectnav. arXiv preprint arXiv:2104.04112, 2021.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli,, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. Proceedings of the International Conference on Learning Representations (ICIR), 2020. 4
Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. Proceedings of the International Conference on Learning Representations (ICLR), 2015. 7
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 8
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See the supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplementary materials
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We use 95% bootstrapped confidence intervals.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]-
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [Yes] See Appendix F
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See Appendix F
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Appendix F
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "zaQ5ZEB7hQL", "reviewer": "Reviewer_45yB", "summary": "This paper proposes a scaling technique for reinforcement learning (RL) to speed up sampling and learning in on-policy RL methods. The paper claims to blend the benefits of SyncOnRL and AsyncOnRL to increase both throughput and sample-efficiency. The main approach is to enable multiple environments to collect a variable number of steps simultaneously without synchronization. The paper performs experiments on several simulation scenarios and analyzes the performance of several baselines.", "questions": "+ The theoretical novelty and contribution of the paper are not clear. Why the proposed technical can theoretically improve both throughput and sample-efficiency?\n\n+ The speed increase of VER on 8 GPU is only 70% faster than 1 GPU. Is the problem big enough to use the power of 8 GPU? Or is the algorithm for the 8 GPUs well parallelized? If 8 GPUs are not fully utilized, the motivation of using the proposed method on multiple GPUs seems not essential.\n\n+ What is the difference between VER and AsyncOnRL? Theoretically/mathematically (not only quantitatively), why does VER perform better than AsyncOnRL?\n\n+ The organization of the paper needs improvement. Experiment results are scattered throughout the paper, from introduction to the four sections of experiments. Some experiment setups also appear in the approach section. This increases the difficulty of understanding the theoretical novelty of the paper.\n\n+ More details need to be provided to explain several concepts including TP-SRL and the architecture in Line 226.\n", "limitations": "No potential negative societal impact is perceived in the paper. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper tries to address an important problem in RL to speed up sampling and learning.\n\n+ The paper introduces an engineering scaling method that aims that takes advantages of SyncOnRL and AsyncOnRL. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "jGjAPVli4w", "reviewer": "Reviewer_5YLN", "summary": "This paper proposes an approach for bridging the gap between asynchronous and synchronous onpolicy RL to leverage the best of both worlds, preserving the throughput of the former and the  sample efficiency of the latter. They achieve this by parallely simulating N environments for different number of timesteps, thus avoiding being bottlenecked by the slowest environment. The authors evaluate their approach on habitat environments (including the more complex rearrange tasks from habitat 2.0), and empirically validate their claims. \n", "questions": "Questions - \n\n-> Does the emergent navigation skill use described in section 6.2 also happen when using prior methods like DD-PPO, SampleFactory etc, even given more data (up to an order of magnitude more)? \n\n-> Further analysis on the induced curriculum due to the relationship between simulation speed and difficulty of environments would be interesting. Does the proposed approach suffer in cases where the difficult environments are harder to simulate? How can this be mitigated?\n", "limitations": "Limitations are adequately discussed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Significance \n\n-> Performance in terms of sample efficiency and training speed are critical for RL agents. This paper provides an intuitively simple approach (i.e having a budget of samples for all environments each time before the policy update, as opposed to a fixed budget for each one of the environments) that improves on both metrics. This idea can be broadly adopted by the community due to its simplicity, and thus has potential to expedite iteration cycles for all RL researchers, pushing the field forward. Further the analysis performed by the authors comparing the sample efficiency and throughput to that of prior methods (pure async or pure sync) is quite thorough and exhaustive (including a comparison to a version of their method with all micro-optimizations, except the main idea - i.e having a fixed budget for all environments). \n\n-> The authors include an interesting result that with access to skills not directly required for a task, the agent is able to leverage these to perform the task better given more data (Specifically, given access to navigation skills, the agent is able to perform pickplace better, even though navigation isn’t necessary for this task). It would be interesting to see if this emergent behavior can also be seen by running previous approaches (eg DD-PPO), even given more data. If prior methods require a lot more data to discover similar interesting emergent behavior, this would make the argument for adopting the proposed approach even stronger.\n\n-> An interesting aspect of the approach is that it naturally induces a curriculum. Since easier environments are faster to simulate, samples from these are seen more often earlier on, and this helps learning. The authors mention this in passing, but I think there can be further analysis here, and perhaps a design change to specifically create a curriculum between easier / harder environments (difficulty might sometimes not vary exactly  with simulation speed, and I wonder if there are some trade-offs here that can be analyzed).  \n\n\nQuality                                                                                  \n-> The empirical evaluation in terms of throughput and sample efficiency are the most critical components in establishing the author’s claims, and this is done quite exhaustively, even taking account variability in the number of GPUs available for training. Evaluation is included on simpler established navigation tasks from habitat1, and also on more complex longer horizon rearrangement tasks from habitat2 (which include multiple pick-place steps). The grasp action in these environments requires contact with the object and the robot to issue a command, as opposed to the ‘magic grasp’ commonly used in habitat environments (where the object is automatically grasped when close enough to the arm). This makes the control problem significantly harder, and brings the approach closer to real world settings (though that gap is still quite large, but addressing that is outside the scope of this paper). \n\n\nOriginality \n\n-> The idea proposed is quite simple. The focus of the paper is not on algorithmic novelty but rather on a highly performant RL system, which will be quite useful. The related work adequately covers related work in the field, including relevant asynchronous and synchronous approaches. \n\n\nClarity \n\n-> The paper is clearly written and motivated, and easy to follow. In order for this work to have an impact on the community the code must be released (since the focus of this work is a better RL system, and there are some micro-optimizations included that might be critical). The code is included in the supplementary, so I think the authors intend to release it. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gfaSuQaGLgc", "reviewer": "Reviewer_ep2X", "summary": "The authors present VER, a method that could both tackle the challenge in synchronous RL that the policy updates need to wait for all steps done in all environments (straggler effect), and the challenge in asynchronous RL that we need to explicitly tackle stale samples when do on-policy learning. Specifically, VER is an environment-dependent variable step size training method that uses variable step numbers according to environments. Evaluation experiments are done sufficiently.", "questions": "Please check the 2 points stated above in the weakness section.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---------------------------------\nThis paper discusses an important problem in the RL community, i.e., what techniques should we use in the context that we usually vectorized a batch of environments in simulations to speed up the training time. Overall, the reviewer enjoys reading the author's analysis of problems in AsyncOnRL and SyncOnRL. And the motivation for variable step size rollout makes sense to me. Experiments are sufficient, and efforts are obvious.\n\nWeakness\n---------------------------------\n- (1) The authors fail to describe how exactly the step length is decided. As a comparison, the baseline DD-PPO [1] paper clearly describes the whole training solution.\n - (2) Although several tasks are evaluated to test the efficiency of VER, it's hard for the reviewer to determine the claimed performance, for example, \"in Habitat 2.0 VER is 150% faster on 1 GPU and 200% faster on 8 GPUs\". At least VER's performance gain is not obvious in Fig. 4.\n\nRef:\n-----------------\n- [1] Wijmans E, Kadian A, Morcos A, et al. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames[J]. arXiv preprint arXiv:1911.00357, 2019.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AScXMIxnP_-", "reviewer": "Reviewer_68WX", "summary": "The paper focuses on the problems pertaining to SyncOnRL and AsyncOnRL and proposes a solution that combines the advantages of these methods. SyncOnRL suffers from the straggler effect, where the slowest worker decides the overall speed of execution, and the system capability is not utilized properly. AsyncOnRL solves this by taking action in an environment as soon as it is ready. Although this leads to efficient system utilization, it leads to reduced sample efficiency. \nThe proposed method, Variable Experience Rollout aims to mitigate these issues. The paper points out that collecting an equal number of time steps T per environment is easier to implement but is not a necessary requirement for reinforcement learning. VER collects a variable number of steps across the different environments. This overcomes staggler effect while at the same time is more sample efficient than AysnOnRL. ", "questions": "Regarding Emergent Skills\n* Would these skills not emerge when trained with AsyncOnRL on SyncOnRL?\n* Is there something specific to VER that leads to the emergence of these skills? Or any method trained for a sufficient amount of time can lead to the emergence of these skills?\n", "limitations": "Limitations are not discussed properly. \n", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths : \n* The paper works on an exciting and vital problem of accelerating on-policy RL algorithms while maintaining sample efficiency. \n* The idea is simple and original based on an ingenious observation about the training of the Reinforcement learning algorithm. It is a relatively novel idea. \n* The amount of time required to train the RL algorithms when compared to the fastest SyncOnRL is much much less while maintaining the same sample efficiency. Making these algorithms more accessible to the general audience. \n* The experimental details are comprehensive about the system capabilities, which is essential, especially for a paper that is mainly about utilizing system capabilities. \n* Clear writing and neat plots. \n\nWeaknesses\n* Although the idea is exciting and novel, the presentation can be improved by including the following plot\n    * it would be constructive to see one plot where : \n       * x-axis: Policy training time (in hours or any other suitable time unit) \n       * y-axis: Success Rate \n       * Baselines: Fastest AsyncOnRL, Fastest SyncOnRL, VER\n    * It is not clear whether VER leads to improvement in training time (ie the time required for the policy to converge to a high success rate). \n\n* Emergent skills \n    * Although it is an interesting result, it is not very clear how the results support the main idea of the paper. In my opinion, the analysis of these skills should be part of the appendix rather than the main paper. \n    \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept