PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators

Authors: kuo-hao  zeng
zichen  zhang
kiana  ehsani
rose  hendrix
jordi  salvador
alvaro  herrasti
ross  girshick
aniruddha  kembhavi
luca  weihs
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 22:25:07

Abstract: We present PoliFormer (Policy Transformer), an RGB-only indoor navigation agent trained end-to-end with reinforcement learning at scale that generalizes to the real-world without adaptation despite being trained purely in simulation. PoliFormer uses a foundational vision transformer encoder with a causal transformer decoder enabling long-term memory and reasoning. It is trained for hundreds of millions of interactions across diverse environments, leveraging parallelized, multi-machine rollouts for efficient training with high throughput. PoliFormer is a masterful navigator, producing state-of-the-art results across two distinct embodiments, the LoCoBot and Stretch RE-1 robots, and four navigation benchmarks. It breaks through the plateaus of previous work, achieving an unprecedented 85.5% success rate in object goal navigation on the CHORES-S benchmark, a 28.5% absolute improvement. PoliFormer can also be trivially extended to a variety of downstream applications such as object tracking, multi-object navigation, and open-vocabulary navigation with no finetuning.

Main paper: 
 introduction: 
 Reinforcement Learning (RL) has been used extensively to train embodied robotic agents to complete a variety of indoor navigation tasks. Large-scale, on-policy, end-to-end RL training with DD-PPO [1] enables near-perfect PointNav 1 performance when using a shallow GRU-based [2] architecture. However, this approach fails to result in the same breakthroughs for harder navigation problems like Object Goal Navigation (ObjectNav [3] ) where an agent must explore its environment to locate and navigate to an object of the requested type. RL approaches for ObjectNav have generally not advanced beyond shallow GRU architectures due to challenges presented by training instability and unreasonably long training times with wider and deeper models, such as scaled-up transformers [4] .In a departure from on-policy RL, which is sample inefficient and often uses complex reward shaping and auxiliary losses [5] , Imitation Learning (IL) has recently shown promise for ObjectNav. Ehsani et al. (2023) [6] demonstrated that the transformer-based SPOC agent, when trained to imitate heuristic shortest-path planners, can be trained stably, is sample efficient, and is significantly more effective than prior RL approaches on their benchmark. SPOC, however, ultimately falls short of mastery, plateauing at a success rate of ∼57%. Critically, as we show in our experiments, the performance of SPOC does not seem to improve significantly when further scaling up data and model depth; we suspect this is a consequence of insufficient state-space exploration as expert trajectory datasets frequently contain few examples of error recovery, which can lead to sub-optimal performance due to compounding errors [7] or non-trivial domain shifts during inference. 1 Point Goal Navigation is the task of navigating to a goal location using privileged GPS coordinates.8th Conference on Robot Learning (CoRL 2024), Munich, Germany. In contrast to IL, RL requires that agents learn via interactive trial-and-error, allowing for deep exploration of the state space. This exploration has the potential to produce agents that can learn behaviors that are superior to those in the expert demonstrations. This raises the question: can we bring together the modeling insights from SPOC-like architectures and the exploratory power of RL to train a masterful navigation agent? Unfortunately, this cannot be done naively due to RL's sample inefficiency and complexities in using deep (transformer) architectures with RL algorithms. In this work, we develop an effective method for training large transformer-based architectures with RL, breaking through the plateaus of past work, and achieving SoTA results across four benchmarks.Our method, POLIFORMER, is a transformer-based model trained with on-policy RL in the AI2-THOR [8] simulator at scale, that can effectively navigate in the real world without any adaptation. We highlight three primary design decisions that make this result possible. (i) Scale in Architecture: Building on the SPOC architecture, we develop a fully transformer-based policy model that uses a powerful visual foundation model (DINOv2 [9] , a vision transformer), incorporates a transformer state encoder for improved state summarization, and employs a transformer decoder for explicit temporal memory modeling (Fig. 1 , top-right). Importantly, our transformer decoder is causal and uses a KV-cache [10] , which allows us to avoid huge computational costs during rollout collection and makes RL training affordable. (ii) Scale in Rollouts: We leverage hundreds of parallel rollouts and large batch sizes, which leads to high training throughput and allows us to train using a huge number of environment interactions (Fig. 1 , top-middle). (iii) Scale in Diverse Environment Interactions: Training POLIFORMER with RL at scale in 150k procedurally generated PROCTHOR houses [11] using optimized Objaverse [12, 13, 14] assets results in steady validation set gains (Fig. 1 , top-left).POLIFORMER achieves excellent results across multiple navigation benchmarks in simulation. On CHORES-S, it achieves an impressive 85.5% Success Rate, a higher than +28.5% absolute improvement over the previous SoTA model [6] . Similarly, it also obtains SoTA Success Rates on Proc-THOR (+8.7%), ArchitecTHOR (+10.0%) and AI2-iTHOR (+6.9%). These results hold across two embodiments, LoCoBot [15] and Stretch RE-1 [16] , with distinct action spaces (Fig. 1 , bottomleft). In the real world (Fig. 1 , bottom-right), it outperforms ObjectNav baselines in the sim-to-real zero-shot transfer setting using LoCoBot (+13.3%) and Stretch RE-1 (+33.3%). 2We further train POLIFORMER-BOXNAV that accepts a bounding box (e.g., from an off-the-shelf open-vocab object detector [18, 19] or VLMs [20, 21] ) as its goal specification in place of a given category. This abstraction makes POLIFORMER-BOXNAV a general purpose navigator that can be "prompted" by an external model akin to the design of Segment Anything [22] . It is extremely effective at exploring its environment and, once it observes a bounding box, beelines towards it. POLIFORMER-BOXNAV is a leap towards training a foundation model for navigation; with no further training, this general navigator can be used in the real world for multiple downstream tasks such as open vocabulary ObjectNav, multi-target ObjectNav, human following, and object tracking.In summary, our contributions include: (i) POLIFORMER, a transformer-based policy trained by RL at scale in simulation that achieves SoTA results over four benchmarks in simulation and in the real world across two different embodiments. (ii) A training recipe that enables effective end-to-end policy learning with large-scale neural models via on-policy RL. (iii) A general purpose navigator, POLIFORMER-BOXNAV, that can be used zero-shot for multiple downstream navigation tasks. 
related work: 
 IL and RL on Embodied Navigation. Recent advancements include Point Goal Navigation [1] , Object Goal Navigation [3, 23, 24, 25] , Exploration [26, 27, 28] , and Social Navigation [29] , as well as successful sim-to-real transfer [6] and high performance in various downstream applications [16, 30, 31, 32, 33, 34, 35] . The prevalent approaches to building capable navigation agents include both end-to-end training [1, 6] and modular methods that leverage mapping [36, 37, 38] or off-the-shelf foundation models [39, 40, 41, 42, 43] . In these frameworks, the policy models are typically optimized via Imitation Learning (IL) using expert trajectories [44, 45, 46, 47, 48, 49, 50] or Reinforcement Learning (RL) within interactive environments [29, 51, 52, 53, 54, 55, 56, 57, 58, 59] and with carefully tuned reward shaping and auxiliary losses [5, 60, 61, 62, 63] .Transformer-based Policies for Embodied Agents. Recent works use transformer-based architectures for embodied agents. Decision Transformer (DT [64] ) learns a policy offline, conditioning on previous states, actions, and future returns, providing a path for using transformers as sequential decision-making models. ODT [65] builds on DT and proposes to blend offline pretraining and online finetuning, showing competitive results in D4RL [66] . More recently, MADT [67] shows few-shot online RL finetuning on the offline trained DT in multi-agent game environments. The Skill Transformer [68] learns a policy via IL for mobile manipulation tasks. While the focus of these works is control towards specific tasks (relying on IL pre-training), we train from scratch using pure RL targeting a navigation policy that can be used zero-shot for many downstream tasks.There has been a huge effort on improving the scale and training stability of transformer policies. GTrXL [69] proposes to augment causal transformer policies with gating layers towards stable RL training in [70] . PDiT [71] proposes to interleave perception and decision transformerbased blocks showing its effectiveness in fully-observed environments. GATO [72] learns a largescale transformer-based policy on a diverse set of tasks, including locomotion and manipulation. Performer-MPC [73] proposes learnable Model Predictive Control policies with low-rank attention transformers (Performer [74] ) as learnable cost functions. SLAP [75] learns a policy for mobile manipulation based on a hybrid design. Radosavovic et al. [76] learn a causal transformer using RL and IL for real-world humanoid locomotion. However, their input is proprioceptive state, without visual observations, and their learned policy is intended to be stationary (walking steadily in the real-world). NavFormer [77] learns image-goal conditioned transformer policies using offline-RL for target navigation. In contrast to existing approaches, we achieve efficient large-scale transformerbased on-policy RL training for a partially-observed, long-horizon, navigation task using RGB sensor inputs and show that our policy can be seamlessly deployed in the real world. We accomplish this without complex inductive biases or multi-task dataset aggregation due to the sheer amount of procedural scene layouts and visual variety of objects in simulation.Toward Navigation Foundation Models. Many recent works attempt to produce general-purpose "foundational" navigation models. The causal transformer for humanoid outdoor locomotion [76] , relying on proprioceptive state, shows emergent behaviors like arm swinging and in-context adaptation. GNM [78] trains image goal-conditioned policies (i.e., conditioning on an image at the desired goal location) by IL across various embodiments, which allows them to scale up the size of their , models state belief b t over time, employing a causal transformer decoder, and, finally, predicts action logits a t and a value estimation e t via linear actor and critic heads, respectively. For rollout collection and inference, we leverage the KV-cache [10] as our temporal cache strategy to prevent recomputing the forward pass for all prior timesteps at each new timestep, saving memory and speeding up both training and inference.training data and manages to generalize to controlling unseen embodiments. NOMAD [79] , which extends ViNT [80] uses a diffusion policy and also uses image goal conditioning. Unlike these works, we navigate from rich RGB image inputs and specify goals with natural-language text (or with bounding boxes), as such goals are far more easily available in real-world settings.3 Method RL, while effective and intuitive for training policies, has not yet been scaled in embodied AI to the same degree as in other domains. RL has primarily been applied to short-horizon tasks [81, 82] , smaller environments [83, 84] , or tasks utilizing privileged knowledge [1] . We scale learning in three directions; (i) Network Capacity, via POLIFORMER's deep, high-capacity, model architecture (Sec. 3.1); (ii) Parallelism and Batch size via our highly-parallelized training methodology that leverages large batch sizes for efficiency (Sec. 3.2), and (iii) Training Environments via our optimization of the simulation environment to support high-speed training (Sec. 3.3). 
the poliformer architecture: 
 We now detail POLIFORMER's transformer architecture (see Fig. 2 ), which is inspired by SPOC [6] . While we make several subtle, but important, changes to the SPOC architecture (detailed below), our largest architectural difference is in the transformer decoder: we replace the standard transformer decoder block [4, 85] with the implementation used in the Llama 2 LLM [86] . This change has dramatic implications for training and inference speed.At each timestep t, POLIFORMER takes an ego-centric RGB observation i t as input and employs a frozen vision transformer model to extract a visual representation r t . This representation, along with an embedding g of the goal, are summarized by a transformer state encoder to produce the state representation s t . The causal transformer decoder then encodes the state feature s t (along with s 0 , . . . , s t-1 ) into a belief b t . Finally, the linear actor and critic heads project b t to predict action logits a t and a value estimate e t , respectively.Vision Transformer Model. Inspired by prior work [6, 87] , we choose DINOv2 [9] as our visual foundation backbone, because of its remarkable dense prediction and sim-to-real transfer abilities. The visual backbone takes an ego-centric RGB observation i ∈ R H×W ×3 as input and produces a patch-wise representation r ∈ R a ℓ × h matrix, ℓ = H • W/196, and project the representation to produce v ∈ R ℓ×d , where d is the input dimension to the transformer state encoder. For effective sim-to-real transfer, it is important to keep the vision transformer model frozen when training in simulation.Goal Encoder. For fair comparison, when training agents on ObjectNav benchmarks using the LoCoBot, we follow EmbCLIP [39] and use a one-hot embedding layer to encode the target object category. On benchmarks using the Stretch RE-1 robot, we follow SPOC [6] and use a FLAN-T5 small [88, 89] model to encode the given natural language goal and use the last hidden state from the T5 model as the goal embedding. Before passing the goal embedding to the transformer state encoder, we always project the embedding to the desired dimension d, resulting in g ∈ R 1×d .In select experiments (detailed in Sec. 4), we specify the goal object via a bounding box (b-box), either in addition to or as an alternative to text. In this case, the goal encoder processes the b-box using both the sinusoidal positional encoding and the coordinate-type embeddings to embed the topleft, bottom-right b-box coordinates and its area (5 values in total), followed by an MLP to project the hidden feature to the desired dimension d, resulting in g b ∈ R 5×d .Transformer State Encoder. This module summarizes the state at each timestep as a vector s ∈ R d . The input to this encoder includes the visual representation v, the goal feature g (and/or g b ), and an embedding f of a STATE token. We concatenate these features together and feed them to a noncausal transformer encoder. This encoder then returns the output corresponding to the STATE token as the state feature vector. Since the transformer state encoder digests both visual and goal features, the produced state feature vector can also be seen as a goal-conditioned visual state representation.Causal Transformer Decoder. We use a causal transformer decoder to perform explicit memory modeling over time. This can enable both long-horizon (e.g., exhaustive exploration with backtracking) and short-horizon (e.g., navigating around an object) planning. Concretely, the causal transformer decoder constructs its state belief b t using the sequence of state features s = {s j | j=t j=0 } within the same trajectories.Unlike RNN-based causal decoders [90] which, during rollout collection, require only constant time to compute the representation at each timestep, standard implementations of causal transformers require t 2 time to compute the representation at timestep t. This substantial computational cost makes large-scale on-policy RL with causal transformer models slow. To overcome this challenge, we leverage the KV-cache technique [10] to keep past feed-forward results in two cache matrices, one for Keys and one for Values. With a KV-cache, our causal transformer decoder only performs feedforward computations with the most current state feature which results in computation time growing only linearly in t rather than quadratically. While this is still mathematically slower than the constant time required for RNN-based decoders, empirically, we found only a small difference between KV-cache-equipped causal transformers and RNNs in overall training FPS for our setting. Compared to other temporal cache strategies, we found that KV-Cache offers the most significant speed improvements. Please see App. B.3 and Fig. 5 for a discussion of the impact on training speed resulting from different cache strategies.While KV-cache accelerates the causal transformer, this alone is not sufficient to enable efficient and fast RL training. In this section, we describe the methodology we use to achieve faster training. Although each of these individual findings may have been discussed in other works, we emphasize the critical importance of these hyperparameters for efficient training and, consequently, stellar results.We parallelize the training process using multi-node training, increasing the number of parallel rollouts by a factor of 4 compared to previous approaches [30, 59, 87] (we use 192 parallel rollouts for Stretch RE-1 agents and 384 for LoCoBot agents). We employ the DD-PPO [1] learning algorithm across 4 nodes, utilizing a total of 32 A6000 GPUs and 512 CPU cores. This scaling accelerates the training speed by approximately 3.4×, a near-linear gain compared to single-node training; we train for ∼4.5 days with 4 nodes, while this would have required 15.3 days with a single node.Our batch size during training, in number of frames, is equal to the number of parallel rollouts (R) multiplied by the length of these rollouts (T ) and thus, by increasing R as above, we multiplicatively increase the total batch size. We follow SPOC [6] and use a small constant learning rate of 2 • 10 -4 throughout the experiments. Instead of annealing the learning rate during training, we instead follow PROCTHOR [11] and increase the batch size by changing the rollout length from T =32, to T =64, and finally to T =128 (resulting in a final batch size of 49,152 frames for LoCoBot agents). We make these increases every 10M steps until we reach T =128.For the experiments on LoCoBot, we use the original PROCTHOR-10k houses for training for a fair comparison to baselines. With the KV-cache technique, our training steps per second is ∼2.3k for training POLIFORMER, using 384 rollouts on 32 GPUs. For the experiments on Stretch, following [6] , we use the PROCTHOR-150k houses with ∼40k annotated Objaverse 3D assets. As on-policy RL requires the agent to interact with the environment on-the-fly, we found the continuous physics simulation used by the Stretch RE-1 agent in AI2-THOR to be too slow to efficiently train at scale. To overcome this, we discretely approximate the agent's continuous movement via a teleportation-with-collision-checks approach. In particular, to move the agent, we perform a physics cast using a capsule collider representing the agent along the desired movement direction. If the cast does not hit any object (suggesting no potential collisions), we teleport the agent to the target location, otherwise we let AI2-THOR simulate continuous movement as usual. For rotations, we first teleport the agent to the target rotation pose. If the agent's capsule collides with any object, we teleport the agent back and let the AI2-THOR simulate the continuous rotation. For our tasks, these approximations increase simulation speed by ∼40%. We also found that AI2-THOR scene loading and resetting was a significant bottleneck when using Objaverse assets. We streamline AI2-THOR's Objaverse 3D asset loading pipeline by saving objects in the Msgpack format (natively usable with Unity's C# backend) as opposed to Python-only Pickle files. This change dramatically decreases the loading time of a new scene from ∼25s to ∼8s. With all these changes, our training steps per second increases from ∼550 to ∼950 using 192 rollouts on 32 GPUs for training Stretch RE-1 agent, reducing the training time by ∼42%. 
results: 
 We now present our experimental results. In Sec. 4.1 we demonstrate that scaling RL with PO-LIFORMER produces SoTA results on four simulation benchmarks across two embodiments (Lo-CoBot and Stretch RE-1). Then, in Sec. 4.2, we show that POLIFORMER, despite being trained purely in simulation, transfers very effectively to the real world, outperforming previous work; we again show these results on the above two embodiments. In Sec. 4.3, we provide ablations for various design choices. Finally, we qualitatively show that POLIFORMER-BOXNAV can be extended to a variety of downstream applications in a zero-shot manner in Sec. 4.4.Baselines. For our baselines, we chose a set of prior works in both imitation learning and reinforcement learning. SPOC [6] is a supervised imitation learning baseline trained on shortest path expert trajectories in AI2THOR. SPOC* is similar to SPOC, but is trained on more expert trajectories (2.3M vs. 100k). We build this baseline to verify if SPOC easily scales with more data. Emb-SigLIP [6] is an RL baseline also used by [6] , but trained with comparable GPU hours with SPOC. The baselines [11, 63, 87] for LoCoBot-embodiment are all trained by RL using the same training steps used by POLIFORMER. We have three experiment configurations for our studies, specifying the goal as (a) natural language instruction text, (b) a b-box for the target object, and (c) b-box & text. Following [6] , b-boxes in simulation are the ground-truth b-boxes, and so such models are not comparable fairly with RGB-only models. In the real world we replace GT b-boxes with outputs from the open-vocabulary object detector Detic [18] , which uses only RGB images as input, and so all comparisons are fair.Implementation Details. For LoCoBot benchmarks, we follow [87] Table 1a shows that, on the CHORES-S benchmark, POLIFORMER achieves 28.5% absolute gain in success rate over the previous best model. EmbSigLIP baseline results are taken from [6] ; we suspect that its poor performance is, in part, due to its training budget being limited to match the training budget of SPOC.POLIFORMER works across embodiments. Table 1b shows that the remarkable performance of POLIFORMER is not limited to one embodiment. The LoCoBot and Stretch-RE1 robots have different body sizes, action spaces, and camera configurations. Nevertheless, POLIFORMER is able to achieve 8.7%, 10.0%, and 6.9% absolute gain over the best baseline on PROCTHOR-10k (val), ARCHITECTHOR, and AI2-iTHOR.We evaluated POLIFORMER on two real-world benchmarks for two different embodiments to show its sim-to-real transfer capabilities. We used the same evaluation set of 15 tasks for LoCoBot (3 different starting poses with 5 different targets), and 18 tasks for Stretch-RE1 (3 different starting poses with 6 different goal specifications), similar to [17] and [6] , respectively. We do not use any realworld finetuning, and the testing houses were not seen during training. In the text-only setting, PO-LIFORMER achieves a remarkable improvement of 33.3% and 13.3% over the baselines for Stretch-RE1 and LoCoBot respectively (Tab. 2b). POLIFORMER even outperforms Phone2Proc [17] , a baseline that has access to privileged information about the layout of the environment for finetuning.The recipe for building the effective RL navigation model is the end product of several design decisions guided by experimentation, the combination resulting in our SoTA performance. Table 2a , shows the process of scaling POLIFORMER. Row 1 vs. Row 2: Moving from the CLIP ResNet-50 visual encoder to the ViT-small DINOv2 improves the success rate by an average of 5.2%. Has POLIFORMER performance saturated? Figure 1 (top-left) shows CHORES-S ObjectNav validation success rate as we train POLIFORMER for ∼700M environment steps. As that plot shows, POLIFORMER's performance does not appear to have converged, and we expect that performance will continue to improve with more compute. This suggests that achieving near-perfect ObjectNav performance may only require further scaling our training approach.By specifying POLIFORMER's goal purely using b-boxes, we produce POLIFORMER-BOXNAV. While POLIFORMER-BOXNAV lacks the ability to leverage some helpful priors about where objects of certain types should reside (comparing rows 6 & 7 in Table 1a we see POLIFORMER-BOXNAV performs slightly worse than a model having both b-box and text goal specifications), this design decision makes POLIFORMER-BOXNAV a fully general-purpose, promptable, navigation system that can navigate to any goal specifiable using a b-box. Figure 3 shows four qualitative examples where, by using an open-vocabulary object detector (Detic [18] ) and a VLM (GPT-4o [91] ), POLIFORMER-BOXNAV is able to, zero-shot, navigate to (1) a book with a particular title, (2) a given room type, (3) multiple objects sequentially, and (4) a toy car as the car is driven around an office building. Simply by specifying goals as b-boxes and leveraging off-the-shelf systems, we obtain complex navigation behaviors that would otherwise need to be trained for individually: a painful process requiring new reward functions or data collection. Finally, we encourage readers to visit our website to view more qualitative results presented in video format. 
discussion: 
 Limitations: Training RL agents for long-horizon tasks with a large search space requires extensive compute and demands careful reward shaping. While we believe POLIFORMER is capable of scaling to other tasks, it requires crafting new reward models for novel tasks such as manipulation. More discussion on limitations in App. E. Conclusion: In this paper we provide a recipe for scaling RL for long-horizon navigation tasks. Our model, POLIFORMER, achieves SoTA results on four simulation benchmarks and two real-world benchmarks across two different embodiments. We also show that POLIFORMER has remarkable potential for use in downstream everyday tasks.These appendices contain additional information about our:• Zero-shot real-world applications (App. A),• Training procedure (App. B),• Environment, benchmarks, and quantitative real-world experiments (App. C),• Simulation evaluations (App. D), and• Limitations (App. E).Please find our project website (see the poliformer.allen.ai) that contains• Six real-world qualitative videos where POLIFORMER performs the everyday tasks of Section 4.4 (recall also Figure 3 ), and• Four qualitative videos in simulation showing our POLIFORMER's behavior in the four benchmark environments (CHORES, PROCTHOR, AI2-iTHOR, and ARCHITEC-THOR).A Details about Zero-shot Real-world Downstream Applications using an Open-Vocab Object Detector and VLMBy specifying POLIFORMER's goal purely using b-boxes, we produce POLIFORMER-BOXNAV. POLIFORMER-BOXNAV is extremely effective at exploring its environment and, once it observes a bounding box, takes a direct and efficient path towards it. We now describe how we utilize this behavior to apply POLIFORMER-BOXNAV zero-shot to a variety of downstream applications by leveraging an open vocabulary object detector (Detic [18] ) and a VLM (GPT-4o [91] ).Open Vocabulary ObjectNav. To perform open vocabulary object navigation (i.e., where one must navigate to any given object type), we simply prompt the Detic object detector with the novel object type, for example, Bicycle. As POLIFORMER-BOXNAV relies on the b-box as its goal specification, it finds a bicycle in the scene smoothly.Multi-target ObjectNav. To enable multi-target object goal navigation, we make a few simple modifications to the inputs and output of the Detic detector. On the input side, we query with multiple prompts simultaneously (one for each object type); for instance, HousePlant, Toilet, and Sofa, as shown in Fig. 3 (bottom-left). We then, on the output side, only return the b-box with the highest confidence score. Since the returned b-box also contains the predicted object type, we know what the target object the agent finds is when issuing a Done action. Therefore, we remove the found target from the list of target types, and reset the POLIFORMER's KV-cache. If the agent issues a Done action without a detected b-box, we terminate the episode and consider it a failure. As a result, the agent is required to find all the targets from the list of target types to succeed in an episode.Human Following. We change the Detic prompt to Person. Once a b-box is detected, PO-LIFORMER drives the agent to approach it. Our experiment participant continues to walk away, so the agent keeps approaching them to minimize the distance.Object Tracking. In this example, we control a remote control car that moves in the environment, and prompt the agent to find the car. Similar to Human Following, we change the prompt to Toy Truck in this example. As a result, the agent keeps trying to move closer to the detected b-box of the RC car, while avoiding collisions with objects in the dynamic scene.Room Navigation. In this example, shown in Fig. 3 (middle-left), we provide no detections to the agent. As the agent sees no detections, it continuously explores the scene. As the agent explores, we 0 1 0 1 2 3 4 5 0 1 2 0 1 2 0 1 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 1 1 1 1 0 00 0 1 1 1 1 1 0 0 0 1 1 1 1 1 11 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 0 0 0 0 0 0 1 1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 1 1 st rollout 2 nd rollout 1 st episode 2 nd episode 1 st episode 2 nd episode 3 rd episode Instance Description Navigation. In this example, shown in Fig. 3 (upper-left), the agent is prompted to find a specific book titled "Humans". Detic can generate open-vocabulary bounding boxes using instance-level descriptions but we found that doing this alone leads to high false-positive rates. To reduce these errors, we use GPT4-o to filter positive detections from Detic. In particular, a sample filtering prompt is "Is there a book titled "Humans" in this image? Please return Yes or No.". We find this combination works well in practice. The agent, not GPT-4o, remains responsible for deciding when it has successfully completed its task, and in the Fig. 3 example sees many books in its search but perseveres and eventually finds the correct one.For reward shaping, we follow EmbCodebook [87] and PROCTHOR [11] and use the implementation in AllenAct [92] : R penalty + R success + R distance , where R penalty = -0.01 encourages an efficient navigation, R success = 10 when the agent successfully completes the task (= 0 otherwise), and R distance is the change of L2 distances from target between two consecutive steps. Note that we only provide a nonzero R distance if the new distance is less than previously seen in the episode. We do not enforce a negative reward for increasing distance. This formulation encourages exploration.During training, to ensure that the causal transformer decoder cannot access observations or states across different episodes, we construct the episodic attention mask to only allow the past experiences within the same episode to be attended. In Fig. 4 , we show a couple of possible rollouts collected during training. With the episodic attention mask, observations and states in an episode can only attend to previous ones within the same episode, in contrast with a naive causal mask where they could also potentially attend to observations and states in previous episodes.Besides the KV-Cache, in this subsection, we ablate four different temporal cache strategies (shown in Fig. 5 (i) No-Cache, we cache the raw frames (visual observation i t ) to provide past experiences for the causal transformer decoder. Therefore, POLIFORMER has to rerun the feedforward across all modules with the cached frames and the latest visual observation at each timestep.(ii) Feature-Cache, we cache the features, including visual representation v t and goal embedding g t (as well as g t b ). In this case, POLIFORMER needs to recompute the transformer state encoder and causal transformer decoder with the cached features and the latest feature at each timestep.(iii) State-Cache, we cache the state feature vector s t . Since it is right before the causal transformer decoder, POLIFORMER only requires to pass the cache state features and the latest state feature vector to the decoder, without recomputing the visual transformer model, goal encoder, and transformer state encoder.(iv) KV-Cache, as described in Sec. 3.1, we cache the Keys and Values inside the causal transformer decoder, further reducing the required computation time for the transformer decoder from t 2 to t theoretically. In addition, since this strategy operates at very end, all the recomputations required by modules preceding causal transformer decoder can be saved.The speed profile results are shown in Fig. 5 bottom. It clearly shows that placing the cache closer to the causal transformer decoder improves the training efficiency significantly.Tab. 3 lists the hyperparameters used in our training and model architecture design. Please find more details such as scene texture randomization, visual observation augmentations, and goal specification randomization when using text instruction in our codebase. indicator of success for episode i, l i is the shortest travel distance to the target, and p i is the actual travel distance. SEL is defined similarly:1 N N i=1 S i wi max(wi,ei), where w i is the shortest number of steps to find the target, and e i is the actual number of steps used by the agent. By definition, SPL focuses on how far the agent has traveled, while SEL focuses on how many steps the agent has used (which also penalizes excessive in-place rotation). SPL can be derived by computing the geodesic distance between the agent's starting location and the target's location, while SEL needs a planner with privileged environment information to calculate the number of steps of expert trajectories. Therefore, we follow ProcTHOR [11] to report SPL to evaluate the LoCoBot agent, since those benchmarks do not provide planner, while we follow CHORES-S [6] to report SEL, since expert trajectories are available.Real-world Experiment Setup. For the experiments using LoCoBot, we follow Phone2Proc [17] to use the same five target object categories, including Apple, Bed, Sofa, Television, and Vase, and the three starting poses, shown in Fig. 6 (a). Among those target categories, Apple can be found in the Living room and Kitchen, Bed can only be found in the Bedroom, Sofa and Television can only be found in the Living room, and Vase can be found in the Livingroom, Corridor, Office, and Kitchen. For the experiments using Stretch RE-1, we follow SPOC [6] to use the same six target object categories, including Apple, Bed, Chair, HousePlant, Sofa, and Vase, and the three starting poses, shown in Fig. 6 (b ). Among the categories not mentioned above, Chair can be found in the Living room, Office, and Kitchen, and HousePlant can be found in the Living room, Office, Bathroom, and Kitchen. Note that we use the hardware design intorduced in SPOC [6] for Stretch RE-1. Instead of using the off-the-shelf camera equipped on the Stretch RE-1 (due to its narrow field of view), we use an Intel RealSense 455 fixed camera, which has a vertical field of view of 59 • and a resolution of 1280 × 720. The camera is mounted facing forward but pointing downward, with the horizon at a nominal angle of 30 • . Please find more details in App. C.1. in SPOC [6] .First, during the training and testing stages, the possible target object types the agent is tasked with searching for include AlarmClock, Apple, BasketBall, Bed, Bowl, Chair, GarbageCan, HousePlant, Laptop, Mug, Sofa, SprayBottle, Television, Toilet, and Vase. We then outline the differences between the environments. Performance Variance. On CHORES-S, since we follow SPOC [6] to apply test-time data augmentation and non-deterministic action sampling, we found that performance varies even using the same checkpoint, especially given that we are only evaluating on 200 episodes. As a result, we re-evaluate our POLIFORMER and SPOC* 4 for movement and σ = rotation degree 3 for rotation. The LoCoBot agent has a step size of 0.25m and rotation degree of 30 degrees by default.To evaluate the robustness of PoliFormer, we inject the gaussian noise into the action only at the evaluation stage. The results using LoCoBot across ProcTHOR-val, ArchitecTHOR, and AI2-iTHOR are shown in Tab. 6. POLIFORMER is robust to action noise even when experiencing substantial perturbations (3 × σ is a very large and unrealistic amount of noise).Depth Sensor. It is important to note that POLIFORMER is not equipped with a depth sensor (which has been proven to be effective for manipulation). While the lack of the depth sensor does not affect our agent's performance on navigation, we acknowledge that integrating the depth sensor into our visual representation is an interesting direction for future work, especially when considering mobile-manipulation extensions.Discretized Action Space. To have a fair comparison with baselines, we use the same discretized action space in this work (see App. C). The discretized action space might not be efficient and realistic in many real-world scenarios where the agent must act in a timely manner.Cross-embodiment. In this paper, we demonstrate that we can train POLIFORMER using LoCoBot and Stretch RE-1. However, we have not yet explored training a single POLIFORMER for both embodiments. We leave this interesting research direction as future work.Further Scaling. Our training and validation curves strongly suggest that even further scaling of model parameters and training time may lead to even more masterful models than those we have trained in this work. This perspective is exciting and we hope to enable further scaling with more computation resources and better visual foundation models in the near future.Failure Analysis. The main mode of failure for POLIFORMER is the agent's limited memory. PO-LIFORMER clearly demonstrates memorization capabilities and is able to perform long-horizon tasks by exploring large indoor scenes without access to explicit mapping. However, as the trajectories get longer (specifically after visiting more than 4 rooms in an environment), the agent's recollection of the rooms it has explored deteriorates and the robot might re-visit rooms that it has explored previously.Extensive Use of GPUs and Significant Training Time. We acknowledge that PoliFormer requires a large number of GPUs to train efficiently. Our main focus in this work is on pushing the performance of navigation agents to its limit without constraints on computing resources. Given the trend of GPU improvements over the last 10 years, we believe that the resources used in this work will become relatively commonplace in the near future. Empirically, we found that training with a smaller number of GPUs (e.g., 8 A6000 in a single host) could yield similar final performance, with the downside of slower training speed (∼ 3.5× longer to train compared to 32 A6000 in four nodes). Moreover, there are many interesting research directions in improving training speed/efficiency; for example, since our submission, we have found that removing the average pooling from PoliFormer's visual encoder can increase convergence speed by up to 2×. E.g., for the LoCoBot agent, we keep the number of tokens (e.g., 256 tokens) from DINOv2, instead of spatially average pooling them to 7 × 7 (resulting in 49 tokens). This implies that a better visual encoder with a larger receptive view over the visual observation could even further speed up training. Furthermore, we believe that incorporating more advanced techniques such as mixed precision training, flash attention, and advanced GPUs optimized for transformer models could reduce the training resource requirements. Sample Efficiency. We hypothesize that an "IL pretraining +RL finetuning" paradigm could further improve sample efficiency. Early in on-policy RL training, policies trained from scratch are effectively random and thus produce a vast amount of marginally useful experiences. Pretraining with IL may improve sample efficiency by providing helpful priors on navigation and thus produce meaningful RL gradients faster. While it is nontrivial to implement this two-stage pipeline, we hypothesize that training PoliFormer using IL on expert data (e.g., the shortest path trajectories from SPOC) first and finetuning using on-policy RL could be an option to improve the sample efficiency. Since we focus on obtaining SoTA performance in this work, we believe that improving sample efficiency, by an IL+RL pipeline or with new RL algorithms such as SAPG [93] , is an exciting research direction for future work.POLIFORMER even outperforms Phone2Proc[17], a baseline finetuned in3D-reconstructed test scenes.We report ProcTHOR results from[87] as these are the most up-to-date for recent AI2-THOR versions.SPOC* is similar to SPOC but is trained on more expert trajectories (2.3M vs. 100k). 
 
 Appendix:  


Decision: Strengths:
- Use of transformer encoder / decoder for scaling RL
- Well written, with demonstrations across four benchmarks and two embodiments
- Rare mix of transformers and RL, all strong accepts from reviewers

Weaknesses
- Reliance on on-policy RL makes method less sample efficient. Method is highly compute intensive making it less accessible.

Reviewers unianimously agree on a strong accept for the paper after rebuttal phase.

Reviews: review: title: title: Official Review of Paper243 from Reviewer bADd
paper_summary: paper_summary: The paper introduces PoliFormer, an innovative reinforcement learning (RL) model for indoor navigation tasks that leverages a vision transformer encoder and a causal transformer decoder for enhanced long-term memory and reasoning. Trained extensively in simulation, PoliFormer achieves state-of-the-art results in object goal navigation, significantly outperforming previous models. The model excels in both simulated and real-world environments, demonstrating strong sim-to-real transfer capabilities without additional fine-tuning. Key contributions include scalable on-policy RL training using hundreds of parallel rollouts, effective use of transformers for navigation, and robustness across diverse tasks and environments. The paper also highlights potential extensions of PoliFormer to various downstream applications such as multi-object navigation and object tracking.
main_review: main_review: ## Quality
The paper exhibits a high level of technical quality, with a well-thought-out experimental design and thorough evaluation. The use of extensive training data and multiple benchmarks strengthens the validity of the results. The authors also provide detailed ablation studies to dissect the contributions of various components of PoliFormer.

## Clarity
Overall, the clarity of the paper is commendable. The authors present the complex concepts and methodologies in a structured manner. However, certain sections, such as the specifics of reward shaping and the precise hyperparameters used, could benefit from additional detail to aid in reproducibility and deeper understanding.

## Originality
PoliFormer’s integration of transformer architectures with on-policy RL is a novel contribution. This work pushes the boundaries of what has been achieved in embodied navigation tasks, particularly by leveraging the strengths of transformers in handling long-term dependencies and providing a robust framework for policy learning.

##Significance
The contributions of this work are significant in the field of embodied AI. By achieving state-of-the-art results and demonstrating robust sim-to-real transfer, PoliFormer sets a new standard for navigation tasks. Its ability to generalize to real-world scenarios without additional fine-tuning is particularly impactful, suggesting broader applicability and practical relevance.

##Strengths
- The combination of vision transformer encoders and causal transformer decoders represents a significant advancement in leveraging transformer architectures for RL tasks.
- PoliFormer achieves substantial improvements over existing models across multiple benchmarks, demonstrating robust performance.
- The model’s ability to perform well in real-world environments without additional fine-tuning highlights its robustness and practical applicability.

## Weaknesses
- On-policy RL’s inherent sample inefficiency requires extensive interactions, making it less practical for applications with limited computational resources. The significant computational resources needed for training (e.g., 32 GPUs, multi-node setups) limit the method's accessibility and scalability.
- While successful in navigation tasks, the model's generalization to other embodied AI tasks, such as manipulation or multi-agent interactions, remains unexplored and could require substantial re-engineering.
questions: questions: - Consider discussing potential strategies to improve sample efficiency, such as exploring off-policy methods or hybrid approaches that combine the strengths of on-policy RL with more sample-efficient techniques.
- The paper lacks detail on the specifics of reward shaping, which is crucial for long-horizon tasks. Including more information on the reward functions used and any techniques implemented to ensure effective exploration, such as intrinsic motivation methods or curiosity-driven exploration, would enhance the clarity and reproducibility of your work.
- Transformers introduce significant memory and computational overhead. Discuss any model compression techniques you have considered, such as pruning, quantization, or knowledge distillation, to mitigate these issues. This would be particularly relevant for deployment on resource-constrained robots.
- The evaluation primarily focuses on success rates and SPL. Adding additional metrics, such as collision rates, error recovery, and robustness under noisy conditions, would provide a more comprehensive assessment of the model's performance and robustness. This would offer a more complete picture of the model's capabilities and limitations.
- The paper does not discuss "NavFormer: A Transformer Architecture for Robot Target-Driven Navigation in Unknown and Dynamic Environments," which covers similar topics. Please revise your paper to include this work and clearly differentiate your contributions from it. Highlighting the unique aspects and advancements of PoliFormer in comparison to NavFormer will strengthen your paper.
limitations: limitations: 2
review_summary: review_summary: I strongly recommend the acceptance of this paper. The authors present PoliFormer, an innovative approach leveraging transformer architectures for on-policy reinforcement learning in indoor navigation tasks. The work demonstrates significant advancements in both architectural design and performance, achieving state-of-the-art results across multiple benchmarks. The model's robustness, scalability, and strong sim-to-real transfer capabilities are particularly noteworthy. Despite minor areas for improvement, such as enhancing sample efficiency and detailing reward shaping strategies, the paper's contributions are substantial and impactful. The thorough experimental validation and potential for broad applicability make this work a valuable addition to the field of embodied AI.
title: title: A solid advance in navigation using a flexible transformer-based method with a good, standardized evaluation.
paper_summary: paper_summary: The paper presents a technical advance in training transformer-based architectures for navigation problems, evaluates it to show strong improvement on standardized baselines, and refactors it to provide a useful foundation model based on bounding boxes.
main_review: main_review: Transformer-based methods are increasingly used in robotic navigation problems. Building on a wide variety of prior work and ideas, the paper presents PoliFormer, which combines efficiency innovations in how the transformers are structured with large-scale training (both in policy rollouts enabled by efficiency, and on broad training in ProcThor, also empowered by work done to speed it up). Overall, the system shows a marked improvement on standardized navigation benchmarks. In addition, the paper presents a refactoring of the system which is slightly less performant, but serves as a foundation model for any task that can be specified with a bounding box. Nicely done.

Quality: Solid. Very good grasp of prior work, clear presentation of methods and advances, solid evaluations, and an innovative new structuring in PoliFormer-BoxNav.

Clarity: Good. Overall, the paper is well written and easy to follow. Section 3.1, however, appears to refer to itself in a few places (line 132, 134).

Originality: Good. The ideas are an advance over several existing lines of research, but a lot of work was put into each component to enable the increase in performance shown.

Significance: Solid. The paper has three key advances: a better training recipe (on top of a better architecture), an improvement to the state of the art, and PoliFormer-BoxNav as a new paradigm for formulating navigation foundation models.
questions: questions: The paper looks pretty good, though it might be worth checking it for minor typos, e.g.:
- Section 3.1, appears to refer to itself in a few places (line 132, 134).
limitations: limitations: 3
review_summary: review_summary: The paper includes a better method, an improvement to the state of the art, a new formulation of the nav problem, evaluation against standard benchmarks, and real-world tests.
title: title: Novel idea supported by strong experiments, good paper!
paper_summary: paper_summary: This paper proposes an architecture and corresponding training recipe for end-to-end indoor navigation using reinforcement learning. The training process was fully conducted in simulation, and the rollouts in the real robots demonstrated its transferability into the real world. Experiments across two robot embodiments and four benchmarks show that the proposed method significantly outperformed baselines.
main_review: main_review: **Quality:** Good. The proposed architecture and scaling recipe look reasonable and technically solid, and the experiment results look promising.

**Clarity:** Good. The structure and writing are good and easy to follow.

**Originality**: Good. The work with large-scale on-policy RL papers on robotics tasks is quite rare.

**Significance**: Good. This work can encourage more work in this direction to scale up on-policy RL on robot tasks.

**Strengths:**

1. RL with modern Transformers architectures is quite rare. That is partially because those architectures are more data-consuming, which is contrary to the long-perusing goal of the RL community to reduce the sample complexity. This work demonstrates that with large-scale training in simulation, RL agents can also achieve zero-short Sim2Real adaptation in navigation tasks.
 
2. The experiment results across four benchmarks and two robot embodiments show significant improvement in comparison with previous methods.

3. The ablation studies demonstrate the importance of design choices for the important components in architecture.

**Weakness**

1. As a reader who is not familiar with robot navigation, I feel confused about the differences and challenges of the selected benchmarks. It would be nicer to include a short description for each benchmark.
questions: questions: 1. According to the description in Section 4, the agents were only trained in ProcTHOR environments and evaluated across four different benchmarks. Could the author explain a little bit about how different the training and evaluation environments are?
limitations: limitations: 3
review_summary: review_summary: I highly recommend accepting this paper because of its novel idea and well-executed experiments.
title: title: Impressive results, pushing SOTA, with questions on inference speed
paper_summary: paper_summary: This paper addresses the objectnav problem, where an embodied agent with a monocular camera navigates to an object specified by a bounding box. Unlike the simpler pointnav problem, which is considered solved with large-scale Reinforcement Learning (RL), objectnav benchmarks lag due to reliance on imitation learning, leading to stable but limited state space exploration. To overcome this, the authors propose training large transformers with RL. They use DinoV2 for image encoding, a transformer state encoder to combine goal specifications with image embeddings, and a transformer decoder for memorizing previous states, utilizing a KV cache for feasible large-scale RL training. The AI2-Thor environment generates diverse indoor scenarios, leading to state-of-the-art results across four benchmarks. The paper also suggests bounding boxes for goal specification, enhancing real-world applicability. Additionally, ablation studies and real-world experiments with two embodiments demonstrate successful zero-shot sim-to-real transfer.
main_review: main_review: This is an excellent paper that is clearly written, understandable, and presents groundbreaking results advancing the state-of-the-art (SOTA).

Strengths:
* Training transformers in RL has been infeasible for a long time, and using a KV cache is an innovative approach to addressing the computational complexity problem for transformers.
* The clear improvement in SOTA results is remarkable.
* The real-world experiments with zero-shot sim-to-real transfer are particularly notable, especially given that the training environments are simulated. It is commendable that DinoV2 facilitates this zero-shot sim-to-real transfer.

Weaknesses:
* Despite the impressive real-world experiments, the fact that the experiments are sped up 20x raises questions. It would be beneficial to understand why the model operates at such a slow speed and what could be done to enhance inference speed.
* Training large models at this scale is commendable, but the extensive use of GPUs and significant training time could make reproducibility challenging for most research labs.
* There is no mention of releasing the code and/or trained models.
questions: questions: * L134: Reference to section within section should be fixed.
* Would linear transformers (state space models) be a viable alternative?
* Is a code-release planned? Are trained models planned to be released? This would be essential, especially since this work redefines SOTA.
* Please address limitations on inference speed, and on continuous control. What needs to be done to get a model that actually runs in real-time on the robot?
limitations: limitations: 2
review_summary: review_summary: This paper presents a compelling and well-executed approach to addressing the objectnav problem, leveraging large transformers and RL to achieve impressive results. Minor issues need to be addressed, such as improving inference speed and ensuring code and model release for reproducibility. Overall, the paper makes a significant contribution to the field.