PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators
Authors: Kuo-Hao Zeng
Zichen Zhang
Kiana Ehsani
Rose Hendrix
Jordi Salvador
Alvaro Herrasti
Ross Girshick
Aniruddha Kembhavi
Luca Weihs
Venue: CoRL 2024 (2024)
Keywords: Embodied Navigation
On-Policy RL
Transformer Policy

Abstract: We present PoliFormer (Policy Transformer), an RGB-only indoor navigation agent trained end-to-end with reinforcement learning at scale that generalizes to the real-world without adaptation despite being trained purely in simulation. PoliFormer uses a foundational vision transformer encoder with a causal transformer decoder enabling long-term memory and reasoning. It is trained for hundreds of millions of interactions across diverse environments, leveraging parallelized, multi-machine rollouts for efficient training with high throughput. PoliFormer is a masterful navigator, producing state-of-the-art results across two distinct embodiments, the LoCoBot and Stretch RE-1 robots, and four navigation benchmarks. It breaks through the plateaus of previous work, achieving an unprecedented 85.5% success rate in object goal navigation on the CHORES-S benchmark, a 28.5% absolute improvement. PoliFormer can also be trivially extended to a variety of downstream applications such as object tracking, multi-object navigation, and open-vocabulary navigation with no finetuning.

PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators
Kuo-Hao Zeng
Zichen Zhang
Kiana Ehsani
Rose Hendrix
Jordi Salvador
Alvaro Herrasti
Ross Girshick
Aniruddha Kembhavi
Luca Weihs
PRIOR @ Allen Institute for Al
poliformer.alen.ai
Abstract: We present POLIFORMER (Policy Transformer), an RGB-only indoor navigation agent trained end-to-end with reinforcement learning at scale that gen- eralizes to the real-world without adaptation despite being trained purely in simu- lation. POLIFORMER uses a foundational vision transformer encoder with causal transformer decoder enabling long-term memory and reasoning. It is trained for hundreds of millions of interactions across diverse environments, leveraging par- allelized, multi-machine rollouts for efficient training with high throughput. Po- LIFORMER masterful navigator, producing state-of-the-art results across two distinct embodiments, the LoCoBot and Stretch RE-1 robots, and four naviga- tion benchmarks. It breaks through the plateaus of previous work, achieving unprecedented 85.5% success rate in object goal navigation the CHORES-S benchmark, a 28.5% absolute improvement. POLIFORMER can also be trivially extended to a variety of downstream applications such as object tracking, multi- object navigation, and open-vocabulary navigation with no finetuning.
Keywords: Embodied Navigation, On-Policy RL, Transformer Policy
Introduction
Reinforcement Learning (RL) has been used extensively to train embodied robotic agents to com- plete variety of indoor navigation tasks. Large-scale, on-policy, end-to-end RL training with DD- PPO enables near-perfect PointNav' performance when using shallow GRU-based [2] architec- ture. However, this approach fails to result in the same breakthroughs for harder navigation problems like Object Goal Navigation (ObjectNav [3]) where an agent must explore its environment to locate and navigate to an object of the requested type. RL approaches for ObjectNav have generally not advanced beyond shallow GRU architectures due to challenges presented by training instability and unreasonably long training times with wider and deeper models, such as scaled-up transformers [4].
In departure from on-policy RL, which is sample inefficient and often complex reward shap- ing and auxiliary losses [5], Imitation Learning (IL) has recently shown promise for ObjectNav. Ehsani et al. (2023) [6] demonstrated that the transformer- based SPOC agent, when trained to im- itate heuristic shortest-path planners, can be trained stably, is sample efficient, and significantly more effective than prior RL approaches on their benchmark. SPOC, however, ultimately falls short of mastery, plateauing at success rate ~57%. Critically, as we show in our experiments, the performance of SPOC does not seem to improve significantly when further scaling up data and model depth; suspect this is consequence of insufficient state-space exploration expert tra- jectory datasets frequently contain few examples of error recovery, which can lead to sub-optimal performance due to compounding errors [7] or non-trivial domain shifts during inference.
Point Goal Navigation the task of navigating goal location using privileged GPS coordinates.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Scaling On-Policy RL Training with Transformers
Centinual improvement with Scale
Hundreds of Poralel Rollouss
Trendforrw-bbeed Policy with Visaa Fourdation Modw
10
STATE TOKtH
Transtormat
Stale Enscoder
Cousal Transformer Dacodar
Vision Transformar
KV cachc
Laye
Search
Goal Encoder
Hundrecs of Millors of Mode Parameters
Simulation
Real
LoCoBot
Clipc Yor
Figure 1: POLIFORMER, a transformer-based policy trained using RL scale in simulation, achieves significant performance improvements in simulation (bottom-lefi) and the real world (bottom-right), across two embodiments. SR denotes Success Rate. We scale on-policy RL training across multiple dimensions: (top-left) we observe continual performance improvement with scaling RL training: (top-middle) we leverage hundreds of parallel rollouts for higher throughput; (top-right) we develop a transformer-bas policy scaling model parameters to hundreds of millions.
In contrast to II, RL requires that agents learn via interactive trial-and-error, allowing for deep exploration of the state space. This exploration has the potential to produce agents that can learn behaviors that are superior to those in the expert demonstrations. This raises the question: can we bring together the modeling insights from SPOC-like architectures and the exploratory power of RL to train a masterful navigation agent? Unfortunately, this cannot be done naively due to RL's sample inefficiency and complexities in using deep (transformer) architectures with RL algorithms. In this work, we develop effective method for training large transformer-based architectures with RL, breaking through the plateaus of past work, and achieving SoTA results across four benchmarks.
Our method, POLIFORMER, is a transformer-basee model trained with on-policy RL in the A12- THOR [8] simulator at scale, that can effectively navigate the real world without any adaptation. We highlight three primary design decisions that make this result possible. (i) Scale in Architecture: Building on the SPOC architecture, we develop a fully transformer-based policy model that uses powerful visual foundation model (DINOv2 [9], vision transformer), incorporates transformer state encoder for improved state summarization, and employs a transformer decoder for explicit tem- poral memory modeling (Fig. 1, top-right). Importantly, our transformer decoder is causal and uses a KV-cache [10], which allows us to avoid huge computational costs during rollout collection and makes RL training affordable. (ii) Scale in Rollouts: We leverage hundreds of parallel rollouts and large batch sizes, which leads to high training throughput and allows us to train using a huge num- ber of environment interactions (Fig. 1, top-middle). (iii) Scale in Diverse Environment Interactions: Training POLIFORMER MER with RI. at scale in 150k procedurally generated PROCTHOR houses [11] using optimized Objaverse [12, 13, 14] assets results in steady validation set gains (Fig. top-lett..
POLIFORMER achieves excellent results across multiple navigation benchmarks simulation. On CHORES-S, it achieves an impressive 85.5% Success Rate, a higher than +28..% absolute improve- ment over the previous SoTA model [6]. Similarly, also obtains SoTA Success Rates on Proc- THOR (+8.7%), ArchitecTHOR (+10.0%) and A12-iTHOR (+6.9%). These results hold across two embodiments, LoCoBot [15] and Stretch RE-1 [16], with distinct action spaces (Fig. bottom- left). In the real world (Fig. 1, bottom-right), it outperforms ObjectNav baselines in the sim-to-real zero-shot transfer setting using LoCoBot (+13.3%) and Stretch RE-1 (+33.3%),
We further train POLIFORMER- BEEERREEER-B-NNNN BOXNAV that accepts a bounding box (e.g., from an off-the-shelf ooen-vocab object detector [18, 19] or VLMs [20, 21]) as its goal specification place of a given
PPOLIFORMER outperforms Phone2Proc [17]. baseline finetuned in 3D 3D-rcconstructedd test scenes.

---

category. This abstraction makes POLIFORMER BAXNAV a general purpose navigator that can be "prompted" by an external model akin to the design of Segment Anything [22]. It is extremely effective at exploring its environment and, once it observes a bounding box, beelines towards it. POLIFORMER-BOX.
a leap towards training a foundation model for navigation; with no fur- ther training, this general navigator can be used in the real world for multiple downstream tasks such as open vocabulary ObjectNav, multi-target ObjectNav, human following, and object tracking.
In summary, our contributions include: (i) POLIFORMER, a transformer-baseee policy trained by RL at scale in simulation that achieves SoTA results over four benchmarks simulation and in the real world across two different embodiments. (ii) A training recipe that enables effective end-to-end policy learning with large-scale neural models via on-policy RL. (iii) A general purpose navigator, POLIFORMER-BOXXE that can be used zero-shot for multiple downstream navigation tasks.
2 Related Work
II and RL on Embodied Navigation. Recent advancements include Point Goal Navigation [1], Object Goal Navigation [3, 23, 24, 25]. Exploration [26, 27, 28], and Social Navigation [29], as well as successful sim-to-real transfer [6] and high performance in various downstream applica- tions [16. 30, 31. 32. 33. 34, 35j. The prevalent approaches to building capable navigation agents include both end-to-end training [1, 6] and modular methods that leverage mapping [36, 37, 38] or off-the-shelf foundation models [39, 40, 41, 42, 433. In these frameworks, the policy models are typ- ically optimized via Imitation Learning (IL) using expert trajectories [44, 45, 46, 47, 48, 49, 50] or Reinforcement Learning (RL) within interactive environments [29, 51, 52, 53, 54, 55, 57, 58, 59] and with carefully tuned reward shaping and auxiliary losses [5, 60, 61, 62, 63].
Transformer-basee based Policies for Embodied Agents. Recent works use transformer-based archi- tectures for embodied agents. Decision Transformer (DT [64]) learns policy offline, conditioning on previous states, actions, and future returns, providing path for using transformers as sequen- tial decision-making models. ODT [65] builds on DT and proposes blend offfine pretraining and online finetuning, showing competitive results in D4RL. [66]. More recently, MADT [67] shows few-shot online RL finetuning on the offline trained DT in multi-agent game enviromments. The Skill Transformer [68] learns a policy via IL for mobile manipulation tasks. While the focus of these works control towards specific tasks (relying on II. pre-training), we train from scratch using pure RL targeting a navigation policy that can be used zero-shot for many downstream tasks.
There has been a huge effort improving the scale and training stability of transformer poli- cies. GTrXL [69] proposes to augment causal transformer policies with gating layers towards stable RL training in [70]. PDiT [71] proposes to interleave perception and decision transformer- based blocks showing its effectiveness in fully-observed environments. GATO [72] learns a large- scale transformer-based policy on a diverse set of tasks, including locomotion and manipulation. Performer-MPC [73] proposes learnable Model Predictive Control policies with low-rank attention transformers (Performer [74]) as learnable cost functions. SLAP [75] learns policy for mobile manipulation based on a hybrid design. Radosavovic al. [76] learn causal transformer using RL and II. for real-world humanoid locomotion. However, their input is proprioceptive state, without visual observations, and their leamed policy is intended to be stationary (walking steadily in the rorldw NavFormer [77] learns image-goal conditioned transformer policies using offline- RL for target navigation. In contrast to existing approaches, we achieve efficient large-scale transformer- based on- on-policy RL training for a partially-observe
Long-horizon, navigation task using RGB sen- sor inputs and show that our policy can be seamlessly deployed in the real world. We accomplish this without complex inductive biases multi-task dataset aggregation due the sheer amount of procedural scene layouts and visual variety of objects in simulation.
Toward Navigation Foundation Models. Many recent works attempt to produce general-purpose "foundational" navigation models. The causal transformer for humanoid outdoor locomotion [76), relying on proprioceptive state, shows emergent behaviors like arm swinging and in-context adapta- tion. GNM [78] trains image goal-conditioned policies (i.e., conditioning goal location) by IL across various embodiments, which allows them to scale up the size of their
image the desired
3

---

Vision Transformen
Model
MLP
Bluas
Fael
Aoloon
Tima
Va Estmilio
Traneformer State Encoderr
Value cache
d² o/-
Goal Specification
Gool Sprcification
Search for sofa
MLP
Rasennhan Loyer
Key cache
Oplional b-box
Coonrram Tep Biibo
A1.Y0 11
area
Exooding
Noi
Figure 2: POLIFORMER is a fully transformer-based policy model. At each timestep f, it takes an ego-centric RGB observation it, extracts visual representations r using vision transformer model, further encodes state features s using the visual representations and goal features (and optional detected bounding box goal features Ø)), models state belief 1/ over time, employing causal trans- former decoder, and, finally, predicts action logits a² and a value estimation et via linear actor and critic heads, respectively. For rollout collection and inference, temporal cache strategy to prevent recomputing the forward pass for all prior timesteps at each new timestep, saving memory and speeding up both training and inference.
leverage the KV-cache [10] as our
training data and manages to generalize to controlling unseen embodiments. NOMAD [79], which extends ViNT [80] uses a diffusion policy and also uses image goal conditioning. Unlike these works, navigate from rich RGB image inputs and specify goals with natural-languse text (or with bounding boxes), such goals are far more easily available in real-world settings.
3 Method
RL, while effective and intuitive for training policies, has not yet been scaled embodied AI the same degree as in other domains. RL has primarily been applied to short-horizon tasks [81, 82], smaller environments [83, 84], tasks utilizing privileged knowledge [1]. We scale learning in three directions; (i) Network Capacity, via POLIFORMER'S 's deep, high-capacity, model architecture (Sec. 3. 1); (ii) Parallelism and Batch size via our highly-parallelized training methodology that lever- ages large batch sizes for efficiency (Sec. 3.2), and (iii) Training Environments via our optimization of the simulation environment to support high-speed training (Sec. 3.3).
3.1 The POLIFORMER Architecture
We now detail POLIFORMER'S 's transformer architecture (see Fig. 2), which is inspired by SPOC [6]. While we make several subtle, but important, changes to the SPOC architecture (detailed below), our largest architectural difference is in the transformer decoder: replace the standard transformer decoder block [4, 85) with the implementation used in the Llama 2 LLM [86]. This change has dramatic implications for training and inference speed.
At each timestep f, POLIFORMER takes an ego-centric RGB observation if as input and employs a frozen vision transformer model to extract a visual representation r'. This representation, along with an embedding g of the goal, are summarized by a transformer state encoder produce the state representation s², The causal transformer decoder then encodes the state feature s' (along with s0 s'-1,, into belief 6'. Finally, the linear actor and critic heads project b² to predict action logits a' and value estimate e', respectively.
Vision Transformer Model. Inspired by prior work [6, 87], we choose DINOv2 [9] as our visual foundation backbone, because of its remarkable dense prediction and sim-to-real transfer abilities. The visual backbone takes an ego-centric RGB observation i E H x3 as input and produces 24 x xh. where H and W are the observation height and width, and h is the hidden dimension of the visual representation. We reshape the visual representation into
patch-wise wise representation

---

a f X h matrix, f H W/196, and project the representation to produce e E R where d is the input dimension to the transformer state encoder. For effective sim-to-real transfer, it is important to keep the vision transformer model frozen when training in simulation.
Goal Encoder. For fair comparison, when training agents ObjectNav benchmarks using the LoCoBot, we follow EmbCLIP [39] and use one-hot embedding layer encode the target object category. On benchmarks using the Stretch RE-1 robot, we follow SPOC [6] and use FLAN-TS small [88, 89] model to encode the given natural language goal and use the last hidden state from the T5 model as the goal embedding. Before passing the goal embedding to the transformer state encoder, we always project the embedding to the desired dimension d, resulting in E IR 1xd
In select experiments (detailed in Sec. 4), we specify the goal object via bounding box (b-box), either in addition to an alternative to text. In this case, the goal encoder processes the b-box using both the sinusoidal positional encoding and the coordinate-type embeddings to embed the top- left, bottom-right b-box coordinates and its area (5 values in total), followed by an MLP to project the hidden feature to the desired dimension d, resulting in Go E Rxxd
Transformer State Encoder. This module summarizes the state each timestep as vector s E RR The input to this encoder includes the visual representation v. the goal feature (and/or g6), and an embedding ʃ of STATE token. We concatenate these features together and feed them to non- causal transformer encoder. This encoder then returns the output corresponding to the STATE token as the state feature vector. Since the transformer state encoder digests both visual and goal features, the produced state feature vector can also be seen as a goal-conditioned visual state representation.
Causal Transformer Decoder. We modeling over time. This can enable both long-horizon (e.g., exhaustive exploration with back- tracking) and short-horizon (e.g., navigating around an object) planning- Concretely, the causal transformer decoder constructs its state belief 6' using the sequence of state features = {s> {s within the same trajectories.
causal transformer decoder perform explicit memory
Unlike RNN-based causal decoders [90] which, during rollout collection, require only constant time compute the representation at each timestep, standard implementations of causal transformers require 1 time to compute the representation at timestep f. This substantial computational cost makes large-scale on-policy RL with causal transformer models slow. To overcome this challenge, we leverage the KV-cache technique [10] to keep past feed-forward results in two cache matrices, one for Keys and one for Values. With a KV-cache, our causal transformer decoder only performs feedforward computations with the most current state feature which results in computation time growing only linearly in rather than quadratically. While this is still mathematically slower than the constant time required for RNN-based decoders, empirically, between K <V-cache-quuppped causal transformers and RNNs in overall training FPS for our setting. Compared to other temporal cache strategies, we found that K KV-Cache offers the most significant speed improvements. Please see App. B.3 and Fig. 5 for discussion of the impact on training speed resulting from different cache strategies.
found only a small difference
3.2 Scalable RL Training Recipe
While KV-cache accelerates the causal transformer, this alone is not sufficient to enable efficient and fast RL training. In this section, we describe the methodology we use to achieve faster training. Al- though each of these individual findings may have been discussed in other works, emphasize the critical importance of these hyperparameters for efficient training and, consequently, stellar results.
We parallelize the training process using multi-node training, increasing the number of parallel roll- outs by a factor of compared to previous approaches [30, 59, 87] (we use 192 parallel rollouts for Stretch RE-1 agents and 384 for LoCoBot agents). We employ the DD-PPO [1] learning algorithm across 4 nodes, utilizing a total of 32 A6000 GPUs and 512 CPU cores. This scaling accelerates the training speed by approximately 3.4x, near- lenr-r gain compared single-node training; we train for ~4.5 days with 4 nodes, while this would have required 15.3 days with a single node.
5

---

Our batch size during training, in number of frames, is equal to the number of parallel rollouts (R) multiplied by the length of these rollouts (7) and thus, by increasing R as above, we multiplicatively increase the total batch size. We follow SPOC [6] and use a small constant learning rate of 2 10-4 throughout the experiments. Instead of annealing the learning rate during training, we instead follow PROCTHOR [11] and increase the batch size by changing the rollout length from T-32, T-64, and finally to T=128 (resulting in final batch size of 49,152 frames for LoCoBot agents). We make these increases every 10M steps until we reach 128.
3.3 Scaling Environment Interactions
For the experiments on LoCoBot, we use the original PROCTHOR-IOK houses for training for a fair comparison to baselines. With the KV-cache technique, our training steps per second is ~2.3k for training POLIFORMER, using 384 rollouts 32 GPUs. For the experiments on Stretch, fol- lowing [6], we use the PROCTHOR-ISS --0k houses with 440k annotated Objaverse 3D assets. As on-policy RL requires the agent to interact with the environment on-the-fly, we found the continu- ous physics simulation used by the Stretch RE-1 agent in AI2-THOR to be too slow to efficiently train at scale. To overcome this, we discretely approximate the agent's 's continuous movement via a www.eioooiiit--oolssiooo---o-oee
approach. In particular, to move the agent, we perform a physics cast using capsule collider representing the agent along the desired movement direction. If the cast does not hit any object (suggesting potential collisions), location, otherwise we let A12-THOR simulate continuous movement as usual. For rotations, we first teleport the agent to the target rotation pose. If the agent's s capsule collides with any object, we teleport the agent back and let the AI2-THOR simulate the continuous rotation. For our tasks, these approximations increase simulation speed by ~40%. We also found that AI2-THOR scene loading and resetting was significant bottleneck when using Objaverse assets. We streamline A12- THOR's Objaverse 3D asset loading pipeline by saving objects in the Magpack format (natively usable with Unity's C# backend) opposed to Python-only Pi Pickle files. This change dramatically decreases the loading time of a new scene from ~55 to 8s. With all these changes, our training steps per second increases from <550 to ~550 using 192 rollouts on 32 GPUs for training Stretch RE-1 agent, reducing the training time by 42%%
teleport the agent to the target
4 Results
We now present our experimental results. In Sec. 4.1 we demonstrate that scaling RL with Po- LIFORMER produces SoTA results four simulation benchmarks across two embodiments (Lo- CoBot and Stretch RE-1). Then, in Sec. 4.2, show that POLIFORMER, despite being trained purely in simulation, transfers very effectively to the real world, outperforming previous work; we again show these results on the above two embodiments. In Sec. 4.3, we provide ablations for vari- ous design choices. Finally, we qualitatively show that POLIFORMER - a variety of downstream applications in a zero-shot manner in Sec. 4.4.
BOXNAV can be extended to
Baselines. For our baselines, chose set of prior works in both imitation learning and reinforce- ment learning. SPOC [6] is supervised imitation learning baseline trained shortest path expert trajectories in AI2THOR. SPOC* is similar to SPOC, but is trained on more expert trajectories (2.3M vs. 100k). We build this baseline to verify if SPOC easily scales with more data. Emb- SigLIP [6] is an RL baseline also used by [6], but trained with comparable GPU hours with SPOC. The baselines [11, 63, 87) for LoCooot-emboddmmen are all trained by RL. using the same training steps used by POLIFORMER. We have three experiment configurations for our studies, specifying the goal as (a) natural language instruction text, (b) b-box for the target object, and (c) b-box & text. Following [6]. b-boxes in simulation are the ground-truth b-boxes, and such models not comparable fairly with RGB-only models. In the real world we replace GT b-boxes with outputs from the open-vocabulary object detector Detic [18], which uses only RGB images as input, and so all comparisons are fair.
Implementation Details. For LoCoBot benchmarks, we follow [87] train our model and base- lines 10k training scenes in ProcTHOR houses for 435M training steps. The evaluation sets consist of 800 tasks in 20 scenes for A12-iTHOR, 1200 tasks in 5 scenes for ARCHITECTHOR,

---

Inpuis
CHORES ObjectNav
Inpues
PROCTHOR-10L
ARCHERCTHOR
AI2-ITHOR
(SEL)
Success
SPOC [6]
B.
57.0 (46.3)
PROCTIOR (11)"
(38.3
(5.0)7
RGB-
EmbSigL.I POLIFORMER
(24.5)
RGB-MeA
EnRCudehook (87)
[63]
20.3
78.7
53.8
(35.6)
(61.2)
POLIPORNER
(584)
(45.1)
(72.7)
RGB Hostebb-
POLIFORMER
(71.4)
RGB texteb-box
POLIFORMER
90.4 (66.5)
(55.6)
(83.5)
RGB+b-boo
POLIFORMER
POLIFORMEE
(78.6)
(a) Stretch RE-1 on CHORES
(b) LoCoBot on ProcTHOR- (val), ArchitecHOR and A12-ITHOR (test)
Table 1: Across four ObjectNav benchmarks, POLIFORMER obtains SoTA performance. (a) Results on the CHORES-S ObjectNav benchmark, which uses the Stretch RE-1 embodiment, POLIFORMER dramatically outperforms the previous SoTA, IL-ttrined SPOC. (b) On three LoCoBot-embodiment test suites, POLIFORMER outperforms all prior work (all trained using RL).
Model
PROCTHOR-10K (val)
ARCHIECTHOR
Model
Stretch RE-1
LoCoBot
Backbone
Encoder
ProcTHOR [11]
26.7
CLIP (ReaNar50)
Ix
GRR
55.8
Phone2Proc [17)
66.7
DENOv2 (STT))
SPOC
59.5
POLIFORMER (ours)
83.3
80.0
DENDv2:
DINO
SPOC-Detic [6]
83.3
DINO (VITH)
PoLIFORMER Detic (ours)
88,9
(a) Ablations on design choices for scaling model capacity
(b) Real-world results- Success
Table We present (a) ablation studies on design choices for scaling up model capacity; and (b) the real-world results, on two different embodiements.
and 1500 tasks in 150 scenes for PROCTHOR-I0K, For Stretch-REI, we train our model and base- lines on 150k ProcTHOR houses populated with Objaverse assets processed by ObjaTHOR. Then, we evaluate on 200 tasks in 200 scenes as in [6]. Training takes 4.5 days on 32 GPUs. Please see App. for more details about training, inference, and environment setups.
4.1 POLIFORMER Achieves SoTA on four Benchmarks
Table la shows that, on the CHORES-S benchmark, POLIFORMER achieves 28.5% absolute gain in success rate over the previous best model. EmbSigLIP baseline results are taken from [6]; we suspect that its poor performance is, in part, due to its training budget being limited to match the training budget of SPOC.
POLIFORMER works across embodiments. Table 1b shows that the remarkable performance of MER is not limited to one embodiment. The LoCoBot and Stretch-RF1 robots have different body sizes, action spaces, and camera configurations. Nevertheless, POLIFORMER is able to achieve 8.7%, 10.0%, and 6.9% absolute gain over the best baseline on PROCTHOR- 10k (val), ARCHITECTHOR, and A12-iTHOR.
4.2 POLIFORMER Generalizes to the Real World
We evaluated POLIFORMER on two real-world benchmarks for two different embodiments to show its sim-to-real transfer capabilities. We used the same evaluation set of 15 tasks for LoCoBot (3 dif- ferent starting poses with 5 different targets), and 18 tasks for Stretch-REI (3 different starting poses with 6 different goal specifications), similar to [17] and [6], respectively. We do not use any real- world finetuning, and the testing houses were not during training. In the text-only setting, Po- LIFORMER achieves remarkable improvement of 33.3% and 13.3% over the baselines for Stretch- REI and LoCoBot respectively (Tab. 2b). POLIFORMER even outperforms Phone2Proc [17], baseline that has access to privileged information about the layout of the environment for finetuning.
4.3 Ablation Studies
The recipe for building the effective RL navigation model is the end product of several design de- cisions guided by experimentation, the combination resulting in our SoTA performance. Table 2a, shows the process of scaling POLIFORMER. Row 1 vs. Row 2: Moving from the CLIP ResNet- 50 visual encoder to the ViT-small DINOV2 improves the success rate by an average of 5.2%.
We report ProcTHOR results from [87] these the most da-to-date for recent AI2-THOR versions.
7

---

Find the book tided 'Humans"
Fallow the
HUMANS
Robot sees book but continues to find the right book.
Find the kitchen
Robot's View
Find housoplant, toilet, and sofa
Robot's View: isques -bj-ooojeeee jct goal
Dynamic Environmants
Novel
People
Glass
Figure 3: We POLIFORMER-BEE -ARNAAV zero-shot find book with a particular title, navigate to kitchen, navigate to multiple objects sequentially, and follow a toy car around an office building.
Row 3 vs. Row 5: Using a 3-layer transformer decoder instead of 3-layer GRU increases the per- formance by 5.5%. Row vs. Row 5: Increasing the number of decoder layers from to 3 increases the performance by 4%. Row 5 vs. Row 6: Using larger capacity visual encoder (VTT-b instead of VīT-s), results 3.2% gain. In all, these changes result in 13.6% absolute avg. improvement.
Has POLIFORMER performance saturated? Figure 1 (top-left) shows CHORES-S ObjectNav validation success rate train POLIFORMER for 700M environment steps. As that plot shows, POLIFORMER's 's performance does not appear to have converged, and we expect that performance will continue to improve with more compute. This suggests that achieving near-perfect ObjectNav performance may only require further scaling our training approach.
4.4 Scaling POLIFORMER to Everyday Tasks
By specifying POLIFORMER'S goal purely using b-boxes, we produce POLIFORMER BERFORRMER-BBNAAV..... lacks the ability to leverage some helpful priors about where ob- jects of certain types should reside (comparing rows 6 & in Table we see POLIFORMER- BOXNAV performs slightly worse than model having both b-box and text goal specificationss, this design decision makes DOLLFORMER-BBXIA
While POLIFORMER--BXX
fully general-purposee promptable, navigation system that can navigate to any goal specifiable using a b-box. Figure 3 shows four qualitative ex- amples where, by using an vocabuacyyllary object detector (Detic [18]) and VLM (GPT-4o [91]), POLIFORMER- -ALLORRRRR-BOANA is able to, zero-shot, navigate to (1) a book with a particular title, (2) a given room type, (3) multiple objects sequentially, and (4) a toy car as the car driven around an office building. Simply by specifying goals as b-boxes and leveraging off off-the-shelf systems, we obtain complex navigation behaviors that would otherwise need be trained for individually: painful process requiring reward functions data collection. Finally, we encourage readers to visit our website to view more qualitative results presented in video format.
5 Discussion
Limitations: Training RL agents for long-horizon tasks with a large search space requires extensive compute and demands careful reward shaping. While we believe POLIFORMER is capable of scaling to other tasks, it requires crafting new reward models for novel tasks such as manipulation. More discussion on limitations in App. E. Conclusion: In this paper we provide a recipe for scaling RL for long-horizon navigation tasks. Our model, POLIFORMER, achieves SoTA results four simulation benchmarks and two real-world benchmarks across two different embodiments. We also show that FORMER has remarkable potential for use in downstream everyday tasks.

---

References
[1] E. Wijmans, A. Kadian, A. Morcos, S. Lee, I. Essa, D. Parikh, M. Savva, and D. Batra. DD- PPO: learning near-perfect pointgoal navigators from 2.5 billion frames. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpp:://oppnnreview http:////ppppppeeee...eaeeeaaaaa
[2] K. Cho, B. van Merrienboer, Ç. Gülçehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Ben- gio. Learning phrase representations using RNN encoder-decoder for statistical machine trans- lation. In A. Moschitti, B. Pang, and W. Daelemans, editors, Proceedings of the 2014 Con- ference Empirical Methods in Natural Language Processing, EMNLP 2014, October 25- 29, 2014, Doha, Qatar, A meeting of SIGDAT, Special Interest Group of the ACI., pages 1724-1734. ACL, 2014, doi: 1et.100..///// 1179. https:/:://do/ org 30.55/1/1 d14-1179.
[3] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans. ObjectNav revisited: On evaluation of embodied agents navigating to objects. CoRR, abs/2006. 13171, 2020. URL https: //txiiv////xiiv///////// wwaeziivvorr//eio /tt.//20.......
[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and Polosukhin. Attention all you need. In Guyon, U. von Luxburg. S. Ben- gio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017. URL https: /prooeeein/occciggs... neurips.... wwt..ees..c.pappeee...eeeeee pttp:pp.cc/pppee////////eaaeee /2a////h////h/a//// Abstract. html.
[5] J. Ye, D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectnav. CoRR, abs/2104.04112. 2021. URL https://arxiv: http:///axxvvvvog////// aat/abb//21000 04112.
[6] K. Ehsani, T. Gupta, R. Hendrix, J. Salvador, L. Weihs, K.-H. Zeng, K. P. Singh, Y. Kim, W. Han, A. Herrasti, et al. Imitating shortest paths simulation enables effective navigation and manipulation in the real world. In CVPR, 2024.
[7] N. Rajaraman, Han, L. F. Yang, K. Ramchandran, and J. Jiao. Provably breaking the quadratic error compounding barrier in imitation learning, optimally. ArXiv, abs/2102.12948, 2021. URL https: / //pii.pp... semanticacholar. wwwpccholarrrr/oooopiiiee
232046264.
[8] E. Kolve, R. Mottaghi, W. Han, VanderBilt, L. Weihs, A. Herrasti, M. Deitke, K. Ehsani, D. Gordon, Y. Zhu, A. Kembhavi, A. K. Gupta, and A. Farhadi. A12-THOR: An Interac- tive 3D Environment for Visual AI. ArXiv, abs/1712.05474, 2017. URL https://api.
semanticscholar org/CorpusuIi 28328610.
[9] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, R. Howes, P-Y. Huang, H. Xu, V. Sharma, S.-W. Li, W. Galuba, M. Rabbat, M. Assran, N. Ballas, G. Synnaeve, I. Misra, H. Jegou, J. Mairal, Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision, 2023.
[10] R. Pope, S. Douglas, A. Chowdhery, J. Devlin, J. Bradbury, J. Heek, K. Xiao, S. Agrawal, and J. Dean. Efficiently scaling transformer inference. In MLSys, 2023.
[11] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, Ehsani, Salvador, W. Han, Kolve, A. Kembhavi, and Mottaghi. ProcTHOR: Large-scale embodied AI using procedural gen- eration. In NeurIPS, 2022. URL http: /tatt:://papere mips wwpp.cc/ppaaaaeaee wwtppp/pppppeeeeeee_eeepeppp wttprr/fllesssepeeeeeeeee p2ppp/202///////////// haap:22266hteeetrreeeeaaaaaatttn
htmm..

---

[12] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A Universe of Annotated 3D Ob- jects. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13142-13153, 2022. URL https://appii htt.:/pp...mmmticcccholl.aa. wttpsscchhhaarro/rrrroraoe :D: 254685588.
[13] Y. Yang, F. Sun, L. Weihs, E. VanderBilt, A. Herrasti, W. Han, J. Wu, N. Haber, R. Krishna, L. Liu, C. Callison- Burch, M. Yatskar, A. Kembhavi, and C. Clark. Holodeck: Language
guided generation of 3d embodied AI environments. CoRR, abs/2312.09067, 2023. doi:
48550/ARRXIV.212.00007 URL https: pestts://doo /ttp::///oooo....../oooo org/10.48550 48550/ arriv. 2312. 09067..
[14] A. I. for AL. ObjaTHOR: Python package for importing and loading external assets into ai2thor. https://githi: com/a com/alleema hat/a/llaaaoobaahhaaaa/e 2024.
[15] A. Gupta, A. Murali, D. Gandhi, and L. Pinto. Robot learning in homes: Improving gener- alization and reducing dataset bias. In NewrlPS, 2018.
[16] C. C. Kemp, A. Edsinger, H. M. Clever, and B. Matulevich. The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environments. In 2022 In- ternational Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 3357. IEEE, 2022. doi:10.1 10910009//A4449920000000 19922. URL https: http:p://doo.org///./
[17] M. Deitke, R. Hendrix, L. Weihs, A. Farhadi, K. Ehsani, and A. Kembhavi. Phone2Proc: Bringing robust robots into our chaotic world, 2022.
[18] X. Zhou, R. Girdhar, A. Joulin, Krähenbühl, and Misra. Detecting wenty-thousand classes using image-level supervision. In S. Avidan, G. J. Brostow, M. Cissé, G. M. Farinella, and T. Hassmer, editors, Computer Vision ECCV 2022 17th European Conference, Tel Avis, Israel, October 23-27, 2022, Proceedings, Part IX, volume 13669 of Lecture Notes in Com- 1007/978-3-0311200000 000-737712077-.211
puter Science, pages 350-368. Springer, 2022. htttp::/ddo.org///
URL
[19] M. Minderer, Gritsenko, and N. Houlsby. Scaling voca-vocy object detection. In NeurlPS, 2023.
[20] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
[21] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. In NeurIPS, 2023.
[22] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao. S. Whitehead, A. C. Berg, W. Lo, Dollár, and B. Girshick. Segment anything. IEEE/CVF Interna- tional Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages IEEE, 2023. 10L11100//CCC1110000220... 10. 1D.1109/IIC 1t111099/CCC551000.22222 2023. 1037..
URL https://doi...
[23] A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffman, and D. Batra. ZSON: zero-shot object-goal navigation using multimodal goal embeddings. In NeurIPS, 2022.
URL http://papers nips cw/papcr_ppppeeeeeeeecceeeee httppppppeefeeeppeeeeeeepepeeeep www.:eewlpepleeeeeeee httpp///p//ppaa/// Abstract Conference .tml.
[24] S. Wani, S. Patel, U. Jain, A. X. Chang, and M. Savva. MultiON: Benchmarking Seman- Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: https::/proceediiiiiiie www.rpp.cccppppeee///..... wc/ppppe//220000000 220h//ash/////////////a///a 601188880000.17775110..111111113
tie Map Memory using Multi-Object Navigation.
Abstract.htmm
10

---

[25] R. Ramrakhya, E. Undersander. D. Batra, and A. Das. Habitat-webh Learning embodied object- search strategies from human demonstrations at scale. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 5163-5173. IEEE, 2022. doi: 10. 109/CCPRS268..0022.00 10. 109.9///PVP268B..... 2022. 00511.
URL https: http:://dooo.rrg./..
[26] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural slam. ICI.R 2020.
[27] B. Yamauchi. A frontier-based approach for autonomous exploration. In Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA "EAA''9."Towwe Towards New Computational Principles for Robotics and Automation', pages 146- 151. IEEE, 1997.
[28] Ye, D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectnav. CoRR, abs/2104.04112, 2021. URL
wwweiii.oorgaaaee wwggabs///eee 04112.
[29] X. Puig, E. Undersander, A. Szot, M. D. Cote, T. Yang, R. Partsey, R. Desai, A. W. Clegg, M. Hlavac, S. Min, V. Vondrus, T. Gervet, Berges, J. M. Turner, O. Maksymets, Kira, M. Kalakrishnan, Malik, D. S. Chaplot, U. Jain, Batra, A. Rai, and R. Mottaghi. Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots. CoRR, abs/2310.13724, 2023. doi: 10. 10.48550/ARXIV.2311 13724. URL https://doi. org/10. wtrgg/..48550/// att.a..ivviioooa 13724.
[30] S. Yenamandra, A. Ramachandran, K. Yadav, A. Wang, M. Khanna, T. Gervet, T. Yang, V. Jain, A. W. Clegg, J. M. Turner, Z. Kira, M. Savva, A. X. Chang, D. S. Chaplot, D. Ba- tra, R. Mottaghi, Y. Bisk, and C. Paxton. Homerobot: Open-vocabulary mobile manip- ulation. CoRR, abs/2306.
113300..1150 11565, 2023. 06::1008550AARNV2333.1.11 //doi. //doi.org/10 10///00.44 200.48850//r/a wtt.://................ 11565.
URL https:
[31] K. Ehsani, A. Farhadi, A. Kembhavi, and R. Mottaghi. Object manipulation via visual target localization. In S. Avidan, G. J. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, Computer Vision ECCV 2022 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, volume 13699 of Lecture Notes Computer Science, pages 321-337. Springer, 2022. doi:10.1007/998-333 031 19842-7 -0----3----112-711111 1007
URL https://://do org/100...1...
[32] N. Yokoyama, A. Clegg, E. Undersander, S. Ha, D. Batra, and A. Rai. Adaptive skill co- ordination for robotic mobile manipulation. ArXiv, abs/2304.00410, 2023. URL https: //api. aapi.semanticschola. www.:ollarrrrrr/ooooooooooo porg/CorrpssDD.. :218886222.6126
[33] A. Goyal, Xu, Guo, V. Blukis, Y. Chao, and D. Fox. RVT: Robotic View Transformer for 3D Object Manipulation. CoRR, abs/2306. 14896, 2023. N00:1048550AAAX1.2336.... 14896. URL https: org/10. /000000.22011 48550/ arXiv 2306 14896.
[34] Gu, D. S. Chaplot, H. Su, and J. Malik. Multi-skill Mobile Manipulation for Object Re- arrangement. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL httpss// pst/pdffid=Z3111 bzvP.
[35] J. Yang. C. Glossop, A. Bhorkar, D. Shah, Q. Vuong, C. Finn, D. Sadigh, and S. Levine. Pushing the limits of cross-embodiment learning for manipulation and navigation. In arXiv preprint arXiv:2402.19433 2024.
[36] D. S. Chaplot, R. Salakhutdinov, A. Gupta, and S. Gupta. Neural topological slam for visual navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[37] M. Chang, T. Gervet, M. Khanna, S. Yenamandra, D. Shah, S. Y. Min, K. Shah C. Paxton, S. Gupta, D. Batra, et al. GOAT: Go to any thing. arXiv preprint arXiv:231 2023.

---

[38] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating to objects in the real world. Science Robotics, 2023.
[39] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: CLIP embeddings for embodied AI. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 14809-14818. URL
IEEE, 2022. doi: 10. CVPR52688 2022. 2022.01441.
1109/
[40] M. Shridhar, L. Manuelli, and D. Fox. CLIPort: What and where pathways for robotic manip- ulation. In CoRL, 2022.
[41] A. Majumdar, K. Yadav, S. Arnaud, Y. J. Ma, C. Chen, S. Silwal, A. Jain, V. Berges, P. Abbeel, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaran, and F. Meier. Where are we in the search for an artificial visual cortex for embodied intelligence? CoRR, abs/2303. 18240, 2023, doi: (0.48550/ARXX12333... 18240. URL bttpp://doi.org/// 48550/arXiv 2303. 18240.
[42]
B. Ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, Sermanet, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L.. Luu, K. Lee, Y. Kuang, Jesmonth, N. J. Joshi, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakrishnan, B. David, A. Zeng, and C. K. Fu. Do as I can, not as say: Grounding language in robotic affordances. In K. Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 287-318. PMLR, 2022. URL pettps:/prroceeeee ichter23a html.
mlr, press/v205/
[43] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang. I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. PIVOT: Iterative visual prompting elicits actionable knowledge for vlms. In arXiv preprint arXiv:2402.078772 2024.
[44] S. Srivastava, C. Li, M. Lingelbach. R. Mart in-Mart'in in, F. Xia, K. Vainio, Z. Lian, C. Gokmen, S. Buch, C. K. Liu, S. Savarese, H. Gweon, J. Wu, and I.. Fei-Fei. BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environments. In Con- ference on Robot Learning, 2021. URL https: //api. semanticscholas org/CorpusID: 236957374.
[45]
C. Li, R. Zhang, J. Wong, C. Gokmen, S. Srivastava, R. Martín-Martín, C. Wang, G. Levine, M. Lingelbach, J. Sun, M. Anvari, M. Hwang, M. Sharma, A. Aydin, D. Bansal, S. Hunter, K. Kim, A. Lou, C. R. Matthews, I. Villa-Renteric J. H. Tang, C. Tang, F. Xia, S. Savarese, H. Gweon, K. Liu, J. Wu, and L. Fei-Fei. BEHAVIOR-IK IK: A Benchmark for Embodied AI with 1, 000 Everyday Activities and Realistic Simulation. In K. Liu, D. Kulic, and J. Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 80 PMLR, 2022. URL https: http://pooceeennneeeeee/e mlr pww.erpeaaaaee http///00/1222/ html.
[46] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K. Lee, S. Levine, Y. Lu, U. Malla, D. Man- junath. I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. S. Ryoo, G. Salazar, P. R. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. T. Tran, V. Vanhoucke, S. Vega, Q. Vuong. F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-1: robotics transformer for real-world control at scale. In E. Bekris, K. Hauser, S. L. Herbert, and Yu, editors, Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023. https: / /doi. org/10. 10. 10601/1677/00 /RSS. 2023. 025.
20X. URL
12

---

[47] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T. E. Lee, S. Levine, Y. Lu, H. Michalewski, Mordatch, K. Pertsch, K. Rao, K. Rey- mann, M. S. Ryoo, G. Salazar, P. Sanketi, Sermanet, J. Singh, A. Singh, R. Soricut, H. T. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-2: vision-language-attinn models transfer web knowledge to robotic control. CoRR, abs/2307.15818 2023.
URI.
https://doi.. htttp::/ddi..org/1.... 48550/ 485500/0Xiiv 15818.
[48] O. X.-F. Collaboration, A. O'Neill, A. Rehman, A. Gupta, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, A. Tung, A. Bewley, A. Her- zog, A. Irpan, A. Khazatsky, A. Rai, A. Gupta, A. Wang. A. Kolobov, A. Singh, A. Garg, A. Kembhavi, A. Xie, A. Brohan, A. Raffin, A. Sharma, A. Yavary, A. Jain, A. Balakrishna,
B. Kim, B. Schölkopf, B. Wulfe, B. Ichter, C. Lu, C. Xu, C. Le, C. Finn, C. Wang, C. Xu, C. Chi, C. Huang, C. Chan, C. Agia, C. Pan, C. Fu, C. Devin, D. Xu, D. Morton, D. Driess, D. Chen, D. Pathak, D. Shah, D. Büchler, D. Jayaraman, D. Kalashnikov, D. Sadigh, E. Johns, H. Foster, F. Liu, F. Ceola, F. Xia, F. Zhao, F. V. Fru- jeri, F. Stulp, G. Zhou, G. S. Sukhatme, G. Salhotra, G. Yan, G. Feng, G. Schiavi, G. Berseth, G. Kahn, G. Yang. G. Wang. H. Su, H.-S. Fang, H. Shi, H. Bao, H. B. Amor, H. I. Christensen, H. Furuta, H. Bharadhwaj, H. Walke, H. Fang, H. Ha, I. Mordatch, Radosavovic, I. Leal, Liang, J. Abou-Chakra, J. Kim, J. Drake, Peters, J. Schneider, J. Hsu, J. Vakil, Bohg, J. Bingham, Wu, J. Gao, J. Hu, J. Wu, J. Wu, J. Sun, J. Luo, J. Gu, J. Tan, J. Oh, Wu, J. Lu, J. Yang, J. Malik, J. Silvério, J. Hejna, J. Booher, J. Tompson, J. Yang. J. Salvador, J. J. Lim, J. Han, K. Wang. K. Rao, K. Pertsch, K. Hausman, K. Go, K. Gopalakrishnan K. Goldberg, K. Byrne, K. Oslund, K. Kawaharazuka, K. Black, K. Lin, K. Zhang, K. Ehsani, K. Lekkala, K. Ellis, K. Rana, K. Srinivasan, K. Fang, K. P. Singh, K.-H. Zeng, K. Hatch, K. Hsu, I. Itti,
A. Wahid, B. Burgess-Limeries
L. Y. Chen, L. Pinto, L. Fei-Fei, L. Tan, L. Fan, L. Ott, L. Lee, L. Weihs, M. Chen, M. Lepert, M. Memmel, M. Tomizuka, M. Itkina, M. G. Castro, M. Spero, M. Du, M. Ahn, M. Yip, M. Zhang, M. Ding, M. Heo, M. K. Srirama, M. Sharma, M. J. Kim, N. Kanazawa, N. Hansen, N. Heess, N. J. Joshi, N. Suenderhauf, N. Liu, N. D. Palo, N. M. M. Shafiullah, O. Mees, O. Kroemer, O. Bastani, P. R. Sanketi, P. T. Miller, P. Yin, P. Wohlhart, P. Xu, P. D. Fagan, P. Mitrano, P. Sermanet, P. Abbeel, P. Sundaresan, Q. Chen, Q. Vuong, R. Rafailov, R. Tian, R. Doshi, R. Mart Mn-Mart in, R. Baijal, R. Scalise, R. Hendrix, R. Lin, R. Qian, R. Zhang, R. Mendonca, R. Shah, R. Hoque, R. Julian, S. Bustamante, S. Kirmani, S. Levine, S. Lin, Moore, S. Bahl, S. Dass, S. Sonawani, S. Tulsiani, S. Song, S. Xu, S. Haldar, Karamcheti, S. Adebola, S. Guist, S. Nasiriany, S. Schaal, S. Welker, Tian, Ramamoorthy, S. Dasari, S. Belkhale, S. Park, S. Nair, S. Mirchandani, T. Osa, T. Gupta, T. Harada, T. Matsushima, T. Xiao, T. Kollar, T. Yu, T. Ding, T. Davchev, T. Z. Zhao, T. Armstrong, T. Darrell, T. Chung, V. Jain, V. Kumar, V. Vanhoucke, W. Zhan, W. Zhou, W. Burgard, X. Chen, X. Chen, X. Wang, X. Zhu, X. Geng, X. Liu, X. Liangwei, X. Li, Y. Pang, Y. Lu, Y. J. Ma, Y. Kim, Y. Chebotar, Y. Zhou, Y. Zhu, Y. Wu, Y. Xu, Y. Wang, Y. Bisk, Y. Dou, Y. Cho, Y. Lee, Y. Cui, Y. Cao, Y.-H. Wu, Y. Tang, Y. Zhu, Zhang, Y. Jiang, Y. Li, Y. Li, Y. Iwasawa, Matsuo, Z. Ma, Z. Xu, Z. J. Cui, Z. Zhang, Z. Fu, and Z. Lin. Open X-Embodiment: Robotic learning datasets and RT-X models. In ICRA, 2024.
[49] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et DROID: A large-scale in-the-wild robot manipulation dataset. In arXiv preprint ar 2xxx:2403..12445 2024.
[50] C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild wild robot teaching without in-the-wild robots. In RSS, 2024.
[51] Gu, F. Xiang, X. Li, Z. Ling, X. Liu, T. Mu, Y. Tang, S. Tao, X. Wei, Y. Yao, X. Yuan, P. Xie, Z. Huang, R. Chen, and H. Su. ManiSkil12: A Unified Benchmark for General- izable Manipulation Skills. In The Eleventh International Conference on Learning Rep-
13

---

resentations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
https://openrevies net/ wwwp//epptddpbbeeeee. dabaiiibbb__aDyrD...
[52] C. Li, F. Xia, R. Martin-Martin M. Lingelbach, S. Srivastava, B. Shen, K. E. Vainio, C. Gok- men, G. Dharan, T. Jain, A. Kurenkov, C. K. Liu, H. Gweon, J. Wu, L. Fei-Fei, and S. Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In A. Faust, D. Hsu, and Neumann, editors, Conference on Robot Learning, 8-11 November 2021, London, UK. volume 164 of Proceedings of Machine Learning Research, pages 455- mlr, htt.e/eiiiin.......araeee//e//// cieesssvv64/1122b6 html.
465. PMLR, 2021. URL htwph:ttp////t httptt::ppppccceeiiigee
[53] B. Shen, F. Xia, C. Li, R. Martin-Martin, L. Fan, G. Wang, C. Pérez-D' Arpino, S. Buch, Sri- vastava, L. Tchapmi, M. Tchapmi, K. Vainio, J. Wong, L. Fei-Fei, and S. Savarese. iGibson 1.0: A Simulation Environment for Interactive Tasks in Large Realistic Scenes. In IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Re- public, September 27 Oct. 1, 2021, pages 7520-7527. IEEE, 2021. doi: 109//R09// 2021.9636667. URL https: /twi//oiooooo org/ 10. 10.0..1..0000 11111110/RRRR II0S5110055518 2021 9636667.
[54] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. M. Turner, N. Maestre, M. Mukadam, D. S. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, Meier, W. Galuba, A. X. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training Home Assistants to Rearrange their Habitat. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Annual Conference Neu- ral Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 251-266 2021. URL https: httt:::/pproceeding. neurips cc/paper/202/ /0/0//hhhh////// Abstract html.
[55] F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A SimulAted Part-based Interactive ENvi- ronment. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
[56] X. Puig, K. Ra, M. Boben, Li, T. Wang. S. Fidler, and A. Torralba. VirtualHome: Sim- ulating Household Activities via Programs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pages 8494-8502. Computer Vision Foundation IEEE Computer Society, 2018. doi: 109/CVPR. 2018.00886. URL URL.http::/opeeaccess thecvf com/ wttp//ttttttpppeeeeeeee Sttpi___v__________ hrppp_221///tmm/uugggg www.:ammlaaiingooooooieeoeeeeeee 2018 paper html.
[57] C. Gan, J. Schwartz, S. Alter, D. Mrowea, M. Schrimpf, J. Traer, J. D. Freitas, J. Kubilius, A. Bhandwaldar, N. Haber, M. Sano, K. Kim, E. Wang, M. Lingelbach, A. Curtis, K. T. Feigelis, D. Bear, D. Gutfreund, D. D. Cox, A. Torralba, J. J. Di- Carlo, J. Tenenbaum, J. H. McDermott, and D. Yamins. ThreeDWorid: A Platform for Interactive Multi-Modal Physical Simulation. In J. Vanschoren and Yeung, edi- tors, Proceedings of the Neural Information Processing Systems Truck on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https http://tttatttttbbbmmrrrteeeeeee prwceedings wwtppps.ccppppppeee///////e httpppper/222//aae 7355000456812266633fff788883 oatp:a/deeeccc.oooooooeeeeeerrrr .tmll.
[58] Z. Zhang and L. Weihs. When learning is of reach, reset: Generalization in autonomous visuomotor reinforcement learning. arXiv preprint arXiv: Arxiv-2303. 17600, 2023.
[59] M. Khanna, Y. Mao, H. Jiang, S. Haresh, B. Schacklett, D. Batra, A. Clegg, E. Undersander,
A. X. Chang, and M. Savva. Habitat synthetic scenes dataset (hssd-200): An analysis of 3d scene scale and realism tradeoffs for objectgoal navigation. In CVPR, 2024.
[60] Z. D. Guo, M. G. Azar, B. Piot, B. A. Pires, and R. Munos. Neural predictive belief represen- tations. ICLR, 2019.
14

---

[61] Z. D. Guo, B. A. Pires, B. Piot, J.-B. Grill, F. Altché, R. Munos, and M. G. Azar. Boot- strap latent-predictive representations for multitask reinforcement learning. In International Conference on Machine Learning, 2020.
[62] K.-H. Zeng, L. Weihs, A. Farhadi, and Mottaghi. Pushing it out of the way: Interactive vi- sual navigation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[63] K. P. Singh, J. Salvador, L. Weihs, and A. Kembhavi. Scene Graph Contrastive Learning for Embodied Navigation. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 10884-10894, October 2023.
[64] L. Chen, K. Lu, A. Rajeswaran, Lee, A. Grover, M. Laskin, Abbeel, A. Srini- vas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence mod- eling. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processigg Systems 2021, NeurIPS 2021, December 6-14, 2021, vir- tual, pages 15084-15097, 2021. URL https:
http://rrreeeee..ee/eeee/e https///rrcceeeengggg hash/11499ff42204110077777333300
ceuuips..c/papeeee. Abstract html.
[65] Q. Zheng, A. Zhang, and A. Grover. Online decision transformer. In ICML, 2022.
[66] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for deep data-driven reinforcement learning. ArXiv, abs/2004.07219, 2020. URL https://api.
semanticscholar: org/CorpusID:2200 :118222711..
[67] L. Meng, M. Wen, C. Le, X. Li. D. Xing, W. Zhang, Y. Wen, H. Zhang, J. Wang, Y. Yang, al. Offline pre-trained multi-agen aaelt- decision transformer. Machine Intelligence Research, 2023.
[68] X. Huang, D. Batra, A. Rai, and A. Szot. Skill transformer: A monolithic policy for mobile manipulation. In CVPR, 2023.
[69] E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning. In ICML, 2020.
[70] C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. Küttler, A. Lefrancq, S. Green, V. Valdés, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, Legg, and S. Petersen. Deepmind lab. ArXiv, abs/1612.03801, 2016. URL attps://appi http:p////pp.semmaticccchaaaaa wtt.rrccccooaaooo/rrrrreeee :ng/CorppuII::3222 3221339.
[71] H. Mao, R. Zhao, Z. Li, Xu, H. Chen, Y. Chen, Zhang, Xiao, J. Zhang, and J. Yin. PDÍT: Interleaving perception and fecision-makinn transformers for deep reinforcement learn- ing. In AAMAS, 2024.
[72] S. Reed, K. Zolna, E. Parisotto, S. G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, T. Springenberg, et al. generalist agent. In TMLR, 2022.
[73] X. Xiao, Т. Zhang, K. Choromanski, E. Lee, A. Francis, J. Varley, Tu, Singh, P. Xu, F. Xia, al. Learning model predictive controllers with real-time attention for real-world navigation. In CoRI, 2023.
[74] K. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, Sarlos, P. Hawkins, Davis, A. Mohiuddin, L. Kaiser, al. Rethinking attention with performers. In ICLR, 2021.
[75] P. Parashar, V. Jain, X. Zhang, J. Vakil, S. Powers, Y. Bisk, and C. Paxton. Spatial-language attention policies for efficient robot learning. In CoRL, 2023.
15

---

[76] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 2024.
[77] H. Wang, A. H. Tan, and G. Nejat. Navformer: A transformer architecture for robot target- driven navigation in unknown and dynamic environments. In IEEE Robotics and Automation Letters, 2024.
[78] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: A general navigation model to drive any robot. In ICRA, 2023.
[79] A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration. In ICRA, 2023.
[80] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. Vint: A foundation model for visual navigation. In CoRL, 2023.
[81] T. Haarnoja, A. Zhou, S. Ha, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep rein- forcement learning. ArXiv, rBg/CorpusID:5718800
1103, 2018. URL httpp:://ap...mmmiicce..eeeee www.ssiiscmoolle......
[82] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies.
J. Mach. Learn. Res., 17:39:1-39:40, 2015. URL pettps//appes http://apppseeaniieeeeeeeee semanti cscholar
semanticschollarr org/
[83] W. Yang, X. Wang, A. Farhadi, A. K. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. ArXiv, abs/1810.06543, 2018. URL petps://apt: semanticscholar. wrg/CorppsID:55116004.
[84] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. A. Ried- miller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013. URL https://app http:://api.emmatiicehhee
semanticscholatt Stcchholarrrrg/oooooee parg/CorpusID.. 15238391.
[85] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
[86] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Es- iobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshom, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Ko- renev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, Mihaylov, Mishra, Molybog, Y. Nie, Poulton, Reizenstein, R. Rungta, Sal- adi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. Tan, B. Tang, Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Zhang, A. Fan, M. Kambadur, S. Narang. A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.092888, 2023. doi: doi:10.4855000 00110048500AARXX123000000 https://doi. org/10. 0rrg/0..8550000 aariiv.iiiivvi/iii 2307 09288.
URL
[87] A. Eftekhar, K.-H. Zeng, J. Duan, A. Farhadi, A. Kembhavi, and R. Krishna. Selective vi- sual representations improve convergence and generalization for embodied ai. arXiv preprint arXiv:2311.04192. 2023.
[88] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. In JMLR, 2020.
[89] H. W. Chung, Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Li, X. Wang, M. Dehghani,
S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin,
16

---

A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling instruction 2022. URL hwt.h:tt////aet //axxiv org/abs/ aa/aabba222111 201b//211..116.
instruction-finctuned finetuned language models,
[90] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio. Empirical evaluation of gated recurrent neural
networks sequence modeling. In arXiv preprint arXiv:1412.3555, 2014.
[91] OpenAl. GPT-4 technical report. CoRR, abx/2303.08774, 2023. doi: 10.48550/ARXXV.23000 08774. URL https://doli org/ 10. 400.48500/arii. 230..087744
[92] L. Weihs, J. Salvador, K. Kotar, U. Jain, K.-H. Zeng, R. Mottaghi, and A. Kembhavi. AllenAct: framework for embodied research. arXiv preprint arXiv:2008. 12760, 2020.
[93] J. Singla, A. Agarwal, and D. Pathak. Sapg: Split and aggregate policy gradients. In arXiv preprint arXiv:2407.20020, 2024.
17

---

Appendices for PoliFormer: Scaling On-Policy RL with Transformers Results in Masterful Navigators
These appendices contain additional information about
Zero-shot real-world applications (App. A),
Training procedure (App. B),
Environment, benchmarks, and quantitative real-world experiments (App. C),
Simulation evaluations (App. D), and
Limitations (App. E).
Please find our project website (see the poliformeraaleen.ii) that contains
Six real-world qualitative videos where POLIFORMER performs the everyday tasks of Sec- tion 4.4 (recall also Figure 3), and
Four qualitative videos in simulation showing our POLIFORMER's behavior in the four benchmark environments (CHORES, PROCTHOR, A12-ITHOR, and ARCHITEC- THOR).
A Details about Zero-shot Real-world Downstream Applications using an Open-Vocal Object Detector and VLM
By specifying POLIFORMER's goal purely using b-boxes, we produce FOLIFORMER-BOXNEE POLIFORMER-BOXNAY is extremely effective at exploring its environment and, once it observes a bounding box, takes a direct and efficient path towards it. We now describe how we utilize this behavior to apply POLIFORMER- BoXNAV zero-shot to a variety of downstream applications by leveraging an open vocabulary object detector (Detic [18]) and VLM (GPT-40 [91]).
Open Vocabulary ObjectNav. To perform open vocabulary object navigation (i.e., where one must navigate to any given object type), we simply prompt the Detic object detector with the novel object type, for example, Bicycle. As IETTORMER-BOXNA. relies on the b-box as its goal specification, it finds a bicycle the scene smoothly.
Multi-target ObjectNav. To enable multi-target object goal navigation, make few simple modifications the inputs and output of the Detic detector. On the input side, we query with multiple prompts simultaneously (one for each object type); for instance, HousePlant, Toilet, and Sofa, as shown in Fig. 3 (bottom-leftt. We then, on the output side, only return the b-box with the highest confidence score. Since the returned b-box also contains the predicted object type, we know what the target object the agent finds is when issuing a Done action. Therefore, we remove the found target from the list of target types, and reset the POLIFORMER'S KV-cache. If the agent issues a Done action without detected b-box, terminate the episode and consider it failure. As result, the agent is required to find all the targets from the list of target types to succeed in an episode.
Human Following. We change the Detic prompt to Person. Once a b-box is detected, Po- LIFORMER drives the agent to approach it. Our experiment participant continues to walk away, so the agent keeps approaching them to minimize the distance.
Object Tracking. In this example, we control a remote control car that moves in the environment, and prompt the agent to find the car. Similar to Human Following, we change the prompt to Toy Truck in this example. As a result, the agent keeps trying to move closer to the detected b-box of the RC car, while avoiding collisions with objects the dynamic scene.
Room Navigation. In this example, shown in Fig. 3 (middle-left), provide detections to the agent. As the agent sees no detections, it continuously explores the scene. As the agent explores, we
18

---

1 episode
2nd episode
1 episode
2nd episode
3rd episode
0
2
3
5
0
2
2
0
0
0
O
0
1
0
0
0
0
O
0
0
0
0
1
0
0
1
0
0
0
0
0
0
0
1
0
0
0
0
0
0
0
0
0
0
1
0
0
0
0
1
0
0
0
1 rollout
2nd rollout
Figure 4: Attention Masks for training with block lower triangular structure.
query GPT4-o every 5 timesteps with the prompt in a Kitchen? Please return Yes or No. with the most recent visual observation. Once GPT-4o returns Yes, the agent issues a Done action end the episode.
Instance Description Navigation. In this example, shown in Fig. 3 (upper-left), the agent is prompted to find a specific book titled "Humans". Detic can generate open-vocabulary bounding boxes using instance-level descriptions but we found that doing this alone leads to high false-positive rates. To reduce these errors, we use GPT4-o filter positive detections from Detic. In partieular, sample filtering prompt "Is there book titled "Humans" in this image? Please return Yes or No.". We find this combination works well in practice. The agent, not GPT-4o, remains responsible for deciding when it has successfully completed its task, and in the Fig. example sees many books in its search but perseveres and eventually finds the correct one.
B Additional Training Details
B.1 Reward Shaping
For reward shaping, we follow EmbCodebook [87] and PROCTHOR [1I] and use the implementa- tion in AllenAct [92]: Reenalty Rencenss + Rdistances where Rpenatty -0.01 encourages an efficient navigation, Reuccess 10 when the agent successfully completes the task (= otherwise), and Reistance is the change of L.2 distances from target between consecutive steps. Note that only provide nonzero Ratistance if the new distance is less than previously seen in the episode. We do not enforce negative reward for increasing distance. This formulation encourages exploration.
B.2 Episodic Attention Mask
During training, ensure that the causal transformer decoder cannot access observations or states across different episoeente
construct the episodic attention mask to only allow the past experiences within the same episode to be attended. In Fig. 4, show couple of possible rollouts collected during training. With the episodic attention mask, observations and states in an episode can only attend to previous ones within the same episode, in contrast with naive causal mask where they could also potentially attend to observations and states in previous episodes.
B.3 Different Temporal Cache Strategies
Besides the KV-Cache, in this subsection, ablate four different temporal cache strategies (shown in Fig. top):
19

---

oonarvatinn
SIAEE
HLP
Specitiation
Searrhh for sola
Alle Laye
Key oache
Tansfarel
Yt
Training Step per Second (SPS)
3000
(i) No-Cache
[ Feature-Cace
State-Cache
2250
KV-Cache
1500
750
Stretch RE-1
LoCoBot
Different Temporal Cache Strategies
Figure 5: Different temporal cache strategies and their impact on the training speed. We ablate four different cache strategies, including (i) No-Cache, (ii) Feature-Cache (iii) State-Cache, and (iv) KV-Cache, shown at top. The bottom chart shows the training Step per Second (SPS) achieved by different strategies, on both LoCoBot and Stretch RE-1 agents.
(i) No-Cache, we cache the raw frames (visual observation i²) to provide past experiences for the causal transformer decoder. Therefore, POLIFORMER has to rerun the feedforward across all modules with the cached frames and the latest visual observation at each timestep.
iii) Feature-Cache, cache the features, including visual representation of and goal embed- ding g" (as well as g¹). In this case, POLIFORMER needs to recompute the transformer state encoder and causal transformer decoder with the cached features and the latest feature at each timestep.
(iii) State-Cache, we cache the state feature vector st, Since it is right before the causal trans- former decoder, POLIFORMER only requires to pass the cache state features and the latest state feature vector to the decoder, without recomputing the visual transformer model, goal encoder, and transformer state encoder.
(iv) KV-Cache, as described in Sec. 3, we cache the Keys and Values inside causal trans- former decoder, further reducing the required computation time for the transformer decoder from f to t theoretically. In addition, since this strategy operates very end, all the computations required by modules preceding causal transformer decoder can be saved.
The speed profile results shown in Fig. bottom. It clearly shows that placing the cache closer to the causal transformer decoder improves the training efficiency significantly.
B.4 Hyperparameters for Training
Tab. lists the hyperparameters used in our training and model architecture design. Please find more details such scene texture randomization, visual observation augmentations, and goal specification randomization when using text instruction in our codebase.
20

---

Training and Model Details
Parameter
Value
Allowed Steps
600 (Stretch RE-1). 1), 500 (LoCoBot)
Total Rollouts
192 (Stretch RE-1), 384 (LoCoBot)
Learing Rate
0.002
Mini Batch per Update
Update Repeats
Max Gradient Norm
0.5
Discount Value Factor 7
0,99
GAE X
0.95
PPO Surrogate Objective Clipping
0.1
Value Loss Weight
0.5
Entropy Loss Weight
0,01
Training Stages
Steps for PPO Update Stage 1
32
Steps for PPO Update Stage 2
64
Steps for PPO Update Stage 3
128
Transformer State Encoder Layers
Transformer State Encoder Hidden Dims
512
Transformer State Encoder Heads
Causal Transformer Deocder Layers
3
Causal Transformer Deocder Hidden Dims
512
Causal Transformer Deocder Heads
Table Hyperparameters for training and model architecture.
C Additional Details about Environment, Benchmarks, and Real- Experiments
Action Space. Following prior work using AI2-THOR, we discretize the action space for both LoCoBot and Stretch RE-1. For LoCoBot, we discretize the action space into actions, including (MoveAhead, RotateRight, RotateLeft, LookUp, LookDown, Done), where MoveAhead moves the agent forward by 0.2 meters, RotateRight rotates the agent clockwise 30 around the yaw- axis, RotateLeft rotates the agent counter-clockwise by 30 around the yaw-axis, LookUp rotates agent's 's camera clockwise by 30 around the roll-axis, LookDown rotates agent's camera counter- clockwise by 30 around the roll-axis, and Done indicates that the agent found the target and ends an episode. We follow previous works [11, 17, 87] to use the same action space for LoCoBot for a fair comparison. For Stretch RE-1, we remove the LookUp and LookDown camera actions, and add MoveBack, RotateRightSmall, and RotateLeftSmall the action space, where MoveBack moves the agent backward by 0.2 meters, RotateRi ghtSmall rotates the agent clockwise by 6 around the yaw-axis, and RotateLeftSmall rotates the agent counter-clockwie by 6" around the yaw-axis. Again, this action space is identical to the one used in prior work [6] for fair comparison.
Success Criteria. We follow the definition of Object Goal Navigation defined in [3], where an agent must explore its environment locate and navigate to an object of interest within allowed number of steps n. The agent has issue the Done action indicate it found the target. The environment will then judge if the agent is within distance d from the target and if the target be seen in the agent's view. An episode is also classified as failed if the agent runs more than steps without issuing any Done action. Across different benchmarks, n and d vary depending on the scenes size and complexity and agent's capabilities. We follow ProcTHOR [11] to use n = 500 and d = meter for LoCoBot, and follow CHORES-S [6] to use 7 600 and
meters for Stretch RE-1.
SPL and SEL. Success Weighted by Path Length (SPL) and Success Weighted by Episode Legnth (SEL) are two popular evaluation metrics evaluate how efficient agent is to find the target. SPL. is defined as N ΣN S m. where N is the total number of episodes, S; is binary
21

---

Bedroom
Office
Bathroom
Kitchen
Storage
Livingroom
Start 2
Corridor
Start 3
Start
Starting Poses for LoCoBot
Bedroom
Office
Bathroom
Kitchen
Storage
Livingroom
Start 2
Corridor
Start 3
Start
(b) Starting Poses for Stretch RE-1
Figure 6: Starting Poses (a) LoCoBot and (b) Stretch RE-1 used the real world experiments. The arrow direction indicates where the agent faces with.
indicator of success for episode i, 4 is the shortest travel distance to the target, and is the actual w. travel distance. SEI is defined similarly: N >i= S maa(w.ee. where w; is the shortest number of steps to find the target, and e; is the actual number of steps used by the agent. By definition, SPL focuses how far the agent has traveled, while SEL focuses on how many steps the agent has used (which also penalizes excessive in-place rotation). SPL can be derived by computing the geodesic distance between the agent's starting location and the target's location, while SEL needs a planner with privileged environment information to calculate the number of steps expert trajectories. Therefore, we follow ProcTHOR [11] to report SPI. evaluate the LoCoBot agent, since those benchmarks do provide planner, while follow CHORES-S [6] report SEL, since expert trajectories available.
Real-world Experiment Setup. For the experiments using LoCoBot, we follow Phone2Proc [17] to use the same five target object categories, including Apple, Bed, Sofa, Television, and Vase, and the three starting poses, shown in Fig. (a). Among those target categories, Apple can found in the Living room and Kitchen, Bed can only be found in the Bedroom, Sofa and Television only be found in the Living room, and Vase be found in the Livingroom, Corridor, Office, and Kitchen. For the experiments using Stretch RE-1, we follow SPOC [6] to the same six target object categories, including Apple, Bed, Chair, HousePlants Sofa, and Vase, and the three starting poses, shown in Fig. (b). Among the categories not mentioned above, Chair can be found in the Living room, Office, and Kitchen, and HousePlant can be found in the Living room, Office, Bathroom, and Kitchen. Note that we use the hardware design intorduced SPOC [6] Stretch RE-1, Instead of using the offetff camera equipped on the Stretch RE-1 (due its narrow field of view), we use an Intel RealSense 455 fixed camera, which has vertical field of view of 59 and resolution of 1280 x 720. The camera is mounted facing forward but pointing downward, with the horizon at a nominal angle of 30". Please find more details in App. C.I. in SPOC [6].
Details about Different Benchmarks
First, during the training and testing stages, the possible target object types the agent is tasked with searching for include AlarmClock, Apple, BasketBall, Bed, Bowl, Chair, GarbageCan, HousePl ant, Laptop, Mug, Sofa, SprayBottle, Television, Toilet, and Vase. We then out- line the differences between the environments.
ii) A12-iTHOR: This environment consists of 60 training scenes, 20 validation scenes, and 20 test scenes, and our training and evaluation never touch the training scenes. All the scenes
22

---

are created by human designers, including the structure, layout, and object placements, producing 750 episodes for validation and 800 episodes for testing. Moreover, iTHOR has 4 different styles of scenes, including LivingRoom, Kitchen, Bathroom, and Bedroom, where each style contains the objects which are semantically associated with it. Since each iTHOR scene only contains a single room, it doesn't require much exploration but rather focuses recognition capability.
(ii) ArchitecTHOR: This environment consists of 10 high-quality large interactive houses de- signed by human designers, including for validation and for testing. Both validation and testing scenes have 1, 200 episodes each. It includes multiple rooms, larger navigable spaces, and more objects to explore. Since they are much larger than iTHOR scenes, they serve as a better benchmark to test the agent's exploration ability. Since Architec is designed by human designers, also conforms to human priors on room layout and object placement.
(iii)
ProcTHOR-val: This environment contains 1, 5,0 episodes across 150 procedurally gen- erated houses. The way these houses are generated follows the same pipeline used in ProeTHOR, Therefore, the distribution of layout styles, number of rooms, and object place- ments respects the ProcTHOR-100 training houses' distribution, where our LoCoBot agent is trained.
(iv) CHORES-S. S: This environment contains 200 episodes across 200 procedurally generated houses, using the same approach as ProcTHOR. However, it includes the Objaverse 3D as- sets, presented by SPOC, which introduced 41, 133 assets into the ProcTHOR-IIe creating a more diverse scene distribution. Our Stretch Agent is trained on 80, 000 houses out of the ProcTHOR 150k training houses; the 200 CHORES test houses and episodes are not seen during training.
houses,
D More Simulation Evaluations
Performance Variance. On CHORES-S, since we follow SPOC [6] to apply test-time data augmen- tation and non-deterministic action sampling, we found that performance varies even using the same checkpoint, especially given that we are only evaluating on 200 episodes. As result, we re-evaluate our POLIFORMER and SPOC*4 16 times and report mean success rate (mSR) and standard deviation (std). POLIFORMER achieves 82.5% mSR with 1.897 std, while SPOC" achieves 56.7% 7% mSR with 2.697 std. This result indicates that POLIFORMER not only achieves a higher mSR than SPOC*, but also exhibits more reliably consistent behavior, i.e. a lower std, when run on the same episodes multiple times.
EasyObjectNav
RegularObjectNav
HardObjectNav
Inputs
Model
Loss
Success (SEL)
Success (SEL)
Success (SEL)
SPOC [6]
IL
62.9 (40.5)
48.2 (38.9)
34.1 (27.4)
RGB+text
SPOC*
IL
69.7 (43.3)
53.5 (34.3)
31.0 (19.6)
POLIFORMER
RL
89.0 (62.1)
82.6 (71.8)
72.3 (62.8)
RGB
SPOC
90.3 (67.7)
78.7 (62.6)
70.6 (52.5)
Hext+b-box
POLIFORMER
RL
98.1 (86.5)
90.4 (79.6)
86.0 (75.0)
RGB+b-box
POLIFORMER
RL
97.1 (83.2)
91.9 (79.8)
87.6 (75.0)
Table 4: Large-scale evaluation results with different difficulty tiers. We evaluate performance on 2,000 episodes per tier.
Larger Scale Simulation Benchmark using Stretch RE-1. To further analyze POLIFORMER'S performance through different difficulty settings, construct 3 different levels of Object Goal Navigation benchmarks, BasyObjectlue RegularObjectNan and HardObjectNar where each
*SPOC* is similar SPOC but is trained more expert trajectories (2.3M 100k).
23

---

level contains 2k episodes, using Stretch RE-1. We construct these differentiated tasks by ensuring the oracle expert path length between the agent and target is to 3 meters long for EasyObjectNav, greater than meters for RegularOjjectNan and larger than 10 meters for HardObjectNav, The results are shown in Tab. 4. We observe that every model performs better as the agent is closer to the target at the episode start. In addition, on EasyObjectWav the agent barely needs exploration to find the target. Thereby, we find that POLIFORMER lagging behind POLIFORMER-BE MOAFFORRER--BOONVVV by 9% could result from Recognition Issue. Moreover, the gap HardObj HardObjectNav ectNav is widened to ~ 13.7%, and it could result from an additional Exploration Issue. The performance gap between HardObjecthan ectNav and EasyObjectNav could also support that an Exploration Issue exists, but not just the Recognition Issue.
Average Collision Rate (%)
PoliFormer w. Text goal
PoliFormer w. Box goal
PoliFormer w. Box Text goal
ProcTHOR-val ArchitecTHOR
2.2
2.9
2.8
3.0
3.6
3.7
A12-TTHOR
2.9
3.5
3.2
Table 5: Average Collision Rates for Different Benchmarks.
Average Collision Rate. shown in Tab. We measure average collision rate achieved by POLIFORMER using the LoCoBot agent in ProcTHOR-val, ArchitecTHOR, and A12-iTHOR. #colission
We compute the collision rate by #steps within an episode and we average the rate across all evaluation episodes to obtain an average collision rate.
ProcTHOR-10k val
PoliFormer w. Text goal
PoliFormer w. Box goal
PoliFormer Box Text goal
82.4 (58.5)
87.4 (56.2)
90.4 (66.6)
x=l
82.2 (56.7)
85.1 (55.0)
89.8 (65.3)
x=2
80.1 (52.9)
82.7 (51.1)
86.1 (59.7)
74.5 (41.9)
75.4 (42.3)
83.4 (50.1)
ArchitecTHOR
PoliFormer Text goal
PoliFormer Box goal
PoliFormer Box Text goal
x=0
68.3 (45.1) I)
85.7 (47.6)
81.9 (55.6)
x=l
67.0 (43.5)
85.1 (47.2)
81.9 (53.3)
x=2
63.3 (39.5)
78.9 (43.6)
78.2 (50.0)
x=3
60.1 (31.8)
69.6 (34.5)
71.5 (39.9)
A12-ITHOR
PoliFormer Text goal
PoliFormer w. Box
PoliFormer Box Text goal
goal
x=0
85.3 (72.7)
92.1 (78.6)
94.9 (83.5)
x=l
85.3 (71.0)
91.6 (78.3)
93.4 (81 (81.4)
x=2
85.1 (67.0)
90.0 (74.1)
92.6 (78.1)
x=3
80.8 (59.1)
86.0 (64.3)
90.6 (69.6)
Tabll 6: Success Rate and SPI. achieved by POLIFORMER with different level of noise across different benchmarks. We inject the gaussian noise G(O,, x o into the action, where is variable control the scale of the noise, and o
for movement and rotation degree for rotation. The LoCoBot agent has step size of 0.25mm and rotation degree of 30 degrees by default.
Robustness POL.FFORMER. To evaluate the robustness of PoliFormer, we inject the gaussian noise into the action only the evaluation stage. The results using LoCoBot across ProcTHOR-val. ArchitecTHOR, and AI2-TTHOR are shown in Tab. 6. POLIFORMER is robust to action noise when experiencing substantial perturbations (3 x is very large and unrealistic amount of noise).

---

E
Additional Discussion on Limitations
Depth Sensor. It is important to note that POLIFORMER is not equipped with a depth sensor (which has been proven to effective for manipulation). While the lack of the depth sensor does not affect our agent's performance on navigation, we acknowledge that integrating the depth sensor into visual representation is an interesting direction for future work, especially when considering mobile-manipulation extensions.
Discretized Action Space. To have a fair comparison with baselines, we use the same discretized action space in this work (see App. C). The discretized action space might not be efficient and realistic in many real-world scenarios where the agent must act in timely manner.
Cross-embodiment. In this paper, demonstrate that and Stretch RE-1. However, we have not yet explored training a single POLIFORMER for both embodiments. We leave this interesting research direction as future work.
train POLIFORMER using LoCoBot
Further Scaling. Our training and validation curves strongly suggest that even further scaling of model parameters and training time may lead even more masterful models than those we have trained in this work. This perspective is exciting and hope to enable further scaling with more computation resources and better visual foundation models in the near future.
Failure Analysis. The main mode of failure for POLIFORMER is the agent's limited memory. PO- LIFORMER clearly demonstrates memorization capabilities and is able to perform long- long-horizon tasks by exploring large indoor scenes without access explicit mapping. However, as the trajectories get longer (specifically after visiting more than 4 rooms in environment), the agent's recollection of the rooms it has explored deteriorates and the robot might re-visit rooms that it has explored previously.
Extensive Use of GPUs and Significant Training Time. We acknowledge that PoliFormer re- quires a large number of GPUs train efficiently. Our main focus in this work is on pushing the performance of navigation agents its limit without constraints on computing resources. Given the trend of GPU improvements over the last 10 years, believe that the resources used in this work will become relatively commonplace in the near future. Empirically, we found that training with smaller number of GPUs (e.g., 8 A6000 in single host) could yield similar final performance, with the downside of slower training speed (~~ 3.5x longer to train compared to 32 A6000 in four nodes). Moreover, there are many interesting research directions in improving training specd/efficiency; for example, since our submission, we have found that removing the average pooling from PoliFormer's visual encoder can increase convergence speed by up 2x. E.g., for the LoCoBot agent, we keep the number of tokens (e.g., 256 tokens) from DINOv2, instead of spatially average pooling them to 7 x 7 (resulting in 49 tokens). This implies that better visual encoder with a larger receptive view over the visual observation could even further speed up training. Furthermore, we believe that incor- porating more advanced techniques such as mixed precision training, flash attention, and advanced GPUs optimized for transformer models could reduce the training resource requirements.
Sample Efficiency. We hypothesize that 'IL pretraining +RL finetuning" paradigm could further improve sample efficiency. Early in on-policy RL training, policies trained from scratch are effec- tively random and thus produce vast amount of marginally useful experiences. Pretraining with IL may improve sample efficiency by providing helpful priors navigation and thus produce meaning- ful RL gradients faster. While it is nontrivial to implement this two-stage pipeline, we hypothesize that training PoliFormer using IL on expert data (e.g., g., the shortest path trajectories from SPOC) first and finetuning using on-policy RL. could be an option to improve the sample efficiency. Since we focus on obtaining SoTA performance this work, we believe that improving sample efficiency, by an IL+RL pipeline with new RL algorithms such as SAPG [93], is an exciting research direction for future work.
25


Reviews:
[{"review_id": "IYPajwvnBa", "reviewer": "Reviewer_bADd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces PoliFormer, an innovative reinforcement learning (RL) model for indoor navigation tasks that leverages a vision transformer encoder and a causal transformer decoder for enhanced long-term memory and reasoning. Trained extensively in simulation, PoliFormer achieves state-of-the-art results in object goal navigation, significantly outperforming previous models. The model excels in both simulated and real-world environments, demonstrating strong sim-to-real transfer capabilities without additional fine-tuning. Key contributions include scalable on-policy RL training using hundreds of parallel rollouts, effective use of transformers for navigation, and robustness across diverse tasks and environments. The paper also highlights potential extensions of PoliFormer to various downstream applications such as multi-object navigation and object tracking.", "summary_of_recommendation": "I strongly recommend the acceptance of this paper. The authors present PoliFormer, an innovative approach leveraging transformer architectures for on-policy reinforcement learning in indoor navigation tasks. The work demonstrates significant advancements in both architectural design and performance, achieving state-of-the-art results across multiple benchmarks. The model's robustness, scalability, and strong sim-to-real transfer capabilities are particularly noteworthy. Despite minor areas for improvement, such as enhancing sample efficiency and detailing reward shaping strategies, the paper's contributions are substantial and impactful. The thorough experimental validation and potential for broad applicability make this work a valuable addition to the field of embodied AI.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "- Consider discussing potential strategies to improve sample efficiency, such as exploring off-policy methods or hybrid approaches that combine the strengths of on-policy RL with more sample-efficient techniques.\n- The paper lacks detail on the specifics of reward shaping, which is crucial for long-horizon tasks. Including more information on the reward functions used and any techniques implemented to ensure effective exploration, such as intrinsic motivation methods or curiosity-driven exploration, would enhance the clarity and reproducibility of your work.\n- Transformers introduce significant memory and computational overhead. Discuss any model compression techniques you have considered, such as pruning, quantization, or knowledge distillation, to mitigate these issues. This would be particularly relevant for deployment on resource-constrained robots.\n- The evaluation primarily focuses on success rates and SPL. Adding additional metrics, such as collision rates, error recovery, and robustness under noisy conditions, would provide a more comprehensive assessment of the model's performance and robustness. This would offer a more complete picture of the model's capabilities and limitations.\n- The paper does not discuss \"NavFormer: A Transformer Architecture for Robot Target-Driven Navigation in Unknown and Dynamic Environments,\" which covers similar topics. Please revise your paper to include this work and clearly differentiate your contributions from it. Highlighting the unique aspects and advancements of PoliFormer in comparison to NavFormer will strengthen your paper.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QRPajPnN5G", "reviewer": "Reviewer_Ax82", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a technical advance in training transformer-based architectures for navigation problems, evaluates it to show strong improvement on standardized baselines, and refactors it to provide a useful foundation model based on bounding boxes.", "summary_of_recommendation": "The paper includes a better method, an improvement to the state of the art, a new formulation of the nav problem, evaluation against standard benchmarks, and real-world tests.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The paper looks pretty good, though it might be worth checking it for minor typos, e.g.:\n- Section 3.1, appears to refer to itself in a few places (line 132, 134).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "koy8TLpyaF", "reviewer": "Reviewer_jeoX", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an architecture and corresponding training recipe for end-to-end indoor navigation using reinforcement learning. The training process was fully conducted in simulation, and the rollouts in the real robots demonstrated its transferability into the real world. Experiments across two robot embodiments and four benchmarks show that the proposed method significantly outperformed baselines.", "summary_of_recommendation": "I highly recommend accepting this paper because of its novel idea and well-executed experiments.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. According to the description in Section 4, the agents were only trained in ProcTHOR environments and evaluated across four different benchmarks. Could the author explain a little bit about how different the training and evaluation environments are?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c0kY5bjpeb", "reviewer": "Reviewer_sGZ1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper addresses the objectnav problem, where an embodied agent with a monocular camera navigates to an object specified by a bounding box. Unlike the simpler pointnav problem, which is considered solved with large-scale Reinforcement Learning (RL), objectnav benchmarks lag due to reliance on imitation learning, leading to stable but limited state space exploration. To overcome this, the authors propose training large transformers with RL. They use DinoV2 for image encoding, a transformer state encoder to combine goal specifications with image embeddings, and a transformer decoder for memorizing previous states, utilizing a KV cache for feasible large-scale RL training. The AI2-Thor environment generates diverse indoor scenarios, leading to state-of-the-art results across four benchmarks. The paper also suggests bounding boxes for goal specification, enhancing real-world applicability. Additionally, ablation studies and real-world experiments with two embodiments demonstrate successful zero-shot sim-to-real transfer.", "summary_of_recommendation": "This paper presents a compelling and well-executed approach to addressing the objectnav problem, leveraging large transformers and RL to achieve impressive results. Minor issues need to be addressed, such as improving inference speed and ensuring code and model release for reproducibility. Overall, the paper makes a significant contribution to the field.", "technical_quality": 5, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* L134: Reference to section within section should be fixed.\n* Would linear transformers (state space models) be a viable alternative?\n* Is a code-release planned? Are trained models planned to be released? This would be essential, especially since this work redefines SOTA.\n* Please address limitations on inference speed, and on continuous control. What needs to be done to get a model that actually runs in real-time on the robot?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- Use of transformer encoder / decoder for scaling RL
- Well written, with demonstrations across four benchmarks and two embodiments
- Rare mix of transformers and RL, all strong accepts from reviewers

Weaknesses
- Reliance on on-policy RL makes method less sample efficient. Method is highly compute intensive making it less accessible.

Reviewers unianimously agree on a strong accept for the paper after rebuttal phase.
Decision: Accept