LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Video
Authors: Noriaki Hirose
Catherine Glossop
Ajay Sridhar
Oier Mees
Sergey Levine
Venue: CoRL 2024 (2024)
Keywords: Language-conditioned navigation policy
data augmentation
TL;DR: Data augmentation for in-the-wild video for learning language-conditioned navigation policy.

Abstract: We present our method, LeLaN, which uses action-free egocentric data to learn robust language-conditioned object navigation. By leveraging the knowledge of large vision and language models and grounding this knowledge using pre-trained segmentation and depth estimation models, we can label in-the-wild data from a variety of indoor and outdoor environments with diverse instructions that capture a range of objects with varied granularity and noise in their descriptions. Leveraging this method to label over 50 hours of data collected in indoor and outdoor environments, including robot observations, YouTube video tours, and human-collected walking data allows us to train a policy that can outperform state-of-the-art methods on the zero-shot object navigation task in both success rate and precision.

LeLaN: Learning A Language-Conditione Navigation Policy from In-the-Wild Videos
Noriaki Hirose Catherine Glossop!t, Ajay Sridhar Oier Mees Sergey Levine
University of California Berkeley 2 Toyota Motor North America cure contributors
noriaki. www.se@eereeeeeeeeeeeeeeeeee .dd
Abstract: The world filled with wide variety of objects. For robots to be useful, they need the ability find arbitrary objects described by people. In this paper, we present LeL.N (Learning Language-conditioned Navigation policy), novel approach that consumes unlabeled, action-free egocentric data to learn scalable, language-conen conditione object navigation. Our framework, LeLaN lever- ages the semantic knowledge of large vision-language models, as well as robotic foundation models, to label in-the-wild data from a variety of indoor and outdoor environments. We label over 130 hours of data collected in real-world indoor and outdoor environments, including robot observations, YouTube video tours. and human walking data. Extensive experiments with over 1000 real-world tri- als show that approach enables training a policy from unlabeled action-free videos that outperforms state-of-the-art robot navigation methods, while being ca- pable of inference at 4 times their speed on edge compute. We open-source our models, datasets and provide supplementary videos on our project page
Keywords: Language-conditiong navigation, Data augmentation
Augmentation
Object Navigation Policy
"iiht bench:
VIM
LLM
Filters
"Jeak silver Fridgge
halory oloy.
Depth
Pobject
Robotic foundation model
Rekutactons
"a
Hunansvcleing Daloeet
Figure 1: LeLaN leverages foundation models label in-the-wild video data with language instructions for object navigation. We train a sfate-of the-art robot policy on this data for solving challenging zero-shot language-conditioned object navigation tasks across variety of indoor and outdoor environments.
1 Introduction
One of the main challenges in robotics is enabling robots to interpret and execute arbitrary user- specified language instructions. For instance, in the case of navigation, a mobile robot should un- derstand that "go to the white and grey chair" and "go to the office chair next to the white desk and black computer monitor" could be asking the same thing in different ways. major bottleneck of training language-conditioned control policies access to high-quality and diverse language- and action-annotated robotics datasets, as acquiring human annotations for such datasets is an ex- pensive process. Some large datasets exist for training snnguuag--coditioone manipulation policies [1, 2, 3, 4]. but minimal data exists for training language-conditione conditioned navigation policies. Addi- tionally, existing language-annotatee robotics datasets are often not large enough to train end-to-end language-condition policies that exhibit strong generalization capabilities akin to foundation mod- els in domains such as natural language processing and computer vision. [5, 6, 7].
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Past works try to circumvent the lack of annotated robotics data by applying foundation models, such as vision language models (VLMs) and large language models (LIMM) in a zero- shot manner. This includes higher-level planning [8, 9, 10, 11, 12, 13], code-generated policies [14, 15, 16, 17], and zero-shot control [18]. The issue these methods face that the data distribution used to train these foundation models vastly different from the distribution of embodied robot experience. Additionally, these foundation models often too computationally expensive to on edge, which is important for mobile robotics. Other methods try to fine-tune these models on more narrow distribution of in-domain robotics data [19]. However, it is unclear if these model retain their full generalization capabilities after fine-tuning. In this paper, we approach the problem of grounding the abilities of foundation models in embodied experience in the context of language- conditioned policies for visual navigation by using scalable data augmentation.
In this paper, we leverage foundation models for vision, language, and robot actions, to annotate egocentric navigation experience with physically grounded language and action labels. Our dataset consists of egocentric navigation data from indoor ground robots, first-person view (FPV) tours on YouTube, and FPV data of a person walking in indoor and outdoor environments. We segment each image in the dataset with a segmentation model to retrieve objects interest for our language in- structions. We use a VLM to generate a description of the objects and an LLM to produce various language instructions for object navigation. We then generate counterfactual action labels that nav- igate the robot to the object in question using a robot foundation model (RFM) for navigation [20] and 3D estimated point cloud information, which is generated from monocular depth estimation model [21, 22). The complete process can be fully automated on any egocentric data and the re- sulting augmented dataset combines the strengths of spatially- and semantically-awaat foundation models. Policies trained on this data demonstrate broad generalization and robust physical ground- ing for language-conditioned navigation. Our contributions are the following:
1. A general data labeling method guided by foundation models that can consume unlabeled videos of action-free data,
2. Empirical results showing that the produced labels enable training a state-of-theearr policy that is more robust for noisy instructions, dynamic target object, and obstacles avoidance,
3. Data-level ablations showing the importance of including in-domain robotics data ground our trained policies and the effectiveness of our data augmentation,
4. We open source over 120 hours of egocentric data annotated with language and actions, including over 15 hours of human-collected video from 11 cities in 3 countries.
2 Related Works
Conditioned Policies for Robotics: Learning end-to-end language-conditionee policies for robotics has been studied extensively [23, 24, 19, 25, 26, 27, 28]. Learned methods are appealing because of their ability generalize beyond objects and tasks not seen in the training data. However, they often require language- and nction-annotated data, which can be expensive to collect and anno- tate in the real world. It may be cheaper to automate data collection in simulation [16, 29, 30), but policies trained in simulation often transfer poorly to unstructured real-world environments, which is commonly referred to as the sim2real gap [31, 32].
Language-Drive Zero-Shot Object Navigation: In this paper, we explore the problem of language- driven zero-shot object navigation (L-ZSON) [5], where agent navigates toward object speci- fied by a language command. The language input is often unstructured and could refer to an object that the agent has never seen before. Several methods attempt to apply foundation models zero-shot for language-guide in our experiments is CLIP on Wheels (CoW) [34], which uses CLIP, a multi-modal encoder that embeds text and images in shared latent space. CoW uses a CLIP [5] image encoder to compute similarity scores with CLIP text embedding corresponding the target object. Unlike many previ- ous ZSON works, which also evaluate search efficiency [35, 34, 36, 37], we focus the case where the object is within the line of sight of the robot [38]. This is because our system could be easily
vision-basse navigation [12, 33]. However, the primary baseline we consider

---

Indoor Nav. Dalasee
Segment Anything
Image Crops
CogVLM
Object Descriptions
YouTube Dataset
MeenccDD Depth350
pprwer Y Object Poses
Temporal robot actions
Human walking Dataset
NoMaD
Figure 2: Data Annotation. To generate diverse language instructions for object navigation, we generic egocentric image observations through series large pre-trained model filters that extract object bounding boxes and masks. We then use a VLM to describe the object(s) in the bounding boxes, and use an LLM to produce several diverse object navigation labels.
paired with a learned [39, 20, 37] or classical beachhbasse system [40, 41]. We address last-mile precision object navigation, the last portion of the L--SOON task.
Learning from In-the-Wild Videos: One major bottleneck for training policies for robotics is collecting in-domain data. Several works in robot learning have studied the use of in-the-wild video data for representation and reward learning [42, 43, 44, 45, 46] or affordance and direct policy learning [47, 48, 49]. One limitation of using mn-the-wild video data is the embodiment gap and lack of action labels. However, the scale and diversity of in-the- wild video data can lead to level of policy generalization that is hard achieve with narrow in-domain robot data.
3 Language-Condittone Navigation from Pethe-Wilde Videos
In this paper, study the task of "last-mile navigation" [38), where the robot must precisely igate to an object location that is visible from afar. Complementary to the well-studied task of semantic exploration [37, 50, 51, 10, 34], last-mile navigation focuses on fine-graines semantic understanding and precise navigation to the target object. We also show that our last-mile naviga- tion method can be combined with a high-level planner and long-term memory to navigate both partially seen and unseen environments.
We generate diverse language instructions and counterfactual robotic actions for action-free egocen- tric data. We use these instructions and actions to train robust annuage-conditione policy. In this section, we describe the LeLaN data augmentation pipeline and training methodology.
3.1 Labeling In-the-Will Videos with Foundation Models
major challenge in training language-conditonee control policies is access to high-quality and verse robot data with language and action annotations. Moreover, teleoperated robot trajectories for general navigation often lack diversity, traveling only down the center corridors and rarely heading toward objects. Therefore, even if human-made labeling used existing data, obtaining rich and useful labels for object navigation remains challenging.
We propose a novel method that leverages foundation models to label in-the-wild video data for training a language-conditione
navigation policy. Fig. shows the overview of our data augmenta- tion approach. Our method (i) scalable, as it uses abundant and readily available aetion-free data sources, (ii) efficient, it leverages all captured objects in the image view during data augmenta- tion, and (iii) generalizable, as it distills state-of seffthheart art large vision and language models to diverse and semantically meaningful annotations.
Language annotation: First, the in-the-wild video frames are passed through Segment Any- thing [52] to localize objects within the scene and produce masks and bounding boxes. The bounding boxes then used create image crops, which may include object of interest, We then feed the image crops into open-source visual language model (VLM) [53], which produces description of the object or objects in the crop. This description is passed to GPT-3.5-ture large language

---

model (LLM). If the LLM determines that the VLM confidently detected an object in the prompt, the LLM is then tasked with generating a prompt "Go "Go to X", where X are various descriptions of the object in the image crop based on the VLM's description.
In principle one could directly use VLM, such as GPT-4V [54], generate multiple prompts directly from image crops. However, at the time of the submission, both the cost of the closed- source VLMs and their inference time were high, while the open-source VLMs resulted in poor quality prompts. Therefore. we leverage an open-source VLM [53] to generate image descriptions, and GPT-335-turbo, which is fast and inexpensive, to generate prompts from the description.
Pose estimation and action generation: To ground the observation in the real 3D scene, we es- timate the 3D pose corresponding to the objects of interest as the goal pose and generate robot actions to move toward it without collision. For the pose estimation,
an approach similar to CoW [34] where we approximate the depth of the scene using Metric3D [22] for the narrow FOV camera image and Depth360 [21] for the wide FOV camera image and project the result into esti- mated point clouds. By masking the estimated point clouds with masks from Segment Anything [52] and taking the median values, we can obtain the 3D pose corresponding to an object for which we have matching language instructions from the VLM and LLM filtering.
To learn avoiiiion-avoidant behavior, we leverage an RFM [20] for navigation to generate the robot actions with this behavior to act as supervision when training our policy. By feeding in a context of observations and the cropped goal image, we can generate the sequence of the robot actions for each object that avoids collision and moves toward the target object, from our pre-trained policy. For more details on how the RFM was used, please refer to Appendix A.
Applying our approach to the i-th image o; in the video data, we have objects with the corre- sponding estimated poses {v: j=11..m and the sequences of the robot actions { [h] h for each object. M is the control horizon of the RFM. The number of the objects m is determined by the number of detections generated from Segment Anything with the GPT-3.5-turbe confidence filter. Each object has n prompts as ( Viggg=t..... where ni depends on the number of generated prompts.
3.2 Policy Architecture & Training
We train our language-condttioned navigation policy on our augmented dataset. We define the language-conditione navigation policy as {Uk, wk w= = ==((o))>=-L.... 1). Our policy generates the sequence of the virtual velocity commands {Te, k= N-. to move toward the tar- get object pose po by feeding the current and previous observations k= and the prompt I corresponding to the target object. Here, 2kk and te are the linear and angular velocity commands respectively. {ox k==-L... and / are taken from our labeled dataset by randomly selecting as the current state and j for the target object and for the instruction prompt. Depending on the selected i, j and g. the object pose and robot action are also uniquely given, such as pa and [p[k]]k=o...MM
Objective: We define a following overall objective incorporating three objectives, Jpuses Jeu and Janoah to train me encouraging object-reaching collision avoidance and smooth trajectories:
M
N
min J(O) = (po PN DN +
[p(h) - 22)
(1u++1 - 02)² + (WK+1 - wx) (I)
6
10
0=
(ia(H)
(a
where (pce) F (auna((()))))))))) W/: wk:) )o....))) is the 2D virtual robot poses from our policy, P. is the 2D object pose, which is the projected p. on the horizontal plane. Here, frobox(.) is the differen- tiable kinematic model. And N2 M) is the control horizon of our policy. Jpoce is for encouraging target object reaching, Jeol is for discouraging collisions, and Jamooth is for enabling smooth trajecto- ries. Since our pre-trained policy learns to beeline toward the target object location without Jeat, the robot can not avoid collision with objects between the robot and target objects. Hence, in addition to the base objectives Jpose + Jamooth introduce Jeal to distill the RFM, which contains an under- standing of collision-avoidant behavior learned from diverse navigation datasets [20, 55]. However,

---

since Jest will continue to encourage collision avoidance near target objects, we use to mask out Joo in the case, Pol < 1.0. Our training process is further described in Appendix and G.
3.3 Training Data
We use wide variety of egocentric datasets to train including: 1) Indoor Navigation Dataset:
image observations from mobile robot trajectories in office building environments, 2) YouTube Tour Dataset: YouTube video data of tours in various countries, and 3) Human-walking Dataset: data collected from walking with a camera in an indoor setting and outdoor city environments. A breakdown of this data is included in Table 1. More analysis of this dataset is shown in Appendix D.
Indoor Navigation Dataset:
Table I:
Breakdown of the datasets annotated with LeLaN YouTube Tour and Human-walking Dataset are sampled at FPS due to the faster walking speed compared to robot speed. GS, YT, and HW indicate Go
The robot dataset consists of an indoor general navi- gation dataset [56, 57, 58] which contains autonomous and teleoperated robot trajec- tories, none of which explic- itly contain object-reaching behavior. We employ three publicly available datasets, the GO Stanford 2 (GS2) Dataset [56], the GS4 Dataset [57], and the SACSoN (HuRoN) Dataset [58]
Stanford, YouTube Tour, and Human-walking. respectively.
Datasets
Hours [b]
Images [*]
Objects [4]
Prompis [8]
Indoor Nav. Dalasel
All
31.0
111570
594045
3418944
GS2 [56]
(17.6)
(63194)
(306041)
(1775073)
GS4 [57]
(10.5)
(37769)
(234266)
(1338117)
SACSoN [58]
(2.9)
(10607)
(53738)
(305754)
YT Dataset [59]
82.5
593727
3343956
14623072
HW Dataser
15.7
534544
231 1467
to cover a variety of office buildings and settings. Note that our data augmentation method is not limited to robot data, but includes to increase data diversity and provide additional grounding.
YubTube Tour Dataset: Incorporating YouTube videos into our dataset allows to enhance both the diversity and scalability of our approach. We curate 82.5 hours of YouTube videos that include vastly diverse domains and are significantly different from the university campus building settings seen in the Indoor Navigation Dataset. These videos cover a broad spectrum of objects, scenes, and camera heights and types, contributing to more generalized model performance. Specifically, we select videos that include home tours, outside walking (sightseeing) tour videos, and shopping mall tour videos from 32 different countries. Additionally, our dataset captures variety of weather conditions (e.g. snowy, rainy, and sumny) and time of day (e.g. morning, afternoon, and night), and diverse environmental types, such as urban, suburban, and rural settings.
Humannwalking Dataset: YouTube video can give us a diverse training set with many kinds of objects in various environments. However, we noticed that most of the cameras used to record YouTube videos have narrow field of view (FOV). However, wide FOV camera is useful for robot control [39, 57, 60]. To address this limitation, wide FOV (fisheyc) camera and collect dataset by walking in inside and outside environments. The Human- Human-walking Dataset includes 15.7 hours of data across 11 cities in 3 different countries.
collect dataset of additional data by holding
Experiments
Our experiments evaluate LeLaN in the real world, studying the following research questions:
Q1. Can LeLaN successfully navigate to novel objects conditioned on simple and noisy or com- plex language instructions?
Q2. Can LeLaN avoid collision with obstacles between the robot and target objects?
Q3. Can LeLaN generalize to different embodiments?
Q4. How does in-the-wild video improve the learned policy performance?
To isolate the advantages from our data labeling and the collision avoidance with the RFM in eval- uation. we evaluate our policy with the simplest implementation not considering Joa as well as not feeding the history images (I 0) in the evaluation. We include this objective when studying Q2
5

---

Table 2: Quantitative results in navigation using prototype real robot. show the goal success rale in 150 trials with 28 objects. Success is determined by the robot reaching within 0.2 [m] radius of the target object. YT and HW indicate the You Tuue Tour Dataset and summan-wllking Dataset, respectively. Inference time indicates the calculation time on Nvidia Jetson Orin to generate an action given the current observation. indicates that inference is performed only with the initial observation after which the classical controller takes over and i indicates incorporate the classical motion planner generate the velocity commands.
Method
Total
Simple premptts
Noisy prompts
Multiple objects
Dynamic object
Inferenee time [s]
OpNa [61]]
0.43
0.38
0.46
0.31
0.00
22,0*
OWL ViT [62]+ VINT
0.47
0.22
CoW [34]]
0.58
0.73
0.51
0.44
0.17
0.22
OWL-v2 [63] Zocdepth [64]f
0.67
0.73
0.63
0.62
0.00
2.82
method 0.0%. HW 0.0%)
0.66
0.64
(YT 20.0%, HW 43,5%))
0.85
0.91
0.82
0.70
0.83
0.054
(YT 102.0% HW 100.0%)
0.89
0,91
0.88
0.81
0.83
0.054
Our method
Baseline: CoW
prrbagebin
parbage
Starr Fcalfiss
Starl scariso
gros box with
pertage his with groon plants
Target object
Geen
method
Baseline:
Stort positoo
poaiiion
sevitier stacier
tentier poit
Target object
hand sontiizer reot
B: building entrance
Visualization of LeLaN performance. We conduct each experiment with 5 different prompts (right side) to visualize the robustness of LeLaN against noisy prompts. Our policy can navigate the robot toward the target object along very similar trajectories, showing its performance is highly reproducible.
Figure 3:
4.1 Evaluation on Diverse Language Instructions
We evaluate our trained policy on target object navigation, which tasks the robot with navigating toward a visible target object from the current robot pose. We evaluate our policy well four baseline methods for comparison. Please see Appendix F, G and H for details on the procedure and training hyperparameter settings, baseline methods, and prototype robot in evaluation.
We evaluate our method on a diverse set of language instructions. We run evaluations in five different environments, including 3 indoor floors and 2 outdoor spaces, across a set of 28 objects or structures in the scene. For each of these objects, we generate set of 5-6 language instructions of the form "Go to X" The form of X in these instructions ranges from simple, including only the object with no additional descriptors, to complex, where adjectives and relationships to other objects close to the main object in the scene are included. To evaluate the robustness of our method with a variety of instructions, we perturb the instruction in two ways: 1) by adding incorrect descriptors or similar but incorrect terms for the object and 2) by describing the object implicitly rather than explicitly. Our instructions are provided in the far right of Fig. and Appendix
Table shows the quantitative analysis from over 1050 evaluations. We categorize each instruc- tion into two categories, simple and noisy prompts, and show the success rate for each category to demonstrate the advantage of our method. Our method achieves an average success rate of 89% compared to 67%, the performance of the strongest baseline. The model is especially robust to noisy prompts, achieving an average of 25% higher success rate on our real-world test set against the baselines. Our method still performs well on simple, correct instructions that can be challenging for the baselines. In addition, we observe that our method outperforms the baselines on prompts with multiple objects, such as "Go to the chair next to the black suitcase". More detailed discussion and analysis are provided in Appendix B and E.

---

Figure 3 shows a visualization of our method deployed in two scenes. In each scene, we navigate the robot towards the same object, but with 5 different prompts shown on the rightmost side. Even though we provide noisy prompts, our method successfully navigates the robot towards the same objects. However, the baseline CoW fails navigation when provided with noisy prompts.
4.2 Capability Analysis on Challenging Settings
To analyze the capability of LeLaN, we conduct three more challenging evaluations: 1) collision avoidance for the obstacles between the initial robot location and the target objects, 2) long-distance navigation, which the robot can not capture the target object at the initial position, and 3) following the dynamic objects such as the pedestrians.
Collision Avoidance: For Q2, we evaluate LeLaN in environment with obstacles to de-
Table 3: Target object arrival rate in navigation with and without the obstacles. i indicates using the classical motion planner cosidering collision avoidance.
termine the effectiveness of our proposed ob- jective, Jeat, for distilling collision-avone avoidaan be- havior from the RFM. We run our policies with and without Jest as well as the strongest base- line, OWL-v2 + Zoedepth. We conduct 15 navigation experiments for different target ob- jects with various obstacles and show the target reaching accuracy in Table 3.
Method
with obstacle
wo obstacle
OWL-v2 [63] + Zoedepth [64]]
0.40
0.67
method wo Jost
0.13
0.89
Our method with Jad
0.60
0.77
Since collision avoidance is a challenging task when performing language-conditioned navigation, the accuracy of our method with Joal is 0.6. However, we can confirm that there is an explicit advantage gap compared to the best baseline method. In most cases, our method without Jool collides with the obstacle between the robot and the target object. To add collision-avoidant behavior to the OWL-v2 Zoedepth baseline, we implement classical motion planning in Appendix K. This baseline often fails navigation due to its slow inference speed, After passing the obstacles without collision, this baseline not change the trajectory toward the target objects. The robot behaviors are shown in the supplemental video.
Long-distance Navigation:
The LeLaN is designed for last-mile navigation, which assumes that the target object seen the initial robot position. However, we can extend the LeLaN to long- distance navigation by leveraging topological memory [65, 55, 66, 57]. We select the node image including the target object by scoring each node with the OWL-v2 [63] and navigate the robot toward its node image location. Then we switch the policy to the LeLaN to move toward the exact target object location. The details of our implementation are shown in Appendix J. We visualize the robot behaviors in the supplemental materials.
Dynamic Object:
We evaluate our method and the baselines on object navigation with six dynamic objects. Here "dynamic" entails moving the object after the robot starts navigating. Due to its robust performance and fast inference speed, our method significantly outperforms the baselines as shown in Table 2. The relatively small network size allows us to implement our policy at a higher frame rate onboard (4 times faster than the fastest baseline, CoW), which is a significant advantage over the baselines that attempt to run larger models on edge compute.
4.3 Cross Embodiment Analysis
Given that our proposed method leverages in-the-wild videos recorded by a variety of cameras at different poses for training, it is inherently capable of generalizing to different embodiments. To rigorously evaluate our policies' cross-embodiment capabilities for Q3, we test our policy on the four different robot setups as shown in Fig. 4, [a] a Unitree Gol quadruped robot with different fisheye camera, [b] the same wheeled mobile robot as used in our extensive quantitative evaluation but with a different fisheye camera, [c] the same wheeled mobile robot with a narrow FOV camera, and [d] the same wheeled mobile robot with a spherical camera at higher height. We find that our policy can robustly navigate all of these robots toward target objects, demonstrating its ability to transfer across novel morphologies and cameras. We show each examples of the robot performance in the supplemental materials.
7

---

Yellow sign
Black sheff trash bins
Dark green couch
whiteboard
enbodmennt cuadruped robot
[b] Diffarenn camera: fishaye
ent comera: conectal
[d] height
Figure 4: Overview of cross embodiment evaluation. We conduct three type experiments evaluate the gen- eralized performance of our policy. [a] quadruped robot with PCB-mounted fisheye camera, [b] PCB-mounted fisheye camera, [c] canonical camera, and [d] spherical camera at higher pose
4.4 Data Ablations
For Q4, we conduct a dataset ablation study for our YouTube Tour Dataset, our Human-walking Dataset and the Indoor Navigation Dataset. We report the mean square pose error between the target object and the gen- erated trajectories on the test dataset in Fig. 5. We start with base mixture of 100% of the Indoor Navigation Dataset, 20% of the YouTube Tour Dataset, and 43% of the Sluman-walking Dataset. We train the policy with dif- ferent ablations of this mixture to analyze the impact of increased data volume from each dataset.
Mean square pose error
2.4
Indoor Nav. Dataset
on best detaset
2.2
YouTube Tour Dataset
walking
2.0
1.8
1.6
Number of image each training dataset [*]
Figure 5: Data Ablation. An ablation
of each dataset included in training data mixture, while keeping the entirety of the other datasets the data mixture.
By including more data sources our augmented dataset, the performance of the trained innguag--conditione nav- igation policy improves. However, performance improvements saturate with the Indoor Navigation Dataset and the Human- Dataset around 100K frames. The YouTube Tour Dataset signifi- cantly improves the performance beyond 100K frames and enables us to train the best policy with over 500K frames. The broad distribution of environments and objects captured in the YouTube data is the key reason that the policy's performance continues to improve. Table 2 shows an addi- tional data ablation on navigation in the real world using our prototype robot. When increasing the proportion of these in-the-wild datasets, our method improves all test cases.
5 Conclusion
We present LeLaN, novel approach that consumes unlabeled, action-free egocentric data and lever- ages foundation models to learn scalable, language-conditioned object navigation. Our work aims to take advantage of the snternet-scale knowledge of foundation models such as monocular depth estimation models, VLMs, and LLMS to add language and action annotations to actionless egocen- tric mobile robot and human navigation data. We release a dataset of over 100 hours of in-the-wild video data, which was automatically augmented using LeLaN. Our policies can effectively seale diverse data sources due to the generality of LeLaN, We show the importance of training our policies with in-domain robotics data. However, also show our policies improve by including out-of-distributie from sources such FPV walking video footage and YouTube house tours. Our policies outperform the SoTA baselines regarding precise line-of-sight object navigation with lan- guage commands of varying quality and granularity.
While our empirical results show that our policy is robust to varying environments, objects, instruc- tions, camera types and embodiments, there still number of limitations that be addressed in future work. For instance, same object using spatial reasoning alone. If there were two doors in the scene, and the instruction asked for the robot to "go to the leftmost door," the policy often gets confused among the doors. Additionally, collision avoidance remains a challenging task when training navigation policies. Learning to accurately distinguish target objects and understanding 3D geome- try for planning will continue be explored future work.
policy has trouble distinguishing between multiple copies of the
8

---

Acknowledgments
This research was supported by Berkeley Al Research the University of California, Berkeley and Toyota Motor North America. And, this work partially supported by ARL DCIST CRA W911NF-1172-0133 NSF IIS IIS-2246811, I, and NSF IIS-2 We thank Dhruv Shah for advising the implementation for the long-distance navigation using the topological memory and the objective design for collision avoidance and Pranav Atreya for discussing the LLMs and VLMs filtering to have accurate labeling.
References
[1] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXi:22403.12945, 2024.
[2] H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, Hansen-Estruch, Q. Vuong, A. He, V. Myers, K. Fang, C. Finn, and S. Levine. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023.
[3] H.-S. Fang, H. Fang, Z. Tang, Liu, C. Wang, Wang, H. Zhu, and C. Lu. Rh20c: A comprehensive robotic dataset for learning diverse skills in one-shot, 2023.
[4] H. Kim, Y. Ohmura, and Y. Kuniyoshi. Multi-task robot data for dual-arm fine manipulation. arXiv preprint arXiv:2401.07603, 2024.
[5] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, Askell, Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[6] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding
by generative pre-training. 2018.
[7] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
[8] M. Ahn, Brohan, N. Brown, Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakr- ishnan, K. Hausman, A. Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, L. Luu, C. Parada, P. Pastor, J. Quiambao. K. Rao, J. Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, Xu, S. Xu, M. Yan, and A. Zeng. Do as can and not as say: Grounding language in robotic affordances. In arXiv preprint arXiv:2204.0169, 2022.
[9] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. CoRR, abs/2201.072077 2022. URL
abs///bb///// / 222122200000 07207.
[10] V. Dorbala, G. Sigurdsson, Piramuthu, Thomason, and G. S. Sukhatme. Clip-nav: Using clip for zero-shot vision-and- -annaauad navigation, 2022.
[11] D. Shah, B. Osinski, B. Ichter, and S. Levine. Lm-nav: Robotic navigation with large pre- trained models of language, vision, and action, 2022.
[12] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.

---

[13] C. Huang, O. Mees, A. Zeng, and W. Burgard. Audio visual language maps for robot naviga- tion. Proceedings of the International Symposium on Experimental Robotics (ISER), Chiang Mai, Thailand, 2023.
[14]
Liang, W. Huang, Xia, Xu, K. Hausman, B. Ichter, Florence, and A. Zeng. Code as policies: Language model programs for embodied control. In arXiv preprint arXiv:2209.07753 2022.
[15] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
[16] H. Ha, P. Florence, and Song. Scaling up and distilling down: Language-guider robot skill acquisition. In Proceedings of the 2023 Conference Robot Learning, 2023.
[17] O. Mees, J. Borja-Diaz, and W. Burgard. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Au- tomation (ICRA). London, UK, 2023.
[18] S. Nasiriany, F. Xia, W. Yu, T. Xiao, J. Liang, I. Dasgupta, A. Xie, D. Driess, A. Wahid, Z. Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. arXiv preprint arXiv:2402.07877 2024.
[19] D. Driess, F. Xia, M. S. M. Sajjadi, Lynch, A. Chowdhery, B. Ichter, Wahid, J. Tompson, Q. Vuong, Yu, W. Huang. Y. Chebotar, P. Sermanet, D. Duckworth, Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.003378 2023.
[20] A. Sridhar, D. Shah, C. Glossop, and S. Levine. NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration. arXiv pre-print, 2023.
2310. xxXX.
[21] N. Hirose and K. Tahara. Depth360: Self-supervised learning for monocular depth estima- tion using learnable camera distortion model. In 2022 IEEE/SS International Conference on Intelligent Robots and Systems (IROS), pages 317-324, IEEE, 2022.
[22] W. Yin, Zhang, H. Chen, Cai, G. Yu, K. Wang, Chen, and C. Shen. Metric3d: Towards zero-shot metric prediction from single image. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9043-9053, 2023.
[23] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning, 2022.
[24] S. Stepputtis, J. Campbell, M. Phiclipp. S. Lee, C. Baral, and H. B. Amor. Language- conditioned imitation learning for robot manipulation tasks, 2020.
[25] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: from natural lan- guage instructions to feasible plans, Autonomous Robots. 4778):1455-1365, Nov. 2023. ISSN 1573-7527. doi: 1007/110514-0231110000 s10514- 00:514--23-10 1e0e3-11333 10131-7.
URL http:::://.......o//////0000
[26] W. Liu, C. Paxton, Hermans, and D. Fox. Structformer: Learning spatial structure for language-guided semantic rearrangement of novel objects. CoRR, abs/2110. 10189, 2021. URI. https: //ar:///rxivvia wweexiv..org/aaae /t1.://../......11..............
[27] O. Mees, L. Hermann, and W. Burgard. What matters in language conditioned robotic imitation learning unstructured data. IEEE Roboties and Automation Letters (RA-L), 7(4): 1205- 11212. 2022.
10

---

[28] H. Mei, M. Bansal, and M. R. Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. CoRR, (0s//150600088, 2015. URI.htp::/ /ttpttppp///xxi...// abs/ 1106. 04089.
[29] L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Qin, B. Wang, H. Xu, and X. Wang. Gensim: Generating robotic simulation tasks via large language models, 2024.
[30] O. Mees, L. Hermann, E. Rosete-Bettt Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters (RA-L), 7()):7527-73344 2022.
[31] A. Kadian, J. Truong, Gokaslan, A. Clegg, E. Wijmans, S. Lee, M. Savva, Chernova, and D. Batra. Are we making real progress simulated environments? measuring the sim2real gap in embodied visual navigation. CoRR, abs/1912.06321, 2019. URL http: //arivv///xv.. org/ abs//bb/// / 1912.06321.
[32] T. Gervet, S. Chintala, D. Batra, J. Malik, and D. S. Chaplot. Navigating to objects the real world, 2022.
[33] B. Chen, F. Xia, Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open- Open-vocabulary queryable scene representations for real world planning. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 11509-11522. IEEE, 2023.
[34] S. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Base- lines and benchmarks for language-driven zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 23171- 23181, June 2023.
[35] A. Khandelwal, L. Weihs, R. Mottaghi, and A. Kembhavi. Simple but effective: Clip embed- dings for embodied ai, 2022.
[36] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects, 2020.
[37] D. Shah, M. Equi, B. Osinski, F. Xia, B. Ichter, and S. Levine. Navigation with large language models: Semantic guesswork as heuristic for planning. In 7th Annual Conference on Robot Learning, 2023. URL ht httptt:::/pppppeepeeeeee net/ forum htts:/ooummmddp//// wid.ssd65000ttppppp
[38] Wasserman, K. Yadav, Chowdhary, A. Gupta, and U. Jain. Last-mile embodied visual navigation, 2022. URL https://arxiv.. org/ /abb/222111 11746.
[39] D. Shah, B. Eysenbach, N. Rhinchart, and S. Levine. Rapid Exploration for Open-Wort World Navigation with Latent Goal Models. In 5th Annual Conference on Robot Learning, 2021. URL https::///// /oppnreveeeeeeeeeeoeeeeeeeeooeee net wtt.f/frrmmiiiee 2ow. SWWJhyKVV.
[40] B. Kuipers and Y.-T. Byun. robot exploration and mapping strategy based on a semantic hierarchy of spatial representations. Robotics Auton. Syst., 8:47-63, 1991. URL https: //api. wer.iisemmtticccchceec
org/ argg/ooppuIDD:213333
[41] F. Dellaert, D. Fox, W. Burgard, and S. Thrun. Monte carlo localization for mobile robots. In Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No.99CH36288C), volume 2, pages 1322-1328 vol.2, 1999. 100:10.1100/RRRR 100.00000//00001999 772544.
[42] S. Karamcheti, S. Nair, A. S. Chen, T. Kollar, C. Finn, D. Sadigh, and P. Liang. Language- driven representation learning for robotics, 2023. URL https: /arxivv.../. org/ abs/ 2302. 12766.

---

[43] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: A universal visual represen- tation for robot manipulation, 2022. URL https: /arxivivv.v////////////////aaavv org/ abs /203. 12601.
[44] C. Bhateja, D. Guo, D. Ghosh, A. Singh, M. Tomar, Q. Vuong, Y. Chebotar, S. Levine, and
A. Kumar. Robotic offline from internet videos via value-function pre-training, 2023. URL attps://arziin org/ abs//309. 13041.
[45] Y. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training, 2023. URL https: www.i..og/aaaa..aaoo.ooooaaaa.aa 2210.2211... 00030.
[46] A. S. Chen, S. Nair, and C. Finn. Learning generalizable robotic reward functions from "in- the-wild" human videos, 2021. URL https: aaa/ariivvv.///.// wwwpaaawwwww.aa/a/a/o//aaaaaaaaa / 22... 16177.
[47] S. Bahl, R. Mendonca, L. Chen, U. Jain, and D. Pathak. Affordances from human videos as versatile representation for robotics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11778-13790 2023.
[48] A. Patel, A. Wang, Radosavovic, and J. Malik. Learning imitate object interactions from internet videos. arXiv:2211. 13225, 2022.
[49] A. Brohan, N. Brown, Carbajal, Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman,
A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, Xu, S. Xu, T. Yu, and Zitkovich. Rt-2: Vision-language-action models transfer web knowledge robotic control, 2023. URL https: /ate:///araa /at/ivixviv./////////a/////aaava /tt///aawi...gaa////////00/ 15818.
[50] N. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher. Vifm: Vision-language frontier maps for zero-shot semantic navigation, 2023. URL https:///axxiiiae wbsp/////bbb//.//// 03275.
[51] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects, 2020. URL wwtth::p////aat org/ abs /2....006...... 13171.
[52] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitthead, the IEEE/CVF International
A. C. Berg, W.-Y. Lo, et Segment anything. In Proceedings the Conference on Computer Vision, pages 4015-4026, 2023.
[53] W. Wang. Q. Lv, W. Yu, W. Hong, J. Qi. Y. Wang, Ji. Z. Yang, L. Zhao. Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arX0v:2311.03079, 2023.
[54] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint 3xX1v:2303.08774,20 2023.
[55] D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose, and S. Levine. ViNT: A foundation model for visual navigation. 7th Annual Conference on Robot Learning, 2023.
URL httph:tt///// /ttp///axvvv.v..///a/ org/ abcp//bb/222 22066 14846.
[56] N. Hirose, A. Sadeghian, Xia, Martin-Martin, and S. Savarese. Vunet: Dynamic scene view synthesis for traversability estimation using an rgb camera. IEEE Robotics and Automa- tion Letters, 4(2):2062-2069, 2019.
12

---

[57] N. Hirose, F. Xia, R. Martín-Martin, A. Sadeghian, and S. Savarese. Deep visual mpc-policy learning for navigation. IEEE Robotics and Automation Letters, 4(4):3184-319), 2019.
[58] N. Hirose, D. Shah, A. Sridhar, and S. Levine. Sacson: Scalable autonomous control for social navigation. IEEE Robotics and Automation Letters, 2023.
[59] youtube.
com/ Accessed: 2024-06-06.
[60] C. Chi, 7. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Proceedings of Robotics: Science and Systems (RSS), 2024.
[61] Y. Kuang, H. Lin, and M. Jiang. Openfmnav: Towards open-set zero-shot object navigation via vision-language foundation models. arXiv preprint arXiv:2402. 106700 2024.
[62] M. Minderer, A. Gritsenko, Stone, M. Neumann, D. Weissenborn, Dosovitskiy, Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, al. Simple open-vocabulary object detection. In European Conference Computer Vision, pages 728-755. Springer, 2022.
[63] M. Minderer, A. Gritsenko, and N. Houlsby. Scaling open-vocabulary object detection. In A. Oh, T. Naumann, A. Globerson, Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 72983-73007. Curran Associates, Inc., 2023. URL https::/pproceedngg..nuuueee neurips ca/ http//aper__ae htappper__ieeeeappppeee
2023/ ( Confereee pdf.
[64] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288, 2023.
[65] N. Savinov, A. Dosovitskiy, and Koltun. Semi- cemi-parametric topological memory for naviga- tion. arXiv preprint urXiv:/803.00653 2018
[66] N. Hirose, D. Shah, A. Sridhar, and S. Levine. Exaug: Robot-conditioned navigation policies via geometric experience augmentation, 2022.
Zero-shot [67] S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller. Zenossot transfer by com- bining relative and metric depth, 2023. URL https: http::/t http:::://axiiivi org/ abss /200./23222222 12288.
[68] E. Perez, F. Strub, De Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI conference on artificial intelligence, 32 (1), 2018.
13

---

Appendix
A Collision Avoidance with Supervision from Robotic Foundation Model
In our target task, last-mile navigation, we assume that there are no obstacles between the robot location and the target objects. However, we may collide with the objects that are adjacent to the initial robot location as our policy is trained to move toward the target object as directly as possible. To address this issue, we introduce the following additional objective to distill a robotic foundation model with collision avoidance capabilities into our policy.
M-1
(col
(plk) -p)²
k=0
where [0(4]]-=o...M-11 is the generated trajectory from the robotic foundation model such as [=[k]]==o...M = fNoma((o))i=-L...0 0g). Here, M and L indicate the control horizon of the foundation model and the length of the image history, respectively. Specifically, we employ the No- MaD [20] as the robotic foundation model, [N.MaD() However, as we do not assume access to the goal image Og of the target object the context of the language conditioned object navigation task, we crop the target object from the current observation on and feed in the cropped image as Og to generate {=] in training. To generate reasonable trajectories with the cropped images, we also fine-tune the original NoMaD policy before employing it for this loss, as explained later.
To apply Jeal into our original objective J and properly train our policy, we use the following tech- niques (Note that following techniques are only for considering Jeol in J.)
1. Pre-training the policy without Jeal and then fine-tuning this policy with Jeols
2. Masking out Jost in the case that the target object is close to the robot, and
3. Feeding in the history of observations }
The first technique allows us to learn two competing capabilities, namely collision avoidance and goal-reaching. By first learning a goal reaching behavior, fine-tuning for an obstacle avoidance be- havior is possible without compromising goal-reaching performance. For the second technique, we mask out Jeot when the target object is close to the robot location. As the robot approaches the object, the NoMaD policy will more strongly push the robot to avoid the object to avoid collision, sometimes preventing the object from being reached. The third technique, including an image his- tory. improves the temporal consistency of the policy and performance in real experiments. We therefore follow the implementation in ViNT [55] and modify the network structure.
Fine-tuning NoMaD policy for the cropped image: To fine-tune NoMaD policy for image crops, we provide a crop from the current image as the goal for fifty percent of the batch and otherwise provide the original goal image. The dataset used to train NoMaD includes the 2D pose of the robot the goal. As result, we can randomly select a height and back-project this 3D robot frame point into an image coordinate as [u, v),, using this that corresponds to the target pose. We crop randomly sized boundary box centered on this point from the current observation to use as the goal image. The rest of the training procedure remains the same. Please see the details in their original paper [20].
reference point for determining image crop
B Breakdown of quantitative experiments
We provide a break down of our quantitative experiments to analyze our results with more gran- ularity. We categorize all environments into "Office floor", "Entrance hall", "Home environment" and "Outside" and show the mean of the target object arrival rate for each categories in Table 4. Our method trained on the full dataset outperforms all baselines in almost all categories of environ- ments. The results clearly show the advantage of our data augmentation approach, which enables
14

---

Table Quantitative results breakdown. categorize all experiments into four categories, "Office floor", "Entrance hall". "Home environment" and "Outside" and show the goal success rate. Success is determined by the robot reaching within a 0.2 [m] radius of the target object. YT and HW indicate the YouTubs Tour Dataset and waaking Dataset, respectively. f indicates incorporate the classical motion planner.
Method
Total
Simple prompts
Noisy prompis
Multiple objects
OpenFMINav [61]t
0.23
0.25
0.22
0.17
Office floor
OWL-VIT [62] VINT [55]
0.35
0.38
0,34
0.17
CoW [34]t
0.56
0.69
0.47
0.33
OWT [63] + Znedepth [67]]
0.65
0.63
0.66
0.50
Our method (YT 0.0% IIW 0.0%)
0.77
0.81
0.75
0.67
(YT 20.0%. HW 43,5%)
0.90
1.00
0.84
0.83
(YT 100.0%. HW 100.0%)
0.96
1.00
0.94
1.00
OpenPMNNav [61]f
0.42
0.50
0,38
0.33
Entrance hall
OWT.-VT [62] + VINT [55]
0.35
0.40
0.31
0.17
CoW [34]]
0.50
0.70
OWL-v2 [63] Zoedspth [67]
0.57
0.70
0.50
0.83
Our method (YT
0.0%. HW 0.0%)
0.50
0.40
0.56
0.33
(YT 20.0%, HW 43.5%
0.85
0.90
0.81
0.83
(YT 100.0%, HI 100.0%)
0.85
0,90
0.81
0.83
OpenFMNNav [61]F
0.60
0.58
0.61
0.0
OWL-VIT [62] + VINT [55]
0.23
0.42
0.11
0.10
Home CNV.
CoW [34]t
0.60
0.67
0.56
0.40
OWL-v2 [63] + Znedepth [67]]
0.77
0.83
0.72
0.80
Our method (YT 0.0%, IIW 0.0%)
0,53
0.42
0.61
0.70
(YT 20.0%. HW 43.5%
0.83
0.83
0.83
0.70
(YT 100.0%. HW 100.0%)
0.87
0.83
OpenFMNav [61]]
0.52
0.67
0.43
0.70
OWL-VIT [62] VINT [55]
0.59
0.56
0.61
0.30
Outiide
CoW [34]t
0.65
0.78
0.57
0.40
OWL-v2 [63] + Zoedepth [67]f
0.67
0.78
0.61
0.60
Our method (YT
0.0%. HW 0.0%)
0.65
0.72
0.61
0.80
(YT 20.0%, HW 43.5%)
0.80
0.83
0.79
0.40
(YT 100.0% 100.0%)
0.87
leverage the in-the-wild videos to learn generalized policy. It is evident that even with just the Indoor Navigation dataset, which does not include at-home and outdoor data, our method still exhibits clear performance gap between the baselines. However, the gap narrows between our method and the baseline OWL-v2 + Zoedepth because it is strong in home environments. The base- line methods are relatively weak in the office floor and entrance hall environments, which are more semantically sparse compared to the home environment. This weakness leads to large gap in the mean score between our method and the basclines across all evaluations.
C Model Ablations
We also study how ablations of our model architecture impact the performance of our policy while training on the LeLaN dataset. For the visual encoder, we replace "ResNet-FiLM" in our method with "ViT-B32" and "ViT-ResNet50" of the pre-trained CLIP. For the text encoder, we employ a larger pre-trrined CLIP text encoder "ViT-ResNet50" instead of "ViT-B32".
In this ablation study, evaluate each model on the test dataset. Similar the objective training, calculate the mean square error between the generated virtual robot pose from the control pol- icy and the target object pose. From Table 5, the pre-trained visual encoders from CLIP are worse than the ResNet-FiLM trained on our augmented dataset. The visual features from the CLIP visual encoder are insufficient to derive time-series velocity commands because they do not include geo- metric information. Furthermore, the ResNet-FiLM inserts the text features from the text encoder for low-level visual features, which helps to understand the target objects in the image view. In addition, the larger CLIP text encoder helps with learning precise control policy. However, the advantage of larger encoder is not significant the test dataset. Furthermore, when navigating with real robot, its difference was trivial and, in fact, increased the computational load on the robot
15

---

controller. This not only reduced the frame rate, but also increased battery consumption. Therefore, we used the the pre-trained text encoder from the "ViT-B32" CLIPP model for the main model the paper.
Table 5: Ablation study of our model architecture. We use the pre-trained weights of ViT-B32 and ViT- ResNet50 from CLIP for both the visual and text encoders. When using ResNct-FiL.M, train our model from scratch.
Visual encoder ReLM ViT-B32 ViT-ResNeL50
Text encoder
MSE
ViT-B32 ViT-ResNeL50
1.291
1.202
1,690
1.673
D Frequency of the object noun in our augmented dataset
We visualize the frequency of object nouns our augmented dataset in Fig 6. We can confirm that our dataset includes wide variety of nouns allowing us to train a generalized policy, which navigate towards various objects. The YouTube Tour Dataset effectively augments the category distribution with objects from home environments and outdoors, for which the Indoor Navigation Dataset is not sufficient.
Indoor Naviqulio on Cataset
Yuuuube Your Dataset
Hummnn collected Walking Datsset
Nambar
0.5
Figure 6: Top 100 nouns in generated instructions. Blue, green, and orange bars indicate the Indoor Navi- gation Dataset, YouTube Tour Dataset, and Human- Human-walking Dataset, respectively.
E Target Objects and Prompts in Evaluation
In our evaluation, we select 18 objects in the university campus environment (inside and outside) and 10 additional objects in the home environment and prompt the robot to navigate towards the target objects. For each object, feed 5 or 6 trials with different prompts (some of which noisy) and evaluate the robustness of the policy. Here, we show the overview of the target objects and the prompts in our evaluation. The first 18 objects are from the university campus and the rest are from a home environment.
First, two prompts for each object are for the simple prompts and the others are for noisy prompts, which includes wrong adjectives (red) long prompts, or the prompts without the target object's noun. The red border in the image indicates the presence of multiple corresponding objects in the experimental environment. If the objects can be distinguished by prompts, success is considered only the robot reaches the correct object; the objects cannot be distinguished by prompts, success is considered if the robot reaches one of the objects that fits the description of the prompt.
couch
triges
smell rounded purple couch
shiny fridge with disponser anched fridge with weter server shiny appliance with two doors
purple carpet
round plush place fumiture
purple oouch on the grey concrete
Fridge next to wooden sheff
flor. next the blue grey gillow
dark silver something storage food
pirkish purple qushion-ike stool
16

---

office thair
closed door
grey and white onirel thai with amn reats and wheels
greyy brown door with black handle black doer with a sllver bar grey wooden surface with black part woocenn coor the while pancled wall grey panol
metal cheir black arm rests
white and grey structure
slees, orgonomic dee gn
white and gray something to
white and groy chair black sulicase
painting
colorful small painting
sluffed
painting of vegetables
stufeed
brant canues
peinting on the plain white well smrell poster
sitting the
sheep
person
person wearing black pants and
black sh
person wearingg dark color pants and gray shirts
roflective
person stting in the office chair
(lrst name) (lamily namee
whisbborrd green box
ruflective
stairs
dooes
concrete stairs with black hands
white double doors
andiad doore
that stairs with the motal railings
doors next the gerbage
piltar
concrate pillar
short metel stool thin metal and wood place fumitrr
round pilaar
sucporting rectanguler strudtre
pilarr with Tyers and papers on
stool next to the while table somethhnn to
hand sanitzor
garbage bin
white hand soritizer station
blue garbage
santiizer post
grey
hand santtzor next white
garbage with grean
post lamp poet
green
pole
structure
large leaves
lamp post mext the trees and rocks
yel
long grean plants
trianguler sign
yellow yollow sign
pllar
bicyclo
dumpster
black bicycle
grean dumpeter with black
electric tikes
ourved cumpotor
vehicle with two wheels
hax with text Syd-3ort on dumpstor along the concrotio wall
bicycle with the green plants
black
block minivan
modern sieck black minivan
round
Toyata Sionna block car with four whoels
wooden fumilure
wooden many shoes
toilel white tolle
orange toys orange
scuare tallet
orange bin
white something for peeing tollet next the bathtut
orange box,
double
backpack
double
black backpeck
cark color backpack
entry way
sicck squar backpack
block backpack with blue shitty band
display
electric sse movie
growing green
large black TV along while
Figure 7: Overview of target objects and various prompts in our evaluation.
17

---

Network architecture:
Figure shows the overview our network architecture, Rp. We encode the prompt with a frozen CLIP text encoder. Using a visual encoder with ResNet-FiLM [68], we have the visual feature con- ditioned the prompt In addition, similar to [55, 20], take the visual feature of {= with EfficientNNet-O0 and feed these extracted features to of Transformer and fully connected MLP layers to produce the sequence of linear and angular velocities {Ux, wi Je= The ResNet- FiLM encoder allows the model to learn low-level and high-level between the image observation and the text embedding. This helps the model generate the proper velocity commands to move toward the target object pose corresponding to the prompt
Observations (0g12a-2:00
[Vc.
Self Alf-AAttetion
Self- Attentitin
ElfccentNet-BB
Current Olservation
Og
ResNet-FiLM
Mayeay (De, (0g.
guagage prompt
person
L.
CLIP
Figure 8: Architecture. The observations and language prompt are passed into the model. The prompt encoded by the CLIP text encoder and passed to the ResNet-FiLM which also takes in the current observation. TheThe embedding is passed an MLP which produces sequence 24 linear angular velocity pairs which can be integrated into trajectory.
G Implementation details
We show the details of our training and the evaluation setup using a real prototype robot for language-conddtioned navigation.
G.1 Training
To train our control policy, we randomly choose 256 observations from our whole dataset. Since one observation contains multiple objects and each object contains multiple prompts, in almost all cases, we randomly select the object and prompt (which is based on the object). We select = 1 to feed only the previous observation (1 second before in YouTube Tour Dataset) to learn the temporal consistency.
By feeding the observations and the prompts into the model, calculate model and generate sequence of the velocity commands for N(=24) steps. Then, we estimate the virtual robot pose N steps in the future via our kinematic model (integration of the velocity commands in our case). Finally, we calculate the objective J in Eqn. 2 and update our policy т.. We freeze the CLIP text encoder ("CLIP" in Fig. 8) to keep the original performance during training. Our training with an Adam optimizer using learning rate 0.0001 on workstation with Intel i9 CPU, 96GB RAM, and NVIDIA RTX 4090 GPU.
Similar to the other machine learning works with multiple objectives, we required some trial and error to find good weighting between the goal-reaching and collision avoidance terms of our ob- jective. For example, much larger weight for Jeel causes degradation of the goal reaching perfor- mance, which should be learned via Jpuse- Moreover, we mask out Jeal by setting n to less than a certain distance threshold. Since the actions from the RFM try avoid collisions. with target objects, 7) prevents this behavior from dominating close the goal. Although our approach does show explicit ad advantage against the baselines, the target object reaching rate is not
|pal
18

---

sufficient for the practical use. Hence, we still believe that reliable collision avoidance with precise language-condttonned navigation a remaining challenge for our method.
G.2 Robot experiments
Figure 9 shows the overview of a prototype mobile robot in navigation. We calculate the control policy on the edge robot controller, Nvidia Orin AGX, with the best frame rate each method. We mount the ommidirectional camera, RICOH Theta the robot and only use the front-side fisheye camera as the observation in our base evaluation. Since we learn the visual encoder from seratch, there are no restrictions the camera on the robot. In the cross embodiment analysis, we replace a RICOH Theta S with a PCB-mounted fisheye camera and narrow FOV camera (Intel D435i) and additionally change the camera height. Our policy works well across these various camera configurations, but a wide FOV camera generally better for reducing blind spots and making object detection easier.
In the evaluation, provide the language instruction the policy once at the beginning of naviga- tion to reduce the computational load in each step, calculating the text encoding only once with the CLIP encoder. To control the real robot, we repeatedly calculate our control policy at the best frame rate on the robot edge controller and feed the first step velocity command (vo, wp } in the generated sequence the velocity commands (Uk: Wk k==u..., similar to the receding horizon control.
Omnidrractior
Robot controlle (Mvici a Onin ASX)
Mobile base (Robbtt Rooman!
iconie NUF-5K)
Figure 9: Overview of the prototype mobile robot. Note that we only use the front side camera of the omnidirectional camera, Ricoh Theta to navigate the robot. And we replace the Ricoh Theta S with various camera types in the cross embodiment analysis.
H Baseline Methods
In our evaluation, we conduct comparative evaluation with four strong baselines trained on the internet scale datasets. In addition, perform evaluations with policies trained ablated data mixtures, which we discuss in the main paper. Here explain the details of each of the four baseline implementations.
CLIP on Wheels (CoW) [34] We implement the best-performing CoW baseline with the OWL- VIT B/32 detector [62]. Similar to our method, we feed the current observation and the prompts corresponding the target object into the OWL-ViT B/32 detector [62], which was trained an intemet-scale dataset estimate the boundary box for the target object. We crop the estimated point clouds by the estimated boundary box and take the median value as the target object pose. To have fair comparison with our method, which uses single camera image, (no depth camera and LiDAR), we estimate the depth with Depth360 [21] and project it to estimate the point clouds. To control the robot toward the detected object, we use state lattice motion planner to generate the linear and angular velocity commands. The details of the implemented state lattice motion planning are shown below in this appendix. We limit the scope of instructions object navigation for objects within view from the starting point of the robot trajectory. Therefore, we do not implement the exploration portion of CoW [34].
19

---

OWL-v2 [63] Zoedepth [67] In addition to the CoW implementation, we implement version with more performant models, including OWI-v2 as the fine-tuned VLM and Zoedepth [67] for monocular depth estimation to detect the target object pose from the current observation and the prompt, instead of the OWL-VIT and the Depth360 [21] ] CoW [34]. To move toward the detected object locations, we use the same state lattice motion planner as with our CoW implementation the foundation model for vision-based navigation, which can navigate the robot towards a goal posi- tion conditioned on a goal image view. To take a goal image view corresponding a target object, this baseline leverages OWL-ViT, a VLM trained on internet-scale data, and combine it with ViNT [55], a control policy trained on multiple dataset collected by various mobile robots. Specifically, we feed the cropped image from the bounding boxes from OWL-ViT into ViNT the goal image.
to generate linear and angular velocity commands. Due to the larger patch size in OWL-v2, the inference speed is lower than the CoW.
OWL-VVT [62] ViNT [55] To compare our method with learning- learning-based method, we leverage
OpenFMNav [61] We implement another zero-shot method for object navigation that targets bust performance with noisy language instructions. While this method also implements a method to explore semantic frontiers to locate goal objects, we simplify the task to our scope, meaning that we modify the method for last-mile navigation. With this modified version, the language instruction is fed into a VLM, in this case GPT-4o, to generate list of potential objects to look for the scene. Then, the observation from the robot is fed into VLM identify objects in the scene, including objects that may be are semantically similar the target objects. We use the exact same prompts used in the original OpenFMNav implementation. These sets of objects are grouped and fed into Grounded-ST SMM to get potential object segmentations. We use the same depth estimation model, D60th360 [21], and the same state lattice motion planner as in the CoW [34] baseline. We imple- mented two versions of this method, one where inference was performed on each observation and another where the state lattice planner takes over after the object is located. As the first implemen- tation's 's inference speed was extremely slow and less performant then the second option, we use the second our baseline evaluations.
I YouTube Video List
We list all URLs of the YouTube video in our YouTube Tour Dataset below.
https httpas/sswanrryauuuecccooo///aa3
https:
https
https:
https:
20

---

httpf///wwww..ooooooooooo/ooaoaa https:
ottee///www.oouuuoooooo//hhhha13
https///wwww.pooorooooooo///2223
https:
https:
21

---

ottee///www.oouuuuooooo///a22213
ottee///www..ouuuuuecc/////22222
http:///www...oouuuooooo//////22
https:
https:
Long-distance navigation
Before running LeLaN with topological memory for long-distance navigation, we collect the topo- logical memory (a sequence of images) by telcoperating through the environment. To navigate toward the target object corresponding to the prompt, we search all node images with OWL-v2 [63]. By feeding the node image and the prompt into Owl-v2, we are provided a score that corresponds the probability of the object's existence each node image. Then, we select the node ID number with the highest score and navigate the robot toward its node ID with the vision-based navigation system [65, 55, 66, 57]. In implementation, employ ViNT as the vision-based navigation system.
22

---

When arriving at the selected node, we switch the policy from VINT to LeLaN and control our robot toward the target object position. Note that the position of the selected node ID is generally not adjacent to the target object. We utilize ViNT's temporal distance estimation to switch the policy. Specifically, when the closest node the selected node, we switch the policy.
K
State lattice motion planning
We implemented sampling-based motion planning as the local motion planner CLIP on Wheels (CoW), OpenFMNav and OWL-v2 Zoedepth baselines. We generated motion primitives assuming steady linear and angular velocity commands for steps (2.664 s). The pairs of linear and angular velocity commands are (V,, w.) = (0.0, 0.0). (0.2, 0.0), (0.2, 0.3), (0.2, 0.6), (0.2, 0.9), (0.2, - -..3,, (0.2, -0.6), (0.2, -0.9), (0.5, 0.0), (0.5, 0.3), (0.5, 0.6), (0.5, 0.9), (0.5, 0.3), (0.5, -0.6), (0.5, 0.9). We selected these 15 motion primitives by balancing computational load and navigation performance.
By integrating these velocity commands for steps, obtained 15 trajectories such as "tpp//:....s>=== where 'p' is the i-the virtual robot pose the j-th motion primitive. To select the best motion primitive, we calculated the following cost value for each primitive.
J min,( "p?)² Cob
Here, Po indicates the estimated target object pose in each baseline. The first term calculates the squared errors between all 8 poses in the j-th motion primitive and the goal pose to consider the goal reaching. The second term Cos is constant value used to filter out trajectories that collide with static obstacles. We calculate Cob as follows:
Cob {
1000.0
(if d, <rr) (otherwise)
(4)
0.0
where d, is the minimum distance between all 8 poses in the j-th motion primitive and the estimated point clouds corresponding to the static obstacles. Similar to SACSoN [58] and ExAug [66). collision is determined when the distance between the static obstacle and the robot is less than the robot's radius ry. To ensure fair evaluation with other methods that only uses an RGB camera, we utilize estimated point clouds from the current observation of the RGB camera in each method. We choose the motion primitive with the minimum {J/ j=...s and assign the corresponding velocity commands vs and w. to control the robot during navigation.
23


Reviews:
[{"review_id": "B0jlsVwiu3", "reviewer": "Reviewer_DySg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces LeLan, a methodology to leverage large-scale unlabelled egocentric data to improve language-conditioned object navigation. The method works by leveraging large VLMs. The authors demonstrate that the proposed methodology outperforms existing baselines.", "summary_of_recommendation": "The paper has many advantages and disadvantages and questions, but the disadvantages and questions seem to outweigh the advantages (see above).", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- The authors write: \"The relatively small network size allows us to implement our policy at a higher frame rate onboard, which is a significant advantage over the baselines that try to run larger models on edge compute.\" The baselines were not necessarily designed for edge compute -- if the baselines were given the same number of attempts at inference, how would they perform? Is this difference only for dynamic objects?\n- The distribution of the training objects has a long-tail -- what about the testing objects? How diverse are these? Furthermore, was something special done to tackle this challenge, e.g. balanced sampling during training?\n- Where does the supervision for the velocity commands come from?\n- Would this work across robot embodiments? If so, could this be demonstrated? Given the short height of the robot, to looks as though many viewpoints from the datasets (e.g. YouTube videos) may have been from an out-of-distribution viewpoint...\n- What is the field of view of the camera? In the person-chasing demo in the video, seems like the camera needs to have a large vertical field of view to be able to see the orange jacket the person is wearing. On a related note, it looks like a fisheye camera is used on the robot -- how does the visual policy transfer to such a camera?\n- The demonstrations in the dataset don't have to be optimal right? Wouldn't offline RL be better than behavior cloning for something like this? Could this be an additional baseline?\n- Does the policy output a STOP action? If the supervision is coming from ego-centric non-robot videos, then even when the target object is reached, the ground truth velocity may not be zero, correct? In many of the videos, it appears as if the object either crashes into the target object, or the video is clipped before this can happen\n- Is the capacity of the model the same as that of the baselines? Is it possible that the gain in performance is coming a better visual backbone (ResNet-FiLM)?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FsyfiB7eN3", "reviewer": "Reviewer_J4Ez", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "LeLaN is an approach to language-conditioned navigation that leverages large vision-language models (VLMs) to automatically generate instructions and action labels for diverse egocentric video data. This method enables the creation of scalable, language-annotated navigation datasets from various sources, including robot observations, YouTube videos, and human-collected walking data. By utilizing the knowledge of foundation models, LeLaN aims to ground language understanding for embodied navigation. The approach demonstrates good performance in robustness to noisy and diverse language instructions, as well as precision in reaching target objects. Empirical results highlight the effectiveness of incorporating diverse data sources, including out-of-distribution data. The authors argue that this method overcomes limitations of expensive human-annotated datasets and bridges the gap between foundation model knowledge and embodied robotic experiences. The research contributes a dataset generation technique, a navigation policy, and data ablation studies that emphasize the importance of in-domain robotics data.", "summary_of_recommendation": "This paper presents LeLaN, a novel approach to language-conditioned navigation using large vision-language models to generate datasets from diverse video sources. While the method shows promise in handling noisy instructions and improving upon some baselines, there are concerns about its practicality and state-of-the-art claims. The limited scope of the task (navigating to visible objects) and the complexity of the data generation process may not justify the benefits over simpler, zero-shot approaches. The comparison to only two baselines, which may not represent current state-of-the-art, is a notable weakness. Additionally, the lack of information on computational requirements and limited exploration of failure cases should be addressed to understand scalability and robustness. While the paper contributes to the field, addressing these concerns and providing a more comprehensive comparison with recent methods would significantly strengthen the work. The authors should consider revising their claims and expanding their experimental comparisons to better position their contribution within the current state of object navigation research.", "technical_quality": 2, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "This method seems overly-complicated, while the task seems too simple. It is not clear to me that this approach has benefits over other existing approaches that work in the real world (see references below), several of which don’t require generating a dataset or training (i.e., are zero-shot). It seems like this method could easily be replaced with a combination of a metric depth estimator and a referring expression model (e.g., ONE-PEACE, UNI-NEXT, MDETR). These models are lightweight, and can also be run onboard a robot. Once the pose of the object is computed, it is simple to navigate to its coordinates. What benefits does generating the LeLaN dataset and training on it have over an approach like this?\n\nThe paper claims to achieve state-of-the-art ObjectNav performance, but only compares against two baseline methods, one of which is CoW, which is a relatively older method that has since been outperformed by several papers that cite it. And I don’t think ViNT is a state-of-the-art approach for ObjectNav, I believe its focus was more on ImageNav. I understand that you are feeding ViNT image chips from an object detector, but ViNT was designed to take in a full image, rather than just a box of one object. Further, if you are already assuming there is only one target object that is in view, a referring expression model would be more appropriate than an object detector. Could you add experiments to compare against other relevant state-of-the-art approaches?\n\nPlease provide details on the computational resources required for dataset generation. How many GPUs were used, and what was the total time taken? This is important for assessing the scalability and reproducibility of the proposed method.\n\nReferences:\nInstructNav: https://arxiv.org/abs/2406.04882 \nOpenFMNav: https://arxiv.org/abs/2402.10670\nVLFM: https://arxiv.org/abs/2312.03275", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MowAUdYzJF", "reviewer": "Reviewer_W8xr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method, LeLaN, that uses action-free egocentric data to learn language-conditioned object navigation. Specifically, it leverages LLM and vision foundation models to label in-the-wild data for augmentation and trains a policy that can outperform state-of-the-art methods. Experiments validate the efficacy of the proposed method.", "summary_of_recommendation": "The paper targets the problem of leveraging in-the-wild data for a stronger navigation policy. The pipeline and generated data are validated effective via both quantitative benchmarks and hardware experiments. Although it does not tackle more challenging problems such as local planning that can avoid obstacle collision or long-distance navigation, it indeed makes progress in the corresponding niche and shows great improvement compared to previous works. Therefore, I vote for weak acceptance and recommend the author carefully consider the reviews and strengthen the submission during the rebuttal phase.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The authors propose a method for using large-scale, unlabeled data for last-mile robot navigation using VLMs. They compare it to a number of baselines, including a recent state of the art method, and show theirs is superior. They test in several diverse real-world environments. The approach itself seems straightforward and leverages large-scale offline data. The approach is clear, and the paper is solidly written.

The authors did a good job with their rebuttal, in particular making it clear that the baselines were stronger than it seemed, and adding more real-world experiments, as well as clarifying the paper's contributions to last-mile navigation.

Strengths:
- Using video from e.g. YouTube is a very powerful idea and it's good the authors were able to achieve this
- Thorough experiments and ablations
- Useful data for future research
- Good baselines

Weaknesses:
- Unclear how challenging the navigation tasks are - if the robots start very close to target objects it is not very interesting
- Needs more discussion of how much it cost and how well the method scales
- Limitations in data collection; can it handle obstacle avoidance, for example
Decision: Accept