LeLaN: Learning A Language-Conditioned Navigation Policy from In-the-Wild Video

Authors: noriaki  hirose
catherine  glossop
ajay  sridhar
oier  mees
sergey  levine
indoor nav dataset
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:51:32

Abstract: We present our method, LeLaN, which uses action-free egocentric data to learn robust language-conditioned object navigation. By leveraging the knowledge of large vision and language models and grounding this knowledge using pre-trained segmentation and depth estimation models, we can label in-the-wild data from a variety of indoor and outdoor environments with diverse instructions that capture a range of objects with varied granularity and noise in their descriptions. Leveraging this method to label over 50 hours of data collected in indoor and outdoor environments, including robot observations, YouTube video tours, and human-collected walking data allows us to train a policy that can outperform state-of-the-art methods on the zero-shot object navigation task in both success rate and precision.

Main paper: 
 introduction: 
 One of the main challenges in robotics is enabling robots to interpret and execute arbitrary userspecified language instructions. For instance, in the case of navigation, a mobile robot should understand that "go to the white and grey chair" and "go to the office chair next to the white desk and black computer monitor" could be asking the same thing in different ways. A major bottleneck of training language-conditioned control policies is access to high-quality and diverse languageand action-annotated robotics datasets, as acquiring human annotations for such datasets is an expensive process. Some large datasets exist for training language-conditioned manipulation policies [1, 2, 3, 4] , but minimal data exists for training language-conditioned navigation policies. Additionally, existing language-annotated robotics datasets are often not large enough to train end-to-end language-conditioned policies that exhibit strong generalization capabilities akin to foundation models in domains such as natural language processing and computer vision. [5, 6, 7] .Past works try to circumvent the lack of language-annotated robotics data by applying foundation models, such as vision language models (VLMs) and large language models (LLMs) in a zeroshot manner. This includes higher-level planning [8, 9, 10, 11, 12, 13] , code-generated policies [14, 15, 16, 17] , and zero-shot control [18] . The issue these methods face is that the data distribution used to train these foundation models is vastly different from the distribution of embodied robot experience. Additionally, these foundation models are often too computationally expensive to run on edge, which is important for mobile robotics. Other methods try to fine-tune these models on a more narrow distribution of in-domain robotics data [19] . However, it is unclear if these model retain their full generalization capabilities after fine-tuning. In this paper, we approach the problem of grounding the abilities of foundation models in embodied experience in the context of languageconditioned policies for visual navigation by using scalable data augmentation.In this paper, we leverage foundation models for vision, language, and robot actions, to annotate egocentric navigation experience with physically grounded language and action labels. Our dataset consists of egocentric navigation data from indoor ground robots, first-person view (FPV) tours on YouTube, and FPV data of a person walking in indoor and outdoor environments. We segment each image in the dataset with a segmentation model to retrieve objects of interest for our language instructions. We use a VLM to generate a description of the objects and an LLM to produce various language instructions for object navigation. We then generate counterfactual action labels that navigate the robot to the object in question using a robot foundation model (RFM) for navigation [20] and 3D estimated point cloud information, which is generated from a monocular depth estimation model [21, 22] . The complete process can be fully automated on any egocentric data and the resulting augmented dataset combines the strengths of spatially-and semantically-aware foundation models. Policies trained on this data demonstrate broad generalization and robust physical grounding for language-conditioned navigation. Our contributions are the following:1. A general data labeling method guided by foundation models that can consume unlabeled videos of action-free data, 2. Empirical results showing that the produced labels enable training a state-of-the-art policy that is more robust for noisy instructions, dynamic target object, and obstacles avoidance, 3. Data-level ablations showing the importance of including in-domain robotics data to ground our trained policies and the effectiveness of our data augmentation, 4. We open source over 120 hours of egocentric data annotated with language and actions, including over 15 hours of human-collected video from 11 cities in 3 countries. 
related works: 
 Language-Conditioned Policies for Robotics: Learning end-to-end language-conditioned policies for robotics has been studied extensively [23, 24, 19, 25, 26, 27, 28] . Learned methods are appealing because of their ability to generalize beyond objects and tasks not seen in the training data. However, they often require language-and action-annotated data, which can be expensive to collect and annotate in the real world. It may be cheaper to automate data collection in simulation [16, 29, 30] , but policies trained in simulation often transfer poorly to unstructured real-world environments, which is commonly referred to as the sim2real gap [31, 32] .Language-Drive Zero-Shot Object Navigation: In this paper, we explore the problem of languagedriven zero-shot object navigation (L-ZSON) [5] , where an agent navigates toward an object specified by a language command. The language input is often unstructured and could refer to an object that the agent has never seen before. Several methods attempt to apply foundation models zero-shot for language-guided vision-based navigation [12, 33] . However, the primary baseline we consider in our experiments is CLIP on Wheels (CoW) [34] , which uses CLIP, a multi-modal encoder that embeds text and images in a shared latent space. CoW uses a CLIP [5] image encoder to compute similarity scores with a CLIP text embedding corresponding to the target object. Unlike many previous ZSON works, which also evaluate search efficiency [35, 34, 36, 37] , we focus on the case where the object is within the line of sight of the robot [38] . This is because our system could be easily Image CropsGo towards the orange fenceblack pants walking on the right side of the hallway.Go to man with the white and black t-shirt.Go to the man.Go towards the black car next to black pants walking on the right side of the hallway.Go to man with the white and black t-shirt.Go to the man.Go towards the man with the black pants walking on the right side of the hallway.Go to man with the white and black t-shirt.Go to the man.x person , y person A orange construction fence for A black car with a yellow license plateA man with the white and black t-shirt A man.A man with the black pants walking on the right side of the hallway A man with the white and black t-shirt A man.https://openai.com/index/chatgpt/ https://general-navigation-models.github.io/nomad/ https://www.youtube.com/ https://arxiv.org/pdf/2311.03079 Figure 2 : Data Annotation. To generate diverse language instructions for object navigation, we pass generic egocentric image observations through a series of large pre-trained model filters that extract object bounding boxes and masks. We then use a VLM to describe the object(s) in the bounding boxes, and use an LLM to produce several diverse object navigation labels.paired with a learned [39, 20, 37] or classical search-based system [40, 41] . We address last-mile precision object navigation, the last portion of the L-ZSON task.Learning from In-the-Wild Videos: One major bottleneck for training policies for robotics is collecting in-domain data. Several works in robot learning have studied the use of in-the-wild video data for representation and reward learning [42, 43, 44, 45, 46] or affordance and direct policy learning [47, 48, 49] . One limitation of using in-the-wild video data is the embodiment gap and lack of action labels. However, the scale and diversity of in-the-wild video data can lead to a level of policy generalization that is hard to achieve with narrow in-domain robot data. 
language-conditioned navigation from in-the-wild videos: 
 In this paper, we study the task of "last-mile navigation" [38] , where the robot must precisely navigate to an object or location that is visible from afar. Complementary to the well-studied task of semantic exploration [37, 50, 51, 10, 34] , last-mile navigation focuses on fine-grained semantic understanding and precise navigation to the target object. We also show that our last-mile navigation method can be combined with a high-level planner and long-term memory to navigate in both partially seen and unseen environments.We generate diverse language instructions and counterfactual robotic actions for action-free egocentric data. We use these instructions and actions to train a robust language-conditioned policy. In this section, we describe the LeLaN data augmentation pipeline and training methodology.A major challenge in training language-conditioned control policies is access to high-quality and diverse robot data with language and action annotations. Moreover, teleoperated robot trajectories for general navigation often lack diversity, traveling only down the center corridors and rarely heading toward objects. Therefore, even if human-made labeling is used on existing data, obtaining rich and useful labels for object navigation remains challenging.We propose a novel method that leverages foundation models to label in-the-wild video data for training a language-conditioned navigation policy. Fig. 2 shows the overview of our data augmentation approach. Our method is (i) scalable, as it uses abundant and readily available action-free data sources, (ii) efficient, as it leverages all captured objects in the image view during data augmentation, and (iii) generalizable, as it distills state-of-the-art large vision and language models to diverse and semantically meaningful annotations.Language annotation: First, the in-the-wild video frames are passed through Segment Anything [52] to localize objects within the scene and produce masks and bounding boxes. The bounding boxes are then used to create image crops, which may include an object of interest. We then feed the image crops into an open-source visual language model (VLM) [53] , which produces a description of the object or objects in the crop. This description is passed to GPT-3.5-turbo, a large language model (LLM). If the LLM determines that the VLM confidently detected an object in the prompt, the LLM is then tasked with generating a prompt "Go to X", where X are various descriptions of the object in the image crop based on the VLM's description.In principle one could directly use a VLM, such as GPT-4V [54] , to generate multiple prompts directly from image crops. However, at the time of the submission, both the cost of the closedsource VLMs and their inference time were high, while the open-source VLMs resulted in poor quality prompts. Therefore, we leverage an open-source VLM [53] to generate image descriptions, and GPT-3.5-turbo, which is fast and inexpensive, to generate prompts from the description.To ground the observation in the real 3D scene, we estimate the 3D pose corresponding to the objects of interest as the goal pose and generate robot actions to move toward it without collision. For the pose estimation, we use an approach similar to CoW [34] where we approximate the depth of the scene using Metric3D [22] for the narrow FOV camera image and Depth360 [21] for the wide FOV camera image and project the result into estimated point clouds. By masking the estimated point clouds with masks from Segment Anything [52] and taking the median values, we can obtain the 3D pose corresponding to an object for which we have matching language instructions from the VLM and LLM filtering.To learn collision-avoidant behavior, we leverage an RFM [20] for navigation to generate the robot actions with this behavior to act as supervision when training our policy. By feeding in a context of observations and the cropped goal image, we can generate the sequence of the robot actions for each object that avoids collision and moves toward the target object, from our pre-trained policy. For more details on how the RFM was used, please refer to Appendix A. Applying our approach to the i-th image o i in the video data, we have m objects with the corresponding estimated poses {p j i } j=1...m and the sequences of the robot actions {p j i [h]} h=0...M -1 for each object. M is the control horizon of the RFM. The number of the objects m is determined by the number of detections generated from Segment Anything with the GPT-3.5-turbo confidence filter. Each object has n prompts as {l j ig } g=1...n , where n depends on the number of generated prompts.We train our language-conditioned navigation policy on our augmented dataset. We define the language-conditioned navigation policy as {v k , ω k } k=0...N -1 = π θ ({o k } k=-L...0 , l). Our policy generates the sequence of the virtual velocity commands {v k , ω k } k=0...N -1 to move toward the target object pose p o by feeding the current and previous observations {o k } k=-L...0 and the prompt l corresponding to the target object. Here, v k and w k are the linear and angular velocity commands respectively. {o k } k=-L...0 and l are taken from our labeled dataset by randomly selecting i as the current state and j for the target object and g for the instruction prompt. Depending on the selected i, j and g, the object pose and robot action are also uniquely given, such as p o and {p[k]} k=0...M -1 .Objective: We define a following overall objective J incorporating three objectives, J pose , J col and J smooth to train π θ encouraging object-reaching, collision avoidance and smooth trajectories:min θ J(θ) := (p o -pN-1 ) 2 Jpose(θ) +ε M -1 k=0 (p[k] -pk ) 2 Jcol(θ) + N -1 k=0 ((v k+1 -v k ) 2 + (ω k+1 -ω k ) 2 ) Jsmooth(θ).where {p k } k=0...N -1 = f robot ({v k , ω k } k=0...N ) is the 2D virtual robot poses from our policy, po is the 2D object pose, which is the projected p o on the horizontal plane. Here, f robot (.) is the differentiable kinematic model. And N (≥ M ) is the control horizon of our policy. J pose is for encouraging target object reaching, J col is for discouraging collisions, and J smooth is for enabling smooth trajectories. Since our pre-trained policy learns to beeline toward the target object location without J col , the robot can not avoid collision with objects between the robot and target objects. Hence, in addition to the base objectives J pose + J smooth , we introduce J col to distill the RFM, which contains an understanding of collision-avoidant behavior learned from diverse navigation datasets [20, 55] . However, since J col will continue to encourage collision avoidance near target objects, we use ε to mask out J col in the case, |p o | < 1.0. Our training process is further described in Appendix F and G.We use a wide variety of egocentric datasets to train on, including: 1) Indoor Navigation Dataset: image observations from mobile robot trajectories in office building environments, 2) YouTube Tour Dataset: YouTube video data of tours in various countries, and 3) Human-walking Dataset: data collected from walking with a camera in an indoor setting and outdoor city environments. A breakdown of this data is included in Table 1 . More analysis of this dataset is shown in Appendix D. Indoor Navigation Dataset:The robot dataset consists of an indoor general navigation dataset [56, 57, 58] which contains autonomous and teleoperated robot trajectories, none of which explicitly contain object-reaching behavior. We employ three publicly available datasets, the GO Stanford 2 (GS2) Dataset [56] , the GS4 Dataset [57] , and the SACSoN (HuRoN) Dataset [58] to cover a variety of office buildings and settings. Note that our data augmentation method is not limited to robot data, but includes it to increase data diversity and provide additional grounding.YouTube Tour Dataset: Incorporating YouTube videos into our dataset allows us to enhance both the diversity and scalability of our approach. We curate 82.5 hours of YouTube videos that include vastly diverse domains and are significantly different from the university campus building settings seen in the Indoor Navigation Dataset. These videos cover a broad spectrum of objects, scenes, and camera heights and types, contributing to more generalized model performance. Specifically, we select videos that include home tours, outside walking (sightseeing) tour videos, and shopping mall tour videos from 32 different countries. Additionally, our dataset captures a variety of weather conditions (e.g. snowy, rainy, and sunny) and time of day (e.g. morning, afternoon, and night), and diverse environmental types, such as urban, suburban, and rural settings.Human-walking Dataset: YouTube video can give us a diverse training set with many kinds of objects in various environments. However, we noticed that most of the cameras used to record YouTube videos have a narrow field of view (FOV). However, a wide FOV camera is useful for robot control [39, 57, 60] . To address this limitation, we collect a dataset of additional data by holding a wide FOV (fisheye) camera and collect a dataset by walking in inside and outside environments. The Human-walking Dataset includes 15.7 hours of data across 11 cities in 3 different countries. 
experiments: 
 Our experiments evaluate LeLaN in the real world, studying the following research questions:Q1. Can LeLaN successfully navigate to novel objects conditioned on simple and noisy or complex language instructions? Q2. Can LeLaN avoid collision with obstacles between the robot and target objects? Q3. Can LeLaN generalize to different embodiments? Q4. How does in-the-wild video improve the learned policy performance?To isolate the advantages from our data labeling and the collision avoidance with the RFM in evaluation, we evaluate our policy with the simplest implementation not considering J col as well as not feeding the history images (L = 0) in the evaluation. We include this objective when studying Q2. We evaluate our trained policy on target object navigation, which tasks the robot with navigating toward a visible target object from the current robot pose. We evaluate our policy as well as four baseline methods for comparison. Please see Appendix F, G and H for details on the procedure and training hyperparameter settings, baseline methods, and a prototype robot in evaluation.We evaluate our method on a diverse set of language instructions. We run evaluations in five different environments, including 3 indoor floors and 2 outdoor spaces, across a set of 28 objects or structures in the scene. For each of these objects, we generate a set of 5-6 language instructions of the form "Go to X". The form of X in these instructions ranges from simple, including only the object with no additional descriptors, to complex, where adjectives and relationships to other objects close to the main object in the scene are included. To evaluate the robustness of our method with a variety of instructions, we perturb the instruction in two ways: 1) by adding incorrect descriptors or similar but incorrect terms for the object and 2) by describing the object implicitly rather than explicitly.Our instructions are provided in the far right of Fig. 3 and Appendix E.Table 2 shows the quantitative analysis from over 1050 evaluations. We categorize each instruction into two categories, simple and noisy prompts, and show the success rate for each category to demonstrate the advantage of our method. Our method achieves an average success rate of 89% compared to 67%, the performance of the strongest baseline. The model is especially robust to noisy prompts, achieving an average of 25% higher success rate on our real-world test set against the baselines. Our method still performs well on simple, correct instructions that can be challenging for the baselines. In addition, we observe that our method outperforms the baselines on prompts with multiple objects, such as "Go to the chair next to the black suitcase". More detailed discussion and analysis are provided in Appendix B and E.Figure 3 shows a visualization of our method deployed in two scenes. In each scene, we navigate the robot towards the same object, but with 5 different prompts shown on the rightmost side. Even though we provide noisy prompts, our method successfully navigates the robot towards the same objects. However, the baseline CoW fails navigation when provided with noisy prompts.To analyze the capability of LeLaN, we conduct three more challenging evaluations: 1) collision avoidance for the obstacles between the initial robot location and the target objects, 2) long-distance navigation, which the robot can not capture the target object at the initial position, and 3) following the dynamic objects such as the pedestrians. Collision Avoidance: For Q2, we evaluate LeLaN in an environment with obstacles to determine the effectiveness of our proposed objective, J col , for distilling collision-avoidant behavior from the RFM. We run our policies with and without J col as well as the strongest baseline, OWL-v2 + Zoedepth. We conduct 15 navigation experiments for different target objects with various obstacles and show the target reaching accuracy in Table 3 .Since collision avoidance is a challenging task when performing language-conditioned navigation, the accuracy of our method with J col is 0.6. However, we can confirm that there is an explicit advantage gap compared to the best baseline method. In most cases, our method without J col collides with the obstacle between the robot and the target object. To add collision-avoidant behavior to the OWL-v2 + Zoedepth baseline, we implement classical motion planning in Appendix K. This baseline often fails navigation due to its slow inference speed. After passing the obstacles without collision, this baseline can not change the trajectory toward the target objects. The robot behaviors are shown in the supplemental video.Long-distance Navigation: The LeLaN is designed for last-mile navigation, which assumes that the target object is seen at the initial robot position. However, we can extend the LeLaN to longdistance navigation by leveraging topological memory [65, 55, 66, 57] . We select the node image including the target object by scoring each node with the OWL-v2 [63] and navigate the robot toward its node image location. Then we switch the policy to the LeLaN to move toward the exact target object location. The details of our implementation are shown in Appendix J. We visualize the robot behaviors in the supplemental materials.Dynamic Object: We evaluate our method and the baselines on object navigation with six dynamic objects. Here "dynamic" entails moving the object after the robot starts navigating. Due to its robust performance and fast inference speed, our method significantly outperforms the baselines as shown in Table 2 . The relatively small network size allows us to implement our policy at a higher frame rate onboard (4 times faster than the fastest baseline, CoW), which is a significant advantage over the baselines that attempt to run larger models on edge compute.Given that our proposed method leverages in-the-wild videos recorded by a variety of cameras at different poses for training, it is inherently capable of generalizing to different embodiments. To rigorously evaluate our policies' cross-embodiment capabilities for Q3, we test our policy on the four different robot setups as shown in Fig. 4 , [a] a Unitree Go1 quadruped robot with a different fisheye camera, [b] the same wheeled mobile robot as used in our extensive quantitative evaluation but with a different fisheye camera, [c] the same wheeled mobile robot with a narrow FOV camera, and [d] the same wheeled mobile robot with a spherical camera at higher height. We find that our policy can robustly navigate all of these robots toward target objects, demonstrating its ability to transfer across novel morphologies and cameras. We show each examples of the robot performance in the supplemental materials. For Q4, we conduct a dataset ablation study for our YouTube Tour Dataset, our Human-walking Dataset and the Indoor Navigation Dataset. We report the mean square pose error between the target object and the generated trajectories on the test dataset in Fig. 5 . We start with a base mixture of 100% of the Indoor Navigation Dataset, 20% of the YouTube Tour Dataset, and 43% of the Human-walking Dataset. We train the policy with different ablations of this mixture to analyze the impact of increased data volume from each dataset.By including more data sources in our augmented dataset, the performance of the trained language-conditioned navigation policy improves. However, performance improvements saturate with the Indoor Navigation Dataset and the Human-walking Dataset around 100K frames. The YouTube Tour Dataset significantly improves the performance beyond 100K frames and enables us to train the best policy with over 500K frames. The broad distribution of environments and objects captured in the YouTube data is the key reason that the policy's performance continues to improve. Table 2 shows an additional data ablation on navigation in the real world using our prototype robot. When increasing the proportion of these in-the-wild datasets, our method improves on all test cases. 
conclusion: 
 We present LeLaN, a novel approach that consumes unlabeled, action-free egocentric data and leverages foundation models to learn scalable, language-conditioned object navigation. Our work aims to take advantage of the internet-scale knowledge of foundation models such as monocular depth estimation models, VLMs, and LLMs to add language and action annotations to actionless egocentric mobile robot and human navigation data. We release a dataset of over 100 hours of in-the-wild video data, which was automatically augmented using LeLaN. Our policies can effectively scale to diverse data sources due to the generality of LeLaN. We show the importance of training our policies with in-domain robotics data. However, we also show our policies improve by including out-of-distribution from sources such as FPV walking video footage and YouTube house tours. Our policies outperform the SoTA baselines regarding precise line-of-sight object navigation with language commands of varying quality and granularity.While our empirical results show that our policy is robust to varying environments, objects, instructions, camera types and embodiments, there are still a number of limitations that can be addressed in future work. For instance, our policy has trouble distinguishing between multiple copies of the same object using spatial reasoning alone. If there were two doors in the scene, and the instruction asked for the robot to "go to the leftmost door," the policy often gets confused among the doors. Additionally, collision avoidance remains a challenging task when training language-conditioned navigation policies. Learning to accurately distinguish target objects and understanding 3D geometry for planning will continue to be explored in future work. 
 
 Appendix: In our target task, last-mile navigation, we assume that there are no obstacles between the robot location and the target objects. However, we may collide with the objects that are adjacent to the initial robot location as our policy is trained to move toward the target object as directly as possible.To address this issue, we introduce the following additional objective to distill a robotic foundation model with collision avoidance capabilities into our policy.EQUATIONwhere {p[k]} k=0...M -1 is the generated trajectory from the robotic foundation model such as{p[k]} k=0...M -1 = f NoMaD ({o i } i=-L...0 , o g ).Here, M and L indicate the control horizon of the foundation model and the length of the image history, respectively. Specifically, we employ the No-MaD [20] as the robotic foundation model, f NoMaD (). However, as we do not assume access to the goal image o g of the target object in the context of the language conditioned object navigation task, we crop the target object from the current observation o 0 and feed in the cropped image as o g to generate {p[k]} k=0...M -1 in training. To generate reasonable trajectories with the cropped images, we also fine-tune the original NoMaD policy before employing it for this loss, as explained later.To apply J col into our original objective J and properly train our policy, we use the following techniques (Note that following techniques are only for considering J col in J.)1. Pre-training the policy without J col and then fine-tuning this policy with J col , 2. Masking out J col in the case that the target object is close to the robot, and 3. Feeding in the history of observations {o i } i=-L...0 .The first technique allows us to learn two competing capabilities, namely collision avoidance and goal-reaching. By first learning a goal reaching behavior, fine-tuning for an obstacle avoidance behavior is possible without compromising goal-reaching performance. For the second technique, we mask out J col when the target object is close to the robot location. As the robot approaches the object, the NoMaD policy will more strongly push the robot to avoid the object to avoid collision, sometimes preventing the object from being reached. The third technique, including an image history, improves the temporal consistency of the policy and performance in real experiments. We therefore follow the implementation in ViNT [55] and modify the network structure.Fine-tuning NoMaD policy for the cropped image: To fine-tune NoMaD policy for image crops, we provide a crop from the current image as the goal for fifty percent of the batch and otherwise provide the original goal image. The dataset used to train NoMaD includes the 2D pose of the robot at the goal. As a result, we can randomly select a height and back-project this 3D robot frame point into an image coordinate as [u, v], using this as our reference point for determining an image crop that corresponds to the target pose. We crop a randomly sized boundary box centered on this point from the current observation to use as the goal image. The rest of the training procedure remains the same. Please see the details in their original paper [20] .We provide a break down of our quantitative experiments to analyze our results with more granularity. We categorize all environments into "Office floor", "Entrance hall", "Home environment" and "Outside" and show the mean of the target object arrival rate for each categories in Table 4 .Our method trained on the full dataset outperforms all baselines in almost all categories of environments. The results clearly show the advantage of our data augmentation approach, which enables us to leverage the in-the-wild videos to learn a generalized policy. It is evident that even with just the Indoor Navigation dataset, which does not include at-home and outdoor data, our method still exhibits a clear performance gap between the baselines. However, the gap narrows between our method and the baseline OWL-v2 + Zoedepth because it is strong in home environments. The baseline methods are relatively weak in the office floor and entrance hall environments, which are more semantically sparse compared to the home environment. This weakness leads to a large gap in the mean score between our method and the baselines across all evaluations.We also study how ablations of our model architecture impact the performance of our policy while training on the LeLaN dataset. For the visual encoder, we replace "ResNet-FiLM" in our method with "ViT-B32" and "ViT-ResNet50" of the pre-trained CLIP. For the text encoder, we employ a larger pre-trained CLIP text encoder "ViT-ResNet50" instead of "ViT-B32".In this ablation study, we evaluate each model on the test dataset. Similar to the objective in training, we calculate the mean square error between the generated virtual robot pose from the control policy and the target object pose. From Table 5 , the pre-trained visual encoders from CLIP are worse than the ResNet-FiLM trained on our augmented dataset. The visual features from the CLIP visual encoder are insufficient to derive time-series velocity commands because they do not include geometric information. Furthermore, the ResNet-FiLM inserts the text features from the text encoder for low-level visual features, which helps to understand the target objects in the image view. In addition, the larger CLIP text encoder helps with learning a precise control policy. However, the advantage of a larger encoder is not significant on the test dataset. Furthermore, when navigating with a real robot, its difference was trivial and, in fact, increased the computational load on the robot controller. This not only reduced the frame rate, but also increased battery consumption. Therefore, we used the the pre-trained text encoder from the "ViT-B32" CLIP model for the main model in the paper.Table 5 : Ablation study of our model architecture. We use the pre-trained weights of ViT-B32 and ViT-ResNet50 from CLIP for both the visual and text encoders. When using ResNet-FiLM, we train our model from scratch.ResNet-FiLM ViT-B32 ViT-ResNet50 ViT-B32 ViT-ResNet50 ✓ ✓ 1.291 ✓ ✓ 1.202 ✓ ✓ 1.690 ✓ ✓ 1.673We visualize the frequency of object nouns in our augmented dataset in Fig 6 . We can confirm that our dataset includes a wide variety of nouns allowing us to train a generalized policy, which can navigate towards various objects. The YouTube Tour Dataset effectively augments the category distribution with objects from home environments and outdoors, for which the Indoor Navigation Dataset is not sufficient. In our evaluation, we select 18 objects in the university campus environment (inside and outside) and 10 additional objects in the home environment and prompt the robot to navigate towards the target objects. For each object, we feed 5 or 6 trials with different prompts (some of which are noisy) and evaluate the robustness of the policy. Here, we show the overview of the target objects and the prompts in our evaluation. The first 18 objects are from the university campus and the rest are from a home environment.First, two prompts for each object are for the simple prompts and the others are for noisy prompts, which includes wrong adjectives (red) long prompts, or the prompts without the target object's noun. The red border in the image indicates the presence of multiple corresponding objects in the experimental environment. If the objects can be distinguished by prompts, success is considered only if the robot reaches the correct object; if the objects cannot be distinguished by prompts, success is considered if the robot reaches one of the objects that fits the description of the prompt. F Network architecture:Figure 8 shows the overview of our network architecture, π θ . We encode the prompt l with a frozen CLIP text encoder. Using a visual encoder with ResNet-FiLM [68] , we have the visual feature conditioned on the prompt l. In addition, similar to [55, 20] , we take the visual feature of {o k } k=-L...0 with EfficientNet-B0 and feed these extracted features to a set of Transformer and fully connected MLP layers to produce the sequence of linear and angular velocities {v k , ω k } k=1...N . The ResNet-FiLM encoder allows the model to learn low-level and high-level between the image observation and the text embedding. This helps the model generate the proper velocity commands to move toward the target object pose corresponding to the prompt l.Language prompt TheThe embedding is then passed to an MLP which produces a sequence of 24 linear angular velocity pairs which can be integrated into a trajectory.We show the details of our training and the evaluation setup using a real prototype robot for language-conditioned navigation.To train our control policy, we randomly choose 256 observations from our whole dataset. Since one observation contains multiple objects and each object contains multiple prompts, in almost all cases, we randomly select the object and prompt (which is based on the object). We select L = 1 to feed only the previous observation (1 second before in YouTube Tour Dataset) to learn the temporal consistency.By feeding the observations and the prompts into the model, we calculate our model and generate a sequence of the velocity commands for N (=24) steps. Then, we estimate the virtual robot pose N steps in the future via our kinematic model (integration of the velocity commands in our case). Finally, we calculate the objective J in Eqn. 2 and update our policy π θ . We freeze the CLIP text encoder ("CLIP" in Fig. 8 ) to keep the original performance during training. Our training is with an Adam optimizer using a learning rate 0.0001 on a workstation with a Intel i9 CPU, 96GB RAM, and an NVIDIA RTX 4090 GPU.Similar to the other machine learning works with multiple objectives, we required some trial and error to find good weighting between the goal-reaching and collision avoidance terms of our objective. For example, a much larger weight for J col causes degradation of the goal reaching performance, which should be learned via J pose . Moreover, we mask out J col by setting η to 0 or 1 |p o | is less than a certain distance threshold. Since the actions from the RFM try to avoid collisions, even with target objects, η prevents this behavior from dominating close to the goal. Although our approach does show an explicit advantage against the baselines, the target object reaching rate is not sufficient for the practical use. Hence, we still believe that reliable collision avoidance with precise language-conditioned navigation is a remaining challenge for our method.Figure 9 shows the overview of a prototype mobile robot in navigation. We calculate the control policy on the edge robot controller, Nvidia Orin AGX, with the best frame rate for each method. We mount the omnidirectional camera, a RICOH Theta S on the robot and only use the front-side fisheye camera as the observation in our base evaluation. Since we learn the visual encoder from scratch, there are no restrictions on the camera on the robot. In the cross embodiment analysis, we replace a RICOH Theta S with a PCB-mounted fisheye camera and a narrow FOV camera (Intel D435i) and additionally change the camera height. Our policy works well across these various camera configurations, but a wide FOV camera is generally for reducing blind spots and making object detection easier.In the evaluation, we provide the language instruction to the policy once at the beginning of navigation to reduce the computational load in each step, calculating the text encoding only once with the CLIP encoder. To control the real robot, we repeatedly calculate our control policy at the best frame rate on the robot edge controller and feed the first step velocity command {v 0 , ω 0 } in the generated sequence of the velocity commands {v k , ω k } k=0...N , similar to the receding horizon control.Figure 9 : Overview of the prototype mobile robot. Note that we only use the front side camera of the omnidirectional camera, Ricoh Theta S, to navigate the robot. And we replace the Ricoh Theta S with various camera types in the cross embodiment analysis.In our evaluation, we conduct a comparative evaluation with four strong baselines trained on the internet scale datasets. In addition, we perform evaluations with policies trained on ablated data mixtures, which we discuss in the main paper. Here we explain the details of each of the four baseline implementations.CLIP on Wheels (CoW) [34] We implement the best-performing CoW baseline with the OWL-ViT B/32 detector [62] . Similar to our method, we feed the current observation and the prompts corresponding to the target object into the OWL-ViT B/32 detector [62] , which was trained an internet-scale dataset to estimate the boundary box for the target object. We crop the estimated point clouds by the estimated boundary box and take the median value as the target object pose. To have a fair comparison with our method, which uses a single camera image, (no depth camera and LiDAR), we estimate the depth with Depth360 [21] and project it to estimate the point clouds. To control the robot toward the detected object, we use a state lattice motion planner to generate the linear and angular velocity commands. The details of the implemented state lattice motion planning are shown below in this appendix. We limit the scope of instructions to object navigation for objects within view from the starting point of the robot trajectory. Therefore, we do not implement the exploration portion of CoW [34] .OWL-v2 [63] + Zoedepth [67] In addition to the CoW implementation, we implement a version with more performant models, including OWL-v2 as the fine-tuned VLM and Zoedepth [67] for monocular depth estimation to detect the target object pose from the current observation and the prompt, instead of the OWL-ViT and the Depth360 [21] in CoW [34] . To move toward the detected object locations, we use the same state lattice motion planner as with our CoW implementation to generate linear and angular velocity commands. Due to the larger patch size in OWL-v2, the inference speed is lower than the CoW.OWL-ViT [62] + ViNT [55] To compare our method with a learning-based method, we leverage the foundation model for vision-based navigation, which can navigate the robot towards a goal position conditioned on a goal image view. To take a goal image view corresponding a target object, this baseline leverages OWL-ViT, a VLM trained on internet-scale data, and combine it with ViNT [55] , a control policy trained on multiple dataset collected by various mobile robots. Specifically, we feed the cropped image from the bounding boxes from OWL-ViT into ViNT as the goal image.OpenFMNav [61] We implement another zero-shot method for object navigation that targets robust performance with noisy language instructions. While this method also implements a method to explore semantic frontiers to locate goal objects, we simplify the task to our scope, meaning that we modify the method for last-mile navigation. With this modified version, the language instruction is fed into a VLM, in this case GPT-4o, to generate a list of potential objects to look for in the scene. Then, the observation from the robot is fed into a VLM to identify objects in the scene, including objects that may be or are semantically similar to the target objects. We use the exact same prompts as used in the original OpenFMNav implementation. These sets of objects are grouped and fed into Grounded-SAM to get potential object segmentations. We use the same depth estimation model, Depth360 [21] , and the same state lattice motion planner as in the CoW [34] baseline. We implemented two versions of this method, one where inference was performed on each observation and another where the state lattice planner takes over after the object is located. As the first implementation's inference speed was extremely slow and less performant then the second option, we use the second in our baseline evaluations.We list all URLs of the YouTube video in our YouTube Tour Dataset below. Before running LeLaN with topological memory for long-distance navigation, we collect the topological memory (a sequence of images) by teleoperating through the environment. To navigate toward the target object corresponding to the prompt, we search all node images with OWL-v2 [63] . By feeding the node image and the prompt into Owl-v2, we are provided a score that corresponds to the probability of the object's existence in each node image. Then, we select the node ID number with the highest score and navigate the robot toward its node ID with the vision-based navigation system [65, 55, 66, 57] . In our implementation, we employ ViNT as the vision-based navigation system.When arriving at the selected node, we switch the policy from ViNT to LeLaN and control our robot toward the target object position. Note that the position of the selected node ID is generally not adjacent to the target object. We utilize ViNT's temporal distance estimation to switch the policy. Specifically, when the closest node is the selected node, we switch the policy.We implemented sampling-based motion planning as the local motion planner in CLIP on Wheels (CoW), OpenFMNav and OWL-v2 + Zoedepth baselines. We generated 15 motion primitives assuming steady linear and angular velocity commands for 8 steps (2.664 s). The pairs of linear and angular velocity commands are (v s , ω s ) = (0.0, 0.0), (0.2, 0.0), (0.2, 0.3), (0.2, 0.6), (0.2, 0.9), (0.2, -0.3), (0.2, -0.6), (0.2, -0.9), (0.5, 0.0), (0.5, 0.3), (0.5, 0.6), (0.5, 0.9), (0.5, -0.3), (0.5, -0.6), (0.5, -0.9). We selected these 15 motion primitives by balancing computational load and navigation performance.By integrating these velocity commands for 8 steps, we obtained 15 trajectories such as {{ s p j i } i=1...8 } j=1...15 , where s p j i is the i-the virtual robot pose on the j-th motion primitive. To select the best motion primitive, we calculated the following cost value for each primitive.EQUATIONHere, po indicates the estimated target object pose in each baseline. The first term calculates the squared errors between all 8 poses in the j-th motion primitive and the goal pose to consider the goal reaching. The second term C ob is a constant value used to filter out trajectories that collide with static obstacles. We calculate C ob as follows:C ob = 1000.0 (if d s < r r ) 0.0 (otherwise) ,where d s is the minimum distance between all 8 poses in the j-th motion primitive and the estimated point clouds corresponding to the static obstacles. Similar to SACSoN [58] and ExAug [66] , a collision is determined when the distance between the static obstacle and the robot is less than the robot's radius r r . To ensure fair evaluation with other methods that only uses an RGB camera, we utilize estimated point clouds from the current observation of the RGB camera in each method. We choose the motion primitive with the minimum {J j s } j=1...15 and assign the corresponding velocity commands v s and ω s to control the robot during navigation.https://learning-language-navigation.github.io/ 8th Conference on Robot Learning (CoRL2024), Munich, Germany. 


Decision: The authors propose a method for using large-scale, unlabeled data for last-mile robot navigation using VLMs. They compare it to a number of baselines, including a recent state of the art method, and show theirs is superior. They test in several diverse real-world environments. The approach itself seems straightforward and leverages large-scale offline data. The approach is clear, and the paper is solidly written.

The authors did a good job with their rebuttal, in particular making it clear that the baselines were stronger than it seemed, and adding more real-world experiments, as well as clarifying the paper's contributions to last-mile navigation.

Strengths:
- Using video from e.g. YouTube is a very powerful idea and it's good the authors were able to achieve this
- Thorough experiments and ablations
- Useful data for future research
- Good baselines

Weaknesses:
- Unclear how challenging the navigation tasks are - if the robots start very close to target objects it is not very interesting
- Needs more discussion of how much it cost and how well the method scales
- Limitations in data collection; can it handle obstacle avoidance, for example

Reviews: review: title: title: Review
paper_summary: paper_summary: This paper introduces LeLan, a methodology to leverage large-scale unlabelled egocentric data to improve language-conditioned object navigation. The method works by leveraging large VLMs. The authors demonstrate that the proposed methodology outperforms existing baselines.
main_review: main_review: Strengths:
- Well-written. The paper is well written and easy to follow.
- Visualizations. The figures very much aid in understanding the paper.
- Experimental Results. The paper compares to and outperforms a number of strong baselines
- A number of ablations are conducted which validate the various design choices made.
- Qualitative evaluation is included

Weaknesses:
- Problem statement. The authors write: "We limit the scope of instructions to object navigation for objects within view from the starting point of the robot trajectory." Even from the visualization, in Figure 5, it looks like the robot starts quite close to the target object. This seems to greatly simplify the problem, while recent works in navigation are tackling semantic exploration real world environments.
- Weak baselines. The authors write: "The relatively small network size allows us to implement our policy at a higher frame rate onboard, which is a significant advantage over the baselines that try to run larger models on edge compute." The baselines were not necessarily designed for edge compute -- if the baselines were given the same number of attempts at inference, how would they perform? Furthermore, is this difference only for dynamic objects?
- Limited real world evaluation. The paper claims to do over 300 evaluations across three different environments -- testing across a larger number of environments would provide more confidence in the results.
- The result of the ablation is not surprising at all: because the test metric used here is performance on the Indoor Navigation Dataset, the more of this exists in the training set, the smaller the error will be. A better ablation may have been perhaps the augmentation technique -- how much does the model suffer on the original test set, if the data augmentation is not done?
questions: questions: - The authors write: "The relatively small network size allows us to implement our policy at a higher frame rate onboard, which is a significant advantage over the baselines that try to run larger models on edge compute." The baselines were not necessarily designed for edge compute -- if the baselines were given the same number of attempts at inference, how would they perform? Is this difference only for dynamic objects?
- The distribution of the training objects has a long-tail -- what about the testing objects? How diverse are these? Furthermore, was something special done to tackle this challenge, e.g. balanced sampling during training?
- Where does the supervision for the velocity commands come from?
- Would this work across robot embodiments? If so, could this be demonstrated? Given the short height of the robot, to looks as though many viewpoints from the datasets (e.g. YouTube videos) may have been from an out-of-distribution viewpoint...
- What is the field of view of the camera? In the person-chasing demo in the video, seems like the camera needs to have a large vertical field of view to be able to see the orange jacket the person is wearing. On a related note, it looks like a fisheye camera is used on the robot -- how does the visual policy transfer to such a camera?
- The demonstrations in the dataset don't have to be optimal right? Wouldn't offline RL be better than behavior cloning for something like this? Could this be an additional baseline?
- Does the policy output a STOP action? If the supervision is coming from ego-centric non-robot videos, then even when the target object is reached, the ground truth velocity may not be zero, correct? In many of the videos, it appears as if the object either crashes into the target object, or the video is clipped before this can happen
- Is the capacity of the model the same as that of the baselines? Is it possible that the gain in performance is coming a better visual backbone (ResNet-FiLM)?
limitations: limitations: 3
review_summary: review_summary: The paper has many advantages and disadvantages and questions, but the disadvantages and questions seem to outweigh the advantages (see above).
title: title: Nice demonstrations, but the simplicity of the task and comparison to existing work could be improved
paper_summary: paper_summary: LeLaN is an approach to language-conditioned navigation that leverages large vision-language models (VLMs) to automatically generate instructions and action labels for diverse egocentric video data. This method enables the creation of scalable, language-annotated navigation datasets from various sources, including robot observations, YouTube videos, and human-collected walking data. By utilizing the knowledge of foundation models, LeLaN aims to ground language understanding for embodied navigation. The approach demonstrates good performance in robustness to noisy and diverse language instructions, as well as precision in reaching target objects. Empirical results highlight the effectiveness of incorporating diverse data sources, including out-of-distribution data. The authors argue that this method overcomes limitations of expensive human-annotated datasets and bridges the gap between foundation model knowledge and embodied robotic experiences. The research contributes a dataset generation technique, a navigation policy, and data ablation studies that emphasize the importance of in-domain robotics data.
main_review: main_review: Quality: Good. Novel approach to language-conditioned navigation. Several experiments and analysis. 

Clarity: Well-structured and clearly written.

Originality: Several recent works have shown how datasets that have been pseudo-labeled using VLMs and LLMs can be used to train capable models. However, the authors apply this approach specifically to the task of ObjectNav, which is not as common.

Significance: Not sure how useful the task is. The approach is mainly for navigating towards objects that are already in front of the robot and in view. The authors recognize this and acknowledge that several existing works go farther; these other works address the case in which the target object is farther away and not in view.

Strengths:
- Use of diverse data sources, including YouTube videos and human-collected data
- Improvements over baseline methods
- Thorough experimentation and ablation studies
- Release of a large, annotated dataset for future research
- Addresses the challenge of expensive human-annotated datasets
- Demonstrates robustness to noisy and complex language instructions

Weaknesses:
- Limited comparison to only two baseline methods. The authors state that LeLaN surpasses methods that are state-of-the-art in ObjectNav (thus attaining a new state-of-the-art in ObjectNav), but I don’t think this is quite the case. There have been several works that have shown open-vocabulary ObjectNav performance in the real world, and unlike LeLaN, these methods can search for the object even when it isn’t in view (see references below). Many of these methods have shown greatly superior performance on the ObjectNav task within benchmarks such as HM3D compared to CoW. Furthermore, these methods are zero-shot, and do not require generating a bespoke dataset for ObjectNav nor any training or fine-tuning at all.
- Lack of discussion on computational requirements and scalability of the approach. What was the cost of using a closed-source LLM like GPT 3.5 to label all of the samples? How many GPUs were used to generate the dataset, and how long did it take?
- The simplicity of the policy’s architecture suggests that it will not fare well for more complex ObjectNav tasks that require temporal reasoning. The LeLaN policy only considers information that is available in the current time step.

References:
InstructNav: https://arxiv.org/abs/2406.04882 
OpenFMNav: https://arxiv.org/abs/2402.10670
VLFM: https://arxiv.org/abs/2312.03275
questions: questions: This method seems overly-complicated, while the task seems too simple. It is not clear to me that this approach has benefits over other existing approaches that work in the real world (see references below), several of which don’t require generating a dataset or training (i.e., are zero-shot). It seems like this method could easily be replaced with a combination of a metric depth estimator and a referring expression model (e.g., ONE-PEACE, UNI-NEXT, MDETR). These models are lightweight, and can also be run onboard a robot. Once the pose of the object is computed, it is simple to navigate to its coordinates. What benefits does generating the LeLaN dataset and training on it have over an approach like this?

The paper claims to achieve state-of-the-art ObjectNav performance, but only compares against two baseline methods, one of which is CoW, which is a relatively older method that has since been outperformed by several papers that cite it. And I don’t think ViNT is a state-of-the-art approach for ObjectNav, I believe its focus was more on ImageNav. I understand that you are feeding ViNT image chips from an object detector, but ViNT was designed to take in a full image, rather than just a box of one object. Further, if you are already assuming there is only one target object that is in view, a referring expression model would be more appropriate than an object detector. Could you add experiments to compare against other relevant state-of-the-art approaches?

Please provide details on the computational resources required for dataset generation. How many GPUs were used, and what was the total time taken? This is important for assessing the scalability and reproducibility of the proposed method.

References:
InstructNav: https://arxiv.org/abs/2406.04882 
OpenFMNav: https://arxiv.org/abs/2402.10670
VLFM: https://arxiv.org/abs/2312.03275
limitations: limitations: 3
review_summary: review_summary: This paper presents LeLaN, a novel approach to language-conditioned navigation using large vision-language models to generate datasets from diverse video sources. While the method shows promise in handling noisy instructions and improving upon some baselines, there are concerns about its practicality and state-of-the-art claims. The limited scope of the task (navigating to visible objects) and the complexity of the data generation process may not justify the benefits over simpler, zero-shot approaches. The comparison to only two baselines, which may not represent current state-of-the-art, is a notable weakness. Additionally, the lack of information on computational requirements and limited exploration of failure cases should be addressed to understand scalability and robustness. While the paper contributes to the field, addressing these concerns and providing a more comprehensive comparison with recent methods would significantly strengthen the work. The authors should consider revising their claims and expanding their experimental comparisons to better position their contribution within the current state of object navigation research.
title: title: Original Review for LeLaN: decent contribution and improvement but can target more challenging key problems
paper_summary: paper_summary: This paper proposes a method, LeLaN, that uses action-free egocentric data to learn language-conditioned object navigation. Specifically, it leverages LLM and vision foundation models to label in-the-wild data for augmentation and trains a policy that can outperform state-of-the-art methods. Experiments validate the efficacy of the proposed method.
main_review: main_review: Strengths:
- The basic idea is easy to follow, and the motivation is clearly presented.
- The overall writing is fine enough to convey the key points and ideas, except for several typos that should be polished afterward.
- The proposed pipeline is scalable for leveraging in-the-wild data, which is basically automatic.
- The experimental results show its effectiveness compared to previous works, with a significant improvement shown in the benchmark.

Weaknesses:
- There are obvious limitations regarding the data generation pipeline. For example, it does not take into account the obstacle avoidance problem in this process. In addition, from the visualization examples, the navigation target is typically in the local region, without touching more challenging long-distance navigation with more complex localization and mapping situations. Therefore, from the perspective of the overall navigation system, it does not tackle key challenges in long-distance planning or (even static) obstacle avoidance in local planning. This is a little disappointing and weakens the contributions significantly. The key problem, from my view, is the way to generate the navigation ground truth path, that can be improved in the future.
- The experiments are a little weak, including only two baselines on one main benchmark and only one ablation about the used data. I find new results in the supplementary material but may still miss some important ablations, such as the performance with different foundation models for the data generation pipeline.
- The overall paper seems to be a rushed one, considering the typos and the overall organization of materials in the main paper and supplemental. There should be a large room to be refined and I would recommend the author carefully make the revision according to the reviews during the rebuttal phase.
questions: questions: See weaknesses.
limitations: limitations: 3
review_summary: review_summary: The paper targets the problem of leveraging in-the-wild data for a stronger navigation policy. The pipeline and generated data are validated effective via both quantitative benchmarks and hardware experiments. Although it does not tackle more challenging problems such as local planning that can avoid obstacle collision or long-distance navigation, it indeed makes progress in the corresponding niche and shows great improvement compared to previous works. Therefore, I vote for weak acceptance and recommend the author carefully consider the reviews and strengthen the submission during the rebuttal phase.