NaVILA: Legged Robot Vision-Language-Action Model for Navigation

Authors: An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Xueyan Zou, Jan Kautz, Erdem Biyik, Hongxu Yin, Sifei Liu, Xiaolong Wang
Primary area: applications to robotics, autonomy, planning
Keywords: Vision Language Action, Legged Robots, Vision Language Navigation

Abstract:
This paper proposes to solve the problem of Vision-and-Language Navigation with legged robots, which not only provides a flexible way for humans to command but also allows the robot to navigate through more challenging and cluttered scenes. However, it is non-trivial to translate human language instructions all the way to low-level leg joint actions. We propose NaVILA, a 2-level framework that unifies a Vision-Language-Action model (VLA) with locomotion skills. Instead of directly predicting low-level actions from VLA, NaVILA first generates mid-level actions with spatial information in the form of language, (e.g., "moving forward 75cm"), which serves as an input for a visual locomotion RL policy for execution. NaVILA substantially improves previous approaches on existing benchmarks. The same advantages are demonstrated in our newly developed benchmarks with IsaacLab, featuring more realistic scenes, low-level controls, and real-world robot experiments.

Decision: reject
Reviewer scores: 3, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1729482990675

thread_type:
official_review

<thread_object>
creator: reviewer_mxea

<content>
summary:
This paper proposes NaViLA, a hierarchical framework for vision-language-based legged robot navigation tasks. The framework consists of a high-level vision-language model (VLM) and a vision-based locomotion policy. The high-level VLM converts historical camera observations and user instructions into commands for the low-level locomotion policy to follow. It takes current and previous observations as input and outputs language-based planning instructions. The VLM is fine-tuned from an existing model using an augmented online dataset. The locomotion policy is trained with reinforcement learning (RL) in a physics-based simulation. The proposed method is compared with various benchmarks, and the results show that it outperforms baseline methods. Additionally, a new benchmark for evaluating vision-language navigation (VLN) in a physics-based simulation is introduced. Finally, the proposed method is successfully implemented on a real-world robot.

strengths:
- The proposed method looks reliable and easy to be reproduced.
- Baselines are compared extensively.
- The real world demonstration look good.
- Paper structure is clear and easy to follow.

weaknesses:
- The motivation for using a legged robot is unclear to me. Since the VLN model is trained separately, the low-level control policy should not impact the high-level model, as long as the low-level policy can achieve the desired goal. 

- The experimental analysis could be improved. For example, in the 'High-fidelity VLN-CE-Isaac Benchmark' section, the robot with vision outperforms the blind policy. But why is this the case? Are there obstacles that need to be avoided? Should the VLN handle obstacle avoidance? Are there any examples that illustrate the different behaviors of these two policies?

- The overall contribution is limited. From my perspective, the main contribution is fine-tuning a VLM that processes vision input and outputs several predefined intermediate actions.

questions:
Please review the weaknesses section. For the overall contribution, I would appreciate a more detailed summary during the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730093399656

thread_type:
official_review

<thread_object>
creator: reviewer_qoup

<content>
summary:
This paper introduces a large vision-language-action model for enabling vision-language navigation for a legged robot. Instead of inferring joint states from image context, authors follow a two step process: (i) predicting the mid-level actions in textual form and (ii) providing these actions to a low-level policy for synthesizing joint action commands. Authors observe the limitations of video-language models (due to lack of large scale data set for the task) and hence take the approach of adapting a VLM to predict actions based on sampled frames from a video. The authors modify prior work VILA to the VLN task by (i) pretraining on generic data sets, (ii) SFT using  VLN data sets and (iii) incorporating auxiliary tasks such as summarization and QA in the associated navigation data. The low-level policy architecture operates on a specific set of skills such as moving forward, turning left etc. and is trained by taking in low-level point cloud maps. Experiments show effectiveness across several benchmarks. Notably performance gain is higher on complex instructions. Finally, the authors also release a new benchmark for the research community.

strengths:
- The paper contributes insights into how to adapt a VLM to predict actions conditioned on present and past visual context. The paper shows how to factor the navigation task as predicting the mid-level actions described in a text form followed by learning a low-level policy. The paper describes the training strategy and SFT data blend and as well as inclusion of auxiliary tasks during training.  

- Authors demonstrate their results on common benchmarks, extensively comparing their results to other approaches. Further, a new benchmark is proposed, specifically suited to the problem setting. 

- Results indicate higher performance gains on complex instructions. Strong results on legged platforms. Indicative results on the humanoid.

weaknesses:
A key motivation for NaVILA is the ability to generalise using the separation between high-level action prediction and the presence of a low-level policy. The experiments demonstrated are impressive in their sequentiality but do not fully highlight the generalization capacity. Specifically, where the low-level plan changes with high-level inputs and visa versa.

questions:
- The authors emphasize the need for including both the current view and a random sample of past frames (a memory) for predicting actions. In this regard, how much memory (number of samples) is needed for the tasks considered. Presumably, the length would depend on the horizon of the underlying navigation task. 

- How does NaVILA respond to action failures. Consider asking a robot to move forward by a distance but the path is either blocked half way or the robot develops a fault. In this case, does the VLA component monitor action execution and replan? More generally, do the low-level actions inform adapation of the high-level plan?

- The core contribution of NaVILA is one of factoring the navigation action generation into predicting mid-level actions in textual form, followed by a low-level join control policy opens doors for behaviour adapation via reasoning. To what extent can the robot perform semantic queries on the world while deciding high-level actions. For example, consider a modification of the action sequence show in top row of figure 1. If one modifies the second phrase to “if you see the stair bar then turn left else go right”, would NaVILA be able to adapt?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730530442839

thread_type:
official_review

<thread_object>
creator: reviewer_lzwz

<content>
summary:
This paper present NaVILA, a framework that integrates vision-language models with locomotion skills for navigation tasks. It uses high-level language commands and a real-time policy for obstacle avoidance. NaVILA maintains reasoning through language actions, avoiding overfitting and allowing broader task training.

strengths:
1. This paper extends NaVid to legged robot scenarios and demonstrates strong performance in real-world applications.
2. The framework effectively integrates 3D scenes into the policy, enhancing robustness, particularly in real scenarios. This is a commendable attempt.

weaknesses:
1. One issue that remains is the inference time, particularly in the demo where the quadruped robot must wait for the large model to complete inference before executing tasks. This waiting period for model inference can lead to instability in motion, which is a point worth considering for resolution.

2. It is unclear whether the model's performance has been tested in an open environment. Currently, the demos presented are tests conducted in a semi-open environment. The authors could consider introducing environments with greater variance from the dataset to test the model's generalizability in real-world scenarios.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534876505

thread_type:
official_review

<thread_object>
creator: reviewer_djqu

<content>
summary:
This paper presents a method to perform navigation tasks with legged robots. To do so, they train a VLM to translate high-level language instructions into low-level navigation commands, such as "move forward 75 cm" or "turn left 30 degrees," based on current and past single-view RGB image observations. These commands are then executed by a legged robot using a locomotion policy that is trained beforehand in a physics simulator through reinforcement learning to follow velocity commands and avoid obstacles. 
The performance of the VLM is evaluated on standard benchmarks for navigation in continuous environments, and a benchmark designed to evaluate navigation and obstacle avoidance is introduced. Finally, the system is deployed on a real Go2 robot.

strengths:
- Real-World Deployment: The authors have successfully implemented their navigation policy on a real legged robot, proving its capability to execute complex navigational tasks in diverse real-world scenarios, including both indoor and outdoor settings.
- Good experimental results in simulated navigation environments: The proposed method outperforms previous approaches for navigation in continuous environments when using only a single RGB camera.

weaknesses:
A significant issue with the paper is that, despite its emphasis on legged locomotion, the navigation skills demonstrated do not inherently require this form of mobility. All the navigation tasks demonstrated could likely be accomplished by wheeled robots without altering the proposed framework. In contrast, prior works on VLA models for legged robots have incorporated capabilities unique to legged robots, such as climbing and crawling under obstacles [1, 2, 3, 4]. The action space of the proposed VLA model is limited to basic navigation commands: “move forward X cm”, “'turn left/right X degrees”, and “'stop”, which abstracts away skills unique to legged robots. The real-world deployment is evaluated in environments where wheeled robots could also navigate effectively. Low-level obstacle avoidance is not exclusive to legged robots. Additionally, it is unclear if the proposed framework for the VLA, which involves supervised fine-tuning from navigation datasets, can handle these kinds of actions only legged robots can perform. The proposed benchmark in simulation, focusing mostly on navigation and collision avoidance, also does not address challenges specific to legged robots.

References:
- [1] Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models
- [2] Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models
- [3] QUAR-VLA: Vision-Language-Action Model for Quadruped Robots
- [4] Helpful DoggyBot: Open-World Object Fetching usingLegged Robots and Vision-Language Models

questions:
Could the proposed approach be extended to navigation tasks that demand more advanced locomotion skills?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732133447945

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response (1/2)

comment:
We thank the reviewers for recognizing the significance of our results (`djqu`, `Lzwz`, `QouP`, `mxeA`), appreciating our efforts toward real-world deployment (`djqu`, `Lzwz`, `mxeA`), and acknowledging the insights provided by our work (`QouP`). Below, we address the reviewers' common feedback, particularly regarding the motivation for using legged robots (`djqu`, `mxeA`) and NaVILA’s generalizability to real-world environments (`Lzwz`).

NaVILA’s strong generalization capabilities are enabled by its modular framework design, which provides exceptional scalability and adaptability to increasingly complex scenes. This design makes NaVILA straightforward to integrate new data sources to enhance performance. For example, we can seamlessly incorporate egocentric human touring videos from YouTube into the VLA training pipeline. These videos were processed into step-wise high-level actions using camera pose estimation, allowing the model to learn effectively from diverse and realistic environments. To validate this, we conduct experiments on the R2R-CE benchmark to measure the performance gains from incorporating YouTube touring videos. As shown in the table below, these experiments demonstrated substantial improvements, with approximately 5% gains in OS, SR, and SPL metrics. This result highlights the scalability and effectiveness of NaVILA’s framework design.

---

| R2R Unseen | NE ↓ | OS ↑ | SR ↑ | SPL ↑ |
|------------|------|-------|-------|--------|
| NaVid | 5.47 | 49.0 | 37.0 | 35.0 |
| NaVILA | 5.37 | 57.6 | 49.7 | 45.5 |
| NaVILA (+ human videos) | **5.22** | **62.5** | **54.0** | **49.0** |

---

Building on this, we conduct more challenging real-world experiments across three environments: Laboratory, House, and Outdoor. These environments included long-horizon tasks traversing narrow passages [[video]](https://navila-iclr.github.io/response/videos/general/indoor_long_horizon.mp4), transitions between indoor and outdoor [[video]](https://navila-iclr.github.io/response/videos/general/indoor_outdoor_transition.mp4), and uneven terrain with small rocks, holes, and troughs in open environments [[video]](https://navila-iclr.github.io/response/videos/general/outdoor_troughs.mp4). In addition to the original experiments, we evaluated 25 instructions, each repeated three times. The quantitative results and accompanying videos below illustrate the complexity of the tasks and the efficacy of our approach. **While NaVILA is compatible with wheeled robots, the motivation for using legged robots lies in their superior ability to handle challenging terrains such as stairs** [[video]](https://navila-iclr.github.io/response/videos/general/outdoor_stairs.mp4), **slopes** [[video]](https://navila-iclr.github.io/response/videos/general/outdoor_hydrant_turn_left.mp4), **and obstacles** [[video]](https://navila-iclr.github.io/response/videos/general/outdoor_parking_lot.mp4)**, where wheeled robots often struggle** [1,2,3]. This advantage makes legged robots a promising platform for advancing vision-language navigation in real-world tasks and has sparked growing industry interest in humanoid and quadruped systems. However, their more complex action space presents greater challenges compared to wheeled robots, making it significantly harder to generalize their behavior to diverse real-world settings. Despite these difficulties, NaVILA achieves strong generalization on legged robots, demonstrating its ability to address these complex scenarios effectively.


| | Lab Simple | Lab Complex | House Simple | House Complex | Outdoor Simple | Outdoor Complex |
|-----------------|------------:|-------------:|---------------:|----------------:|-----------------:|-----------------:|
| GPT-4o | 10/15 | 5/15 | 8/15 | 0/15 | 6/9 | 3/6 |
| NaVILA | 15/15 | 12/15 | 15/15 | 10/15 | 9/9 | 5/6 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes NaVILA, a vision-and-language navigation (VLN) framework that integrates a vision-language-action (VLA) model with locomotion skills for legged robot locomotion. Rather than mapping natural language instructions to low-level actions, NaVILA adopts a two-stage architecture in which the vision-language model reasons over the instructions and the robot's observations to infer primitive (mid-level) text-based actions that then serve as input to a low-level policy that generates joint action commands. For the high-level VLM, NaVILA fine-tunes VILA, an existing image-based VLM, based on available navigation datasets. The low-level locomotion policy is pre-trained in simulation via reinforcement learning. The paper evaluates NaVILA on a series of simulation-based benchmarks and demonstrates its performance on a variety of challenging, real-world legged locomotion tasks.

The paper was reviewed by four referees and received a healthy amount of attention during the rebuttal and the reviewer-AC discussion phases. The paper is highly topical---the use of vision-language models for robot reasoning continues to receive a lot of attention in the robot learning community. All four reviewers agree that a key strength of the paper is NaVILA's successful deployment on real robots, with demonstrations of robots locomoting in a number of challenging scenarios. The authors expanded on these demonstrations with results on humanoid robots during the rebuttal period, which the AC and reviewers appreciated as being no small feat. At least three of the reviewers further emphasized NaVILA's strong performance relative to contemporary baselines on simulation-based benchmarks. Others appreciated the means by which the low-level policy incorporates 3D scene information as well as their perception that the method can be easily deployed in new scenarios.

However, several reviewers raised concerns about the significance of the paper's contributions relative to the current state-of-the-art and the motivation for using legged robots. Some reviewers felt that the paper over-states the novelty of the bi-level architecture and the application to legged robots. There was concern that the contributions relative to NaVid are relatively incremental and while NaVid was applied to wheeled robots, it is not clear that it could not readily be extended to legged robots using a similar low-level policy given the similar nature of the mid-level action space. Additionally, the reviewers questioned the contributions over other works that similarly employ a bi-level architecture that builds a vision-language model on top of low-level policies. That said, it is clear to the AC that the real robot scenarios considered here are more complex than those demonstrated in other papers, however what was not made sufficiently clear is what unique aspects of NaVILA make this possible. In their rebuttal, the authors emphasize NaVILA's modularity as being key to its generalizability, however it is not obvious why a framework like NaVid can not be considered to be similarly modular. The AC acknowledges that that NaVILA application to legged robots (compared with NaVid) and its demonstrations on more challenging locomotion scenarios (compared with existing two-stage legged robot methods), but would like to see a clearer discussion of the significance of this contribution. Related, several reviewers felt that while the paper developes NaVILA in the context of legged robots, it is not apparent in what ways the framework is specific to legged locomotion as opposed to using legged robots as platform for experimental evaluation. Indeed, the authors acknowledge the general-purpose nature of NaVILA and claim that this generality is critical in light of the challenges of legged locomotion, however the reasons for this are not sufficiently clear. Less critical, there was a discussion with the reviewers about the claim that the paper proposes the first vision-based RL policy that employs LiDAR in light of the work of Hoeller et al. [1].

Overall, the AC believes that if the paper were clearer in how it is positioned in the context of contemporary methods that adapt vision-language models for (legged) robotics---whether it is with regard to how NaVILA is specific to legged locomotion or that legged robots serve as the platform for experimental evaluation---it would provide a valuable contribution to the community.


[1] D. Hoeller, N. Rudin, D. Sako, and M. Hutter, "ANYmal Parkour: Learning Agile Navigation for Quadrupedal Robots", arXiv preprint arXiv:2306.14874