InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment
Authors: Yuxing Long
Wenzhe Cai
Hongcheng Wang
Guanqi Zhan
Hao Dong
Venue: CoRL 2024 (2024)
Keywords: Generic Instruction Navigation
Zero-shot
Unexplored Environment

Abstract: Enabling robots to navigate following diverse language instructions in unexplored environments is an attractive goal for human-robot interaction. However, this goal is challenging because different navigation tasks require different strategies. The scarcity of instruction navigation data hinders training an instruction navigation model with varied strategies. Therefore, previous methods are all constrained to one specific type of navigation instruction. In this work, we propose InstructNav, a generic instruction navigation system. InstructNav makes the first endeavor to handle various instruction navigation tasks without any navigation training or pre-built maps. To reach this goal, we introduce Dynamic Chain-of-Navigation (DCoN) to unify the planning process for different types of navigation instructions. Furthermore, we propose Multi-sourced Value Maps to model key elements in instruction navigation so that linguistic DCoN planning can be converted into robot actionable trajectories. With InstructNav, we complete the R2R-CE task in a zero-shot way for the first time and outperform many task-training methods. Besides, InstructNav also surpasses the previous SOTA method by 10.48% on the zero-shot Habitat ObjNav and by 86.34% on demand-driven navigation DDN. Real robot experiments on diverse indoor scenes further demonstrate our method's robustness in coping with the environment and instruction variations.

InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment
Yuxing Long! 123* Wenzhe Cai", Hongcheng Wang 123 Guanqi Zhan and Hao Dong! 123f CCFCS School of Computer Science, Peking University 2PKU-Agibot Lab Nattooaal Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University *School of Automation, Southeast University University of Oxford
Abstract: Enabling robots to navigate following diverse language instructions unexplored environments is an attractive goal for human-robot interaction. How- ever, this goal is challenging because different navigation tasks require different strategies. The scarcity of instruction navigation data hinders training an instruc- tion navigation model with varied strategies. Therefore, previous methods are all constrained to one specific type of navigation instruction. In this work, we propose InstructNav, generic instruction navigation system. InstructNav makes the first endeavor to handle various instruction navigation tasks without any navigation training or pre-built maps. To reach this goal, introduce Dynamic Chain-of- Navigation (DCoN) unify the planning process for different types of navigation instructions. Furthermore, propose Multi-sourced Value Maps to model key ele- ments in instruction navigation so that linguistic DCoN planning can be converted into robot actionable trajectories. With InstructNav, we complete the R2R-CE task in a zero-shot way for the first time and outperform many task-training methods. Besides, InstructNav also surpasses the previous SOTA method by 10.48% on the zero-shot Habitat ObjNav and by 86.34% on demand navigation DDN. Real robot experiments on diverse indoor scenes further demonstrate our method's robustness in coping with the environment and instruction variations. The project webpage is https: / sites. google. com/ com/ciew//i//c wwwee/inssruuttar
Keywords: Generic Instruction Navigation, Zero-shot, Unexplored Environment
1
Introduction
Instructing the robot navigate in the unexplored indoor scene via natural language is user-friendly, which has drawn considerable interest within the robotic community [1][][[][]][5] A plausible future direction is to develop the robot that can understand and execute a large variety of instructions. This will greatly expand the application scenarios of instruction navigation robots. However, different types instructions emphasize different navigation strategies. For example, object goal navigation approaches [6][7] primarily concentrate on performing efficient exploration to find the target object in unseen environments; visual language navigation methods [8][9] focus on following step-by- step instruction; demand-driven navigation [5] works geared towards conducting demand-based commonsense reasoning. There are significant differences among their navigation strategies, which makes training an instruction navigation model that can follow diverse instructions difficult. The scarcity of instruction navigation data exacerbates this difficulty. Therefore, almost all previous works are limited to executing one type of navigation instructions and cannot adapt to other types. This raises the question- Can we develop a zero-shot system for generic instruction navigation in the unexplored environment?
The first challenge lies in how to unify different types of instructions. To reach this goal, we propose a generic navigation planning paradigm called Dynamic Chain-of-Navigation (DCoN). The
Joint first authors Corresponding Author (hao.dong pku.edu.en)
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

DCoN models critical elements in navigation actions and landmarks, as well as their consequential relationship. It inherently corresponds to the Chain-of- Thought thinking process of the large language model (LLM). This alignment allows the LLM to convert navigation instructions to DCoN without manual annotations. More importantly, DCoN not a static and simple instruction decomposition but generic navigation strategy that updates with the newly explored environment. During the navigation, the next navigation action and landmarks in DCoN will be dynamically updated every Docision-making step with observed objects considering LLM's inner commonsense about room layout and human habits. This way, DCoN can inspire InstructNav to align semantic labels, efficiently explore the unseen environment, and conduct commonsense reasoning about landmarks.
"Find some flowers"
"Find safa."
"Go forward to the chair and right the trash bin."
and furn left find the abitue""
"Wals forward papar box Then, turn right to find potted plant""
"Go straight the office area and help no ceek comething refreshing to drisk"
Twwnnt wash my clathes'
forward turn get
plant the poster"
"Wah forward and left
Object Goal Navigation
Demand-drives Navigation
Visual Language Nav gatton
VLN ObjectMov
VLN + DDN
Figure 1: InstructNav can follow different types of navigation instructions in diverse indoor scenes.
With unified DCoN planning, the ensuing challenge is how to control the robot by this linguistic planning. To address this problem, propose Multi-sourced Value Maps, which represents key elements in instruction navigation, including action, landmark, and history trajectory, on four value maps to decide the next waypoint and actionable trajectories. The Action Value Map and Semantic Value Map are created according to the next DCoN action and landmarks. They can encourage the robot to follow the specified action and move toward navigation landmarks. The Trajectory Value Map is established based on the navigation trajectory to avoid repetitive movement. Although these value maps can deal with simple instructions, there still exist difficulties when navigation instructions require multimodal reasoning. For example, the Semantic Value Map struggles to figure out which table on the map is "the front dining table" while the Action Value Map fails represent "walk between". To improve InstructNav's 's capability on multimodal reasoning, we further design an Intuition Value Map. The multimodal large model [10]'s prediction on the next navigation area is projected on the Intuition Value Map to guide the robot. At each decision step. these value maps are synthesized to plan the next waypoint. Their collaboration form InstrcutNav's generie instruction navigation capability, which single semantic map- methods like [11] [12] cannot realize.
In this work, our primary contribution is InstructNav, the first generic instruction navigation system that can execute different types of instructions in the continuous environment without any navigation training or pre-built maps. To reach this goal, we introduce Dynamic Chain-of- -avi-ggtiiiiiiioonnnnn to unify different types of navigation instructions into a standard planning paradigm. We create Multi-sourced Value Maps to model the effect of key elements in instruction navigation. This way, the linguistic DCoN planning can be converted into robot actionable trajectories. With InstructNav, we first achieve zero-shot performance on the R2R-CE task [3] and outperform wide array of task-training methods. Furthermore, InstructNav achieves 10.48% and 86.34% improvement the zero-shot

---

Habitat ObjNav and fomandd-drvve navigation DDN compared with state-of- models. In the real world, InstrtuctNav demonstrates robustness in diverse kinds of instructions and indoor scenes including apartment, office, library, gallery, and teaching buildings.
Related Work
2.1 Instruction-guided Navigation
Controlling a navigation robot through natural language instructions in unexplored environments is user-friendly interaction mode. Many research works have been conducted in this area. According to the instruction type, these works can be categorized into object goal navigation, visual language navigation, and demand-driven navigation. The object goal navigation meth- ods
can find one specific object in the scene. The visual language navigation approaches [][[5][[]][]][88]]]]] can follow one step-by-stey bt- instruction to reach a speci- fied destination. The demand-driven navigation models [5] can satisfy human demand by searching for related objects in the scene. Although these methods can perform pre-defined type of instruction, all of them are incapable of executing different kinds of instructions, which significantly limits their application scenarios. Compared with them, our InstructNav can follow diverse types of instructions to navigate with convincible success rates.
2.2 Large Models in Robotics Navigation
With snternet-scale training data, large models including large language models [19][0[[[11[22] and multimodal large models [23][24][25] [26][27] have emerged with powerful capabilities, which include instruction following, task planning, and visual perception. These capabilities are closely related to instruction navigation. The development of large models has motivated their application in robotic navigation. Some [12][][[8][2][L] directly used visual perception features encoded by multimodal large models (e.g., CLIP[23] and BLIP2[24]) retrieve target position while oth- ers [30][3][[2]]33]]]]] just simply leveraged large language models make high-level navigation planning. Although these methods are all based on large models with strong generalization capabili- ties, none of them support different kinds of navigation instructions. Our InstructNav aims to relive the potential of large models to navigate with different kinds instructions a zero-shot way.
3 METHODOLOGY
3.1 Problem Formulation and Method Overview
Problem Formulation The generic instruction navigation task requires the robot to follow one instruction / in natural language format to reach the target location in the unexplored continuous environment. At each step, the robot can observe egocentric RGB image V, and depth image D,. The robot also knows its camera pose P, With observations O, D, D, P,), the robot needs to execute low-level action a, to move towards the target. No pre- built maps are allowed in this task.
Method Overview We propose to handle the unified instruction navigation via a new planning paradigm: Dynamic -avviato- (DCoN) in Section 3.2. Then, we create Multi-sourced Value Maps (Section 3.3) to model key elements in generic instruction navigation. By deciding with these value maps, our method can plan actionable trajectories for low-level movement (Section 3.4).
3.2 Dynamic Chain-of-Navigation Planning
After analyzing instruction -uided navigation processes, we discern that the navigation robot should follow one action to move towards specific navigation landmarks. Consequently, the navigation instructions can be transformed into "Action 1 Landmark 1 - Action 2 Landmark 2 - schema, which is similar to the chain-of-thoupe thinking process of the large language model [34]. Therefore, this paradigm can be aptly named "Chain-of-Navigation (CoN)". The conversation from raw navigation instruction to CoN can be obtained through L.L.M.
However, generic navigation planning cannot be realized by directly extracting actions and landmarks from raw instructions. Firstly, the landmarks specified in the instruction like "arched wooden doors" may not align with the object labels produced by the semantic segmentation model like "Doorway".
3

---

Textual Prompt
Part Robot Definition
Part II: Nav. Strategy <Candidate Navigation Actions
Part IlI: Pred. Format
Part IV: Episode Info.
[ACL Ex.xxllled
<Baackgroudd Descriptions
<Prediction Requirement>
[Novigation Instruction)
<Novigation Task Definition>
<Requirements for Landmarks>
<Qutput Format>
[Previous Plan)
<Choin of Novigation Definition>
<trategy Description
<lermination Condition
[Observed Scene Objects]
Object "Find a sola"
Explore Dest
Explore
Television
Approach - Sofa (Stop)
LLM C Cuddan
Visual Language Nov: sland)
Approach Plano
Move Forward-
DDorway
Right Burhtu (Stop)
bathi.b
Demand-driven Nav: am thiney"
Sexre Ofjects from Nep
Eaplore Couch
Approach
Diving
Approach Bottle (Stop)
Navigtiion
Step
Step
Masiggttion Instruction
Planner
Dynamic Chain of Navigatin
The workflow of Dynamic Chain-of-Naviaation (DCoN). Different types of navigation instructions can be unified into DCoN by LLM. The next action and landmarks will be updated based on observed scene objects at every decision step. Beyond extracting actions and landmarks, DCoN achieves semantic label aligmment, common-sent -ense reasoning, and environmental exploration for navigation planning. Complete prompt and output examples are shown in Supplementary Materials.
Figure 2:
which hinders the retrieval of landmarks on the map. Secondly, the target landmarks may not be observed the explored environment. At this time, navigation planning should inspire efficient environment exploration. Thirdly, instruction with abstract human demand like "I am thirsty" cannot be decomposed into concrete actions and landmarks all.
To address these problems, we propose Dynamic Chain-of- -aaiiattt- (DCoN) as shown in Figure 2. Rather than make the whole plan at one time, the DCoN will be re-inferred at each decision-making step to update the next action and landmarks based on observed scene objects. The textual prompt for DCoN is composed of "Robot Definition". "Navigation Strategy", "Prediction Format" and "Episode Information" four parts. The < <Candidate Navigation Actions> defines common navigation actions like "Explore". "Approach", "Move Forward".. ... choose from. The <Requirements for Landmarks> prioritizes the observed objects when planning the next landmarks so that they are more likely to be retrieved in the creation of the Semantic Value Map. Strategies for different instruction navigation tasks are defined in <Strategy Description> to infer the next landmark considering given navigation instruction, common house layout, and human habits. The I.L''s prediction is formatted as ('Reason':.. 'Action':.. 'Landmark':.. 'Flag":... :...). As Figure examples, DCoN can align "arched wooden doors" with "Dourway". guide the robot to explore areas with TV to find a sofa and approach a bottle of water to relieve thirst. With DCoN, InstructNav realizes landmark alignment, environment exploration, and commonsense reasoning for different types of navigation instructions.
3.3 Multi-sourced Value Maps
The DCoN planning is in language format, which cannot directly control the robot's movements. To convert the linguistic DCoN planning into robot actionable trajectories, we create Multi-sourced Value Maps (Figure 3). These value maps model key factors in the generic instruction navigation, including action, landmarks, and navigation history. The next waypoint and actionable trajectories can be decided based on Multi-sourceed Value Maps. During the navigation, RGB-D and camera pose are utilized to build scene point cloud PCD. The area the explored ground, free from obstacles, is selected as navigable area PCD The next DCoN action and landmarks to be executed are denoted as A; and L, Each value map is initialized with all-zero values and detailed in the following.
3.3.1 Semantic Value Map
Accurately mapping observed objects' locations and semantic information is critical for navigating toward DCoN-specifies landmarks. To this end, create a Semantic Value Map m, for DCoN landmarks L, In the navigation process, the 2D semantic segmentation mask [35] is lifted to 3D by depth and camera pose to obtain scene semantic point cloud PCD The semantic value Csem of m, is calculated based on the normalized minimal distances between each navigable area position E PCD and the L, positions E PCD following Equation 1 and 2.
d mingepCD I| all Vp E PCD Rav
(I)

---

Czem
dyo minn (cem))) max(()ew) - -min(daew)
(2)
As the above equations, the areas near the landmarks L will have relatively higher values than others. 3.3.2 Action Value Map
To endow InstructNav with the capability execute concrete movement actions and explore the frontiers, we propose the Action Value Map Ma. A value assignment operation is conducted on the initial all-zero value map by the DCoN A type. The operation specifics are detailed below:
Move forward Turn around / Turn right Turn left: Values of one are assigned to the front back right/ left sector area at the robot current location. Each region corresponds to one-quarter of the panoramic Field of View (FOV) at the current location.
Explore: Values of one are set for the boundaries of the present explored environment.
Enter / Exit: Given that entering or exiting any room necessitates crossing door-shaped regions, the "Enter" or "Exit" actions will be replaced by a "Approach" action and a "Doorway" landmark will be integrated into the DCoN planning.
Approach: This action does not necessitate any operation on the Action Value Map. It can realized directly through the Semantic Value Map.
This way, the action A related areas will have higher values than others on Ma-
Navigation Intruction
cacaring
rertined
Step
Scene Ofject Labels
action thoi Aucion, ECoN Lendvant
Posa
20 10000
Trgjetory Map
Dynamic of
LLM
Current Panorama
Action
HOW
DOGA Action
Action Valua
Landmark
Next Waypoint
Action
DOUR L
Serrantc Unlun Map
Decision-making Value Map m
Segrerartee
Poso
m, m, m. m,
Figure 3: The system framework of Nav. The next Action i and Landmarks are obtained from DCoN. Scene semantic point cloud is created from the RGB-D observation and 2D semantic segmentation. With this information, Multi-sourced Value Maps mg. my, M, and m; can be established. Areas with redder colors represent higher 1 values, while bluer colors indicate lower 4 values. By synthesizing them into a decision maasiinnmmainng value map m, InstructNav can plan the next waypoint.
3.3.3 Trajectory Value Map
To encourage more diverse navigation trajectories, we design Trajectory Value Map m, for Instruct- Nav. During the navigation, the robot's positions are continually recorded as history trajectory PCDrraj. The trajectory value Craj of m, is calculated as the normalized minimal distances between each navigable area position p PCD and every history position h PCDrraj
dmmj mingerCD || h|1. Vp PCD
(3)
dmaj mmin-ran(draa)) max(dmaa) -minn(dra
Ciraj
As Equation 3 and 4, the areas far from history trajectory have relatively higher values navigate.
3.3.4 Intuition Value Map
To further improve InstructNav's capabilities on multimodal semantic reasoning, we propose an Intuition Value Map m;. The next navigation area predicted by the multimodal large model (MLM) is projected this map with values of one to guide the robot to move. Figure 3 displays how InstructNav leverages GPT-4V to predict the next navigation area. For the visual input, N RGB
5

---

observations from different directions at the current position are sampled at equal intervals and concatenated into panorama image P. The effect of N is studied in the Ablation Study. Each observation image is annotated with the corresponding direction ID. For the textual input, we define task and provide a complete navigation instruction I, next action A; and landmarks L, from Dynamic Sraain-o--avvigttinn In response, the large model will first conduct chain-of-thought CoT; analyze visual information in each direction and then make decisions the next movement direction Dir, Refer to Supplementary Materials for details about input prompt and output examples. The multimodal large model makes inferences as Equation 5.
(CoT, Din) (P: PELMPPILAAA L.)
(5)
The FOV at direction Dir; is projected on the Intuition Value Map as MLM predicted navigation area. there are navigable positions in this area, they will be assigned values of one. Otherwise, the failure feedback will be transmitted back to the large model, instigating re- prediction.
3.4 Navigation Process with Multi-sourced Value Maps
As Equation 6, a decision makiinn value map m is obtained by summing over all four value maps.
mi + Ma + my + my
zero for obstacle avoidance. Then, following Equation 7, the navigation goal (x))).2) is set to be the point with the highest value the and the robot trajectory is planned by A algorithm, which is also based the generated value map m. In the simulator, the action space follows discrete setting, we use simple rotate-then- [366 to track the planned path and while in the real-world, we directly control the robot speed. The navigation stopped when DCoN sets "Flag" to "True" or GPT-4V outputs "Stop" judgment.
Obstacles areas on the decision-making value map m are
yi, 2i) = arg max (sxx)em (xx,),z)
Table 1: Comparison with SOTA methods HM3D object goal navigation.
Mathod
Training Free
Novel Object
Support Instruction
Novigotion Error
Snccess Rute
SPL.
SemExp (37)
X
ObjectNav
2.94
37.9
18.8
FixelNav [7]
/
ObjectNar
37.9
20.5
Hahitar-Wed Weh
ObjentNav
16.0
(VR]
ObjectNav
ZSON [40]
V
ObjeciNav
25.5
12.6
ESC [6]
/
ObjectNav
39.2
22.3
ObjectNav
LSMVN [14]
ObjectNav
4,43
50.4
23.1
VLFM [11]
ObjectNav
52.5
30.4
InstructNav (Ouns)
Generic
2.58
20.9
Table Comparison with SOTA methods on R2R-CE visual language navigation. For fairness, only methods free from MP3D dataset-specific waypoint predictors are studied.
Method
Training Free
Support Instruction
Trijectory Length
Nivigation Frror
Success Rate
SPI.
Saomi [42]
VLN
7,80
8.32
24
22
Seg2Sen
VIN
7.27
25
CWP-CMA
7.54
CWP.ReBEET
VLN
7.42
7.66
23
Ego2Map NaViT [9]
VLN
8.00
7.25
30
29
CMA [3]
VIN
864
7.37
32
50
NaVid (43)
VLN
7.03
5.47
37
InstructNav (Ours)
Generic
7.74
6.89
Table 3: Comparison with SOTA methods on DDN demand-driven navigation.
Method
Training Free
Novel Object
Support Instruction
Trajectory Longth
2SON-3mmed (5)
x
DDN
3.5
2.4
VTN-C1IP--emen C1-P-dd
x
DDN
9.1
3.9
DDN
ChamPiT-Prompt [19]
MinKPP4 (44)
/
/
DDN 0.74
2.9
2.0
InstractNav (Ours
/
/
Generic
4,44
30.0
14,2

---

EXPERIMENTS
4.1 Experiment Setup
For object goal navigation, we evaluate our method on the HM3D [45] dataset in Habitat simulator
following Habitat ObjectNav challenge's setting [1]. For visual language navigation, we test our method on the R2R-CE [3] dataset val-unseen split with the Habitat simulator. For demand-driven navigation, we evaluate our method on the DDN [5] dataset based on A12Thor [46] simulator and ProcTThrr [47] scenes following its unseen scenes and instructions setting. As previous work [3] [1] [5], we take Trajectory Length (TL), Navigation Error (NE), Success Rate (SR), Oracle Success Rate (OSR), and SR penalized by Path Length (SPL) as evaluation metrics.
4.2 Implementation Details
We utilize GPT-4 (gpt-4-0613) [21] to plan Dynamic Chain-of-Navigation and adopt GPT-4V (gpt-4- I 1e1--vision-rreviee [10] to judge navigation directions for Intuition Value Map. Their parameters all the OpenAl default settings. The number of RGB observations in the visual prompt (ie., N) is set 6 following the ablation study result. When creating semantic point cloud, deploy GLEE [35] on one RTX 4090 GPU to conduct semantic segmentation on the RGB image. Note that our InstructNav is completely free from navigation training and pre-built maps.
4.3 Simulation Experiments
Object Goal Navigation on HM3D We compare our method with state-of-the-art object goal navigation models on HM3D datasets. Table shows that our InstructNav outperforms all zero-shot methods on success rate and is comparable to the best trained object navigation model OVRL [39].
Visual Language Navigation on R2R-CE Many previous methods rely on waypoint predic- tor [36] [48] specifically trained the Matterport3D topological map improve their performances on MP3D-related navigation tasks. Our method predicts the next waypoint based on our designed Multi-sourced Values Maps in zero-shot way. Therefore, We follow [8]. [9] and [43] to compare our InstructNav with VLN models free from waypoint predictor trained on MP3D topological map. From Table 2, can be observed that InstructNav is the first model that completes visual language navigation task in a zero-shot way and outperforms wide array of task-trained models.
Demand-driven Navigation on DDN The baselines of the DDN task include task-adapted large models. We compare our method with these trained and non-trained models on the unseen scenes and unseen instructions. As Table 3, InstructNav outperforms all baselines by large margin.
4.3.1 Ablation Study
To conduct the ablation study, randomly sampled 100 instructions for each task respectively.
The effect of N RGB observations in MLM's visual prompt
We test three differentt values of N to study its influence on InstructNav. The "N = 4" concatenates front, right, back, and left (i.e., Direction 1, 4. 7, 10) RGB observations as MLM's visual prompt while "N = 12" concatenates all twelve RGB observations. The "N 6" selects 6 out of all 12 RGB observations with an interval as visual input MLM. From Figure
Dahae
50
sccess Rate
40
30-
20
observe that "N 6" consistently produces hetter performance on all three navigation tasks. After analyzing failure cases, found that smaller N may miss some key visual information, while larger N may increase the understanding burden of MLM. Therefore, we set N to in our implementation.
10
N-12 Figure 4: Effect of N RGB observations.
The effect of DCoN and Multi-sourced Value Maps To verify the effectiveness of DCoN and Multi-soureed Value Maps in our method, we ablate DCoN and four value maps in Table 4. Through the experiment, find that DCoN is critical to the InstructNav. Ablating DCoN results in significant decrease in the success rate all three tasks. Our method relies on DCoN to achieve unified planning for different instruction navigation tasks. Besides, all four Multi-sourced Value
7

---

Maps show improvement as expected. The ablation of any one value map will weaken InstructNav's s performance on all tasks. The Multi-sourced Value Map is a suitable representation of key factors involved in the decision of generic instruction navigation.
Table 4: Ablation study about DCoN and Multi-sourced Value Maps in our InstructNav system.
Mothod
Object Nav, on HM3D
Visual Language Nav. on R2R CE
Demand-driven Nav, on DDN
NE
SR
SPI.
TI.
NF:
OSR
SR
SPI.
T1.
SR
SPI.
InstrueNNov
6.04
4.27
we Dynamic CoN (DCoN)
3.09
11
19.4
7.42
7.49
46
23
18
4.51
22
10.8
w/o Action Valne Map
3.26
51
19.1
6.82
6.97
39
28
22
4.76
15.9
w/o Semanic
Map
21
w/o Trajectory Value Mup
3.00
52
21.7
5.54
6.83
31
19
16
3.81
11.4
wio Intuition Value Map
3.13
54
21.2
9.22
9.18
47
17
11
5.62
20
11.1
The effect of open-source large models We further replace GPT models in InstructNav with
Llama3 70B [22] and 1LaVA1.6 34B (i.e., LLaVA-NeXT 34B) [27] explore the feasibility of driving InstructNav by open-source models. Specifically, Llama3 is utilized plan the DCoN while LL. 6 is adopted to create the Intuition Value Map. From Table 5, we observe that open-source models can achieve comparable performances to GPT models in portion of tasks (e.g., ObjectNav). This benefits from the robust design of DCoN and Multi-sourced Value Maps in the InstructNav. However, open-source models still exhibit weakness compared with GPT models.
Table 5: Comparison between open-soured and close source large models in InstructNav.
Large Models InstructNav
Object Nav. on HMBD
Visual Language Nav. on R2R-CE
Demand-driven Nav. on DDN
(Langugee + Mulliodall
NB
SR
SPL.
TL
NE
OSR
SR
SPL
TL
SR
SPL
GPT4 + GPT4V
2.11
56
22,5
6.74
6.04
42
22
4.27
15.6
Llamaà 70B GPT4V
3.19
50
18.9
6.89
7.02
40
23
19
4.09
21
11.0
GPT4 aVAA 34B
19.4
7.97
13
5.45
13.9
70B LLaVA1.6
178
5.82
8.34
5.30
4.4 Real Robot Experiments
We perform real robot experiments based on the Turtlebot 4 mobile robot. Our robot is equipped with an ORBBEC Astra Pro Plus RGB-D camera connected to the ThinkPad E14 laptop computer and an RPLIDAR-A/ lidar connected to Raspberry Pi 4B as sensors. All processors are installed on the robot and communicate with each other through a portable WiFi. To accelerate, InstructNav is deployed on a remote RTX 4090 workstation. We utilize the SLAM Toolbox [49] for self-localization and the Navigation2 [50][51] for point-to-point navigation with dynamic obstacle avoidance.
The real robot experiments are conducted in representative indoor scenes including large-scale offices and multi-room apartments, open library. gallery, and teaching building. To demonstrate the effectiveness of our approach, every instruction is independently executed without any pre-b map. For each scene, we create diverse navigation instructions covering different instruction types and navigation goals. Figure displays the scenes and part of navigation instructions in our real robot experiments. Videos about real robot experiments can be found in the Supplementary Materials.
Conclusion
In this work, we focus on developing the first generic instruction navigation system InstructNav in the continuous environment without any navigation training or pre-built maps. To reach this goal, we propose the Dynamic -hain-of-Navigatioo to unify different navigation instructions and model key elements in instruction navigation through Multi-sourced Value Maps. This way the linguistic DCoN planning can be converted into robot actionable trajectories. Extensive experiments on the simulators and the real robot demonstrate the generalization and effectiveness of our training-free free method.
Limitation and Future work The current InstructNav system still relies on close source large models to achieve the best performance. The inference speed depends on the network quality. Besides, the API calls produce cost. In the future, we will try to design a data generation pipeline to overcome data scarcity and develop an end-to-en mnd-t--end model for generie instruction navigation.
8

---

References
[1] K. Yadav, J. Krantz, R. Ramrakhya, S. K. Ramakrishnan, J. Yang, A. Wang, J. Turner, A. Gokaslan, V.-P. Berges, R. Mootaghi, O. Maksymets, A. X. Chang, M. Savva, A. Clegg, D. S. Chaplot, and D. Batra. Habitat challenge 2023, 2023.
[2] Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and Savarese. Gibson env: Real-world perception for embodied agents. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9068-907777 2018.
[3] Krantz, E. Wijmans, A. Majundar, D. Batra, and S. Lee. Beyond the nav-graph: Vision and language navigation in continuous environments. In European Conference on Computer Vision (ECCV), 2020.
[4] A. Ku, P. Anderson, Patel, E. le, and Baldridge. Room-across-room: Multilin- gual vision-aand-aagguage navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954 2020.
[5] H. Wang, A. G. H. Chen, X. Li, M. Wu, and H. Dong. Find what you want: Learning conditonon object attribute space for demand-drive navigation. Advances in Neural Information Processing Systems, 2023.
[6] K. Zhou, K. Zheng, C. Pryor, Y. Shen, H. Jin, L. Getoor, and X. E. Wang. Esc: Exploration with soft commonsense constraints for zero-shot object navigation. In /CMI.'2,, 2023.
[7] W. Cai, S. Huang, G. Cheng, Y. Long, P. Gao, C. Sun, and H. Dong. Bridging zero-shot object navigation and foundation models through pixel-guided navigation skill, 2023.
[8] Y. Hong, Z. Wang, Q. Wu, and S. Gould. Bridging the gap between learning in discrete and continuous environments for vision-and- languageg navigation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[9] Y. Hong, Y. Zhou, R. Zhang, F. Dernoncourt, T. Bui, S. Gould, and H. Tan. Learning navigational visual representations with semantic map supervision, 2023.
[10] Z. Yang. I.. Li, K. Lin, J. Wang, C.-C. Lin, 7. Liu, and I.. Wang. The dawn of Imms: Preliminary explorations with gpt-4v(ision), 2023.
[11] N. H. Yokoyama, S. Ha, D. Batra, J. Wang, and B. Bucher. Vlfm: Vision-language frontier maps for zero-shot semantic navigation. In 2nd Workshop on Language and Robot Learning: Language as Grounding, 2023.
[12] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[13] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and E. Wijmans. ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects. In wrXXi:2206613/11 2020.
[14] B. Yu, H. Kasaci, and M. Cao. L3mvn: Leveraging large language models for visual target navigation. arXiv preprint arXiv:2304.0550/. 2023.
[15] A. Majumdar, A. Shrivastava, S. Lee, Anderson, D. Parikh, and D. Batra. Improving vision- and-language navigation with image-text pairs from the web. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 259 274.. Springer, 2020.
[16] A. Moudgil, Majumdar, H. Agrawal, Lee, and D. Batra. Soat: A scene-and object-aware transformer for vision-and laaion-an-- navigation. Advances in Neural Information Processing Systems, 34:7357-7367, 2021.

---

[17] P.-L. Guhur, M. Tapaswi. S. Chen, Laptev, and C. Schmid. Airbert: In-domain Pretraining for Vision-and-Langguaee Navigation, 2021.
[18] Y. Goel, N. Vaskevicius, L. Palmieri, N. Chebrolu, and C. Stachniss. Predicting dense and context-aware cost maps for semantic robot navigation. arXiv preprint arXiv:2210.08952. 2022.
[19] B. Brown, B. Mann, Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, Neelakantan, Shyam, G. Sastry, Askell, et Language models are few-shot learners, arXiv preprint 20l:200051416,, 2020.
[20] OpenAl. Introducing chatgpt, 2022.
[21] OpenAL. Gpt-4 technical report, 2023.
[22] AI@ Meta. Llama 3 model card. 2024. URL 11ama3/blob/ http///oo///a////aoa/aattttaaatt main/MODEL,.......
/
md.
[23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021. URL https: http///arxvvvoo/o/ abs/ 2203/ 00020.
[24] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping Eiguage-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301112597, 2023.
[25] W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. C. H. Hoi. Instructblip: Towards abs/2305.06555 2023. doi: 10..555/aaX...335.66666333
vision-language models with instruction tuning. CoRR,
[26] H. Liu, Li, Q. Wu, and Y. Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08885 2023.
[27] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL
[28] J. Chen, G. Li, S. Kumar, B. Ghanem, and F. Yu. How to not train your dragon: Training- embodied object goal navigation with semantic frontiers, 2023.
[29] D. Shah, B. Osińskii S. Levine, et al. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In Conference on Robot Learning, pages 492-504. PMLR. 2023.
[30] G. Zhou, Y. Hong, and Q Wu. Navgpt: Explicit reasoning in vision-and-lant ann-uagg navigation with large language models, 2023.
[31] K. Rana, J. Haviland, S. Garg, J. Abou-Chakra I. Reid, and N. Suenderhauf. Sayplan: Ground- ing large language models using 3d scene graphs for scalable task planning. 7th Annual Conference on Robot Learning, 2023. URL https: /ttp///peeeeeeeeeeeee/eeeee net forum?id= wMpOM00Ss7a.
[32] Y. Long, X. Li, W. Cai, and H. Dong. Discuss before moving: Visual language navigation via multi-expert discussions, 2023.
[33] A. Rajvanshi, K. Sikka, X. Lin, B. Lee, H.-P Chiu, and A. Velasquez. Saynav: Grounding large language models for dynamic planning to navigation in new environments, 2023.
[34] Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q V. Le, Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 04:24824-24877 2022.
10

---

[35] J. Wu, Y. Jiang, Q. Liu, Z. Yuan, X. Bai, and S. Bai. General object foundation model for images and videos at scale, 2023.
[36] D. An, H. Wang, W. Wang, Z. Wang, Y. Huang, K. He, and L. Wang. Etpnav: Evolving topological planning for vision lainunla navigation in continuous environments. arXiv preprint arXiv:2304.088. 00344, 2023.
[37] D. S. Chaplot, Gandhi, A. Gupta, and Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. In In Neural Information Processing Systems (NeurIPS), 2020.
[38] R. Ramrakhya, E. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object-search strategies from human demonstrations scale. In CVPR, 2022.
[39] K. Yadav, R. Ramrakhya, A. Majumdar, V.-P. Berges, S. Kuhar, D. Batra, A. Baevski, and O. Maksymets. Offline visual representation learning for embodied navigation. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023.
[40] A. Majumdar, G. Aggarwal, B. Devnani, J. Hoffmman, and D. Batra. Zson: Zero-shot object-goat navigation using multimodal goal embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[41] Wu, Y. Mu, B. Wu, Hou, Ma, Zhang, and Liu, Voronav: Voronoi-based zero-shot object navigation with large language model, 2024.
[42] M. Z. Irshad, N. Mithun, Z. Seymour, H.-P. Chiu, S. Samarasekera, and R. Kumar. Semantically-aware spatio-temporal reasoning agent for vision-and-languag navigation in continuous environments. In 2022 26th International Conference on Pattern Recognition (ICPR), pages 4065-4071 IEEE, 2022.
[43] J. Zhang, K. Wang, R. Xu, G. Zhou, Y. Hong, X. Fang, Q. Wu, Z. Zhang, and W. He. Navid: Video-based vlm plans the next step for vision-am aad--and-lnn und-language navigation. arXiv preprint arXiv:2402. 15852, 2024.
[44] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[45] Ramakrishnan, Gokaslan, E. Wijmans, O. Maksymets, A. Clegg, J. M. Turner, E. Un- dersander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and Batra. Habitat- matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL wwtpt:ttt///.c....m /ataaaivvv.//// org/ abs 2109. 08238.
[46] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi. A12-THOR: An Interactive 3D Environment for Visual AL. arXiv, 2017.
[47] M. Deitke, F. VanderBilt, A. Herrasti, L. Weihs, J. Salvador, K. Ehsani, W. Han, E. Kolve, A. Farhadi, A. Kembhavi, and R. Mottaghi. ProcTHOR: Large-Scale Embodied AI Using Procedural Generation. In NewrIPS, 2022. Outstanding Paper Award.
[48] D. An, Y. Qi, Li, Y. Huang, I. Wang, T. Tan, and Shao. Bevbert: Multimodal map pre-training for language-guided navigation. Proceedings of the IEEE/CVF International Conference Computer Vision, 2023.
[49] S. Macenski and I. Jambrecic. Slam toolbox: Slam for the dynamic world. Journal of Open Source Software, 6(61):2783, 2021.

---

[50] S. Macenski, T. Moore, D. Lu, A. Merzlyakov, and M. Ferguson. From the desks of ros maintainers: A survey of modern capable mobile robotics algorithms in the robot operating system 2. Robotics and Autonomous Systems, 2023.
[51] S. Macenski, F. Martín, R. White, and J. Ginés Clavero. The marathon 2: A navigation system. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), 2020. URL https:////// ///gtgttulll com/ ros- palssslplannggggaaee pllemning/maviieeioon. naw.gatiooi....
12


Reviews:
[{"review_id": "z0VPQewUby", "reviewer": "Reviewer_3AfH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes InstructNav, an approach to robot navigation that can handle generic instructions in unexplored environments. InstructNav uses Dynamic Chain-of-Navigation to enable a LLM to decompose a generic navigation instruction expressed in natural language into a chain of common navigation actions along with landmarks, and then uses the action and landmarks to generate multiple value maps that are composed into a cost function for a planner to generate navigation paths that accomplish the task.", "summary_of_recommendation": "I like this approach and the paper overall, but I do have some concerns about what kind of commands are possible with the current proposed approach.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "As mentioned in the main review: How does this system handle the following type of instructions:\n- Avoiding landmarks (“Go to the kitchen but avoid going near the tables”)\n- Instructions involving back-tracking (“go to the sofa, then go to the chair, then go back to the sofa”)\n- Cyclic commands (“continuously patrol around the room”)\n- Object-centric spatial references  (“Go to the left side of the bed”).\n\n- If a user instructed the robot to move a specific amount forward (e.g: 5 feet forward, 10 meets to the right), how does the action value map take the numeric component into account? Are the locations relative to the robot that are assigned 1 in the move forward / turn right / etc. hard-coded, or are the distances extracted from the instruction?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "08TpSN1AZb", "reviewer": "Reviewer_dRHf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "InstructNav is a system for generic instruction navigation in unexplored environments, designed to execute diverse language instructions without task-specific training or pre-built maps. Its key innovation, Dynamic Chain-of-Navigation (DCoN), unifies various navigation instructions into a standard planning paradigm, while Multi-sourced Value Maps convert linguistic plans into actionable robot trajectories. The system excels in Object Goal Navigation, Visual Language Navigation, and demand-driven navigation tasks, outperforming existing methods in simulations and real-world experiments. InstructNav demonstrates superior performance on HM3D object goal navigation, achieves zero-shot performance on R2R-CE visual language navigation, and surpasses state-of-the-art on demand-driven navigation. It was tested extensively in simulations and with a Turtlebot 4 in diverse indoor environments.", "summary_of_recommendation": "InstructNav is a flexible approach for embodied navigation that handles multiple task types without task-specific training or pre-built maps. The core innovations - Dynamic Chain-of-Navigation and Multi-sourced Value Maps - show potential in unifying navigation instructions and converting linguistic planning into actions, attaining competitive performance with state-of-the-art zero-shot methods across multiple benchmarks. While the paper is well-structured and clearly explains key components, it has limitations including reliance on large language models, lack of quantitative real-world results, and limited discussion of failure modes. To strengthen the work, the authors should provide a detailed analysis of failure modes, and clarify the ablation study methodology (particularly regarding how the system functions without Dynamic Chain-of-Navigation).", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The paper lacks a discussion of failure modes. Please provide a detailed analysis of scenarios where InstructNav struggles or fails, including examples from both simulated and real-world experiments. It is important to show how the design choices made by the authors result in behavior that is distinct from previous methods, both in cases that result better performance than other methods and worse performance.\n\nClarification needed on ablation study methodology:\nTable 4 presents an ablation study of the InstructNav system, including a row 'w/o Dynamic CoN (DCoN)'. However, the paper describes the Dynamic Chain-of-Navigation as necessary for generating inputs for the Multi-sourced Value Maps, particularly the Action Value Map and Semantic Value Map. It's unclear how these maps could be utilized without DCoN.\nPlease address the following:\n- Explain how the system functions without DCoN in this ablation.\n- Clarify how inputs for the value maps are generated in the absence of DCoN.\n- If there's an alternative method being used, please describe it in detail.\n- If there's an error in the table or its description, please correct and discuss any potential impact on the interpretation of results.\nFor the ObjectNav results, the authors state that their results are “comparable to the best trained object navigation model OVRL”, however I believe that OVRL is a relatively older method that is no longer the ‘best trained model’. From what I remember, PIRLNav outperformed OVRL, and got 70.4% success and 34.1% SPL (vs. 58% and 20.9% from InstructNav).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RUzUG5qcaI", "reviewer": "Reviewer_nMyn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a system enabling robots to navigate diverse natural language instructions without prior training or maps. It presents Dynamic Chain-of-Navigation (DCoN), which unifies planning by dynamically updating navigation plans based on observed objects and environment context. Additionally, Multi-sourced Value Maps convert linguistic plans into actionable robot trajectories, incorporating action, semantic, trajectory, and intuition maps for comprehensive guidance. InstructNav achieves notable zero-shot performance on multiple tasks, outperforming several trained models, and demonstrates robustness in diverse real-world settings, such as apartments, offices, and libraries. This system marks a significant advancement in robotic navigation, offering versatile and effective operation across various environments and instructions without specific training.", "summary_of_recommendation": "The paper presents a generic instruction navigation method without training any model. This method can be easily applied to multiple platforms, and can have more impact if authors illustrate more about the above rebuttal questions", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "Please consider answering the following questions:\nWhy is it necessary to select several discrete actions instead of continuous linear and angular velocities?\nHas the authors consider using panorama views instead of taking images multiple times for a single inference? Would be great to add a quick ablation in the short rebuttal phase.\nIf there are multiple floors that robot should go to, are there ways to modify current maps to navigate robots to go to different floors?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: # Strengths
1. The paper introduces a novel zero-shot method for handling diverse navigation instructions.
1. The real-world demonstrations showcase the effectiveness of the approach.
1. The authors evaluated InstructNav across multiple benchmarks.
1. The paper provides a comprehensive evaluation, including ablation studies.

# Weaknesses
1. Although real-world experiments are conducted, quantitative results from these experiments are missing.
1. The proposed method relies mainly on multimodal large language models, which can be impractical due to computational costs, latency, and network issues.
1. Failure mode analysis is insufficient.

### Post-rebuttal comment
The reviewers agree in their recommendation to accept the paper because most of the concerns have been addressed. I agree with their consensus.
Decision: Accept