ViNT: A Foundation Model for Visual Navigation
Authors: Dhruv Shah
Ajay Sridhar
Nitish Dashora
Kyle Stachowicz
Kevin Black
Noriaki Hirose
Sergey Levine
Venue: CoRL 2023 Oral (2023)
Keywords: visual navigation
multi-task learning
planning
generalization

Abstract: General-purpose pre-trained models (``foundation models'') have enabled practitioners to produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is available for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), a foundation model that aims to bring the success of general-purpose pre-trained models to vision-based robotic navigation. ViNT is trained with a general goal-reaching objective that can be used with any navigation dataset, and employs a flexible Transformer-based architecture to learn navigational affordances and enable efficient adaptation to a variety of downstream navigational tasks. ViNT is trained on a number of existing navigation datasets, comprising hundreds of hours of robotic navigation from a variety of different robotic platforms, and exhibits positive transfer, outperforming specialist models trained on narrower datasets. ViNT can be augmented with diffusion-based goal proposals to explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with a technique inspired by prompt-tuning, where the goal encoder is replaced by an encoding of another task modality (e.g., GPS waypoints or turn-by-turn directions) embedded into the same space of goal tokens. This flexibility and ability to accommodate a variety of downstream problem domains establish ViNT as an effective foundation model for mobile robotics.

ViNT: A Foundation Model for Visual Navigation
Dhruv Shah', Ajay Sridhar', Nitish Dashora', Kyle Stachowicz, Kevin Black, Noriaki Hirose, Sergey Levine UC Berkeley
Abstract: General-purpose, pre-trained models ("foundation models") have en- abled practitioners produce generalizable solutions for individual machine learning problems with datasets that are significantly smaller than those required for learning from scratch. Such models are typically trained on large and diverse datasets with weak supervision, consuming much more training data than is avail- able for any individual downstream application. In this paper, we describe the Visual Navigation Transformer (ViNT), foundation model that aims to bring the success of general-purpose pre-trained models vision-based robotic navi- gation. ViNT is trained with general goal-reaching objective that be used with any navigation dataset, and employs a flexible Transformer-baat based architec- ture to learn navigational affordances and enable efficient adaptation to variety of downstream navigational tasks. ViNT is trained on number of existing naviga- tion datasets, comprising hundreds of hours robotic navigation from a variety different robotic platforms, and exhibits positive transfer, outperforming special- models trained on narrower datasets. ViNT can be augmented with diffusion- based goal proposals explore novel environments, and can solve kilometer-scale navigation problems when equipped with long-range heuristics. ViNT can also be adapted to novel task specifications with technique inspired by prompt-tuning. where the goal encoder is replaced by an encoding of another task modality (c.g.. GPS waypoints or tum-by-turn directions) embedded into the same space of goal tokens. This ficxibility and ability to accommodate a variety of downstream prob- lem domains establish ViNT as an effective foundation model for mobile robotics.
Keywords: visual navigation, multi-task learning, planning, generalization
kf Veleeeer Tepkrasi
Coverag Mapping
Triining
Foundiaiionn
Zero-Sha Shott Deployment
Adapt
Figure 1: Overview of the VINT foundation model. VINT generalizes zero-shot across environments and robot embodiments, can directly applied to tasks including exploration and navigation around humans. ViNT can also be fine-tuned with small amount data to expand its capabilities to new tasks.
Introduction
Recently, machine learning methods have achieved broad success in natural language processing [1], visual perception [2-4], and other domains [5, 6) by leveraging Internet-scale data to train general- purpose "foundation" models that can be adapted to new tasks by zero-shot transfer, prompt-tuning. or fine-tuning on target data [7-10]. Although this paradigm has been successful in many domains,
Lead Authors. Unabridged paper, videos, code: general-naviiaaionn-oool modells hit.aiaggaaiiioooommiiiuu.....o.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

it is difficult to apply in robotics due to the sheer diversity of environments, platforms, and applica- tions. In this paper we ask the question: what is required of a foundation model for mobile robotics?
In this paper, we define a robot foundation model as a pre-trained model that can be (i) deployed zero-shol in novel, useful settings (e.g., different sensors, robot embodiments, environments etc.), and (ii) adapted to downstream task of choice (e.g.. different objectives, goal specification types, behaviors etc.). We specifically consider the problem of visual navigation, where the robot must navigate its environment solely using egocentric visual observations. A general pre-trained robot navigation model should enable a wide range of navigation applications, readily allow fine-tuning to downstream tasks, and generalize to a broad range of environments and robotic platforms. Such a model should provide a broadly capable navigation policy on top of which applications specific domains can be constructed, giving a base level of generalization and capabilities new robotic platforms in zero shot that can be further improved after fine-tuning with small amount of data.
To this end, we propose the Visual Navigation Transformer, or ViNT: a cross-embodiment foun- dation model for visual navigation with strong zero-shot generalization. We train ViNT to reach goals specified by camera images, providing a very general pre-training objective that can be ap- plied to almost any mobile robot dataset. We propose novel exploration algorithm for the visual navigation paradigm using a diffusion model to propose short-horizon goals, and demonstrate that it enables ViNT to navigate in novel environments. ViNT can control new robots in zero-shot, explore previously unseen environments, perform indoor mapping, and navigate kilometer-scale outdoor environments without interventions. Furthermore, we show that ViNT can be fine-tuned on a small amount of data to achieve high performance with new task specification modalities such as GPS waypoints or high-level routing commands- allowing ViNT to serve as a foundation for a wide va- riety of navigation applications. Lastly, we qualitatively analyze some emergent behaviors exhibited by ViNT, such implicit preferences and navigation around dynamic pedestrians.
We hope that ViNT represents step towards such be deployed on a wide range of robots, and on a wide range of tasks, and serve as a foundation for diverse mobile robotic applications. Model weights for ViNT as well as training and deployment code will be released on our project page:
purpose robot foundation models that can
2 Related Work
Learning from large, diverse robotic datasets has been studied for various robotic applications where data sharing across similar robots provides larger training set for more generalizable models [11- 13). However, for applications in mobile robotics, with varying dynamics and camera configurations (e.g., focal length, field of view, and extrinsics), current approaches tend to rely on learning either from small real-world datasets, which are only representative of a single robotic platform, or from simulation, with paired robot and environment models to transfer learned policies [14-16]. Instead, our paper follows the paradigm of learning navigation behavior from data collected across multiple different real-world robotic systems [17-19), while focusing on training foundation model that can be adapted for variety of downstream tasks in zero shot or with small amounts of data.
Our goal is to train an effective visual navigation policy that can solve a range of downstream tasks, such as navigating to GPS goals [20], goal images [21], and skill-conditioned driving [22] Following a large body of research in visual navigation, we use combination of topological graphs for main- taining a spatial representation of the environment and learned policies for low-level control [23- 28], and use learned heuristies to guide the robot in novel environments [15, 29]. But unlike these works, our goal is to train single generalist model rather than specialist solutions to each of these problems, showing how single high-capacity model can be adapted for diverse tasks.
The closest related works to ViNT are RT-1, 12O, and GNM [15, 19, 30J, which study broad gen- eralization across environments and embodiments for robots deployed in real-world settings. While RT-1 demonstrates impressive performance in following diverse instructions, our focus is on adapt- ing single model across many robots to solve different tasks, by fine-tuning with small amounts of data. 120 and related efforts [15, 16] show impressive transfer from simulation real-world envi-

---

Observations
Posbandd
EtfciantNet BD
Temporal Distance F- Continnn
Sel Alllen
3M4 facorstor
Net
Transformer
Normalized Actions
Observation
Tookee 453240
25M/aassies
Actions
Figure 2: VINT Model Architecture. ViNT uses two EfficientNet encoders vh e to generate input tokens to a Transformer decoder. The resulting sequence is concatenated and passed through fully-connected network to predict (temporal) distance the goal well sequence of H 5 future actions.
ronments, but we emphasize that our aim is orthogonal to the specific choice of algorithm: we focus on learning capable navigation policy that can be efficiently adapted solve different downstream tasks. GNM [19] demonstrates policy learning from heterogeneous RGB datasets, but focuses the singular task of reaching image goals in the zero-shot setting. Instead, ViNT trains single generalist policy with emphasis on adaptation to new embodiments and tasks in downstream applications, though it can also be used zero-shot to great effect (Sec. 6.1).
The ViNT Model
Our model is trained for image-goal navigation, providing general navigational capabilities that can then either be utilized directly, or serve as pre-trained foundation for downstream fine-tuning with other task specifications.
the image-goal navigation task, the robot tasked with navigating to a subgoal specified by an image observation (i.e., the robot's observation at the goal). Unlike alternative mechanisms for goal specification such as PointGoal [31], GPS navigation, or semantic objectives [32], model can be trained for image-goal navigation with minimal assumptions, uti- ground-truth localization,
lizing any data that contains videos and actions, without requirements semantic labels, or other metadata. This makes it practical to train on large and diverse dataset sourced from many different robots, facilitating broad generalization.
ViNT takes as input current and past visual observations ot- Pit and a subgoal image os, and predicts (i) the number of time steps needed to reach the subgoal (the dynamical distance), and (ii) sequence with length H of future actions leading towards the subgoal. Our 31M- M-parameter model, ViNT, is built on the Transformer architecture [33] and is optimized for: (i) fast and efficient inference on eesource-constraingg robots, and (ii) the ability to prompt and fine-tune for downstream tasks. We initialize all networks from scratch and train them end-to-end with a maximum likelihood objective. The model architecture is summarized in Figure 2, and described in detail Appendix A.
Tokenization: The ViNT architecture (Fig. 2) first tokenizes its inputs into embedding of size dmodel 512. ViNT independently tokenizes current and P past visual observations by encod- ing them with EfficientNet-BO [34] model, which takes 85 x 64 x 3 images as input and outputs a flattened feature vector (o,) from the final convolutional layer [30].
We found that naïvely extracting features from the goal image d(o,)) using an Effi cientNet encoder ø led to poor performance, often ignoring the goal entirely (see Appendix A). We hypothesize that effective features for image-based goal-reaching tasks are often relative, encoding the difference between the current observation and the goal rather than an absolute representation of the goal itself. Hence, we use separate goal fusion encoder Φ(0,, o2) to jointly encode the current and goal observations. We stack the two images along their channel dimensions, pass them through a second EfficientNet--OO encoder, and flatten to obtain the goal token.
Goal fusion:
Transformer:
The + observation and goal tokens are combined with positional encoding and fed into a Transformer backbone f. We decoder-only Transformer with n 4 multi-hee heati- attention blocks, each with nh 4 heads and dep 2048 hidden units.
3

---

Current Observation
Ungrounded Subgoals
Grounded Subgoals
Goal
Diffusion
VINT
Heuristic Beauisttc-Baseed Topalagical Planrer
Figure 3: Long-horizon navigation in unseen environments with ViNT. We use physical search with a topological graph-based planner to explore the environment. An maage-to-image diffusion model proposes diverse exploration targets which are spatially grounded using ViNT (yellow), and scored using goal-directed heuristic Subgoals are added to topological graph M executed using the ViNT policy.
Training data: We train ViNT using large-scale dataset of heterogeneous navigation trajectories from diverse set of environments and robotic platforms with varying dynamics, camera parame- ters, and behaviors. The training dataset contains over 100 hours of real-world trajectories sourced entirely from existing datasets, spanning distinct robotic platforms with varying speeds and dy- namics. For more details about the dataset, see Appendix C.
Embodimennt-ggnssiee action space: To effectively train single model across robots of varying sizes, speeds, and dynamies, we follow Shah et al. [19] and choose embodiment-agnun -agnossti action space for ViNT. To abstract away low-level control, ViNT uses relative waypoints as its action space à; to account for the large variance in speeds and sizes of the robots, we normalize these waypoints by scaling them according to the robot's top speed. During deployment, a robot-specifc controller is used to un-normalize and track these waypoints using low-level control.
4 Long-Horizon Navigation with ViNT
While the goal-conditioned policy learned by ViNT captures a general understanding of navigational affordances and obstacles, it has limited applicability its own. Many practical tasks either not defined by goal images, or require much longer horizon than what ViNT directly supports. We apply ViNT to several downstream applications by combining it with an episodic memory formed by a topological graph, which provides short-horizon subgoals for reaching faraway locations. In previously unseen environments, we can further augment this graph-based planner with exploratory subgoal proposals, which can drive ViNT explore new environment and discover a path to the goal. We consider multiple such proposal mechanisms and find that maximum performance is attained by an image diffusion model that samples diverse future subgoal candidates conditioned on the current observation.
These subgoals are scored with a goal-directed heuristic to identify the best subgoal that makes progress towards the goal using a process akin to physical A search [29]. Past observations and unexplored frontiers are stored nodes topological graph, with their connectivity determined by the distances predicted by ViNT. During exploration, build this topological graph on-the-fly the robot explores the environment. During later deployments it may be used for discovering shortcuts to arbitrary goals in the environment. We first describe the high-level algorithm that plans on top of subgoal candidates, and then discuss the process for obtaining these subgoal candidates.
4.1 High-Level Planning and Exploration
Let's assume that we have access to subgoal candidates Os, E S available ViNT for planning. We incorporate these subgoal candidates into an exploration framework for goal-directed exploration in novel environments, where the user provides a high-level goal G, which may be arbitrarily far away. We largely follow prior work [29], but swap out the learned models with ViNT and the diffusion model. We summarize the system here, and provide more complete discussion in Appendix B.3.
We construct a topological graph M online to act as episodic memory, with each node as an in- dividual subgoal observation and edges representing paths between two subgoals, added when the path is taken by the robot, the model predicts subgoal be reachable from another node. We frame goal-directed exploration a search problem, where the robot incrementally builds M while searching for the goal. To guide search towards the goal, the robot uses goal-directed heuristic

---

h(,, Os., C) to score subgoal candidates according to their likelihood of reaching the goal, given additional context C - for example, a floor plan or satellite imagery [15, 29). This heuristic may be geometric (e.g., Euclidean distance) or learned (see Appendix B.3).
During deployment in a new environment, the robot uses the diffusion model to generate subgoal candidates S from ot, spatially grounds them using ViNT, and scores them using the goal-directed heuristic h(.). The robot then selects the best subgoal 0₂- according to this heuristic using -lkkiee planner, adds it to M, and drives towards it using ViNT (Figure 3). During subsequent deployments in the same environment, the robot can use M to discover shortcuts to arbitrary goals in the environ- ment. Please see Appendix B.3 for more details about the planner and heuristics. In our experiments, we consider two candidate search heuristics: geometric heuristic based on positions of the robot and the goal, and a learned heuristic based additional context in the form of a satellite image.
4.2 Subgoal Generation with Diffusion
The physical search algorithm presented above relies on the ability to propose subgoal candidates that are both diverse and reachable from the current observation of the robot o,. This amounts to sampling from a high-dimensional, multimodal distribution of RGB images.
To do so, we train a conditional generative model g(ox, lo.) on the ViNT training data. Specifically, apply an image-to- -imae diffusion model [35, 36], generative model class that is well-suited for producing diverse samples over high- nigh-dimensional spaces such as RGB images. We train the model using randomly-sampled future observations from trajectories in the ViNT dataset (Appendix B.2), and sample K subgoal candidates S = (81,..., 88) from the model at inference time.
However, these subgoal generations are not spatially grounded: they do not include an actionable re- lationship to o₁. We ground these candidates by using ViNT to compute temporal distances d(s;, oi) and action rollouts a(s,, ot), yielding set of grounded subgoals as in Fig. 14. While the samples generated by the diffusion model do not necessarily match any real observation (see Figure 3), they preserve sufficient relative features from or to be plausible, and we find that ViNT generalizes well to generated subgoals. We further study the behavior of this diffusion model in Section F.
ViNT: A Foundation Model For Downstream Tasks
Beyond its core functionality as an image goal-conditioned model, we show that the strong naviga- tional priors learned by ViNT can be adapted to a variety of downstream tasks, beyond image goals, by fine-tuning part or all of the model in novel environments or with new modalities of data.
Full model fine-tuning: While ViNT demonstrates strong zero-shot general- ization new environments and robots, we can further improve on-task perfor- mance by fine-tuning the entire model with the same objective but using on-task data. This allows ViNT to quickly learn new skills, forming continually im- proving model. ViNT can master new environments and embodiments with as little as hour of navigation data, transferring the capabilities of the original model new setting without retraining from scratch.
"turn
Adapting to modalities: While specifying image goals gives a general pre-training objective, ViNT can easily be adapted to other forms of goal- specification by learning a "soft prompt" mapping from the desired goal modality to the ViNT goal token [10]. We build on the Transformer architecture's ability to Given a
Figure 4: Adapt-
attend to multimodal inputs projected into shared token space subgoal a in a new modality (such as 2D coordinates or routing directions [22]), we train a small neural network & that maps the subgoal this shared token space as Shown in Figure 4, and replace o(o,, oe). This allows adaptation to new tasks with minimal data, while still leveraging the performance and generalization of ViNT. Appendix B.4 includes additional details.
ing ViNT to differ- ent goals using a tunable goal token.
5

---

Indoor
Outdoor
Method
Success
Success
End-to-End BC
0.72
0,44
End-to-End GCG [39]
0.61
RECON [40]
0.19
0.23
VINT-R (Random Subgoals)
0.81
VINT
0.94
1.00
VINT
VINTR R
BC
RECON
Table ViNT paired with our physical search algorithm consistently outperforms baselines for the task of undirected goal-rcaching in indoor and outdoor environments (left). By effectively planning diffusion subgoal proposals, ViNT able to find an efficient path to the goal. Other baselines struggle to explore large indoor environments, shown by trajectories overlaid on an indoor floor plan (right).
Real-world Evaluation
We deployed our VINT foundation model on five distinct robotic platforms, including a drone, a quadruped, and two other novel robots which are not present in the training data. We designed our experiments answer the following questions:
Q1. Can ViNT efficiently explore previously unseen environments and incorporate heuristics?
Q2. Does ViNT generalize to novel robots, environments, and obstacles?
Q3. Can ViNT be fine-tuned to improve performance in out-of-distribution env anvironments?
Q4. Can the ViNT policy be adapted to handle new task specification and modalities?
Please Appendix D for more details platforms used in the training data and in evaluation.
6.1 Navigation Performance
Towards understanding Q1, we deploy our full graph-based navigation pipeline (Section 4.1) in variety challenging indoor and outdoor environments, previously unseen in the training data. We evaluate the performance of ViNT on two challenging tasks: (i) coverage exploration, where the objective is maximally explore an environment in search of goal whose location is unknown, and (ii) guided exploration, where the objective is reach a goal using contextual information such GPS coordinates or satellite image (see Figure 11 for task examples). We compare ViNT to a variety of baselines, including end-to-end policies trained with imitation or RL [15, 39], prior graph- based approach using VIB for exploration [40], and an ablation of ViNT that samples random images from the training set to use as goals rather than generating subgoals with a diffusion model. See Appendix E. for details about the experimental setup.
For the coverage exploration task, the agent is placed in unknown environment and tasked with exploring the environment maximally in search of the goal without any additional cues. Table summarizes the success rates for this task in indoor and outdoor environments. We find that, while end-to-end baselines avoid collisions with their surroundings, they fail to explore new areas and often get stuck in small areas of the environment. Graph- based methods avoid this trap by explic- itly reasoning about coverage with the search objective, leading to a high success rate for ViNT. Qualitative analysis (Table 1-right) shows that planning over the diverse subgoals proposed using diffusion leads more efficient paths, whereas other baselines take winding paths while exploring. Figure II illustrates the egocentric rollouts of the coverage exploration task in challenging indoor environments. ViNT-R performs respectably despite the lack of valid subgoal proposals.
This observation extends to the position-guider navigation task (Table 9), where the robots are tasked with reaching a 2D goal position in a previously unseen environment. The robots have ac- cess to onboard wheel odometry (indoors), GPS coordinates (outdoors), or passive satellite imagery (outdoors), to track their position and use goal-directed heuristic. Compared to baseline of the previous state of the art [29], we find that the various sub-goal predictions from the diffusion model

---

paired with the graph-baser based scoring scheme lead to higher success rate and a greater distance trav- eled without collisions. ViNT is also more effective at avoiding collisions in crowded indoor spaces, and more efficient at reaching goals in outdoor spaces (captured by the SPL metric), owing to the implicit affordances and preferences learned by the large-scale pre-training (see further analysis in Section F. ViNT also requires fewer interventions, observed by the larger distance before observed collisions. Figure 12 illustrates a rollout of physical search in outdoor environment with ViNT using satellite image as context (also see Figure 11).
6.2 Zero-Shot Generalization: a Single Policy to Drive Any Robot
Towards answering Q2, we deploy the same pre-trained ViNT policy on four distinct robotic plat- forms without any fine-tuning for the task of undirected exploration. We report the maximum dis- placement of the robot (in meters) from its starting position, without interventions, reaching arbitrarily distant goals in complex environments in Table 2. Most notably, ViNT success- fully generalizes zero-shot to control a Go quadruped, which does not appear during training.
proxy for
We compare ViNT trained across all the com- bined datasets and robots the best single-robot baseline model trained using data only from the target environment
Model
LoCoBot
Vizbot Jackal
Single-Robot
40
12
40
184
GNM [19]
20
427
as well as the GNM model [19] trained on all datasets. We observe that policies trained across robot embodiments can not only match, but also outperform, single-robot mod- els across all the embodiments we studied. We also find that the larger capacity of ViNT leads to improved generalization compared to the smaller GNM model, especially on robots that do not appear in the training dataset (e.g., Go 1). Crucially, we also find that ViNT demonstrates positive transfer for in-domain robots (Vizbot), greatly outper- forming a specialist model trained on only the target robot and setting, an emergent phenomenon not present in smaller models. This indicates that the model generalizes between tasks to improve performance, a key property of a foundation model.
VINT
120
45
110
438
Table 2: In coverage tasks, ViNT drives dif- ferent robots for 100s of meters (reported maxi- mum displacement without intervention). beating lower-capacity models (GNM) and specialist mod- els trained on single robot dataset.
6.3 Broader Generalization via Fine-Tuning
To answer Q3, we consider the problem of fine-tuning ViNT in the low data regime. In this setting, the entire ViNT model is fine-tuned end-to-end with a reduced learning rate of 1 x 10-4 over nep = epochs (Section 5). We assume access to small amount of on-task data (at most hours, with successful results in 1-2 hours of data), and study the the efficiency of learning from subsets of this data using ViNT. We study fine-tuning for the task of autonomous driving in the CARLA simulator for two reasons: (i) the simulated CARLA environment is perceptually distinct from the real-world data used to train ViNT (Fig. 13), and (ii) the on-road driving task requires very specific semantic behavior, i.e., driving in a lane and making smooth turns, that is not present in our real-world training data. We show that ViNT can be fine-tuned on a small amount of data (under hour) to achieve strong performance by effectively leveraging the navigational priors encoded the model.
Table 3 summarizes findings. We report fractional progress towards the goal "success", and the fraction of trajectories where the agent drives within the driving lane as "in lane". While pre- trained visual representations significantly improve task performance over a policy trained entirely from scratch, we observe that the learned policies suffer from frequent collisions and poor perfor- mance. GNM [19] outperforms these baselines due to its strong navigation prior, but the lower- capacity model is unable to generalize fully to the task. ViNT, on the other hand, is able to achieve strong performance, achieving substantially higher success rate than other baselines. Sweeping over fine-tuning dataset size (Table 3-right) shows that ViNT achieves strong performance with under hour of fine-tuning data, demonstrating its ability to efficiently generalize to new environments.
7

---

Images
Positions
Routing Success
VNT
Method
Success
In Lane
Success
Scratch
0.45
0.74
0.79
0,43
Succeess Rttte 39
ImageNet
0.22
0.71
0.59
0,45
SimCLR [7]
0.21
0.63
0.70
0.64
VC-1 [41]
0.19
0.65
0.49
0.38
GNM [19]
0.49
0.66
0.45
0.49
VINT
0.82
0.82
0.89
0.72
Table 3: Left: ViNT can be fine-tuned end-to-end (Images) adapted downstream tasks (Positions and Routing), and outperforms training from scratch and other pre-training methods. Right: VINT can transfer navigational affordances novel tasks (40% success without fine-tuning). and efficiently masters the task (80% success) with less than hour of fine-tuning data.
6.4 Adapting ViNT to Downstream Tasks
To evaluate Q4, we investigate whether ViNT can serve as a foundation model for a broader range of downstream tasks by considering goal modalities beyond subgoal images (see Section 6.4). We consider the same CARLA driving task but with two different high-level planners: (i) a position- based planner that commands sequence of GPS waypoints, and (ii) routing planner with similar functionality to Google Maps that commands high-level navigation directions (left/right/straightt) to the policy [22]. We compare the pre-trained navigational priors learned by ViNT to the baselines discussed earlier, corresponding to pre-trained visual representations and policies, each adapted to the downstream task using the same on-task data (see Appendix E.3 for more details).
Table summarizes our results for the two tasks. We again find that general pre-trained visual rep- resentations, such as ImageNet VC-1, are not sufficient extract navigational affordances for challenging downstream tasks, suggesting that effective generalization requires more than general visual representations [41, 42]. We also find that unlike fine-tuning, GNM struggles with the adap- tation task, suggesting that the architecture and increased capacity of ViNT are essential for broad generalization and adaptation.
7 Discussion
We presented ViNT, a robot foundation model that is trained for generic image-goal navigation task on diverse data from many different robots, and can then support different navigation functionalities. ViNT can be deployed for oogghoorzzon navigation in combination with a high-level planner, explore new environments with goals proposed by a diffusion model, be fine-tuned new domains (such as autonomous driving), and be adapted to new task specification methods, such turn-by cum--y--tunn routing commands. Our results show that ViNT can successfully generalize across robots and environments, outperforms prior navigational models, can be efficiently tne-tuned to new domains and tasks, and shows promising emergent behaviors such as navigating through dynamic pedestrians.
Limitations and Future Work
As with many large-scale models, ViNT carries heavier computational burden at inference time, which present challenge for power-constrained platforms such quadcopters. While our de- sign aims to enable efficient inference, our Transformer- model is still significantly costlier to run at deployment time than simpler feedforward convolutional networks. Additionally, although ViNT generalizes effectively across robots in our experiments, it assumes a degree of structural sim- ilarity. More broadly, while ViNT illustrates the promise general-purpose purpose and broadly reusable navigational foundation model, we believe that the most exciting developments for general-purpose cross-robot models are still ahead: larger and larger multi-robot datasets are assembled, perhaps will see even broader generalization and more flexible specification with increasingly powerful and general-purpont robotic models. We hope that ViNT represents step in this direction.

---

Acknowledgments
This research was partly supported by ARL DCIST CRA W911NF-17-2-0181, ARO W911NF-21- 1-0097, IIS-2150826, and compute from Google TPU Research Cloud, NVIDIA, NSF CloudBank, and the Berkeley Research Computing program. The authors would like thank Yunhao Cao, Seung-Hyun Kwon, Hrish Leen, Laura Smith, Medini Tolpadi, and Christopher Yu, for help with setting up experiments; Ted Xiao, for help with reproducing RT-1; Chethan Bhateja, Devendra Singh Chaplot, Kuan Fang, Adrien Gaidon, Dibya Ghosh, Saurabh Gupta, Haresh Karnan, Vijay Kumar, Hrish Leen, Fangchen Liu, Arjun Majumdar, Carlos Nieto, Aravind Rajeswaran, Ethan Stump, Jie Tan, Joanne Truong, and Tingnan Zhang, for useful discussions and feedback during various stages of this research. The authors would also like to thank Byron Boots, Gregory Kahn, Haresh Karnan, Xiangyun Meng, and Xuesu Xiao, for their help in aggregating the dataset used for training ViNT.
References
[1] A. Radford, K. Narasimhan, T. Salimans, Sutskever, et al. Improving language understanding
by generative pre-training. 2018. 1
[2] J. Deng, W. Dong, R. Socher, L..J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierar- chical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
[3] Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
[4] K. Grauman et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 1
[5] A. Ramesh, Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022. 1
[6] M. Chen et al. Evaluating large language models trained on code. arXiv:2107.03374, 2021.
arXiv preprint
[7] T. Chen, S. Korbllihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. International Conference on Machine Learning (ICML), 2020. 1, 8, 20
[8] D. Shah, B. Osiński, B. Ichter, and S. Levine. Robotic navigation with large pre- preitrained models of language, vision, and action. In Conference on Robot Learning (CoRL, 2022.
[9] X. Liu, Y. Li, C. Liang. and X. Li. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[10] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. 1,5
[11] C. Devin, Gupta, T. Darrell, Abbeel, and S. Levine. Learning modular neural network policies for multi-task and multi-robot transfer. In 2017 International Conference on Robotics and Automation (ICRA), 2017. 2
[12] S. Dasari, F. Ebert, et al. Robonet: Large-scale multi-robot learning. In Conference on Robot Learning (CoRL), 2020.
[13] Yu, H. Chen, X. Wang, W. Xian, Y. Chen, F. Liu, Madhavan, and T. Darrell. Bdd100k:
A diverse driving dataset for heterogeneous multitask learning. In Conference Computer Vision and Pattern Recognition (CVPR), June 2020. 2, 18

---

[14] P. Anderson, A. Shrivastava, J. Truong, A. Majumdar, D. Parikh, D. Batra, and S. Lee. Sim- to-real transfer for vision-and-language navigation. arXiv preprint arXiv:201 I. 103807, 2020.
2
[15] Truong, A. Zitkovich, Chernova, Batra, T. Zhang, Tan, and W. Yu. Indoorsim-to- outdoorreal: Learning navigate outdoors without any outdoor experience. arXiv preprint arXiv:2305.01098, 2023. 2,5,
[16] A. Kadian, J. Truong, A. Gokaslan, A. Clegg. E. Wijmans, S. Lee, M. Savva, S. Chernova, and
D. Batra. Sim2Real Predictivity: Does Evaluation in Simulation Predict Real-World Perfor- mance? IEEE Robotics and Automation Letters, 2020.
[17] A. Loquercio, A. Maqueda, C. R. del Blanco, and D. Scaramuzza. Dronet: Learning to fly by driving. IEEE Robotics and Automation Letters, 2018.
[18] N. Hirose, D. Shah, A. Sridhar, and S. Levine. ExAug: Robot-Conditioned Navigation Poli- cies via Geometric Experience Augmentation. In International Conference on Robotics and Automation (ICRA), 2023.
[19] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. GNM: General Navigation Model Drive Any Robot. International Conference on Robotics and Automation (ICRA), 2023. 2, 4, 7, 19, 21
[20] Manolis Savva*, Abhishek Kadian Oleksandr Maksymets*, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embod- ied AI Research. In International Conference on Computer Vision (ICCV), 2019. 2
[21] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. International Confer- ence on Robotics and Automation (ICRA), 2017.
[22] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-End Driving Via Conditional Imitation Learning. In International Conference Robotics and Automation (ICRA), 2018. 2, 5, 8, 16, 20
[23] N. Savinov, A. Dosovitskiy, and Koltun. Cemi-Parametric Topological Memory for Naviga- tion. In International Conference on Learning Representations (ICLR), 2018. 2
[24] Bruce, N. Sunderhauf, Mirowski, R. Hadsell, and M. Milford. Learning deployable nav- igation policies kilometer scale from single traversal. In A. Billard, A. Dragan, J. Peters, and J. Morimoto, editors, 2nd Conference Robot Learning, 2018.
[25] A. Faust, K. Oslund, O. Ramirez, A. Francis, L. Tapia, M. Fiser, and J. Davidson. Pmm-rl: Long-range robotic navigation tasks by combining reinforcement learning and sampling-basee planning. In 2018 International Conference on Robotics and Automation (ICRA), 2018.
[26] X. Meng, N. Ratliff, Y. Xiang, and D. Fox. Scaling Local Control Large-Scale Topological Navigation. In International Conference on Robotics and Automation (ICRA), 2020.
[27] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. ViNG: Learning Open- World Navigation with Visual Goals. In International Conference on Robotics and Automation (ICRA), 2021.
[28] N. Hirose, F. Xia, R. Martin-Martin, A. Sadeghian, and Savarese. Deep Visual MPC-Policy Learning for Navigation, IEEE Robotics and Automation Letters, 2019. 2, 18
[29] D. Shah and S. Levine. ViKiNG: Dision-Based Kilometer-Scall Navigation with Geographic Hints. In Robotics: Science and Systems (RSS), 2022. 2, 4, 5, 6, 15, 16, 18, 19, 22
10

---

[30] A. Brohan et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2023. 2, 13, 14
arXiv preprini [31] Anderson et al. On Evaluation of Embodied Navigation Agents.
arXiv:1807.06757, 2018.
[32] S. Wani, S. Patel, U. Jain, A. Chang, and M. Savva. Multion: Benchmarking semantic map memory using multi-object navigation. In Neural Information Processing Systems (NeurIPS), 2020.
[33] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and Polosukhin. Attention all you need. In I. Guyon, U. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://procceeeee neurips ppaper_filee/ppper///aaa 3fae223547aas1111b0ccccccaaaaa aefap.r..f..... 3
[34] M. Tan and Q. Le. EfficientNer: Rethinking model scaling for convolutional neural networks. In K. Chaudhuri and Salakhutdinov, editors, Proceedings of the 36th International Con- ference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105-61 14. PMLR, 09-15 Jun 2019. URL https: htt.https:/pproceeeeeee http///cpeednga..//////// .ml...g..mm......mmmmmm......m.m www.mmlr....///ee.m press/v97/ tan19a. html. 3, 13
[35] Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. In Neural Information Processing Systems, 2020. 5, 14
[36] C. Saharia, W. Chan, H. Chang, C. Lee, J. Ho, T. Salimans, D. Fleet, and M. Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPHI 2022 Conference Proceedings, pages 1-10, 2022. 5. 14
[37] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.0333 2023. 5
[38] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Y. Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint
arXiv:2210.03094. 2023.
[39] G. Kahn, A. Villaflor, B. Ding, P. Abbeel, and S. Levine. Self-Supervised Deep RL with Gen- eralized Computation Graphs for Robot Navigation. In International Conference on Robotics and Automation (ICRA), 2018. 6, 18, 19
[40] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine. Rapid exploration for open-world navi- gation with latent goal models. In Conference on Robot Learning (CoRL), 2021. 6, 18, 19
[41] A. Majumdar, K. Yadav, S. Arnaud, Y. Ma, C. Chen, Silwal, A. Jain, V.-P. Berges, Abbeel, J. Malik, D. Batra, Y. Lin, O. Maksymets, A. Rajeswaram, and F. Meier. Where in the search for artificial visual cortex for embodied intelligence? arXiv preprint arXiv:2303.18244 2023. 8, 21
[42] Parisi, A. Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. arXiv preprint arXiv:2203.03580 2022.
[43] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019. 15
[44] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020. 15
arXiv preprint

---

[45] P. von Platen, S. Patil, A. Lozhkov, P. Cuenca, N. Lambert, K. Rasul, M. Davaadorj, and T. Wolf. Diffusers: State-of-the-art diffusion models. https: bttps://github. www.taww.....gggggggaaaea cow/huggingface/// diffusers, 2022. 14
[46] D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Neural Information Processing Systems (NeurIPS), 2021. 14
[47] Ho and T. Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022. 15
[48] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, Karagol Ayan, T. Salimans, al. Photorealistie text-to-image diffusion models with deep language understanding, Neural Information Processing Systems (NeurIPS), 2022. 15
[49] A. v. d. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.088. 1807.03748, 2018. 16
[50] H. Karnan et al. Socially CompliAnt Navigation Dataset (SCAND): A Large-Scale Dataset Of Demonstrations For Social Navigation. IEEE Robotics and Automation Letters, 2022. 18
[51] A. Shaban, X. Meng, Lee, B. Boots, and D. Fox. Semantic terrain classification for off-road autonomous driving. In Conference Robot Learning (CoRL), 2022. 18
[52] S. Triest et al. TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models.
In International Conference on Robotics and Automation (ICRA), 2022. 18
[53] A. Agha et al. Nebula: Quest for robotic autonomy in challenging environments; team costar at the darpa subterranean challenge. arXiv preprint arXiv:2103. 2021. 18
[54] N. Hirose, D. Shah, A. Sridhar, and S. Levine. Sacson: Scalable autonomous control for social navigation. arXiv preprint arXiv:2306.01877 2023. 18
[55] Niwa, S. Taguchi, and Hirose. Spatio-temporal graph localization networks for image- based navigation. In International Conference on Intelligent Robots and Systems (IROS), 2022.
18
[56] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[57] I. Radosavovic, T. Xiao, S. James, P. Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training In Conference on Robot Learning (CoRL), 2022. 21
12

---

Appendix
A
VINT Model Architecture
Table 4 shows the ViNT model architecture in detail. We use all 18 layers of the EfficientNet-BO convolutional encoder [34], initialized from scratch, to tokenize the observation and subgoal images into 512-dimensional embeddings each. We utilize an observation encoder to tokenize the past and current observation images and a joint observation and goal encoder to tokenize the subgoal image fused with the current observation image channel-wise. For tokenizing the joint observation and subgoal token, images On and O₂ concatenated along their channel dimension, yielding 6 x 85 x 64 tensor per training data point.
Layer
Input [Dimensions]
Output [Dimensions]
Layer Details
O, [64, 85, 3] If [64, 85, 6]
I [64, 85, 6]
Concatenate observations and goal
2
Eú [1, 1000]
Goal EfficientNet encoder
Oct-P [P+I, 64, 85
Feg-r [P+1, 1000]
Context EfficientNet encoder
E; [1, 1000]
E7 [12]
Goal embedding compression
5
Ext-P [P+1, 1000]
Ees-r [P+1,
Context embedding compression Concatenate
6
Ei.- P
EY [12]
S (122
7
S (122,511)
$ [1]
Feed into Transformer f Predict temporal distance Predict future actions à
$ [1, 32]
d
9
$ [1, 321
à, [1, T,
Table 4: Architectural Details of ViNT The inputs to the model are RGB images Ou P E [0, 1 Px3x86 and o, E [o, 1j1xx5xx66 representing the current, past, and goal images. We seek to predict H future actions à and the temporal distance d.
A.1 Goal-Conditioning Architectures
We considered different mechanisms for conditioning ViNT with features from the subgoal image, as illustrated in Figure 5.
1. Late Fusion: Extract the observation and goal features independently and fuse them in the multi-head attention layers. To achieve this effect, we avoid any channel-win -aine con- catenation between any of the observation and goal images before inputting them into the model.
2. Early Fusion: Jointly extract observation (context) and goal features and fuse the observa- tion and goal features before we tokenize them. We achieve this by concatenating the goal image with every observation image along the channel dimension. We remove the goal token in this setup since information about the goal is already in every observation token.
3. FiLM (RT-1): Following the FiLM+EfficieetNet encoder (Brohan et al. [30]), encode each observation image separately. For conditioning on visual goals, we replace the "Universal Sentence Encoder" with an EfficientNet encoder. We remove the goal token in this setup since information about the goal already in every observation token.
Our observations are summarized in Table 5. While FiLM works well for language, we found that training was unstable for image-based navigation tasks. Instead, we directly encode each observa- tion independently and pass them to a Transformer. Ideally, the goal would be encoded separately and then combined with the observations in the Transformer layers, allowing the entire goal encoder later be swapped out for different goal modalities. Unfortunately, (which we term "late fusion", as the goal and observations are not fused until after encoding them) performs poorly: in baage-based navigation, it is the relative features between the observation and
found that this approach
13

---

FILM
Multi-Headde Attention
Mutt-Headed Attention
Mu Id- U--Headed Attention
Our Model
Late Fusion
Early Fusion
FILM (RT-)
Figure 5: Different goal--ondiiioning architectures considered for VINT.
Method
Performance
Adaptation
Late Fusion
X
V
Farly Fusion
X
FiL.M (RT-1) [30]
ViNT
Table 5: Comparing merits (V) and demerits (X) of different goal-conditioning architectures. While "Early Fusion" works the best for the core navigation task, it does not support downstream adaptation (Section 5). "Late Fusion" ideal adaptation, but does not perform well our tasks, Our goal fusion architecture able to closely match the performance of early fusion, while also supporting adaptation.
goal images that are important, rather than absolute goal features. An "early fusion" architecture would fuse the goal image with all the past and current observation images immediately, which allows for learning joint features between the goal image and current state. However, this architec- ture is inflexible as the observation encoder would have to be learned entirely from scratch when adapting to new goal modality. ViNT avoids this issue by using two distinct types of encoders: an observation-only encoder used to tokenize each observation image, and joint observation and goal encoder that should extract relative goal features. This latter encoder can be replaced to allow alternative goal specifications in downstream tasks, as described in Appendix B.4. Specifically, we adapt new tasks by learning the final token conditioned on the new task goal information place of the joint observation/goal encoder.
B Implementation Details
B.1 Training VINT
See Table for detailed list of hyperparameters for training the ViNT foundation model.
B.2 Subgoal Diffusion
For generating subgoals, we use an image-to-image diffusion model. takes an image o, as input and produces samples from g(os. o,). where os, are candidate subgoal images reachable from o,- To produce training pairs for the diffusion model, first select o, uniformly at random from the training data and then select Os, fall between 5 and 20 timesteps in the future from ot-
Following Saharia et al. [36], we implement image conditioning as simple channel-wise concatena- tion to the U-Net input. We use the Flax U-Net implementation from the diffusers library [45] with textual cross-attention removed since we do not condition on text inputs.
We use the continuous-time diffusion formulation from Kingma et al. [46] with fixed linear noise schedule rather than a learned one. Also unlike Kingma et al. [46], we use the unweighted training objective, called Limmle in Ho et al. [35, Equation 14] and Kingma et al. [46, Appendix KJ. We
We used variety of workstations equipped with different GPU configurations over the course of this research. including 4x9090, 3x Titan Xp. 4xP100, 8x 1080Ti, 8x V100, and 8x With the model archi- tecture fixed, the batch size and training time varies significantly across these devices, and the entry in Table 6 representative of our most common training configuration.
14

---

Hyperparameter
Value
Hyperparameter
Value
VINT Model
Diffusion Training
# Parameters
31M
Dropout
0.1
RGB Resolution
x 64
Batch Size
128
Encoder
EfficientNee-B0
Optimizer
AdamW W
Token Dimension
512
Warmup Steps
1000
Attn. hidden dim.
2048
Learning Rate
le-4
# Attention Layers n
4
LR Schedule
Cosine
# Attention Heads n
Adam 31
0.95
Temporal Context P
5
Adam $2
0.999
Prediction Horizon H
5
Adam
le-8
MLP layers
(256, 128, 64, 32)
Weight Decay
0.001
VINT Training
EMA Inv. Gamma
1.0
# Epochs Rep
30
EMA Power
0.75
Batch Size
300
EMA Max Decay
0.9999
Learning Rate
5 10-4
CFG Mask Proportion
0.2
Optimizer
Adam W [43]
Train Steps
250,000
Warmup Epochs
4
Training Time
hours
LR Schedule
Cosine 10
Compute Resources
v4-8 TPU board
Scheduler Period
Diffusion Sampling
Compute Resources
8xV100
Sampler
DDIM [44]
Training Time
hours
DDIM r)
0.0
Fine-tuning LR
1 10
Sampling Steps
200
Guidance Weight
1.0
Diffusion Model
Other
# Parameters
318M
Maximum distance
20
Resolution
128 x 128 4 32, 16, 8 2
Distance tradeoff
0.01
# Up/Down Blocks
Attn. Resolutions
Layers per Block
Attn. Head Dim
Channels
(128, 128, 256, 512, 640) continuous time linear
Diffusion Type
Noise Schedule
Table 6: Hyperparameters for training ViNT and the diffusion model.
employ classifier-fres free guidance [47] and find that it helps produce subgoals with better visual fidelity, which is consistent with prior work [48].
B.3 Long- Horizon Physical Search via Topological Graphs
As in Shah and Levine [29], we implement physical search similarly to standard A algorithm, by keeping track of an open set Ω of possible unvisited subgoals (generated by our diffusion model) and following Alg. 1.
Nodes visited according a costing function (s) that depends on the distance from the current state o, to the parent node s (measured along the graph), the predicted distance from to 8, and a heuristic function h (similar to that of A*) providing long-horizon navigation hints:
fs)) dm(or, s (s ,8) h(s,, G,C)
In general, the heuristic can be any function providing notion of distance between a subgoal and the long-horizon goal G, optionally with some context C. For experiments, considered three heuristics to demonstrate the flcxibility of our approach:
Coverage exploration: We have no long-horizon guidance for coverage exploration, and thus, use h(s) = 0.
Position-guided: For long- horizoo GPS goals (outdoors) and 2D position goals (indoors), we use Euclidean distance h(s) ||s G'|.
15

---

Inpul Image
Outpul Image
166x256
8fx512
8xx440
8Px312
16*x256
32"x128
32"x
64 128
64Px229
ResiNtt DoonserpteUUpsrppi
x28'x
128'x x3
Sell-Atlertion Downsorple/Upsaree
Reshett: Self-Attendiar
Skip Cornection (Concatenate))
Figure 6: Subgoal diffusion model U-Net architecture. Fach ResNet consists of 2 residual blocks. Downsam- pling and upsampling is done with strided convolutions.
Algorithm 1: Long-Horizon Navigation via Topological Graph
1: while goal not reached do
2: mi))(
3: < ShortestPath,M, or, 87)
4: for (s, s') P do
5: ViNT.GoToGoal(S
6: end for
7: ViNT.GoToGoalle
8: Or + Observe();
9: AddNode(MM, oz, parent: s s>);
10: Sample 8; ~ g(s()rr;;
11: Add(S2, s:);
12:
end while
Satellite-uiided: In the context-guided experiments, train learned heuristic function that uses the satellite image an input to learn a heuristic for "good" subgoals. We train a convolutional neural network on the overhead image to predict the probability that the subgoal is included on a trajectory from og to G, trained using a contrastive objective [49]. Additional information can be found in Shah and Levine [29].
B.4 Fine-tuning ViNT
CARLA fine-tuning experiments, on-task data was collected using a rule-based oracle agent,
with start and end locations sampled randomly up 900 meters apart. We collect 181 training trajectories (roughly 4 hours) in CARLA's Town 01 environment, and further 52 trajectories (1 hour) in the held-out Town 02 environment. Inspired by Codevilla al. [22], we further augment this dataset by allowing the rule-based agent to correct its position and re-center to the lane after perturbation.
Image Fine-tuning:
Architecture: We utilize the exact same architecture ViNT with changes.
Training: For fine-tuning the image-goal directed model, we utilize the same training process for VINT with a learning rate of 0.0001, AdamW optimizer, but no warmup or cosine scheduler. We do not mix any prior data for fine-tuned training.
GPS-Adaptation:
16

---

GPS Goal
Command
V
V
Figure 7: Adaptation architectures for ViNT. Left: GPS-adaptation architecture. The local coordinates of the goal are concatenated to the fixed latent z. Right: command-adag adaptation architecture, using latent selected by command label index i.
Architecture: To adapt to GPS-style goals, we cut off the goal encoder block from ViNT. We then learn a fixed tensor of size 3000 and concatenate it to the GPS- uPS-command goal in ego-centric coordinates. We then pass this into a 2-layer MI.P which outputs the prediction of the final token for the transformer. The architecture shown in Figure 7.
Training:
During training, instead of randomly sampling future images to serve as goals, we sample goals from future odometry information. Once we have a future goal coordinate for self-supervision, we convert to local coordinates and pass into our architecture, fine- tuning with the same objective as VINT. We use cosine scheduler with learning rate warmup to 0.0001 for epochs. We also sample goal points from between 1.25s and 1.75s rather than from 0.5s to 2.5s.
Command-Adaptation
Architecture: For discrete command goals, we adopt similar approach for GPS-style goals. We learn a fixed tensor for each discrete command and use the command index to select the corresponding latent to pass into a 2-layer MLP for predicting the final token. In this way, we learn a dictionary of latents, each corresponding to a distinct command. This architecture is illustrated in Figure 7.
Training: For our experiments, we use "left", "right", and "straight" as our discrete com- mands. We assume training data is not labelled with the discrete command, so we label dataset trajectories with the corresponding commands retroactively by sampling future position (as in GPS-Adaptation) and then selecting command based on its lateral devia- tion. For our experiments we bin samples with lateral coordinate greater than 0.05 as "left" or "right" and label the remaining samples as "straight"". We again use a cosine scheduler with a learning rate warmup to 0.0001 for epochs.
C Training Dataset
The ViNT training dataset contains over 100 hours of real-world navigation trajectories, sourced entirely from existing datasets. The dataset consists of a combination of tele-operated and au- tonomous navigation behaviors collected across 8 distinct robotic platforms, including 4 commer- cially available platforms (TurtleBot, Clearpath Jackal, Warthog and Spot) and several custom plat- forms (Yamaha Viking ATV, RC Car, passenger automobiles). The trajectories contain widely vary- ing robot dynamics and top speeds, ranging between 0.2 and 10m/s, operating in a diverse set of environments (e.g., office buildings, hallways, suburban, off-road trails, university campuses, etc.). All data is either publicly available, or collected by other researchers for past projects; no additional training data was collected specifically for training VINT.
17

---

Dataset
Platform
Speed
Total Hrs.
Hrs. Used
Environment
1
GoStanford [28]
TurtleBo12
0.5m/s
17h
14h
office
2
RECON [40]
Jackal
Im/s
25h
25h
off-road
CoryHall [39]
Car
1.2m/s
2h
hallways
Berkeley [29]
Jackal
2m/s
4h
4h
suburban
5
SCAND-S [50]
Spot
1.5m/s
8h
4h
sidewalks
6
SCAND-J [50]
Jackal
2m/s
1h
1h
sidewalks
Seattle [51]
Warthog
5m/s
1h
1h
off-road
TrivanDrive [52]
ATV
10m/s
7h
off-road
9
NeBula [53]
ATV
10m/s
10h
10h
off-road
10
SACSoN [54]
TurtleBo12
0.5m/s
75h
10h
office
11
BDD [13]
Car(s)
20m/s
10h
4h
on-road
Total
160h
80h
Table 7: The ViNT training datasct contains 150 hours of navigation data in challenging indoor, outdoor,
off-road environments across different robots of varying sizes, speeds, and capabilities.
Remember to mention: total size, number of robots, conversion to number of frames and so on.
D Robotic Platforms for Evaluating ViNT
Vizbot: A custom-buil robot platform inspired by the design of Niwa et al. [55], based on a Roomba. It is equipped with an off-the-shelf "CB-mounted fisheye camera.
Unitree Go 1: A commercially available quadruped robot equipped with the original forward-facing camera. There is no training data from a Go I in the training dataset. Athough SCAND includes data collected on a Boston Dynamics Spot, which also a quadrupedal robot, the two platforms practically have very different characteristics.
Clearpath Jackal UGV: A commercially available off-road platform equipped with off-the- shelf PCB-mounted fisheye camera. This system resembles the daia collection platform used for the RECON, Berkeley, and SCAND-J datasets, but has a different camera and mounting height.
LoCoBot: A popular open-source platform based on a Kobuki equipped with an off-the-shelf PCB- mounted fisheye camera. This robot is not present in the training dataset, although GS collected on similar TurtleBot2 with different spherical camera lower height.
E Evaluation Setup and Details
E.1 Navigation Performance
E.1.1 Indoor Experiments
For setting up the indoor coverage exploration experiments, we use the LoCoBot and Vizbot robotic platforms. We choose random starting point and goal an enclosed environment, and keep these locations consistent across baselines we test. For the coverage exploration task, we ensure that the environments are "enclosed" and block any glass walls and stairwells, which are beyond the capabilities of the robots. Experiments terminated when either (i) the robot is unable to reach the goal within pre-specified time limit of 10 minutes, or (ii) the robot becomes physically stuck (e.g., collides and is unable to recover).
For setting up the indoor guidance exploration experiments on the LoCoBot, we mark the start and goal locations in large office building and note their 2D positions. The goal location is conveyed to the robot the context, and is available to the search algorithm. The system onboard wheel odometry to track position.
the robot's
18

---

E.1.2 Outdoor Experiments
For the coverage exploration experiments, we follow the setup of Shah et al. [40] and use the Clearpath Jackal UGV. We choose random start and goal location in confined outdoor environ- ments and obtain a goal image observation for the robot to seek. Experiments are terminated either when (i) the robot is unable to reach the goal within a pre-specified time limit of 20 minutes, (ii) the robot collides with an obstacle in the environment.
For the guided exploration experiments, we closely follow the setup of Shah and Levine [29]. For the GPS guided experiments, the robot has access to the GPS location of the goal, in addition to a goal image. For the satellite-guided experments, the robot further has access to an overhead satellite image centered at its current location and learned heuristic funtion h.
E.1.3 Baselines
For experiments presented in Section 6.1, evaluate baselines against method.
1. End-to-End BC: A modified ViNT model with no goal token, trained end-to-ent for the task of only predicting future actions. This represents typical undirected BC baseline with similar model capacity the other baselines.
2. End-to-End GCG: A model-based algorithm that uses a predictive model to plan quence of actions that reach the goal without causing collisions [39]. Since this model requires collision labels for training, it is only trained on a subset of the training data (RE- CON, CoryHall, Berkeley) that has these labels; hence, this baseline is only evaluated outdoors.
3. RECON: variant of the physical search algorithm RECON [40], which uses latent goal model to represent reachable goals and plans over sampled subgoals explore the environ- ment in similar manner to ours. This baseline uses variational information bottleneck to sample latent subgoals, rather than diffusion model sampling subgoal images.
4. ViNT-R: An ablation our method that uses subgoals randomly sampled from the training data, instead of samples from a conditional diffusion model, as subgoal candidates.
E,2
Multi-robot Generalization Experiments
The setup for multi-robot generalization experiment is same as the coverage exploration experi- ments. The only differences are the baselines we evaluate.
E.2.1 Baselines
For experiments presented in Section 6.2, we test three baseline low-level policies each robot. Each baseline uses the graph-based exploration scheme described in Section 4.1. We the follow- ing baselines:
1. Single-Robot: We train single-dataset policy model (ViNT architecture) and diffusion model on the two largest datasets (RECON for outdoor, and SACSoN for indoor), and evaluate them on each of our robots to identify the best single-dataset model for each robot. Note that we do not have comparable magnitudes of training data of visual locomotion on the Go 1.
2. GNM: We use the pre-trained model checkpoint from the authors of GNM [19] coupled with our diffusion model (since GNM is not compatible with the exploration task) to eval- uate each robot.
3. ViNT: We use our pre-trained ViNT policy and image diffusion model (no fine-tuning) to evaluate each robot.
19

---

E.3 Fine-tuning and Adaptation
This section describes the setup and implementation details for ViNT fine-tuning and adaptation experiments in the CARLA autonomous driving simulator, as presented in Sections 6.3 and 6.4.
E.3.1 CARLA Data Collection
We collect expert trajectories with an oracle rule-based self-driving agent and gather odometry and RGB information across the trajectory at 4 Hz. These trajectories have random spawn points and random destination points up to 900 meters in length. We collect 52 trajectories in the CARLA Town 02 for held-out testing, and collect 181 trajectories in Town 01 for training. This makes for dataset size of 5 hours for the autopilot control data. Inspired by [22], we also collect short trajectories of the agent correcting back onto the right lane after drifting off course in Town 01 and Town 02 for training and testing, respectively. This data is 4 hours long, and we add it to the autopilot data for training.
E.3.2 Fine-tuning Experiments
To test the fine-tuning system which trains VINT with the same goal specification but in new do- mains, we utilize the collected test trajectories as sequences of goal images to follow. Each Town 02 test trajectory creates graph which every node timestamped odometry point corresponding an image. To evaluate a model on a test trajectory, we spawn the same start point and localize on the trajectory's 's map. We then query the image for the goal node which corresponds to node 1.5s after the current node. This goal image is sent to ViNT along with the 4Hz image context to compute a short-range trajectory. This is tracked by simple PID controller. The average progress towards the goal before collision is collected and reported across all trials. Table summarizes the results of these experiments with multiple baselines and data sizes.
E.3.3 Adaptation Experiments
To test the new tasks, we adopt a similar evaluation setup to the fine-tuning experiments, but rely on the odometry position for the selected goal node rather than the image. For positional-addapatiion, we move the goal coordinates into local frame and send to ViNT, For routing-adaptationn we de- termine the difference lateral coordinates between the current node and the goal node. We choose the current node as reference ensure open-loop experiment and to allow for pre-computation of the command signals to be sent. We then apply the same binning strategy during training using 0.05 normalized distance the boundary between "left", "right", and "straight", The control system downstream of this is identical to image fine-tuning and the experiment terminates when at the goal or when colliding. The progress towards the goal before collision is collected and averaged across all trials in Table 3.
E.3.4 Baselines
We have the following baselines for the CARLA experiments:
1. Scratch: VINT trained from scratch on the CARLA on-task dataset.
2. Pre-trained Visual Representations
(a) ImageNet: ViNT initialized with the EfficientNet-BOO weights pre-trained on Ima- geNet, other parameters initialized from scratch, and fine-tuned with the CARLA on- task dataset.
(b) SimCLR: ViNT initialized with the EfficientNet-BO weights pre-trained with Sim- CLR [7] on the training data described in Section C, other parameters initialized from scratch, and fine-tuned with the CARLA on-task dataset.
(c) VC-1: ViNT initialized with pre-trained ViT-B model checkpoint from the authors of VC-1 [41] and frozen, other parameters initialized from scratch, and fine-tuned
20

---

Figure 8: Samples from the diffusion model may be invalid subgoals, but ViNT is robust to such proposals.
with the CARLA on-task dataset. The VC-1 encoder is pre-trained on combina- tion of Ego4D, manipulation, navigation, and ImageNet images using Masked Auto- Encoding [56, 57].
3. GNM: The pre-trained embodiment-agnostic model checkpoint from the authors of GNM [19], fine-tuned with the CARLA on-task dataset. Note that GNM has 8.7M trainable parameters, compared to ViNT's 31M.
We note that the VC-1 baseline's weak per- formance in Section 6.4 may be explained by the fact that it is frozen, while all other visual encoders were free to fine-tune. This is rep- resentative of typical downstream usage [41]. Despite training on multiple, diverse datasets, the visual representation's 's general-purpost fea- tures not optimized for the navigation task, hampering zero-shot transfer to out-of-domant tasks. To provide a fair comparison of the qual- ity of pre-trained visual features, we compare this performance to ViNT-FE (a pre-trained ViNT model that has it's visual encoder frozen). ViNT-FE has an equal number of trainable pa- rameters the VC-1 baseline, and frozen visual representations (see Table 8).
Method
Images
Positions
VC-1 [41]
0.19
0.49
ViNT-FE
0.32
0.78
ViNT
0.82
0.89
Table 8: Evaluation of ViNT fine-tuning with and with- out frozen encoder, compared to general purpose visual encoder. Even when frozen, ViNT's navigation- relevant features appear to transfer more readily to out- of-distribution inputs than general-purpose features.
F Emergent Behaviors
One of the most exciting aspects of large-scale machine learning is the potential for emergent be- havior that arises from the training of large models on diverse datasets. Despite the simple self- supervised training objective used by ViNT, it shows a number of emergent behaviors, which we describe qualitatively in this section and present as examples on the project page and supplemental videos: cawaall-aviiaiionnnmooeo .otthg...tt.iu............
Implicit navigation affordances: Ideally, we would like robot foundation model to exhibit some desirable "default" behavior, while providing mechanism for downstream applications to adapt this behavior as needed. We find that VINT has this property vis-a-vis collision-avoidan piece of evidence is its behavior when provided with random subgoals from locations that are not reachable by the robot, studied quantatively via the ViNT-R baseline in Table 1. In this case, despite the subgoals being invalid and duitr--diittitution (ViNT was only trained to reach subgoals), ViNT succeeds at exploring the environment and reaches the goal 80% of the time, outperforming all baselines. This suggests that ViNT takes collision-fie free actions when provided with meaningless goals (i.e. the above "default"), while still attempting to follow reachable subgoals.
One
Indeed, although the "full" version of our method augmented with the diffusion model performs bet- ter, the subgoals generated by this model are often of low quality with many artifacts, and sometimes do not match any real reachable state (Figure 8). Nonethcless, because of this "default" behavior, ViNT is able to successfully leverage the valid subgoals, while ignoring the bad ones, and demon- strate collision-fret navigation in previously unseen environments.
21

---

Figure 10: Robustness to dynamic pedestrians. ViNT successfully navigate around crowd of dynamic pedestrians and reach the goal behind them, despite simple self-supervised training objective.
Position Guided (200ml
Coverape
Coverage (354in)
Satelitt Guided (1.24cm)
Figure ViNT accomplishes long-horizon navigation with variety objectives indoor and outdoor environments; example trajectories between start (orange) and goal (green) visualized here. Goal-reaching behavior can be achieved with a goal-directee heuristic (optionally guided by satellite imagery), while removing this heuristic allows for undirected exploration maximally
workspace.
Implicit navigation preferences: Yet another interesting property exhibited by ViNT its implicit preference to follow paved roads (out- doors), and drive smoothly in the middle of hallways (indoors), as demonstrated in Figure 9 and in the supplemental video. This is particu- larly interesting since a large chunk of the pre- training dataset contains suboptimal, weavy tra- jectories, and suggests that ViNT can learn "good" default behavior from the diverse train- ing behaviors. This preference helps ViNT efficiently explore previously unseen environments, where other baselines tend to explore the environment haphazardly (see Table 1 (righ()).
Figure 9:
ViNT exhibits an implicit preference for fol- lowing paved roads (lefi) and hallways (right).
Robustness to dynamic pedestrians: While ViNT trained only on offline data with a simple, self- supervised training objective, we find that its collision avoidance capabilities generalize to dynamic obstacles and pedestrians. Figure 10 exhibits an instance where the robot is tasked with navigating goal behind two pedestrians. ViNT selects actions that avoid the pedestrians and recovers the original path, successfully reaching the goal.
G
Additional Results
Please see Table 9, and Figures 11 and 12 for more experiment rollouts.
Indoor: Position
Outdoor: GPS
Outdoor: Satellite
Method
Success
Distance
Success
SPL
Distance
Success
SPL
Distance
ViKiNG [29]
0.60
56m
0.64
0.42
720m
0.77
0.68
780m
ViNT
0.90
91m
0.95
0.84
1270m
1.00
0.94
1040m
Table 9: ViNT can effectively utilize goal directed heuristics, such as 2D goal positions and satellite images, to explore novel silometer-scak environments successfully and without interventions.
22

---

(a)
(b)
(c)
(d)
lel
Figure 12: Satellite-uuided physical search with ViNT. We visualize 765m rollout of ViNT with satellite image-based heuristic from start (orange) goal (green). The future action samples â obtained by spatially grounding the subgoal candidates for five instances in the trajectory are shown yellow. uses the heuristic to pick the best subgoal (corresponding à marked in blue), guiding the robot to the goal.
-like planner
Right
Straight
Figure 13: The CARLA environment (top), and bird's eye view showing high-level routing commands for the routing task.
Figure Visualizing ViNT exploration rollouts in challenging indoor environments using Vizbot (top) and LoCoBot (bottom) rohotic platforms. Future action samples â ohtained hy spatially grounding the subgoal candidates are shown in yellow, with the best actions corresponding to the best candidate marked in blue.
23


Reviews:
[{"review_id": "rhuIX_e7K-", "reviewer": "Reviewer_ikk2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a foundation model for visual navigation tasks. The model is trained with a visual goal navigation task on multiple datasets. The paper also presents a diffusion-based goal generation module, a search-based planning module to enable long-range exploration. The paper also shows the model can take different prompts such as waypoints and turn-by-turn directions.", "summary_of_recommendation": "I think the work is interesting and could inspire more work in this direction. I hope the authors be a little more elaborated about its limitations and lessons learned from the process.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. This paper uses \"hundreds of hours\" of data to train the model. I have always been curious about how much data it requires to train a robot foundation model. I would love to hear the authors' opinions, do you think the size and diversity of the navigation data is still a bottleneck in this study? \n2. Similar to GNM, this work doesn't take into account the difference in camera intrinsics (FoV) and extrinsics (the install angle, etc.). Do you think this affects the generalization ability of the model? \n3. When fine-tuning on other types of prompts, i.e. the GPS waypoints and turn-by-turn instructions, the prompts are mapped to a shared space with the vision inputs. I wonder how the authors make sure the token space is shared? After fine-tuning, does the model still work for the visual token? \n4. The diffusion sub-goal generation model is a key difference in this paper compared to prior work. But there are not a lot of details about how that is trained. The author mentioned in 6.5 that the model generates invalid sub-goals. I wonder at about how much percent the model generates the reasonable sub-goals? In the video and paper, I could find any visualization about it. \n5. One difference of this paper with the GNM is that using a transformer model. How much difference has it made?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes an interesting problem of training a foundation model for visual navigation. It is currently a scorching topic to train \"foundation\" models for everything. Naturally, this paper caught wide attention as soon as it appeared on Arxiv. But looking deeply into the paper, it looks more like a stitch work of the authors' two prior works of ViKing and GNM, rather than one groundbreaking \"foundation model\". \n\nStrengths: \nThis is an interesting problem and the authors show interesting hardware results. \n\nWeaknesses:\nThe contribution of this paper is not highlighted. As far as I understand, the main improvement of this paper over the prior work is 1) a diffusion model that generates the sub-goal images, 2) adding other prompt formats. The authors have not shown very convincing results on how these two improvements contribute to the navigation foundation model. Actually, these two improvements are not about the foundation model, but more about how to use the foundation model in different navigation tasks.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "LZDn43PtexX", "reviewer": "Reviewer_EDM6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes ViNT – a foundation model designed for vision-based robotic navigation. It is trained on large and diverse datasets, enabling efficient adaptation to various navigational tasks and outperforming specialist models. The proposed methodology exhibits positive transfer, can explore novel environments, and solve kilometer-scale navigation problems with the ability to accommodate different problem domains. Results also show zero-shot transfer capabilities across new robots. \n", "summary_of_recommendation": "The paper proposed an intuitive idea and is well-presented. The experiments are detailed and exhaustive which makes a strong contribution to the paper. Some minor issues/questions shall be addressed, as highlighted above. I recommend accepting the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please check the \"Strengths and Weaknesses\" section for all the questions that need to be addressed. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n\n- The introduction of the paper is effectively crafted, providing a clear and compelling rationale for developing a system capable of generalization in the robotics context. The idea's intuitive appeal is well-explained, giving readers a strong motivation to delve into the proposed approach.\n- One particularly intriguing aspect of the research is its focus on the system's adaptability to novel modalities. This unique application warrants further investigation, as it has the potential to yield valuable insights and advancements in the field.\n- The website's inclusion of both code and pre-trained weights for the proposed model is undoubtedly beneficial for the wider community. By facilitating straightforward testing and adoption of the architecture, researchers can easily integrate and build upon the work.\n- The real-world experiments in the paper serve as a powerful demonstration of the system's utility. Showcasing the efficacy of the proposed approach in practical scenarios is highly commendable and adds credibility to the research findings.\n\nWeaknesses:\n\n- Related work sections mostly outline robotics based foundations models but miss mentioning various relevant papers that perform general visual navigation using GPS goals or goal images.\n- Various works that utilize advances in generative models (like diffusion models) to obtain sub goals could be included in the related works section\n- Question: the approach mentions that a unique output from ViNT is obtained in the form of a temporal distance. It would be interesting to see how the model could adapt/perform in cases where the goal is not reachable. This could be tested in simple block world examples for simplicity as well. In such cases, the current setup might lead to an ill posed problem.\n- The model also includes an output where a sequence of future actions is provided. It would be interesting to see how the model adapts in case of changes in the environment. For example, if certain parameters of the environment are varied the model should be equipped with the ability to modify the previous served action sequence. \n- An ablation study of the various components of the proposed approach in the main paper would be helpful for the readers.\n- The current model size is relatively large in comparison to the hardware resources available to most of the community. This might make it restrictive in terms of who can use this for actual deployment. Also, the impact of increasing model size is not studied. The tradeoff between the number of parameters and performance could be an interesting addition for the readers. Another smaller model (with slightly inferior performance) could also be provided alongside for those with limited hardware resources.\n- The paper specified that the robot is deployed at 4 Hz frequency which might act as a bottleneck in some cases.\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "KDs5q_0bkSl", "reviewer": "Reviewer_LWck", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose a method for training a foundation model for robot navigation. They create a large dataset by combining several open research datasets on robot navigation and train a transformer model to perform image goal navigation on the big dataset. To apply this to other downstream applications, they combine the trained model with a diffusion model for sampling subgoals (images) and a heuristic for determining which subgoals to use. The proposed approach is demonstrated to generalize to new environments, new robots, and even new tasks (with finetuning), outperforming all baselines.", "summary_of_recommendation": "Overall, I like the premise of the paper a lot. I also think the approach overall seems reasonable (at least Sec. 3). Moreover, the experimental questions that are asked are particularly important and exciting for the community if we can successfully do all those things with a single model. What's holding this paper back for me is the paper clarity (e.g. many important details are left in the appendix, and the appendix seems to be missing details too), and the experiments. Rather than answering many questions fairly quickly, I'd rather see fewer questions answered but with high quality experiments to support the claims. This can be done by demonstrating the claim against stronger baselines and on more tasks (e.g. the guided exploration task seems cherrypicked where there are not many baselines).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "See above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n* It's a very exciting and trending idea to apply foundation models to robotics (and specifically robot navigation).\n* Sec. 3 is written fairly clearly\n* Questions asked in Sec. 6 are very useful questions to ask. It's particularly exciting that the proposed model can generalize to new environments and to new robots. It's also nice that it can be fine-tuned for out of distribution environments.\n* Lots of experiments. The authors run a lot of experiments to answer several fundamental questions. Each of these experiments is valuable and answering relevant questions.\n\nWeaknesses & Suggestions\n* Please specify that the subgoals are represented as subgoal (RGB) images (line 147)\n* Suggestions for related work which use topological maps for navigation, and also utilize waypoint prediction + low-level controller to reach navigation subgoals.\n    * Chaplot et al. Neural Topological SLAM for Visual Navigation. CVPR 2020.\n    * Chen et al. Topological Planning with Transformers for Vision-and-Language Navigation. CVPR 2021.\n* Some important details are left out of the main paper and included in the appendix instead.\n    * Line 173 skips over important details wrt the planner and includes them in the appendix instead. More details in the main paper would be appreciated.\n    * Overall, I would appreciate more details in Sec. 4.2. \n* Some important details are left out of the appendix (and main paper)\n    * Line 556 seems to be added by accident: \"Remember to mention: total size, number of robots, conversion to number of frames and so on.\"\n        * Many details from the training dataset are missing, such as the above info. Were the training sets simply concatenated? What kind of processing was applied on the datasets and how was the data sampled (e.g. uniformly)?\n    * What is d_M and d_pred in line 499 of appendix?\n    * Where is f() defined? It seems to have two different definitions in the main paper vs in appendix.\n    * Please include more details in Appendix B.4.\n        * For example, it's unclear if ground truth odometry is used, and if there is a supervision signal for the goal encoder adapter \\tilde{phi}().\n        * What does it mean to \"learn a fixed tensor\" (lines 522, 533 of appendix)? Are you representing GPS commands as a latent learnable representation? How is this supervised?\n* More qualitative results, especially for the baseline models as well, would be useful.\n* Please clarify the output action space for all baselines (especially if they differ from the waypoints action space of the proposed model).\n* The baselines are weak (e.g. end-to-end BC). Recommended baselines include:\n    * Chaplot et al. Neural Topological SLAM for Visual Navigation. CVPR 2020.\n    * Chaplot et al. Learning to Explore using Active Neural SLAM. ICLR 2020.\n    * Chen et al. Learning Exploration Policies for Navigation. ICLR 2019.\n* More common tasks for Sec 6.1. Since guided exploration with GPS or satellite images is chosen, there are not many baselines to compare against here. For example, image goal navigation can be used and demonstrated for unseen environments. Thus, the method can be compared with a variety of image goal navigation approaches (e.g. neural topological SLAM). Or, point goal navigation can be compared against DD-PPO (Wijmans et al. ICLR 2019).\n    * While it is great to see generalization to unseen environments, it's also important to show that there is no degradation in performance in seen environments (compared to baselines).\n* Table 1 only shows a single metric for coverage. The single metric is insufficient, since it only measures distance traveled before collision rather than measuring coverage itself (e.g. the robot could be traversing in circles.\n* Please clarify Sec. 6.4 / Appendix E3.2. Overall, it is unclear. For example, is the finetuning done with behavior cloning? How is the graph constructed? Is a graph pre-constructed ie. prior to inference/evaluation time? How dense are the nodes? Do all baselines use the graph for navigation as well?\n* The heuristic seems hand-engineered, arbitrary, and brittle. How much tuning is involved with creating the heuristic? How sensitive is the model to the heuristic? e.g. how does performance differ if the weightings of the individual values change in the equation at line 499 appendix", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Dt6slBd4nbn", "reviewer": "Reviewer_Q7NJ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this work, the authors present Visual Navigation Transformer (ViNT), which aims at building a foundation model for vision-based navigation. ViNT tokenizes observations over the past 5 steps as well as the goal/observation states via EffcientNet-50. A Transformer is then applied to predict the 5 future actions and temporal distance to the goal. ViNT is pretrained on over 100 hrs heterogeneous navigation trajectories. The authors also propose implementation of diffusion model to generate subgoals and physical search for long-horizon planning. Through rigorous experiments, ViNT demonstrates (1) zero-shot generalization to unseen environments, (2) zero-shot generalization to different robotic platforms, (3) adaptation to low data regime via fine-tuning, and (4) adaptation to different modalities with different encoders. ", "summary_of_recommendation": "As mentioned above, this paper attempts to build foundation models for vision-based navigations. The paper presents a Transformer-based architecture as well as a pretraining technique that can be applied to heterogeneous trajectories. It also applies a diffusion model to generate subgoals to enable long-horizon navigation. Rigorous experiments that demonstrate the generalizability and adaptability of the proposed ViNT in navigations under a wide variety of settings. Overall, this paper can be a valuable contribution to building foundation models for various robotic tasks. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Major questions: \n1. How do authors determine to choose observations over the past 5 timesteps as input and future actions over 5 future timesteps? Will predicting more future actions be helpful?\n2. When applying the diffusion model in generating downstream tasks, is it fine-tuned on downstream tasks? If not, will the diffusion model still generate plausible samples?\n3. Long-horizon navigation relies on ViNT itself to grounding spatially. Will that lead to the accumulation of errors in cases where ViNT fails to accurately predict the temporal distances?\n4. As shown in Table 4, pretraining strategies like ImageNet and SimCLR achieve worse performance than training scratch. What could be the cause of the results? Since these pretraining methods usually perform well on vision tasks. \n4. As the authors also mention in the discussion, Transformer model can be computationally costly. I'm curious what the inference time of ViNT is compared with other baselines.\n\nMinor questions: \n1. In Appendix Table 7, FF dim of ViNT model is not correctly shown.\n2. In Appendix section C, there seems to be a note sentence that should be deleted. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The motivation to build a foundation model is inspiring. Robot learning usually involves various tasks, platforms, and environments, which limits the available data for different scenarios. This paper can be a valuable attempt to generalist foundation models in robotics.\n2. ViNT presents pre-training via predicting future actions and temporal distances. It demonstrates that such methods can be applied to heterogeneous real-world videos, which enlarge the available training data. \n3. The authors implement a diffusion model to generate subgoals for long-horizon navigation which greatly broaden the application of the pretrained model. \n4. The authors conduct rigorous experiments on different navigation settings and robotic platforms which demonstrate the effectiveness of the proposed method. Also, details regarding training and deployment are provided. \n5. The paper is well-written and easy to follow. \n\nWeaknesses:\n1. The paper focuses on vision-based navigation. Though it shows adaptation to different modalities in some experiments, there are still other modalities that may not be plugged in directly. ", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes how to learn generalist policy for image goal navigation that generalised (i) across robots using a robot-agnostic actions representation and (ii) new task specifications such as following waypoints or high-level routing commands. 

Authors are requested to take the detailed reviewer feedback into consideration. Specifically address the following: (i) improve clarity on the technical gap the paper is trying to fill (ii) why the contribution extends beyond putting two prior works together and (iii) improving the technical presentation of the high-level navigation system.
Decision: Accept (Oral)