Lifelong Autonomous Improvement of Navigation Foundation Models in the Wild
Authors: Kyle Stachowicz
Lydia Ignatova
Sergey Levine
Venue: CoRL 2024 (2024)
Keywords: Navigation
Reinforcement Learning
Lifelong Learning
TL;DR: We propose a framework for autonomously improving offline RL navigation policies with online RL with minimal-to-no human supervision

Abstract: Recent works have proposed a number of general-purpose robotic foundation models that can control a variety of robotic platforms to perform a range of different tasks, including in the domains of navigation and manipulation. However, such models are typically trained via imitation learning, which precludes the ability to improve autonomously through experience that the robot gathers on the job. In this work, our aim is to train general-purpose robotic foundation models in the domain of robotic navigation specifically with the aim of enabling autonomous self-improvement. We show that a combination of pretraining with offline reinforcement learning and a complete system for continual autonomous operation leads to a robotic learning framework that not only starts off with broad and diverse capabilities, but can further improve and adapt those capabilities in the course of carrying out navigational tasks in a given deployment location. To our knowledge, our model LiReN is the first navigation robot foundation model that is capable of fine-tuning with autonomous online data in open-world settings.

Lifelong Autonomous Fine-Tuning of Navigation Foundation Models in the Wild
Kyle Stachowicz', Lydia Ignatova', Sergey Levine UC Berkeley (kstachowicz, lydiaignatova, svlevine) (aciine)ebell @berkeley.dde
Recent works have proposed a number of general-purpon robotic foun- dation models that can control a variety robotic platforms to perform a range of different tasks, including in the domains of navigation and manipulation. How- ever, such models are typically trained via imitation learning, which precludes the ability to adapt autonomously through experience that the robot gathers on the job. In this work, aim is to train euderal-purpose robotic foundation models in the domain of robotic navigation specifically with the aim of enabling autonomous self-improvement. We show that a combination of pretraining with offline rein- forcement learning and complete system for continual autonomous operation leads to a robotic learning framework that not only starts off with broad and di- verse capabilities, but can further specialize and adapt those capabilities in the course of carrying out navigational tasks in given deployment location. To knowledge, this result demonstrates the first navigation robot foundation model continually learn via autonomous interaction in open- open-world settings.
Abstract:
Keywords: Reinforcement Learning. Robot Foundation Model
Pretrained RL Generalist
Deployable Lifelong Learning
Diverse Data
Improved Policy
Policy
Online
RL
Offline RL
Proposs
Ensura Continull
High-Lavel Supercisor
Figure LiReN navigation foundation model that improves autonomously during deployment via reinforcement learning. Offline RL pretraining provides broad generalization capabilities, while lifelong fine-tuning allows the model to get better and better through deployment.
1 Introduction
Many recent developments in machine learning have been driven by the advent of large-scale foun- dation models that leverage pretraining on large-scale Internet data. The same general recipe pretraining on diverse datasets and then adapting to new tasks via finetuning or in zero-shot has delivered impressive results on wide variety tasks in natural language processing [1. 2], com- puter vision [3, 4J, and other domains [5]. Analogously, robot, foundation models defined broadly policies trained on diverse robot data that can be further fine-tuned for downstream applications have shown promising results by training behavioral cloning policies on diverse multi-embodiment robot datasets [6. 7J. The process of fine-tuning these models typically requires training on expen- sive human demonstrations the target environment.
Equal contribution. Videos, code, and models: httpe/act.cittabiiiii//eeeeoon-e
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

This paradigm cannot deliver models that improve or adapt without human data for example, to respond to changes in the target environment or wear and tear on the robot itself. Ideally, a robot foundation model could instead continually improve via training on autonomous data collected while deployed. Such a model could adapt on the job, becoming better and better the more is used with- out relying on human supervision. However, this requires learning objective capable of learning from suboptimal data. While reinforcement learning is natural formulation for this problem, it is nontrivial to directly fine-tune BC policies with RL. The sample-efficient RL algorithms that are most practical for use in the real world [8. 9] require learning critic (88, a). which requires differ- ent leature-learming capabilities than BC [10].
In this paper we ask: how can we enable practical autonomous fine-tuning of generalist navi- gation policies in the real world? We posit that continually improving generalist policies in the real world requires two main ingredients: scalable training objective that is able successfully incorporate suboptimal data, and a supervisor framework for proposing tasks, computing rewards, and managing real-world deployment concerns at level of abstraction above the policy itself.
We instantiate these components in the visual navigation setting with our approach, Lifelong Reinforcement learning for Navigation (LiReN, Fig. 1). Like prior work in foundation models for visual navigation, LiReN performs pretraining on large, cross-embodiment visual navigation dataset. However, unlike existing methods, LiReN is trained with offline reinforcement learning to enable further training online. We wrap our foundation model with framework to enable life- long learning and show that LiReN is capable of autonomously improving its performance through online interactions over long timescales, increasing goal-reaching performance the downstream environment from 40% to 75% without additional human data.
Related Work
Mobile robots and visual navigation. We study the visual navigation problem, where a mobile robot navigates using only egocentric image observations reach goal pose, also specified by an egocentric image [11]. The task must be done minimal time without colliding with obstacles. Previous work applied BC to visual navigation in both simulation [12, 13] and real-world environ- ments [14]. Also related to our deployment setting is the CoBots system [15, 16, 17], mobile robot navigation system successfully deployed in real environments, although it does not include learn- ing components or online improvement. However, to our knowledge, LiReN is the first generalist navigation policy that can be continually improved during deployment with online RL.
Robot foundation models. Several recent works have proposed robot foundation models, defined in this work models trained on large-scale, diverse robot data (typically aggregated from many embodiments). The RT- series of models propose training generalist robot manipulation models via BC on single-embodiment (RT-1 [18], RT-2 [19]) and multi-embodiment (RT-X [20]) datasets. Sim- ilarly, Octo [6] proposes an open-source BC model for manipulation based on a scalable diffusion- based architecture. ViNT [7], the most relevant such model, trains a generalist policy on a broad navigation dataset and deploys on several mobile robot embodiments. While BC enables a simple learning objective, fine-tuning these models is challenging: further improvement requires expert tra- jectories, and fine-tuning BC policies with online RL is difficult without reward-speeific features [10]. Pre- Tre-TTrinnng for Roboties [21] applies offline RL diverse data and fine-tunes (also with offline RL) on demonstration data. While we similarly pre-train with offline RL, we also include an autonomous phase in which we fine-tune our offline foundation model with new online data.
Online reinforcement learning. Another approach to learning an end-to-end robot model starts from scratch, learning a policy using only online RL. While traditional wisdom suggests training in sim for data efficiency reasons [22], recent work has enabled sample-efficient RL operating directly in the real world [9, 23]. Zhu al. [24] further studies how make such an online learning problem autonomous. The closest navigation work, FastRLAP [10], learns a high-speed navigation policy using actor-critic algorithm applied to features from RL-pretrained encoder. While this provides

---

strong prior in the form of relevant features, the policy is not initialized with any notion of collision avoidance, goal-reaching, or other navigation behaviors. We instead train the entire policy and critic on offline data, providing a strong behavioral prior upon which the RL algorithm can improve.
Autonomous improvement for robotics. RoboCat [25] achieves autonomous improvement in the manipulation setting by collecting thousands of policy rollouts and then re-training the model from scratch on this dataset, filtered for successes. Isele et al. [26] learns low-level controllers for mobile robots in simulation. In contrast to these works which consider a closed- -woldd, instrumented environment in order to enable autonomous learning - we study RL-based autonomous fine-tuning in real office building in the navigation setting, which is inherently open- open-world. We thus develop supervisory framework to enable autonomous learning for our online fine-tuning system.
Preliminaries
We begin by formalizing our problem statement and discussing several prior works upon which our proposed framework builds:
Visual navigation. We formalize the visual navigation problem as a goal-conditioned partially- observed Markov decision process in which observations are given by egocentric images from the robot's camera and actions correspond to linear and angular velocity commanded to the robot, or the robot's relative position K frames ahead for the offline data in which raw actions are not avail- able [27]- We specify goals via future observation from a pose would like the robot reach, and consider trajectory success when the robot is within a threshold of this pose.
Sample-efficient RL. To learn from sub-optimal data collected autonomously in the real world, tools from reinforcement learning. We express our problem as a (goal-conditioned, par- tially observed) Markov decision process M composed of observation and action spaces O, A and goals G. We want to learn a policy S x g + AA through interactions with the environment. In visual navigation tasks we define the observation space as a single image frame of input and consider goal-reaching tasks considered on observations taken from the desired state.
We use actor-critic methods for their ability to learn from off-policy data. These methods learn Q-function corresponding the expected sum of discounted returns starting in state-action pair, and policy æ. In deep RI., Q and are approximated with neural network. The policy æ is trained to maximize (s,, a). which is learned via approximate dynamic programming:
y + By--Maaaaaaaa) [rr + ((s(,a')); min Lq (Q(s.a) -))
Actor-critic methods require continuous online interaction to avoid catastrophic overestimation [28]. When the policy held stationary or very slow-moving compared to data collection this can be viable, but sample-efficiency typically increases with a high update-to-data (UTD) ratio between gradient steps and environment steps. Stabilizing high-UTD RI. requires regularization techniques such clipped double-Q [29], ensembling [8], entropy regularization [30], and critic normalization [31]. In LiReN's 's implementation. we use a combination of all four of these techniques (following e.g. [10]), as well as several other techniques described in Sec. and App. A. We consider a special case (similar in spirit to hindsight relabeling [32]) such that some additional information about s becomes knowable later and provide this privileged information to the critic (but not the actor).
Offline RL with conservative Q-learning. To provide a high-quality initialization for our off- policy actor-critic algorithms, we require a pre-training method that is capable of learning both a policy and a critic function Q. One such choice of pretraining method is offline reinforcement learning, which attempts to solve the same maximum-expecte--rrwar problem online RL but with static dataset rather than through online interaction. This requires even further regularization to ensure the policy's ' actions remain in the approximate support of the original dataset.
3

---

Goal Selection
Online RL Training
Recovery Behavior
Trainer
Offiien Data
Robot
Battery Management
Keepout Zones
Figure 2: Requirements for autonomous improvement. Autonomous improvement mechanisms for proposing and evaluating goals, providing resets, and enabling pseudo-resets enabled by randomized perturbations after failures, and safe behavior corresponds to avoiding designated zones and maintaining battery charge.
safe operation. In our setting,
In particular, we apply conservative Q-learning [33], which proposes conservative regularizer artificially depress the Q-values of out-of-distribution actions compared to dataset actions. CQL adds the following term (blue) the standard temporal-differene, based Q-learning objective:
min Be.o-D [[Qo(s,a) a) y)² + En'wa [Q(s, -((sss))
Autonomously improving robot foundation models. Training reinforcement learning algorithms in the wild presents significant challenges not present in simulation. In the standard reinforcement learning framework, we assume that we have access to episodic resets which occur at regular in- tervals and ensure adequate state coverage, and each episode we are given a plausible task. Task success is automatically verified and yields ground-truth rewards for the task we are attempting to accomplish. In the real world, none these are true by default! Instead, we must implement them our- selves, as well as any task- task-specific safety features that are necessary to avoid accidentally interfering with humans or damaging the robot the environment.
4 Autonomously Improving Navigation Foundation Models
We apply RL train navigation foundation models end-to-end with the following recipe:
1. Apply offline RL large, multi-embodiment dataset of diverse navigation data, to get goal-conditioned policy with broad generalization.
2. Use navigation-specific "autonomy supervisor" to provide tasks, resets, and guardrails for exploratory behavior during deployment.
3. Learn online with sample-effiiien actor- critic algorithm fine-tune the model with au- tonomous data from deployment to enable in-the-wild improvement.
See Fig. 3 for a detailed diagram explaining how all of the following components interact. We provide an implementation of LiReN and our autonomy supervisor, along with sample data collected during online RL deployment, in our supplementary material.
4.1 Training Setup
We perform both (offline) pretraining and (online) fine-tuning with reinforcement learning, meaning that the same RL objective throughout. We train using CQL [33] both online and offline; all relevant hyperparameters are detailed in App. A.

---

Reward objective. We penalize the final distance to the goal, at the end of trajectory. with terminal value function (s) = - || gll- This could be accomplished with a single sparse reward for goal-reaching, but we equivalently express with a dense reward:
(8)) = -|s -
اlو = Σ (IIs: gll - 11s++1 - gll) ~
Σ
6i
(3)
gl
where e is the velocity vector and sÿ is the vector in the direction of the goal. This step reward function r(8, a, s', g) ||s - 9 - ||s ggl corresponds to the speed-made-good [10], maximizing (long-term) speed towards the objective. Collisions are penalized with a negative reward Terash-
Goal relabeling. In offline data, goals are selected from both future observations the same tra- jectory (positives, sampled from an exponential distribution) and observations from other trajectories (negatives. sampled uniformly from other trajectories from the same robot embodiment), as hind- sight experience replay [32] or goal-conditioned supervised learning [34]. During online training, keep the initially commanded ("on-policy") goals with 50% probability and otherwise relabel with positive goals. When relabeling, rewards are recomputed with the new goal.
Privileged critic information. In actor-critic reinforcement learning, it is crucial to have accurate critic values in order to learn high-quality policy. We again make particular use of the problem structure of goal-conditioned navigation. While at evaluation time the policy does not require a ground-truth goal displacement vector, we can use this vector when training on a completed trajec- tory by integrating locally-accurate odometry measurements. We take advantage of this property by supplying the critic network (but not the actor) with relative goal vectors sg.
Dataset. Single-embodiment robot models train on a relatively small dataset, collected specifically for the downstream task and environment. However, including diverse data tends to improve gener- alization and robustness [21]. Following ViNT [7], we train on the GNM dataset [27], compilation of several robot navigation datasets in distinct environments and embodiments. This allows LiReN to benefit from diverse data on which train its initial policy and learn generalizable features.
In the online phase, we continue to train on the offline dataset but draw half of each batch from the online replay buffer. This ensures that we do not experience catastrophic forgetting of behaviors learned from the original dataset, and generally stabilizes training.
4.2 Deployment and Autonomous Fine-tuning
We further fine-tune LiReN "on-the-job" during online deployment. this phase, we equip LiReN with several components to propose goals, avoid getting stuck, and ensure stable operation.
Proposing and evaluating goals. To propose goals (represented by images) and evaluate when they are completed, we make use of the spatial structure navigation. Prior deployment we map the area create a graph where nodes represent states (image observations) and edges represent environment transitions. During deployment, when goal is reached, future goal image can be selected randomly from its successors in the graph. We typically select goals 10-20 meters into the future according to an exponential distribution. We keep a static goal grpah, though if desired the graph can be updated online as in Shah et al. [11].
Recovery behavior. If the robot becomes stuck in a small area without exploring the rest of the environment, continuing to train on the new data will not increase overall performance. We avoid getting caught in short "goal loops" by biasing the goal graph's 's construction away from short cycles. Additionally, to avoid the case when the robot unable reach any goal because it stuck or in collision with an object, we inject pseudo-resets [10. 24] to perturb the state when the robot collides with object (detected by the robot's bumper). When it does not reach its goal for set period T, it is considered "stuck" and goal is selected from the graph.
5

---

Robot
Susers Plte
Conticker
Offline Stage
Datis
Reacted
Thoeel
Fol stre
COL
Goal Selection
Sran
Deck Change
Panuds staaet jBack upl
Offine
Onitne
Exacution
Crask
Online Stage
Ralkune.
Keepost
Zane
Figure 3: System diagram of LiReN. We use offline dataset to train generalist policy and Q- function on a broad array of navigation data. During fine-tuning the robot continually accepts new network weights from the training process and sends back data, while deciding whether to run the policy, recovery behavior, or docking fallbacks depending on the state of the robot. Goal selection accomplished by sampling nearby goals from pre-recorded goal graph.
Keepout zones. While exploring the state space, there are some zones in the environment the robot cannot enter for a variety of reasons. To protect privacy, ensure robot safety, and maintain reasonable localization, we mark areas such as bathrooms. When the robot detects that it is in a keepout zone, it triggers pseudo-reset.
Calling for help. If the robot is unable exit its current state (as measured by repeatedly becoming stuck crashing at least ten times in row) it will "call for help" by messaging human over the internet. While this capability is necessary for truly lifelong learning, we find that it is triggered extraordinarily rarely (c.g. it is physically stuck); typically the combination of pseudo-resets and reward-guided exploration is enough for the robot to escape any local minimum.
Battery charge monitoring. To enable indefinite operation for lifelong learning, we continuously monitor the battery's charge. We place charging stations in the deployment environment; the battery level drops below set threshold, it will dock for charging the next time passes a station.
Robot hardware. We use an iRobot Create 3 with an inexpensive fisheye camera. Inference performed with a Jetson Orin NX at 3Hz. We also equip our robot with a LiDAR sensor and map of the environment for localization, though we emphasize that this is an artifact of our goal selection mechanism, is not used by the policy, and could be avoided with different choice of training task.
Asynchronous training system design. During fine-tuning, training is handled asynchronously by server (a single Google TPUv4 node) which receives data from the robot in real time and continously updates the robot's model parameters over the network. The training server is robust to unstable network conditions, allowing gracefully recovery from disconnects.
5 Experiments
We design our experiments answer the following questions:
Q1. Does our combination of offline pretraining with autonomous online fine-tuning yield signif- icant improvements in quantitative and qualitative navigation behavior?
Q2. Is offline RL necessary? That is, can we achieve similar autonomous fine-tuning with goal- conditioned behavior cloning (GCBC) or via online RL from scratch?
Q3. How performant is the offline policy learned by offline RL, in comparison with existing state- of-the-art foundation models for navigation?

---

100%
60%
Noportton
Figure Qualitative improvement dur- ing online fine-tuning. The pre-trained model cannot differentiate the out-of- distribution white obstacle from the walls. Encountering the obstacle later during fine- tuning, LiReN has learned from its experi- ences and successfully avoids it.
5.5
1.5
Time (rouns)
Figure 4: Lifelong learning with LiReN. Our model continues train successfully without human interventions or demonstrations, improving from less than 40% initial goal-reaching performance to 75% after a few hours of autonomous online training.
5.1 Evaluating Online Learning
We deploy LiReN with our autonomous improvement framework in real-world office building. Operating in this regime, the robot must contend with diverse scenes, significant changes in lighting, and people working in the space and reconfiguring it might expect in an office building. We track two key metrics: (i) success rate, the percentage of goals successfully reached as moving average over a window of T = 12 minutes (corresponding to roughly the time taken to traverse the whole environment) and (ii) coverage, the percentage of free space that has been explored: F (e F llpt pll S r}}, where F is free space. important to consider both of these metries, as policy that cannot escape an "easy" region may have high success rate but low coverage.
Using our fine-tuning framework leads to significant quantitative improvements in policy perfor- mance during online deployment: LiReN trained online significantly outperforms the base policy. Figure shows steady increase in policy performance throughout the course of training.
We also see specific qualitative improvements during training. Figure 5 shows one such challenging case: the white obstacle difficult to distinguish from the white walls, and the base model fails to navigate the situation when encountered for the first time. However, after period of online fine-tuning, LiReN successfully recognizes and avoids the obstacle.
5.2 Should we really fine-tune with RL?
Why not GCBC? Interestingly, this im- provement is not mirrored by attempts to fine- tune GCBC models [34]. In this setting, we train a policy (via BC) to reach relabeled goals [32], chosen from future observations from the same trajectory. We find that perform- ing this "iterated GCBC", in which the policy is trained to match its own successful output, causes the policy's 's performance to degrade over time as shown in Fig. 7. We hypothesize that this is due in part to distribution shift between goals on which the policy is trained (from policy, which come from the fixed goal graph collected different circumstances (c.g. lighting).
LiReN
GCBC
RLPD
Figure 6: Locations visited during online fine- tuning. LiReN is explores the entire environment successfully. Iterated GCBC also exhibits reason- able coverage. Without a generalist policy initial- ization, RLPD becomes stuck in the initial region and cannot reach long-horizon goals.
future states) and the "real" goals provided to the
Why not online RL? We also consider the other end of the spectrum: applying online RL with random initial policy. In this setting, we initialize random policy and apply RL with Prior Data [31]
7

---

17040
Treeut
Method
Time
Success %
Coverage
0016
Cran
LiReN
Oh
32%
GCBC [34]
11%
63%
RLPD [31]
1h
46%
27%
LiReN
1h
61 %
95% %
20%
LiReN
3h
75%
100%
03 62
24
055
2.6
Table 1: Online Fine-Tuning, LiReN can be
fine-tuned with autonomous data great effect. Within a few hours of online training to adapt to new environment, LiReN's ' performance on goal- reaching tasks doubles that of the original policy.
Figure 7: GCBC fine-tuning. While GCBC gives reasonable initial performance. it cannot improve over time and performance eventually collapses. Metries averaged over 12 minutes.
(RLPD), which trains SAC-like policy on a mixture of offline and online data. While RLPD learns surprisingly good collision avoidance behavior in small section of the environment, find that it
strongly overfits to the initial location and is unable generalize. This is reflected in the coverage
metrics shown in Table 1 and in the maps visualized in Figure 6.
Indoor
Outdoor
Avg.
5.3 Measuring Offline Performance
Method
Env.
C
LiReN
34%
97%
98%
93%
81 %
LiReN
We adopt a methodology similar to prior work [27, 7, 35] to compare the quality of the offline policy with existing models along small, con- trolled sequences of goals (rather than the open- world performance reflected by the first row of Tab. 1). For each environment in which we eval- uate offline performance, we collect fixed se- quence of 4-5 short term goals, following evaluations in Shah et al. [27]. We measure each model's progress from consistent starting point, computed as percent of the total distance traveled and averaged over ten trials. In addition indoor experiments, we also provide experiments using the Clearpath Jackal operating in an outdoor setting, demonstrating LiReN's cross-embodiment capa- bilities. We include diagrams of each environment in App. B.
78%
78%
91%
93%
84 %
(fine-tuned)
GCBC
61%
30%
20%
46%
ViNT [7]
71 %
38%
23%
52%
46%
Table Offline Performance. LiReN delivers strong zero-shot visual navigation performance, comparable to cate-of-the-ar art models [7].
While improving pure offline performance is not the focus of this work, we nevertheless find that LiReN achieves strong empirical performance with offline RL in comparison to state-of-the-art mod- els as highlighted in Table 2.
Discussion and Limitations
We presented LiReN, a navigation foundation models that can learn autonomously in lifelong setting while deployed. By pre-training with offline RL on diverse navigation data, we get a sual navigation foundation model comparable state-of-the-an models. By pairing LiReN with an autonomous improvement framework for navigation including goal selection and verification, recov- ery/reset procedures, and guardrails to ensure continuous operation, we can fine-tune our foundation model to continuously adapt to the target environment, learning to handle changes in the downstream environment without requiring additional groundwork for further study into methods for autonomously fine-tuning robot foundation models.
data. We hope that LiReN will lay the
Limitations and future work: current autonomous improvement framework does have several limits: our goal selection procedure currently requires a notion of localization to find and verify nearby locations on the goal graph, and we currently only support a single fine-tuning task. Ad- ditionally, while the robot we used for deployment and online improvement was not itself in the dataset, we did not explore concurrently fine-tuning RL models with multiple new embodiments at once; we expect that this might raise its own challenges.

---

Acknowledgments
This research was partly supported by the NSF under IIS-2150826, ARL DCIST CRA W911NF-17. 2-0181, and ONR 400014-22-1-2773. The authors would like to thank Catherine Glossop, Dhruv Shah, Zhiyuan Zhou and Pranav Atreya for helpful discussions early in the project. We would also like to thank Oier Mees for feedback on an early draft of the manuscript and its figures.
References
[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, N. Gomez, L. Kaiser, and Polo- sukhin. Attention is all you need, 2023.
[2] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
[3] M. Oquab, T. Darcet, Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, Fernandez, D. Haz- iza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, Mairal, Labatut, A. Joulin, and P. Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
[4] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021.
[5] A. Das, Kong, R. Sen, and Y. Zhou. decoder-only foundation model for time-series forecasting, 2024.
[6] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405. 12213, 2024.
[7] D. Shah, A. Sridhar, N. Dashora, Stachowicz, K. Black, N. Hirose, and Levine. Vint: A foundation model for visual navigation. arXiv preprint 1rXiv:2306.14844 2023.
[8] X. Chen, C. Wang, Z. Zhou, and K. Ross. Randomized ensembled double q-learning: Learning fast without model, 2021.
[9] L. Smith, I. Kostrikov, and S. Levine. walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. arXiv preprint arXiv:2208.07888 2022.
[10] K. Stachowicz, D. Shah, A. Bhorkar, I. Kostrikov, and S. Levine. Fastrlap: A system for learning high-speed driving via deep rl and autonomous practicing. In Conference on Robot Learning, pages 3100-3111. PMLR, 2023.
[11] D. Shah, B. Eysenbach, G. Kahn, N. Rhinehart, and S. Levine. Ving: navigation with visual goals. In 2021 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 13215-13222. IEEE, 2021.
Learning open- world
[12] K. Yadav, S. K. Ramakrishnan, J. Tumner, A. Gokaslan, O. Maksymets, R. Jain, R. Ramrakhya, A. X. Chang, A. Clegg, M. Savva, E. Undersander, D. S. Chaplot, and D. Batra. Habitat challenge 2022. https: haaps:///ahabbiatt wwwpiitttt.tt///haaeee wttpcccalalcggg//20222 2022.
[13] D. S. Chaplot, D. Gandhi, A. Gupta, and R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration, 2020.
[14] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine. Rapid Exploration for Open-World Navigation with Latent Goal Models. In 5th Annual Conference Robot Learning, 2021. URL http://ppppreeeeeeee.eee fttp://formmnddddddeeeeed ittt///oorrrr--esSS/hhhtttttt

---

[15] J. Biswas and M. M. Veloso. Localization and navigation of the cobots over long-term deploy- ments. The International Journal of Robotics Research, 32(14):1679-1694 2013.
[16] M. Veloso, J. Biswas, B. Coltin, S. Rosenthal, T. Kollar, C. Mericli, M. Samadi, S. Bran- dao, and R. Ventura. Cobots: Collaborative robots servicing multi-floor buildings. In 2012 JEEE/RSJ international conference on intelligent robots and systems, pages 5446-5447. IEEE, 2012.
[17] M. Veloso, J. Biswas, B. Coltin, and S. Rosenthal. Cobots: Robust symbiotic autonomous mobile service robots. In fwonth-fourth international joint conference artificial intelligence. Citeseer, 2015.
[18] A. Brohan, N. Brown, Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, Haus- man, Herzog, Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. Ri-1: Robotics transformer for real-world control at scale, 2023.
[19] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, L. Lee, T.-W. E. Lee, Levine, Lu, H. Michalewski, Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, Sanketi, Sermanet, Singh, A. Singh, R. Soricut, H. Tran, Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-2: Vision-language-action models transfer web knowledge robotic control, 2023.
[20] O. X.-E. Collaboration. Open x-embodiment: Robotic learning datasets and rt-x models, 2024.
[21] A. Kumar, A. Singh, F. Ebert, M. Nakamoto, Y. Yang, C. Finn, and S. Levine. Pre-training for robots: Offline rl enables learning new tasks from a handful of trials, 2023.
[22] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots,
2021.
[23] J. Luo, Z. Hu, C. Xu, L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine. Serl: software suite for sample-efficient robotic reinforcement learning, 2024.
[24] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine The ingredients of real-world robotic reinforcement learning, 2020.
[25] K. Bousmalis, G. Vezzani, D. Rao, C. Devin, A. X. Lee, M. Bauza, T. Davehev, Y. Zhou, A. Gupta, A. Raju, et al. Robocat: A self-improving foundation agent robotic manipulation. arXiv preprint arXiv:2306. 1706, 2023.
[26] D. Isele, J. M. Luna, E. Eaton, G. V. de la Cruz, J. Irwin, B. Kallaher, and M. E. Taylor. Lifelong learning for disturbance rejection mobile robots. In 2016 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3993-3998 IEEE, 2016.
[27] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model drive any robot. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 7226-7233. IEEE, 2023.
[28) R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[29] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor- critic methods, 2018.
10

---

[30] T. Haarmoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policc maximum entropy deep reinforcement learning with a stochastic actor, 2018.
[31] J. Ball, L. Smith, I. Kostrikov, and S. Levine. Efficient online reinforcement learning with offline data, 2023.
[32] M. Andrychowicz, F. Wolski, Ray, Schneider, R. Fong, Welinder, McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay, Advances in neural information processing systems, 30, 2017.
[33] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances Neural Information Processing Systems, 33:1 179-1191, 2020.
[34] D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. M. Devin, B. Eysenbach, and S. Levine. Learn- ing to reach goals via iterated supervised learning. In International Conference on Learning Representations, 2021. URL https /ttp///vpp//eeeeeee///ee/ee met//foruu
[35] A. Sridhar, D. Shah, C. Glossop, and S. Levine. Nomad: Goal masked diffusion policies for navigation and exploration. arXiv preprint arXiv:2310.07896, 2023.

---

Appendix A Network Architecture and Hyperparameters
Model Architecture
Structure
ResNet18
Encoder
Pooling
spatial softmax, 8 blocks
Structure
MLP+repeat action conditioning [21]
Critic
Normalization
LayerNorm
Hidden dims
(256, 256)
Structure
MLP
Actor
Hidden dims
(256, 256)
Training
Discount
0.97
Target entropy
-1.0
Temperature init
0.1
CQL
DR3 regularization
1e-3
Alpha
0.3
Batch size
256
Optimizer type
adamw
Weight decay
le-4
Optimizer
Agent LR
0.0001
Critic LR
0.0003
Temp LR
0.0003
Data
Image size
64x64
Image transforms
random brightness/contras//uus
Augmentations
random flip, random crop
Positive probability
0.75
Offline dataset goal sampling
Negative probability
0.25
Sampling distribution
Exponential(20)
Original goal probability
0.5
Positive probability
0.375
Online dataset goal sampling
Negative probability
0.125
Data before training
1500
Autonomy
Goal reach distance
0.75 meters
Goals
Goal reach angle error
45 degrees
Sampling distribution
Exponential(S)
Max failures in row
10
Error handling
Low battery threshold
25%
Table 3: Model Architecture and Training Parameters
12

---

Appendix B
Offline Testing Environment Layouts
Env A
Env B
START
OCAL
COAL
START
Env D
Env C
Figure 8: Small environments for evaluation of offline policies. Goal trajectories are depicted, along which 4-5 goal waypoints are placed. When goal achieved in the offline evaluation setting, we advance the goal deterministically to the next waypoint.
13


Reviews:
[{"review_id": "LHfB9eRWIx", "reviewer": "Reviewer_BE4N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a new approach for a navigation foundation model. Offline RL is used to fine-tune a general foundation model, followed by online RL for further improvement. Real-world experiment results demonstrate the effectiveness of the proposed module.", "summary_of_recommendation": "I like the idea and the experiment of this paper. But I do think the writing for the section 3 and 4 needs to be improved before acception.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Since you are discussing a \"Navigation Foundation Model,\" it is important to show that your method can be deployed on different navigation robots, similar to what Vint is doing.\n\n2. I want to see the offline-RL and online-RL parts more clearly explained. Specifically, the methods you mentioned in Section 4—during which stage are they used?\n\n3. I also have a deployment question. From Figure 4, we can see that the performance of navigation improves over time. Does this mean that you are training on the fly, or are you collecting data for 10 minutes and then updating the policy? Perhaps you can provide more details about the deployment process in the appendix.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jErP3xxBp7", "reviewer": "Reviewer_4zSa", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a method to continuously and autonomously improve the capability of navigation foundation models after deployment in a real-world environment. It leverages offline pretraining on diverse navigation data and later adapts the policy on the real robot via sample-efficient online RL.", "summary_of_recommendation": "I like the proposed idea and feel that the experimental results mostly support the claims made by the authors. At a high level, the framework proposed in the paper is quite clear, but it is somewhat difficult to completely grasp all the details because of the absence of a concise description of all the training procedure. Some aspects of the results also need further clarification (e.g. how many independent trials were conducted).", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. How many independent trials were used to calculate the results reported in Figs. 4, 7, and tables 1 and 2?\n\n2. What is the effect of the size of the dataset used for pretraining via offline RL? How much data is sufficient?\n\n3. Line 119: \"... some additional information about $s_t$ becomes knowable later\" - Why does this information become available later? When does it become available?\n\n4. Line 116: The authors write \"In LiReN’s implementation, we use a combination of all three of these techniques ...\" but the previous line mentions 4 techniques: clipped double-Q, ensembling, entropy regularization, and critic normalization. \n\n5. How is a crash detected?\n\n6. Why do you not consider any penalty in the reward formulation (e.g. a penalty for crashing)?\n\n7. Concerning Fig. 7, why does the performance of GCBC collapse after improving initially?\n\n8. Do you map the environment manually before deploying the robot? Please mention this in the paper.\n\n9. In Table 1, how do you calculate the coverage percentage?\n\n10. What are the differences between \"Env A\", \"Env B\" and \"Env C\" in Table 2?\n\n11. Is it a good idea to perform online training continuously? There is a difference between finetuning and lifelong/continual learning. For example, suppose the robot trains during the day (with some natural light) and then continues to adapt (i.e. finetune its parameters) to artificial light at night. Does it need to readapt to natural light the next day? As there is no mechanism to prevent forgetting, I would assume the answer to this question is yes. This is possibly something to look into in the future but I would be interested to know the authors' view on this.\n\n12. *Minor*: Line 164: Is this a typo? \"... it triggers a minimal reset which backs away it is back ...\"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kBWKeCKe6e", "reviewer": "Reviewer_p4fP", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The submission proposes a pipeline for lifelong learning for robot navigation by starting from a pretrained model. In particular, the method uses large-scale data to pretrain a policy and Q-function via offline RL (using conservative Q-learning; CQL [28]0), and then uses the learned models as a starting point for adapting to a set of real-world navigation tasks. The pipeline is fully autonomous: it provides goals for the robot and recovery mechanisms to avoid the need for human interventions except in the most extreme cases where the robot is physically stuck. The experiments demonstrate improvements over existing methods, which also serve as ablation studies for the proposed approach.", "summary_of_recommendation": "I lean toward recommending acceptance for the submission. While in terms of learning the contributions are not significant, they are in terms of proposing a complete pipeline that enables learning with existing methods within a physical platform. The authors should primarily address two concerns: delimit the scope of their contribution more precisely, and describe more clearly the details of their goal relabeling scheme.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Summarizing the points above, here are the main questions/suggestions I have for the authors:\n- Please include the relevant missing references (and discussions about their connection to this work)\n- Please scope the contribution appropriately in terms of specializing or finetuning (instead of the more general \"improving\")\n- What is the exact relabeling process used by the authors? Is this novel or taken from existing work?\n- Please answer my questions above about the experimental setting\n\n######## Additional feedback ########\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\nAbstract\n- It is unclear at this point what \"a complete system\" is\n\nIntro\n- Either define \"behavior cloning (BC)\" and use \"BC\" thereafter, or drop \"BC\" and stick to \"behavior cloning\"\n- \"a critic $Q(s,a)$, which requires different feature-learning capabilities than behavior cloning\" --> could the authors argue this point? Is there existing literature that studies whether this is true, or is there some obvious reason why features learned for BC couldn't be used for Q?\n\nSec 2\n- This is the second time the authors mention a \"supervisory framework to enable autonomous learning\", but this doesn't really mean much on its own. It might be worth to include ~1 sentence in the intro that broadly describes what this entails.\n\nSec 3\n- This section seems quite precise, which is nice.\n\nSec 4\n- I suggest separating Sec 4 into two sections: one focusing on the non-learning aspects that are necessary for safe long-term deployment (roughly p to line 178) and the learning aspects (from line 179 on, plus maybe the goal proposal)\n- There's a sense in which the goal proposal really is just a \"trick\" to enable evaluating these ideas. In the real world, we would want the robot to learn from actual goals that users intend for it (or goals that it sets for itself with some real higher-level objective, like surveillance). It might be worth discussing this point. More generally, this section would benefit from additional clarity around which aspects are necessary for training, for longterm autonomy, or for evaluation.\n- I didn't really understand the \"privileged critic information\" paragraph, but I think this relates to the relabeling.\n- \"we emphasize that this is not necessary for downstream deployment\" --> this is a really odd comment in a lifelong deployment setup. I think the authors mean that the LiDAR is only used to set the goal, which is only an artifact of the evaluation setting and not the actual learning setting. \n\nSec 5\n- I like the design of fig 4, like a continuous stacked bar chart\n- What is the cause for the oscillating pattern in the performance improvement?\n- The fact that RLPD obtains 46% success rate but only covers 27% of the space suggests that success rate over the distribution of proposed goals is not on its own a great metric of performance. But including the results on coverage addresses this.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper presents an online learning approach for a foundation navigation model that can be refined as the robot collects more data. 

Strengths:
- Interesting idea and an effective algorithm
- Convincing experiments that clearly demonstrate the impact of online RL for improvement the performance of previously trained models

Weaknesses:
- More details needed for the results
- Clarity on the method can be improved with pseudo-code or a diagram
- Novelty needs to be better described as the general idea of offline RL followed by online finetuning has been proposed before
- Clarify on how the method improve "foundation models" in all its generality 

The reviewers provided valuable suggestions to improve the paper. I encourage the authors to work with the reviewers in addressing their concerns.
===============================
Pos-rebuttal update:

The authors have addressed the concerns reported and the reviewers now agree that the paper should be accepted.
Decision: Accept