Lifelong Autonomous Improvement of Navigation Foundation Models in the Wild

Authors: kyle  stachowicz
lydia  ignatova
sergey  levine
u c berkeley
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 13:14:33

Abstract: Recent works have proposed a number of general-purpose robotic foundation models that can control a variety of robotic platforms to perform a range of different tasks, including in the domains of navigation and manipulation. However, such models are typically trained via imitation learning, which precludes the ability to improve autonomously through experience that the robot gathers on the job. In this work, our aim is to train general-purpose robotic foundation models in the domain of robotic navigation specifically with the aim of enabling autonomous self-improvement. We show that a combination of pretraining with offline reinforcement learning and a complete system for continual autonomous operation leads to a robotic learning framework that not only starts off with broad and diverse capabilities, but can further improve and adapt those capabilities in the course of carrying out navigational tasks in a given deployment location. To our knowledge, our model LiReN is the first navigation robot foundation model that is capable of fine-tuning with autonomous online data in open-world settings.

Main paper: 
 introduction: 
 Many recent developments in machine learning have been driven by the advent of large-scale foundation models that leverage pretraining on large-scale Internet data. The same general recipepretraining on diverse datasets and then adapting to new tasks via finetuning or in zero-shot -has delivered impressive results on a wide variety of tasks in natural language processing [1, 2] , computer vision [3, 4] , and other domains [5] . Analogously, robot foundation models -defined broadly as policies trained on diverse robot data that can be further fine-tuned for downstream applicationshave shown promising results by training behavioral cloning policies on diverse multi-embodiment robot datasets [6, 7] . The process of fine-tuning these models typically requires training on expensive human demonstrations in the target environment. This paradigm cannot deliver models that improve or adapt without human data -for example, to respond to changes in the target environment or wear and tear on the robot itself. Ideally, a robot foundation model could instead continually improve via training on autonomous data collected while deployed. Such a model could adapt on the job, becoming better and better the more it is used without relying on human supervision. However, this requires a learning objective capable of learning from suboptimal data. While reinforcement learning is a natural formulation for this problem, it is nontrivial to directly fine-tune BC policies with RL. The sample-efficient RL algorithms that are most practical for use in the real world [8, 9] require learning a critic Q(s, a), which requires different feature-learning capabilities than BC [10] .In this paper we ask: how can we enable practical autonomous fine-tuning of generalist navigation policies in the real world? We posit that continually improving generalist policies in the real world requires two main ingredients: a scalable training objective that is able to successfully incorporate suboptimal data, and a supervisor framework for proposing tasks, computing rewards, and managing real-world deployment concerns at a level of abstraction above the policy itself.We instantiate these components in the visual navigation setting with our approach, Lifelong Reinforcement learning for Navigation (LiReN, Fig. 1 ). Like prior work in foundation models for visual navigation, LiReN performs pretraining on a large, cross-embodiment visual navigation dataset. However, unlike existing methods, LiReN is trained with offline reinforcement learning to enable further training online. We wrap our foundation model with a framework to enable lifelong learning and show that LiReN is capable of autonomously improving its performance through online interactions over long timescales, increasing goal-reaching performance in the downstream environment from 40% to 75% without additional human data. 
related work: 
 Mobile robots and visual navigation. We study the visual navigation problem, where a mobile robot navigates using only egocentric image observations to reach a goal pose, also specified by an egocentric image [11] . The task must be done in minimal time without colliding with obstacles. Previous work applied BC to visual navigation in both simulation [12, 13] and real-world environments [14] . Also related to our deployment setting is the CoBots system [15, 16, 17] , a mobile robot navigation system successfully deployed in real environments, although it does not include learning components or online improvement. However, to our knowledge, LiReN is the first generalist navigation policy that can be continually improved during deployment with online RL.Robot foundation models. Several recent works have proposed robot foundation models, defined in this work as models trained on large-scale, diverse robot data (typically aggregated from many embodiments). The RT-series of models propose training generalist robot manipulation models via BC on single-embodiment (RT-1 [18] , RT-2 [19] ) and multi-embodiment (RT-X [20] ) datasets. Similarly, Octo [6] proposes an open-source BC model for manipulation based on a scalable diffusionbased architecture. ViNT [7] , the most relevant such model, trains a generalist policy on a broad navigation dataset and deploys on several mobile robot embodiments. While BC enables a simple learning objective, fine-tuning these models is challenging: further improvement requires expert trajectories, and fine-tuning BC policies with online RL is difficult without reward-specific features [10] . Pre-Training for Robotics [21] applies offline RL to diverse data and fine-tunes (also with offline RL) on demonstration data. While we similarly pre-train with offline RL, we also include an autonomous phase in which we fine-tune our offline foundation model with new online data.Online reinforcement learning. Another approach to learning an end-to-end robot model starts from scratch, learning a policy using only online RL. While traditional wisdom suggests training in sim for data efficiency reasons [22] , recent work has enabled sample-efficient RL operating directly in the real world [9, 23] . Zhu et al. [24] further studies how to make such an online learning problem autonomous. The closest navigation work, FastRLAP [10] , learns a high-speed navigation policy using an actor-critic algorithm applied to features from a RL-pretrained encoder. While this provides a strong prior in the form of relevant features, the policy is not initialized with any notion of collision avoidance, goal-reaching, or other navigation behaviors. We instead train the entire policy and critic on offline data, providing a strong behavioral prior upon which the RL algorithm can improve.Autonomous improvement for robotics. RoboCat [25] achieves autonomous improvement in the manipulation setting by collecting thousands of policy rollouts and then re-training the model from scratch on this dataset, filtered for successes. Isele et al. [26] learns low-level controllers for mobile robots in simulation. In contrast to these works -which consider a closed-world, instrumented environment in order to enable autonomous learning -we study RL-based autonomous fine-tuning in a real office building in the navigation setting, which is inherently open-world. We thus develop a supervisory framework to enable autonomous learning for our online fine-tuning system. 
preliminaries: 
 We begin by formalizing our problem statement and discussing several prior works upon which our proposed framework builds:Visual navigation. We formalize the visual navigation problem as a goal-conditioned partiallyobserved Markov decision process in which observations are given by egocentric images from the robot's camera and actions correspond to linear and angular velocity commanded to the robot, or the robot's relative position K frames ahead for the offline data in which raw actions are not available [27] . We specify goals via a future observation from a pose we would like the robot to reach, and consider a trajectory as a success when the robot is within a threshold ϵ of this pose.Sample-efficient RL. To learn from sub-optimal data collected autonomously in the real world, we turn to tools from reinforcement learning. We express our problem as a (goal-conditioned, partially observed) Markov decision process M composed of observation and action spaces O, A and goals G. We want to learn a policy π : S × G → A through interactions with the environment. In visual navigation tasks we define the observation space as a single image frame of input and consider goal-reaching tasks considered on observations taken from the desired state.We use actor-critic methods for their ability to learn from off-policy data. These methods learn a Q-function corresponding to the expected sum of discounted returns starting in a state-action pair, and a policy π. In deep RL, Q and π are approximated with a neural network. The policy π is trained to maximize Q(s, a), which is learned via approximate dynamic programming:EQUATIONActor-critic methods require continuous online interaction to avoid catastrophic overestimation [28] .When the policy is held stationary or very slow-moving compared to data collection this can be viable, but sample-efficiency typically increases with a high update-to-data (UTD) ratio between gradient steps and environment steps. Stabilizing high-UTD RL requires regularization techniques such as clipped double-Q [29] , ensembling [8] , entropy regularization [30] , and critic normalization [31] . In LiReN's implementation, we use a combination of all four of these techniques (following e.g. [10] ), as well as several other techniques described in Sec. 4 and App. A. We consider a special case (similar in spirit to hindsight relabeling [32] ) such that some additional information about s t becomes knowable later and provide this privileged information to the critic (but not the actor).Offline RL with conservative Q-learning. To provide a high-quality initialization for our offpolicy actor-critic algorithms, we require a pre-training method that is capable of learning both a policy π and a critic function Q. One such choice of pretraining method is offline reinforcement learning, which attempts to solve the same maximum-expected-reward problem as online RL but with a static dataset rather than through online interaction. This requires even further regularization to ensure the policy's actions remain in the approximate support of the original dataset. In particular, we apply conservative Q-learning [33] , which proposes a conservative regularizer to artificially depress the Q-values of out-of-distribution actions compared to dataset actions. CQL adds the following term (blue) to the standard temporal-difference based Q-learning objective:EQUATIONAutonomously improving robot foundation models. Training reinforcement learning algorithms in the wild presents significant challenges not present in simulation. In the standard reinforcement learning framework, we assume that we have access to episodic resets which occur at regular intervals and ensure adequate state coverage, and each episode we are given a plausible task. Task success is automatically verified and yields ground-truth rewards for the task we are attempting to accomplish. In the real world, none these are true by default! Instead, we must implement them ourselves, as well as any task-specific safety features that are necessary to avoid accidentally interfering with humans or damaging the robot or the environment. 
autonomously improving navigation foundation models: 
 We apply RL to train navigation foundation models end-to-end with the following recipe:1. Apply offline RL to a large, multi-embodiment dataset of diverse navigation data, to get a a goal-conditioned policy with broad generalization.2. Use a navigation-specific "autonomy supervisor" to provide tasks, resets, and guardrails for exploratory behavior during deployment.3. Learn online with a sample-efficient actor-critic algorithm to fine-tune the model with autonomous data from deployment to enable in-the-wild improvement.See Fig. 3 for a detailed diagram explaining how all of the following components interact. We provide an implementation of LiReN and our autonomy supervisor, along with sample data collected during online RL deployment, in our supplementary material.We perform both (offline) pretraining and (online) fine-tuning with reinforcement learning, meaning that we can use the same RL objective throughout. We train using CQL [33] both online and offline; all relevant hyperparameters are detailed in App. A.Reward objective. We penalize the final distance to the goal, at the end of a trajectory, with terminal value function V (s) = -∥s -g∥. This could be accomplished with a single sparse reward for goal-reaching, but we equivalently express it with a dense reward:EQUATIONwhere v is the velocity vector and ⃗ sg is the vector in the direction of the goal. This step reward function r(s, a, s ′ , g) = ∥s -g∥ -∥s ′ -g∥ corresponds to the speed-made-good [10] , maximizing (long-term) speed towards the objective. Collisions are penalized with a negative reward r crash .Goal relabeling. In offline data, goals are selected from both future observations in the same trajectory (positives, sampled from an exponential distribution) and observations from other trajectories (negatives, sampled uniformly from other trajectories from the same robot embodiment), as in hindsight experience replay [32] or goal-conditioned supervised learning [34] . During online training, we keep the initially commanded ("on-policy") goals with 50% probability and otherwise relabel with positive goals. When relabeling, rewards are recomputed with the new goal.Privileged critic information. In actor-critic reinforcement learning, it is crucial to have accurate critic values in order to learn a high-quality policy. We again make particular use of the problem structure of goal-conditioned navigation. While at evaluation time the policy does not require a ground-truth goal displacement vector, we can use this vector when training on a completed trajectory by integrating locally-accurate odometry measurements. We take advantage of this property by supplying the critic network (but not the actor) with relative goal vectors ⃗ sg.Dataset. Single-embodiment robot models train on a relatively small dataset, collected specifically for the downstream task and environment. However, including diverse data tends to improve generalization and robustness [21] . Following ViNT [7] , we train on the GNM dataset [27] , a compilation of several robot navigation datasets in distinct environments and embodiments. This allows LiReN to benefit from diverse data on which to train its initial policy and learn generalizable features.In the online phase, we continue to train on the offline dataset but draw half of each batch from the online replay buffer. This ensures that we do not experience catastrophic forgetting of behaviors learned from the original dataset, and generally stabilizes training.We further fine-tune LiReN "on-the-job" during online deployment. In this phase, we equip LiReN with several components to propose goals, avoid getting stuck, and ensure stable operation.Proposing and evaluating goals. To propose goals (represented by images) and evaluate when they are completed, we make use of the spatial structure of navigation. Prior to deployment we map the area to create a graph where nodes represent states (image observations) and edges represent environment transitions. During deployment, when a goal is reached, a future goal image can be selected randomly from its successors in the graph. We typically select goals 10-20 meters into the future according to an exponential distribution. We keep a static goal grpah, though if desired the graph can be updated online as in Shah et al. [11] .Recovery behavior. If the robot becomes stuck in a small area without exploring the rest of the environment, continuing to train on the new data will not increase overall performance. We avoid getting caught in short "goal loops" by biasing the goal graph's construction away from short cycles. Additionally, to avoid the case when the robot is unable to reach any goal because it is stuck or in collision with an object, we inject pseudo-resets [10, 24] to perturb the state when the robot collides with an object (detected by the robot's bumper). When it does not reach its goal for a set period T , it is considered "stuck" and a new goal is selected from the graph. Keepout zones. While exploring the state space, there are some zones in the environment the robot cannot enter for a variety of reasons. To protect privacy, ensure robot safety, and maintain reasonable localization, we mark areas such as bathrooms. When the robot detects that it is in a keepout zone, it triggers a pseudo-reset.Calling for help. If the robot is unable to exit its current state (as measured by repeatedly becoming stuck or crashing at least ten times in a row) it will "call for help" by messaging a human over the internet. While this capability is necessary for truly lifelong learning, we find that it is triggered extraordinarily rarely (e.g. it is physically stuck); typically the combination of pseudo-resets and reward-guided exploration is enough for the robot to escape any local minimum.Battery charge monitoring. To enable indefinite operation for lifelong learning, we continuously monitor the battery's charge. We place charging stations in the deployment environment; if the battery level drops below a set threshold, it will dock for charging the next time it passes a station.Robot hardware. We use an iRobot Create 3 with an inexpensive fisheye camera. Inference is performed with a Jetson Orin NX at 3Hz. We also equip our robot with a LiDAR sensor and a map of the environment for localization, though we emphasize that this is an artifact of our goal selection mechanism, is not used by the policy, and could be avoided with a different choice of training task.Asynchronous training system design. During fine-tuning, training is handled asynchronously by a server (a single Google TPUv4 node) which receives data from the robot in real time and continously updates the robot's model parameters over the network. The training server is robust to unstable network conditions, allowing gracefully recovery from disconnects. 
experiments: 
 We design our experiments to answer the following questions:Q1. Does our combination of offline pretraining with autonomous online fine-tuning yield significant improvements in quantitative and qualitative navigation behavior?Q2. Is offline RL necessary? That is, can we achieve similar autonomous fine-tuning with goalconditioned behavior cloning (GCBC) or via online RL from scratch?Q3. How performant is the offline policy learned by offline RL, in comparison with existing stateof-the-art foundation models for navigation? Encountering the obstacle later during finetuning, LiReN has learned from its experiences and successfully avoids it.We deploy LiReN with our autonomous improvement framework in a real-world office building.Operating in this regime, the robot must contend with diverse scenes, significant changes in lighting, and people working in the space and reconfiguring it as one might expect in an office building. We track two key metrics: (i) success rate, the percentage of goals successfully reached as a moving average over a window of T = 12 minutes (corresponding to roughly the time taken to traverse the whole environment) and (ii) coverage, the percentage of free space that has been explored:1 |F | |∪ t {p ∈ F : ∥p t -p∥ ≤ r}|,where F is free space. It is important to consider both of these metrics, as a policy that cannot escape an "easy" region may have high success rate but low coverage.Using our fine-tuning framework leads to significant quantitative improvements in policy performance during online deployment: LiReN trained online significantly outperforms the base policy.Figure 4 shows a steady increase in policy performance throughout the course of training.We also see specific qualitative improvements during training. Figure 5 shows one such challenging case: the white obstacle is difficult to distinguish from the white walls, and the base model fails to navigate the situation when it is encountered for the first time. However, after a period of online fine-tuning, LiReN successfully recognizes and avoids the obstacle.LiReN GCBC RLPD Why not GCBC? Interestingly, this improvement is not mirrored by attempts to finetune GCBC models [34] . In this setting, we train a policy (via BC) to reach relabeled goals [32] , chosen from future observations from the same trajectory. We find that performing this "iterated GCBC", in which the policy is trained to match its own successful output, causes the policy's performance to degrade over time as shown in Fig. 7 . We hypothesize that this is due in part to distribution shift between the goals on which the policy is trained (from future states) and the "real" goals provided to the policy, which come from the fixed goal graph collected in different circumstances (e.g. lighting).Why not online RL? We also consider the other end of the spectrum: applying online RL with a random initial policy. In this setting, we initialize a random policy and apply RL with Prior Data [31] 0 (RLPD), which trains a SAC-like policy on a mixture of offline and online data. While RLPD learns surprisingly good collision avoidance behavior in a small section of the environment, we find that it strongly overfits to the initial location and is unable to generalize. This is reflected in the coverage metrics shown in Table 1 and in the maps visualized in Figure 6 . We adopt a methodology similar to prior work [27, 7, 35] to compare the quality of the offline policy with existing models along small, controlled sequences of goals (rather than the openworld performance reflected by the first row of Tab. 1). For each environment in which we evaluate offline performance, we collect a fixed sequence of 4-5 short term goals, following evaluations in Shah et al. [27] . We measure each model's progress from a consistent starting point, computed as a percent of the total distance traveled and averaged over ten trials. In addition to indoor experiments, we also provide experiments using the Clearpath Jackal operating in an outdoor setting, demonstrating LiReN's cross-embodiment capabilities. We include diagrams of each environment in App. B.While improving pure offline performance is not the focus of this work, we nevertheless find that LiReN achieves strong empirical performance with offline RL in comparison to state-of-the-art models as highlighted in Table 2 . 
discussion and limitations: 
 We presented LiReN, a navigation foundation models that can learn autonomously in a lifelong setting while deployed. By pre-training with offline RL on diverse navigation data, we get a visual navigation foundation model comparable to state-of-the-art models. By pairing LiReN with an autonomous improvement framework for navigation including goal selection and verification, recovery/reset procedures, and guardrails to ensure continuous operation, we can fine-tune our foundation model to continuously adapt to the target environment, learning to handle changes in the downstream environment without requiring additional human-collected data. We hope that LiReN will lay the groundwork for further study into methods for autonomously fine-tuning robot foundation models.Limitations and future work: our current autonomous improvement framework does have several limits: our goal selection procedure currently requires a notion of localization to find and verify nearby locations on the goal graph, and we currently only support a single fine-tuning task. Additionally, while the robot we used for deployment and online improvement was not itself in the dataset, we did not explore concurrently fine-tuning RL models with multiple new embodiments at once; we expect that this might raise its own challenges. 
 
 Appendix:  


Decision: The paper presents an online learning approach for a foundation navigation model that can be refined as the robot collects more data. 

Strengths:
- Interesting idea and an effective algorithm
- Convincing experiments that clearly demonstrate the impact of online RL for improvement the performance of previously trained models

Weaknesses:
- More details needed for the results
- Clarity on the method can be improved with pseudo-code or a diagram
- Novelty needs to be better described as the general idea of offline RL followed by online finetuning has been proposed before
- Clarify on how the method improve "foundation models" in all its generality 

The reviewers provided valuable suggestions to improve the paper. I encourage the authors to work with the reviewers in addressing their concerns.
===============================
Pos-rebuttal update:

The authors have addressed the concerns reported and the reviewers now agree that the paper should be accepted.

Reviews: review: title: title: Good experiment, but writing is confusing.
paper_summary: paper_summary: This paper presents a new approach for a navigation foundation model. Offline RL is used to fine-tune a general foundation model, followed by online RL for further improvement. Real-world experiment results demonstrate the effectiveness of the proposed module.
main_review: main_review: This paper discusses an interesting problem: how to improve a navigation foundation model during deployment. This approach can serve as a helpful recipe for future research on navigation foundation models.

I find the experiment convincing, as it clearly demonstrates the impact of online RL for improvement and the performance of offline RL pre-training.

However, there are some issues that need to be addressed:

1. Since you are discussing a "Navigation Foundation Model," it is important to show that your method can be deployed on different navigation robots, similar to what Vint is doing.

2. Sections 3 and 4 are somewhat confusing. Your method involves two stages: an offline RL pre-training stage and an online RL fine-tuning stage, conducted in sequence—first offline RL, then online RL. Section 3 gives the impression that you are discussing online RL first, then switching to offline RL, and then back to the online part. In Section 4, it is unclear whether the methods are used for online RL, offline RL, or both. I suggest organizing these two steps (offline-RL, online-RL) in a sequential order and explaining exactly what you did during each step. The offline RL pre-training is a novel part of this paper and should be explained thoroughly.
questions: questions: 1. Since you are discussing a "Navigation Foundation Model," it is important to show that your method can be deployed on different navigation robots, similar to what Vint is doing.

2. I want to see the offline-RL and online-RL parts more clearly explained. Specifically, the methods you mentioned in Section 4—during which stage are they used?

3. I also have a deployment question. From Figure 4, we can see that the performance of navigation improves over time. Does this mean that you are training on the fly, or are you collecting data for 10 minutes and then updating the policy? Perhaps you can provide more details about the deployment process in the appendix.
limitations: limitations: 3
review_summary: review_summary: I like the idea and the experiment of this paper. But I do think the writing for the section 3 and 4 needs to be improved before acception.
title: title: Relevant topic and nice idea
paper_summary: paper_summary: This paper proposes a method to continuously and autonomously improve the capability of navigation foundation models after deployment in a real-world environment. It leverages offline pretraining on diverse navigation data and later adapts the policy on the real robot via sample-efficient online RL.
main_review: main_review: **Quality, clarity, originality, and significance**: 
The paper is easy to read and well-organized, and I was able to follow the arguments made by the authors mostly without problems. This paper addresses an important topic in robotics that is often overlooked: how a robot can continue to improve its capabilities by learning from its own experience after deployment. The authors discuss the motivation behind this work clearly and present relevant real-world experiments. In my view, combining offline pretraining with online adaptation and the use of the "autonomy supervisor" to minimize human involvement are original ideas that can also potentially be suitable for robot modalities beyond navigation.  Though the overall quality of the paper is good, certain aspects need to be improved (see weaknesses, and questions below). 

**Strengths**:
1. The general framework and the idea behind this work are appealing. Combining pretraining on general data with offline RL and adaptation in the real world with sample-efficient online RL is a nice idea that can be relevant to other areas of robotics as well.

2. Reliance on real-world experiments and data.

3. In general, the paper is well-written.

**Weaknesses**:
1. Insufficient details for results (please see questions for rebuttal).

2. A single architectural diagram to describe the framework would help the reader quickly grasp the different components of the proposed framework. This would be easier than deciphering separate figures (e.g. figs. 1,2,3). Also, the textual description of the overall system would benefit.

3. A concise description of LiReN's training algorithm, i.e. in the form of a pseudocode listing, is missing (for both offline and online RL). It may help to improve the clarity of the paper.

4. I appreciate that the authors have provided code and data, but the supplementary material should also describe network architectures, model sizes, hyperparameters, etc. in text/tabular form. 

5. Multiple equations are not numbered (e.g. on pages 3, 5).
questions: questions: 1. How many independent trials were used to calculate the results reported in Figs. 4, 7, and tables 1 and 2?

2. What is the effect of the size of the dataset used for pretraining via offline RL? How much data is sufficient?

3. Line 119: "... some additional information about $s_t$ becomes knowable later" - Why does this information become available later? When does it become available?

4. Line 116: The authors write "In LiReN’s implementation, we use a combination of all three of these techniques ..." but the previous line mentions 4 techniques: clipped double-Q, ensembling, entropy regularization, and critic normalization. 

5. How is a crash detected?

6. Why do you not consider any penalty in the reward formulation (e.g. a penalty for crashing)?

7. Concerning Fig. 7, why does the performance of GCBC collapse after improving initially?

8. Do you map the environment manually before deploying the robot? Please mention this in the paper.

9. In Table 1, how do you calculate the coverage percentage?

10. What are the differences between "Env A", "Env B" and "Env C" in Table 2?

11. Is it a good idea to perform online training continuously? There is a difference between finetuning and lifelong/continual learning. For example, suppose the robot trains during the day (with some natural light) and then continues to adapt (i.e. finetune its parameters) to artificial light at night. Does it need to readapt to natural light the next day? As there is no mechanism to prevent forgetting, I would assume the answer to this question is yes. This is possibly something to look into in the future but I would be interested to know the authors' view on this.

12. *Minor*: Line 164: Is this a typo? "... it triggers a minimal reset which backs away it is back ..."
limitations: limitations: 3
review_summary: review_summary: I like the proposed idea and feel that the experimental results mostly support the claims made by the authors. At a high level, the framework proposed in the paper is quite clear, but it is somewhat difficult to completely grasp all the details because of the absence of a concise description of all the training procedure. Some aspects of the results also need further clarification (e.g. how many independent trials were conducted).
title: title: Good contribution to lifelong robot learning, but somewhat "oversold"
paper_summary: paper_summary: The submission proposes a pipeline for lifelong learning for robot navigation by starting from a pretrained model. In particular, the method uses large-scale data to pretrain a policy and Q-function via offline RL (using conservative Q-learning; CQL [28]0), and then uses the learned models as a starting point for adapting to a set of real-world navigation tasks. The pipeline is fully autonomous: it provides goals for the robot and recovery mechanisms to avoid the need for human interventions except in the most extreme cases where the robot is physically stuck. The experiments demonstrate improvements over existing methods, which also serve as ablation studies for the proposed approach.
main_review: main_review: ######## Update after rebuttal ########
Per the discussion with the authors below, I am updating my recommendation to "Strong accept".

######## Strengths ########

1. This paper is, to my knowledge, the first plausible instantiation of a lifelong learning method on a physical robot.
    - Not many existing works have attempted to learn continually over long deployments in robots. This is the most successful version of such attempts that I am aware of. I do encourage the authors to contextualize their work with respect to such existing attempts (e.g., [1], [2])
    - Compared to those existing works, this submission studies a setting that is both autonomous (there is no human to provide ground-truth rewards or reset the environment at will) and online (the agent learns during deployment while it attempts to solve the tasks that it is intended to solve)
2. The primary technical strength of the work lies in the design of a pipeline that enables _fully autonomous_ learning on a physical robot in a natural environment.
    - What makes this work promising toward true lifelong robot learning is that the authors carefully design a pipeline/framework for completely autonomous learning/evaluation. The framework proposes goals, constructs rewards measured from the agent's own sensors, incorporates resetting, battery charging, error recovery, and (as a failsafe) "asking for help." 
    - The closest design I had seen to this was in CMU's Cobots [3], which the authors should also likely cite. Critically, [3] was not a _learning_ robot, but simply a long-term deployed robot. The current submission steps this up by introducing all the relevant mechanisms to use during online learning. 
3. The experiments are well designed and demonstrate the superiority of the proposed approach over well-chosen baselines.
    - The experiments focus on showing that pure offline training is not enough, that methods pretrained via behavior cloning instead of offline RL underperform, and that pure online training is comparatively slow.
    - The authors include two metrics: success rate over the proposed goals, and space coverage. This is well appreciated, because success rate on its own could be misleading (and, as a reader, I likely would have missed it if not because the authors included the space coverage metric). The key point here is that since the pipeline itself proposes local goals, if the agent fails at goals "on the border" of the local region, then it might never be able explore beyond, while still being able to achieve the "internal" goals within the explored region. It might be worth stating this more plainly up front.
    - Overall, the results are strong, though it seems necessary to include additional details (as described below)

######## Weaknesses ########

1. The authors claim that their approach improves foundation models via lifelong learning, which is much more general than what their method is designed for (or demonstrated to achieve).
    - This is my main concern: I believe that the authors should rephrase their contribution away from "improving robot foundation models."
    - The purpose of a foundation model is to be a general-purpose model to work across a variety of robot morphologies, environments, and objectives. Improving a foundation model would require addressing these dimensions both in the method and in the evaluation, neither of which are done in this submission. 
    - The authors vaguely hint at this in the final section: "we do not support fully multi-task foundation models" and "we did not explore concurrently fine-tuning RL models with multiple new embodiments at once without experiencing forgetting." But this is insufficient: I encourage the authors to rephrase the contributions of their work to more precisely characterize them.
    - In particular, the proposed method permits "adapting", "finetuning", or "specializing" a foundation model to a particular embodiment-environment pair throughout a lifelong deployment. This is already, in my opinion, a strong contribution. 
2. The general idea of "offline RL -> online finetuning" is not novel (though the details of how it is deployed might be).
    - This somewhat limits the novelty of the contribution, though there is still sufficient technical novelty for publication.
    - While the authors cite [22] as an existing approach that pretrains a model via offline RL and adapts it online on a physical robot, I think it might be worth directly stating that the whole motivation for the existence of the offline RL field is precisely to train models on large-scale data. While today the "popular" technique is to use behavior cloning, large-scale pretraining has long been proposed as a key setting for offline RL. I encourage the authors to cite relevant literature in this space and contextualize their contribution accordingly.
3. The method description is insufficient to understand the goal relabeling scheme, which I believe to be a key component of the proposed method.
    - The authors mention a couple of times that they relabel states/goals, but they never explicitly describe this.
    - "we only relabel 20%...", "we _do_ have access to this information when relabeling states..." --> this isn't made explicitly clear, which is why I'm confused: do the authors use something like hindsight experience replay [4] to make some of the online data look like positive data?
    - I encourage the authors to include an additional paragraph in Sec 4 that states what the relabeling process is.
4. There are several missing details to fully understand the experimental settings.
    - "Interestingly, this improvement is not mirrored by attempts to fine-tune goal-conditioned behavior cloning models" --> in prior work or where? Or do the authors mean the results in Fig 7, which is never referenced? This sentence isn't clear. 
        - Assuming that this means Fig 7, how is the behavior cloning model finetuned without demos? Do the authors use the attained goals as the true goals for training?
    - "pure online RL without a base policy (but taking into account prior data)" --> how is prior data "taken into account"?
    - The whole setup of Sec 5.3 is quite unclear -- is "offline performance" the performance of the pretrained models deployed on the robot online without any finetuning? Why is the performance of LiReN different from that in Table 1 at 0h?

[1] Sutton et al. "Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction." AAMAS 2011.

[2] Isele et al. "Lifelong learning for disturbance rejection on mobile robots." IROS 2016.

[3] Biswas and Veloso. "Localization and Navigation of the CoBots Over Long-term Deployments." IJRR 2013.

[4] Andrychowicz et al. "Hindsight Experience Replay." NIPS 2017.
questions: questions: Summarizing the points above, here are the main questions/suggestions I have for the authors:
- Please include the relevant missing references (and discussions about their connection to this work)
- Please scope the contribution appropriately in terms of specializing or finetuning (instead of the more general "improving")
- What is the exact relabeling process used by the authors? Is this novel or taken from existing work?
- Please answer my questions above about the experimental setting

######## Additional feedback ########

The following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.

Abstract
- It is unclear at this point what "a complete system" is

Intro
- Either define "behavior cloning (BC)" and use "BC" thereafter, or drop "BC" and stick to "behavior cloning"
- "a critic $Q(s,a)$, which requires different feature-learning capabilities than behavior cloning" --> could the authors argue this point? Is there existing literature that studies whether this is true, or is there some obvious reason why features learned for BC couldn't be used for Q?

Sec 2
- This is the second time the authors mention a "supervisory framework to enable autonomous learning", but this doesn't really mean much on its own. It might be worth to include ~1 sentence in the intro that broadly describes what this entails.

Sec 3
- This section seems quite precise, which is nice.

Sec 4
- I suggest separating Sec 4 into two sections: one focusing on the non-learning aspects that are necessary for safe long-term deployment (roughly p to line 178) and the learning aspects (from line 179 on, plus maybe the goal proposal)
- There's a sense in which the goal proposal really is just a "trick" to enable evaluating these ideas. In the real world, we would want the robot to learn from actual goals that users intend for it (or goals that it sets for itself with some real higher-level objective, like surveillance). It might be worth discussing this point. More generally, this section would benefit from additional clarity around which aspects are necessary for training, for longterm autonomy, or for evaluation.
- I didn't really understand the "privileged critic information" paragraph, but I think this relates to the relabeling.
- "we emphasize that this is not necessary for downstream deployment" --> this is a really odd comment in a lifelong deployment setup. I think the authors mean that the LiDAR is only used to set the goal, which is only an artifact of the evaluation setting and not the actual learning setting. 

Sec 5
- I like the design of fig 4, like a continuous stacked bar chart
- What is the cause for the oscillating pattern in the performance improvement?
- The fact that RLPD obtains 46% success rate but only covers 27% of the space suggests that success rate over the distribution of proposed goals is not on its own a great metric of performance. But including the results on coverage addresses this.
limitations: limitations: 3
review_summary: review_summary: I lean toward recommending acceptance for the submission. While in terms of learning the contributions are not significant, they are in terms of proposing a complete pipeline that enables learning with existing methods within a physical platform. The authors should primarily address two concerns: delimit the scope of their contribution more precisely, and describe more clearly the details of their goal relabeling scheme.