SELFI: Autonomous Self-Improvement with RL for Vision-Based Navigation around People
Authors: Noriaki Hirose
Dhruv Shah
Kyle Stachowicz
Ajay Sridhar
Sergey Levine
Venue: CoRL 2024 (2024)
Keywords: online reinforcement learning
vision-based navigation
TL;DR: Online model-free RL coorperating offline model-based learning objective and its application to vision-based navigation

Abstract: Autonomous self-improving robots that interact and improve with experience are key to the real-world deployment of robotic systems. In this paper, we propose an online learning method, SELFI, that leverages online robot experience to rapidly fine-tune pre-trained control policies efficiently. SELFI applies online model-free reinforcement learning on top of offline model-based learning to bring out the best parts of both learning paradigms. Specifically, SELFI stabilizes the online learning process by incorporating the same model-based learning objective from offline pre-training into the Q-values learned with online model-free reinforcement learning. We evaluate SELFI in multiple real-world environments and report improvements in terms of collision avoidance, as well as more socially compliant behavior, measured by a human user study. SELFI enables us to quickly learn useful robotic behaviors with less human interventions such as pre-emptive behavior for the pedestrians, collision avoidance for small and transparent objects, and avoiding travel on uneven floor surfaces. We provide supplementary videos to demonstrate the performance of our fine-tuned policy.

SELFI: Autonomous Self-Improvement with RL for Vision-Based Navigation around People
Noriaki Hirose!.2, Dhruv Shah', Kyle Stachowicz', Ajay Sridhar', Sergey Levine 1 University California Berkeley 2 Toyota Motor North America noriaki. wirki.irrose www.ssebereeley.cc.ooo edu
Abstract: Autonomous self-improving robots that interact and improve with ex- perience are key the real-world deployment of robotic systems. In this paper, we propose online learning method, SELFI, that leverages online robot expe- rience to rapidly fine-tune pre-trained control policies efficiently. SELFI applies online model-free reinforcement learning on top of offline model-based learning to bring out the best parts of both learning paradigms. Specifically, SELFI stabilizes the online learning process by incorporating the same model-based learning ob- jective from offline pre-training into the Q-values learned with online model-free reinforcement learning. We evaluate SELFI in multiple real- world environments and report improvements in terms of collision avoidance, well more socially compliant behavior, measured by human user study, SELFI enables us to quickly learn useful robotic behaviors with less human interventions such as pre-emptive behavior for the pedestrians, collision avoidance for small and transparent objects, and avoiding travel on uneven floor surfaces. We provide supplementary videos to demonstrate the performance of our fine-tuned policy on our project page
Keywords: online reinforcement learning, vision-based navigation
Introduction
Reinforcement learning (RL) provides an appealing algorithmic approach for autonomously improv- ing robotic policies in unpredictable and complex real-world settings [1-4]. For example, in the in- door navigation scenario depicted in Fig. the robot needs to not only avoid obstacles, but also deal with unpredictable and hard-to-model situations, like the interaction with the pedestrian. Model- based control methods can struggle with the unpredictable elements in the scene [5-7], and RL in principle provides an appealing alternative: learn directly from real-world experience, sidestepping the need for highly accurate modeling. However, directly performing end-to-end RI. from scratch in the real world can be difficult: discovering a high-quality policy may require a large number of trials and encounter catastrophic failures during the training process [8-10]. This is especially prob- lematic when not possible to provide external instrumentation that avoids catastrophic failures - for example, with human interactions where failures might be inconvenient even dangerous.
In this work, we propose framework for robotic learning that aims to address this challenge by utilizing model-free RI fine-tuning on top of learning-enabled model-based policy that is pre- computed offline. Our approach initializes the robot from a policy that already exhibits basic com- petency in its environment, and from there further improves its behavior in the particular setting where it situated through trial-and-error. While number of prior works have examined the use of model-free RL fine-tuning strategy top of pre-trained pre-computed policies [11-14), this is often complicated by the fact that initializing sample-efficient model-free RL methods requires not only policy but also a critic [3, 4, 15-18]. The policy initialization can often be derived from a prior policy (either classic or learned), but in modern actor-critic methods, the policy is trained
sites htt.ee.ccom/rwwe/efiiii..-
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

rapidly to maximize the critic's 's value, and this quickly overrides any actor initialization if the critic is not also pre-trained [3, 4, 17, 18). Our key observation is that model-based methods that maxi- mize some sort of planning objective can also be used to initialize the critic, such that model-free RL Q-values are given by linear combination of learned model-free critic and a model-based trajectory value estimate. In this design, as long as the critic is initialized to produce small values, the RL process starts off by maximizing the model-based value estimate (i.e., model- control), and then improves further through trial-and-error interaction.
We illustrate this design in Fig. 1, and we instantiate our system in the con- text of social navigation: the prob- lem of navigating an indoor space while avoiding undesirable behavior around pedestrians, such as inter- ruptions, collisions, or invasion of their intimate distance. This problem domain is a good fit for validating our framework because model-based policies can be derived from geomet- ric models of the world and rough predictive models of pedestrians, but these policies can be significantly improved through model-free trial-and-error, both because the pedestrian models might be inaccurate, and because the robot can adapt directly to the behavior of the pedestrians in a specific downstream deployment environment. In this setting, our model-based policies are derived from the previously proposed SACSoN [5], which constructs policies by opti- mizing a trajectory value using a 3D reconstruction of training environments and predictive models of humans. This model-based procedure also provides trajectory value estimates that can bootstrap the model-free RL critic. During the real-world model-free RL phase, we improve the model-based policy by learning residual value critic and applying actor-critic methods as described above.
Workstation
1" stop: Model bared learning on offline Maniricau
J
Model
stap: Our online learning SELF!
Rooor
Manintiiia
JJ J Q
Figure Overview our proposed online learning system, SELFI. Our method fine-tunes a pre-trained control policy trained with model-based objective by incorporating this objective into a Q-value function to maximize during online model-free RL.
The main contributions of this paper is to propose a framework, SELFI, that takes advantage of the best aspects of online RL and offline model-based learning. Specifically, SELFI uses online Q-learning to fine-tune a control policy trained with offline model-based learning. SELFI rapidly improves the performance of pre-trained policy in the target environment without needing sig- nificant human intervention during online learning. In our evaluations, SELFI improves the per- formace of the pre-trained policy in multiple vision-based navigation tasks, greatly outperforming policies trained purely offline as well as standard end-to-end model-free offline-to-online RL fine- tuning methods. Within only two hours of fine-tuning, our policy learns complex robotic behaviors, e.g. pre-emptive behavior for navigating around pedestrians, collision avoidance for unseen small and transparent obstacles, and preferences for smooth and easily traversable surfaces. The robotic behaviors learned by SELFI are shown in the supplemental videos.
2 Related Work
We review the related learning methods as well as the navigation methods on the evaluation task.
Online learning: There are various data-driven methods for adapting control policies their real- world environments through interactions. In learning- based settings, DAgger [19] is general frame- work that iteratively trains control policy with expert labeled demonstrations. Additionally, model- based learning can fine-tune control policies by utilizing differentiable dynamic forward models to define an objective function with visual foresight [20-22], reward prediction [23], and state predic- tion [5, 24. 25]. This type of learning can be combined with an optimization algorithm to generate action commands online, just as in model predictive control [26-29]. However, for any model-based approach, the performance of the learned control policy is limited the accuracy of the model and the quality of the dataset. Model-based RL, which learns model from interactions with the real world, is subject to similar limitations [30-32].

---

Model-free RL accumulates data, including rewards from real-world interactions, and trains a con- trol policy to maximize the expected sum of discounted rewards from future timesteps [1-4]. Al- though model-free RL does not suffer from modeling errors, running model-free RL from scratch requires a significant amount of time for for data collection and learning. In addition, certain interac- tions with the environment can be dangerous for the robot itself and humans. Prior work has studied how offline learning addresses this issue by pre-training policies in simulation [33, 34] in the real world using behavior cloning [25, 35, 36] or offline RL [11, 12, 37, 38). Hybrid approaches use learned dynamics model or a control policy learned with model- RI to initialize a model-free learner, making it more sample efficient [39, 40J.
Our proposed approach is closely related to Residual RL [14], which decomposes the policy's 's output into a solution from an existing controller and the actions from a residual policy trained with model- free RL. By leveraging the existing controller, residual RL stabilizes the robot's 's behavior during the early stages of online learning and learns the target behavior via the flexibility of model-free RL. However, the confusion between the existing and learned control policies due to composing them in the action space restricts the performance of residual RI. Different from these previous works, our proposed method, SELFI, seamlessly composes model- -baed learning and model-free RL in the objective space. SELFI a flexible and stable method for online fine-tuning with model-free RL because it incorporates the objective used in offline model-based learning in the learned Q-values.
Social navigation: Social navigation has been extensively explored in [41-43]. Model-based ap- proaches relying on dynamic pedestrian models have been utilized for behavior modeling [6, 7, 44 46). Many existing techniques determine the robot's actions based on predicted pedestrian behav- ior [47-53]. Furthermore, social navigation has also been explored using model-free learning ap- proaches, such as reinforcement learning [54-60) Unlike vision- navigation [21, 25, 61-63] as our evaluation task, which only uses RGB camera observations, these approaches rely on detected pedestrian poses and/or multiple metric sensors such LiDARs and/or depth cameras [64, 65].
The most related work, SACSoN [5], utilizes the model-based learning architecture to learn a so- cially unobtrusive policy in dision-based navigation setting. However, even simple socially compli- ant behaviors are still challenging to learn due to the modeling errors. Hence, we apply our proposed method, SELFI, fine-tune the pre-trained SACSoN policy to improve its performance. We con- duct comparisons with baseline methods as well other online fine-tuning methods in the same task setting, vision-based navigation our method.
3 Combining Model-based Control with Online Model-Free RL
3.1 Preliminaries
We apply our hybrid model-based and model-free learning algorithm to a Markov Decision Process (MDP) M. We first briefly explain model-free RL and model-based learning, respectively.
Model-free RL: In RL, we want to maximize the expected sum of discounted rewards. Q- learning algorithms [4, 66] solve this task by learning a function approximating (TTT(ss, a) = -(s,,a)~-- Din 7'r(s,, az) where trajectories with states & and actions a, are sampled from the closed-loop dynamics of M under the policy kg. 7 indicates the discount factor for future rewards r. Here, mathematical symbols without subscript representing time show the current time state and action, e.g., s so. = ag. The Q-function for the optimal policy obeys the Bellman equation, and it can be trained to minimize the TD error 8 r(s, a) + + max @=*(s' a') QT (s, a) where s'' and a' indicate the next step and a, respectively. In the actor-critic setting, we learn both an approximation for the action-value function (s, a) and for the policy that maximizes the action- value function Qss, a) as (s)... This enables tractable optimization over large action spaces where the max in the Bellman equation cannot be efficiently computed.
Model-based learning: In the model-based learning setting, we optimize objective over open- loop sequences of virtual actions {āc)t=o...H- using approximate dynamics model and
3

---

reward estimate as arg max)= [((.T) E'to'' (f((S,, āc) s.t. 8111 = f(ee, āz) where St is the predicted state under the approximate dynamics described by f. which can be either learned or given. åe is the 1-th virtual action, which is estimated at the current time state. Note that 8n - and 0n = a in Js,, T). We assume that the dynamics $ = f(,,, âe) and rewards f(3,, âe) are easy to compute and differentiable, allowing us to directly optimize the sequence of actions online with gradient-based trajectory optimization [26, 28]. Instead of optimizing this objective online (analogous traditional nonlinear model-predictive control methods), model-based learning amor- tizes this optimization by learning the parameters 0 of control policy T = (s) to optimize this objective offline. This distills the optimization problem into an offliee base policy To. represented as a neural network mapping observations to sequences of actions т. At runtime, actions are then sampled from the learned policy kg. A number of prior methods have proposed similar methods for model-based learning [20, 21, 23, 25].
To(S)
3.2 SELFI learning architecture
In SELFI, we combine the strengths of model-based learning with the strengths of model-free learning to enable finetuning in the real world. We wish to decompose the critic value Q into a model-based objective J and a learned residual objective Q as Q(s,T) = J(s T) + (s,,a) where a is the first action âg in the sequence virtual actions T = . - We train Q(s, a), which corresponds to the part the overall objective that is not considered by model- based learning. We fine-tune tne-tune the pre-trained control policy (s(s) to maximize the combined Q- function (s, Accordingly, we first assume r+*t; ae) in timesteps 0... H has the form, r(st, ax) f(3,, ât) + F(sr, at) where f($;, âz) is the model-based reward and F(st, oz) is the un- modeled residual reward. Hence, we have:
H-1
H
Q(s,T) = (r) Σ 7/s,, ax)
Σ t=
Fss 0.) + Σ )r)
(1)
t=0
1=0
t=
J(s.7)
Q(+.a)
The model-free Q includes i) the reward terms that cannot be directly modeled and ii) the value of the long-horizon returns that are ignored by the limited-horizon model-based learning. As usual, Q(s, r) should obey the Bellman equation Q(s.T) = r(s, a) + 7Q(ss, (o(s)) and the critic for Q(s,a) a) can be trained to minimize the TD error 8:
y + r(s, a) + (Q(ss, (s)))): 8 - Q(s, T).
(2)
In practice we use a delayed copy of Q as a target network to compute target values [18]. During online learning, we calculate the gradient of Ty by mack-propagation to maximize the hybrid objec- tive Q(s, r). By including the model- based objective term [(s, 7) in Q(s, T), the initial estimate of Q(s, T) already provides reasonable values with a suitably initialized Q(s, a) (e.g., with small initial weights), which significantly stabilizes performance early in training. Then, Q(s, a) can be further trained with online interactions to learn the target robotic behaviors.
3.3 SELFI implementation
As our underlying RL algorithm, we use a variant of twin delayed deep deterministic policy gradi- ents (TD3) [18], which a sample-efficient and stable algorithm for training deterministic control policies. Fig. I shows the system overview of SELFI for real-world learning. Similar to [13], policy training is implemented on an workstation while the robot's onboard computer is used for inference.
On the workstation, we train with batch data from both the offline and online dataset to avoid over- fitting to the online data. Half of the batch data is chosen from the offline dataset, and the other half from online data, which is collected by the robot. We update the actor once for every two updates to the critic, maximizing Q(s, r) = T) + Q(s, a) with respect to the policy.
On the robot, calculate the learned policy to obtain the sequence of actions and execute the first action dg. adding Gaussian noise to encourage exploration. Then, the robot
{āg f=)...

---

step Offline model-bazed learning. SACSoN
step: Our Online Jearnins. SELE!
Critic
GOD Inags 1
Encoder (rosse)
Actor
Model
Model
Murch
Jasces
Iluron
Figure 2: SELFI architecture overview, Before online learning, we train the encoder and the actor by maximizing the differentiable model-based objective. In the online phase, we combine the oftline objective with the learned QQ Q-value from model-free RL fine-tune the actor.
sends the collected data the workstation, where it is stored in the replay buffer. Periodically, the updated actor weights sent from the workstation to the robot so it can use the latest policy.
4 SELFI System Setup
We evaluate SELFI on for a bisond social navigation task, where a robot navigates an indoor environment with pedestrians. We employ SACSoN [5] the offline model-based objective. In the online phase, SELFI fine-tunes the pre-trained policy learn socially-compliant behavior including 1) pre-emptive avoidance of oncoming pedestrians, 2) collision avoidance for the small or transpar- ent objects, and 3) avoiding travel on uneven floor surfaces. These behaviors are difficult to learn purely from offline model-based learning due to the modeling errors and insufficient information in the offline dataset. this section, we describe the implementation of SELFI on top of SACSoN.
4.1 Offline Learning with SACSoN
We briefly describe the learning procedure in SACSoN. Details are shown in the original paper [5].
Offiie model-based objectives: As shown in Fig. 2, represent the control policy by en- coder 96 coupled to actor network TT. We train the policy to maximize the model-based ob- jective [5], Jsacson (s, 7) = - - ,pose + fge0n + pppd + 1, where the frose reward encourages goal-reaching behavior, fason [25] penalizes collision with static obstacles (via signed point-cloud distance), pped [5] is to learn socially unobtrusive behavior and freg acts regularization term to encourage smooth motion. All objectives in Jsacson are differentiable with respect to the action sequence 7 and can therefore calculate the gradient of go and via and learn them.
The offline policy нæ and Jo are trained on the 80-hour HuRoN dataset [5] for dision-based navigation including over 4000 human-robod interactions. Observations consist of 2-second sequence of six 128x256 omnidirectional camera images from a Ricoh Theta S, together with the goal image 19, and predict sequence of eight future actions. To allow the critic to handle any actuator delays in the system, we concatenate the previous action with the extracted image features as shown in Fig. 2.
Offline training:
4.2 Online Learning with SELFI
We demonstrate an instantiation of SELFI for the vocially-complian navigation, following Sec, 3.2 and Sec. 3.3. Here, we describe the specific procedure we use for online RI. finetuning.
Learning setting: To obtain the target robotic behavior fast as possible during online learning, SELFI fine-tunes the actor me with frozen encoder Io shown in Fig. 2. It corresponds to defining the extracted feature from the encoder feature s and the sequence of action 7 to the critic, shown in right part of Fig. 2. By sharing the encoder with the actor, we efficiently learn both actor and critic. In the TD error calculation, we assume J(8,7) T) RS [s'', 7') to train the small critic network during the brief online learning phase.
in formulation of SELFI. To estimate Q, we feed the
Reward design: The reward is designed as r(s, a) T ÿ + C, + C. to incentivize smoothness and progress towards the goal, while avoiding collisions: The first term maximizes velocity towards the next goal, where = (x ug. 0,) and ग7 = [v, 0, w] are goal direction and velocity vectors
5

---

Table 1: Closed-loop Evaluation of trained control policies. IDV is intimate distance violation duration, NCO is hear-collision duration, and UFS is duration on uneven floor surface, CP is the number of collision for pedestrians, CO is the number of collision for the tiny objects, Int is the number of interventions by teleoperators, SPL is Success weighted by Path Length [70] and STL is Success weighted by Time Length [71]. SPL and STL are calculated by assuming that the robot reaches the goal position even regardless of there being human intervention. * indicates using the ground truth goal pose for generating the velocity commands.
Method
IDV [s] L
NCO [s]
UFS [s]
CP [|4)
CO L
[4]
Int 1 [0]]
SPL
STL
Sampling-based motion planning*
21.512
11.633
10.212
1.333
3.333
0.733
0.808
0.652
Residual RL
28.731
12.365
8.073
3:200
4.333
2.067
0.817
0.641
SACSoN with fine-tuning Ours
18.018
11.229
3.578
0.800
2.667
0.733
0.838
0.59%
7,978
3,938
3.067
0.200
1.400
0.267
0,918
0.739
expressed in the robot's ss current frame [13]. To obtain the local goal pose 9. we build an approximate localization system that incorporates visual odometry and AR markers along the robot's 's trajectory.
The specific choice of AR markers a design decision to facilitate easy online learning, and other mechanisms for localization based on visual odometry(see Appendix F). v and w are the linear and the angular velocity commands for the two-wheelddrive robot in åg- We use C, and Cs denote the rewards for avoiding static obstacles and dynamic obstacles (pedestrians) respectively:
-Ce
(if collision is True)
C. c.-1-c0
-Ch (if dv <0.5 r,) (otherwise)
Cb
(else if bumpy is True)
Ca =
0.0
0.0
(otherwise)
Here, Cc. Cb, and Cn are positive constants that penalize undesirable behaviors. We set C. Cp 0.3 and C₁ 0.1, and do not tune these values for our experiments.
The robot triggers a collision event using the robot's bumper sensor, and a bumpy event (caused by an uneven floor) when the measured accelcration exceeds a fixed threshold. Note that these sensors are not mandatory and can be substituted by the other sensors commonly used navigation [13]. To detect intimate distance violations, we estimate the distance d, the closest pedestrians using combination of semantic segmentation [67, 68] and monocular depth estimation [69]. r, 0.5 m is the radius of the circular robot footprint with margin.
Others:
We set the discount factor y = 0.97, accounting for long trajectories of human- robot inter- action behaviors. addition, the workstation sends the policy model parameters to the robot every 50 training steps (approx. minute wall clock time). Since we conduct online learning for maxi- mum of two hours, the maximum number of training steps is about 6000. On online learning, we set the batch size 76. The learning rate of Adam optimizer is set as 0.0001. All other parameters follow the authors' implementation of TD3 [18] and SACSoN [5].
5 Evaluation
Our experiments evaluate SELFI in the real world, studying the following research questions:
Q1. Does SELFI lead to better final policy performance than existing approaches?
Q2. Does SELFI reduce interventions during the fine-tuning process (without degradation)?
Please see Appendix for more details on a prototype robot and whole navigation system for our evaluation. And we show the details of our online training and evaluation setup Appendix
Performance analysis of the fine-tuned control policy: We compare to the strongest baselines, Residual RL and SACSoN with fine-tuning. We fine-tune our method as well as these baselines and evaluate the fine-tuned policy. In addition, we evaluate Sampling-based motion planning as the classical motion planning. The details of these baseline methods and the online training setups are shown in Appendix B and G. In evaluation, it is difficult to conduct extensive and repro- ducible comparisons the highly populated natural environments, because they are uncontrolled, with pedestrians walking and out of the scene at random. Therefore, to conduct more controlled and reproducible comparison with each of the baselines, addressing Q1, we focus on the "organized" environments specifically.

---

Residual RL
SACSoN with fine-tuning
Ours
10
1/
did rabol's batawiar??
the thenobett bahasier? did heel reur rebot?
robol's movements odhere to Nurnon social narmal
have
behawe?
robut?
personal spoce?
nakod'' movemnnnss the roborts
Figure 3: Evaluation of socialness human rating. The height of each bar indicates the and the range line indicates standard deviation. Larger is better ratings. * indicates statistical significance of our method on the t-test with < 0.05. indicates scale reversed for analyses.
Dumn
SACSOR with Bre-dusing
Ouri
Sustng
[b) cosc E
Figure 4: Visualization the robot behavior when interacting with the pedestrian. Robots and pedestri- ans with the same color indicate the same time. The black dotted lines indicate the time of closest proximity.
run our robot five laps each in three different environments, with a control policy fine-tuned by our methods and two selected baselines. For more details regarding the experimental setups, please refer to the appendix C.I. Table shows quantitative analysis with method and selected the three strongest baselines (The measured metrics are shown the caption).
Our method has the best scores for all metrics. In particular, our method reduces IDV and NCO by more than 50% against the strongest baseline and increases SPL and STL by about 10 %. and CO are improved by adding a negative reward for collisions detected by the mechanical bumper sensor. Although we penalize the collision in the model-based objective, it based on the estimated poses, which less accurate than using the bumper sensors. Hence, learned values signif- icantly improve CP and CO. The reward design for the pedestrians can help prevent the robot from stacking in front of the pedestrians, which may worsen STL. Moreover, online learning could improve basic goal-reaching performance and reduce human interventions. These positive aspects suppress the undesired deviations and significantly improve the SPL.
To evaluate how well our method behaves around humans, we also conduct additional experiments with twelve human subjects. We recruited twelve subjects from among graduate students, visiting scholars, and staff members campus. We consider the balance to be as diverse in gender (6 male, 6 female), professional-leve (7 student, 5 non-student), and origin (4 North America, 5 Asia, Europe, Latin America) as possible. For more details, please refer to the appendix C.2. Following [72], subjects were asked nine qualitative questionnaires [72] with 7-point Likert scale from "Not all" to "Very much" after each experiment. To mitigate bias in favor of the first method, we present the questionnaires [72] in Fig. before running evaluations with the first method.
Figure 3 shows the means and standard deviations of the scores from this study. Note that we flip the score for the last question such that higher is better for the entire bar plot. Residual RL often violates the intimate distance and occasionally collides with the human subject. As the result, Residual RL has the worst score. SACSoN with fine-tuning shows similar behavior in navigation and the closest score our method. However, our method performs better overall. The statistical significance of method is confirmed in four questions, (4), (6), (8) and (9) on the t-test with p 0.05.
Figure 4 visualizes the robot behavior when the pedestrian and robot pass each other. In these time lapse illustrations, the color of the pedestrian and robot indicates the timestep (i.e., a yellow pedestrian and a yellow robot indicate the same point in time). In addition, we use a black dotted line for the robot and the pedestrians when robots and people are in closest proximity. all cases, the pedestrian gets stuck in front of or behind the robot running the baseline method (right), and the robot penetrates the intimate distance for extended length of time. Especially in case A and D. the baseline fails to reach the goal position and collides with obstacles. With our method, an

---

evasive maneuver is initiated at an early stage and succeeds in smoothly passing pedestrian without getting stuck (left). Although the robot is close to the pedestrians when passing, our control policy minimizes how long the robot penetrates the pedestrian's 's intimate distance.
Figure 5 shows the time lapsed images when avoiding the small unseen objects and the un- even rubber mat. Our method naturally avoids colliding with the small objects, though this presents a challenge for the initial SACSoN policy. In addition, our methods avoids travel- ing on the uneven mat since we give a negative reward C. during online training. Please
[a] case A
[al osee B
Figure 5: Visualization of the robot behavior of avoidance for small obstacles and uneven floor mat.
supplemental materials for more details.
Interventions during online training: Reducing human interventions is important in online train- ing to enable autonomous adaptation in the real world. However, it known that the data distribution shift between offline and online training causes performance degradation, leading navigation fail- ures and a lot of interventions to keep online learning in navigation. To evaluate the online training process, we count the number of human interventions in each navigation loop for Q2
Figure 6 shows mean and standard deviation of the interventions in three environments. Our method gradually decreases the number of in- terventions during online training, and it almost reaches to zero at laps. It means that SELFI can consistently improve the performance with- out degradation. On the other hand, the number of interventions increase for the basclines over the first few laps. Afterwards, the baselines de- crease the number of interventions. However, the baselines still need a few human interven- tions to complete navigation.
the inteventtin
-e-Ourt -a- SACSoN with fine-tuning -0- Residual
baps
Figure The number intervention on online learning in three different environments. The lines indicate the mean and the areas indicate the standard deviation. The horizontal axis indicates the number of times the robot laps the loop reference.
Please additional evaluation with more baselines in Appendix E and supplemental videos.
6 Discussion
We proposed online self- iiff-iippoiing method, SELFI, to quickly fine-tune control policy pre- trained with model-based learning. SELFI combines model-based learning and model-free RI. its training objectives to take advantage of the best parts of both approaches. The same objectives used in offline learning are introduced into online learning to stabilize the learning process. The performance of the pre-trained policies are improved via Q-functions from online model-free RL.
In the evaluation, SELFI was implemented to fine-tune the SACSoN policy [5] for vision-based nav- igation. SELFI enables us to quickly learn complex robotic behavior, such as pre-emptive collision avoidance for pedestrians, collision avoidance for the small and transparent obstacles, and prefer- ences for traversing on smooth surfaces. These behaviors are difficult to learn on offline training due to the modeling errors and data distribution shift. In addition, compared to various baseline methods, SELFI did not require much human intervention during online learning. The performance of the trained control policy by SELFI also visualized the supplemental materials.
While our method enables us quickly fine-tune pre-trained policy, it has some limitations. For effective online learning, the balance between objectives from model- learning and the learned Q-function is important, but this balance cannot be predicted in advance and requires some trial- and-error with real robots. Although the reward for socialness given only for intimate distance violations, human-in-the- looo online learning with human evaluations can lead to better behavior. And more diverse multiple public datasets on offline training is practically required for robustness.

---

Acknowledgments
We consulted the Committee for Protection of Human Subjects at our home institute, and it was determined that the study does not meet the definition of research with human subjects as outlined in Federal Regulations at 45 CFR 102. This research was supported by Berkeley Deve at the University of California, Berkeley, and Toyota Motor North America. And. this work was partially supported by ARL DCIST CRA W911NF-17-2-018/ and ARO W911NF-21-1-009, We thank Qiyang Li and Mitsuhiko Nakamoto for advising the mathematical formulation. We would like to express their gratitude to Roxana Infante, Ami Katagiri, Pranav Atreya, Stephan Allenspach, Lydia Ignatova, Toru Lin, Charles Xu, Kosuke Tahara, Katsuhiro Kutsuki and Catherine Glossop for their valuable assistance in evaluating SELFI.
References
[1] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, al. Human-level control through deep rein- forcement learning. nature, 518(7540):59-533,, 2015. 3,
[2] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (if.::v777006447 2017.
[3] P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 150.000///090/1 2015. 1,2
[4] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018. 2,
[5] N. Hirose, D. Shah, A. Sridhar, and S. Levine. Sacson: Scalable autonomous control for social navigation. IEEE Robotics and Automation Letters, 91)):49-56, 2024. doi: 10. 100.1009// 3329626. 1, 2, 3, 5, 6, 8, 14, 15, 19
[6] D. Helbing and P. Molnar. Social force model for pedestrian dynamics. Physical review E, 51 (5):4282, 1995. 3
[7] J. Van Den Berg, S. J. Guy. M. Lin, and D. Manocha. Reciprocal n- n-body collision avoidance. In Robotics Research: The 14th International Symposium ISRR, pages 3-19. Springer, 2011.
1.3
[8] Kober, A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey, The 2013.
International Journal of Robotics Research,
[9] Levine, C. Finn, Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):13341333, 2016.
[10] Levine, Pastor, A. Krizhevsky, Ibarz, and Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research, 37(4-5):421-453 2018.
[11] A. Kumar, Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. Advances in Neural Information Processing Systems, 191117--1191 2020. 1,
3
[12] I. Kostrikov, A. Nair, and S. Levine. Offline reinforeement learning with implicit lelamnnng. arXiv preprint arXi::2110006169 2021. 20211 3

---

[13] K. Stachowicz, D. Shah, A. Bhorkar, I. Kostrikov, and S. Levine. Fastrlap: A system for learning high-speed driving via deep and autonomous practicing. arXiv preprint (rXiv:2304.0983/, 2023. 4, 6, 14
[14] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. In 2019 International Conference on Robotics and Automation (ICRA), pages 0023-6029, IEEE, 2019. 1,3, 15
[15] V. Konda and J. Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999. 1
[16] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for rein- forcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[17] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
[18] S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018. 1,2, 4, 6, 14
[19] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.7
[20] D. Pathak, P. Mahmoudich, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings the IEEE conference on computer vision and pattern recognition workshops, pages 2050-2053, 2018. 2,4 4
[21] N. Hirose, F. Xia, R. Martín-Martin, A. Sadeghian, and S. Savarese. Deep visual mpc-policy learning for navigation. IEEE Robotics and Automation Letters, A(4):3184-311 2019. 3, 4, 14, 18
[22] N. Hirose, S. Taguchi, F. Xia, R. Martín-Martin, K. Tahara, M. Ishigaki, and S. Savarese. Prob- abilistic visual navigation with bidirectional image prediction. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1539-1546. IEEE, 2021. 2
[23] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: arXiv:1912.00 1912.0/62..0 2019. 2, 4
[24] R. Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning explore via self-supervised world models. In International Conference Machine Learning, pages
5593. PMLR, 2020. 2
[25] N. Hirose, D. Shah, A. Sridhar, and S. Levine. Exaug: Robot-conditioned navigation policies via geometric experience augmentation. In 2023 IEFF International Conference Robolics and Automation (ICRA), pages 4077-4084. IEEE, 2023. 2, 4, 5, 18, 19
[26] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017. 2,
4
[27] F. Ebert, C. Finn, A. X. Lee, and S. Levine. Self-supervised visual planning with temporal skip connections. CoRL, 12:16, 2017.
[28] D. Hafner, T. Lillicrap, Fischer, R. Villegas, Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In International conference machine learning, pages 3555-2565. PMLR, 2019. 4
10

---

[29] P. Atreya, H. Karnan, K. S. Sikand, X. Xiao, S. Rabiee, and J. Biswas. High-speed accurate robot control using learned forward kinodynamics and non-linear least squares optimization. In 2022 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11789-11795. IEEE, 2022.
[30] T. M. Moerland, J. Broekens, A. Plaat, C. M. Jonker, et al. Model-based reinforcement learn- ing: A survey. Foundations and Trends@ in Machine Learning, 16(1):1-11, 2023.
[31] A. S. Polydoros and L. Nalpantidis. Survey of model-based reinforcement learning: Applica- tions on robotics. Journal of Intelligent & Robotic Systems, 86-2):153-173, 2017
[32] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv:I903.. 00374, 2019. 2
[33] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE interna- tional conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017. 3
[34] Xia, A. R. Zamir, Z. He, A. Sax, J. Malik, and Savarese. Gibson env: Real-world per- ception for embodied agents. In Proceedings of the IEEE conference computer vision and paltern recognition, pages 9068-9079, 2018. 3
[35] D. Shah, A. Sridhar, N. Dashora, Stachowicz, K. Black, N. Hirose, and S. Levine, Vint: A foundation model for visual navigation. arXiv preprint arXiv:2306. 14846, 2023. 3
[36] A. Padalkar, A. Pooley, A. Jain, A. Bewley, A. Herzog, A. Irpan, A. Khazatsky, A. Rai, A. Singh, A. Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x mod- els. arXiv preprint arXiv:2310.08864 2023.
[37] M. Nakamoto, Y. Zhai, A. Singh, M. S. Mark, Y. Ma, C. Finn, A. Kumar, and S. Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. arXiv preprint arXiv: arXiv:2303.05479, 2023. 3
[38] Fujimoto and S. Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145,, 2021. 3, 14
[39] A. Nagabandi. G. Kahn, R. S. Fearing, and S. Levine. Neural network dynamics for model- based deep reinforcement learning with model-free fine-tuning, In 2018 IEEE international conference on robotics and automation (ICRA), pages 7559-7566. IEEE, 2018.
[40] R. Rafailov, K. B. Hatch, V. Kolev, J. D. Martin, M. Phielipp, and C. Finn. Moto: Offline pre-training to online fine-tuning for model-based robot learning. In Conference on Robot Learning, pages 3654-3671. PMLR, 2023. 3
[41] C. Mavrogiannis, F. Baldini, A. Wang, D. Zhao, P. Trautman, A. Steinfeld, and Oh. Core challenges of social robot navigation: A survey, ACM Transactions Human-Rober Interac- tion, 12(3):1-39) 2023. 3
[42] E. A. Sisbot, F. Marin Marin-Urias, R. Alami, and T. Simeon. human aware mobile robot motion planner. IEEE Transactions on Robotics, 23(5)88748888 8837, 2007.
[43] J. Mumm and B. Mutlu. Human-robot proxemies: physical and psychological distancing in human-robot interaction. In Proceedings of the 6th international conference on Human-robot
interaction, pages 331-338, 2011. 3
[44] C. Mavrogiannis, K. Balasubramanian, S. Poddar, A. Gandra, and S. S. Srinivasa. Winding through: Crowd navigation via topological invariance. IEEE Robotics and Automation Letters, 8(1):121-128, 2022.

---

[45] G. Ferrer, A. Garrell, and A. Sanfeliu. Robot companion: A social-force based approach with human awareness-navigation crowded environments. In 2013 JEEE/RS International Conference on Intelligent Robots and Systems, pages 1688-1694, IEEE, 2013.
[46] D. Mchta, G. Ferrer, and E. Olson. Autonomous navigation in dynamic social environments using multi-policy decision making. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1190-1197. IEEE, 2016. 3
[47] M. Luber, L. Spinello, J. Silva, and K. O. Arras. Socially-aware robot navigation: A learning approach. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 902-907. IEEE, 2012. 3
[48] B. D. Ziebart, N. Ratiff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. K. Dey, and S. Srinivasa. Planning- based prediction for pedestrians. In 2009 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems, pages 3931-3936. IEEE, 2009.
[49] A. Bajesy, S. L. Herbert, D. Fridovich- Keil, J. F. Fisac, S. Deglurkar, A. D. Dragan, and C. J. Tomlin. A scalable framework for real-time multi-robot, multi-human collision avoidance. In 2019 international conference on robotics and automation (ICRA), pages 936-943. IEEE, 2019.
[50] X.-T. Truong and T.-D. Ngo. "to approach humans?": A unified framework for approaching pose prediction and socially aware robot navigation. IEEE Transactions on Cognitive and Developmental Systems, 10(3):557-572, 2017.
[51] V. Narayanan, B. M. Manoghar, P. RV, and Bera. Ewarenet: Emotion-aware pedestrian intent prediction and adaptive spatial profile fusion for social robot navigation. In 2023 IEEE International Conference on Roboties and Automation (ICRA), pages 7569-7575. IEEE, 2023.
[52] C. Rösmann, M. Oeljeklaus, F. Hoffmann, and T. Bertram. Online trajectory prediction and planning for social robot navigation. In 2017 IEEE International Conference on Advanced Intelligent Mechatronics (AIM), pages 1255-1260. IEEE, 2017.
[53] A. Wang, C. Mavrogiannis, and A. Steinfeld. Group-based motion prediction for navigation in crowded environments. In Conference on Robot Learning, pages 871 1812. PMLR, 2022. 3
[54] Z. Xie and P. Dames. Drl-vo: Learning to navigate through crowded dynamic scenes using velocity obstacles. IEEE Transactions on Robotics, 39(4):2000-2719, 2023.
[55] Liu, P. Chang, W. Liang, N. Chakraborty, and K. Driggs-CCampblly Decentralized structural- mn for robot crowd navigation with deep reinforcement learning. In 2021 IEEE international conference on robotics and automation (ICRA), pages 3517-3524. IEEE, 2021.
[56] Y. F. Chen, M. Everett, M. Liu, and J. P. How. Socially aware motion planning with deep reinforcement learning. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1343-1350. IEEE, 2017.
[57] Y. F. Chen, M. Liu, M. Everett, and P. How. Decentralized con-communicating multiagent collision avoidance with deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 285-292. IEEE, 2017.
[58] M. Everett, Y. F. Chen, and J. P. How. Motion planning among dynamic, decision-making agents with deep reinforcement learning. 20/8 JEEF/RSJ International Conference on In- telligent Robots and Systems (IROS), pages 3052-3059, IEEE, 2018
[59] C. Chen, Y. Liu, S. Kreiss, and A. Alahi. Crowd-robot interaction: Crowd-aware robot navi- gation with ittention-based deep reinforcement learning. In 2019 international conference robotics and automation (ICRA), pages 6015-6022. IEEE, 2019.
12

---

[60] Y.JJ Mun, M. Itkina, S. Liu, and K. Trigg-CCampbell. Occlusion- aware crowd navigation using people as sensors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 12031-12037. IEEE, 2023.
[61] N. Savinov, A. Dosovitskiy, and V. Koltun. Semi cami-paraettric topological memory for naviga- tion. In International Conference on Learning Representations, 2018. 3, 14, 18
[62] K. Chen, J. De Vicente, G. Sepulveda, F. Xia, A. Soto, M. Vázquez, and Savarese. A behavioral approach visual navigation with graph localization networks. arXiv preprint arXiv:1903.00445, 2019.
[63] D. Shah and S. Levine. Viking: Vision-based kilometer-scale navigation with geographic hints. Robotics: Science and Systems, 2022. 3
[64] X. Yao, J. Zhang. and J. Oh. Autonomous human-aware navigation in dense crowds. In Pro- ceedings of the IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), 2019. 3
[65] C.-E. Tsai and J. Oh. A generative approach for socially compliant navigation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2160-2166. IEEE, 2020.
3
[66] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 2013. 3
[67] Redmon, S. Divvala, Girshick, and A. Farhadi. You only look once: Unified, real- time object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 779-788, 2016. 6
[68] Object detection by yolov5. https: http://ggtttttt htwp:h/ww.oouuuutrratttts/////ae http:://///g.......a.////aaaaaa/
[69] N. Himse and K. Tahara. Depth360: Self-supervised learning for monocular depth estima- tion using learnable camera distortion model. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 317-324. IEEE, 2022. 6
[70] P. Anderson et al. On evaluation of embodied navigation agents. arXiv preprint
arXiv:1887.006557 2018. 6
[71] A. Francis, C. Pérez-d' Arpino, C. Li, Xia, A. Alahi, R. Alami, A. Bera, A. Biswas, Biswas, R. Chandra, et Principles and guidelines for evaluating social robot navigation algorithms. arXiv preprint arXiv:2306. 16740, 2023. 6
[72] R. Kirby. Social robot navigation. Carnegie Mellon University, 2010. 7
[73] D. Shah, B. Eysenbach, G. Kahn, N. Rhinchart, and Levine. Ving: Learning open-word world navigation with visual goals. In 202/ IEEE International Conference Robotics and Automa- tion (ICRA), pages 13215-13222. IEEE, 2021. 14
[74] T. Niwa, S. Taguchi, and N. Hirose. Spatio-temporal graph localization networks for image- based navigation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Sys- lems (IROS), pages 3279-3286 IEEE, 2022. 14, 18
[75] T. M. Howard and A. Kelly. Optimal rough terrain trajectory generation for wheeled mobile robots. The International Journal of Robotics Research, 26(2):141-166, 2007. 14
[76] T. M. Howard, C. J. Green, A. Kelly, and D. Ferguson. State space sampling of feasible motions for high- perforerfor mobile robot navigation in complex environments. Journal of Field Robotics, 25(6-7):325-34)) 345, 2008. 14
[77] D. Shah, A. Sridhar, A. Bhorkar, N. Hirose, and S. Levine. Gnm: A general navigation model to drive any robot. In 2023 JEEE International Conference Robotics and Automation (ICRA), pages 7226-7233. IEEE, 2023. 18
13

---

Appendix
A
Robotic system
For online learning in the real-world, we build vision-hased navigation system that uses a topologi- cal graph of the environment, where nodes denote visual observations and edges denote connectivity.
Hardware setup: Figure 7[a] shows the overview of our prototype robot. We use an omnidirec- tional camera to observe (4+tt-MM.. MM..0 and 19, This allows us to observe a 360 view for capturing the pedestrians even behind the robot. The robot is equipped with an NVIDIA Jetson Orin AGX onboard computer, which runs inference of trained models at 3 Hz. We use two additional cam- eras to estimate visual odometry and detect long-term localization fiducials, following the setup of Hirose al. [5]. We to detect collisions. In addition to on-robot compute,
IMU to measure bumpiness and uneven terrain and a bumper sensor
workstation for fast, online training. The workstation is equipped with an Intel i9 CPU, 96GB RAM, and an NVIDIA RTX 3090ti GPU.
Figure 7[b] shows small obstacles, which we place for online learing(left) as well as our evalua- tion(right). In our evaluation, we place the obstacles, which is not seen online training.
Navigation system: Similar to [21, 61, 73], we construct our vision-based navigation system using a topological memory. We update the goal image 1 based on localization in the topological map to navigate towards distant goal position. Before deployment, we collect the map by human teleoperation and record the subgoal images and the corresponding global goal poses at 0.5 as { , >= along the robot's trajectories. Here, L indicate the number of the nodes in the topological map. During inference, number &e the current node by i arg min, l! pll and feed 1 the goal image 19, We estimate with incorporating visual odometry and AR markers as shown in the appedix.
estimate the global robot pose p and decide the closest node
Front dide
Coneicaee and saree
Back side
treining
For Evaluetion
[n] Dverview of The prototvon robot
[o] Overviow of small obstac
Figure 7: Overview of the prototype robot(left) [74] and small obstacles(right).
B
Evaluation setup
We evaluate following basclines in addition our proposed method for comparative evaluation.
Sampling-based motion planning: This baseline generates fifteen motion primitives [75, 76] at every time step and selects the best one considering goal reaching, static and dynamic obstacles such as the pedestrians. To control the robot, we give the velocity commands corresponding to the selected motion primitive. The details are shown in the Appendix G.
TD3+BC TD3 [18, 38]: This baseline uses TD3+BC [38), an offline RL method, to train the encoder, actor, and critic offline. During online training, we fine-tune the pre-trained actor and critic with TD3 [18] while freezing the encoder.
FastRLAP [13]: FastRLAP employs the pre-trained encoder from offline RL and trains the critic and the actor from scratch online while freezing the encoder. Different from the original FastRLAP, use TD3+BC [38] and TD3 offline and online learning algorithms, respectively.
14

---

position
Trajectory
Oojects, which
B
ovioon
APS.
5, In
[e] Environment (42.9
[b] Environment (21.10m)
[c] Environment (62.461m)
Figure 8: Three environments on online training and evaluation. conduct online training three different challenging environments, [a] the open space facing restrooms, elevator hall and café space, [b] the entrance hall with many pedestrians, and, [c] the office area with narrow corridors. [a] and [c] have many glass walls, which difficult for collision avoidance and cause lighting condition changes.
Residual RL [14]: In residual RL, the policy is given by the sum of base policy and learned policy, T ahaaee (s) + (s), where h is the pre-trained control policy and R is the ac- tor trained with online RL (TD3). We evaluate two choices for hasse (1) the pre-trained control policy maximizing only Σ prose to simply move towards the goal position, and (2) the pre- trained SACSoN policy, maximizing the total objective Jsason We label the latter as Residual RL Residual RL provides an alternative way to combine prior policies with online model-free RL, and therefore represents natural prior method for comparing with SELFI.
SACSoN with fine-tuning [5]: This baseline trains the entire control policy by maximizing the SACSoN objective Jessonn on the HuRoN dataset, and then fine-tunes the actor online by maximizing Jaason again. The online objective does not use the additional (non-differen diinffffeeretiable) online reward terms r.
All bearning-based baselines use the same network structure, except that single-step methods (all except Ours and SACSoN with fine-tuning) predict only single action rather than a sequence T. Unless specified, all RL-based methods use the same reward.
We conduct our experiments in three challenging environments in Fig. 8, which are in different regions of the same building. Environment is entrance and café area, which naturally has lot of pedestrians. The environment's 's lighting conditions and furniture placement change significantly over time. This environment also contains glass walls and chairs with thin legs, which can be difficult to detect as obstacles. Environment is the entranceway to an office building, also with significant pedestrian traffic. Environment working spaces. Pedestrians are less frequent in this environment, but the corridors are narrow and require avoiding difficult static obstacles such as glass walls, and present challenge in avoiding pedestrians in confined spaces.
a loop through several hallways, desk areas, and
In these three environments, we design the looped trajectories, which the last node is same pose as the initial node, as shown by red lines in Fig. 8, and feed the first goal image when arriving at the last node to continuously train the control policy online. We conduct online learning while rotating these loops and stop training when the robot run 15 laps or two hours. During online training, we randomly place small objects shown in Fig. 7[b](eett) to learn the collision avoidance behavior for the small objects. Since Environment I includes many challenges for our task, evaluate all methods in this environment and conduct the overall evaluations with the selected baselines in three environments.
C Evaluation with human participants
For Q1, we conduct two types of experiments, 1) comparative evaluations with relevant baselines and 2) human subject experiments to evaluate how well our methods behave. For 1), conduct the experiments in the "organized" environments for reproducible comparisons. For 2), we conduct
15

---

our experiments in the "unorganized" environments to obtain various evaluations of each human subject's senses. We explain the details of the experimental setups each experiment.
C.1 Reproducible comparisons
difficult to conduct extensive and reproducible comparisons in highly populated natural envi- ronments because pedestrians are walking and out of the scene random intervals. Therefore, to conduct more controlled and reproducible comparison with each of the baselines, we focus on the "organized" environments.
Before the experiments, we instruct the human participants to follow pre-defined trajectories (as repeatably as possible) to have consistent testing conditions. However, if the robot interferes with a person's 's path, the pedestrians were asked slow down or stop as needed. If the robot can not give way to the pedestrians, the pedestrians change their path as they deem fit, interact with the robot and then come back the original trajectories. All subjects are asked to follow these instructions across the experiments. We conduct the experiments across different days, as well as times of day, mimicking the range of lighting conditions and changes in environment layout that a robot would experience over several days. In each experiment, we randomly place the small unseen objects shown Appendix A into the scene to increase the clutter the scene and evaluate the collision avoidance performance. However, we conduct the experiments under approximately equivalent conditions for each method.
C.2 Human subject experiments
To incorporate various human subjects into evaluation, ask the human subjects to interact with the robot and evaluate its behaviors without specifying specific ways for the subjects to interact with the robot. This prevents biased behavior and insights from the human subjects. Additionally, we ask the human subjects to have similar interactions with the robot across all methods to maintain fairness the evaluation. To assure the behaviors the pedestrians were as natural as possible, we did not always observe and follow the robot during online training. Subjects were told which method being used for each trial. Before the evaluation, we explain the rough robot route from the start to the goal position.
D Weighting objectives on online training
Similar to the other learning algorithms, approach needs find the best balance of each objective on online training. In our case, we first perform online learning with small weights for the learned objective Q(s, a) and gradually increase the weights. The small weighting for the learned objective makes online learning more stable and facilitates the analysis of learning results. Once we find good weight in one environment, we use the same value the different environments.
E Pre-training with data from the target environment
When robots operate continuously in the same environment, they can directly collect and utilize a large amount of in-domain data. In the supplemental evaluation, we show that a large offline dataset of interactions from the target environment can boost the performance of our method during online offline RI. train actor and critic on the collected dataset from the target environment top of SACSoN. Then, we fine-tune both networks online by SELFI.
training. Specifically,
During offline RL training, we leverage the collected dataset from our study. While finding good hyperparameter for online learning, we collected a dataset of 50 hours of experience in Environment Figure shows the number of intervention for each loop in Environment Even at beginning, our method leverages large dataset, Ours + prior data only needs one intervention and can navigate the robot without any interventions after 5 laps, because of faster training with the pre-trained actor and critic the large dataset. Similar to Table 1, Table 2 shows the mean value of the selected
16

---

metrics on 5 laps in the Environment 1. Ours prior data shows remarkable gap against Ours in every metric except SPI.. Since Ours + prior data takes a larger deviation from the original path to avoid violating the intimate distance of close pedestrians (to decrease IDV), SPL is slightly worsen.
In addition Ours prior data, show the results of the all other baselines including FastRLAP, TD3+BC-IDD TD3 and Residual RL in Fig. 9 and Table 2. FastRLAP and TD3+BC-TTD3 can also improve the performance of the control policy during online training. However, online training for two hours or 15 laps is not sufficient, and these methods require many interventions. Surprisingly, applying Residual RL with the pre-trained SACSoN policy as π. basc actually decreases in perfor- mance during online training, giving worse performance than Residual RL. We hypothesize that in this case Rl. must learn a copy of the SACSoN policy to predict the result of particular action, sat- urating the capacity of the network due to the base policy's high complexity. We find that Residual RL and SACSoN with fine-tuning are the strongest baselines. And think that Sampling-based motion planning can be the proper baseline from non learning-based approach. Hence we prioritize these three baselines in our further evaluations in Sec. 5.
40
number interenniin
35
30
fine-tuninng
25
TD3-BC -DD
Residual
20
Residual RL FasIRLAP
15
10
laps
Figure 9: The number of intervention on online learning in Environment 1. Ours + prior data indicates our method leveraging large dataset same environment. Residual RI, uses the SACSoN policy as Thwe. The horizontal axis indicates the numher of times the rohot laps the loop.
Table 2: Closed-loop Evaluation of trained control policies at Environment 1. Ours prior data indicates our method leveraging large dataset same environment. Residual RI. uses the SACSoN policy as Masse indicates the use of the ground truth goal pose to generate velocity commands.
Method
IDV [s] 4
CP J. |#]
Int 4 [#]
SPL f
STL
Sampling- motion planning TD3 + BC-TD3
23.909
1.800
0.400
0.839
0.678
34.632
0.800
6.600
0.773
0.383
FastRLAP
43.623
2.000
10.000
0.702
0.306
Residual RL.
21.754
4.200
3.000
0.780
0.568
Residual RL
39.627
2,800
9,400
0.598
0.466
SACSoN with fine-tuning
19.534
0.200
1.000
0.788
0.549
Ours
10.712
0
0.600
0.932
0.699
Ours prior data
6.610
0
0.200
0.900
0.713
F
Robot pose estimation with AR markers
For reward calculation, first estimate the robot's global position p. Using p. get an estimate of g. the goal position in the robot's local frame, which we can directly use for reward calculation. Additionally, is useful to localizing the robot's 's position in a topological map, which provides our policy the current subgoal image éc
For localization, we mount a tracking camera, Intel T265 on our robot that measures visual odome- try. However, the Intel T265 can maintain the same globalframe before and after rebooting due to battery replacement. In addition. the shaky robot motion and insufficient visual features deterio- rate the accuracy of visual odometry. To have same global frame and to get more accurate estimate
17

---

of the robot's global position, we place one AR marker every 15 [m] along the robot's trajectory and suppress the localization error for stable online learning.
Figure 10 shows an overview of our localization system using AR markers. Before starting online learning, we collect {T.² TTA.R. )i=1...o in conjunction with a topological map. T. is the robot position matrix computed from visual odometry and T, is the position matrix of i-th AR marker in the robot's local frame when detecting i-th AR marker. Here, assume the robot's s motion during teleoperation is smooth enough for us to have an accurate estimate of T from visual odometry. O is the number of AR markers on the robot's trajectory. When detecting the i-th AR marker during online learning, we calculate the correlation matrix T. to estimate T from the visual odometry T,":
T= T. T
(4)
When detecting the i-th AR marker during online learning, T can be defined as follows,
T T. TAAoo (T.AR.)-1.
(5)
Here I' is on the same global coordinate as TV. Hence, we can obtain T by calculat- ing T T.A.R (Tr T: TTAR.--1. During online learning, we update T every time we detect an AR marker, and we use Tţ to calculate T until the robot detects the next AR marker. The robot pose is uniquely calculated from the robot pose matrix T. And, the goal pose 9 on the robot local co- ordinate be uniquely calculated from T-TTy. where Tg is the global goal pose matrix for the goal image T9.
1t AR marker
TRA
T T.ARg
True current rooot position
noce topo, map
T
T
T
Tuo
Est. robot pas tion viaual adam.
Global coard.
While we use AR markers due to our de- sign decision using only an RGB camera and limitations of the Intel T265, is im- portant note that the usage of AR mark- ers is not mandatory. Our system setup serves as just one example. If potential users relax the restriction of using only the camera, they can leverage other sensors such as GPS, LiDAR, and/or depth cameras to estimate the robot's pose without requiring AR markers. It's worth mentioning that we can run our system without AR markers during inference using other vision- based localization techniques [21, 25, 61, 74, 77], but we use AR markers to identify the corre- sponding goal image ID) focusing evaluating the trained control policy.
Figure 10: Overview of robot's position estimation with AR markers. We update the correlation matrix 77' when the robot detects an AR marker, and we apply it to suppress the error from the noisy visual odometry T." until the robot detects the AR marker.
G Details of sampling-based motion planning
We implemented sampling-based motion planning as the baseline to bridge the learning-based ap- proach with broader robotic motion planning. We generated 15 motion primitives assuming steady linear and angular velocity commands for 8 steps (2.664 s), which is the same horizon method and the strongest baseline, SACSON. The pairs of linear and angular velocity commands are (Va, w.) = (0.0, 0.0), (0.2, 0.0), (0.2, 0.3), (0.2, 0.6), (0.2, 0.9), (0.2, -0.3), (0.2, -0.6), (0.2, -0.9), (0.5, 0.0), (0.5, 0.3), (0.5, 0.6), (0.5, 0.9), (0.5, -0.3), (0.5, -0.6), (0.5, -0.9). We selected these motion primitives by balaneing computational load and navigation performance.
By integrating these velocity commands for steps, obtained 15 trajectories such as (((r]) ) )= where "pl is the i-the virtual robot pose the j-th motion primitive. To select the best motion primitive, we calculated the following cost value for each primitive.
J = min,,p?
"p?)² + Cob + Cped
(6)
18

---

Here, Pi+ indicates the next subgoal pose. The first term on the right-hand hand side calculates the squared errors between all 8 poses in the j-th motion primitive and the goal pose and selects the minimum one to evaluate the goal-reaching performance. Cob is a constant value used to filter out trajectories that collide with static obstacles. We calculate Cob as follows:
Cob { 1000.0
(if d, <r,)
0.0
(otherwise)
where de is the minimum distance between all 8 poses the j-th motion primitive and the esti- mated point clouds corresponding to the static obstacles. Similar to SACSoN [5] and ExAug [25], collision determined when the distance between the static obstacle and the robot less than the robot's radius r,. To ensure fair evaluation with other methods in rision-based navigation that only uses an RGB camera, we utilize estimated point clouds from the current observation of the RGB camera. Additionally, Cped is a constant value used to filter out trajectories that violate the intimate distance with pedestrians. Following SACSoN [5], we predict the future trajectory of pedestrians as pedd {pred)=1... H==...8 and assess whether each motion primitive violates the intimate distance or not.
10000.0 0.0
(if dped +,5++r)) (otherwise)
Cped
Here, dned min; dist(pp"d, =xp)) and dist() is the function used to calculate the distance between two poses. The values of Cob and Cped are set to 1000.0, which is much larger than the first term on the right-hand side in Eqn to filter out inappropriate motion primitives during the selection pro- cess. We choose the motion primitive with the minimum J and assign the corresponding velocity commands v, and wis to control the robot during navigation. Note that we utilize the same point cloud estimator well the pedestrians' trajectory predictor as method and SACSoN to have fair comparison.
Figure 11 shows the implemented sampling-basee motion planning. Since pedestrians are not present in [b], this baseline selects the motion primitive that moves toward the goal position. How- ever, in [a], where the pedestrian's predicted trajectory crosses between the robot's current position the goal position, the baseline selects motion primitive that maintains sufficient distance from the pedestrian's 's predicted trajectory. After the pedestrian passes through the area, the baseline will select trajectory towards the goal. In Fig. 11, the blue lines with the "x" markers are the motion primitives and the black dots are the estimated point clouds.
selected traj robot past traj
selected traj robot past troj goal position
ped. past traj
estimated ped. traj
goal position
pedestrian
2
1
-1.5
-1.0
-0.5
0.0
0.5
1.0
-1.5
-0.5
0.0
1.0
[a] With pedestrian
[b] Without pedestrian
Figure 11: Overview of sampling-basel motion planning.
H Details of network structure
Each raw image from the robot's omnidirectional camera is a front- and back-side fisheye image stiched side. We resize the stiched images and concatenate them in the channel direction
19

---

such that the resulting image data 6 x 128 x 128. To extract temporally consistent features in the observation and goal image, we channel-wise concatenate the goal image and history of observa- tion images as 48 x 128 128 input and feed into the encoder, Ig. Note that 6 channels are for current observation, 6x channels are for the history past observations, and last 6 channels are for the goal image. The encoder has eight convolutional layers with batch normalization and ReLU activation function in each layer extract 12-dimensional feature vector. The feature vector is then fed into the actor ne. full-connected MLP.
The extracted features for the actor are concatenated with the previous action commands á to handle the deadzone, which a result system delay and backlash in the robot hardware. The actor my has three fully connected layers with batch normalization and ReLUs generate т. The last layer has hyperbolic tangent function to limit the velocity commands within upper and lower boundaries.
We concatenate r and the extracted features from the image encoder and feed it into the critic to estimate Q. The critic is designed with five fully connected layers, batch normalization, and ReLUs. The last layer has linear function instead of ReLUs to allow us to estimate negative values.
loyer
Current image Io
back
& letten
ooncat
Jue lppers
Critic
History
{
bock
back
front
front
36x 124122
Encoder
Soma Coninng
Goal
lg
Actor
T
back
Sr128c128
corn, Noyurs
1
with BN and Rooce
Figure 12: Overview of network structure in our implementation.
20


Reviews:
[{"review_id": "UnymR3FT3N", "reviewer": "Reviewer_XMR4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a model-free online fine-tuning method on a model-based offline trained model to increase the robot’s capability in handling small obstacles, bumpy surfaces, and pedestrians.", "summary_of_recommendation": "This paper's quality is high, however it is confusing how humans are modeled in the “organized” environments setting, which raises questions on the quantitative analysis of the social navigation aspect. The sampling-based baseline is also missing in the qualitative analysis.", "technical_quality": 3, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "What does the pedestrian setup look like in the “organized” environments?\nWhy is the sampling-based method not included in the natural evaluation setting with the 12 participants?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZxfGNrP5rS", "reviewer": "Reviewer_agf9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors present a technique for vision-based navigation in mobile robot systems. They achieve this behavior by combining two types of RL based techniques. Firtly, the apply a model-based learning offline and then in a 2nd step additional RL based Online learning. This combination gives the authors to chance to apply it to a small-scale mobile robot and evaluate a navigation task around people. Further, the authors not only evaluate the quality of the navigation task but also evaluate the socialness by humans in order if they have the feeling that the robot is moving socially.", "summary_of_recommendation": "The paper is showing a nice method for visual based navigation especially in social situations. The paper is well written, adresses the methodology in a good way and shows real-world experiments - both qualitative and quantiative.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 1, "questions_for_rebuttal": "Feedback For Improvement Or Clarification: \n- I think that it would be helpful to provide more guidelines or automated methods for balancing the model-based and model-free objectives during online learning to reduce trial-and-error. Maybe it is possible to enter some more information for the authors on how to do that or provide even pre-trained models?\n- Currently, I am missing  a clarification on the reward design process and how it adapts to different tasks and environments\n- Furthermore, can you specify if your system can be adapted to different environments? Since you heavily rely on the AR markers, it is difficult to see how we move the system to a different environment\n- The limitation section is currently missing. I would be good to have your own view on where the limitations of your paper are.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KY2VE1V3rM", "reviewer": "Reviewer_5y57", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces an online learning method SELFI that leverages online experiences to rapidly fine-tune pre-trained control policies. While closely related to residual RL, SELFI considers the linear combination of the offline and online policies through the objectives (i.e., Q-values) rather than their direct actions. The incorporation of the offline model-based learning objective into the online learning process stabilizes the training process and allows quick adaptation to the operational environment.", "summary_of_recommendation": "The paper is well-written and the approach, though simple, is well-supported theoretically and empirically.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. I would prefer if the authors could improve their discussion and evaluation of the results. Currently, much of the paragraph in the evaluation section is a detailed description of the evaluation by human rating. If paper length is a consideration, these details could be relegated to the appendix since much of it is similar to existing works. A more detailed analysis of the results and their interpretation would significantly strengthen this work. (e.g., Why CP, CO, SPL, STL are significantly better than the other baselines?)\n\n2. The implementation of SELFI uses a frozen encoder during the online phase, which implies that the online inputs need to be relatively similar to those seen in the offline dataset. Otherwise, the extracted features may not be accurate. Are there any precautions taken to address this? If the deployed environment differs from the training environment, or if there are new elements (e.g., new types of obstacles) that were not encountered during training, does the approach still perform well?\n\n3. SELFI uses a model-based approach to compute the SACSoN objective even during the online phase. While this objective is beneficial for the initial training phases, is it necessary to maintain it throughout the entire online deployment? As with all model-based approaches, the performance of the control policy depends on the accuracy of the model, and it might be beneficial to reduce such dependencies over time. Have the authors considered the use of, say, a decay on the offline objective to gradually phase out the dependency on the model and transition to a completely model-free approach? Is this approach feasible? Or rather, is there a reason that the offline objective needs to be maintained throughout?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIEplxM5s7", "reviewer": "Reviewer_1Aui", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a model-based + online RL pipeline for vision-based social navigation. The proposed method overcomes the inaccurate models of model-based method and data inefficiency of end-to-end RL. The experiments evaluated both navigation metrics and subjective feedback from a user study. The results show that the proposed method outperforms baselines in terms of navigation metrics, user feedback, and number of human interventions during online learning.", "summary_of_recommendation": "This paper investigates an interesting topic with novel methodology and solid experiments. It would be good if the authors could improve the clarity of presentation.", "technical_quality": 4, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- For readers who do not know SACSoN, the introduction is hard to digest. Please consider filling in the relevant background so that this paper can better stand on its own. In addition, some other parts are unclear:\n  - Citation is needed in line 22 after “Model-based control… in the scene.”\n  - In line 42-43, I don’t understand why RL can work as expected “as long as…small values.”\n  - In line 68, please point out what type of model-based learning is used. “Model-based learning” is a very general phrase.\n- The authors defined the reward $r$ in line 214. But $\\hat{r}$ is mentioned but not clearly defined in line 194. Also, what is $\\overline{r}$ used in the experiments?\n- In online learning stage, are pedestrians real or arranged? If real, how to deal with safety issues when the robot fails?\n- In experiments, what is the range of robot’s travel distance, the range of the number of pedestrians, and number of environments/scenarios for training and testing?\n- In line 279, why “our method performs better overall”?\n- Are human intervention signals used for training? If not, how could this be done in future work?\n\nMinor problems:\n- In Sec. 2, \n  - the authors may consider citing the following papers as examples of model-based methods:\n    - Van Den Berg et al, \"Reciprocal n-body collision avoidance,\" Robotics Research, 2011.\n    - Mavrogiannis et al. \"Winding through: Crowd navigation via topological invariance,\" RA-L 2022.\n  - And consider citing the following papers as RL methods:\n    - Liu et al, \"Decentralized Structural-RNN for Robot Crowd Navigation with Deep Reinforcement Learning,\" ICRA 2021.\n    - Xie et al, \"DRL-VO: Learning to Navigate Through Crowded Dynamic Scenes Using Velocity Obstacles\", T-RO 2023.\n- Table 1 should be moved to page 7.\n- In Fig. 4, the yellow robot is hard to see in the left 2 subfigures, and the green human is missing in the right two subfigures.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Strengths:
- A critical and timely challenge is addressed. 
- The proposed approach is technically sound and well-validated.  
- The presentation is clear and the paper is well-written. 

Weaknesses:
- There is some unclear parts in the evaluation setting. 
- Dependency on a specific marker could be a limitation for exentions.
Decision: Accept