Multi-Strategy Deployment-Time Learning and Adaptation for Navigation under Uncertainty
Authors: Abhishek Paudel
Xuesu Xiao
Gregory J. Stein
Venue: CoRL 2024 (2024)
Keywords: policy selection
domain adaptation
navigation under uncertainty
TL;DR: We present an approach that enables a robot to deploy multiple learning/adaptation strategies during deployment and pick the best one.

Abstract: We present an approach for performant point-goal navigation in unfamiliar partially-mapped environments. When deployed, our robot runs multiple strategies for deployment-time learning and visual domain adaptation in parallel and quickly selects the best-performing among them. Choosing between policies as they are learned or adapted between navigation trials requires continually updating estimates of their performance as they evolve. Leveraging recent work in model-based learning-informed planning under uncertainty, we determine lower bounds on the would-be performance of newly-updated policies on old trials without needing to re-deploy them. This information constrains and accelerates bandit-like policy selection, affording quick selection of the best-performing strategy shortly after it would start to yield good performance. We validate the effectiveness of our approach in simulated maze-like environments, showing improved navigation cost and cumulative regret versus existing baselines.

Multi-Strategy Deployment-Time Learning and Adaptation for Navigation under Uncertainty
Abhishek Paudel, Xuesu Xiao, and Gregory J. Stein
Department of Computer Science
George Mason University (apaude14, xiao, gjstein//gmu.edd edu
Abstract: We present an approach for performant point-goal navigation in unfa- miliar partially-mappe environments. When deployed, our robot runs multiple strategies for deployment-time learning and visual domain adaptation in parallel and quickly selects the best-performing among them. Choosing between poli- cies they are learned or adapted between navigation trials requires continu- ally updating estimates of their performance as they evolve. Leveraging recent work in model-based learning-infonen lower bounds the would-be performance of newly-updated policies old trials without needing to re-deploy them. This information constrains and accelerates bandit-like policy selection, affording quick selection of the best-performing strat- egy shortly after it would start to yield good performance. We validate the effee- tiveness our approach in simulated maze-like environments, showing improved navigation cost and cumulative regret versus existing baselines.
planning under uncertainty, determine
Keywords: policy selection, domain adaptation, navigation under uncertainty
Introduction
Consider robot deployed in an unknown environment and tasked to reach an unseen point goal. The robot leverages learning to make decisions about where to go most quickly find the goal. How- ever, depending upon how different the training and deployment environments are, learning may always inform good behavior and the robot may demonstrate poor performance during deployment. Thus, to perform well in a wide variety of environments, the robot must improve its behavior during deployment, via strategies such as visual domain adaptation or online policy learning. Problemat- ically, there is no one-size-fits-al approach to deployment-te time learning or adaptation: each such strategy typically only suitable for addressing a subset of the types of changes a robot may en- counter during deployment or are slow to converge, risking poor performance during a potentially lengthy or error-prone learning phase and limiting most such approaches in general.
If robots in this goal-directed navigation scenario are to perform well when deployed in arbitrary unfamiliar environments, they could ideally run many strategies for adaptation and online learning in parallel and rely on run-time monitoring to select the best-performing policy or strategy. Bandit algorithms [1, 2, 3) are potentially useful, yet are prohibitively slow to converge this setting. Other existing work in the space of run-time monitoring [4. 5, 6. 7, 8] or policy selection for reinforcement learning [9, 10, 11, 12] similarly limited to short time horizons or fully-known environments and are not straightforwardly applicable. Moreover, such strategies typically presume that the policies are static, posing risk when the policies improve during deployment: deciding to not select a policy that was deemed to perform poorly early on-yet has improved considerably since then-results in poor overall performance. Instead, we require an approach that can reliably select between family of continually changing (non-stationary) policies being learned or adapted during deployment.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Goal: Select the best of an ensemble of learning-iffrrmed visual navigation policies, even as many of those evolve during deployment via online learning or visual domain adaptation.
Performance of uwnr greatly improves after round of visual domain adaptation at Trial 10:
We use deployment-time timmenttttmmeeeeee data to "replay" old trials, updating otherwise-outdated performance estimates:
Adaptation
E Coet (oracle)
Round (7100)
Our uppraach understandss
R Rears has imprevved
Rolling Averige
Trials
Deploymer Selected Polley
duta illutrative parpones
Partial map i mmngees
Statonary Rece
Deploy Ranme
Deploy
Even if provided privileged information about
Our approach affords deploying multiple policies or stralegies for online learning or domain adaptation in parallel and quickly choosing the best performing among them during deployment.
policy performance over time, selection via a rolling average fails quickly switch Raor
Figure 1: Overview of our approach for fast selection between non-stationary policies.
In this work, we present an approach for data-efficient and reliable policy selection capable of choos- ing the best-performing policies from an ensemble of policies, even when many of these policies are being learned or adapted during deployment (Fig. 1). We leverage insights from the recent work by Paudel and Stein [13] whose offline all-policy replay uses data collected by the robot from navigation trial to simulate how another policy would have performed, yielding lower bounds on its performance that constrain and improve policy selection. As this approach does not consider that policies may change during deployment, the rolling estimates of the performance of each policy quickly diverge from their true expected performance as old performance estimates become stale. Instead, we leverage offline alt-policy replay to refresh old data and so compute up-to-date bounds on the would-be performance of each policy, allowing us quickly select the best performing policy even the robot's policies improve during deployment.
We deploy our approach in simulated maze-like environments where visual cues indicate promis- ing routes to the unseen goal. Via our approach, our robot is deployed with six policies-a static pre-trained policy, a non-learned baseline policy, two policies being continually adapted via visual domain adaptation, and two policies being trained from scratch using deployment-time data and quickly chooses the best performing ones among these when deployed. Notably, our approach cor- rectly avoids the non-stationary policies early in deployment before they learn to understand the environment yet quickly switches to them once they improve, capability that allows approach to outperform all single-policy strategies and existing policy selection baselines.
2 Related Work
Domain Adaptation Advances in visual domain adaptation [14, 15, 16, 17, 18, 19] have made it possible to adapt systems trained in one domain to a different domain during deployment. Visual domain adaptation approaches have been used compensate for differences in the robot's visual observations between the training and deployment environments [20, 21, 22, 23, 24]. However, visual domain adaptation approaches, such as those that leverage CycleGAN [19], suitable for improving performance, especially when domains are drastically dissimilar, requiring careful consideration for their uses on robotic systems where reliability is of critical importance.
always
Deployment-Tinne Adaptation The idea of adapting robots during deployment has been explored widely in the robotics literature. Many approaches focus on short-horizon robot behavior such as adapting lowelevel motor controls or locomotion in diverse terrains [25, 26, 27, 28, 29, 30]. As such, they are not concerned with long-horiae behavior where the robot should consider the impacts of its immediate actions far into the future, the focus of this work.

---

Policy Selection can be thought of as model selection applied to choosing between robot behav- iors [13]. In reinforcement learning, model selection [9, 10, 11, 12] is often treated as a generalized form of multi-armed bandit problem [31]. Using bandit algorithms [1, 2, 3]-which are often black- box-for policy selection requires trade off between exploitation and exploration and hence the robot has potentially go through multiple trials with poor behavior before a better policy can be identified. White-box policy selection [13] has shown recent promise in data efficiency, yet far only consider stationary policy selection, ill-suited for policies that evolve during deployment.
3 Preliminaries
For robots to perform well across a variety of unfamiliar environments, they must have the abil- ity to simultaneously apply multiple deployment-time learning and domain adaptation techniques in parallel, so that they may choose the best-performing of them when deployed. We model this scenario as an instance of policy selection. While the policies themselves are cin-stationary-thee change over time as the robot is deployed and collects data-we must first discuss the fundamentals of goal-directed navigation in partially-mapper environments and strategies for stationary (static) policy selection in this domain, needed to understand our approach.
Goal-Directed Navigation in Partially-Mapped Environments For each trial, our robot is placed in a partially-maped environment and tasked to reach an unseen point-goal in minimum expected cost, measured units of distance. Planning under uncertainty is often formulated as a partially observable Markov decision process (POMDP) [32]. As planning via the POMDP model directly is computationally intractable, many planning strategies in this domain rely on learning to anticipate what may lie in unseen space and thus inform good behavior. The robot's policy sumes belief state bi-including the robot pose qt, the partial map m,, and visual observations collected onboard the robot 0, andand returns a primitive action specifying the robot's behavior. For all policies in this work, images are used to inform learning, which makes predictions about unseen space to guide the robot more quickly to the unseen goal.
Policy Selection over Multi-Tria Deployments A single deployment consists of sequence of trials, each single traversal from start to goal in previously-unseen map. We consider the scenario in which the robot is deployed with an ensemble of policies II = (2, , }. each defining different behavior. The objective of policy selection is to determine, during a deployment, which of those policies performs best the deployment-timg time environments:
π = argmin B[C(T))
(1)
#Ell
where [[]( is the expected cost of policy т the deployment environment distribution. In practice, the expected cost is approximated by deploying the policy multiple times and averaging the performance: [B[[(T)] C(T)), the average cost of the trials in which policy x was deployed.
3.1 Black-box Policy Selection via Multi-Armed Bandits
Selecting the policy that minimizes ((+)) is unwise in general, since the randomness inherent in the deployments means that the best-performing policy may be incorrectly ruled out early on and never selected again to improve the estimate of B[C(T)) with no recourse to recover. Instead, the upper confidence bound (UCB) aumti- bandit [1] balances exploitation with exploration, incentiviz- ing selection of policies that have been chosen less often to improve the estimate of [E[C(=)). For trial l + UCB bandit selection specifies one choose policy ((k=)) according to
In k
(41) argmin *Ell
Cx((T) - n& (T)
where (T) is the average cost over trials 1-through-k in which policy was selected, ny (T) is the number of times policy selected through trial k and 0 is parameter controlling the rate of exploration. In practice, each trial of goal-directed navigation is expensive and the resulting cost
3

---

samples have high variance. Thus, black-box policy selection approaches such as this are problem- atically slow to converge, making them impractical for policy selection in this domain despite their desirable guarantees on asymptotic sub-linear regret.
3.2 Data-Efficient Policy Selection via Offline Alt-Policy Replay
central challenge of black-box model selection approaches their data inefficiency, as each deployment yields only a sin- gle data point for single policy. If it were instead possible to use the images and partial map collected during trial to determine how well other policies could have performed bound on performance-woe we could use it to inform policy se- lection and so accelerate convergence to the hest-performing policy, Paudel and Stein [13] present an approach they term offline alt- alt-policy replay in which data collected during deploy- ment of policy ı is used to simulate the would-be behavior of another policy n' placed in the same environment.
Though we deploy policy we we might want to know how well unother policy x,, could have done:
Deployment of
Offline Replay [13] of policy
policy
Guall
Goal
T Start
Would have
here.
Start
Using data from deployment of Femoo offline alt-policy replay [13] computes lower bound ("a the would-be cost of deploying another policy n,..
Under this approach, deployment of policy т during trial yields a record 2g of all its poses and observations of the environment. This partial snapshot of the environment is used to simulate how another policy n' would have acted, letting it navigate and reveal the environment and receive the image ob- servations it needs make predictions about unseen space that inform its behavior. Offline play produces lower bound on the performance of x' had it instead been deployed: (T') OFFLINEREPLAY(T 2k) S (TT).. Thus, each trial yields information about the performance of all policies π E II. The average lower bound after & trials, (xr'', constrains bandit-like policy selection:
Figure 2: Offline Replay Overview
max (CE(T).
In d: (1)((T)
x 004) argmin sell
Offline alt-policy replay results in dramatically faster convergence to the best-performing policy while still preserving guarantees on asymptotic sub-linear regret. However, like the bandit algorithm before it, this approach presumes that the policies themselves are stationary, and so is not well-suited to select between policies being learned or adapted online, the focus of this work.
3.3 Learning over Subgoals Planning: Cearning-Augmentte Model-Based Planning
A key enabler of offline alt-policy reply is the require- ment that planning be done via a policy amenable to coun- terfactual reasoning-i.e. What would policy n' have done if it had instead been placed in this situation? with which the behavior of policy x' can be simulated. As such, our policies are based on the learning-ifoormed model-based learning over subgoals planning (LSP) by Stein et al. [33], which satisfies this requirement.
In lcarning over subgoals planning (LSP), actions correspond navigating to frontiers:
Gool
Action
Action leads to goal:
Action
Ps= R5=40 40.
Start
Action leads to dead end: Ps=0, R,=30. 30.
Goal
Learning estimates the likelihood (Pt) that the fronfier-action leads the goal associated costs (Rş and R2). Using these estimates, planning done Eq. (4).
Learning over subgoals planning (LSP) a high-level planning framework for bearning-informe' model-based navigation in partially-mapped environments. Under this abstraction, subgoals correspond to frontiers, each a boundary between free and unknown space; high-level actions a correspond to navigation to subgoal and then exploration beyond to attempt to reach the unseen goal. The robot uses a learned model to make predictions about unseen space beyond each subgoal, information it uses to decide where to reveal next. The learned subgoal property estimator, neural network No parameterized by 0 and trained via supervised learning, consumes panoramic images in the vicinity of each subgoal to estimate the likelihood that each subgoal-action will suc-
Figure Overview of learning over subgoals planning (LSP).

---

cessfully reach the goal Ps.o and the expected costs associated with success Rs.0 and failure R to reach the goal in unseen space. Upon failing to reach the goal via a high-level action or, the robot must select another action a E AA \ ae. Planning is model-based, with the expected cost of a high-level action defined by a Bellman Equation:
min Q0(8% an)
Qb,,, a4) D(be, ax) Ps,o Rs.o(a:) (1-Ps.0(aa)) Rr.o(a:)
(4)
aEA(be)aa
where by is the robot's belief, (b,, (x) is the travel cost to reach the subgoal via known space, and b2 is the approximate updated belief, which reflects that the robot has moved to the subgoal Planning seeks to find the action or that minimizes expected cost: TT(b)) argmin, Qa a); the robot replans whenever the map is updated, proceeding until the goal is reached.
As our approach builds upon the insights of offline replay, all navigation policies in this work rely on the learning over subgoals planning abstraction. Planning for each is done via Eq. (4), and policy selection in this context can be thought of as choosing the network model No whose predictions about unseen space {Ps,e, Rs,,, RF } result in the best performance when deployed.
4 Multi-Strategy Deployment-Time Learning and Adaptation
4.1 Problem Formulation: Non-stationary Policy Selection
We seek achieve minimum-expected--cot performance for navigation in partially-revealed envi- ronments, Our robot is deployed with an ensemble of learning-informet policies-many of which are being learned or adapted after each trial inal-add seeks to pick the best performing strategy during deployment. We formulate this problem as an instance of non-stationary policy selection. A deploy- ment is a sequence of trials, each a navigation from start to goal in previously unseen map. As many of the policies are learned or adapted and so evolve over time, we add an additional subscript k denote the policy after trial k. Thus, before trial k + the robot has access to N policies T and seeks pick the one that minimizes expected cost via
I (u T2k.-
7
argmin ([C((T,))] Elll
Though Eq. (5) resembles Eq. (I), computing the expected cost of policies in Eq. (5) challenging because the policies themselves are continually being updated via learning or adaptation during deployment. The policy selection strategies discussed in Sec. (Eq. (2) & (3)) use rolling average to estimate [[(((T.))] which quickly diverges from the true estimate for policies that improve via deployment- -ime training or adaptation, reducing the robot's performance. Instead, if selection is to quickly and reliably converge to the best performing policy, there is a need for an approach that can compute accurate bounds on the performance of each policy n. even as they improve.
4.2 Approach: Selection over Non-Stationary Policies being Continuously Learned or Adapted During Deployment
We present an approach that performs data efficient policy selection over a set of policies that are continually learned or adapted during deployment. Our policy selection approach chooses policies based on the selection strategy of Eq. (3), yet rather than using rolling averages of the lower-bound costs Tt and the deployment costs C3 that fail consider the evolving nature of the robot's policies, instead rely upon OFFLINER to refresh the estimates of the robot's performance.
Computing Up-to-Date Bounds on Performance Whenever one of the robot's policies is up- dated, via domain adaptation or learning with data it collects during deployment, its behavior may have changed. Thus, (T(TT))) * Cs(Taxi) * ... * C*(T,,) in general and so using a rolling average performance will result in poor selection performance and miss out on choosing policies that may have dramatically improved. Just as offline alt-policy replay can be used determine the would-be be performance of alternate policies after each trial, we can leverage this approach revisit old trials to determine how well the robot's 's updated policies would have performed. We use the data
5

---

computes the robot collects until trial & (the records (Z...=,,....k)) to replay how each of the robot's updated policies would have behaved in older trials, letting us get a much more accurate estimate of that policy's 's expected performance and performance bounds. For each updated policy Tak E Iī,, the updated lower bound performance computed by replaying its behavior for all trials:
& OFFLINEREPLAA( TTkk 2,)
1 K
Ch (TaK)
(6)
The average cost of deployed policies CA(F is recomputed similarly. We note that our prob- lem setting envisions that robots will not be in constant operation and can perform much of this computation for training re-evaluation while idle and waiting for its next navigation objective.
Policy Selection using Updated Performance Estimates We use the selection approach of Eq. (3) to perform policy selection, yet use our updated estimates of the performance of the robot's policies-loverr-ound costs C% and the deployment costs C2 computed via Eq. (6)-in place of their rolling averages. After each trial k, the robot initiates a procedure to learn or adapt its non- stationary policies using the data it has so far collected {Z, Before trial k updated performance estimates for each of its updated policies T.a E II via Eq. (6) and selects the policy to deploy via Eq. (3). Since data from new trials are continually added to record 2 for training and offline replay. these policies will asymptotically converge to static policies after many trials, and we preserve bandit-like guarantees on sub-linear asymptotic regret.
4.3 Deploying an Ensemble of Policies
Facilitated by our approach, we deploy our robot with an ensemble of policies; some are unchanging (e.g.. the robot's pre-trained policy) and others evolve and improve during deployment as more data collected- e.g., those that rely on visual domain adaptation or are trained from scratch. Policies in the ensemble are chosen so as to capture potentially distinct scenarios during deployment. Each such policy depends on the learning over subgoals planning (LSP) approach discussed in Sec. 3.3, and policy selection and domain adaptation in this context can be thought of simultancously improving and choosing between feed-forward models that make predictions about unseen space. Here, we discuss the different strategies employed by each policy our robot is deployed with.
The Pre-Trained Searning-Informee Policy [Stationary] AercTRAIN The robot is equipped with static policy trained in advance of deployment, a learning over subgoals planning (LSP) policy well-suited to good performance in the training environments. If we were know in advance that the deployment-time environments matched the training environments, would expect this policy to perform well and so no policy selection would be necessary. However, this work considers the more general case wherein the deployment environments may differ non-trivially compared to those seen during training, leading to poor performance when relying on TYRETRAIN-
The Non-Learned Optimistic Policy [Stationary] TNonLearned We also deploy the robot with an optimistic planning strategy: common non-leamndd strategy in which unseen space is assumed to be unoccupied. The non-learned optimistic policy can be thought of as special case of I.SP, in which Ps = 1 and Rs, R = 0, so that the robot simply selects the shortest path through the partial map to the unseen goal, replanning when necessary as unseen space is revealed. This strategy of optimism under uncertainty is unlikely to outperform Teretrain when the deployment environments match those seen during training. However, when deployed in an unfamiliar environment enn that differs significantly from the training environment, the learned model underpinning TeAETRAN may be misled by visual features it cannot properly understand, making TNoNLEARNED reasonable backup strategy.
Visual Domain Adaptation via CycleGAN [Non-Stationary] TCycLeGAN Our robot will use im- ages it collects during deployment to perform visual domain adaptation via the CycleGAN algo- rithm [19, 21J. Using unlabeled images from the training and deployment environments, CycleGAN learns a mapping from one the other, as shown in Fig. 4. This mapping is used as a preprocess- ing step: deployment-time images are made look like training-time images and then fed to the robot's pre- traitrained policy TeaETRANN The resulting policy Teyclegan can thus compensate for visual

---

dissimilarities between training and deployment, and so can improve performance when such changes are what limits the robot's behavior. However, visual domain adaptation cannol perform well in all deployment envi- romments, something policy selection must judge during deployment. For our experiments, one for 50 epochs TCvcLeGANço and one for 100 epochs TevennGA and allow policy selection to choose be- tween them. Additional details of the CycleGAN ap- proach and training can be found in the Appendix.
Visual Domain Adopiation uses CycleGAN to learn a mapping from deploymeti-linne imagees Iraining-lini -iiiinn--g-meee images. Se fotaay
Ioe Dipleaee
CycleGAN
the mupprng using unilabelled imsages.
TOYLLGGNN
train two models,
G
SraxTows
Action
Mapping used overcome visual shin.
Adaptation of a pre-trained policy using visual domain adaptation.
Figure 4:
Training from Scratch during Deployment | TeCratCh Using data it collects during deployment, the robot additionally train LSP-style policy from scratch while deployed. For LSP, the robot's learned model estimates for each subgoal (associated with a boundary between free and unseen space that may lead to the unseen goal) the likelihood Ps that the subgoal success- fully leads to the goal and the expected costs associated with reaching the goal Rs or needing to explore and turn back Ry. As the robot navigates, it can label image data using the partial map to train an LSP model: routes the robot discovered to reach the goal correspond to Ps = 1 and dead- ends Ps 0. Ambiguous routes-potential routes to the goal the robot did not explore during deployment can also be included in the training data under either optimistic prior, i.e., where all ambiguous routes are labeled with Ps 1. conservative prior, i.e., assumption that un- explored space is simply-connected, so that Ps- = 0. During deployment, we train one such policy for each of the optimistic prior Tecrathhоtt and conservative prior TTCRATChCON- While these policies will be invariably slow-to- conooo-c towards good performance, they have the potential to success- fully improve performance even in environments where visual domain adaptation not well-suited. We use 5-fold cross validation to determine an unbiased estimate of the updated lower bound on performance and deploy a policy trained on all trials.
Experimental Results
We demonstrate the effectiveness of our approach in simulated maze-like environments, procedu- rally generated so that each navigation trial sees a unique map. Each deployment consists of 50 navigation trials, each a traversal from start to goal through previously-unsenn map. As de- scribed in Sec. 4.3, the robot has an ensemble of six policies to choose from during deployment:
II (NNALLEARND, TRUETRAN:: #CrcuzdiNço; TTruuatiAN TSScAnCuDrr; WScKAvenCON J; to save computation, the non-stationary policies are updated via learning adaptation only every 10 trials. In addition to our approach for non-stationary policy selection CONSTATIONARYREPL... we include results with UCBBANDIT and ROLLINGREPLAY [13] policy selection, which performs selection via Eq. (3). See the Appendix for more details on the choice of these baselines.
We deploy in three maze variants, each with differing (often conflicting) visual that signal routes the unseen goal. Results (Fig. 5) show average performance and cumulative regret of each selection strategy over time. We additionally show (Fig. 5 bottom) our CONSTATIIONAYRERS estimate of the cost lower bound Thhh over time, lending insight into its selection process.
Deployment in Maze-Green: green path signals routes the goal, the floor is gray, and blue path leads to dead-ends. The stationary TaataAI is trained offline in held-out maps from this en- vironment and so is the best. While the UCBBANDIT converges slowly, both NONSTATIONARY REPLAY and ROLLINGREPLAY expectedly select PaETTAI quickly, achieving near-optimal performance.
Goal
Deployment Maze-Gray: A gray path signals routes to goal. the floor is green, and the blue path leads to dead ends. Mis- led by the green flooring, TeeETRAIN performs poorly and so both NonSiAtIONaRyEEPlAY and ROLLINGREPLAY, initially select the non-learned TNonleeaned- How- ever, only our approach considers that policies may evolve during deployment. After 10 trials, the
Goal
7

---

Deployment Environment
ROLLING REPLAY [13]
NONSTATIONARY REPLAY (ours)
Our Improvement vs. ROLLING
BANDIT
UCB
Avg.
Cost
Maze-Green
187.9
161.6
161.6
14.0%
0.0%
Maze-Gra
183.7
189.3
165.0
10.2%
12.8%
Maze-Blue
243.5
202.3
196.5
19.3%
2.9%
Cumul. Regret
Gaze-Greenn
1375,2 1848.6
61.2
61.2
95.6%
0.0%
Maze- Maze-Gray
2131.2
916.2
50.4%
57.0%
Maze-Blue
2955.6
898.2
606.6
79.5%
32.5%
Maze-Green
Maze-Gray
Maze- Blue
Anarge Deplan ieleeeeddd Croi.
718
400
Rovauates performs pearly
100
100
Commbative
4000
6008
Degget
NONSTATIOON SATATIONAE HEPLAY
2000
cornectly pld6 Tpremoss
2000
7000)
Avg Booker
Coet (C")
adkytation, covbeci colitssio
100
Towaetia
100
Nun
Thinls
Trids
Pulley Solestion Apprnnds
UCBBANDIT
ROLLNNREPLAY
AONPEA (Ouns)
Single Policy. Duployment
TPIRTRAIN
#CymaGAN50
#eymEGANN00
Figure 5: Our NONSTATIONARYREPLAN policy selection approach outperforms both UCBBANDIT and ROLLINGREPLAY [13] baselines in both Average Navigation Cost and Cumulative Regret.
TevclegaN policies improve via visual domain adaptation and our approach quickly switches to them, achieving 57% lower cumulative regret compared to RollinGreplay Notably, selecting Tevclsaia before improves would reduce performance; thus, our NONSTATIONARYREEN outperforms all single-policy selection strategies.
Deployment in Maze-Blue: A blue path on the ground signals routes to goal, the floor is gray, and the green path leads to dead ends. TevhhGgA cannot resolve the visual dissimilarities between training and deployment even after many trials, and so the policies trained from scratch Teehaich are the most promising approaches despite their slow convergence. After 30 trials, TecratchCon improves sufficiently and our NONSTATIONARY REPLAY selects it, resulting in 32% lower regret compared to Rol.LINGRE number that would continue to grow with further trials.
Goal
6 Conclusion, Limitations, and Future Work
We present approach to monitor and quickly select between Searningiinfrmme navigation poli- cies, many of which are being continuously learned adapted during deployment. Our approach facilitates deploying multiple-such strategies for learning and visual domain adaptation in parallel, allowing our robot to choose the best among them over time as demonstrated in simulated visual maze-like environments which we outperform state-of-the-art selection strategies by a consider- able margin. In future, we aim apply our approach to other domains like multi-robot planning and task planing under uncertainty since these problems also show promise for formulation in way that allows counterfactual reasoning despite uncertainty and so hold promise for applying our approach.
Limitations Although our policy selection approach not particularly computationally intensive on its own, each strategy for learning/adaatation relies on training deep neural network, and so re- quires considerable computation as the number of non-stationary policies grows (see the Appendix for further discussion). Moreover, our policy selection method relies on policies amenable to coun- terfactual reasoning of which learning subgoals planning (LSP) is one- one-and may not be directly applicable for selection between model-free navigation methods, limiting its broad utility.

---

Acknowledgments
This material based upon work supported by the National Science Foundation (NSF) under Grant No. 2232733. This work done at Robotic Anticipatory Intelligence & Learning (RAIL) Group and RobotiXX Laboratory at George Mason University. RobotiXX research is supported by National Science Foundation (NSF, 2350352), Army Research Office (ARO, W91 INF22202224 I91122220004 W91 US Air Forces Central (AFCENT), Google DeepMind (GDM), Clearpath Robotics, and Raytheon Technologies (RTX).
References
[1] L. Lai, H. Robbins, et Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 1985.
[2] W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 1933.
[3] J. C. Gittins. Bandit processes and dynamic allocation indices, Journal of the Royal Statistical Society: Series B (Methodological), 1979.
[4] Q. M. Rahman, P. Corke, and F. Dayoub. Run-time monitoring of machine learning for robotic perception: A survey of emerging trends. IEEE Access, 2021.
[5] Mallozzi, Castellano, P. Pelliccione, G. Schneider, and K. Tei. A runtime monitoring framework enforce invariants on reinforcement learning agents exploring complex environ- ments. In 2019 IEEE/ACM 2nd International Workshop on Roboties Software Engineering (RoSE), 2019.
[6] W. Zhou, S. Berrio, S. Worrall, and E. Nebot. Automated evaluation of semantic segmen- tation robustness for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 2019.
[7] S. Daftry, S. Zeng, J. A. Bagnell, and M. Hebert. Introspective perception: Learning to predict failures in vision systems. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016.
[8] H. Liu, S. Dass, R. Martin-Martnn, and Y. Zhu. Model-based runtime monitoring with interac- tive imitation learning. IEEE International Conference on Robotics and Automation (ICRA), 2024.
[9] Lee, A. Pacchiano, V. Muthukumar, W. Kong, and E. Brunskill. Online model selection for reinforcement learning with function approximation. In International Conference Artificial Intelligence and Statistics. PMLR, 2021.
[10] J. Reisinger, P. Stone, and R. Müikkulainen. Online kernel selection for bayesian reinforcement learning. In Proceedings of the 25th International Conference on Machine Learning, 2008
[11] A. Ghosh and S. R. Chowdhury. Model selection in reinforcement learning with general func- tion approximations. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, 2022.
[12] A. Pacchiano, Dann, Gentile, and Bartlett. Regret bound balancing and elimination for model selection in bandits and RL. arXiv preprint arXiv:2012.13045 2020.
[13] A. Paudel and G. Stein. Data-efficient policy selection for navigation in partial maps via subgoal-bart based abstraction. In International Conference Intelligent Robots and Systems (IROS), 2023.

---

[14] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 2016.
[15] M. Long, Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. International Conference on Machine Learning, 2015.
[16] K. Bousmalis, N. Silberman, Dohan, and D. Erhan. Unsupervised pixel-level domain adap- tation with generative adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[17] Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros, and T. Darrell. Cy- CADA: Cycle-consistent adversarial domain adaptation. In International Conference Ma- chine Learning. 2018.
[18] Y. Taigman, A. Polyak, and L. Wolf. Unsupervised cross coss-domain image generation. In Inter- national Conference on Learning Representations, 2017.
[19] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
[20] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M. Kalakrishnan, L., Downs, J. Ibarz, Pastor, Konolige, al. Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
[21] G. J. Stein and N. Roy. GeneSIS-RT: Generating synthetic images for training secondary real- world tasks. In 2018 IEEE International Conference on Robotics and Automation (ICRA), 2018.
[22] M. Wulfmeier, A. Bewley, and I. Posner. Addressing appearance change in outdoor robotics with adversarial domain adaptation. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.
[23] Palazzo, D. C. Guastella, L. Cantelli, P. Spadaro, Rundo, Muscato, D. Giordano, and C. Spampinato. Domain adaptation for outdoor robot traversability estimation from RGB data with pafety-preserving loss. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
[24] D. Bradley and J. A. D. Bagnell. Domain adaptation for mobile robot navigation. In NeurIPS 2009 Learning from Multiple Sources with Applications to Robotics Workshop, December 2009.
[25] A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid motor adaptation for legged robots. In Robotics: Science and Systems, 2021.
[26] Y. Zhu, P. Thangeda, M. Ornik, and K. Hauser. Few-shot adaptation for manipulating granular materials under domain shift. Proceedings of Robotics: Science and Systems, July 2023.
[27] A. S. Chen, Chada, L. Smith, Sharma, Z. Fu, Levine, and C. Finn. Adapt on-the- go: Behavior modulation for single-life robot deployment. In NeurlPS 2023 Robot Learning Workshop, 2023.
[28] L. Smith, J. C. Kew, X. B. Peng. Ha. Tan, and Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In 2022 International Conference on Robotics and Automation (ICRA), 2022.
10

---

[29] N. Hansen, R. Jangir, Y. Sun, G. Alenyå, P. Abbeel, A. A. Efros, L. Pinto, and X. Wang. Self- supervised policy adaptation during deployment. International Conference on Learning Representations, 2021.
[30] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learn- ing to adapt in dynamic, real-world environments through meta-reinforcement learning. In International Conference on Learning Representations, 2019.
[31] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introdaction. MIT Press, 2018.
[32] L. P. Kaelbling, M. L. Littman, and A. Cassandra. Planning and acting in partially observ- able stochastic domains. Artificial Intelligence, 1998.
[33] G. J. Stein, C. Bradley, and N. Roy. Learning over subgoals for efficient navigation of struc- tured, unknown environments, In Conference on Robot Learning. PMLR, 2018.
[34] K. Arndt, M. Hazara, A. Ghadirzadeh, and V. Kyrki. Meta reinforcement learning for sim-to- real domain adaptation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), 2020.
[35] A. Ajay, Gupta, D. Ghosh, S. Levine, and Agrawal. Distributionally adaptive meta reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[36] G. Schoettler, A. Nair, A. Ojea, S. Levine, and E. Solowjow. Meta-reinforreemen learning for robotic industrial insertion tasks. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
[37] J. Shin, A. Hakobyan, M. Park, Y. Kim, G. Kim, and I. Yang. Infusing model predictive control into meta-reinforcement learning for mobile robots dynamic environments. IEEE Robotics and Automation Letters, October 2022.

---

Multi-Strategy Deployment-Time Learning and Adaptation for Navigation under Uncertainty: Appendix
A CycleGAN Implementation and Training
We use the official PyTorch implementation of CycleGAN provided by Zhu et al. [19] on GitHub'. As mentioned in Sec. 4.3, we train two models: one for 50 epoch and another for 100 epochs. We use the default parameters for training except that we use a batch size of and no learning rate scheduling done. We use learning rate of 0.0002 with Adam optimizer. The images are of size 512 128 and we perform default resizing and cropping as a preprocessing step. 1261 images from 10 distinet maps in Maze-Greet (where TeRETRAI is trained) are set aside target domain images, and we sample 1300 images from the source domain (either Maze-Gray or Maze-Blue) collected during deployment for training the CycleGAN model. Using only 1300 images from deployment- time environments additionally helps to get an unbiased estimate of performance when replaying MeceGGN-adaaptdd policy Teyclegan on older trials from which the images were collected.
B Learning over Subgoals Planning: Subgoal Property Estimator Network Implementation and Training
As discussed in Sec. 3.3, all Bearinnginfforeer policies rely on the learning over subgoals planning (LSP) abstraction for planning, a model-based approach that relies on a learned model to estimate subgoal properties: statistics of unseen space associated with each of the robots temporally-extended high-level actions to explore unseen space. The subgoal property estimator network No correspond- ing to TehETRINN is trained with data-collected in 500 distinct maps in Maze- Geeee where the robot navigates using the Tnonlearned policy. Data labelling procedure is similar to the one described for TSCRATen (Sec. 4.3) except that at training time the underlying map is known and so can be used to generate ground truth labels for subgoal properties Ps, Rs and RF corresponding to all subgoals.
The subgoal property estimator network Ng is trained via supervised learning using the data col- lected during offline training phase. Our neural network architecture and training procedure resemble that of Paudel and Stein [13]. The network takes as input 512 x 128 panoramic image centered on a subgoal, relative distance to the subgoal and relative distance to the goal. The image is encoded by passing through 4 convolutional layers and then concatenated with features corre- sponding to relative distances to subgoal and goal after which the concatenated features are passed through convolutional layers and finally 5 fully connected layers to output subgoal properties Ps, Rs and Rr. We use learning rate of 0.002 with a decay factor of 0.5 every epoch and train for 8 epochs with Adam optimizer. We use cross-entrops loss for learning logits associated with P's and L.2 loss for learning Rs and Ry. The deployment-time training of subgoal property estimators for SCRATCCH follows a similar architecture and procedure.
C Offline Alt-Policy Replay Details
offline alt-policy replay to replay the behavior of a policy without deploying it. To replay the behavior of policy n', we leverage the record Z, collected during trial k under a deployed policy At every time step during offline alt-policy replay, the robot leverages the final partial map Minii observed in trial k to simulate the laser scan and updates its observed map as the robot moves. The frontiers-boundartie between free and unknown space- revealed the observed map corresponds to the subgoal-actions that the robot can take to explore the region. To get the robot-view panoramic image corresponding subgoal-action, we retrieve
As discussed in Sec. 3.2,
http:p/gg....booooo/o//////////s anddixxpp
12

---

existing image in record 2 that is closest to and in line of sight to the subgoal and recenter it at the subgoal. This image is used to estimate the subgoal properties Ps, Rg and Rs using a neural network corresponding to x' which then used to compute the next high-level action using Eq. (4). The robot then simulates the low-level motion primitive to move towards the selected subgoal and this process is repeated. At any point during simulated navigation, if the robot attempts enter region that is unknown in the final partial map Miaal via frontier, mask that frontier and force the robot to pick different subgoal-action. The net distance travelled to reach the goal via this procedure is the replay cost of policy n².
D Cross Validation for Reevaluating Older Trials
As mentioned in Sec. 4.3, we use 5-fold cross validation to get unbiased estimate of the per- formance of updated policies. Since our policies trained during deployment from scratch (TScRAtcH) are based on the same data in record 2 that is also used for offline alt-policy replay to reevaluate older trials after updating the policies, such volssvalliationn approach overcomes the risk of overes- timation of performance during replay due to data leak. With 5-fold cross validation, 5 policies are trained, each the data from four-fifth of older trials, and replayed on the remaining one-fifth of policy is trained
the trials to get the revised performance estimates for all older trials. Finally, on data from all previous trials and made available for the robot to choose from in the next trial.
E
Ablation Studies
We study the effect of removing the most effective learning/adaptation strategies from Maze- Gray and Maze-Green and see how the performance varies. Specifically, we remove CycleGAN- based policies (FevcusGANço and TovclegAN1oo) from Maze-Gray and policies trained from scratch (ТsсилтснсоU and ТsCиAтCнон)) from Maze-Blue. The results are shown Table 1. We observe that in both environments, removing the corresponding best learning/adappationn strategies leads our NoN- STATIONARYREPLAY approach still outperforming or on-par with baselines. We also observe that the average navigation cost for SONStATIONARY ing/adaptation strategies are removed compared to those in Fig. 5 where these strategies were in- cluded.
approach increases when the best learn-
Table 1: Results of ablation with removing best leamiin/addaptaioon strategies corresponding to Maze-Grap and Maze-Blue
Environment/ Ablation
UCB BANDIT
ROLLING REPLAY [13]
NONSTATIONARY REPLAY (ours)
Our Improvement vs. UCB ROLLING
Maze-Gray
Average Cost
190.8
197.1
176.3
7.6%
10.56%
(w/o #CVeLEgAN)
Maze-Blue
234.6
202.3
202.3
13.8%
0.0%
(w/o Tscaaien)
Maze-Gray
CCumulative
Regret
(w/o WeveLrgAN)
1917.0
2232.0
1191.6
37.8%
46.6%
Maze-Blue
2053.8
441.0
441.0
78.5%
0.0%
(w/o Wsensten)
F Additional Discussion of Trends in Results
Adaptation of TeyelegAN in Maze-Gray and Maze-Blue As is mentioned in Sec. 5, Maze-Blue has blue path signaling routes goal and green path leading dead ends, and is deliberately de- signed to be indistinguishable from Maze-Green based only on visual observations in absence of the
13

---

Figure Sample Trajectories for PRRTTRAIN Maze-Blue
FEG
KEDE
Figure 7: Sample Trajectories for TPPETRAIN in Maze-Gray
information about goal location. To clarify, CycleGAN does learn reasonable visual mapping be- tween the two environments: it learns an approximate identity mapping (see images corresponding to Maze-Blue in Fig. 8) from the visual observations from each. However, we note that this mapping not helpful for resolving the best path to the goal since the robot is still drawn to follow the green path, which no longer leads the direction of the unseen goal. Hence, the performance of TeyaleGAN poor in Maze-Blue and say that "visual domain adaptation fails" to properly adapt. On the other hand, in Maze-Gray, the color of the floor and goal-routes CycleGAN learns mapping between the two that "swaps back" the colors of floor and path, map- ping that then allows the robot to correctly identify the best path to the goal, resulting in improved performance of TeyclGAN in Maze-Gray.
swapped compared to Maze-Green.
Performance Differences of FaretRaIn in Maze-Blue and Maze-Gray The TearTRAN policy trained in Maze-Green would have learned to follow the green path to find the goal and avoid the blue path to dead ends. As such, deploying it in Maze-Blue where green path leads to dead ends and blue path leads to goal (as illustrated in the respective environment's 's figures in Sec. 5) would often mislead Taactrann to navigate towards the dead ends increasing the cost to find the goal (see example trajectories in these environments as shown in Fig. 6 and 7). In Maze-Gray, this phenomenon is less severe since the dead ends are still blue and are often avoided by TEETRRIN-
Performance Similarities of TereTraI and TeyclEGAN in Maze-Blue As discussed in the afore- mentioned paragraph, the visual observations from Maze-Blue and Maze-Green are similar in ab- sence of the knowledge about goal. This causes CycleGAN generator in TeveleGAN to effectively learn an identity mapping (see images for Maze-Blut in Fig. 8) and therefore RevelbaAa policies are effectively equivalent to Tiretrain (see Fig. 4 on how TereTraIn is used by TCyclegAN) resulting in very similar performances over trials Fig 5.
G Sample Images from Deployment-Time Visual Domain Adaptation
Image samples transformed from deployment-time environments (Maze-Gray or Maze-Blue) the training- -ime environment (ieze) are shown in Fig, 8. The CycleGAN models trained after 40th trial are used to generate the images.
14

---

Maze-Green Maze-Green
Maze-Gray to Maze-Green
Maze-Bi Blue Maze-Green
Figure 8: Images transformed from deployment-ting time environments (input) to look like training -imm environments (output) with CycleGAN based visual domain adaptation.
H
Additional Related Work
Runtime Monitoring Runtime monitoring approaches aim to evaluate the reliability learned models used to guide the robot's behavior during deployment [4, 5, 6, 8] and decide whether fallback policy should instead be used. However, these approaches focus primarily on evaluating certain subsystems (e.g., the perception system) [5, 8] scrutinizing learned model's raw output instead of the robot's task performance [6, 7]. thus limiting their applicability for evaluating the robot's long-horizon behavior for navigation in partially-mapper environments.
Adaption with Rentoreeinorreemmett Learning Many problems in robotics have leveraged meta- reinforcement learning (meta-RL) approaches for learning robot policies that can adapted to new tasks scenarios [30, 34, 35, 36, 37]. While meta-RL is most effective when provided access during training distribution over environments to which the deployment-time environments may belong, our work is geared more towards reliably identifying and deploying existing general-purpose online training and domain adaptation tools meant to handle systematic differences between training and deployment-time environments. As such, our approach allows for integrating a wide range of existing leamiin/addaptaiion strategies as demonstrated in our experiments with a mix of different Srlciiss-somethinne that meta-RI. is not geared towards. Additionally, our approach makes no assumptions about similarities between training and deployment environments while still benefiting from asymptotic performance guarantees.
I Choice of Baselines
We compare NONSTATIONARYREPLAYY approach with UCBBANdit and ROLLINGREPLAY baselines both of which assume that the policies are stationary. While it would have been suitable to
15

---

compare our approach with non-stationary policy selection approaches, existing potential baselines in this area either assume policies are stationary or leverage knowledge we do not have access to in the context of robot navigation, e.g. underlying distribution of a policy's navigation cost. As such, ours the first to propose such non-stationary policy selection algorithm for robot navigation under uncertainty, and hence our baselines only include the most suitable stationary policy selection approaches from the literature.
J Scalability and Computational Limitations
Our approach scales linearly in terms of both the number of policies and the number of trials: the ad- dition of new trial requires that offline alt-policy be run for the policies that were not deployed and, as policies are retrained, each old trial is re-evaluated. Such replay isn't computationally intensive- replaying policy in single typical trial takes roughly 20 seconds-and hence scales well over larger number of trials and policies. Additionally, our policy selection approach not particularly computationally demanding on own, but that the underlying strategies for training or adapting policies online (e.g., via CycleGAN) can be expensive, costs that are not specific to our policy selec- tion approach, and should not be seen limitation specific our policy selection approach.
In real-world settings, the computation associated with training policies and selecting between them will not preclude using policy selection via our approach; specifically, our problem setting envisions that robots will not be in constant operation and can perform much of this computation and policy selection while idle and waiting for next navigation objective. Thus, such computation associated with our approach would not necessarily impede navigation performance. For very lengthy deploy- ments spanning thousands of trials, indeed approach might run into practical limitations since determining accurate estimate of the robot's performance requires replaying all previous trials. In future work, one could consider only selectively replaying specific trials or randomly sampling a subset of trials to replay to save on computation while preserving our asymptotic performance guarantees.
K Compute Platform and Execution Time
We ran our experiments on Intel i9 CPU with NVIDIA RIX A6000 GPU. As mentioned earlier, the most computational intensive tasks our experiments are training the CycleGAN and the neural networks. While deploying the robot with our approach, training policy from scratch (8 epochs using data from 50 maps) would take around minutes, and training CycleGAN (100 epochs with about 1300 images in each domain) would take around 3 hours. By contrast, performing offline alt- alt-policy replay for single policy on single typical trial takes about 20 seconds.
16


Reviews:
[{"review_id": "XvAzlmreJs", "reviewer": "Reviewer_UHRD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach for efficient point-goal navigation in unfamiliar partially observable environments. The key idea is to deploy multiple strategies for deployment-time learning and visual domain adaptation in parallel, and then quickly select the best-performing strategy. A single deployment here is defined as a sequence of trials. The authors formulate this as a non-stationary policy selection problem and propose a novel method to estimate up-to-date performance bounds using offline alt-policy replay for policies that are continuously evolving during deployment . The approach is evaluated in simulated maze-like environments and shows promising results compared to baselines.", "summary_of_recommendation": "I think this paper explores an interesting research problem that will become more and more common as we start to have multiple competing policies for the same task in robotics. While the experiments of the paper are in simple simulated environments, I think they are sufficient to validate the main contributions of the work.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "- The approach is heavily dependent on the Learning over Subgoals Planning framework as offline alt-policy reply requires planning be done via a policy amenable to counterfactual reasoning. Could you elaborate on why this would mean a hard requirement of LSP?\n- How realistic is a multi-trial per deployment assumption for a real world setting?\n- How scalable is this approach, both in terms of number of policies in the pool and longer deployments with hundreds or thousands of trials?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFh34ehzog", "reviewer": "Reviewer_rHoK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for online policy selection from an ensemble of evolving, nonstationary policies being updated. The proposed method leverages offline alt-policy replay to compute would-be performances of policies, and perform selection based on these metrics.", "summary_of_recommendation": "Although the core components of the proposed approach are based on prior work, the combination of them still address a very challenging problem in planning, therefore I recommend weak accept. After rebuttal, the authors answered my questions on the results, and some ratings are updated accordingly.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "1. What are the limits on the size of the ensemble of policies? Is the scalability only related to computation resources available? If not then would having more policies in the ensemble potentially help performance online?\n2. Are there any insights into why CycleGAN adapts to Maze-Gray but not Maze-Blue?\n3. It seems like the pretrained policy only exhibits a high cost in Maze-Blue but not in Maze-Gray from Figure 5 but in the main text (Sec. 5) it was mentioned that pretrained also performs poorly in Maze-Gray. Shouldn’t the performance be similar in both of these scenarios? \n4. Related to Question 3, it seems like the performance of pretrained and CycleGAN policies are somewhat mirroring each other. Are there any explanations for this phenomenon?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4JKt9wePrN", "reviewer": "Reviewer_APKz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel method for online selection of policies as an adaptation method to new visual navigation environments. The proposed method assumes access to an ensemble of policies, and takes into account their non-stationarity when selecting which policy to use. The selection algorithm reevaluates current policies on previous experience, relying on the counterfactual property of the LSP framework. Based on a previous approach, the algorithm proposed in this paper inherits lower bounds on cumulative regret.", "summary_of_recommendation": "While this paper proposes a valid approach to solving an interesting problem, it seems like its scope is limited, and it is hard to see how it can be applicable to real-world robotic applications beyond a specific use-case. Therefore, I recommend rejecting the paper.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "See Review section above for questions and concerns.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 3 fairly confident reviews. The manuscript presents an approach for efficient point-goal navigation in unfamiliar partially observable environments. The key idea is to deploy multiple strategies for deployment-time learning and visual domain adaptation in parallel, and then quickly select the best-performing strategy.

To address in rebuttal:
- Add results that isolate the contributions of individual components (ie, an ablation)
- Address scalability of approach wrt number of trials and number of policies; computational limitations
- Further address the nature (interpretation) of the results, as pointed out by reviewers.

Post-rebuttal:
The authors address many of the concerns raised raised by the reviewers; some doubts remain about the applicability of this approach to real-world robotic scenarios (the experiments of the paper are in simulated environments). Overall, the paper explores an interesting research problem in cases where multiple competing policies exist for the same task).
Decision: Accept