DROID: Learning from Offline Heterogeneous Demonstrations via Reward-Policy Distillation
Authors: Sravan Jayanthi
Letian Chen
Nadya Balabanska
Van Duong
Erik Scarlatescu
Ezra Ameperosa
Zulfiqar Haider Zaidi
Daniel Martin
Taylor Keith Del Matto
Masahiro Ono
Matthew Gombolay
Venue: CoRL 2023 Poster (2023)
Keywords: Learning from Heterogeneous Demonstration
Network Distillation
Offline Imitation Learning

Abstract: Offline Learning from Demonstrations (OLfD) is valuable in domains where trial-and-error learning is infeasible or specifying a cost function is difficult, such as robotic surgery, autonomous driving, and path-finding for NASA's Mars rovers. However, two key problems remain challenging in OLfD: 1) heterogeneity: demonstration data can be generated with diverse preferences and strategies, and 2) generalizability: the learned policy and reward must perform well beyond a limited training regime in unseen test settings. To overcome these challenges, we propose Dual Reward and policy Offline Inverse Distillation (DROID), where the key idea is to leverage diversity to improve generalization performance by decomposing common-task and individual-specific strategies and distilling knowledge in both the reward and policy spaces. We ground DROID in a novel and uniquely challenging Mars rover path-planning problem for NASA's Mars Curiosity Rover. We also curate a novel dataset along 163 Sols (Martian days) and conduct a novel, empirical investigation to characterize heterogeneity in the dataset. We find DROID outperforms prior SOTA OLfD techniques, leading to a $26\%$ improvement in modeling expert behaviors and $92\%$ closer to the task objective of reaching the final destination. We also benchmark DROID on the OpenAI Gym Cartpole environment and find DROID achieves $55\%$ (significantly) better performance modeling heterogeneous demonstrations.

DROID: Learning from Offline Heterogeneous Demonstrations via Reward-Policy Distillation
Sravan Jayanthi Letian Chen st.1, Nadya Balabanska', Van Duong², Erik Scarlatescu',
Ezra Ameperosa', Zulfiqar Zaidi', Daniel Martin', Taylor Del Matto', Masahiro Ono?, Matthew Gombolay
School of Interactive Computing, Georgia Institute of Technology
Jet Propulsion Laboratory, California Institute Technology
Equal Contribution, 'Corresponding Author: letian. chen@gatech. edu
Abstract: Offline Learning from Demonstrations (OLID) is valuable in domains where trial-and-error learning is infeasible or specifying cost function is diffi- cult, such as robotic surgery, autonomous driving, and path-finding for NASA's Mars rovers. However, two key problems remain challenging in OLfD): 1) hetero- geneity: demonstration data can be generated with diverse preferences and strate- gies, and 2) generalizabiliyy: the learned policy and reward must perform well in unseen test settings beyond the limited training regime. To overcome these chal- lenges, we propose Dual Reward and policy Offline Inverse Distillation (DROID) that leverages diversity to improve generalization performance by decomposing common and individual-speeific strategies and distilling knowledge in both the reward and policy spaces. We ground DROID in novel and uniquely chal- lenging Mars rover path-planning problem for NASA's Mars Curiosity Rover. We curate novel dataset along 154 Sols (Martian days) and conduct a novel, em- pirical investigation to characterize heterogeneity in the dataset. We find DROID outperforms prior SOTA OLfD techniques, leading to a 21% improvement in mod- eling expert behaviors and 90% closer to the task objective of reaching the final destination. We also benchmark DROID on the OpenAl Gym Cartpole and Lunar Lander environments and find DROID achieves 23% (significantly) better perfor- mance modeling unseen holdout heterogeneous demonstrations.
Learning from Heterogeneous Demonstration, Network Distillation, Offline Imitation Learning
Keywords:
Introduction
Deep Reinforcement Learning (Deep RL. has achieved success in generating nigh-performance con- tinuous control behaviors but requires high-fidelity simulator 4,5, 6, 7, 8). A counterexample is the Mars Path-Planning (MPP) problem where one must construct a path through series of waypoints for the Mars rover to traverse towards a destination without an explicit notion of reward or simulator. Not only is this challenging due to the chaotic terrain but also because of factors such as physical limitations the rover's capabilities and unobservable terrain information [9, 10]. Expert human Rover Planners (RPs) at NASA design paths under time and safety constraints based their expertise crafted over years of experience knowledge that has yet to be codified [9]. Efforts have been made to automate the process with symbolic and connectionist (e.g., Deep RL) approaches [11, 12, 13]. However, these methods do not match the human RPs' success because it is difficult to codify experts' knowledge into cost function [13, 14], and large homogeneous dataset is usually required to learn robust policies [9]. Such challenges not only ex- ist in the MPP problem but are prevalent in other robotic applications such as surgery, search and rescue, self-driving, and elderly care [15, 16, 17, 18, 19, 20, 21].
dataset [1, 2, 3,
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Sol
Sol
Sol
4
Smooth Terrain
Rocky Terrain
Rough Terrain
Choatic Terrain
Driver 1
Driver 1
Driver 2
Driver 2
Diverse Demonstration
Modeling
Driver
Driver 2
R₁ = Ry R51
R₂ Ry + R52
e Ør Qsi
Q. = Qr + Qsz
Bellman Equation
Reward Distillation
Aggressive Strategy
Risk-Averse Strategy
Shared Reward := Ry
Common Task Policy := Qr
Policy Distillation
Figure 1: This figure shows how DROID infers (blue arrows) the underlying expert strategies given the MPP dataset with varying preferences (e.g., Aggressive vs. Risk-Averse) across heterogeneous states (Smooth to Chaotic Terrains). DROID performs knowledge distillation to a common task policy (black arrows) shared by all demonstrators, infers reward that encodes each expert's latent preference (orange arrows), and identifies the shared reward across demonstrators (green arrows).
Learning from Demonstration (LfD) is promising paradigm to address this challenge: LiD meth- ods learn by having users demonstrate the desired behavior on the robot, removing the need for cost function specification [22]. However, most LID approaches are limited by the need for many envi- ronment interactions [23, 24, 25]. robotic applications, the exploratory environment interactions could be costly (e.g., damaged or lost rovers), unethical (e.g., surgery), or unsafe. Appropriately, Offline LfD (OLfD) has been proposed as framework that allows for training a robot policy solely from pre-recorded demonstrations with assumption about viable simulator [26].
OLID relaxes the requirement for reward function and simulator, it however faces several al- gorithmic challenges that limit its full potential [26, 27]. First, key challenge for OLID is het- erogeneity within the demonstration set. Each expert has individual preferences stemming from varying cognitive biases [28, 29] or different latent goals [30, 31] for accomplishing a given task. the LID algorithm assumes homogeneity about heterogencous data, the robot may fail infer the expert intention [28, 32] and the learned policy may perform poorly [28, 33]. On the other hand, learning each heterogeneous behavior separately is data-inefficient and prone to overfitting. The second critical challenge is learning from limited dataset [34]. The limited data makes it difficult for the learned policy and reward functions to capture the user's latent intentions and to generalize beyond the demonstrated setting [15, 35, 36, 37].
In this paper, propose novel OLID approach, Dual Reward and Policy Offline Inverse Distillation (DROID), that simultaneously distills common task policy and reward from diverse demonstrators, while modeling individual preferences along both strutegy-specifie policies and re- wards. This approach allows us to extract an unbiased task objective while understanding various styles of accomplishing the task (Figure 1). Our contributions are three-fold:
1. We curate a novel dataset with RP-designed Mars Curiosity Rover paths for 154 Sols (Martian days) covering various terrains on Mars. We conduct a novel, empirical investigation to character- ize heterogeneity in the dataset, motivating the need for a OLID approach robust to heterogeneity.
We propose DROID, a framework that simultaneously distills knowledge through the learned policy and reward. We also introduce two improvements (Augmented Regularization Reward Maximization) the underlying IRL algorithm [15] to improve generalization performance.
We show DROID achieves 29% and 17% better modeling performance (measuring the distance between expert demonstration and the generated trajectory) than previous SOTA in Cartpole and Lunar Lander, respectively. On the MPP problem, also find DROID outperforms SOTA and gets 90% closer to the goal point (an important objective in the Mars path planning domain).

---

2 Related Work
In this section, we discuss related works in offline LiD, reward and policy distillation, and the Mars Path Planning problem. There has been extensive work for LfD in robotics and learning from heterogeneous demonstrations [30. 29, 33, 38. 39, 40, 41, 42, 43] but our focus offline learning from diverse, limited demonstrations, an unexplored problem setting.
Offline LID- - Few previous OLfD approaches model demonstration heterogeneity [44, 45, 46, 47, 48), which could cause the learned reward function and policy to fail at generalizing beyond the original demonstrated setting [35, 36] and capturing the individual preference of the expert [37]. Some work in OLfD attempts to tackle multimodal expert behaviors by increasing model capacity through Diffusion [49], BeT (Behavior Transformers) [50], and representation learning [51], but each fails to overcome the fundamentally mode-seeking behavior of LfD. DROID's explicit policy and reward decomposition is critical to success in modeling heterogeneity.
Reward and Policy Distillation - Several frameworks consider commonalities among reward func- tions across heterogeneous demonstrations [29, 38, 41J. However, these methods rely on online in- teractions with the environment, which is infeasible in many robotic domains. Policy distillation has been studied to improve policy transfer performance [52, 53). However, DROID is the first study simultaneous reward and policy distillation, particularly in the challenging setting of OLID.
Mars Path Planning - For NASA, it requires much human effort to plan paths for the Mars Curios- ity Rover [9]. Current approaches such as Autonomous Navigation (AutoNav) [54] do not consider all hazards that humans deem dangerous to the rover [55]. Hedrick et al. [56] proposes efficient Martian path planning, and Rover-IRL [57] learns cost function from demonstration, but both fail plan under missing/occluded terrain maps, which is key obstacle in the Mars domain [9]. Our al- gorithm, DROID, instead directly learns from how previous RPs drive in the midst of occluded info and has the ability to model an RP's s behavior when planning along unknown parts of the terrain.
3 Preliminaries
In this section, we introduce preliminaries Markov Decision Processes (MDP), Offline Learning from Demonstration (OL.D), and Multi-Strategy Reward Distillation (MSRD).
and correspond to the state and action spaces, R(s, a) the reward, and T(s'|s, ()) a) the transition probability for state s' after performing action a in state s. Y E (0, 1) is the discount factor and po denotes the initial (a]s) represents the probability of choosing action in state s. The state distribution. The policy л
Markoy Decision Process - A MDP, M, is a 6-tuple, (8, A, R.T.
Q-value is defined as (2, a) Er,T E 7" R(ss, as]ls0 5, ag a), denoting the expected discounted cumulative return following in the MDP under reward R.
Offline Learning from Demonstration IRL considers an MDP sans reward function (MDP)R) and infers the reward function R based on set of demonstration trajectories D (7 .., TN } where N is the number of demonstrations. Our method leverages Approximate Variational Reward Imitation Learning (AVRIL) [58] as the underlying IRI. approach. AVRIL. considers distribu- tion over the reward function and approximates the posterior, p(R). with variational distribution, qs (R). AVRIL introduces a second variational approximation for QF (s, a) with Qa a) and ensures the variational reward distribution, 4o [R), is consistent with the variational Q function, Qo a), by Bellman equation. We show the final objective function of AVRIL in Equation I.
exp(8Qu(s, a)) Ere exp(3Qo
LavaL
Σ
log
DE. Daulss (R.((s()))l)(()) + log (Qo(s, - "Ro(s',o)) (1)
Multi-Strategy Reward Distillation We propose general reward distillation framework based on previous online IRL technique, MSRD [38]. MSRD decomposes the per-strategy reward, R; for strategy as a linear combination of a common task reward and a strategy-only reward with neural network parameters TTak and ds R4 Rona + Reg_,. MSRD leverages regularization loss to distill common knowledge into @rak and retain personalized information in Øs-i.
3

---

Onboard Images
Gaming Area
DROTD Designed Trajectory
Figure 2:
This figure shows the dataset curation process for the MPP problem. We unify the height maps created by onboard cameras into single "gaming area" (middle figure) and then plan the driving path based on features calculated on the gaming area height map.
4 Mars Curiosity Rover Path Planning Problem
In this section, we introduce the Mars Path Planning (MPP) problem and how we 1) curate the dataset, 2) construct an MDP for OLfD, 3) analyze heterogeneity present across RPs.
Dataset Curation The raw data consists of height maps created by photos captured by Curiosity across multiple Sols (Martian days). The multi-resolution height maps are processed into sin- gle 64x64 "gaming area" by interpolation of overlapping height maps and scaling along each axis (Figure 2). The processed gaming area is then used to calculate nine features identified by RPs: (1) distance to the goal point, (2) unknown data percentage, (3) average roughness, (4) maximum roughness, (5) average pitch, (6) average roll, (7) maximum pitch, (8) maximum roll, and (9) turning angle. More details for the features are available in the Supplementary.
MDP Problem Setup We create novel formulation to convert the MPP problem into an MDP: a state contains the terrain information of the Sol and the current and target locations for the path planning. The action space, A, consists of all possible next waypoints in the gaming area. The reward function is constructed as a function of features associated with the path specified by a on the terrain 8: R(s, a) = f(b(,, a)) where & (S x A) + R the path feature mapping.
Analysis of Heterogeneity We perform PER- MANOVA test with
150
0.05 and the Holm method for the correction of multi-tests with the 37 different RPs to answer whether differ-
RP 2
100
RP 22
RP
12
TSNE feature
50
ent strategies exist among drivers in the fea-
RP 10
RP
ture space. The test shows significant differ-
RP 14
ences along seven pairs of RPs, particularly along the paths designed by RP 26 with spect to 5 other RPs, as shown in Figure 3 with TSNE [59] to reduce features into two dimen- sions (further explanation provided in the sup- plementary). This result shows heterogeneity in the RP-generated paths from differences ter- rains of the Sols and the expert RP strategies, motivating the need for offline learning from heterogeneous demonstration approaches.
-50
RP 19
-100
RP 26
-150
-150
-100
-50
50
100
150
TSNE feature
Figure 3: This figure shows heterogeneity in MPP dataset. Comparisons with p < .05 are repre- sented by connecting lines. RP IDs are labeled and marked red.
Methods
The challenges of heterogeneity and limited data prevalent for OLiD in many robotic applica- tions, particularly in the MPP problem shown in Section 4. To overcome these challenges, we propose our algorithm, DROID, for learning high-performing policies by distilling common infor- mation across heterogeneous demonstrations in both the policy space and the reward function space.
5.1 Reward Distillation
We propose a general reward distillation approach for OLfD. We model the reward distributions as mean-field Gaussian distributions partitioned on each state-action pair and let the reward neural networks output the mean and standard deviation of the Gaussian distributions. The advantage of Cauusian-distribation models for task reward (9 (s, a) ~ Jona (R) A (Prask (8, a), of (8. a)))

---

and strategy rewards (Rs-i(s. ~ (R) [ ((ss-((s,a) (s, is that the summation of two Gaussian distributions is still Gaussian distribution, as shown in Equation 2, where Ri denotes the random variable for the reward distribution of strategy i.
Rs, a) Rrask a) + Rs-s(s,a) a) ~ (Nrakkss a) + (ss a), oin (s, + (s, (2)
Assuming the number of strategies, M, and the strategy label, C, for T, is known priori, we can perform reward distillation on the strategy reward, as shown in Equation LRD((Os-r))=)
= [( [ll (s, a)ll
(3)
Intuitively, LRD pushes the strategy reward to output O and therefore encourages common knowledge to flow to the shared task reward and each individual strategy reward only captures preferences. The MSRD reward distillation formulation is a special case where the reward distribution for each strategy collapses a Dirac delta function.
5.2 Policy Distillation
We propose DROID to leverage commonalities in both reward and policy spaces. As policies are implicitly defined via the Q function in AVRIL by (aas)) = arg maXaea Q(s, a), we construct the Q function for each strategy as a combination task Q function and strategy Q function: Q, = Qraa + Qos :. As such, we propose to regularize the output of the strategy Q values, Qs-i. as in Equation 4, to encourage common information to be distilled into the task Q-value, QTask-
= (rr,,)) JD [llQs-a, (x.a)]l]
(4)
DROID's explicit knowledge distillation across diverse policies aids in improving generalization performance as the shared policy benefits from modeling all demonstrations in the offline dataset.
5.3 Enhancing DROID for Offline LID
We present two enhancements to construct the inductive bias useful for learning more accurate rewards and better-perfomming policies in OLiD. Improvement 1: Augmenting Dataset for Regularization. We introduce an augmented dataset, D' = {(s,,b) E D E A}, for regularizing (R) to a prior R) compared with AVRIL's KL divergence regularization only within the demonstration (Equation 1). By extending the operation of DKL to be on the entire action space, we are encouraging a conservative estimate the reward for any action that is not taken by demonstrators, following the pessimistic principle in OLfD [60].
L (D:
Σ
DRI DR..
eED,aEA
Intuitively, Equation 5 could be viewed as data augmentation technique to regularize reward learn- ing across the entire action space. In contrast, AVRIL's variational lower-bound only regularizes (xL(Ø) E(t,)eED DKL (g [(s, a)llp((((ss))))))) a))). on (s, a) samples from D. We provide a lemma describe the effect of this augmented regularization in the supplementary.
Improvement 2: Reward Maximization. The second improvement we propose is to maximize the reward given to the demonstrated action, as shown Equation 6.
Lynax action reward (Ø: D)
Σ rss,a) where r(s, ~ (R(ss)
(s.a)DD
In AVRIL, the reward function learning relies on the two-stage process of 1) Q function learning (first-term of Equation 1) and then 2) reward learning by compatibility (third-term of Equation 1). Our proposed T-max action instead directly encourages high reward for demonstrated state-action pairs, allowing the reward learning to be faster without reliance on a successful Q function learning. Com- bining these two enhancements and distillations on reward and policy, we summarize the loss func- tion for DROID in Equation We also provide more details regarding the two improvements and pseudocode for DROID in the supplementary.
-= Σ log EneA exp(3Qo(s,b)) (a
exp ())
DE (H(s a))) + Lenax-action (d; D); RED,UCA
Σ
Σ
AlogssQo((s "Qo(s', + Lad (an(( Led (m;
(s.a,e' D
5

---

6 Results
In this section, we show that DROID achieves strong performance in two OpenAl Gym environ- ments (CartPole and Junarl.andery [61] J and the more difficult Mars Path Planning problem com- pared to prior works (Section 6.1-6.2) and DROID's own ablations (Section 6.3). We focus our analysis on three questions that address the 3 challenges Offline LfD: heterogeneity, policy gen- cralizabilityy to unseen task settings, and reward transferrability to downstream tasks.
Q1: Diverse Demonstration Modeling How well does DROID perform at modeling different preferences from heterogeneous demonstrations in Offline LID?
Q2: Policy Generalizability How well can the learned policies perform in an unseen holdout test dataset (e.g., modeling unseen demonstrations in CartPole, planning unseen terrains in MPP)?
Q3: Reward Generalizability - How successful are the learned rewards in encoding experts' latent objectives and inducing nigh-performing downstream policies on unseen test settings?
For the benchmark experiments, we compare DROID against collection of OLID baselines: a) Behavior Cloning (BC) Batch: a single BC model across the dataset [44, 45], b) BC Single, which trains a BC model for each demonstrator, c) Diffusion model [49] which is generative modeling technique that leverages denoising to model multimodal demonstrations, d) Behavior Transformers (BeT) [50], which models next actions conditioned on the sequence observations, e) AVRIL Batch, a single AVRIL model on all data, f) AVRIL Batch XL, which increases model capacity fourfold to better model the heterogeneous dataset, g) AVRIL State Representation Learning (SRL) [51] which implicitly models multimodal data by inducing representation space trained by predicting the corresponding action, h) AVRIL Single, training separate models for each expert and i) MSRD- Offline: MSRD with reward distillation adapted from DROID.
6.1 Cartpole and LunarLander
We evaluate DROID against baselines on four metrics: Frechet Distance [62], KI. Divergence [63]. Undirected Hausdorff Distance [64]. and Average Log Likelihood. We train each method on a dataset of 60 trajectories from 20 distinct strategies generated by jointly optimizing an environment reward and diversity reward from DIAYN [65]. Diverse CartPole strategies include swinging to different ends of the track and oscillating at varying periodicity while diverse Lunar Lander (LL.) strategies include different landing attack angles and touchdown techniques. More experiment de- tails and videos of demonstrations and learned policies are provided in the supplementary.
Q1: Diverse Demonstration Modeling. Table summarizes the results of modeling and imitation on the training demonstrations. We find DROID performs significantly (p < .05) better on the Undi- rected Hausdorff (19% in Cartpole, 20% in LL) and Frechet Distance (32% in Cartpole) compared to the best baselines, showing that DROID models heterogeneous behaviors better and minimizes deviation between the learned policy's s behavior and the expert demonstrations.
Q2: Policy Generalizability. We study how well each method's policy performs on unseen demon- strations. Results in Table section "Policy Generalizabilitt"" show DROID significantly outper- forms (p < .05) baselines along Frechet and Undirected Hausdorif. Especially, DROID's per- formance gain over MSRD-OFIE Offflin being larger generalization than imitation shows the policy distillation in DROID is essential to learn generalizable policy.
Q3: Reward Generalizability, As a further analysis of generalization performance, we study how successful the learned reward functions are at inducing high-performing policies. We train offline RL policies with CQL [66] and compare performances on the holdout test set. The results in Table I section "Reward Generalizability" show DROID achieves significantly better (p .05) perfor- mance log- likeliioddood (underperforms Frechet Distance and Undirected Hausdorff), showing DROID's 's reward function can induce a similarly well-performing policy.
6.2 Mars Path Planning
With the success of DROID on Cartpole, we further test it against benchmarks on the more challeng- ing MPP problem, where RPs optimize for a complex objective considering goal locations, strate- gies, and safety constraints. Since there is no clear ground truth reward in the MPP problem, study the performance along four metrics: Distance from (expert) Waypoint, Final Distance (from

---

Table 1: This table shows performance comparisons and significance of DROID and baselines in Cartpole (left) and Lunar Lander (right). Bold denotes the best-performin model for the metric. denotes significance of p .05 against the second-best model.
CartPole
I Lunar Lander
Benchmark
Frechet Undirected
Log
Bonchmark
Frechet Undirected
Log
GT
Method
Divergence Distance Hausdorff Likelihood
Method
Distance Hausdorff Likelibood Reward
Diverse Demonstration
Modellngg (n1
40)
Diverse Demonstration Modeling
40)
BC Batch
10.046
1.192
0.969
22.599
BC Batch
5.496
5,400
-122,774
-24.613
BC Single
9,440
1.176
0.923
-24.729
BC Single
3.847
3,787
-124.510
-19.736
Diffasion
13.687
2.922
2.867
1165.517
Diffusion
4.213
4.174
299.733
-5.409
Bel'
13.755
2.901
2,836
-138.622 1,38.629
BeT
3,980
3.926
-277.258
-54.614
AVRIL, Batch
7.608
0,933
0.729
118.113
AVRIL Batch
6.209
5,436
-275.345
-72.539
AVRIL, Batch XL
8.840
1,023
0.775
-45.991
AVRIL Batch XL
1.380
1.342
-154,998
-7.132
AVRIL SRI.
11.335
1.395
1,069
.28.713
AVRIL SRI.
2,012
1.919
-578800
-13.275
AVRIL Single
10.051
1.294
0.895
-48.910
AVRIL Single
1.488
1.459
-151,956
-1.329
MSRD-Offlie
7.479
0.621
0.476
40.453
MSRD Offline
3.643
3,353
244.261
-3.652
DROID (ours)
6.047
0.425'
0.261
-37,948
DROID
1.153
1.063'
-54.941
-6.637
Policy Generalizability (n = 20)
Policy Generalizability (r1
20)
BC Batch
10.792
1.237
1,026
-33.079
BC Batch
5,186
5.020
-198 198.297
-218872
BC Single
9,330
1.111
0.881
-32.018
BC Single
3.434
3.356
-197.667
-17,448
Diffusion
13.825
3.050
2,911
-164.980
Diffusion
4,165
4.146
-299.734
-47,071
Bell
2.959
2,853
BeT
3.717
2277.258
447.544
AVRIL Batch
8,367
1.004
0,786
-52.843
AVRIL Batch
5.621
5.131
-275,345
-64.985
AVRIL
7,878
0.950
0.738
AVRIL Batch
1.374
-185.511
4.492
AVRIL SRL
11.320
1.449
1,056
-34.710
AVRIL SRL
2,080
1.979
-873.198
-12,429
AVRIL Single
7,960
AVRIL Single
1.544
-182.000
-5.313
MSRD-Ortinee
7.582
0.584
0.458
-44.173
MSRD-Offlne
3.476
3.244
-251.386
-28.495
DROID (ours)
0.412
-38.057
(ours)
1.158
1.061
-139.133
-2.020
Reward Generalizabilit (n = 20)
Reward Generalizability (v)
20)
AVRIL
8,923
1.197
1.152
-180.305
AVRIL, Batch
4.042
3.835
.326.569
-28.436
AVRIL Batch XL
8,809
1.099
0.994
-180.770
AVRIL Batch XL
4.079
4.049
-378.684
-66.989
AVRII. SRI.
8.732
1.124
1.030
AVRII, SRI.
3.474
3.332
-327.652
-27 760
AVRIL Single
9.017
1,418
1,280
1178.544
AVRIL Single
4,305
3.159
1328.189
-26.141
MSRD-Offine
8.368
1.403
1,274
-179
MSRD-Offline
4,870
4.662
-327.508
-33.656
DROID (ours)
8.048
1.441
1,336
-175.528*
DROID (ours)
3,438
3.326
-328.853
-25.271
desired goal point), Undirected Hausdorff Distance (how closely the generated path and expert path align), and Average Log Likelihood. We train each technique on a dataset of 1 117 distinct training sols (each with three waypoints: start point, midpoint, and goal point) from 37 RPs and hold out one sol per RP (37 sols) as a test dataset to evaluate generalization performance. Note data is extremely limited each RP is associated with only 2-5 Sols, and we treat each RP a unique expert strategy. We provide more experiment details in the supplementary.
Q1: Diverse Demonstration Modeling. We show in Table 2 that DROID is more successful the imitation objective with respect to baseline approaches. DROID outperforms (p < .05) baselines by 95% on reaching the goal point (Final Distance) along with 20% better modeling on the strategic preference (Log Likelihood). Policy distillation ensures DROID accomplishes the task goal while allowing it to model expert waypoint preferences well.
Q2: Policy Generalizability. On a holdout set of unseen Sols, Table 2 shows that DROID achieves better (p < .05) performance on the Undirected Hausdorff (21%) along with Final distance (90%) compared to best baselines. Despite limited and heterogeneous data, DROID's s learned policy gener- alizes well to achieve high performance and model expert preferences closely compared to baselines.
Q3: Reward Generalizability. Similar the OpenAl Gym experiments, evaluate how success- ful the learned reward functions are at inducing high-performing policies by training CQL policies. The results in Table 2 section "Reward Generalizability" show DROID's learned reward successfully induces policies with significantly better (p < .05) performance along the Undirected Hausdorff metric (12%) and the Final Distance metric (36%).
6.2.1 Qualitative Analysis
We visualize the task reward and strategy reward learned by DROID for a randomly selected Sol in supplementary Figure to understand RPs' preferences. We observe that the task reward encodes the common goal of converging at the goal point by giving high rewards to the goal area. In contrast, the strategy reward correctly identifies the midpoint preference. The illustration shows DROID can successfully decompose the shared goal along with modeling latent strategies unseen domains.
7

---

Table 2: This table shows performance comparisons of DROID and baselines in MPP. Bold denotes the best-performing model. denotes < .05 significance against the second-best method.
Benchmark
Undirected Hausdorff
Distance from
Final Distance
Log Likelihood
Methods
Waypoint
Diverse Demonstration Modeling (n
114)
BC Batch
8.924
9.40R
5.020
-4,192
BC Single
14,824
13.200
-192.530
Diftusion
11,397
8.421
9.591
-33894
AVRIL Batch
11.825
10.517
1.428
-9.129
AVRIL Batch XL
10.066
1.643
-6.306
AVRIL SRL
9.096
8.401
6,313
-264225
AVRIL Single
MSRD-Offing
8.162
7,580
4,476
-27.667
DROID (ours)
6.780"
4.592
0.070
-7.261*
Policy Generalizability (n
49)
BC Batch
7.791
9,562
3.197
-39.923
BC Single
186.310
Diffusion
10.886
8,775
3,506
43,175
AVRIL Batch
8.387
10.020
3,878
-22.623
AVRIL Batch XL
11,032
8.589
4.205
-6.561
AVRIL SRL
8.885
6.082
3,935
23.756
AVRIL Single
8.336
8.888
5,951
-17,357
MSRD-Offline
9.732
8,886
7.462
-148.964 964
DROID (ouns)
Reward Generalizabiliyy (n
49)
AVRIL Batch
9.385
9.502
1,823
1155501
AVRIL Batch XI.
9.396
9.502
1,925
-15.653
AVRIL SRL
10,34]
9,753
0.928
-16.475
Single
MSRD Offine
8.799
8.867
1,268
-14.515
DROID (ours)
6.3 Ablation
In this section, we perform ablation studies to evaluate the utility of different components of DROID. Ablation 1-6 corresponds to the following: 1) DROID without AVRIL improvements and 2; 2) DROID without AVRIL Improvement 1; 3) DROID without AVRIL Improvement 2; 4) DROID without distillation; 5) DROID without policy distallation; 6) DROID without reward distillation.
Supplementary Table shows the results of our ablation study in Cartpole and MPP domains. DROID outperforms Ablation on all metrics, showing the importance of distillation across dif- ferent strategies on the Diverse Demonstration Modeling task. Ablation 5 overfits to the demonstra- tions and achieves poorer generalization performance compared to DROID, which demonstrates the benefit of policy distillation to improve policy generalizability. Ablation 6 achieves worse Log Like- lihood, suggesting that reward distillation's is ability to share knowledge among demonstrations helps DROID understand expert behaviors. Ablation 1-3 perform worse than DROID in all tasks except Log likelihood in MPP, demonstrating that two AVRIL entancemetns effective in improving DROID's ability to learn demonstrator's 's latent intention. Overall, ablation study highlights the importance of both reward and policy distillation, well as the two enhancements, in achieving state-of-thed -htte-rt performance. More ablation study details are included in the supplementary.
7 Conclusion, Limitations, and Future Work
In this paper, we introduce an OLiD technique, DROID, that expands the applicability of OLiD with heterogeneous and limited data by novel decomposition of the policy and reward models. Our results on both simulated and real-world data demonstrate that DROID outperforms SOTA methods, particularly in capturing difficulitoo-articllate knowledge from rover path planners NASA.
There are several limitations with DROID. Firstly, DROID assumes experts have stationary pref- erences across demonstrations. Secondly, DROID assumes access to the number of strategies and the strategy label for each demonstration, which may be non-trivial to obtain. Thirdly, in the MPP domain, we extract nine features, which may not encompass all features an RP considers. future work, we plan to explore modeling nonstationary strategies for demonstrators. with RP's workflow, explore automatic feature extraction for Mars terrain, and relax the assumption about known strategy labels with behavior clustering [30] or online inference [29].
DROID's utility

---

Acknowledgments
We wish to thank our reviewers for their valuable feedback in revising manuscript. This work was supported by the National Institutes of Health (NIH) under Grant IR01HL157457, by NASA Early Career Fellowship under Grant 80NSSC20K0069, and by the National Science Foundation (NSF) under Grant #2219755.
References
[1] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and S. Levine. Soft actor-critic algorithms and applications. CoRR, abs/1812 182.05905, 2018.
[2] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1.07,0077770034777 2017.
[3] R. Paleja, Y. Niu, A. Silva, C. Ritchie, S. Choi, and M. Gombolay. Learning interpretable, high-performing policies for continuous control problems. arXiv preprint arXiv:220..02352, 2022.
[4] E. Seraj, Z. Wang, R. Paleja, D. Martin, M. Sklar, A. Patel, and M. Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1173- 1182, 2022.
[5] A. Silva, N. Moorman, W. Silva, Z. Zaidi, N. Gopalan, and M. Gombolay. Lancon-learn: Learning with language to enable generalization in multi-task manipulation. IEEE Robotics and Automation Letters, 12)):1635-164) 2022. doi:10.1 110.1111099AA2222113399666
[6] E. Seraj, L. Chen, and M. C. Gombolay. A hierarchical coordination framework for joint pereeption-action tasks in composite robot teams. IEEE Transactions Robetics,38(1)1339 158, 2021.
[7] S. Konan, E. Seraj, and M. Gombolay. Iterated reasoning with mutual information in coopera- tive and byzantine decentralized teaming. arXiv preprint arXiv:2201.0840 2022.
[8] S. G. Konan, E. Seraj, and M. Gombolay. Contrastive decision transformers. In 6th Annual Conference Robot Learning, 2022.
[9] D. M. Gaines, R. C. Anderson, G. B. Doran, W. Huffman, H. Justice, R. M. Mackey, G. R. Rabideau, A. R. Vasavada, Verma, T. A. Estlin, L. M. Fesq. M. D. Ingham, M. W. Maimone, and A. D. Nesnas. Productivity challenges for mars rover operations, 2016.
[10] A. R. Vasavada. Mission Overview and Scientific Contributions from the Mars Science Lab- oratory Curiosity Rover After Eight Years of Surface Operations. Space Sci Rev, 218(3):14, 2022.
[11] D. I. Koutras, A. C. Kapoutsis, A. A. Amanatiadis, and B. Kosmatopoulos. Marsexplorer: Exploration of unknown terrains via deep reinforcement learning and procedurally generated environments. Electronics, 10(22), 2021. ISSN 2079-9292. doi: 10.33990eleetrroniss102222 1022222... URL setttp:://ww.i wwt.:/99...o///00////// (00e//99992220000 22/2/11.
[12] R. Hu and Y. Zhang. Fast path planning for long-range planetary roving based on a hicrarchical framework and deep reinforcement learning. Aerospace, 9(2), 2022. ISSN 2226-4310. doi: 10.33900/arropace990000011. URL https:: //wp::///www... http:://wwwwp....... cttppi.comm22222/ httm:2226-443300//////...
[13] J. Zhang, Y. Xia, and G. Shen. A novel deep neural network architecture for mars visual navigation. CoRR, abs/ 808.008000 000808895, 2018. URL http: htap:://axxiv.ee/e http://xxxv.oogabba////a/o 08395.

---

[14] T.-H. Cheng, C.-P. Wei, and V. S. Tseng. Feature selection for medical data mining: Compar- isons of expert judgment and automatic approaches. 19th IEEE symposium on computer- based medical systems (CBMS)06), pages 163-170. IEEE, 2006.
[15] A. J. Chan and M. van der Schaar. Scalable bayesian inverse reinforcement learning, 2021.
[16] Jarboui and V. Perchet. Offline inverse reinforcement learning. CoRR, abs/2106.05068,
2021. URL https: htap::///rxiiie http//rxivv.rrr// http:/..org/aa///////// 05068.
[17] S. A. Murphy, M. J. van der Laan, and J. M. Robins. Marginal Mean Models for Dynamic Regimes. J Am Stat Assoc, 96(456):1400-1423 Dec 2001.
[18] A. Kumar, A. Singh, S. Tian, C. Finn, and S. Levine. A workflow for offline model-free robotic reinforcement learning. CoRR, abs/2109.10813, 2021. URL https://arxiv.. org/ abs/2109.10811
[19] X. Fang, Q. Zhang, Y. Gao, and D. Zhao. Offline reinforcement learning for autonomous driving with real world driving data. 2022 IEEE 25th International Conference on Intelli- gent Transportation Systems (ITSC), pages 3417-3422, 2022. doi:10. 109//TSCC55140.2222. 9922100.
[20] R. F. Prudencio. M. R. O. A. Maximo, and E. L. Colombini. A survey on offline reinforcement learning:
Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, pages 1-0, 2023, doi: 10. 10::10.1119/// (0.1109/mmls2202333522222222 org/10. 11099/Fttnlss 2023. 3250269.
URL btttpa://do:
[21] M. Fatemi, M. Wu, J. Petch, W. Nelson, S. J. Connolly, A. Benz, A. Carnicelli, and M. Ghas- semi. Semi-markov offline reinforcement learning for healthcare, 2022.
[22] S. Schaal. Learning from demonstration. In M. C. Mozer, M. Jordan, and Petsche, editors, Advances in Neural Information Processing Systems, volume 9. wnw.:eeeeddiinnnn.urrsr.........
T. MIT Press, 1997. URL https://proceednngs.. neurips http://w....ooii..cccpp///////// /tc/ppper/196//filee/ .pff..
[23] Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[24] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne. Imitation learning: A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
[25] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-acto critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint 1809.00880 2018
[26] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. CoRR, abs/2108.03298 2021. URL https: //t::i/////iv/v/ee www.wwwwwwwwa...../o 03298.
[27] G. Tucker. Tackling open challenges offline reinforcement learning, Aug 2020. URL https: //ai. hai.ggoglebbog. com/2020/ httpe/////taakkkaaee wtn.taacclnnnnnn.eeeeeeeeealnnne open- wwteeecchaaaaaaaaeeee inntssi-nnooffifoo html.
[28] E. F. Morales and C. Sammut. Learning to fly by combining reinforcement learning with behavioural cloning. In Proceedings of the International Conference on Machine Learning
(ICML), page 76, 2004.
[29] L. Chen, S. Jayanthi, R. Paleja, D. Martin, V. Zakharov, and M. Gombolay. Fast lifelong adaptive inverse reinforcement learning from demonstrations, 2023.
10

---

[30] S. Nikolaids,, R. Ramakrishnan, K. Gu, and J. Shah. Efficient model learning from joint-action demonstrations for human-robot collaborative tasks. In 2015 10th ACM/IEEE International Conference on Human-Robor Interaction (HRI), pages 189-196. IEEE, 2015.
[31] L. Chen, R. R. Paleja, M. Ghuy, and M. C. Gombolay. Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation. CoRR, abs/2001.00503, 2020.
URL http:://axxi..oo// wttgg//g///////oo/o////////////e 00503.
[32] S. Amershi, M. Cakmak, W. B. Knox, and T. Kulesza. Power to the people: The role of humans in interactive machine learning. Al Magazine, 35(4):105-120, Dec. 2014. doi: 10o:100.6699/amaa.3355555 v35.4.2513. URL. https://ojs. aaai org/index. http///ndx..pp/aaiaaaaaee shhpaammgazzine/ article/view/2513
[33] R. Paleja, A. Silva, L. Chen, and M. Gombolay. Interpretable and personalized apprentice- ship scheduling: Learning interpretable scheduling policies from heterogeneous user demon- strations. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 6417-6428. Curran Associates, Inc., 2020. URL https://proceedii: .eworieee.pp.. cc/papper/200///ile/ pdf.
[34] X. Chen, A. Ghadirzadeh, T. Yu, J. Wang, A. Y. Gao, W. Li, L. Bin, C. Finn, and C. Zhang. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learn- ing. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Ad- vances in Neural Information Processing Systems, volume 35, pages 36902-36913. Curran Associates, Inc., 2022. URL https://ppooceedings meurips wttp:ig.....rppppppppper.r.. filesr/ppeeeseee////eeepppp 202211100c10022300tt77866111a212 Papeee Conference. Caceroo--pepr---oofeeeeneeeeee
[35] A. Szot, A. Zhang, D. Batra, Z. Kira, and F. Meier. BC-IRL: Learning generalizable reward functions from demonstrations. In The Eleventh International Conference Learning Repre- sentations, 2023. URL
[36] S. Yue, G. Wang, W. Shao, Z. Zhang, S. Lin, J. Ren, and J. Zhang. Clare: Conservative model-basee reward learning for offline inverse reinforcement learning, 2023.
[37] J. Maghakian, P. Mineiro, K. Panaganti, M. Rucker, A. Saran, and C. Tan. Personalized reward learning with interaction-groundde learning (igl), 2023.
[38] L. Chen, R. R. Paleja, M. Ghuy, and M. C. Gombolay. Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation. In Proceedings of the International Conference on Human-Robot Interaction (HRI), 2020.
[39] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3, 2020.
[40] A. Correia and L. A. Alexandre. A survey of demonstration learning, 2023.
[41] Y. Li, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. Advances in Neural Information Processing Systems, 30, 2017.
[42] R. R. Paleja, A. Silva, L. Chen, and M. Gombolay. Interpretable and personalized ap- prenticeship scheduling: Learning interpretable scheduling policies from heterogeneous user demonstrations. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 2020.
[43] L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self- supervised reward regression. In Proceedings of Conference Robot Learning (CoRL), 2020.

---

[44] S. Ross, G. J. Gordon, and J. A. Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/1011 .6686, 2010. URL. http://arxiv, org/abst 1011. 0686.
[45] M. Bain and C. Sammut. A framework for behavioural cloning. Machine Intelligence 15, Intelligent Agents, 15, 03 2000.
[46] I. Kostrikov, O. Nachum, and J. Tompson. Imitation learning via off-polliy distribution match- ing. CoRR, abs/1912.05002,, 2019. URL http::t////tttte wwwp:.rggggaao..a.. htrg/aba/1te 2ab2. 05032..
[47] I. Kostrikov, K. K. Agrawal, Levine, and J. Tompson. Addressing sample inefficiency and reward bias in inverse reinforcement learning. CoRR, abs/1809.02925, 2018. URL http:
// //arxiv.co org/ aaaaaa/s/1000 02925.
[48] D. Jarrett, I. Bica, and M. van der Schaar. Strictly batch imitation learning by energy-based distribution matching, 2021.
[49] T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, and S. Devlin. Imitating human behaviour with diffusion models, 2023.
[50] N. M. M. Shafiullah, J. Cui, A. Altanzaya, and Pinto. Behavior transformers: Cloning modes with one stone, 2022.
[51] H. Zang, X. Li, J. Yu, C. Liu, R. Islam, R. T. D. Combes, and R. Laroche. Behavior prior representation learning for offline reinforcement learning, 2023.
[52] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation, 2016.
[53] Xing, T. Nagata, X. Zou, E. O. Neftci, and J. Krichmar. Policy distillation with selective input gradient regularization for efficient interpretabilityy. ArXiv, abs/2205.08688 088658 2022.
[54] S. Daftry, N. Abcouwer, T. Del Sesto, S. Venkatraman, J. Song, L. Igel, A. Byon, U. Rosolia, Y. Yue, and M. Ono. Minav: Learning to safely navigate on martian terrains. IEEE Robotics and Automation Letters, 7(2):5661 2022.
[55] E. Hilgemann. How to drive rover, Dec 2020. UNL.ttpp://nediium.coo//rre how-to-dr -
[56] G. Hedrick, N. Ohi, and Y. Gu. Terrain-aware path planning and map update for mars sample return mission. IEEE Robotics and Automation Letters, 5(4):5181-5188, 2020. 1oi:10.1109/ L.A.2020.300512.3
[57] M. Pflueger, A. Agha, and G. S. Sukhatme. Rover-irl: Inverse reinforcement learning with soft value iteration networks for planetary rover path planning. IEEE Robotics and Automation Letters, 4(2):1387-1394, 2019,
[58] A. Chan and M. van der Schaar. Scalable bayesian inverse reinforcement learning. arXiv preprint arXiv: 102.06002.6483, 2021.
[59] L. Van der Maaten and G. Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11). 2008
[60] L. Shi, G. Li, Y. Wei, Chen, and Y. Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity, 2022.
[61] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. arXiv preprint arXiv: arXiv:1606.01540 2016.
12

---

[62] K. Toohey and M. Duckham. Trajectory similarity measures. SIGSPATIAL Special, 7:43-50, 05 2013. doi: 10. (5/114////77777977777777
[63] Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 86, 1951.
[64]
Hausdorff.
Grundzüge der Mengenlehre.
Chelsea, 1914.
[65] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
URL https: / wwwpoo/wwww..ne////ooo rett/foruumidd-S5x63]]....
[66] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforce- ment learning. CoRR, abs/2006.04779, 2020. URL WRl.httss://arxii httrpp////arrv...og/aaa////.... 04779.
[67] B. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey, Maximum entropy inverse reinforcement learning. In Proceedings the National Conference on Artificial intelligence (AAAI), pages 1433-1438, 2008.
[68] M. Hessel, J. Modayil, H. van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. CoRR, abs/1710.00298,, 2017. URL http:///rxxiv.or/aatteee
02298.
[69] L. Kozachenko and N. N. Leonenko. Sample estimate of the entropy of a random vector. Probl. Inf. Transm., 23(1-2):95-100 1987.
[70] W. Brown. Nasa begin attempts free sand-trapped rover, Nov 2009. URL https: //www. nasa.
http://wwwe//22201010002 html.
[71] L. S. Shapley. A value for n-person games. In H. W. Kuhn and A. W. Tucker, editors, Con- tributions to the Theory of Games II, pages 307-317. Princeton University Press, Princeton, 1953.
[72] W. Gong, X. Xie, and Y.J. Liu. Human experience-inspire, path planning for robots. International Journal of Advanced Robotic Systems, 15:172988141875704, 02 2018. doi: 10.1177/117988141877744
[73] R. Ramón- Vigo. N. Pérez-Higueras, F. Caballero, and L. Merino. Transferring human naviga- tion behaviors into a robot local planner. In The 23rd IEEE International Sympositm on Robot and Human Interactive Communication, pages 774-779, 2014. 10/.0110///OMA..000
6926347.
[74] Dergachev, K. Muravyev, and K. Yakovlev. 2.5d mapping, pathfinding and path following for navigation of differential drive robot in uneven terrain, 2022.
[75] B. Hao, H. Du, J. Zhao, J. Zhang, and Q. Wang. A Path-Planning approach based potential and dynamic Q-Learning for mobile robots in unknown environment. Comput Intell Neurosci, 2022:2540546, June 2022.
[76] E. Murphy. Planning and exploring under uncertainty, Jan 2010. URL htt.hhtss///or.cco ox. uk/ uk/objeee htt//beectt/uuii8Bbb3355555
117b-4f5e--- 92ab=b6acc87aff77
[77] Yu, B. Deng. J. Gui, Zhu, and W. Yao. Efficient informative path planning via normalized utility in unknown environments exploration. Sensors (Basel, Switzerland), 22, 2022.
[78] L. Cuevas. M. Ramírez, I. Shames, and C. Manzie. Path planning under risk and uncertainty of the environment. 2021 American Control Conference (ACC), pages 4231-4236 2021.
13

---

[79] Y. Yin, Z. Chen, G. Liu, and J. Guo. A mapless local path planning approach using deep reinforcement learning framework. Sensors, 23:2036, 02 2023. dbo:100.339///2244200...
[80] K. Weerakoon, A. J. Sathyamoorthy, U. Patel, and D. Manocha. Terp: Reliable planning in uneven outdoor environments using deep reinforcement learning. 2021.
[81]
Meng, L.. Chen, Ma, Wang, and Q.-H. Meng. Nr-rrt: Neural risk-aware near-optimal path planning in uncertain nonconvex environments. arXiv preprint arXiv: 2205.06951 2022.
14

---

A Offline LID Enhancements Detail
AVRIL considers a distribution over the reward function and approximates the posterior, p (D),, with a variational distribution, (R).. It is trained by maximizing the Evidence Lower BOund (ELBO), shown in Equation 8, where p( R) is the prior distribution for the reward function and w the expert policy. The second equation follows by the assumption of Boltzmann rationality of the demonstrator [67].
ELBO(Ø) B [log ((D[R)] (g (1)((c()))
cxp (BQR (s, a))
(8)
-gg
Σ
log
DxL (xc((x(())l((R)))
ΣSEA exp (BQR (s.b))
(s.a)ED
Directly optimizing ELBO not feasible as the gradient of Q (s, a) with respect &i intractable. Therefore, AVRIL introduces second variational approximation for Q% (s, a) with Qo a) and ensures the variational reward distribution, (R), is consistent with the variational function, Qo(8,a), by Bellman equation, i.e., R(s, a) = E' (0h a) (s"(s''
Here, we present a lemma to show how AVRIL Enhancement 1 (i.e., extending KL-divergence regularization on all actions) impacts reward learning. This enhancement could be viewed as data augmentation technique encouraging small distance to the prior distribution for any action. We formalize the intuition in Lemma 1.
Lemma 1. Assume the prior reward distribution, p(R(s, a)), is a Gaussian distribution partitioned on each state and action pair, minimizing Lgg results in (a(R(s. a)) p(R((s for each operated (8,a).
Following Lemma and our extended operation over E A. we have the following observation.
Corollary 1.1. Assume we choose the prior reward distribution pl R(s, s, a)) to be Standard Gaussian distribution, N (0, for E D, E & s.t. (s,b) & D, optimizing LAVRIL leads to (s(ss b) = 0 and (s, b) 1. The proof follows immediately by observing gs (R(s,b) b)) only gets gradient from T.KL. and the optimal solution of Let is that pa pa a) and o% (8, a) 1.
B MPP Heterogeneity Analysis Details
In our analysis, we seek compare the variance of path features within each RP to the variance of path features across RPs, this would help quantify how diverse expert demonstrations demonstration for each RP is multivariate and not normal (we tested for normality and homoscedas- ticity), necessitating the use of the PERMANOVA test, which is non-parametric and can compare multivariate data. More specifically, it tests the null hypothesis that the centroid and dispersion for the two groups are equivalent. To apply this test to the RP data, we tested each possible pair of RPs to see which ones have statistically significant differences their distributions. The Bonferroni-Holne method was used account for the fact that many hypothesis tests are performed.
The
C Experimental Details
For fair comparisons with all baseline techniques, we share the same network architecture for each policy and reward with two hidden layers of 64 units, along with GELU activation functions. The training is with Adam Optimizer for 1000 iterations. For downstream policies, we train offline Con- servative Q Learning [66] for 1000 iterations. Conservative Q-learning is an offline RL algorithm that guards against overestimation while avoiding explicitly constructing a separate behavior model. We leverage several improvements, including Dueling Double Q Networks and Distributional RL from Rainbow [68] improve the CQL training. We list hyperparameters used in all algorithms in Table 3.
15

---

Hyperparameters
Values
Training Iterations
1000
Learning Rate
0.0001
State Only Reward
False
State Dim
4, 9
Action Dim
2
Gamma
0.99
Lambda
1.0
Train Test Split
0.8
Min Number of Test Sols
1
Linear Reward
False
Offline CQL Training Itrs
1000
Strategy Reward Regularization Coeffficient (MSRD and DROID)
0.01
Strategy Q Function Regularization Coeffficient (DROID)
0.001
Table 3: This table shows the hyperparameters we use for DROID and all benchmark algorithms. All values separated with commas are for CartPole (LL.) and MPP, respectively.
To showcase the statistical significance of our results on the Cartpole, LL, and MPP domains, we perform tests for normality and homoscedasticity and find that our metrics do not satisfy the as- sumptions of the parametric ANOVA test. Thus, we instead perform a non- non-parametric Friedman test followed by posthoe Semenyi-Damico-- Wolfe (Nemenyi) test. We show significance by aligning demonstration strategies between treatments (benchmark techniques).
C.1
CartPole / Lunar Lander
C.1.1
Video Demonstrations
We include demonstrations of heterogeneous behaviors along with each technique's learned policies http////ododddarrpoo.eeee.oo
in CartPole and Lunar Lander in the link:
C.1.2 Metries
Here, we describe the motivation behind each of the metrics, evaluated from rollouts of the policies with respect to expert demonstrations.
1. Frechet Distance [62]] Compare the spatial and temporal differences of the trajectory from the agent's policy with the expert trajectory to quantify how well the agent captures the motion pattern of the expert.
2. KL Divergence [63] (CartPole): By estimating the state distribution within trajectory by the kernel density estimator [69], KL divergence quantifies how well the learned policies state visitation matches the expert's.
3. Undirected Hausdorff Distance [64]: This measures the maxima between the two Directed Hausdorff distances: one mapping our learned policy's trajectory to the expert trajectory, and the other mapping the expert trajectory to our learn policy's trajectory. This metric studies how far the agent's trajectory is from the expert's trajectory.
4. Average Log-Likelihoodd This measures the likelihood of expert demonstration under the learned policy.
5. Ground Truth Rewards (Lunar Lander): The environment reward for Lunar Lander (all benchmarks in CartPole achieve near-maximal reward).
C.1.3 Analysis
We show Friedman Chi-square and Post-hoc Nemenyi statistical test metrics in CartPole for the three experiments (Diverse Demonstration Modeling, Policy Generalizability, and Reward General-
16

---

Table 4: This table shows the APA-style statistical test results for Friedman (a 0.05, d.o.f.=3) and Posthoe Nemenyi (a = 0.05) of DROID with respect to baselines Cartpole. All reported test statistics are significant other than the italicized metrics (if the Friedman test results are insignificant, no posthoc analysis performed).
CartPole
Benchmark
KL Divergence
Frechet Distance
Undirocted Hausdorff
Log Likelihood
Method
Diverse Demonstration Modeling
Friedman
174.82
222.92
240.52
339.68
DROID vs BC Batch
5,20
4,80
6.06
8.71
DROID vs BC Batch Large
5.13
4.73
6.13
9.2.
Diffusion
4.62
DROID vs BeT
9.42
10.71
11.52
1.66
DROID vS AVRIL Batch
2.84
3.36
4.21
2.66
DROID vs AVRIL Batch
3.32
DROID AVRIL SRL
6.87
5.83
6.06
7.87
DROID vs AVRIL Single
5.13
4.99
5.24
1.85
DROID vs MSRD-Offline
1.77
3.05
3.29
5.50
Policy Transferability (n 20)
Friedman
82.53
115.51
122.72
148.93
3,97
4.23
DROID vs BC Batch Large
2.77
3.86
4.28
5.69
DROID vs Diffusion
6,37
8.20
8.25
2.77
DROID vs BeT
6.74
7,99
DROID AVRIL Batch
2.30
3.39
3,86
1,36
DROID vs AVRIL Batch XL
2.25
3.45
3.34
2.56
DROID va AVRIL SRL
4.96
4.70
4.07
4.49
DROID AVRIL Single
2.45
2,66
0.78
DROID vs MSRD-Offine
1.62
2.72
2.94
3.71
Reward
Friedman
2.22
0.54
0,3
13.38
DROID vs BC Batch
N/A
N/A
N/A
N/A
vs
Large
N/A
DROID Diffusion
N/A
N/A
N/A
N/A
DROID vs BeT
N/A
N/A
N/A
N/A
DROID vs AVRIL Batch
N/A
DROID AVRIL Batch XL
N/A
N/A
N/A
2.87
DROID vs AVRIL SRL
N/A
N/A
N/A
3.01
DROID vs AVRIL Single
N/A
N/A
2,08
DROID MSRD-Offine
N/A
NIA
N/A
3.43
izability, e.f. main paper Results Section Q1-Q3) in Table 4. In the training task, DROID generates rollouts that align closer with expert behaviors, evidenced by stronger Undirected Hausdorff perfor- mance. Likewise, DROID does significantly better on Frechet and Undirected Hausdorff distance compared to the best baselines in both the Demonstration modeling and Policy Transferability tasks. Common eeward-poicyy distillation helps guide DROID's policies and rewards to better model ex- pert preferences and, thus, better capture diversity in expert behaviors.
Likewise, show the statistical test results for Lunar Lander in Table 5. DROID demonstrates clear superiority over several baseline methods. For instance, when compared to BC Batch and BC Batch Large, DROID achieves significantly better Frechet Distance values. This indicates that DROID's 's trajectory predictions align more closely with ground truth demonstrations than those of the standard BC baseline methods. Similarly, in the "Policy Transferability" benchmark, DROID counterparts. Compared to BC Batch, DROID attains substantially lower Frechet Distance and Undirected Hausdorff Distance than the best baselines in AVRIL SRL and AVRIL Batch XL. This suggests that DROID's 's explicit strategy decomposition and knowledge sharing improve overall performance.
consistently outperforms
17

---

Table 5: This table shows the APA-style statistical test results for Friedman (a 0.05, d.o.f.=3) and Posthoc Nemenyi (a = 0.05) of DROID with respect to baselines Lunar Lander. All reported test statistics significant other than the italicized metrics (if the Friedman test results are insignificant, no posthoc analysis is performed).
Lunar Lander
Benchmark
Frechet Distance
Undirected Hausdorff
Log Likelihood
Ground Truth
Method
Diverse Demonstration Modeling
40)
Friedman
171.61
170.31
322.40
207.27
DROID vs BC Batch
6.94
7.98
3.84
4.91
DROID vs BC Batch Large
6.24
6.13
4.10
3.91
DROID vs Diffusion
12.56
DROID BeT
8.9
9.08
11.08
8.75
DROID vs AVRIL Batch
3.40
4.14
5.28
0.37
DROID VS AVRIL Butch XL
DROID AVRIL SRL
4.73
5.39
0.15
3.32
DROID vs AVRIL Single
3.43
2.73
5.80
0.44
DROID vs MSRD-Offline
8.27
7.90
9.60
Policy Transferability (n 20)
Friedman
71.93
76.90
122.76
87.25
Batch
DROID vs BC Batch Large
3.86
3.92
1.62
3,08
DROID vs Diffusion
6,27
6.74
6.68
5.69
DROID vs BeT
5,74
5.48
5.43
DROID AVRIL Batch
2.14
2,04
0.89
0.78
DROID vs AVRIL Batch XL
2.67
1.98
1.36
0.11
DROID vs AVRIL
2.66
7.15
2.87
AVRIL Single
2.40
2.30
1.15
DROID vs MSRD-Offline
5.33
5.64
3.97
4.44
Reward
Friedman
32.02
29.15
DROID BC Batch
N/A
N/A
N/A
NIA
DROID va BC Batch Large
NIA
N/A
N/A
NIA
DROID ys Diffusion
N/A
N/A
DROID BcT
N/A
N/A
N/A
N/A
DROID vs AVRIL Batch
N/A
N/A
N/A
NIA
DROID VS AVRIL Butch XL
N/A
N/A
N/A
N/A
vs
DROID vs AVRIL Single
N/A
N/A
N/A
N/A
DROID vs MSRD-Offinee
N/A
N/A
N/A
N/A
C.2 Mars Path Planning
C.2.1 Domain Introduction
Exploring Mars has been a fascinating and challenging endeavor for space agencies worldwide. The Curiosity Rover is the longest active autonomous vehicle NASA has sent to Mars to study the climate, geology, and potential habitability of the planet [10]. The Rover has been operation for the past ten years, and its path planning has been done by manual labor of Rover Planners (RPs) on Earth.
There are several factors RPs consider when designing paths, including the change in elevation, distance to the desired destination, uncertainty about missing data on the terrain, etc. We study dataset of Curiosity Rovers curated paths from 154 sols (a sol being a Martian day, approximately 24.6 hours). We demonstrate in Section of the main paper that there significant heterogeneity between RP's paths. Each RP has a specific priority among safety, efficiency, risk, and mission constraints that inform their path design. This motivates to design autonomous path-plac planninnnnn approach that learns from these heterogeneous experts.
C.2.2 Dataset Curation
The data consists of features that were created through a series of interviews with RPs, scientists, and engineers that ideally capture the decision-makee process for rover path planning. The features engineered to codify the reasoning behind the RP's decisions. For example, RPs would visually
18

---

analyze the terrain and map waypoints to avoid "rough" terrains without quantifiable measures of what is considered rough. We identify the following features to encode the mental models and strategies of RPs with considerations of risks, efficiency, safety, and mission requirements.
Distance Feature- The distance feature measures the percent added distance the rover must take by adding intermediate waypoints in relation to the direct distance between the start and end waypoints. The aim of this feature is to drive the rover's necessary additional distance. It is assumed that the path between waypoints is driven straight as RPs rarely drive curved paths and rather set additional intermediate points if the rover needs to avoid hazards between waypoints.
Unknown Data Feature- When constructing the height maps, data could be missing where the cameras cannot terrain beyond hill obstructions like large rock or the rover itself. By traversing terrains with missing data, the RP places the rover at higher risk of damage. The design of the Unknown Data Feature is to minimize the distance the rover traverses over terrain without data. We compute the unknown data feature as the percent data missing in the height map for proposed trajectories.
Roughness Feature Rover Planners ideally drive on relatively smooth surfaces, avoiding rough terrain that could potentially damage the rover's hardware. Similarly, Rover Planners also look to avoid terrain that is too soft to prevent a similar fate as Spirit getting stuck in sand [70]. Here, the roughness is computed as the difference of consecutive surface angles as the rover traverses to the goal point. The maximum roughness and the average roughness over proposed trajectories are measured to avoid large holes or rocks and minimize traveling on rough terrains.
Pitch and Roll Feature Pitch and roll of the rover add another level of safety checks that ensure that the rover will not face terrains that risk the rover rolling over.
Turning Trajectory- We include the turning trajectory as a feature to track. This feature calculates the angle the rover must turn at intermediate waypoints. With this feature, the cost of taking sharp turns considers the rover's hardware and long-term health.
Waypoint Grid Construction The 64x64-sized waypoint grid is constructed by scaling the terrain height map along each axis and sampling the terrain map height at each (x.y) coordinate in the scaled grid. We do so according to an inverse weighted distance from each point along the four nearest points with height map data. We perform this scaling to limit the size of the action space of possible waypoints.
H v)
H represents the height evaluated at each point in the gaming area. (x, y) represent the correspond- ing coordinates, and h; d: represent the height and distance away from the evaluated point in the dataset, respectively.
C.2.3 Description of Policy
The action space exists on the 64 by 64 discrete grid of 4096 possible successor waypoints, which was chosen to be large enough to have high precision when selecting waypoints. The average dis- tance between grid points ranges from 0.01m to 0.7m for different Sols. We define our learned policy in Equation from our learned Q-function Qo.
(s) = max Qo(s,a) aEA
As mentioned in the main paper. we consider the three-waypoint planning problem and therefore, an action, a, (i.e., the intermediate waypoint) determines the trajectory as from the current point the intermediate waypoint and then from the intermediate waypoint the ending waypoint. We calculate the features of the action (i.e., next waypoint) for each of the two segments of the trajectory (current point to next waypoint, and next waypoint to goal point).
19

---

Table 6: This table shows the APA-style statistical test results for Friedman (a 0.05, d.o.f.=3) and Posthoc Nemenyi (a = 0.05) of DROID with respect to baselines in MPP. All reported test statistics significant other than the italicized metries (if the Friedman test results are insignificant, posthoc analysis is performed).
Mars Path Planning
Benchmark Methods
Undirected Hausdorff
Log Likelihood
Waypoint
Distance
Diverse Demonstration Modeling
114)
Friedman
112.98
44.91
408.32
786.47
DROID BC Batch
2.19
0.15
4.33
DROID BC Batch Large
2.29
0.67
14.45
4.23
DROID vs Diffusion
4.22
1.11
4.77
4.04
DROID AVRIL Butch
6,29
4,00
4.84
12.07
DROID AVRIL Batch XI.
4.09
0.07
5.07
3.17
DROID vs AVRIL SRL
5.18
0.83
6.43
7.57
DROID AVRIL Single
6.19
3.86
12.74
12.16
DROID MSRD-Offine
2.30
0.24
10.08
6.51
Policy Transferability (n 49)
Friedman
74.09
15.08
153.77
345,76
DROID
2.14
N/A
DROID BC Batch Lurge
2,89
N/A
3.41
1.42
DROID vs Diffusion
2.55
N/A
3.73
3.91
DROID AVRIL Batch
4.15
N/A
7.43
8.89
DROID AVRIL Batch XL
3.11
N/A
2,87
3.28
DROID vs AVRIL SRI.
3.73
N/A
2.99
6.27
DROID vs AVRIL Single
4.22
N/A
7.14
8.63
DROID Offiine
3.55
N/A
2.72
3.84
Reward (n
49)
Friedman
65.42
172.63
173.50
129.79
DROID BC Batch
N/A
N/A
N/A
N/A
DROID BC Batch Large
N/A
N/A
N/A
N/A
DROID Diffusion
N/A
N/A
N/A
N/A
DROID AVRIL Batch
2.73
6.16
6.51
0.21
7,96
DROID AVRIL SRL
5,97
7.69
7.94
288
DROID vs AVRIL Single
3.16
6.05
6.56
4.59
DROID MSRD-OOine
0.32
2.88
C.2.4 Metrics
Here, we include further description of the metrics we study in the MPP problem:
1. Average Distance from Midpoint:
The average distances from our policies' predicted way- points to the demonstrated waypoints.
2. Distance from Endpoint: The average distance from the final waypoint selected by the path generated by each technique to the goal point.
3. Undirected Hausdorff Distance [64]] This metric measures the maxima between the Di- rected Hausdorff distances mapping both our learned policy's set of waypoints to the expert waypoints and vice-versa.
4. Average Log Likelihood: This metric measures the likelihood of expert demonstration un- der the learned policy.
C.2.5 Analysis
We showcase the specific metrics that DROID outperforms baseline techniques on for MPP in Ta- ble 6. Rather than assuming homogeneity across demonstrations or discarding data to design a personalized policy for each RP, DROID takes advantage per-RP modeling and knowledge shar- ing to significantly outperform three out of four metrics in the Diverse Demonstration modeling benchmark. On the policy generalization benchmark, DROID can also model the latent objectives from diverse experts to design trajectory in unseen Sols that align closer to the expert's 's true path while successfully capturing the high-level common task goal. Lastly, DROID is the only technique
20

---

Distance to Goal
-0.03
Rehative Missing Features
Relative Missing Features
+0.03
Relutive Average
-0.11
Relative Average Roughness
+0.04
Relative Maximum Roughness
0.05
Relative Maximum Koughoess
Average Roll
-0.01
Average
Avenge
Average Pitch
+0.97
Max Roil
+0.14
Mar. Roll
+0.05
Max Pitch
+.0.01
Max Pitch
Turning
Tuming
-0.42
-0.50
-0.25
0.00
0.25
0.50
0.5
1.0
1.5
Figure 4: This figure shows Shapley values (the contribution of each of the composite features to the model's s reward estimate) for the learned rewards of RP I (left) and (right) evaluated on Sol 2030's demonstrated path.
3D Map Top Waypoints from Normafized 3D terrain Map of Weypoints from Waypaine 1 Sol 2575
Expert
DROID
BROID
BROID
20
3g
Aatk
Ang
20
Figure 5: These figures show DROID's policy outputs terrain map to plan for the next way- point from the Start point (left) and Waypoint I (right). The orange spheres represent the selected waypoints of DROID and the expert. Highlighted in green above the terrain map are the top 10 highest-rated successor waypoints. The orange labels correspond to DROID's found waypoints, and the red labels correspond to the expert demonstration's 's waypoints.
show significantly better performance on downstream reward transfer, indicating the learned ward is more useful encoding of expert's latent objective and can be used better interpret the salient features of given expert.
C.2.6 Additional Qualitative Analysis
In this section, we discuss the additional contributions of DROID the goal of interpreting expert decision-making and how is valuable the domain of path planning for the Mars Curiosity Rover and future missions.
Feature Contribution Analysis We perform Shapley value analysis [71] on each RP's learned reward function. The analysis measures how adding feature would change the prediction output and is helpful in comparing the relative importance different RPs place on features, even on Sols they have not explicitly planned on. As shown in Figure 4, we evaluated two randomly selected RPs (1 and 5) on Sol 2030. We observe that all drivers value that the path has a low pitch. However, RP prefers to have a smaller turning angle, while RP 5 values a lower pitch. This demonstrates how DROID can model heterogeneous strategies and quantify the influence of specific features on the modeled objective function. We can identify why certain RPs like or dislike given path and understand which features contribute to that assessment.
Learned Reward and Policy First, we analyze our learned Q-function and study how it can pro- vide insight into the decision-making process of our model. We showcase in Figure 5 how our technique can highlight the top 10 highest-rated successor waypoints from the start and midpoint
21

---

10.23
Espert
Export
DROND
SaV
10.00
1.0
9.75
9.00
7.75
8.50
8.50
0.6
DROID
Export
0.4
20
20
30
30
40
Export
BROID
40
DROID
50
Normalized Reward
00
60
40
X Axis
20
10
50
10
Output
Figure 6: This figure visualizes actual 3D Mars terrain map with DROID's mean estimate of the learned task (left) and strategy (right) reward for Sol 2163. X-axis and Y-axis correspond to the surface coordinates, Z-axis corresponds elevation, and heatmap coloring is the normalized reward output. The black line with arrows interlayed represents the path of the rover. The orange labels are DROID's found waypoints and the red labels are ground-truth demonstration waypoints. Point is the starting point, Point is the intermediate waypoint, and Point 2 is the final waypoint selected.
Table This table shows the ablation performance of DROID in Cartpole (Left) and MPP (Right). Bold indicates the best-performing model of the metric.
CartPole
MPP
Benehmark
Frechet Undireeted
Benchmark Undirected Distance Final
Log Distance Likelihood
Method
Divergence Distance Haudorff Likelihood
Methods
Hausdorff'
Waypoint
Diverse Demonstration Modeling (n
10)
Diverse Demonstration Modeling (n.
40)
Ablation
6,758
0.492
0.340
-63.926
Ablation 1
4,871
1.557
8.391
-10.157
Ablation
0.716
Ablation
0.288
Ablation
6.718
0.532
0.387
-63.728
Ablation 3
7.126
0.571
7.287
-8.431
Ablation
9,756
0.992
Ablation
6.720
7.441
Ablation
5.868
0.420
0.263
-42.53
Ablation 5
3.910
4.014
7.498
-225.212
Ablation
6,244
0.444
0.298
-92.006
5.556
9.389
DROID (ours)
6,047
0.425
0,261
-37.948
DROID
4.592
0.070
6.780
-7.261
Policy Generalizabilly
40)
Policy Generalizabilit
40)
Ablation
7.253
0.504
0.358
66.059
Ablation
8,086
1.842
8.945
-15.010
Ablation
8,889
0.743
0.654
-71.426
Ablation 2
8.071
1.615
8.331
-16.334
Ablation
6.455
0.509
0.316
65.631
Ablation 3
9.318
3.933
9.644
-16.503
Ablation
10.045
1.033
0.797
-84.233
Ablation 4
8.518
0.576
7.744
-11.391
Ablation
4.632
0.419
0.276
45.663
Ablation 5
6.162
7.295
9.537
-13.610
Ablation
6.626
0.487
0.284
-92.680
Ablation
8.026
9.078
9.246
30.037
DROID (ours)
5,271
0.412
0.207
+38.057
DROID
6.144
0.277
6.407
-18.483
positions, respectively. Providing multiple options aligned with the expert's latent preference could be beneficial for future assistive tool for RPs. Expert drivers NASA can also study DROID's recommended waypoint and similar waypoints that rated highly by our model. If expert dis- find several additional options that align
agrees with the best action identified by DROID, with that expert's 's latent preferences.
We examine the uncertainty in the reward predictions of our model by plotting the standard deviation of the strategy reward posterior. shown in Figure we can estimate how uncertain our model about different parts of the terrain due to the limited coverage of the dataset. Intuitively, of the state space where the demonstrations have not covered, such as the edges of the terrain, have higher estimate of uncertainty.
22

---

10.25
Expert
10.00
9.75
9.50
9.25
9.00
8.75
8.50
0.8
Export
DROID
30
40
Export
DROID
50
Normalized Reward Output
60
50
20
10
X Axis
Figure 7:
This figure shows a heatmap of Sol 2163 of DROID's Strategy Reward log standard deviation estimate, where higher value represents greater uncertainty in the reward estimation.
Proposed Application to NASA The explainability provided by DROID has significant potential application supplementary planning tool for interplanetary exploration. By leveraging Shapely value analysis of the importance of different factors, such as the change in elevation and uncertainty about missing data on the terrain, rover planners can gain deeper understanding of the objective function our algorithm extracts for modeling rover path planning. Therefore, DROID can explain what features contribute most to its perceived estimate of any human or Al-designed path.
DROID may have application to inform the design of future missions by providing into the limitations of the rover and the types of environments in which it is best suited to operate. As shown on the Curiosity Rover dataset, DROID could be applied to give rapid feedback about paths that best avoid sharp rocks (Rough Terrain), which may damage the open holes on the rover's s wheels [9], thus improving the longevity of the rover. Similarly, with additional data, such as orbital satellite imagery, our approach could used to evaluate the value of landing sites [57] by studying our learned RP objective function on constructed terrain maps.
insight
Furthermore, the ability to model different strategies taken by human drivers could potentially be used in the future by JPL in developing training programs. We hope future application of DROID would be to capture difficult-to-articulae tribal knowledge among rover planners and identify the most important features to trainees. can describe implicitly understood knowledge to help train new drivers NASA faster and with greater efficiency. By letting DROID explain which features contribute most the underlying latent RP strategy, human drivers can better understand what features to consider when navigating other extraterrestrial terrains.
Our hope is that DROID lessens the burden for operators to plan out daily schedules for rovers (since it performs automated path planning that better optimizes operator preferences). Moreover, the algorithm's s ability to reason under uncertainty makes it particularly useful for fast path planning inference, even when occluded information exists from cameras or other sensors. With the DROID's ability to learn diverse expert strategies and plan under uncertainty/occlusionn, algorithm could further advance autonomous rover exploration.
D Additional Related Works
In this section, deseribe additional related works regarding offline path planning under uncer- tainty and navigation beyond the MDP setting.
23

---

Algorithm 1: Dual Reward and policy Offline Inverse Distillation (DROID)
Input
TTraining iterations E,, number of strategies M |P], demonstration dataset for all strategies D = {D₃ {D,} learning rate a. Output Learned policy set II, rewards functions R
Initialize: Initialize all reward function and policy parameters θ {0Tasks {0s-i) i=
Trask [
for to do
Zero Gradients for Shared Reward and Policy Parameters: Task Trakk
for == to M do
Sample minibatch (s,a) a)
Calculate (ss) = (s) + (s).
Combine loss terms to calculate overall loss C according Equation 7
Calculate AVRIL loss LAVRIL (0, 6) according to Equation with the two
improvements introduced in Equation 5 and 6
Calculate reward regularization LRD( )
1.44
according to Equation 3 and
policy regularization terms Lpd( [ D) according to Equation 4.
Calculate the gradient of L with respect all parameters O
10
Update strategy-only reward parameters: Øs-j + Øs-j + a Dés,
ac
Update strategy-only policy parameters: Os.j 0s.j + 30g,
UCL.
Aggregate shared task reward gradients: AΦTask To + ac
Aggregate shared task policy gradients: AdTus < T + ac
13
80
Update shared reward and policy parameters with learning rate Ouakk f- Orakk + 1uSks 1aask & Taskk + aAArrakk
Offline Learning Offline learning has been used to teach robots to perform tasks such as assembly, manipulation, and grasping in manufacturing settings [26]. has also been used in healthcare to assist with tasks such as clinical diagnosis [26]. In medical diagnosis, offline LiD can used to understand how expert clinicians diagnose disease based on temporal indicators of key symptoms and medical reports. DROID's success in offline learning, particularly in complex domain settings, makes it useful for understanding expert recision-making process without risking the safety of humans expensive equipment.
Path Planning Algorithm. Several works use human-inspired admissible heuristic functions to plan paths [72, 73]. Yet, these functions are handerafted and require domain expertise to design. Model Predictive Path Integral (MPPI) is studied for local path following for rovers [74]. However, classical path planning approaches fail without high-fidelity simulator [75, 76]. Other works look the problem of path planning to maximize reward function [77, 78, 79] under uncertainty. How- ever, these techniques leverage exploration to obtain a better estimate of their cost function, which may not be feasible in offline learning. Our algorithm, DROID, learns heterogeneous preferences and policies directly from expert demonstrations without assuming hand-designed reward function or simulator.
Generalization Performance of Navigation Algorithms. Another important factor in offline path planning is the generalization performance of the planning algorithm to novel terrains. To improve the generalization performance, existing work attempts to decouple the training of feature extraction and navigation blocks using Deep RL [80]. However, they perform online planning through 2D navigation to extract an attention map, which is not feasible in the offline setting. Additionally, Meng et [81] proposes path planning algorithm that balances the trade-off between safety and efficiency under uncertainty. However, in contrast with DROID, these approaches do not generalize to unseen environments that contain new or additional obstacles.

---

Global Path Planning for the Martian Domain. Several prior works study path planning in the Martian domain but focus on local path planning (short-range egocentric navigation) and do not address global path planning: long-range end Hedrick et al. [56] proposes efficient Martian path planning and Rover-IRL [57] learns local cost function from demonstration, but both fail to learn an RP objective function that can transfer to new and unexplored terrain spaces, key challenge in the Mars domain [9]. Unlike previous path-planning approaches, DROID infers global RP objective function that can transfer to planning downstream unseen terrains.
E Pseudocode for DROID
We present the pseudocode for DROID in Algorithm 1. For each training iteration E (line 1), we iterate over all M demonstrations (line 3). For each strategy, we sample a minibatch from the strategy's 's dataset, Dj (line 4), We then calculate the overall loss L (line 6), with which we calculate the gradient with respect to all parameters (line 9). We update the strategy-only reward and policy parameters (lines 10-11) and aggregate shared task reward and policy gradients (lines 12-13). After iterating over all M strategies, we update the shared reward and policy parameters with the aggregated gradients (line 14).
25


Reviews:
[{"review_id": "jRVGodL3B6O", "reviewer": "Reviewer_CKuJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper provides a method for offline learning from demonstrations. In particular, the authors choose to look at situations that are heterogeneous (i.e with diverse preferences and strategies) as well as look for generalization (adapting to unseen test settings). Their method is analyzed in two domains, on of which is the novel Mars rover path-planning problem. They show that their method outperforms baselines to a significant extent.", "summary_of_recommendation": "Overall, I reccomend the acceptance of this work. I think the work had algorithmic novelty and tested on some grounded, realistic problem settings.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. For the reward distillation (equation 3), is the assumption that each trajectory is labeled with the type of strategy used $c_t$ and the number of total strategies $M$ used known apriori?\n2. There are two other points of comparison that can be considered to the approach as mentioned. Firstly, Imitation Learning approaches would not need a reward estimation and can learn from prior demonstrations directly. Methods such as BET or Diffusion based methods can deal well with multimodality. Additionally, offline RL methods could be learned in this setting with either goal-conditioned reward formulation or a heuristic reward (such as the last n timesteps of the demonstration data is successful). Could you compare how your method would fare against these methods?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- The problem is well-motivated an appropriate approach is provided to address the problem setting\n- The algorithm is evaluated on two domains, one of which is the Mars rover path-planning problem which is a great setting to ground the algorithm\n- The performance of the algorithm outperforms the existing baselines meaningfully and with statistical significance.\n\nWeaknesses:\n- The paper considers only two additional baselines in this setting. This I believe isn't extensive enough and would encourage the authors to consider adding in additional points of comparison.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "i3B0VJ2wkT", "reviewer": "Reviewer_CWnj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces a new offline learning from demonstrations algorithm, DROID, which includes an explicit modeling for different demonstrator strategy policy and rewards, a novel Gaussian paramerization is used to distill the rewards. Data heterogeneity and generalization are argued as the main motivation explicit model. Comparison to baselines are shown for modeling train and unseen demonstrations and also compared to expert behavior. A new Mars Curiosity Rover path planning dataset is introduced along with an analysis of how demonstrator strategies influence data heterogeneity. Results are also shown for a cartpole problem. An insightful qualitative analysis shows how DROID can model the different strategies in the Mars dataset.", "summary_of_recommendation": "Main rejection is how widely applicable this approach is, should it be applied to most LfD scenarios even when the presence of demonstrator strategies is not clear or explicit. Does this method scale with larger datasets and models where such preferences may be easily learned implicitly.\n\nGiven the additional experiments in the rebuttal phase,  I recommend a weak accept", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* The Mars dataset was explicitly curated to research data heterogeneity and is thus already suited to DROID. Is DROID more generally suitable where it should always be used in LfD problems? Can you show this for more general datasets beyond Cartpole?\n* It was unclear from the algorithm, do the number of strategies need to be set in the algorithm? If so that seems like a big drawback of the algorithm, how did you define this for the Cartpole problem?\n* Two improvements are presented in Section 5.3, are there any ablations to show that these actually improve the method?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* Algorithm is presented in a very realistic setting with Mars path planning\n* The motivation for the algorithm is clear, demonstrators have different styles\n* Qualitative analysis clearly shows that the algorithm addresses the initial motivation\n* Limitations are exhaustive enough\n\n\nWeakness:\n* Typically, adding explicit structure in learning algorithms can be addressed instead by adding modeling capacity which can implicitly model strategy/preferences for example. There is no mention of if increasing modeling capacity like number of model parameters also helps.\n* The preferences are in relatively low-dimensional space in the examples presented. It would bolster their case if more complex datasets/tasks were included to show that this method more generally works in any LfD use case. It is unclear if the work is proposing their algorithm should be used in most LfD scenarios", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "FkbOC6pN6T", "reviewer": "Reviewer_FG7m", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a new OLfD approach that addresses key problems in realistic settings where demonstrations are heterogeneous and limited. DROID modifies the OLfD method AVRIL by separating demonstrator/strategy-specific components and shared task components when inferring the reward function and learning an ensemble of policies that express these different strategies. The authors also introduce a new task/dataset comprised of paths by rover planners for a Martian Path Planning problem.", "summary_of_recommendation": "The paper proposes an impactful problem as well as a solution, but the clarity and some experimental analysis should be improved before publication.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "* Can you benchmark/ablate on a couple other standard domains/tasks other than cartpole?\n* How is $M$ chosen? is it assumed to be known? Relatedly, do you assume the data for different strategies are labeled?\n* Section 5.1 says \"We propose a novel reward distillation approach\", but I was under the impression this was proposed in MSRD. Could you clarify exactly what MSRD did and how this component relates to it?\n* Will the dataset/simulator be made publicly available?\n* Can you show that the metrics chosen are correlated with some quantitative or qualitative measure that indicates task success (like Final Distance in the MPP)?\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I appreciate the paper tackles a realistic problem and proposes a reasonable solution. The paper is well-motivated; however, there are some concerns I have with clarity as well as the experimental results.\n\nRegarding clarity, I found there to be many instances in which I did not adequately understand what was being communicated. Generally, the paper would be much improved by taking care to explain things that are being introduced and with proper justification. I've listed some examples below:\n* I strongly suggest summarizing the overall algorithm using pseudocode/algorithm box for clarity\n* Figure 1 is very confusing without more context, so I would suggest either making this simpler or adding explanation\n* Related work could use some more detail and additional explanation as to how these works relate to DROID specifically. For example, in addition to: \"both fail to plan under missing/occluded terrain maps\", an explanation of how the work addresses this problem would help.\n* Section 5.3 does not contain enough information to fully understand the improvements, nor do the experimental results refer to these improvements\n\nRegarding experiments, I find that the current results do not directly address all the claims made. First, I am not sure what each of the comparisons is *exactly* referring to---please consider making this a lot more explicit and explaining which of the claims each one is meant to test. Are all comparisons benefiting from the improvements? Otherwise, it's hard to understand what is providing the benefit---please make this more clear. Regarding the evaluation metrics, it's hard to judge whether the results are significant without justification for the metrics chosen. The most important issue, however, is that due to the lack of clarity (perhaps) I am not sure what to take away from the experiments and how they indeed validate the claims. For example, since the claim is that doing both is better in DROID, what is the effect of reward distillation only versus policy distillation only? It's currently unclear whether this is exactly being tested since the descriptions of the comparisons are a bit vague. Also, if the authors talk about improvements, it would be nice to include experiments that show their effects/analyze them.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "7n8GsQ90vK", "reviewer": "Reviewer_tYgj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new approach to handle heterogeneous demonstrations to learn generalizable policies in the context of offline imitation learning. The key contribution of the paper is achieving state of the art performance on a new and interesting dataset for Mars rover path planning and in proposing new and general reward and policy distillation methods for offline imitation learning.", "summary_of_recommendation": "Overall I enjoyed the paper and think the strong performance on the MARS path planning domain is quite interesting for the CoRL community. I am leaning towards acceptance, but would be further convinced by the ablations mentioned above and the addition of behavior cloning based baselines.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1) Will the new Mars rover path planning dataset be made public? This would be a valuable contribution to the research community.\n2) Would it be possible to add baselines which do not do direct reward inference but instead simply do behavior cloning (with both unimodal and multimodal policies)? These baselines often perform quite well in practice and would be helpful in establishing whether the complexities of DROID are truly necessary.\n3) Please add ablation studies as mentioned above so the reader can better understand the impact of the design choices made in DROID.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall I am positive on this paper and leaning towards acceptance. Experiments are conducted on a challenging and interesting real world dataset and while the reward and policy distillation approaches build on prior methods, they include interesting innovations such as using mean-field Gaussian distributions for reward distillation. I think the core strengths of this paper are in the challenging and novel experimental domain and the clear description of the methodological improvements over prior work in reward and policy distillation required to make DROID work in practice. One weakness of this work is the lack of appropriately controlled ablations in the experimental evaluation. For example, it would be nice to study DROID but with the exact same reward distillation approach as MSRD to understand the effect of the proposed changes in reward distillation.  Additionally, it would be helpful to study DROID but with the policy distillation process performed exactly as in AVRIL. Another very important comparison that should be included is to behavior cloning methods which do not directly infer reward functions, which are often very strong baselines for offline imitation learning. I appreciate the detailed experimental results otherwise though, and really enjoyed Figure 4.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new approach for Learning from Demonstration in the offline setting with heterogeneous and limited demonstrations. Most reviewers agreed that the approach is interesting and novel, and appreciated the new realistic dataset which the authors introduced for their evaluation. They were also convinced by the experimental evaluation, especially after the additional ablations and comparisons in the thorough rebuttal. Please make sure to include them in the final version of the paper.
Decision: Accept (Poster)