KD-BIRL: Kernel Density Bayesian Inverse Reinforcement Learning
Authors: Aishwarya Mandyam
Didong Li
Andrew Jones
Diana Cai
Barbara E Engelhardt
Venue: Accepted by TMLR ()

Abstract: Inverse reinforcement learning (IRL) methods infer an agent's reward function using demonstrations of expert behavior. A Bayesian IRL approach models a distribution over candidate reward functions, capturing a degree of uncertainty in the inferred reward function. This is critical in some applications, such as those involving clinical data. Typically, Bayesian IRL algorithms require large demonstration datasets to ensure posterior concentration, which may not be available in practice. In this work, we incorporate existing domain-specific data to achieve better posterior concentration rates. We study a common setting in clinical and biological applications where we have access to expert demonstrations and known reward functions for a set of training tasks. Our aim is to learn the reward function of a new test task given limited expert demonstrations. Existing Bayesian IRL methods impose restrictions on the form of input data, thus limiting the incorporation of training task data. To better leverage information from training tasks, we introduce kernel density Bayesian inverse reinforcement learning (KD-BIRL). Our approach employs a conditional kernel density estimator, which uses the known reward functions of the training tasks to improve the likelihood estimation across a range of reward functions and demonstration samples. 
Our empirical results highlight KD-BIRL's faster concentration rate in comparison to baselines, particularly in low test task expert demonstration data regimes. Additionally, we are the first to provide theoretical guarantees of posterior concentration for a Bayesian IRL algorithm. Taken together, this work introduces a principled and theoretically grounded framework that enables Bayesian IRL to be applied across a variety of domains, especially those with limited expert demonstration datasets.

Published in Transactions on Machine Learning Research (10/2024)
KD-BIRL: Kernel Density Bayesian Inverse Reinforcement Learning
Aishwarya Mandyam
am@8stanford.com edu
Deportment of Computer Science Stanford University
Didong Li
didongli@tunc.o edu
Department Biostatistics University of North Carolina
Diana Cai Flatiron Institute
daut@flaaironinskiuutt.oor
Andrew Jones
oj uj13@prinneton.cde
Department of Computer Science
Princeton University
Barbara E. Engelhardt
barbarue @artarue@stanford.cc edu
Gladstone Institutes
Department of Biomedical Data Science Stanford University
Reviewed on OpenReview: https: // http://epppvveee.eeem/ee net/ forum? id= B8OWUNhTAW
Abstract
Inverse reinforcement learning (IRL) methods infer agent's reward function using demon- strations of expert behavior. A Bayesian IRL. approach models a distribution over candidate reward functions, capturing a degree of uncertainty in the inferred reward function. This is critical some applications, such as those involving clinical data. Typically, Bayesian IRL algorithms require large demonstration datasets, which may not be available in practice. In this work, we incorporate existing domain-spocific data achieve better posterior con- centration rales. We study common setting in clinical and biological applications where have access to expert demonstrations and known reward functions for set of training tasks. Our aim is to learn the reward function of a new test task given limited expert demonstrations. Existing Bayesian IRL methods impose restrictions on the form of input data, thus limiting the incorporation of training task data. To better leverage information from training tasks, we introduce kernel density Bayesian inverse reinforcement learning (KD-BIRL). Our approach employs a conditional kernel density estimator, which uses the known reward functions of the training lasks to improve the likelihood estimation across range of reward functions and demonstration samples. Our empirical results highlight KD- BIRL's faster concentration rate in comparison to baselines, particularly in low test task expert demonstration data regimes. Additionally, we are the first to provide theoretical guarantees of posterior concentration for a Bayesian IRL algorithm. Taken together, this work introduces a principled and theoretically grounded framework that enables Bayesian IRL be applied across a variety of domains.

---

Published in Transactions on Machine Learning Research (10/2024)
1 Introduction
Inverse reinforcement. learning (TRL. is strategy within the domain of reinforcement learning (RL) that infers agent's reward function from demonstrations of the agent's behavior. The learned reward fune- tion helps identify the objectives driving the agent's 's behavior. IRL methods have been successfully applied across several domains, including robotics (Ratliff et al.
Kolter et al., 20080, swarm animal move- The carlicst IRL algorithms Icarned point estimates of the reward function [gg & Russell 2000 Abbeel & Ng 2004 and were applied to problems in
2006a;
ment (Ashwood ct al.) 2020), and ICU treatment (Yu ct al. 2019a
path planning (Mombaur et al. 2010 and urban navigation Ziebart et al. 2009
Despite the success of carly IRL approaches, there are limitations to inferring a point estimate of the reward function. First, the reward function is frequently non-identifiable (Abbeel & Ng. 2004 Ratliff et al. 2006b; Ziebart et al.) 2009), meaning there are multiple reward functions that explain the expert demonstrations equally well. Second, for finite demonstration data, the point estimate of the reward function fails to capture the noise in the getta-geneaaing process, and does not model uncertainty in the resulting reward function estimate. A Bayesian approach learns a posterior ristribution over reward functions after observing all samples. This distribution places mass on reward functions proportional to how well they explain the observed behavior, using the information from the samples (Ramachandran & Amir 2007 Michini & How
2012b; Choi & Kim 2012 Balakrishnan et al. 2020 Chan & van der Schaar 2021 Michini & How
2012a
better highlights possible alternative explanations of agent objectives.
Despile their ability to produce a set of candidate reward functions, Bayesian IRL algorithms lypically require large amounts of high-quality expert demonstration data to ensure posterior contraction (Brown ct al. 2020). A Baycsian posterior changes depending on the samples that are observed. "Contraction" is the phenomenon of the posterior concentrating at the true value as more samples are observed. However, in some applications, such as those involving clinical or biological data, expert. demonstrations are expensive to obtain. For example, in a clinical application, collecting expert demonstrations may involve parsing electronic health records (EHRs) (Johnson et al.] 2016). which contain information associated with patient trajectories as well as clinical notes (Johnson et. al.. 2023 However, each clinician treats a limited number of patients, so there are few patient trajectories. In a biological application, we might be interested in studying the factors that drive the hunting behavior of T-cells, which affects how fast they kill tumor cells. For example, certain conditions may induce "pack hunting" in which a T-cell works with other T- cells to kill tumor cells. One common approach to gather information about how cells move under different conditions is genetic perturbation studics (Feldman et al. 2022 which examine the cellular behavior changes following an induced genelic perturbation. Previous work has studied how some genetic perlurbations, such as RASA2 (Carnevale et al., 2022) and CUL5 knockout Liao et al. (2024). affect the efficiency of T cells killing tumor cells. However, cach study is exponsive to run, and thus, a limited number of trajectorics are available. In this work, we use supplementary data sources to improve the posterior contraction rate for Bayesian IRL algorithms in applications with few expert demonstration trajectories.
We assume that we have several training tasks and single test task. We have access to expert demonstrations well as the associated reward function for each training task. This assumption is reasonable in many application areas. Referring to the examples above, in clinical applications, the reward function represents the clinical treatment objective. These treatment objectives can be obtained from EHR clinical notes, which contain Sumician-trrascribeer goals of treatment. In the biophysical movement example, the reward function represents how various factors, such as the density of the tumor cells and the proliferation rate of T-cells, induce a certain type of hunting behavior. In both examples, each training task associated with a known reward function. Our goal is to infer the reward function for the test task, i.e., patient trajectories treated by new clinician or cell behavior trajectories from a new genetic perturbation, with small set of expert demonstrations.
Integrating supplementary data poses challenges within the current landscape of Bayesian IRL algorithms. Existing methods for single-task IRL (Ramachandran & Amir 2007 Michini & How 2012b) are restricted to a framework where the set of expert demonstrations arises from a single task. Alternative formulations, such as meta-IRL methods, can accommodate demonstrations from several distinct tasks. However, meta- IRL methods cannot incorporate information about. the known reward functions from the training tasks.
2

---

Published in Transactions on Machine Learning Research (10/2024)
In both the single-task and meta-IRL settings, one can learn an informative prior to facilitate the reward function inference. A "prior" is the parameterized Bayesian distribution before observing samples. Prior distributions can be uninformative (i.e., uniform), or informative (i.e., Gaussian). Nevertheless, earlier work has shown that using an informative prior may introduce bias [Gelman et al. 2020; or reduce sonsitivity to new demonsttrtions from the lest lask (Kass & Wasserman, 1996.. Ultimately, neither the single-task nor meta-IRL approaches suitable because the methods either cannot incorporate demonstrations from training tasks, or cannot adequately include information about the known reward functions from training tasks.
To incorporate all possible information sources stored within the training tasks, we introduce kernel density Bayesian inverse reinforcement learning (KD-BIRL). In our work, we use conditional kernel density estimation (CKDE) to approximate the likelihood function. The "likelihood" a probability distribution that defines how likely a sample to be observed, given parameter values for a specific probabilistic model. Our CKDE formulation uses kernels to quantify how much each demonstration sample contributes to the likelihood, conditioned on reward fuuccion. The of CKDE allows us to incorporate reward functions from the training tasks into the likelihood estimation stage.
Our work demonstrates that a posterior distribution learned using likelihood estimated by CKDE empir- ically contracts faster than single-task IRI. approaches. This holds true even if the single-task approaches have informative priors. The contributions of our work follow.
1. We propose KD-BIRL, which incorporates supplementary data by using conditional kernel density estimation to approxinate the likelihood (Section 4 To the best of our knowledge, we are the first to identify and formalize a setting in which multiple sets of demonstrations and their corresponding reward functions exist as training tasks, and the first to build a Bayesian IRL algorithm that can take advantage of it.
2. We perform asymptotic analysis, and demonstrate that the posterior distribution contracts to the equivalence class of the true reward function (Section B]. We are also the first to provide asymptotic guarantees of posterior contraction in a Bayesian IRI. setting.
3. With a feature-based reward function (Section 4.2 KD-BIRL infers a posterior distribution over reward functions in high-dimensional state spaces, empirically contracting using fewer samples than baseline approaches (Figure
We first review related work in Section 2 and fundamental topics in Section g Then, we formally introduce our problem and KD-BIRL in Section We provide posterior consistency guarantees in Section 5 In Section D we demonstrate results in a Gridworld environment and a challenging sepsis management task, benchmarking KD-BIRL against existing Bayesian IRL methods. Finally, in Section 7 we discuss broader implications.
2 Related Work
Single-task Bayesian IRL. Single-task Bayesian IRL methods receive as input a set of demonstration data that originates from an expert policy optimizing for an unknown reward function. There have been many
follow-up methods which improve upon the original Bayesian IRL. algorithm (Ramachandran & Amir 2007
To perform inference for more complex reward function structures, several approaches learn nonparametric
reward functions, which use strategics such Gaussian processes (Levine al. 2011 Qiao & Beling 2011
and Indian buffet process (IBP) priors (Choi & Kim 2013). Sošié et al. (2018) decompose complex
global IRL task into simpler sub-tasks. Other approaches reduce computational complexity by either using informative priors (Rothkopl& & Ballard 2013). different sampling procedures (e.g., Metropolis-Hastinng (Choi
& Kim 2012 expectation maximization [Zheng et al., 2014)), ), variational inference (Chan van der Schaar 2021 ) or learning several reward functions that each describe a subset of the state space (Michini How 2012b ab. In contrast, we assume access to demonstration data from experts optimizing for multiple tasks, and the known reward functions of these tasks to facilitate reward function inference for new task.
3

---

Published in Transactions on Machine Learning Research (10/2024)
Meta-IRL and multi-task IRL. Meta-IRL algorithms receive set of unstructured demonstrations that arise from several distinct but unknown tasks, each associated with different reward function. Previous work learns context-conditional reward function from training tasks, which can then be generalized to unseen but similar tasks (Yu et al,, 2019b Xu et al.) 2019 However, these approaches learn a point estimate of the reward function instead of interring a posterior distribution. Additionally, these approaches do not incorporate information about the reward functions associated with each of the training tasks. Multi-
task IRL algorithms (Dimitrakakis & Rothkopl 2012: Gleave Habryka, 2018 Arora et al., 2021 Chen
et al. 2023 Choi & Kim 2012) are similar, but focus learning task-specific reward functions rather than
generalizing to new tasks.
IRL with supplementary information. There are a few methods that use both expert demonstration data and supplementary data sources to solve IRL problems. One method finds that using a combination of sub-optimal demonstrations and high-quality human feedback can improve reward function inference (Ezzod- dinc et al. 2018h. Another approach finds that expert demonstrations of varying quality aids in learning the reward function because the IRL model can learn from non-expert demonstrations AAudillren el. al., 2015 Brown et al. (2020) finds that Bayesian learning can be sped up using human preferences over demonstrations. Our work builds upon these efforts to improve Bayesian IRL using supplementary data sources. Specifically, we reformulate the likelihood function using CKDE to take advantage of a training dataset with both demonstrations and reward functions from multiple expert agents.
3 Preliminaries
3.1 Single-task Bayesian IRL
IRL methods infer agent's reward function given expert demonstrations of its behavior. We model the agent's 's behavior using Markov decision process (MDP). The MDP is defined by P. where S is the state space; A is a discrete set of actions; P(s ax) defines state transition probabilities from time to t + 1; and y E [0, 1] is a constant discount factor.
The IRL algorithm receives as input n expert demonstration samples, ((sf,a5))) and each sample is a 2-tuple that specifies the expert agent's state and chosen action. The demonstration samples arise from an agent following the stationary policy x² S + A, which optimizes for fixed but unknown ground truth reward function R S A + R, where R E R and R is the space of reward functions.
Given these expert demonstrations, Bayesian IRT. algorithms treat R as inherently random and aim to learn a posterior distribution over R that can rationalize the expert demonstrations. By specifying a prior distribution over the reward space R, p(R), and a likelihood function for the observed data, II P(sf. R), Bayesian IRL methods infer a posterior distribution over R using the expert demonstrations as evidence. By Bayes' rule, the posterior density of the reward function is given by:
p(l() IT a(5), p(((s), af))
p (R ((),)
(1)
The earliest formulation of Bayesian IRL used the optimal Q-value function to approximate the likelihood component, p(sf. af|RR R), of Equation Ramachandran & Amir 2007). The likelihood takes the form
p(s,a R) x (aQ"(s.a,1))
(2)
where a > 0 is an inverse lemperature parameter characterizing the optimality of the expert dernonstrations and (s, a, R) is the optimal Q-value function for reward function R, defined
Q* R) max E,
Σ R(S,, Ar) So = s, Ao = 1=0
3.2 Problem Setting
In this work, we assume that we have already encountered several training tasks with different reward functions. For each training task, we have access to both optimal demonstrations from the corresponding
4

---

Published in Transactions on Machine Learning Research (10/2024)
expert RL agent, and know the reward function the expert is optimizing for. Specifically, there samples in the training dataset {(*j,0) djj )=)] where each state-action tuple (5). is demonstration of an expert optimizing for the reward function Rj. We note that there will be several state-action pairs that correspond to the same reward function since there are several samples from each training task; thercfore, R2 is not unique. Our goal is learn the unknown reward function R of a new test task given a limited amount of expert demonstrations of the new test task, {(s), (;))) (n << m)..
3.3 Conditional kernel density estimation
In this work, we propose approximating the likelihood in Equation | using conditional density estimation (CDE). In Section 1.1 we show that CDE allows us incorporate information about both the reward functions and demonstrations associated with training tasks. Given a set of empirical observations, CDE aims to capture the relationship between the responsive variable Y and the covariates X by modeling the probability density function,
p(y22) p(x,8) v) p(x)
With this perspective, any appropriate conditional density estimator can be used to approxinate the like- lihood; examples include the conditional kernel density estimator (CKDE) van der Vaart, 2000p, Gaussian process models (Rihimāki & Vehtari 2012). or diff'usion models Cai & Lee
2023 In our work, we choose CKDE because it is nouparametric, which allows us to model complex reward structures. Additionally, CKDE has a closed form and is straightforward to implement (Holmes et al. 2007 Izbicki Lee 2016
Specifically, the CKDE estimates the conditional density p(ylz)) by approximating the joint distribution p(a,y) and marginal distribution p(æ) separately via kernel density estimation (KDE). Intuitively, KDE estimates the probability densily function of random variable by considering how close each observed data point is to a given location. The likelihood at that location is higher if there are more observations nearby, with closer observations contributing more to the estimate. Given pairs of vector observations ((z,, } the KDE approximations for the joint and marginal distributions are
dhe (1,
(d((w.b)) h'
Σ
K
K'
m
(3)
1
dg (z)
(x)
K
m
where K and K' are kernel functions with bandwidths h, h' > 0 respectively, and de, dy are distance functions that mcasure the similarities between x, Fy y, Uj. respectively. There are many choices for suitable kernels (e.g. uniform, normal, triangular) and distance metrics (e.g. Euclidean distance, cosine similarity, Manhattan distance) (Cawley & Talbot] 2010), which should be chosen depending on the application domain. To approximate the conditional density, the CKDE simply takes the ratio of these two KDE approximatious:
P(a,y) v) p(x))
m K dd (m..m)))))
K'
dg (1-36)
(m(()
-==1
Note that the above approximation is not a distribution because it is not normalized.
4 Methods
There are several concerns with using Equation 2 as the likelihood function to learn the posterior dis- tribution of the reward function of the test task. First, Equation 2 uses the optimal Q-value function, which is difficult to learn given a limited amount of demonstration data from the test task. As result, a large demonstration dataset. is required for posterior contraction. Furthermore, current literature often uses
5

---

Published in Transactions on Machine Learning Research (10/2024)
parametric methods to estimate the optimal Q-function, As result, if the function class is misspecified, the likelihood function is not guaranteed to converge to the true likelihood function.
To guarantee likelihood convergence, we use KDE, which is a non non-paramttric method, to learn a Bayesian IRL posterior. As mentioned earlier, we also use the information stored within the training task datasets to improve posterior contraction with limited test task demonstration data.
4.1 Kernel density Bayesian IRL
To avoid the issues with prior approaches, we propose kernel density Bayesian inverse reinforcement learn- ing (KD-BIRL), which uses CKDE approximate the likelihood. CKDE calculates the likelihood of a reward function sample by weighting the density estimate based the distance observations from the training data. The CKDE allows us to approximate the Bayesian IRL likelihood using the training dataset {$3, aj, R5)) as
(s, a, R) Pm (R)
pm(s, a R)
(d(((1a))1330011)) (R,() K"
(5)
K
h
N
Ee
K'
de(R,Rd)
where h, W/ > 0 are the bandwidth hyperparameters with larger values giving a smoother function estimate,
and d, (S x A) x (S x A) Rzo and d, R R + Rzo, specify the distance between state-action tuples and reward functions, respectively.
Given expert demonstration samples from the test task, m training demonstration samples from the training tasks, and a prior p(R), we now learn a posterior on R as,
(R] (H) & p(R)
pm (5 ot (R)
(6)
m K
K'
(R, 1
= P(R
(7)
E K'
(R)
There are many possible options for the kernel K, including uniform and triangular kernels. In KD-BIRL, we use a Gaussian kernel because it can approximate bounded and continuous functions well. Several prior distributions can be appropriate for the prior distribution, p(R) depending on the characteristics of the MDP, including Gaussian (Qiao & Beling, 2011), Beta [Ramachandran & Amir 2007), or Chinese restaurant process (CRP) (Michini Howl 2012a). define distance metrics on a vector space, one can choose cosine similarity, dot product similarity, Manhattan distance, etc. In our work, we choose Euclidean distance for d,, d, and uniform Gaussian prior depending the application domain. We choose the bandwidth hyperparameters h, ht' using mule-of-thumi) procedures (Silverman 1986). These procedures define the optimal bandwidth hyperparameters as the variance of the pairwise distance between the training data demonstrations and the training data reward functions respectively. Intuitively, one wants to choose the smallest bandwidth hyperparameter the data will allow, to avoid density estimates that are too smooth.
There are many techniques to sample from the posterior distribution (Fquation T such as MCMC sam- pling Van Ravenzwaaij al. (2018) and variational inference [lei et al. (2017). We use a Hamiltonian Monte Carlo algorithm Team 2011 (details in Appendix F and Algorithm D which is suited to large parameter spaces. However, note that KD-BIRL introduces new likelihood estimation approach, and that any pos- terior sampling algorithm (i.c. Markov Chain Monte Carlo (MCMC), Metropolis Hastings, Gibbs Sampling etc.) can be used. A key computational gain of our approach over the original Bayesian IRL algorithm (also a sampling- based approach) is that we avoid the cost of re-estimating the Q-value function with every iteration of sampling. This possible because our posterior distribution (Equation D does not depend on Q". Thus, sampling from the posterior is less computationally intensive than sampling from the standard Bayesian IRL algorithm posterior.

---

Published in Transactions on Machine Learning Research (10/2024)
4.2 natture-based reward function
While KD-BIRT can be applied directly in environments where the reward function has few parameters, kernel density estimation is known to scale poorly to high dimensions. In practice, the computational cost of the CKDE increases substantially the number of reward function parameters and the number of behavior samples increase [Izbicki & Lee 2017p. To allow CKDE to perform reward inference in environments with higher-dimensional reward functions, we choose to re-parameterize the reward function.
We adopt a fealure-bused reward function, which allows us to represent the reward as a linear combination of weight vector and low-dimensional feature encoding of the state (Abbeel Ng 2004 Hadfteld Menell et al.. 2017). Earlier work demonstrates the success of feature-based reward functions imitation learning in
2020 The feature- ((),a),, where E IR9 and Φ Sx R2. Here, é is known function that maps a state-action tuple to a feature vector of length q. This parameterization is advantageous because the dimension of the weights does not scale with the dimension of the state representation or the cardinality of the state space, and does not rely on the state space being discrete. Ideally, q is large enough fully represent the state-action space, but small enough for computational efficiency. Our work assumes that the featurized projection Φ(s, a) fully captures the information in the given state-action tuple. The goal of Bayesian IRL method is now to learn posterior over rather than R.
complex control problems (Ratliff et al. 2006b Ziebart et al. 2008 2009 Brown et al.
based reward function is written as R(s, a)
Under the feature-based reward function paradigm, sample in the training task dataset is now of the form {(S aj, w where wj is the weight vector of length associated with the sample(sj,4) The posterior over the weights w using the CKDE formulation is,
pm(s a, w) Pm (w) m K [clolee))))(s)a))))
(m a (w)
K'
dr dr(w,,)
(8)
Ee K' (ww,))))))
where d, measures the similarity between weight vectors, and d, is the distance between statc-action tuples projected into lower dimension using の. The posterior is then
(W p(w) II
pm (5 of w)
(( K' de (w,w,)
(9)
m K
p(w)
I1
E K' (r(w)))
The procedure for sampling from the posterior identical, except that Equation used calculate the likelihood, and Equation update the posterior.
5 Theoretical Guarantees
Before we investigate our method empirically, we prove asymptotic postcrior consistency. Posterior consis- tency ensures that as the number of samples increases, the posterior distribution centers around the true reward function parameter value. This is important to ensure that our method is reliable incorporating new data, and given enough data, will accurately learn the parameters corresponding to the true reward function.
KD-BIRL uses a probability density function to approximate the likelihood; consequently, we can reason about its posterior's asymptotic bohavior. In particular, we show that the posterior estimate contracts as it receives more samples. This type of analysis also aligns with the behavior that we want in practice; more demonstration samples should shrink the posterior distribution around the true reward function.

---

Published in Transactions on Machine Learning Research (10/2024)
We first focus the likelihood estimation step, and show that, when the size of the training dataset samples, m, approaches no and the samples are associated with reward functions that sufficiently cover R, the likelihood approximation (Equation 5 converges to the true likelihood p(s, R).
Lemma 5.1. Let hm. N'm > be the bandwidths chosen to estimate the joint probability density and marginal probability density respectively. Assume that both the likelihood p(s, a R) and prior p(R) are square-integrable and twice differentiable with square- integrable and continuous second order derivative, and that mh and mhn." v 12
m - 00, where p is the dimension of (8,a, a, R) and p' is the dimension of R. Then,
Pm (n, R)
P 2 R), V(8 a, R) ESSAAX,
where (m is the appronimation of the likelihood using m training dataset samples.
Lemma 5.1 verifies that we can approximate the true likelihood function using CKDE, which opens the door to Bayesian inference. Because the IRL problem non-identifiable, the "correct" reward function may not be unique (Abbeel & Ng 2004 Ratliff et al. 2006b Ziebart all 2009 In this work, assume that any two reward functions that load an agent to behave in the same way are considered equivalent. That is, if a set of demonstration trajectories is equally likely under two distinct reward functions, the functions are considered equal: R1 = R2 if (R)) - p(. R2)ll = O. Using this intuition, we can then define the equivalence class [R**] for R as [AM* = {R E R R = J. Our objective is to learn a posterior distribution that places higher mass on reward functions that are in the equivalence class [R*].
We now show that as n, the size of expert demonstration dataset for the test task, and m, the size of the training demonstration dataset, approach oo, the posterior distribution contracts to the equivalence class of the reward function that best explains the expert demonstration dalaset [R(]]
Theorem 5.2. Assume the prior for R, denoted by P, satisfies P([R: KL(R",R) < > 0 for any > O, where KT. is the Kullback-Leiin functions such that each reward function is within a KL-divergence distance of c from the data-generating reward function Rt Assume R C Bd a compact sel. Let Din be the posterior densily function estimated using test task samples and m training task samples. Then, the posterior measure corresponding Pm (Equation D) denoted by Pm, is consistent w.r.t. the L1 distance; that is,
divergence. That is, prior P defines a non-emply set of possible reword
Pm ((R(( [pp(( p(11R")l,, <()) m03, 1.
Theorem 5.2 verifies that the posterior Pn assigns almost all mass to the neighborhood of [R*]. This means that KD-BIRL asymptotically contracts to the equivalence class of the test task data-generating reward function R Note that this not a statement regarding the posterior contraction rate, just a certification of contraction. Proofs for both Thcorem 5.2 and Lemma 5.1 are in Appendix A
6 Experiments
Building on our theoretical analysis, we soek to answer the following questions with our empirical analyses:
How does KD-BIRL perform in comparison to baseline methods (Section 6.1 under the standard reward function parameterization?
2. How does the featurized reward function parameterization improve performance? (Section 6.4
3. How much quicker can KD-BIRL contract in comparison to baseline methods? iSe
6.4
To answer these questions, we investigate two environments. The first is a Gridworld setting with a discrete state space. We use three grid sizes (2 x 2, 5 x and 10 x 10) to investigate how KD-BIK KD-BIRL's performance scales. The second sctting is a simulated sepsis treatment environment (aaaaentttAmmm (Amirhossein Kiani 2019), which has continuuus state space and is thus, more challenging. The objective of the sepsis environment is administer series of treatment such that patient is successfully discharged.
8

---

Published in Transactions on Machine Learning Research (10/2024)
6.1 Baselines
We evaluate KD-BIRL's empirical performance in comparison to three Bayesian IRL baselines. We focus Bayesian IRI, approaches because the inferred posterior distribution provides uncertainty estimates of the reward function, which frequentist approaches fail to provide. Our first baseline is a single-task Bayesian IRL (BIRL) approach (Ramachandran & Amir 2007,, which we compare to in the environments with smaller state spaces. We avoid this comparison in the larger state-space environments because BIRI. is computationally prohibitive to run. Next, we compare to a more recent single-task Bayesian IRL method, Approximato Variational Reward Imitation Learning (AVRIL) (Chan van der Schaar 2021). AVRIL uses variational inference to approximate the posterior distribution efficient at the cost of approximate inference. As stated carlier, one goal of our evaluations is to verify that incorporating data from training tasks improves the contraction rate of the posterior. Thus, we also compare to a baseline in which AVRIT is initialized using an informative prior learned from the training tasks. In this way, AVRIL can also take advantage of the information stored in the training task dataset. We initialize the mean and variance parameters of the AVRIL prior using the reward function samples from the training dataset. Specifically, given training datasel, {(8), 0y R3)}== p(R) NNoo og), where Po + E/ Rg(s5,a) az) and oî = +. د= (Rz(sj, -a)²².
the reward function, which is more computationally
6.2 Evaluation Metrics
To evaluate the learned posterior distributions of the reward functions, we follow earlier work and use expected value difference (EVD) (Choi & Kim 2012 2013 Brown & Niekum, 2018 Levine et al. 2011). EVD measures the difference in total reward obtained by an agent whose policy is optimal for the true (data-generating) reward function and an agent whose policy is optimal for some estimated reward function. Define the value function of policy x under the reward function as,
VT.
ES
Σ
> R(S,, Ax) So
Then, EVD calculated as
EVD EVD=IV=*.' -V="." (=(1"),
where n² and Tf² the optimal policies for the true reward function R' and reward function rd drawn from the posterior, respectively. EVD is a suitable metric for comparing the performance of methods without directly comparing the learned reward functions, which may have distinct functional forms. The lower the EVD, the better the estimated reward function ما captures the qualities of the true reward function R (details in Appendix H.
In addition to EVD, which evaluates the posterior concentration at the true reward function parameters, we plot the marginalized posterior distributions, which highlight how well KD-BIRL captures uncertainty.
6.3 Original reward function parameterization
In our first set of evaluaaions, we represent It as a vector in which each cell contains a parameter corresponding to the scalar reward in one of the states. For example, a 10 x 10 Gridworld environment, there are 100 distinct states, so the reward function RE 100 is represented as a vector of length 100.
We first demonstrate that KD-BIRL's marginalized posterior distribution is more concentrated than baseline approaches in 2 x 2 Gridworld. Here, the test task demonstrations arise from the reward function R [0, 00,0,0,1] 1]. In addition to these test demonstrations, KD-BIRI. receives training task demonstrations from the reward functions π²2 = [1, O, 0, 0J and 1²2 [0, 1, 1, 0j. To evaluate our method, we visualize the density of the posterior samples for each of the baselines, marginalized at each state.
The posterior samples from KD-BIRL are more concentrated around R' than bascline approaches (Figure) 1 The BIRL posterior samples are also close the ground truth reward, but not as concentrated those of KD- BIRL. In contrast, herther AVRIL nor AVRIT with an informative prior has concentrated posteriors in all of
9

---

Published in Transactions on Machine Learning Research (10/2024)
00.1
Joint Dersity
16
16-
Densky
0.4
0.2
0.0
0.8
00.00
[1.0]
10
0
10
0.4
16-
10
Posterfor Mean
Dennty
0.5
0.4
0.0
0.2
0.0
0.2
Reward
10 -C.8
-0.6
-0.4
KD-SRRL
AVRIL
BIRL
(nfommmta
Figure 2: Using known features to parameterize re- ward enables reward inference in two settings in 10x10 Gridworld. The first column shows w" projected onto the Gridworkd, the second shows the KD-BIRL posterior mean, and the third shows the joint density plots the two weights, with the first on the x-axis and the second on the y-axis. KD-BIRL accurately in- fers the relative magnitude and sign of the individual weights
Figure KD-BIRL learns more concen- trated posteriors towards R than base- line methods. Dashed vertical lines display the true reward. The x and y-axis repre- sents the sampled reward marginalized at the given state and estimated density, re- spectively. A perfect marginalized posterior distribution would have the highest density at the black line (R*) in each state.
the states. Notably, in state (0, 1], these distributions are mostly uniform, when they should be concentrated at zero. This result indicates that using the combination of the training- and test task demonstrations can improve concentration of the posterior, but an informative prior is insufficient to make use of this information. In contrast, incorporating this information using the CKDE-based likelihood formulation can lead to concentrated state-marginalized posterior distributions.
While the 2 x 2 Gridworld experiment is promising, we would like to apply our method in enviromments with much poorly as the number of reward function parameters increase (Izbicki & Lec 2017 To support this claim and motivate the use of a feature-based reward function, we show results under the original reward function parameterization in a 5 x Gridworld environment.
reward functions. As discussed in Section g kernel density estimation scales
We select data-geng reward function R* that has nonzero reward in only one of the states (Figure R Panel 1). The state occupancy of the test task expert demonstrations can be seen in Figure 3 Panel 4. From Panel 4, we see that the test task expert demonstrations all approach the upper right corner of the grid. KD- BIRL also receives training task demonstrations from three other reward functions. The state occupancy density of the training demonstrations can be seen in Figure g Panel 3. In contrast with the test task expert demonstrations, the training task demonstrations cover far more states in the environment. Thus, the training task demonstrations contain information about portions of the state-space that the expert demonst.ratio do not cover, which should ideally improve posterior distribution inference.
In our results, we find that method is able to estimate posterior whose mean is in the equivalence class of R (Figure Pancl 2). In other words, an agent that follows the reward function identified by the postcrior mean would still be directed to the terminal state in the upper right corner of the grid (which contains the highest reward). However, despite having demonstrations from four separate reward functions across the training and test tasks, the postcrior mean is slightly inaccurate. There are states ([2, 3), [4,3] in Figure 3 Panel 2) where the estimated scalar reward is incorrect, which suggests that the CKDE struggles to learn 25 independent reward parameters successfully from these diverse demonstrations. This result motivates the
10

---

Published in Transactions on Machine Learning Research (10/2024)
Training Dataset State Occupancy
Expert Dataset State Occupancy
R
Posterior Mean
0
X
X
X
Figure 3: We study KD-BIRL's performance in a 5 x 5 Gridworld environment in which the reward is positive in one state (upper right corner). The first pancl shows the data generating reward function R", the second panel shows the KD-BIRL posterior mean, and the third and fourth panels show the state occupancy distribution of the training task and expert task demonstrations respectively. The KD-BIRL posterior mean is within the equivalence class of R*, because the darkest square is in the upper right hand corner, but there are incorrect estimates of reward in other states.
use of a. feature-based reward function, which enables KD-BIRI. to perform reward inference in environments with higher-dimensional state spaces.
6.4 Feature-based reward function
Now, investigate feature-based reward function, which enables KD-BIRL to perform posterior inference in environments with high-dimensional state space. To build intuition for this approach, we first investigate 10 x 10 Gridworld environment, where the state space S is the series of vectors of length 100. Here, we select (88 a) = [ar, y] to be a function that maps the state vector to the spatial coordinates of the agent. That is, we treat the coordinates the agent as a known "feature vector". In this setting, the reward function is low-dimensional weight vector that we aim learn. Similar prior
is R" = T domains, KD-BIRL receives training task samples from two other reward functions each parameterized by corresponding weight vectors.
((ss T= a) where
Our results indicate that feature-based reward function allows KD-BIRL to perform reward inference in 10 x 10 Gridworld under two different reward functions. First, we note that the posterior mean, when projected onto the Gridworld, is nearly identical to the ground truth weight vector w" projected onto the Gridworld. This indicates that the learned posterior is concentrated close to w*, which is ideal. Next, we visualize the learned posterior parameters for both weights, and find that KD-BIRI. is able to accurately recover the relative magnitude and sign of the individual weights in two settings with different w* values (Figure 2. Our results in the 10 10 Gridworld setting demonstrate that KD-BIRL is able to infer reward functions in environments with high-dimensional (R R 1000) state spaces using featurized reward function.
Next, we investigate KD-BIRL's performance in the sepsis treatment environment, which presents additional challenges due to its continuous state space. Unlike the previous Gridworld environments, each state in the sepsis environment
vector of length 46, where each element. corresponds a real-valued measurement of given feature. As a result, the reward function parametcrization needs to learn a low-dimensional representation of the state features. To do this, we learn & using a variational auto-encoder (VAE). The use of VAE is appropriate in this situation because do not know in advance which features will be most relevant to reward function inference. The VAE takes as input vectors of length 47 (46 state foatures + treatment action), and projects them into latent space that has 3 features.
Now, we seek to understand how KD-BIRL performss when it receives few test task expert demonstrations in comparison the size of the training task datasct. KD-BIRL reccives expert demonstrations from one unknown reward function, which is parameterized using set of weights w*, and training task demonstrations from four additional known reward functions. Our baselines include AVRIL, which only receives the expert
11

---

Published in Transactions on Machine Learning Research (10/2024)
10, 0.3]
w"=[0.5, 0.5, 0.5]
w"=[0.6, 0.8, 1.0)
w"=[0.8, 0.6, 0.4]
40
40
40
40
EVD
20
20
400 800
400
800
400
800
400
800
n
AVRIL
AVRIL (Informative Prior)
KD-BIRL
Figure 4: The KD-BIRL posterior empirically contracts faster than baselines. Here, we evaluate four sepsis treatment settings, each with a distinct w". The z-,y- axes represent the number of test demonstrations and the EVD of the learned posterior respectively, and the error bars indicate slandard error. We compare to two baselines, AVRIL and AVRIL with an informative prior. Our results indicate that KD-BIRL is able to produce lower or comparable EVD with fewer samples than either bascline in low expert demonstration data regime (n < 1000).
3
Density
2
0.0
0.5
0.5
0.0
0.5
1.0
w[0]
w[1]
w[2]
KD-BIRL
True parameter
Figure 5: The marginal posterior of KD-BIRL for a chosen w in the sepsis environment: The true parameter is displayed in the black dashed line, and the density the posterior marginalized at each parameter is plotted on three plots.
demonstrations, and AVRIL with an informative prior, which uses prior that is calculated based on the mean and variance of the demonstrations in the training task dataset.
In four settings, each with different w*, KD-BIRL's posterior samples generate consistently lower EVD than AVRIL (Figure 1 and notably lower standard error than AVRIL with an informative prior. In particular, KD-BIRL achieves lower EVD in the low-expert demonstration data regime where there are fewer than 1000 samples the test expert demonstration dataset. Lower EVD implics that the posterior more concentrated at w*. Figure g shows that as n increases, the EVD of KD-BIRT, lower than both baselines. which implics that the KD-BIRL postcrior more concentratod with the same number of samples. This indicates that KD-BIRL can learn a more accurate posterior with fewer test demonstration samples by leveraging information from the training tasks. Taken together, this suggests that the KD-BIRL posterior requires fewer samples to contract to w*. We also note that the informative prior that is learned using the training task demonstrations has a large variance, which we believe leads to posterior samples that produce EVDs with high variance.
Finally, as noted carlier, one of the benefits of using Bayesian method is uncertainty quantification. We visualize the learned posterior parameters for a fifth setting with new wt (Figure 0 and find that the posterior concentrated at the true parameter. Our experiments demonstrate that using a VAE to learn
12

---

Published in Transactions on Machine Learning Research (10/2024)
& can effectively summarize state information in continuous and high-dimensional state spaces. This result, coupled with asymptotic posterior consistency guarantees, makes KD-BIRL superior choice for performing IRL when there are few test task expert demonstrations available.
7
Discussion
In this work, present kernel density Bayesian inverse reinforcement learning (KD-BIRL), an IRL algo- rithm that uses CKDE to incorporate existing domain-specifie datasets to improve the posterior contraction rate. We address gap in the Bayesian IRL. literature by leveraging consistent likelihood estimator and translating this estimator into posterior consistency guarantees for this class of methods. Our experiments demonstrate that KD-BIRL can perform TRT, in high-dimensional and continuous state spaces. Notably, KD-BIRL learns a more concentrated posterior distribution with fewer samples, outperforming a leading single-task IRL method, even when this leading method initializod with an informative prior. Our results show that KD-BIRL can learn posterior distributions that capture key characteristios of objectives while providing uncertainty estimates, even with limited data.
Limitations Our work assumes that the expert demonstrations are of high quality; we anticipate that training lask demonstrations that are not optimal w.r.t the corresponding reward function will result in less accurate likelihood estimates. Furthermore, we assume that the features used in the feature-based reward function fully capture the information stored within state-action tuple. Further work is needed to analyze the effect of less representative features. Our work also assumes that the L1 norm is a valid distance metric between posterior densities, and this may not be true depending the application setting.
Future Work Several future directions remain open. The particular choices of distance metrics and hyper- parameters used in the CKDE depend on the enviromment and reward function parameterization; additional experimentation is required to characterize the relationship between these metrics and the quality of the learned posterior. Furthermore, it is of interest to explore other ways in which the CKDE can be used to infer higher dimensional reward functions. This may include exploring how to speed up the CKDE (Holmes et al. 2007). replacing it with another nonparametric conditional density estimator (Rojas al.) 2005 work proposes using density estimation approximate the likelihood. and there several other choices for learning this density function including diffusion models
Hinder al. 2021 Additionally,
(Cai & Lee
2023 and Gaussian processos (D'mmilio et al. 2021
References
Picter Abbeel and Andrew Y. Ng. Apprenticeship lcarning via inverse reinforcement learning. In The Twenly-First International Conference on Machine Learning, 2004.
Peter Henderson Amirhossein Kiani, Tianli Ding. Gymic: An OpenAl gym environment for simulating sepsis treatment for icu patients, 2019.
Saurabh Arora, Prashant Doshi, and Bikramjit Banerjee. Min-max entropy inverse rl of multiple tasks. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 12639-12645, 2021. doi: 10. 10.1109/ICRAA 10110//CCA4500.2200 2961177777
Zoe Ashwood, Nicholas A. Roy, Hyun Bak, and Jonathan W Pillow. Inferring learning rules from animal decision-making, In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 3442-343 3453. Curran Associates, Inc., 2020.
Julien Audiffren, Michal Valko, Alessandro Lazaric, and Mohammad Ghavamzadeh. Maximum entropy semi-supervised inverse reinforcement learning. In Proceedings of the 24th International Conference on Artificial Intelligence, IJCAI'15, pp. 3315-3321. AAAI Press, 2015. ISBN 9781577357384.
Sreojith Balakrishnan, Quoe Phong Nguyen, Bryan Kian Hsiang Low, and Harold Soh. Efficient explo- ration of reward functions in inverse reinforeement learning via Bayesian optimization. orXiv preprint M:Xiv:2011.08.. 2020.
13

---

Published in Transactions on Machine Learning Research (10/2024)
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the Americon stalistical Association, 112(518):859 877,,877- 2017.
Daniel S. Brown and Scott Niekum. Efficient probabilistie performance bounds for inverse reinforcement learning. In AAAI, 2018.
Daniel S. Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast Bayesian reward inference from preferences, 2020.
Maxwell X. Cai and Kin Long Kelvin Lee. p-diffusion: A dlffusiio-based density estimation framework for computational physics, 2023.
Julia Carnevale, Fric Shifrut, Nupura Kale, William Nyberg, Franziska Blaeschke, Yan Chen, Zhongmei Li, Sagar Bapat, Morgan Diolaiti, Patrick O'Leary, Shane Vedova, Julia Belk, Bence Dániel, Theodore Roth, Stefanie Bachl, Alejandro Anido, Brooke Prinzing, Jorge Ibañez- Vega, Shannon Lange, and Alexander Marson. Rasa2 ablation in cells boosts antigen sensitivity and long-term function. Nature, 609:1-9, 08 2022. doi: 10.1038/s415866
05126-w.
Gavin C. Cawley and Nicola C. Talbot. On over-fitting model selection and subsequent selection bias in performance evaluation. Journal of Machine Learning Research, 11(70):2079 2107, 2010.
José E. Chacón and Tarn Duong. Multivariate kernel smoothing and its opplications. Chapman and Hall HRR//CRC, 2018.
Alex James Chan and Mihaela van der Schaar. Scalable Bayesian inverse reinforcement learning, In Inter- national Conference on Learning Representations, 2021.
Jiayu Chen, Dipesh Tamboli, Tian Lan, and Vancet Aggarwal. Multi-task hicrarchical adversarial inverse reinforcement learning, 2023.
Jaodeug Choi and Keo-eung Kim. Nonparametric Bayesian inverse reinforccment learning for multiple reward functions. In Advances in Neural Information Processing Systems, volume 25, 2012.
Jaedeug Choi and Kee-Eung Kim. Bayesian nonparametric feature construction for inverse reinforcement learning. In The Oeenty-Thhrr International Joint Conference on Artificial Intelligence, IJCAI '13, 2013.
Christos Dimitrakakis and Constantin A. Rothkopf. Bayesian Multitask Inverse Reinforcement Learning, pp. 273-284. Springer Berlin Heidelberg, 2012. ISBN 9783642299469..
V D'Emilio, R Green, and V Raymond. Density estimation with gaussian processes for gravitational wave posteriors. Monthly Notices of the Royal Astronomicul Sociely, 508(2):2090) 2097, September 2021. ISSN 1365-2966.
Ali Exzeddine, Nafee Mourad, Babak Nadjar Araabi, and Majid Nili Ahmadabadi. Combination learning from non-optimal demonstrations and feedbacks using inverse reinforcement learning and bayesian policy improvement. Expert Systems with Applications, 112:331-341, 2018. ISSN 0957-4174.
David Feldman, Luke Funk, Anna Le, Rebecca J Carlson, Michael D Leiken, FuNien Tsai, Brian Soong, Avtar Singh, and Paul Blainey. Pooled genetic perturbation screens with image-based phenotypes. Nature prolocols, 17(2):476 512, 2022.
Andrew Gelman, John Carlin, Hal Stern, David Dunson, Aki Vehtari, and Donald Rubin. Bayesion Dala Analysis. 2020. URL http::/www. stat, columbia, edu/-gelman/bookk/
Subhashis Ghosal and Aad van der Vaart. Fundamentals of nonporametric Bayesian inference, volume 44. Cambridge University Press, 2017.
Adam Gleave and Oliver Habryka. Multi-task maximum entropy inverse reinforcement learning, 2018.
Dylan Hadfield- Menell, Smitha Milli, Picter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design, 2017.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Fabian Hinder, Valerie Vaquet, Johannes Brinkrolf, and Barbara Hammer. Fast non-parametric conditional density estimation using moment trees. In 2021 TEEE Symposium Series on Compulational Intelligence (SSCI), pp. 1-7. LEEE, 2021.
Michael P. Holmes, Alexander G. Gray, and Charles Leo Isbell. Fast nonparametric conditional density estimation. In The Twenty-Thite Conference on Uncertainty in Artificial Intelligence, UAI'0,, Arlington, Virginia, USA, 2007. AUAI Press.
Rafael Izbicki and Ann B. Lee. Nonparametrie conditional density estimation in a high-dimensional regression setting. Journal of Computational and Graphical Statistics, 25(4):1297-131) 2016.
Rafael Izbicki and Ann B. Lee. Converting high-dimensional regression to high-dimensional conditional density estimation, 2017.
Alistair Johnson, Tom Pollard, Steven Horng, Leo Anthony Celi, and Roger Mark. Mimic-iv-note: Deiden- tified free-text clinical notes, 2023.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li- Li-Wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, freely accessible critical care database. Scientific data, 3(1):1-9 9, 2016.
Robert Kass and Larry A. Wasserman. The selection of prior distributions by formal rules. Journal of the American Statistical Association, 91:1343 1370, 1996. URL http:::/aapp...maatttccoooooooot semanticcccatt org/ Corpus ID: D: 53645083
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Zico Kolter, Pieter Abbecl, and Andrew Ng. Hicrarchical apprenticeship learning with application quadruped locomotion. In Advances in Neural Information Processing Systems, pp. 769-77 769-776. Citeseer, 2008.
Sergey Levine, Zoran Popović, and Vladlen Koltun. Nonlinear inverse reinforcement learning with Gaussian processes. In The 24th International Conference on Newral Information Processing Systems, NIP$'11, 11, pp. 19 19-27, Red Hook, NY, USA, 2011. Curran Associates Inc. ISBN 97816183099 8395993.
Xiaofeng Liao, Wenxue Li, Hongyue Zhou, Barani Kumar Rajendran, Ao Li, Jingjing Ren, Yi Luan, David Calderwood, Benjamin Turk, Wenwen Tang, Yansheng Liu, and Dianqing Wu. The cul5 e3 ligase complex negatively regulates central signaling pathways in ed8 t cells. Nature Communications, 15, 01 2024. doi: 10.1038/s441 00008///1144--0244485555
Henry B Mann and Abraham Wald. On stochastic limit and order relationships. The Annals of Mathemalical Statistics, 14(3):217-22) 226, 1943.
Bernard Michini and Jonathan P. How. Bayesian nonparametric inverse reinforcement, learning. Tn Peter A. Flach, Tijl De Bie, and Nello Cristianini (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 148 163, Berlin, Heidelberg, 2012a. Springer Berlin Heidelberg.
Bernard Michini and Jonathan P. How. Improving the efficiency of Bayesian inverse reinforcement learning, In 2012 IEEE International Conference on Robolics and Aulomation, pp. 3651-3656, 3656, May 2012b. doi: Tw..1199ICA...1.....0
Katja Mombaur, Anh Truong, and Jean-Paul Laumond. From human to humanoid locomotion an an optimal control approach. Autonomous Robots, 28(3):369 383, 2010.
inverse
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In in Proc. 17th Interna- lional Conf. on Machine Learning, pp. 663 670. Morgan Kaufmann Publishers Inc., 2000.
Qifeng Qiao and Peter A. Beling. Inverse reinforcement learning with Gaussian process. 2011 American Control Conference, pp. 113 118, 2011.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In The 20th International Joint. Conference on Artifical Intelligence, pp. 2586-2591, 2591, 2007.
Nathan Ratliff, David Bradley, J. Andrew Bagnell, and Joel Chestnutt. Boosting structured prediction for imitation learning. In Advances in Neural Information. Processing Systems, pp- 1153-1106 2006a.
Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. Maximum margin planning- The 23rd International Conference on Machine Learning, ICML '06, pp. 729 736, New York, NY, USA, 2006b. Association for Computing Machincry.
Jaakko Rithimäii and Aki Vehtari. Laplace approximation for logistic gaussian process density estimation and regression. Buyesian Analysis, 9, 11 2012. doi: 10.1214/14-BA87.
Alex L Rojas, Christopher R Genovese, Christopher J Miller, Robert Nichol, and Larry Wasserman. Con- ditional density estimation using finite mixture models with an application to astrophysics. Center for Automatic Learning and Discovery, Department of Statistics, Carnegie Mellon University, 2005.
Constantin A. Rothkopf and Dana H. Ballard. Modular inverse reinforcement learning for visuomotor behavior. Biol. Cybern.. 107(4):477-490, aug 2013. ISSN 0340-1200.
Lorraine Schwartz. On Bayes procedures. Zeitschrift für Wohrscheiiliccheesstheeree und Verwandte Gebiete, 4:10 26, 1965.
B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, London, 1986.
Stan Development Team. Stan modeling language users guide and reference manual, version 2.29, 2011.
Aad van der Vaart. Asymptotic Statistics, volume 3. Cambridge University Press, 2000.
Don Van Ravenzwaaij, Pete Cassey, and Scott D) Brown. A simple introduction to markov chain monte-carie sampling. Psychonomic bulletin & review, 25(1):143 154, 2018.
Adrian Sošié, Abdelhak M. Zoubir, Elmar Rueckert, Jan Peters, and Heinz Koeppl. Inverse reinforcement learning via nonparametric spatio-temporal subgoal modcling. Mach. Learn. Res., 19(1):277722211, jan 2018. ISSN 1532-4435.
Matt P Wand and M Chris Jones. Kernel smoothing. CRC press, 1994.
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning prior over intent via mela-inverse reinforcement learning, 2019.
Chao Yu, Jiming Liu, and Hongyi Zhao. Inverse reinforcement learning for intelligent mechanical ventilation and sedative dosing in intensive care units. BMC Medical Informalics and Decision Making, 19, 04 2019a. doi: 10.1186/s12911-019-
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with prob- abilistic context variables. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Curran Associates, Tnc., 2019b.
Jiangchuan Zheng, Siyuan Liu, and Lionel M. Ni. Robust Bayesian inverse reinforcement learning with sparse behavior noise. In Association for the Advancement of Artificial Intelligence, AAAI14, pp. 2198 2205. AAAI Press, 2014.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K. Dey, et al. Maximum entropy inverse reinforeement learning. In Association for the Advancement of Artificial Intelligence (AAAI), volume 8, pp. 1433 1438. Chicago, II., USA, 2008.
Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Human behavior modeling with maximum entropy inverse optimal control. In AAAI Spring Symposium: Human Behavior Modeling, 2009.
16


Reviews:
[{"review_id": "Up4RtgxS1r", "reviewer": "Reviewer_SESx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The problem of inverse reinforcement learning is much relevant in both theory and practice and this work presents a significant step forward in the field. The paper is clear and sound, and the method is sufficiently novel. \n\nOne weakness I point out is minor and is the use of jargon from very early on in the paper. Terms such as \"posterior contraction\", \"posterior concentration\", and perhaps even \"posterior\", \"prior\" and \"likelihood\" may not be perfectly clear to the general reader.\n\nMy other question is regarding the empirical analysis. The author say their third question empirical question is whether their method contracts quicker than baselines (\"How much quicker can KD-BIRL contract in comparison to baseline methods? (Section 6.4)\", bullet 3 in the first paragraph of Section  6). I do not understand which results/figures correspond to that specific question. In other words, I don't see/understand how the contraction rate was analyzed and against which baselines and would appreciate the authors clarifying.\n\nFinally, I also think it would have been important to see discussed how the quality of the expert demonstrations may affect performance, as well as the choice of features.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes KD-BIRL, a bayesian inverse reinforcement learning method that leverages a training set of expert demonstrations and reward functions in order to achieve better performance on estimating a new reward function given a new set of expert demonstrations. The method is based on a kernel density estimator for approximating the likelihood of the observations given a reward function. The method is theoretically shown to converge to the correct equivalence class of reward functions and empirically shown to have a better performance than the baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My minor recommendation is that \"posterior contraction\" and \"posterior concentration\" are either explained or replaced by common-knowledge terms early on in the paper (including the abstract and introduction).\n\nMore relevant, I would also like to see clarified the third bullet I mentioned above in the empirical analysis section (where and how was the contraction rate analyzed and against which baselines).\n\nLastly, I think it would be important to see mentioned/discussed the impact of non-expert demonstrations and rewards on the training sets as well as the impact of the choice of features.", "broader_impact_concerns": "It could be important to refer the implications of the inclusion of non-expert demonstrations and rewards on the training sets, as well as the impact of the choice of features.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nsdKlaHSG2", "reviewer": "Reviewer_bBST", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I want to preface this section by stating that I am not an expert in this area so I apologize for any inaccuracies.\n\n## Strengths:\n- The integration of CKDE into Bayesian IRL allows the method to perform better in low-data scenarios by leveraging additional information from training tasks.\n- The paper provides a solid theoretical basis for the method, though it relies on existing theoretical results, making the analysis relatively straightforward.\n- The experiments demonstrate the method’s effectiveness in both discrete and continuous environments.\n\n## Weaknesses:\n- The computational demands of kernel density estimation might limit the method’s applicability to high-dimensional problems.\n- The use of the \\(L_1\\) norm in Theorem 5.2 and the assumption \\(P(\\{R: KL(R^\\ast, R) < \\epsilon\\}) > 0\\) could be more clearly explained. The assumption might be better expressed as stating that there exists a non-empty set satisfying the condition.\n- The paper does not explicitly discuss the method’s limitations, particularly with respect to scalability and assumptions made in the theoretical analysis.\n- I think the authors could test their KD-BIRL algorithms on more environments to further demonstrate its generalizability and robustness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new method called Kernel Density Bayesian Inverse Reinforcement Learning (KD-BIRL). This method enhances existing Bayesian IRL approaches, particularly in situations where expert demonstration data is scarce. The key idea is to use conditional kernel density estimation (CKDE) to incorporate additional data from training tasks, which improves the accuracy of the learned reward function. The authors also provide theoretical guarantees on the method’s ability to concentrate on the correct reward function as more data becomes available. Empirical results show that KD-BIRL outperforms traditional approaches in both discrete (Gridworld) and continuous (sepsis treatment simulation) environments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses for the details.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ehEBbZsA2z", "reviewer": "Reviewer_SgoL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The Conditional Kernel Density Estimation (CKDE) approach is a well-conceived and reusable framework. It incorporates information from alternative demonstrations, allowing for a more efficient convergence to the true reward function of the target task compared to other known methods.\n- The paper is clearly written, and the experimental results are robust, particularly in high-dimensional settings\n- The workflow presented for addressing high-dimensional IRL problems is both effective and generalizable. The use of a low-dimensional representation of the reward functions such VAEs employed here, combined with CKDE, provides a practical and reusable strategy for similar tasks.\n- The theoretical consistency guarantees provided by the authors offer a strong mathematical foundation for the approach\n\nWeaknesses:\nI believe the main weakness of the paper is the application of this novel method in only simple if not toy settings. The majority of the experiments discussed are variations of Gridworld, while the sepsis setting is still treated through generated reward functions. Given the potential of the method, it would be good to see how it behaves in a non-generated setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new algorithm to solve Inverse Reinforcement Learning tasks in the setting where we have demonstrations as well as full knowledge of the reward functions for some training tasks and only a few expert demonstrations for the target task. Within the Bayesian framework, it uses a Conditional Kernel Density approach to estimate the likelihood function and integrate information from the training tasks. The paper also proves asymptotic consistency of the approach used. Overall the methods presented are novel and the results are promising, especially in an environment where the state-action space is of high dimension.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "These proposed adjustments would strengthen the work:\n- Add further study of the application of the method in real-world and non-generated settings - see Weaknesses section above\n- Allocate more time studying rate of contraction and uncertainty reduction as a function of the number of demonstration\n- Add more explanation about each step of the application of KD-BIRL. Some parts are particularly dense and could benefit from more clarification and formalization (e.g. sections 6.3 and 6.4)\n- Typo in p8: should be $\\sigma_0^2$ instead of $\\sigma_0$", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a kernel-based method for Bayesian inverse reinforcement learning called KD-BIRL. The key idea in the method is to reuse pairs of reward functions and demonstrations from similar tasks to estimate the likelihood of demonstrations in a new task with an unknown reward function. This in turn allows computing the posterior distribution of the reward function in the new task. KD-BIRL is both analyzed and empirically evaluated.

Most comments of the reviewers were minor (clarify terminology, clarify notation, state limitations) and the authors addressed them in the revised manuscript. However, I checked and the revised manuscript does not seem to be submitted. Therefore, I accept this paper with a minor revision, and will make the final decision after I see the changes.

Two more things:

* One comment of the reviewers was that the solved problems are not large scale and realistic by today's standards. I share this sentiment. However, I also believe that the evaluation is sufficient to support the claims of the paper. To address this, I suggest that the authors downplay the large-scale and realistic claims.

* I also looked at the paper. When you introduce kernel-based approximations in (3) and (4), make clear that these are not proper distributions or densities because they have not been normalized.
Decision: Accept (poster)