A Bayesian Approach to Robust Inverse Reinforcement Learning
Authors: Ran Wei
Siliang Zeng
Chenliang Li
Alfredo Garcia
Anthony D McDonald
Mingyi Hong
Venue: CoRL 2023 Poster (2023)
Keywords: Inverse Reinforcement Learning
Bayesian Inference
Robustness

Abstract: We consider a Bayesian approach to offline model-based inverse reinforcement learning (IRL). The proposed framework differs from existing offline model-based IRL approaches by performing simultaneous estimation of the expert's reward function and subjective model of environment dynamics. We make use of a class of prior distributions which parameterizes how accurate the expert’s model of the environment is to develop efficient algorithms to estimate the expert's reward and subjective dynamics in high-dimensional settings. Our analysis reveals a novel insight that the estimated policy exhibits robust performance when the expert is believed (a priori) to have a highly accurate model of the environment. We verify this observation in the MuJoCo environments and show that our algorithms outperform state-of-the-art offline IRL algorithms.

A Bayesian Approach to Robust Inverse Reinforcement Learning
Ran Wei', Siliang Zeng", Chenliang Li', Alfredo Garcia', Anthony McDonald', Mingyi Hong 'eexas A&M University, UUniversity of Minnesota, 'University of Wisconsin {rw422, chenliangli, alfredo. aeroogarciiaeen Stamu. {zeng0176, mhong)@umn. lum,,,, {admcdonald lrtriioonndddiiic.eeeeee @0visc.edu
Abstract: We consider a Bayesian approach to offline model-based inverse rein- forcement learning (IRL). The proposed framework differs from existing offline model-based IRL approaches by performing simultaneous estimation of the ex- pert's reward function and subjective model of environment dynamics. We make use of class of prior distributions which parameterizes how accurate the expert's s model of the environment is to develop efficient algorithms to estimate the expert's s reward and subjective dynamics high-dimensional settings. Our analysis reveals a novel insight that the estimated policy exhibits robust performance when the expert believed (a priori) to have a highly accurate model the environment. We verify this observation in the MuJoCo environments and show that our algorithms outperform state-of-the-an aft offline IRL algorithms."
Keywords: Inverse Reinforcement Learning, Bayesian Inference, Robustness
1
Introduction
Inverse reinforcement learning (IRL) is the problem extracting the reward function and policy of value- maximizian agent from its behavior 2). IRL is important tool in domains where manually specifying reward functions or policies is difficult, such as in autonomous driving [3], when the extracted reward function can reveal novel insights about a target population and be used to device interventions, such as in biology, economics, and human-robot interaction studies [4, 5, 6]. However, wider applications of IRL face two interrelated algorithmic challenges: 1) having access to the target deployment environment or an accurate simulator thereof and 2) robustness of the learned policy and reward function due the covariate shift between the training and deployment environments or datasets [7, 8, 9].
In this paper, we focus on model-based offline IRL to address challenge 1). A notable class of model-based offline IRL methods estimate the dynamics and reward in a two-stage fashion (see Figure 1) [10, 11, 12, 13]. In the first stage, dynamics model is estimated from the offline dataset. Then, the dynamics model is fixed and used as a simulator to train the reward and policy in the second stage. To overcome covariate shift in the estimated dynamics, recent methods design density estimation-basee "pessimistic" penalties to prevent the learner policy from entering uncertainty regions in the state-action space (i.e., space not covered the demonstration dataset) [14, 13, 12].
2-Stage: maxp Ev (log ((a'(e,))) a)), maxy Ep/lor <(ae;; B, P*)) Simultaneous: max,, Bo flog fi(a) R, P) Alog ( []]
Likeiheed
Figure 1: Objectives of the traditional two-stage IRL and the proposed simultaneous estimation ap- proach of Bayesian model-based IRL.
We instead approach IRL from Bayesian modeling perspective, where we simultaneously estimate the expert's 's reward function and their internal model of the environment dynamics. The core idea is that expert decisions convey their beliefs about the environment [15, 16] and thus should affect the update direction of the dynamics model. This Bayesian model of perception and action, related to
7th Conference Robot Learning 2023), Atlanta, USA.

---

Bayesian Theory of Mind [15], has been used to understand human biases encoded in the internal dynamics simple and highly constrained domains [17, 16, 18, 19, 20, 21, 22, 23, 24]. In contrast to these works, we study how a Bayesian model enables learning high- gegh-performanc and robust policies.
We first propose a class of priors parameterizing how accurate believe the expert's model of the environment is. We then present key observation that if the expert is believed a priori to have a highly accurate model, robustness emerges naturally in the Bayesian modeling framework by simultaneous estimation approach in which planning is performed against the worst-case dynamics outside the offiin data distribution. We then analyze how varying the prior affects the performance of the learner agent and pair our analysis with set of algorithms which extend previous simultancous estimation approaches [18, 19] nigh-dimensional continuous-contir settings. We show that the proposed algorithms outperform state-of-the-an (SOTA) offine IRL methods in the MuJoCo environments without the need for designing pessimistic penalties.
2
Preliminaries
2.1 Markoy Decision Process
We consider modeling agent behavior using infinite-horizon entropy-regularizze Markov decision pro- cesses (MDP: [25]) defined by tuple (S, A, P. R. 71 with state space S, action space A, transition probability distribution P(s'(s,a) a) E (SS), reward function R(s, a) IR, initial state distribution u(so) ASS), and discount factor 7 E (0, 1). We denote the discounted occupancy measure as pp((sa) a) = P. 1 P(s; s, az a)] and the marginal state-action distribution as (8, = (1 - 7 (s,a), We further denote the discounted occupancy measure starting from specific state-action pair (s,a) a) with (8, a|s, a). The agent selects actions from an optimal policy (als) E ((A) that achieves the maximum expected discounted cumulative rewards and policy entropy H(((--I)) Σn T (ā] log π (ā s) in the MDP:
7' (R(s).a)+
max Jp(x) E,,, Pix
(1)
The optimal policy satisfies the following conditions [26]:
T(ass) x exp (Q(s,
Q(s, a) R(s, a) + 7EP(11 [V(s']
(2)
((s) log Σ exp (Q(s,a'))
2.2 Inverse Reinforcement Learning
The majority of contemporary IRL approaches have converged on the Maximum Causal Entropy (MCE) IRL framework, which aims to find a reward function Rg(s, a) with parameters 0 such that the entropy-regularizzed learner policy & has matching state-action feature with the unknown expert policy π [27].
A related formulation casts IRL maximum discounted likelihood (ML) estimation [28, 29, 30], subject the constraint that the policy entropy-regulariied. Given dataset of N expert trajectories (s1.7, ai:r) sampled from the expert policy in environment P
each of length T: 'D { with occupancy measure pD = 17. MI-IRL aims to solve the following optimization problem:
max E D
rd log (o((a.s))
s.t. to = arg max Epg. [Ro(s,)) H(+(-1s) (3)
#ell
where the policy implicitly parameterized by the reward parameters 0.
It can be shown that for sufficiently large T oc, MCE-IRL and ML-IRL equivalent under linear reward parameterization [28, 29], however (3) permits non-linear reward parameterization

---

through the following surrogate optimization problem:
max pp [Ro(s, a)] Epp [Rg(s, s.t. tt(aas) arg max #E11 p [Ro(s,a) + H(((-|s)] (4)
(4) can be efficiently solved via alternating training of the learner policy and the reward function, similar to Generative Adversarial Network (GAN)-based) algorithms [31, 32, 33, 34, 35, 36). However, these methods all require access to the ground truth environment dynamics a high quality simulator in order compute or sample from the learner occupancy measure pTp.
2.3 Offline Model-Based IRL & RL
Existing offline model-based IRL algorithms such as [10, 11] adapt (4) using a two-stage process. First, an estimate of the environment dynamics is obtained from the offline dataset, e.g., using maximum likelihood estimation. Then, P is fixed and used in place of P to compute P while optimizing (4). However, this simple replacement incurs gap between (4) and (3) which scales with the dynamics model error and the estimated value [13]. This puts high demand the accuracy of the estimated dynamics.
A related challenge is to prevent the policy from exploiting inaccuracies in the estimated dynamics, which can lead to erroneously high value estimates. This has been extensively studied in both online and offline model-based RI literature [37, 38, 39, 40]. The majority of recent offline model-based RL methods combat model-exploitation via notion of "pessimism", which penalizes the learner policy from visiting states where the model likely be incorrect [37]. These pessimistic penalties often designed based on quantifying uncertainty about transition dynamics through the estimated model [41, 42]. Drawing on these advances, recent offline IRL methods also incorporate pessimistic penalties into their RL subroutine [13, 12, 14]. However, designing pessimistic penalties involves nontrivial decisions to ensure that they can accurately capture -istt--disttibution samples [43].
An alternative approach avoid model-exploitation is to perform policy training against the worst- case dynamics in out- difdrrriibutionn states [44], similar robust MDP [45, 46]. Rigter et al. [47] implemented this idea in the R RAMBO algorithm and showed that it is competitive with pessimistic penalty-basee -bsed approaches while requiring no penalty design and significantly less tuning. We will show that robust MDP corresponds to sub-problem of IRL under the Bayesian formulation.
We list additional related work in Appendix A.
3 A Bayesian Approach to Model-based IRL
We consider model-based IRL under Bayesian framework (BM-IRL), where the observed expert decisions are the results of the unknown reward function Rs, (s, a) and their internal model of the environment dynamics Po. (s'ls,a))
We denote the concatenated parameters with 0 = (01, 02) and condition the policy on 0 as ( s; O) emphasize that the expert configuration is determined by both the reward and dynamics parameters.
Upon observing finite set of expert demonstrations D, BM-IRL aims to compute the posterior distribution (() given a choice of a prior distribution P(0):
N T
P( x ((D()))()) = IIII (..,, ()))
where we have omitted I[ IT P(st.et1ls a.2)) from the likelihood because it does not depend on 0.
We consider class of prior distributions of the form:
log Po, (s.c4aasls aus))
P(0) exp
(6)
3

---

where the prior precision hyperparameter represents how accurate we believe the expert's model of the environment is.
Let [(0) := NT log D)) can be easily verified that
(()) = )'> log # (als; + Alog Poz (s)
In this paper, we consider finding a Maximum A Posteriori (MAP) estimate of the BM-IRI. model by solving the following bi-level optimization problem:
max (()), S.L. # ( s; 0) = arg max of [Ro(s, H(((( #ell
(7)
0
Note that this formulation differs from (3) and the two-stage approach (see Figure 1) because it includes the log likelihood of the internal dynamics in the objective (weighted by A).
should be noted that obtaining the full posterior distribution (or an approximation) is feasible using popular approximate inference methods (e-g., stochastic variational inference Langevin dynamics [48, 49)) and does not significantly alter the proposed estimation principles and algorithms.
3.1 Naive Solution
We start by presenting a naive solution to (7) which can be seen as an extension of the tabular simultaneous reward-dynamies estimation algorithms [18, 19] the nigh-dimensional setting.
Solving (7) requires: 1) computing the optimal policy with respect to 9, and 2) computing the gradient Vo log (a]s; f)) which requires inverting the policy optimization process itself. Both operations can be done exactly in the tabular setting as in prior works but are intractable in high-dimensional settings. We propose to overcome the intractability using sample-based approximation.
In this section, we focus on approximating the gradient of the policy V log {a|ss; 0) and construct- ing surrogate objective similar to (4) to perform simultaneous estimation. We can show that Va log (als;; 0) has the following form (see Appendix B for all proofs and derivations):
V lag 0) VoQo(s, a) (- (Vo (s, a)]
(8)
where VoQa a) = [V, (0() Ve, Qo,Q a)] is the concatenation of reward and dynamics gradients defined as:
V a = E (2,0))))))) Vo, Ro (3, a)
Vo. Ro(s,a) = (2.aaa))))))
Po Po.
(10)
Given (9) and (10) are tractable to compute using sample-based approximation of expectations, we construct the following surrogate objective ((() with the same gradient as the original MAP estimation problem (7):
Z(0) = ((..))-DEE((sd))) (E + -E Po (s)
(11)
where
a) (p.a)a)) [Rg (3, +7 EVo(s, Σ
(12)
Optimizing (11) (subject the same policy constraint) is the as optimizing (7) but tractable.
An interesting consequence of maximizing the first two terms of (11) alone (excluding the prior) is that we both increase the reward and modify the internal dynamics to generate states with higher expected value (EV) upon taking expert actions then following the learner policy â, and we do the opposite when taking learner actions. Intuitively, reward and dynamics play complementary roles in determining the value of actions and thus should be regularized [50, 16. 51]. Otherwise, one cannot disentangle the effect of truly high reward and falsely optimistic dynamics. Our prior (6) alleviates this unidentifiability some extent.

---

3.2 Robust BM-IRL
We now present our main observation that robustness emerges from the BM-IRL formulation under the dynamics accuracy prior (6).
We start by analyzing a discounted, full-trajectory version of the BM-IRL likelihood (7). Note that discounting does not change the optimal solution to (7) under expressive reward and dynamics model class; nor does it require infinite data because we can truncate the summation at T int and obtain nearly the same estimator as with infinite sequence length. Using a result from [13], we decompose the discounted likelihood as follows (see Appendix B.2):
Ep(r)
7f log á (aels);0)
Ep(r)
>'t (Qo(st, Vo(se))
Epp Ro, (st, at)
Ep
Vo(5o) + ppp
R (111)0))) Vo(8') P( Vo
((0)
T1
(13)
where TI corresponds to the value difference under the real and estimated dynamics. We can show that TI is negligible if the estimated dynamics is accurate, e.g. (((s Dx (P(+|s, a)]l P(-ss, a)) S for sufficiently small e, under the expert data distribution, which can achieved by setting a large Then, T1 can be dropped and the discounted likelihood reduces to {()).
(()) highlights the reason why the proposed BM-IRL approach can be robust to limited dataset. It poses the offline IRL problem as maximizing the cumulative reward of expert trajectories in the real environment, and minimizing the cumulative reward generated by the learner in the estimated dynamics with respect to both reward and dynamics. other words, it aims to find performance- matching reward and policy under the worst-case dynamics, which trained adversarially outside the offline data distribution. This connects BM-IRL with the robust MDP approach offline model-based RL [44, 47]. We refer this version of BM-IRL as robust model- IRL (RM-IRL).
3.3 Proposed Algorithms
We now propose two scalable algorithms to find the MAP solution to (7) via the surrogate objective (11). The first algorithm (BM-IRL; 1) applies the naive solution, while the second algorithm (RM-IRL; 2) exploits the observation in section 3.2 to derive a more efficient algorithm for high X.
The estimation problem (7) has an inherently nested structure where, for each update of parameters (the outer problem), we have to solve for the optimal policy & (a|s; 0) (the inner problem). Follow- ing recent ML-IRL approaches [29, 13], we perform the nested optimization using two-timescale stochastic approximation [52, 53], where the inner problem is solved via stochastic gradient updates on faster time scale than the outer problem. For both algorithms, solve the inner problem using Model-Bann -ased Policy Optimization (MBPO; [39]) which uses Soft Accor-Critic (SAC; [26]) in dynamics model ensemble.
BM-IRL: For the BM-IRL outer problem, we estimate the expectations in (11) and (12) via sampling and perform coordinate-ascent optimization. Specifically, for each update step, we first sample of state-action pairs (s, a) D and mini-batch of (fake) actions Gtake f(-Is; 8) and simulate both (x, a) and (s, (rake) forward in the estimated dynamics / to get the real and fake trajectories Tyeal Tiake- We then optimize the reward function first by taking single gradient step to optimize the following objective function:
max ((s.a)-DDof [R, () - E aew B. (1110)) (5.....))))))))))...... Ro, (8, 8:
(14)
Lastly, optimize the dynamics model by taking few gradient steps (a hyperparameter) to optimize the following objective function using on on-policy rollouts branched from mini-batches, of
5

---

expert state- sutte-aciios as in RAMBO [47]:
max
X (aa) [VVo (8, a)] -
[Vo, (8,
82
(15)
+ +2
log Po, (s'|s,
where we have introduced weighting coefficients A and A facilitate tuning the prior precision and dynamics model learning rate. We estimate the dynamics gradient using the REINFORCE method with baseline.
RM-IRL: We adapt the BM-IRI. algorithm slightly for the RM-IRL outer problem, where we only simulate single trajectory for each state in the mini-batch and update the reward using the following objective:
max Ep [Re, (s, a)] - Bpp [Ro a)]
(16)
01
For the dynamics update, set X2 A and drop the first term in (15):
- P (5.0)s.amm.) [FVo, [)) (2 [og Po, (8'1s, aa) (17)
max 82
provide additional details and pseudo code for the proposed algorithms Appendix C.
3.4 Performance Guarantees
In this section, we study how policy and dynamics estimation error affect learner performance in the real environment. Using lemma 4.1 from [54] which decomposes the real environment performance gap between the expert and the learner into their policy and model advantages in the estimated dynamics, we arrive at the follow performance bound (see Appendix B.3): da) log ř [a]ss)] be the policy estimation error and cp a) DKL DP((Ps a)ll((-s)s( a)) be the dynamics estimation error. Let Rmaz = max,,a Ro(s, a)l + log [A). Assuming bounded expert-learner marginal state-action density ratio (x.a)
Theorem 3.1. Let Ex
S C, we have the following (absolute) performance bound for the IRL agent:
(s,0)
2(C + 1)
(p(#)-Jp(p S
1-7 (1 (1-7)²
Ex
22p
(18)
This bound shows that the performance gap between the IRL agent and the expert is linear with respect the discount factor in the policy estimation error and quadratic in the dynamics estimation error. Thus, ensuring small dynamics estimation error is essential and prior simultaneous estimation approaches such as [18] which do not explicitly encourage dynamics accuracy expected to perform well in general.
4 Experiments
We aim to answer the following questions with our experiments: 1) How does the dynamics accuracy prior affect agent behavior? 2) How well does BM-IRL and RM-IRL perform compared to SOTA offline IRL algorithms? We investigate QI using a Gridworld environment. We investigate Q2 using the standard D4RL dataset on MuJoCo continuous control benchmarks.
4.1 Gridworld Example
To understand the behavior of the BM-IRL algorithm, we use a 5x5 deterministic gridworld envi- ronment where the expert travels from the lower left the upper right corner using a policy with discount factor of y 0.7. We represent the reward function the log probability of the target state: log (ss), where the upper right corner has target probability of 1.

---

We parameterize both reward and dynamics using softmax of logits. Using 100 expert trajectories of length 50, we trained 3 BM-IRI. agents with E {0.001, 0.5, 10). As a comparison, we also trained a two-stage IRL agent whose dynamics model was fixed after an initial maximum likelihood pretraining step and its reward was estimated using the same gradient update rule as BM-IRL in (14).
The ground truth and estimated target state probabilities are shown in the first row of Figure 2. All agents correctly estimated that the upper right corner has the highest reward, although not with the same precision as the ground truth sparse reward. BM-IRL agents with À = 0.5 and À = 10 were able to assign high reward only to states close to the true goal state, where as the BM-IRI. agent with 0.001 and the two-stage IRL agent assigned high rewards to state much further away from the true goal state.
Ground
stage
(A=0.001)
BM-IRL
(A=10)
0:0
0:03
0:30
3.50
3.00
0:31
0.55
6:01
1139
9:50
0.29
9:11
0.00
[1:3
0:00
E:30
6:41
0.50
3.00
E.02
0:25
4:4
0033
9.50
083
2:2
000
0.30
0.20
2.00
0.00
9:1
0992
0.99
9:00
6:32
0.00
3.00
0:00
0.92
9:00
5:00
0:00
0892
0.82
8.00
0.00
0.20
0.00
8.00
001
9.00
0:11
0.00
3.00
0.93
0:00
5.00
0.00
0.90
0.00
0.00
0.50
1.00
0.00
9:00
0.01
0.03
0.00
5.00
0.00
3.00
0.00
0.02
0:00
5.00
0.00
0.02
0.85
Start
Endl
Figure 2: Gridworld experiment results. Ground truth and estimated target state distributions (softmax of reward; Row 1) and sample path of estimated policy in estimated dynamics (Row 2) for two-stage and BM-IRL agents with À = [0.001, 0.5, 10). BM-IRL agents with higher À obtain more accurate reward estimates and commit fewer illegal transitions.
We visualize the estimated dynamics models by sampling 100 imagined rollouts using the estimated policies in the second row of Figure 2. This figure shows that the BM-IRL(A = 0.001) agent, which corresponds roughly to Herman et al. 's simultaneous estimation algorithm where no (or a very weak) prior is used [18], and the two-stage IRL agent would take significantly more illegal transitions (i.e., diagonal transitions) than BM-IRL agent with higher A. Comparing among BM-IRL agents, we see that increasing À decreases the number of illegal transitions. contrast to the two-stage IRL. agent whose illegal transitions are rather random, the illegal transitions generated by BM-IRL agents with lower À have a strong tendency to point towards the goal state. This corroborates with our analysis that BM-IRL learns optimistic dynamics under the expert distribution.
4.2 MuJoCo Benchmarks
We study the performance of the proposed algorithms in the MuJoCo continuous control benchmark [55] provided by the D4RL. dataset [56]. Following prior IRL evaluation protocols, our IRL agents maintain two datasets: 1) a transition dataset is used to trained the dynamics model and the actor- critic networks and 2) an expert dataset is used to train the reward function. The transition dataset is selected from one of the following: medium, medium-replay, and medium-expert, where medium- expert has the highest quality with the most complete coverage of expert trajectories. We fill the expert dataset with 10 randomly sampled D4RL expert trajectories. For each evaluation environment (HalfCheetah, Hopper, Walker2D) and transition dataset, we train algorithms offline for fixed number of epochs and repeat this process for 5 random seeds. After the final epoch, we evaluate the agent for 10 episodes in the corresponding environment. For both BM-IRL and RM-IRL, we set A1 = 0.01, 222 = to encourage an accurate dynamics model under the offline data distribution. We provide additional implementation and experimental details in Appendix D.
For the baseline algorithms, we choose Behavior Cloning (BC) and ValueDICE [57], both of which model-free offline imitation learning algorithms and they do not use the transition dataset. For
7

---

Hopper
1.50
A,00
1.50
1.50
1.25
1.25
1.25
3,60
1.00
1,00
Nonmaliee
0.75
0.75
0.75
0.50
0.50
0.50
0.25
mod esplay moctert-copprt
medium
un-cspprt
mcdium
indumnespy med um-expert
Figure 3: MuJoCo benchmark performance using 10 expert trajectories from the D4RL dataset. Bar heights and error bars represent the means and standard deviations of normalized scores, respectively, over 5 random seeds. Baseline algorithm performances are taken from [13].
SOTA offline model-based IRI. algorithms, we compare with the following: 1) ML-IRL uses a similar two-timescale algorithm but operates in a two-stage fashion with a pessimistic penalty which adapts to the learner policy's is state-action visitation [13], 2) CLARE performs marginal density matching with a mixture of expert and transition datasets weighted by the in-distributionness of non-expert state-action pairs [12]. We expect algorithms outperform CLARE whose marginal density matching method requires larger expert dataset to work well shown in [I3, 12]. We also expect to perform similarly to ML-IRL, but better when adversarial model training is more appropriate than ensemble-basee pessimistic penalty for a given dataset. Lastly, we expect BM-IRL to perform better than RM-IRL since BM-IRL solves the simultaneous estimation problem exactly.
Figure 3 shows the results of algorithms and the comparisons reported in [13]. In the Hopper environment, all algorithms performed similarly all datasets, except for BM-IRL and ML-IRL having lower performance with much larger variance on the medium dataset. In the HalfCheetah and Walker2D environments, our algorithms and ML-IRL performed significantly better. BM-IRL and RM-IRL outperformed other algorithms in 6/9 settings, and in 2/9 settings, the differences from the best algorithm were very small. The only case where the best algorithm, ML-IRL, significantly outperformed our algorithms was HalfCheetach medium-rees replay. This is mostly likely because in this environment, the medium-replay dataset has much less coverage on expert trajectories so that adversarial model training might have been too conservative.
As we expected, BM-IRL outperformed RM-IRL in 7/9 settings. The only case where its mean performance was significantly lower was Hopper medium where BM-IRL performance had large variance. However, as we explain the section this is because BM-IRL has higher training instability where its peak performance was in fact on par with RM-IRL.
Limitations
limitation of the proposed algorithms is that BM-IRL can have less stable training dynamics than RM-IRL where its evaluation performance may alternate between periods of near optimal performance and periods of medium performance (thus the larger variance in Figure 3). While stability is known issue for training enssey-based models using contrastive divergence objectives (i.e.. objective (11)) [58], we believe the current issue is related to BM-IRL's two-sample path method having weaker and noisier learning signal. Another source of instability likely introduced by simultaneously training the dynamics model, which may be improved in future work by adding Lipschitz regularizations [59].
Conclusion
We showed that inverse reinforcement learning under Bayesian simultancous estimation framework gives rise robust policies. This yielded set of novel offline model-based IRL algorithms achieving SOTA performance in the MuJoCo continuous control benchmarks without ad hoc pessimistic penalty design. An interesting future direction is to identify appropriate priors robustly infer reward and internal dynamics from sub-optimal and biased human demonstrators.

---

Acknowledgments
A. Garcia would like to acknowledge partial support from the Army Research Office grant W9. 22-1-0213. M. Hong and S. Zeng are supported by NSF grant CIF- The authors would also like to acknowledge Marc Rigter for answering questions about the RAMBO algorithm.
References
[1] A. Y. Ng. S. Russell, et al. Algorithms for inverse reinforcement learning, In leml, volume 1,
page 2, 2000.
[2] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 7(1-2):1-179, 2018.
[3] T. Phan-Minh, F. Howington, T.-S. Chu, U. Lee, M. S. Tomov, N. Li, C. Dicle, S. Findler, Suarez-Ruiz, R. Beaudoin, et al. Driving real life with inverse reinforcement learning. arXiv preprint arXiv:2206. 2000000000000 2022.
[4] S. Yamaguchi, H. Naoki, M. Ikeda, Y. Tsukada, Nakano, I. Mori, and Ishii. Identification of animal behavioral strategies by inverse reinforcement learning. PLoS compulational biology, 14(5)xe000612) 2018.
[5] J. Rust. Optimal replacement of gme bus engines: An empirical model of harold zurcher. Econometrica: Journal of the Econometric Society, pages 999-1033, 1987.
[6] D. Sadigh, N. Landolfi, S. S. Sastry, S. A. Seshia, and A. D. Dragan. Planning for cars that coordinate with people: leveraging effects on human actions for planning and active information gathering over human internal state. Autonomous Robots, 42: 1114-115226 2018.
[7] S. Ross and D. Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 661-668. JMLR Workshop and Conference Proceedings, 2010.
[8] Spencer, S. Choudhury, A. Venkatraman, B. Ziebart, and A. Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint arXiv:2102.02872, 2021.
[9] A. Kuefler, Morton, T. Wheeler, and M. Kochenderfer. Imitating driver behavior with generative adversarial networks. In 2017 IEEE Intelligent Vehicles Symposium (IV), pages 204-211. IEEE, 2017.
[10] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 34:3016-3028, 2021.
[11] N. Das, S. Bechtle, T. Davehev, D. Jayaraman, A. Rai, and F. Meier. Model-based inverse reinforcement learning from visual demonstrations. arXiv preprint arXiv:2010.09034 2020.
[12] S. Yue, G. Wang, W. Shao, Z. Zhang, S. Lin. J. Ren, and J. Zhang. Clare: Conserva- tive model-based reward learning for offline inverse reinforcement learning. arXiv preprins arXiv:2302.04782. 2023.
[13] Zeng, C. Li, A. Garcia, and M. Hong. Understanding expertise through demonstrations: maximum likelihood framework for offline inverse reinforcement learning. arXiv preprint arXiv:2302.07457. 2023.
[14] Chang, M. Uehara, D. Sreenivas, R. Kidambi, and W. Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965-970 2021.

---

[15] C. Baker, R. Saxe, and J. Tenenbaum. Bayesian theory of mind: Modeling joint belief-desire attribution. In Proceedings of the anmual meeting of the cognitive science society, volume 33, 2011.
[16] S. Reddy, A. Dragan, and S. Levine. Where do you think you're going?: Inferring beliefs about dynamics from behavior. arXiv preprint arXiv:1805.080// 2018.
[17] D. Jarrett, A. Hüyük, and M. Van Der Schaar. Inverse decision modeling: Learning interpretable representations of behavior. In International Conference on Machine Learning, pages 4755- 4771. PMLR, 2021.
[18] M. Herman, T. Gindele, J. Wagner, F. Schmitt, and W. Burgard. Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. In Artificial Intelligence and Statistics, pages 102-110. PMLR, 2016.
[19] Z. Wu, P. Schrater, and X. Pitkow. Inverse rational control: Inferring what you think from how you forage. arXiv preprint 1805. 2018.
[20] T. Makino and J. Takeuchi. Apprenticeship learning for model parameters of partially observable environments. arXiv preprint arXiv: 2012.
[21] F. Schmitt, H.-J Bieg, M. Herman, and C. A. Rothkopf. see what you see: Inferring sensor and policy models of human real-world motor behavior. In Thirty-Firss AAAI Conference on Artificial Intelligence, 2017.
[22] Z. Gong and Zhang. What you really want of me? generalized reward learning with biased beliefs about domain dynamics. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 2485-2492, 2020.
[23] L. Chan, D. Hadfield-Menell S. Srinivasa, and A. Dragan. The assistive multi-armed bandit. In 2019 14th ACM/EEEE International Conference on Human-Robot Interaction (HRI), pages 354-363, IEEE, 2019.
[24] N. Desai, A. Critch, and Russell. Negotiable reinforcement learning for pareto optimal sequential decision-making, Advances in Neural Information Processing Systems, 31, 2018.
[25] G. Neu, A. Jonsson, and V. Gómez. A unified view of retroayyrre markov decision processes. arXiv preprint arXiv:1705.07799 2017.
[26] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[27] B. D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
[28] A. Gleave and S. Toyer. A primer maximum causal entropy inverse reinforcement learning. arXiv preprint arXiv:2203. 1400,,
[29] S. Zeng, M. Hong, and A. Garcia. Structural estimation of markov decision processes in high-dimensional state space with finite-time guarantees. arXiv preprint 1rXiv222 2022.
[30] Zeng, C. Li, A. Garcia, and M. Hong. Maximum-likelihodd inverse reinforcement learning with finite-time guarantees. arXiv preprint arXiv:2210.01808. 2022.
[31] S. K. S. Ghasemipour, R. Zemel, and S. Gu. A divergence minimization perspective imitation learning methods. In Conference Robot Learning, pages 1259-1277. PMLR, 2020.
10

---

[32] J. Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[33] L. Ke, S. Choudhury, M. Barnes, W. Sun, G. Lee, and S. Srinivasa. Imitation learning as f-divergence minimization. In International Workshop on the Algorithmic Foundations of Robotics, pages 313-329. Springer, 2021.
[34] C. Finn, P. Christiano, P. Abbeel, and S. Levine. A connection between generative adver- sarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852 2016.
[35] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[36] J. Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprini 1rXiv:1710.1122 1248, 2017.
[37] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01645 2020.
[38] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[39] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-basee policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
[40] T. Jafferjee, E. Imani, E. Talvitie, M. White, and M. Bowling. Hallucinating value: A pitfall of dyna-style planning with imperfect environment models. arXiv preprint arXiv:2006.044. 2020.
[41] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Y. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14129-14142, 2020.
[42] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel: Model-based offline reinforcement learning, Advances in neural information processing systems, 33:21810-21822 2020.
[43] C. Lu, P. Ball, J. Parker-Holder, M. Osborne, and S. J. Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations, 2021.
[44] M. Uehara and W. Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226, 2021.
[45] A. Nilim and L. El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
[46] G. N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005.
[47] M. Rigter, B. Lacerda, and N. Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning, arXiv preprint arXiv:2204.1 12581, 2022.
[48] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 131..61111 2013.
[49] M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-11), pages 681-688, 2011.
11

---

[50] S. Armstrong and S. Mindermann. Occam's 's razor is insufficient to infer the preferences of irrational agents. Advances in neural information processing systems, 31, 2018.
[51] R. Shah, N. Gundotra, P. Abbeel, and A. Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference Machine Learning, pages 1670-5679. PMLR, 2019.
[52] V. S. Borkar. Stochastic approximation with two time scales. Systems & Control Letters, 29(5): 291-294, 1997.
[53] M. Hong. H.-T. Wai, Z. Wang, and Z. Yang. A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic. arXiv preprint arXiv:2007.05//0 2020.
[54] A. Vemula, Y. Song, A. Singh, J. A. Bagnell, and S. Choudhury. The virtues of laziness in model-based rl: A unified objective and algorithms. arXiv preprint arXiv:2303.008. 00694, 2023.
[55] E. Todorov, T. Erez, and Y. Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033, IEEE, 2012.
[56] Fu, A. Kumar, O. Nachum, Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0722 2020.
[57] I. Kostrikov, O. Nachum, and Tompson. Imitation learning via off- off-policy distribution matching. arXiv preprint arXiv: 191.00092..00032 2019.
[58] Y. Du, S. Li, J. Tenenbaum, and I. Mordatch. Improved contrastive divergence training of energy based models. arXiv preprint arXiv:2012.01316. 2020.
[59] K. Asadi, D. Misra, and M. Littman. Lipschitz continuity in modell-ased reinforcement learning. In International Conference Machine Learning, pages 264-273. PMLR, 2018
[60] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. IJCAI, volume 7, pages 2586-2591, 2007.
[61] J. Choi and K.-F. Kim. Map inference for bayesian inverse reinforcement learning. Advances in neural information processing systems, 24, 2011.
[62] A. Chan and M. van der Schaar. Scalable bayesian inverse reinforcement learning. arXiv preprint arXiv:2102.06483 2021.
[63] M. Kwon, S. Daptardar, P. R. Schrater, and X. Pitkow. Inverse rational control with partially observable continuous nonlinear dynamics. Advances in neural information processing systems, 33:7898-79099 2020.
[64] N. Lambert, B. Amos, O. Yadan, and R. Calandra. Objective mismatch in model-based reinforcement learning. arXiv preprint vxXiv:2002.04523 2020.
[65] C. Grimm, A. Barreto, S. Singh, and D. Silver. The value equivalence principle for model-based reinforcement learning. Advances in Neural Information Processing Systems, 33:5541-5552, 2020.
[66] A.-m. Farahmand, A. Barreto, and D. Nikovski. Value-aware loss function for model-based reinforcement learning. In Artificial Intelligence and Statistics, pages 1486-1494. PMLR, 2017.
[67] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05000 2018.
12

---

Supplemental Materials for: A Bayesian Approach to Robust Inverse Reinforcement Learning
A
Additional Related Work
Bayesian IRL; Ramachandran and Amir [60] first proposed a Bayesian formulation of IRL to solve the reward ambiguity problem. A MAP inference approach was proposed [61] and a variational inference approach was proposed in [62]. Their formulations considers con-entropy-regularize policies and the dynamics model is fixed during reward inference. contrast, simultaneous estimation of reward and dynamics can potentially infer the demonstrator's biased beliefs about the environment, which is desirable for psychology and human-robot interaction studies [15, 19, 16]. Despite the attractiveness, simultaneous estimation is challenging because of the need to invert the agent's planning process, especially in continuous domains. Reddy et al. [16] avoids this by representing agent discrete action policies using neural paramo--parmmeterizzed Q functions and regularizing the Bellman enror to be small over the entire acate-action space. This method however cannot be straightforwardly adapted to the continuous action case. Kwon al. [63] avoids this by first training task-conditioned policy on distribution of environments with known parameters using meta reinforcement learning and then the meta-trained policy to guide parameter inference. This precludes the method from being used in general settings with unknown task distributions. To our knowledge, our proposed algorithms are the first to address simultaneous estimation in general environments.
Decision-aware model learning: Decision-aware model learning aims to solve the objective mis- match problem in model-based RL [64]. Many proposed methods in this class value-targeted regression similar to model loss in (15) [65, 66). Our analysis and that of Vemula et al. [54] suggest that value-targeted model objectives may be related to robust objectives. Furthermore, since the set of value-equivalent models only shrink for increasingly larger set of policy and values [65], using value-aware model objective alone may not be optimal and additional prediction based regularizations may be needed.
B
Missing Proofs
B.1 Proofs For Section 3.1
Derivation of BM-IRL Gradients (section 3.1). Recall the definition of the optimal entropy-
regularized policy and value functions:
#(a|s;0) = exp(Qo(s,a))
Σ. cxxp((o(ssa)
Q0(8, a) = Ro, (8, + (1. Vo(8')
(o(')))
(19)
Vo(s) log Σ exp((Q(s.a))
The gradient of the policy log likelihood terms of the Q function gradient is obtained as follows:
Ve log ())s: VeQo(s. a) voVo(s)
voQa(s.a) Zo Vo exp(Qo(s.))
Σ
(20)
VoQo(s, Zo Σ exp(VoQo(s.dd)
VaQo(s, Ba (s:0) VeQo(s.
where Za E.. exp(Qo(s,a')) is the normalizer.
13

---

Recall (3, al,, a.) is the discounted state-action occupancy measure starting from pair (s.a). We define for any function f(s,a):
00
p3 (((. = -B
Σ
f((
80 8, ao = a
(21)
We now derive Q function gradients with respect to the reward parameters 8 and dynamics parameters 02, respectively.
Vo. a)(s,))= = Vo, Ro, (s. + (-laa))) Vo, Vo(s')]
= -o, (3, a) + (1..)) (1a
Vo, Ve, Qo(s', a')]
Ro, (s. و
(22)
Ro, (s', 3 3r (-20 (-lo
Qo(s,, "asss,a"))
Re, (8,0) + E ( (P,#)
Ve, Ro, (sh, an)
80 8, a0
(2.aaa))) Vo, Rg, ($, a)
In line two we used the result that V (o)) for both Φ = 01 and Φ = 02 corresponds to the second term (20).
Va, (o(,0)) = Ve, Ho (s, a) Ve (1). [V((s')
= Σ
Vo( Po a) Po00
($|8, 7 (1l.)))
7 Σ Vo. (3 E 5 (1 V a")]]
Ve,
= r Σ Po (3|s, (P)A))
y h+l Vo (3) 8, ag
Σ Vo(s') (s'18,))
(23)
We make a quick remark on the identifiability simultaneous estimation.
Remark B.1. Simultaneous reward- dynamics estimation of the form (5) without specific assumptions on the prior P(O) is in general unidentifiable.
|SSIA and V R|S be a set of Bellman-consistent reward, dynamics, and value functions in matrix form. Let P & P be an alternative dynamics model. We can always find an alternative reward R' = + AR, where:
Proof. Let R E RIS|A| and P E [R X|S
P.P =
AR = (Q Q) (P'V -V)
(24)
APV
without changing the value functions and optimal entropy-regulaaize policy.
14

---

Remark B.1 implies that existing simultaneous estimation approaches which do not use explicit or implicit regularizations, such as the SERD algorithm by [18), cannot in general accurately estimate expert reward. Paired with theorem 3.1, it shows that these algorithms cannot in general achieve good performance.
B.2 Proofs For Section 3.2
Derivation of discounted likelihood (13).
Ep(r)
Y log (aclss::))
-E((r)
7' -Vo(s)))
=E((r)
Ro (se, ar) + +,B, BB Pos (1 [o(('))
Er(r)
Σ
Vo(si)
1=0
7¹ Ro, (st) aa)
-En Vo((o))
EP(r))
+ Br(()
1+1E B Mo
(r(r)
7 Vo(st)
1=1
=Ep(r)
Σ
7' Roo, (5c, an)
ai)
Ex Vo(su)
+ Ep(r)
1 Pat- ( [Vo(s')]
E (P()
1=
Epp
Ro. (z)
Ey
(o()
+yEp
E Pa Vols') E." Ber
((0)
n1
(25)
The following lemma shows that negligible the estimated dynamics is accurate under the expert distribution, which is available from the offline dataset.
Lemma B.2. Let e PKL a)ll ((-ls,a) and Rmaz max,,a (Ro(s,)) log [A) it holds that
+
RBmax V2c
TI <
(26)
(1 7)2
Proof.
Σ Vo(s') ( (t)) P(s' ar)
T
((o.a))P(F)
[+] ((
(o((s)) P(ssiss, P(s p(s'|se ax)
11
7 (ollVo))llloBBooaaaae ( P(11
[oo(-)lboo22
15

---

where (I) follows from Jensen's inequality, (2) follows from Holder's inequality, and (3) follows from Pinsker's inequality.
Finally, given ((((--s)) E. T(als) S E. TTals) [A]) 1-7
log (Al, we have IAl 1
looc S D 7" (maxs,a ()) log
B.3 Proofs For Section 3.4
We first restate a slight modification of the result from [54], which decomposes the real environment performance gap between the expert and the learner into their policy and model advantages in the estimated dynamics:
Lemma B.3. (Performance difference via advantage in model; Lemma 4.1 in [54]) Let dā denote the marginal state-action distribution fiteributionfolloiinn policy π in environment P. The following relationship holds:
(s,,))yd [log & p(als)] = Eswdy [Ba (p(0, V
(27)
- B
(28)
Performance difference real enviroument
(a.0)) [E E (s")
(29)
Model disndvoonage under learner distribution
+ - VV B
(30)
Model advantage under expert distribution
The performance bound in theorem 3.1 can be obtained from lemma B.3 as follow:
Theorem B.4. (Restate of theorem 3. Let - (s [log 7 ((a]s)) be the policy estimation
error and p = a) (Prl, a)ll (--ss,a)))) be the dynamics estimation error. Let Rynar = Ra (s, a)l + log [A, Assuming bounded expert-learns marginal state-action density ratio
max,,a
(3.0))
S C. we have the following (absolute) performance bound for the IRL agent:
(3,0)
1 7(C + 1m
(Jr(f) - Jr(n))
-r(()))r((T] S 1-7 (1 7)2
c+
2
(31)
Proof.
[Jr(A) Jr(T))
Ean - VB
Ex
1
1-7
ds (Brr -Ba())))Emma/ya E
?
+
E(a,a)dF
1
d)
+
E(
E V8') V
1
(32)
7
(..)))
|| [(..a)ddB P(-1s, a) P(- -PP),,,,)
1
(.)))))
||
B(s.nymaz
P((1s, - -(-
1
7(C + 1)
Ex +
/2Ep
7)²
where the last line uses results from lemma B.2.
16

---

C Further Algorithm Details and Pseudo Code
We estimate the dynamics gradient in (15) and (17) using the REINFORCE method with baseline:
VV, Σ Vo(s') (5'1s,))
((lsa) (Vo(()) b(s,a)) Vo, log (s' (s
Following Rigter et al. [47], set the baseline b(s,a) Qo(s, Rs (s, a) reduce gradient variance and further normalize Vo(s') b(s, a) across the maiii-btchh to stabilize training. In the continuous-contre setting, the value function can be estimated Vo(s) Ea-s, QQ a) log [(aas; s; 0)] with single sample. We apply this gradient for fixed number of steps for dynamics model training, which a hyperparameter.
Pseudo code for the proposed algorithms are listed in Algorithm and Algorithm 2.
Algorithm 1 Bayesian Model-based IRL (BM-IRL)
Require: Dataset = [T), dynamics model Poj (8'18, a), reward model Re, (8, a). hyperparameters
\1. A2
1: for k = 1 K do
2: Run MBPO to update learner policy [a]ss; 0) and value function Qo a) in dynamics P
3: Sample real trajectory Treal starting from (s, a) 'D and following P and
4: Sample fake trajectory Tlake starting from 8 ~ DD Ojake ~ f-|ss; f) and following P and à
5: Sample (s, a, 3 D for dynamics model training
6: Evaluate (14) and take a gradient step
7: Evaluate (15) and take few gradient steps.
8: end for
Algorithm 2 Robust Model-based IRL (RM-IRL)
Require: Dataset D {T), dynamics model Po. (s'ls, a), reward model Re, (s, a), hyperparameters
\1. A2
1: for k == 1: K do
2: Run MBPO to update learner policy (a]ss; s; 0) and value function Qa(s, a) in dynamics P
3: Sample real trajectory Treal ~ D
4:
Sample fake trajectory Trake starting from D and following i and a
5: Sample (s, a, 8') 'D for dynamics model training
6: Evaluate (16) and take gradient step
7: Evaluate (17) and take few gradient steps
8: end for
D Implementation Details
Our implementation' builds on top of the official RAMBO implementation" [47].
D.I MuJoCo Benchmarks
For the MuloCo benchmarks described in section 4.2, follow standard practices in model-based RL.

---

D.1.1 Dynamics Pre-training
We use an ensemble of K neural networks where each network outputs the mean and covariance parameters Gaussian distribution over the difference between the next state and the current state
((k) Bg
(1k)) (8|8, a) (8 (s. a), 2-92 (s,a)) a))
(33)
Each network is 4-layer feedforward network with 200 hidden units and Sigmoid linear unit (SiLU) activation function. For the initial pre-training step, we maximize the likelihood of dataset transitions using batch size of 256 and early stop when all models stop improving for more than percent. We then select the 5 best models in terms of mean-squared-ennet During model rollouts, we randomly pick one of the 5 best models (elites) to sample the next state.
10 % holdout validation set.
Table 1: Shared hyperparameters across different environments
Hyparameter
BM-IRL
RM-IRL
critic learning rate
3e-4
3e-4
actor learning rate
3e-4
3e-4
discount factor (n)
0.99
0.99
SAC + MBPO
soft target update parameter (r)
5e-3
5e-3
target entropy
dim(A)
dim(AA
minimum temperature (ax)
0.1
0.001
batch size
256
256
real ratio
0.5
0.5
model retain epochs
5
5
training epochs steps per epoch # model networks
500
300
1000
1000
# elites
adv. rollout batch size adv. rollout steps adv. update steps adv. loss weighting (A1)
1000
256
Dynamies
10
10
50
50
0.01
0.01
supervised. loss weighting (A2)
learning rate adv. update steps
le-4
le-4
50
50
max reward rollout batch size
1000
64
Reward
rollout steps
40
100
12 penalty
1e-3
le-3
learning rate
1e-4
le-4
update steps
Table 2: Environment--peecfiic hyperparameters
Environment
Hyperparameter
BM-IRL
model rollout batch size
50000
HalfCheetah
model rollout steps
5
model rollout frequency
250
model rollout batch size
10000
Hopper
model rollout steps
40
model rollout frequency
250
model rollout batch size
10000
Walker2d
model rollout steps
40
model rollout frequency
250
18

---

D.1.2 Policy Training
Our policy training process follows MBPO [39] which uses SAC with automatic temperature tuning [67]. Shared hyperparameters across different environments listed in Table and environment- specific hyperparameters are listed in Table 2. For the actor and critic, we use feedforward neural networks with 2 hidden layers of 256 units and ReLU activation. We train the actor and critic networks using a combination of real and simulated samples. We use real ratio of 0.5, which is standard practice in model-based RL and IRL. We found that BM-IRL requires a higher minimum 0.1.
temperature to stablize training, which is set
We found that different MuJoCo environments require different model rollout hyperparameters, simi- lar to what's reported in [43]. Specifically, Hopper and Walker2d only work with significantly larger rollout steps. We decrease their rollout batch size to reduce computational overhead. HalfCheetah on the other hand works better with smaller rollout steps and larger rollout batch size. contrast to Lu al. [43], we did not use different rollout hyperparameters for different datasets.
D.1.3 Reward and Dynamics Training
We use 10 random trajectories from the D4RL MuJoCo expert dataset after removing all expert trajectories that resulted in terminal states.
We use the same network architecture as the actor-critic to parameterize the reward function. We further clip the reward function to a maximum range of 10 and apply 12 regularization on all weights with penalty of 0.001.
As described in the main text, update the reward function by simulating sample trajectories and taking single gradient step. For RM-IRL, randomly sample expert trajectory segments of length "rollout steps" and use the first step as the start of our simulated sample paths.
We update the dynamics using on-policy rollouts branched from the dataset aate-actions. We use the same batch size for reward and dynamics rollouts, which 1000 for BM-IRL and 256 for RM-IRL. Because only the first step in BM-IRL's real sample paths come from the dataset, it requires larger batch size to iterate more data samples. We also train BM-IRL for more epochs than RM-IRL.
To compute the dynamics log likelihood in the REINFORCE gradient in (33), we treat the ensemble as a uniform mixture and compute the likelihood as:
K
Pe. (b) =
1 K
p(k) 82
(8]s,a) a)
(34)
We set the dynamics adversarial loss weighting to A = 0.01 for both BM-IRL and RM-IRL. found this to work better than what's in the official RAMBO implementation, which is X₁ 0.0768. Note that the RAMBO author reported A 3e-4 their paper but forget to average their REINFORCE loss the mini-batch of size 256 in their implementation, which is instead treated default by TensorFlow. We empirically found that small X leads to severe model exploitation.
by
19


Reviews:
[{"review_id": "SesxWOm1ty", "reviewer": "Reviewer_agyf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an offline inverse reinforcement learning (IRL) framework in which both the reward and transition functions are learned simultaneously from observations in an offline manner. The proposed framework is built on the Bayesian Theory of Mind perspective under which expert demonstrations implicitly convey their beliefs about the environment dynamics. Using this idea, the authors propose a class of priors that encode how accurate the learning agent believes the expert model of the environment is and later show how variations in this prior affect the learner agent's performance. Two solutions (naive and robot) to this new formulation are also proposed and evaluated in simulation.\n", "summary_of_recommendation": "I think the core idea is interesting and applicable to varied domains such as robotics and psychology. However, I would like to see a more thorough analysis and/or discussion of how BTOM and RTOM performance changes when there is a discrepancy between the learning agent's belief and the accuracy of the expert's internal dynamics model.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Although the evaluation does a good job of showcasing the performance and advantages of the proposed framework, there are several points for which additional information or details are missing:\n\n1. Figure 1 shows that although agents with higher confidence in the accuracy of the expert's model result in better dynamics estimates, illegal transitions are still present. What are the implications of these results for reward and policy learning? Also, does it mean that a higher lambda will help to mitigate these illegal transitions?\n2. What lambda values were used for the MuJoCO benchmark and how should this value be chosen in practice?\n3. From Table 1, why does ML-IRL achieve better results for the medium replay dataset in the HalfCheetah and Walker2D?\n4. Table 1 shows the quality of the demonstrations can affect the performance of BTOM and RTOM. A more detailed analysis that shows the trade-offs and correlations between the choice of lambda and the coverage of expert demonstrations should be included and discussed in detail. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses an important challenge in IRL: how to mitigate compounding errors in the traditional two-step approaches where dynamic estimates are obtained prior to reward and policy learning. The paper is well-written and the idea of considering IRL under the Bayesian Theory of Mind framework is well-motivated and interesting.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "RafbjQ11dP", "reviewer": "Reviewer_oFYH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes an approach to inverse reinforcement learning that accounts for the theory of mind of the demonstrator. In particular, this accounts for the demonstrator's subjective (and possibly incorrect) estimate of the dynamics. The authors analyze IRL in this framing and show how it has nice connections to robust learning from demonstrations. They provide experiments in a gridworld and on Mujoco benchmarks that demonstrate the approach.", "summary_of_recommendation": "I'm somewhat torn about this paper. I think the idea is quite promising, but the presentation and experiments have substantial room for improvement. With the paper as it is, I weakly recommend rejection. However, I think there are several points that can be addressed that improve my opinion of the paper. \n\nUpdate after discussion/rebuttal: I’m happy with the authors response and changes. I still think that there is substantial room for improvement on the empirical work, but I value the combination of theoretical results and empirical validation. I’ve increased my score to recommend acceptance.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "### Can you add a more complex robotics evaluation domain?\n\nThis would address my primary concern about the paper. I think the method is quite promising but I think that insufficient evaluation will reduce the paper's impact.\n\n### Can you provide a more clear description of the method and baseline?\n\nI think that more guidance for practitioners to implement the method will help the paper substantially. Similarly, in order to make the work more reproducible, it will help to have more descriptions of the baseline method for the experiment.\n\n### Can you describe your changes to improve the clarity of the presentation?\n\nPlease mention the specific points I raised and any other changes you plan to make.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths\n\n## Quality\n\nThe theoretical results in the paper are valuable contributions. I really appreciate the connection to robustness that they identify. I also appreciate the discussion of performance guarantees.\n\n## Originality\n\nThe key idea in this paper: doing IRL in a way that estimates the expert's subjective beliefs is novel and promising.\n\n## Clarity\n\nThe paper does a good job of setting up and proving the theoretical results.\n\n## Significance\n\nDepending on how useful these methods are in practical situations, this paper has the potential for high impact. It leverages a nice insight into IRL, supports that with solid theory, and demonstrates some initial empirical results. \n\n# Weaknesses\n\n## Quality\n\nThe experiments in this paper are not particularly strong. The results are primarily a comparison of two different ToM approaches, with less comparison of ToM approaches to other approaches. It is possible I have missed something with the experiments, but it was hard for me to identify the benefits of ToM in comparison to other IRL approaches (despite the claim in l.209). In either case, comparing to more baseline methods (clearly grounded in the related literature) would help. \n\nBeyond this issue, the experiments are not super convincing as they are in simulation for a gridworld and Mujoco. I'm not sure how easy it would be to fix this, but a complex evaluation domain would help the paper. \n\nFinally, a few related work pieces to include: https://arxiv.org/abs/1901.08654 estimates reward where the observed behavior has a subjective estimate of reward; https://dl.acm.org/doi/10.5555/3327345.3327381 deals with negotiating agents who have different dynamics estimates\n\n## Clarity\n\nIt wasn't very clear how one could replicate the experiments or methods. Consider including pseudocode for the method and more details about, e.g., the baseline method. Furthermore, I found the analysis of the results (lines 254-258) incomplete. I recommend adding some more discussion of how and why performance differs across the conditions.\n\nA few specific lines I found unclear:\n  - l.116-117] Please say more about the distinction with (3); \n  - [l.197] Please add more English description of the implications of the theorem, I had trouble parsing the math into an intuitive understanding", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Q43ehctM3wg", "reviewer": "Reviewer_yDmY", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a model-based offline IRL approach based on Bayesian theory of mind where the learner must estimate the demonstrator's reward function and dynamics. The authors show a connection to robust MDPs and demonstrate their algorithm on a grid world and on D4RL benchmarks.", "summary_of_recommendation": "The empirical results show an improvement; however, the general approach is poorly motivated and not studied in the setting most appropriate for BTOM. It is not clear why the approach works better than other approaches nor whether it will work in cases where the demonstrator does not have access to the true dynamics. The authors only compare against one baseline. Furthermore the mathematical details are often hard to follow.\n\n---\nPost rebuttal: see above notes in review. I think the paper needs more convincing results that there is value to the proposed approach as opposed to ML-IRL. Currently the gains seem mostly marginal and given the error bars it's unclear whether they are statistically significant. I maintain my recommendation of weak reject.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Typos:\nLine 61 R(s,a)\nEq (7) Should be \\hat{P}, right?\n\nWhy use the prior in equation (6)? This is based on actual transitions and assumes that a dynamics parameter setting has higher likelihood if the actual transition is high under the model of the expert's dynamics. However, it is common that an expert can have biases and not understand the dynamics. In this case they would take an action expecting a different next state than the one that actual occurs. The current prior in the paper will assign this low likelihood even if this is the expert's true model of the env. \n\n\nHow do you get from (7) to (11)-(12)? Please add more explanation to the text.\n\nLine 152. Why decompose a likelihood from prior work. I thought this would be decomposing Eq (7) from this paper.\n\nIn Equation (14) it is unclear what expectations are taken wrt the demonstrations.\n\nWhy use length 50 trajectories in a 5x5 grid world?\n\nThe related work section mentions several model-based offline IRL algorithms. Why not compare against them? Adding more baselines would strengthen the empirical results.\n\nIsn't BTOM for when the demonstrator doesn't know the true dynamics. Please test in this setting and discuss why the BTOM mind approach is appropriate when the expert has perfect knowledge of the true dynamics.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n+ Empirical results show that the BTOM and RTOM approaches outperform an ML-IRL baseline.\n+ Authors derive an approach that scales to high-dimensional tasks.\n\nWeaknesses:\n- I do not understand why the authors use a BTOM approach. There is no motivation for this approach and it does not seem to match the experimental domains.\n\n- The authors do not assume that the expert has access to the true parameters of the environment, but all experiments assume this.\n- Some of the derivations are hard to follow or not explained.\n\n- The approach seems identical to other approaches except for the addition of the prior. However, as noted later the prior seems hard to justify in the setting of BTOM.\n\n---\nPost rebuttal: \nI think the paper is stronger, and I think it is better framed by not using TOM wording (unless the authors were to address cases where the expert does not have a good model of the transition dynamics). However, I still do not see a significant win over ML-IRL and as such I'm not sure when this method would be chosen over existing methods. I think the paper needs to more clearly show why/when a Bayesian approach is better. Some ideas are to add experiments showing where ML-IRL has a significant drop in performance but the proposed Bayesian approach maintains high performance because of the robustness property. Other ideas include showing that the proposed approach is more data or computationally efficient. Prior work has shown that reward learning methods that seek point estimates are less robust than Bayesian robust counterparts (https://arxiv.org/pdf/2106.06499.pdf) and I would like to see something similar for this paper. Bayesian methods are also often useful for active learning and uncertainty quantification, but there is no evidence of this benefit in the current paper. I maintain my recommendation of weak reject.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "IYdUYZ88DvG", "reviewer": "Reviewer_zKpN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper introduces a model-based inverse reinforcement learning approach for offline datasets, in which a dynamics model and reward function are estimated simultaneously. The method is motivated by a Bayesian Theory of Mind framework and approaches the problem from a bayesian inference perspective (MAP estimate). Two variants are proposed, BTOM and RTOM, where BTOM solves the full solution and RTOM solves a reduced version under the assumption that the real and estimated dynamics are sufficiently close. Experiments in Gridworld and MuJoCo are performed in which the proposed methods are compared to a prior method, ML-IRL, and show performance improvements in some scenarios.", "summary_of_recommendation": "As the paper currently stands, I would lean towards a weak reject due to the lack of baselines in the experimental results and the lack of clarity in the methodology. I am open to revising my recommendation though given any additional updates in the rebuttal as I think the paper has a lot of potential.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions:\n1) What does BTOM and RTOM stand for? They are not defined anywhere (presumably BTOM is Bayesian Theory of Mind which is also a bit confusing as that is an existing model).\n\n2) I'm not sure I totally understand the bottom row of Fig. 1. What is the input and output of the dynamics model? I am confused as to why the generated transitions are stochastic (ground truth figure) when the gridworld is a 5x5 environment with discrete actions and deterministic transitions. I would have expected a perfect grid for the ground truth dynamics model.\n\n3) Line 255 indicates that the results for ML-IRL are pulled from the paper [13], but this does not appear to be the case as the values don't match. Are these re-generated?\n\n4) How sensitive is lambda to the algorithm performance? Does this value require tuning?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* Using Bayesian Theory of Mind as a motivating framework for model-based inverse reinforcement learning is an interesting perspective. The mapping of the dynamics model to belief and reward to desire is intuitive on its face and differs from that in prior work (although I also feel that this is a bit of aweakness, see below).\n\n* The derivations appear to be correct to my eye, and the connection between the proposed method and prior robust model-based offline RL methods is a nice observation. Intuitively this connection makes sense and provides context for the methods.\n\nWeaknesses:\n* While an interesting perspective, I also feel that there is an over-emphasis on the Bayesian Theory of Mind aspect to a point where it detracts from the paper. Although the dynamics model and reward can be mapped to the beliefs and desires of an agent, this seems to me to simply be a motivation for construction a Bayesian posterior (in Eq. 5) as opposed to a full theory of mind model. For example, this work appears to assume that the expert's model of the environment dynamics is stationary, i.e., maps to a fixed belief, as opposed to rigorous theory of mind models which often consider dynamic beliefs in partially observable (POMDP) environments.\n\n* The relationship to prior methods is not entirely clear. Line 44 indicates that the proposed algorithms are extensions of prior works to continuous spaces, but this connection is never touched on again. The loss function defined in Eq. 7 is clearly similar to that defined in [18] (plus an entropy regularizer), so is the primary contribution the sample-based approximation scheme introduced in Sec. 3.1? Does the proposed Bayesian setup contribute to this, or could the sample-based approximations also have been directly applied to the loss functions defined in prior work?\n\n* The experimental results are not very comprehensive, as the proposed methods are only compared to a single prior method (ML-IRL). Given that connections are drawn to robust methods (specifically RAMBO), I would have expected a comparison to at least this method if not others. In addition, the results seem somewhat ambiguous. In some scenarios BTOM performs best, while others RTOM best, and others ML-IRL. Some analysis as to why this is the case would be helpful, e.g., in what scenarios is RTOM/ML-IRL expected to perform better than BTOM.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers appreciated the novelty of the proposed approach to IRL, the inclusion of theoretical results, and the thorough response in the rebuttal. The additional results with more baselines were very much appreciated, and were felt to make the paper stronger, although some doubts with regard to a clear performance win over ML-IRL remained (and should be addressed).
Decision: Accept (Poster)