Accelerating Inverse Reinforcement Learning with Expert Bootstrapping
Authors: David Wu
Sanjiban Choudhury
Venue: Submitted to ICLR 2023 (2023)
Keywords: inverse reinforcement learning
imitation learning
reinforcement learning

Abstract: Existing inverse reinforcement learning methods (e.g. MaxEntIRL, $f$-IRL) search over candidate reward functions and solve a reinforcement learning problem in the inner loop. This creates a rather strange inversion where a harder problem, reinforcement learning, is in the inner loop of a presumably easier problem, imitation learning. In this work, we show that better utilization of expert demonstrations can reduce the need for hard exploration in the inner RL loop, hence accelerating learning. Specifically, we propose two simple recipes: (1) placing expert transitions into the replay buffer of the inner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learner about high reward states instead of forcing the learner to discover them through extensive exploration, and (2) using expert actions in Q value bootstrapping in order to improve the target Q value estimates and more accurately describe high value expert states. Our methods show significant gains over a MaxEntIRL baseline on the benchmark MuJoCo suite of tasks, speeding up recovery to 70\% of deterministic expert performance by 2.13x on HalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.

Under review as a conference paper at ICLR 2023
ACCELERATING INVERSE REINFORCEMENT LEARNING WITH EXPERT BOOTSTRAPPING
Anonymous authors Paper under double-blind review
ABSTRACT
Existing inverse reinforcement learning methods (e.g. MaxEntRLL f-IRL) search over candidate reward functions and solve a reinforcement learning problem in the inner loop. This creates a rather strange inversion where harder problem, rein- forcement learning, is in the inner loop a presumably easier problem, imitation learning. In this work, we show that better utilization of expert demonstrations can reduce the need for hard exploration in the inner RL loop, hence accelerating learning. Specifically, we propose two simple recipes: (1) placing expert tran- sitions into the replay buffer of the inner RL algorithm (e.g. Soft-Actor Critic) which directly informs the learner about high reward states instead of forcing the learner to discover them through extensive exploration, and (2) using expert ac- tions in Q value bootstrapping in order to improve the target Q value estimates and more accurately describe high value expert states. Our methods show signif- icant gains a MaxEnt baseline on the benchmark MuloCo suite of tasks, speeding up recovery to 70% of deterministic expert performance by 2.1x on HalfCheetah-v2, 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.
INTRODUCTION
The core problem in inverse reinforcement learning (IRL) is to recover a reward function that ex- plains the expert's actions as being optimal, and a policy that optimal with respect to this reward function, thus matching expert behavior. Existing methods like MaxEntIRL (Ziebart et al. 2008) and f-RRL (Ni et al. 2020) accomplish this by rumning an outer-loop that updates reward function and an inner-loop that runs reinforcement learning (RL), usually many steps of policy iteration.
However, running RL in the inner-loop results in high sample and computational complexity com- pared to IL (Sun et al. 2017). Specifically, this requires large numbers of learner rollouts. Learner rollouts can be expensive, especially with high fidelity, complex simulators, and in the real world, where excessive rollouts can lead to elevated risk of damage to the physical agent or system. It therefore important to study methods for accelerating the inner RL loop. Our key insight that instead of treating the inner RL as some black box policy optimization, we can provide valuable information about potentially high reward regions that can significantly accelerate learning.
We propose two simple recipes that applicable to wide class of inner RL solvers, notably any actor-critic approaches (e.g. Soft-Actor Critic (SAC) (Haarnoja et al. 2018)):
1. Place expert transitions into the actor's replay buffer. These transitions contain high re- ward states that accelerate policy learning and reduce the amount of exploration required to discover such high reward states. We call this method expert replay bootstrapping (ERB).
2. Use the expert's next action from each transition in the critic's target Q-value estimator. By leveraging such side information, we more accurately describe high value expert states and improve the estimate of the next state's target value. We call this method expert Q bootstrapping (EQB).
In general, the critic's target value estimate is derived from the actor and the actor's optimization objective is derived from the critic. This creates mutual bond where neither side move forward without the other progressing accordingly. When the policy lagging behind due to its inability to effectively maximize potentially complex Q function surface, the policy's ' action may be quite

---

Under review as a conference paper at ICLR 2023
suboptimal. This creates lower critic target value estimates for expert states and slows learning- However, in the imitation learning setting, we have access to expert demonstrations that allow the critic to progress in its learning without being held back by the actor. In fact, the critic's 's value func- tion surface is exactly derived from these expert demonstrations. hence, the expert demonstrations maximizing value. By leveraging expert Q bootstrapping and providing accurate targets using the expert's 's next action, we allow the critic to progress as if it were working with a stronger policy. Accurate targets allow the Q function to progress in its learning and provide better signals to the policy, further accelerating learning.
We show that our methods are able to accelerate multiple state-of-the-art inverse reinforcement learning algorithms such as MaxEmtIL (Ziebart et al.] 2008) and f-IRL (Ni et al.) (020). We believe that our methods are especially helpful on hard exploration problems (i.e. problems where many actions lead to low reward, while few, sparse actions lead to high reward, like in the toy tree MDP in Section areas of the space through hard exploration, and informing the learner of expert states and actions through expert bootstrapping can significantly accelerate recovery of expert performance.
these types of problems, it difficult rely on the learner to find high reward
In summary, the main contributions of this paper are two recipes, ERB and EQB, which can be added onto ttate-of-the-an art inverse reinforcement learning algorithms (with few lines of code) for acceler- ated learning. Empirically, we show that our techniques yield significant gains the benchmark MuloCo suite of tasks. In addition, we explain when and why our techniques are helpful through the study of simple toy tree MDP.
2 RELATED WORK
2.1 LEVERAGING EXPERT DEMONSTRATIONS IN REINFORCEMENT LEARNING
Reinforcement learning algorithms (e.g. SAC (Haarnoja al. 2018), DQN (Mnih et al. 2013), and PPO (Schulman et al. (017)) aim to find an optimal policy by interacting with an MDP. Solv- ing reinforcement learning problem can require extensive exploration throughout a space to find potentially sparse reward. A standard practice is to bootstrap RL. policies with a behavior cloning policy (Cheng et al., 2018 Sun et al., 2018). Deep Q Learning from Demonstrationss(QQ0) (Hes- ter et al. 2018) and Human Experience Replay (Hosu & Rebedea 2016) propose to accelerate the exploration process by inserting expert transitions into the policy replay buffer in order to inform the learner of high reward states. However, these RL approaches assume access to stationary ground truth rewards, which is not the case in imitation learning where rewards are being learnt over time.
2.2 IMITATION LEARNING
Imitation learning algorithms attempt to find policy that imitates given set of expert demon- strations without access to ground truth rewards. Based on assumptions of available information, imitation learning algorithms can be broadly classified into three categories: offline (c.g. Behavior Cloning), interactive expert (c.g. DAgger (Ross et al.] [201T). VaTe
VaTe (Ross & Bagnelll, 2014).
or interactive simulator (e.g. MaxEntIRI (Ziebart et al. 2008 ). While offline algorithms such as offline IQ-Learn (Garg et al., (021)) and AVRIL (Chan & van der Schaar) 202T) are sample effi- cient at leveraging expert data, they suffer from covariate shift due the mismatch between expert and learner distributions. In this work, we use online inverse reinforcement learning algorithms to combat covariate shift and hence assume access to an interactive simulator.
Inverse reinforcement learning algorithms (e.g. MaxEntIRL (Ziebart et al.) 2008 f-IRL (Ni et al.) 2020 attempt to recover a reward function that explains expert behavior. There are previous meth- ods that aim at recovering reward without solving an inner loop RL problem. For example, (Klein et af. 2012) assumed simple classes of reward functions; (Pirotta & Restelli 2016)] Ramponi et al.) 2020) learned a reward function solely on expert states without considering the learner distribution. In contrast, we focus on general online inverse reinforcement learning methods. Adversarial imi- tation learning methods (e.g. AIRL (Fu et al. 2018), GAIL (Ho & Ermon| 2016)) attempt to find an optimal policy by using a discriminator instead of a reward function, and running policy opti- mization in the outer loop, as opposed to the inner loop in MaXEn Similar to our work, SQIL, or Soft-Q Imitation Learning (Reddy et al., 2019) inserts expert transitions into the learner replay buffer. However, SQIL only uses rewards of 0 for all learner transitions and rewards of for all
2

---

Under review as a conference paper at ICLR 2023
expert transitions, which does not recover reward function that can be used for other purposes, for example with different environments with different dynamics, and settings where the expert actions not realizable (e.g. human hand vs. robot arm). IQ-Learn (Garg et al.] 2021) proposes new formulation which learns a single Q function to take the place of both the reward function and the SAC policy critic. IQ-Leamn also inserts expert transitions into the replay buffer, for the purpose of estimating the expected policy value function V on the initial state distribution and for policy optimization. However, the authors did not provide justification ment. In addition. while the focus of the IQ- paper is the new formulation, the focus of our paper on general purpose recipes for using expert information to accelerate imitation learning.
empirical studies of this treat-
BACKGROUND
The goal of reinforcement learning algorithms is find policy π that maximizes reward in an MDP (S, (S,A,T, T, Po), which is a tuple of states S, actions A. transition dynamics T. reward function r, discount factor 7, and initial state distribution po. The policy is mapping from states to a distribution over actions. The reward function is mapping S x A - IR from states and actions to a reward value. The goal of the policy is to maximize cumulative discounted reward on trajectories sampled from the trajectory distribution induced by policy æ: max; ETTT
Ei ("r((t: at)
In maximum entropy reinforcement learning, the goal is to find a policy π that maximizes entropy- regularized reward: max; E Et= 7'(((st, uz) + H (=+(s=))) The additional entropic regu- larization increases exploration by incentivizing the policy to spread out its probability mass different actions instead focusing its probability mass on a single action.
It can be shown that the optimal policy selects trajectories with probabilities proportional to the ponential of the ground truth cumulative discounted sum of rewards: x <a(ss) = z, exp(@" a)), where n denotes the optimal policy and Q" denotes the optimal action value function.
Soft-Actor Critic (SAC), a popular maximum entropy RL algorithm, alternates between optimizing a Q function, or critic, and optimizing a policy, or actor. SAC calculates target next state values for the Q updates each transition by approximating the optimal action using explicit policy ı and calculating the entropy-regularized Q value for this action in order to create the target (Fquation []. 0 indicating whether the episode is over. For more
In Equation [] is defined as boolean details regarding this version of SAC, please refer to OpenAl Spinning Up (Achiam] 2018).
v(r, Vir,s' =r + d)( min Q (s', log (ā')s')) d' TT(-ls') 112
(I)
The ming- 1,2 Qone.: (8', a') a log Te (a'(s"))) term is essentially estimating V*(s'), or the value on state s' under the current policy. The objective of the Q function is
C(s.a,r, d) (Q(s, a) v(r, s', d))².
The policy's objective is to maximize the entropy regularized Q value max (a [Q a) a log (als))
SAC is essentially alternating between policy evaluation, where the Q function is estimating the Q value under the current policy, and policy improvement, where the policy is trying to improve using the current Q function.
In imitation learning settings, the reward function r not given, and instead set of expert demon- strations 'D is given. The goal in imitation learning is to find policy that imitates the expert. In inverse reinforcement learning, this done by recovering a reward function that explains expert behavior as shown in the expert demonstrations. In the maximum entropy formulation, the expert is assumed to be optimizing the reinforcement learning algorithms such as f-IRL and MaxEntIRL alternate between taking one gra- dient step for the reward and solving the inner-loop RL problem to obtain the optimal policy with respect the current reward function.
cumulative discounted sum of rewards. Inverse
3

---

Under review as a conference paper at ICLR 2023
4 ERB: EXPERT REPLAY BOOTSTRAPPING
In algorithms such MaxEntIRL, the policy typically ignores the expert data when solving the inner-loop RL problem, and instead must act solely based on the blackbox reward. We propose to accelerate the inner-loop RL problem by placing expert samples into the learner replay buffer and thereby informing the learner about high reward expert transitions instead of requiring the learner explore a potentially vast space in order reach the expert distribution. The detailed algorithm is described in Algorithm with the additions to MaxEntIRL highlighted in red.
Algorithm ERB version of MaxEntIRL
Input: Expert demos Dg. epochs N, inner policy steps T
Output: Policy т, critic Q, reward function r
1: Initialize SAC policy æ, critic Q. reward function
2: Initialize learner replay buffer D,.
3: Initialize SAC policy learner replay buffer Dp
4: for i 0 to N do
5: Collect policy rollouts Tz- place into Dr
6: Sample expert and learner batches Bg from DE, By from Dr
7: Update reward with gradient
BB (V.r(ss,)) a)] - Be [V,r((s)))
8:
for j j= 0 do
9:
Collect rollouts from и and place into Dp
10:
Sample policy batch Bp from Dp
Sample expert batch Bg from De
12:
Bp Bg Bp
Use the objective defined in Equations aand 2 update the critic Q
14: Use the objective in Equation 3 to update the policy
end for
16:
end for
17:
return r.
4.1 THIS JUST BEHAVIOR CLONING?
Behavior Cloning trains the learner to pick the expert action expert states. However, there several ways that Behavior Cloning may not function properly. For instance, the expert may not be realizable, in which case the learner may be led wee--eeer--istiiiuui states where it does not have any experience.
Instead, by providing the expert samples in the replay buffer, allow the policy choose parts of the expert transitions that are realizable and helpful in maximizing rewards even when the expert is realizable whole. As result, if the expert takes perfect actions while stochastic perturbations lead the learner to offf-expertddistriuttie states, non-dynamics-awarr BC will not be able to recover while dynamics-aware methods like MaxEntRL will be able to. In similar vein, there is strong evidence from the planning community (Phillips et al. 2012) that shows how expert demonstrations, even segments of it, can be used to construct useful heuristics aid solving large MDPs
We also evaluated Behavior Cloning and MaxEnURL with ERB on modified version of the Tree MDP presented in Section T where the learner has "shaky hands" and with 20% probability takes random action at any given state instead of the learner's chosen action. In addition, one of the actions at any given state is to go back up to the current node's parent, as a means of recovering from the stochastic perturbations. Expert demonstrations are given showing the expert perfectly descending the tree on the left and without "shaky hands". In such a situation, BC suffers from covariate shift, as it is unable to recover after stochastic perturbations have led it off- off-expert-disse -xstrr-distribution states, while dynamics-awer aware ERB is able to recover to the expert path and converge to higher returns, as shown in Figure ((b)

---

Under review as a conference paper at ICLR 2023
5 EQB: EXPERT Q BOOTSTRAPPING
Standard SAC updates used in both reinforcement learning and imitation learning calculate target next state Q values for each transition by approximating the optimal action using an explicit pol- icy π and calculating the regroay-regularizze Q value for this action in order to create the target (Equation I.
However, the policy's approximation of the optimal action may be suboptimal, which can cause target value estimates be significantly below what could be achieved by a stronger Q-optimizing policy. The lower targets cause the Q function to learn lower estimates of value and slows down learning. Ideally, the poliey component of SAC would be able strongly optimize the critic, and create a positive cycle where the policy provides better targets and accelerates the critic's 's progress, hence providing better signals to the policy. Our key observation is that when updating on expert states as in expert replay bootstrapping. the expert's 's near optimal next action is given and can be used improve the target estimate. This extra information creates an opportunity improve. We propose to use combination of the ground truth expert's 's next action and the policy's 's approximated action in order to create beller targets for Q learning on expert states.
Specifically, we modify the target equation on expert states to be
s'(r,s',d) r 711 VEVB VVeoo(s','''''' d'E,,, (5',dE) Bg, a' =o(-1s))
(4)
Qm(5,a) = min Q Quonr ())
(5)
1,2
amlorat)
Amina'
VEQB (s, á', d'g) log(e and the objective is thus
(Q(. ye(r, a d)) if if E BB (Q(s - v(r, s', d))² otherwise
where BE denotes the of expert transitions in our set of demonstrations. Note that VEOB is performing softmax between two discrete actions, so it creates mixture policy between the expert and learner selected actions.
Our final ERB+EQB learning algorithm consists of Algorithm I with Equations 4 and 7 critic objective, replacing Equations and 2 in line 13
defining the
5.1 DERIVATION OF EQB Q TARGET
In maximum entropy inverse reinforcement learning with a SAC policy, the goal of the policy is to maximize the entropy regularized Q value (Equation 8
max (+-=(+ [Q(s, a) a log [[als)]
This policy is then used to select actions when rolling out in the environment and is used to estimate the target state value used to update the Q function (Equation[]) However, as the expert next action is given on expert states, the target value estimate can be improved with the additional information.
At state s', suppose the policy suggests an action ä' and the expert suggests an action d'g. We compose these two actions into a stronger meta policy, and more importantly, the meta policy to derive value targets. The better value targets from the strong meta policy can accelerate the critic's progress and lead to better signals the policy, therefore accelerating both sides and creating virtuous cycle. We consider a class of meta policies 1IM that select between the action chosen by the current policy and the given expert action. Each meta policy is governed by a probability such that the meta policy selects the current policy chosen action with probability w and the expert next action with probability we can explicitly solve for the optimal value of w for meta policy TM that maximizes the discrete entropy-rege regulaazee Q value objective in Equation over the two choices:
w. we consider the policy and expert actions as discrete choices, then
VFOR (s) max [w(Q (8, a') - a log w) + (1 (Q (8, d'e) log((1 - w))]
(9)
aminidgg)
Gmilaaa)
log(e
(10)
5

---

Under review as a conference paper at ICLR 2023
5-
2-
0
0
0
MaxEIdRL= ERB
MarEronLa E88-
200
400
600
800
1000
Figure 1: I() An example toy tree MDP. Reward is only given for taking the leftmost actions. Results for Behavior Cloning, MaxEntRL, ERB, and EQB on the "shaky hands" MDP. Results averaged over 5 seeds, and shaded areas represent standard deviation. The x-axis represents iterations, while the y-axis represents return.
((b)
More details on the derivation can be found Appendix A
INSIGHTS FROM A TOY PROBLEM
To gain more insight into why ERB and EQB are helpful, we evaluated ERB and EQB on toy problem, consisting of an exponentially growing tree MDP with levels. High reward is given for each action on the leftmost path of the tree. Expert demonstrations are given showing the expert taking only the leftmost path. We ran simple SAC policy MaxEntRL algorithm evaluate the effectiveness of ERB and EQB. Though this problem is discrete, we used separate policy from the function, as our methods are specifically designed for the continuous case. continuous setting, it is infeasible to explicitly maximize over the different action Q values at state so we must have an additional separate policy that attempts to maximize the Q function. EQB is helpful when the separate policy is unable to effectively maximize the Q function. The policy, Q function, and reward function are all tabular. The policy æ parameterized by values Vz. which determine softmax distribution over actions. An example tree MDP is shown in Figure I(a)
Our algorithm alternates between performing n SAC updates, each of which consists of a Q update and a policy update, and performing I reward update. The original Q update is done using the following equation:
:= 711 (Q((())))))))))))))) (og(=(('(ss)))) ((-s')
(11)
:- (s, 712 * (y a)s.
(12)
For simplicity, the discount factor y and the entropy weight are both set to 1. The EQB Q update on expert states is done using the modified target
r 711 -d) (log(c (s),a")) + ald (al"a")))
(((da"))) a²))), d' TT-ls').
(13)
The policy update is done by maximizing the expected entropy regularized Q value. The gradients are calculated using Py Torch Autograd.
max Ea (Q(s.)) log (a|s)]
(14)
x
The update is then done:
Ux = Ux + 7x
Ve, (ey [Q(s, a) - log π (a]s)]
(15)
The reward update simply increases reward expert states and decreases reward learner states:
r(s, a) = r(s, a) + (1((s, a) is expert)) - 7r * (1((s, a) is learner)).
(16)
In the experiments in Figure learning rates ni and π7, are both set to 0.01, while 7/ is set to varying values (0.01, 0.001, 0.000T}. Here 7/ serves as an optimization difficulty knob that controls how well the policy optimize the Q function.

---

Under review as a conference paper at ICLR 2023
0 = 0.01
ng = 0.001
0, = 0.0001
Oy Edtt
0:011
0.003
0001
15
(m Srall LEB0-009
Manda 200---L--0000 ER0--000
Figure 2: Top: Results for toy tree MDP task with branching factor 10 and 1/= set to different values. Results averaged over seeds, shaded areas represent standard deviation. Bottom: Results for toy tree MDP task with branching factor 15. X-axis represents iterations, y-axis represents return.
We also tested using a branching factor b = 10 versus 15 in our experiments. A larger branching factor creates harder exploration problem, as there are sparser rewards. This creates more room for improvement from ERB and EQB, as they both cut down on exploration by giving the policy expert information and allow the policy to quickly find expert regions. As shown the bottom half of Figure 2
ERB and EQB yield bigger improvements when the branching factor is larger.
Why is ERB helpful? ERB is helpful in accelerating learning as instead of relying on the policy to explore the exponential tree and find high reward states, it directly informs the learner of high reward expert states and accelerates learning.
Why is EQB helpful? EQB allows the Q function to progress in its learning even when the policy may provide suboptimal target estimates. The policy may be unable to effectively optimize the Q function even though the Q function may be accurately estimating high value in expert transitions. In this case, using the expert action to estimate target value as EQB allows for an accurate target value estimate that is not incorrectly low because of the suboptimality of the policy. Using this target value estimate, the Q function is able to progress in its learning and continue to provide more accurate signals the policy, instead of being held back by the suboptimal policy, hence accelerating learning. This effect is illustrated in the horizontal variation in Figure 2 where we vary the learning rate of the policy to allow for different degrees of maximization of the function measure the effectiveness of EQB. When the policy is able to effectively maximize the Q function, the effect of EQB diminishes. On the other hand, when the policy is unable to effectively maximize the Q function, EQB significantly accelerates learning. Though using only the expert's 's next action to calculate target values a valid approach, we believe that using a combination between the expert's next action and the policy's chosen action is necessary. As the Q function may be erroneous during training, and the policy is explicitly optimizing the Q function, using the policy's chosen action can help catch erroneously high Q values and correct them needed. As there also cases in which the policy's ' chosen action is suboptimal, we believe that using a mixture of the policy and expert Q values defends against either extreme.
The modified target value estimate is no longer V as in SAC, it is estimating target value for the meta policy πs. We believe that this is beneficial it more accurately estimates the high value in expert states and moves the learner towards expert states, which is the ultimate goal.
7 EXPERIMENTS
We evaluated our methods on the benchmark MuloCo (oooooo et al. (012)) suite of tasks in OpenAI Gym (Brockman et al. 2016), building top of Max and / IRR. We built on top of an open-
7

---

Under review as a conference paper at ICLR 2023
Hopper-v2- Deterministic Return
HalfChestah-v2 Deterministic
14000
4000
32000
UN
3000
10000
Maye
8000
2000
6000
4000
1000
MaxEnURL
2000
MaxEndRL MaxEntIRL
MaxEntIRL+ERR
MactnuRL--ERR-E00
MoxEndRL-ERRB-ER
50
100
150
175
200
150
250
350
Welker2d-v2 Deterministic Return
Ant-v2 Deterministic Return
6000
5000
5000
4000
4000
3000
2000
2000
1000
MaxEntIRL
1000
MaxEnURL MaXEntRL+ERD MaxEnuRRL ERS-
MexEntRRLEER MaxEntRL FRS--O8
200
250
300
350
200
250
Figure 3:
Deterministic returns on MuJoCo tasks with a MaxEntRL baseline. X-axis is iterations, where each iteration is 5000 policy learning environment steps.
source implementation (that calculates state-only rewards to disentangle reward from dynamics); https: / con.t/wwwooo/ttttt/tiio.
IRL. git and added expert replay bootstrapping and expert Q bootstrapping as described in dection[4aane Section 5] In all experiments. we report results averaged over 5 seeds, and shaded areas represent standard deviation. In all plots, the x-axis is iterations, and each iteration is 5000 policy learning environment steps, and the y-axis is return. All hyperparameters remain the same as in the original f-IRI. configs except for the entropy weight a on expert state EQB Q updates due to the difference in the scale of entropy in EQB. More details are given Appendix B All experiments with expert replay bootstrapping are run with batches of half expert samples and half learner samples, We study the effect of different ratios of expert samples in Appendix C
The total number of samples used in each batch for updates remains the same.
MaxEntIRL Baseline:
We report results on the four benchmark MuJoCo environments reported in the f-IRL paper with MaxEmuRL baseline. Performance graphs showing deterministic return are given in Figure 3 and performance graphs showing stochastic return given in Figure 5 Expert performance reported in Table Each unit on the x-axis represents 5000 policy learning environ- ment steps, though there an additional 10000 learner rollout steps required by the reward function to update the reward replay buffer of learner states. We further report the amount of iterations to recover 50%, 70%, and 90% of deterministic expert performance Tables T and respectively. It is seen that our methods show significant gains over MaxEntIL baseline on the benchmark MuJoCo suite of tasks, speeding up recovery to 70% of deterministic expert performance by 2.13x on HalfCheetah-v2. 2.6x on Ant-v2, 18x on Hopper-v2, and 3.36x on Walker2d-v2.
fIRRL Ni al., 2020) Baseline: We also evaluated our methods on top of 3 variants of an f-RRL baseline, minimizing Forward KI, Reverse KI, and Jensen-Shannon Divergence. Detailed results are given in Appendix D We found that the variants achieved similar performance with each other and with the MaxEntRL baseline. We found that ERB and EQB gave consistent, significant im- provement on the Reverse KL and Jensen-Shannon Divergence variants for all four environments, and on the Forward KL variant of f-IRL, ERB and EQB gave significant improvement in terms of real return on 3 of the 4 environments. As an example, Deterministic Return for ERB and EQB on top of the Jensen-Shannon variant of f-IRI. is given in Figure However, Walker2d with
8

---

Under review as a conference paper at ICLR 2023
Hopper-v2 Deterministic Return
HalfChestah-v2 Deterministic
14000
4000
12000
3000
10000
8000
2000
6000
1000
ARRL-S
2000
NR
FIRL-JS-ERD
HRR--S-ERB+E00
FIRL -ERR-J--ER--E08
75
100
150
175
200
250
300
Welker2d-v2 Deterministic Return
Ant-v2 Deterministic Return
6000
4000
3000
2000
2000
1000
FIRLIJS
FIRL-JS
1000
JS+ERB
FIRL-JS+EBB-E0
FIRL-JS+ERS-EGO
50
250
Figure 4: Deterministic returns on 4 MuJoCo tasks with a Jensen-Shannon Divergence f-IL base- line. Each unit on the x-axis represents one iteration, 5000 policy update environment steps.
Table I: Number of iterations to recover 70% of deterministic expert performance. Lower is better.
Task
Original
ERB
ERB+EQB
Hopper-v2
126
83
7
HalfCheetah-v2
245
234
115
Walker2d-v2
296
160
88
Ant-v2
242
129
93
Forward KL f-RRL baseline, EQB yielded improvement on the surrogate Forward KL objective that f-IRL-FKL optimizing, but did do well in terms of real return. We suspect this is due a dis- connect between Forward KL and return on Walker2d. Detailed discussion is given in Appendix
CONCLUSION
In summary, have presented two methods for accelerating inverse reinforcement learning algo- rithms (e.g. f-RRL, MaxEntRRL) that are generally applicable to any such IRL algorithms with an off-policy RL algorithm serving as the learner. Our first method, expert replay bootstrapping, con- sists of placing expert transitions into the learner replay buffer and performing learner updates on these transitions. We believe that this informs the learner of high value expert states instead of rely- ing on hard exploration to reach expert states. Our second method, expert Q bootstrapping, consists of using the expert next action to create better target value estimate on expert states. This results in more accurate estimates of value expert states instead of potentially suboptimal policy action approximations that lead the learner away from expert states. Empirically, our methods significantly the benchmark MuJoCo suite of tasks. Promising
accelerate recovery to expert performance future directions include building stronger theoretical foundations for expert bootstrapping and ex- ploring higher dimensional MDPs that can benefit from acceleration.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018. http:: /tt http://ennaaspinniiiee
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
Alex J. Chan and Mihaela der Schaar. Scalable bayesian inverse reinforcement learning. In ICLR, 2021.
Ching-An Cheng, Xinyan Yan, Nolan Wagener, and Byron Boots. Fast policy learning through imitation and reinforcement. arXiv preprint arXiv: 10413, 2018.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse re-
inforcement learning. In International Conference on Learning Representations, 2018. URL
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learm Inverse soft-q learning for imitation. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL htttt:://opeeeeeeeeeee http:rrvvvee.eeeefrrreeeeeee www.:neee/ooommid/eoeee.m
xctb5p
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maxi- mum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning. 2018.
Todd Hester, Matej Vecerík, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arold, John P. Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In AAAI, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In 30th Conference on Neural Information Processing Systems, 2016.
lonel-Alexanrr Hosu and Traian Rebedea. Playing atari games with deep reinforcement learning and human checkpoint replay. ArXiv, abs/1607.05000 2016
Edouard Klein, Matthieu Geist, Bilal Piot, and Olivier Pietquin. Inverse reinforcement learning through structured classification. In NIPS, 2012.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602, 2013.
Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, 2020.
Mike Phillips, Benjamin Cohen, Sachin Chitta, and Maxim Likhachev. E-graphs: Bootstrapping planning with experience graphs. In International Symposium on Combinatorial Search, vol- ume 3, 2012.
Matteo Pirotta and Marcello Restelli. Inverse reinforcement learning through policy gradient mini- mization. In AAAI, 2016.
Giorgia Ramponi, Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, and Marcello Restelli. Truly batch model-free inverse reinforcement learning about multiple intentions. In AISTATS, 2020.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Sqil: Imitation learning via regularized behavioral cloning. ArXiv, 0bs/1905.11108. 2019.
Stéphane Ross and Andrew Bagnell. Reinforcement and imitation learning via interactive no- regret learning. ArXiv, abs/1406.5979, 2014.
10

---

Under review as a conference paper at ICLR 2023
Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statisties, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. ArXiv,
Wen Sun, Arun Venkatraman, Geoffrey Gordon, Byron Boots, and Andrew Bagnell. Deeply ag- grevated: Differentiable imitation learning for sequential prediction. In International conference machine learning, pp. 3309-3318, PMLR, 2017.
Wen Sun, Andrew Bagnell, and Byron Boots. Truncated horizon policy search: Combining rein- forcement learning & imitation learning. arXiv preprint a805. //240 /240, 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012. doi: 10.1
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. AAA/ Conference on Artificial Intelligence, 2008

---

Under review as a conference paper at ICLR 2023
Hopper-v2 Stochastic Return
HalfCheetah-v2 Stochastic Return
1000
12000
10000
8000
2000
6000
4000
1000
MaxEnURL
2000
MaxEndRL
MaxEntIRL+
BoxEndILLEERR
MectnuRL-EERR-EO0
MaxEndRL-ERRB-ER
100
175
200
150
250
350
Walkerrdvy Stochastic Return
Ant-v2 Stochastic Return
5000
4000
4000
3000
3000
2000
2000
MaxEntIRL MaxEntRRLERR MaxEnIRL +RG--08
1000
MaxEnURL MaXEnBRL+ERD
MaxEnuRLL ERS-
200
250
300
350
150
200
250
Figure 5: Stochastic returns on 4 MuJoCo tasks with a MaxEntIRL baseline. X-axis iterations, where each iteration is 5000 policy learning environment steps.
Table 2: Expert Performance for MaxEntIRL. Higher is better.
Task
Average Demonstration Performance
Hopper-v2 HalfCheetah-v2
10.70 12559.94 169.32
Walker2d-v2 Ant-v2
5283. 62:77700
1188877=13.444
A DETAILS OF SECTION 5.1
Substituting into the objective in Equation 3 Jusing the meta policy πM gives us
FT (s) VEOB (s) max Eaw [Qm a log (als))
(17)
STM
where the probability of selecting an action is simply the discrete probability choosing the action multiplied by the continuous density of that action under the original policy (we approximate the expert's density with the current policy). TM (a
(a)ss)
(18)
As the only parameter of the policy is the probability w of selecting the current policy's chosen action, we can expand the expectation and obtain
VEOB (s) max wwQ (m (s, a log π (ā'ls) a log w)+
(1 (.. d'r) log (a'gls)
log(1 w))]
(19)
where ă' and d' the current policy's chosen action and the ground truth expert next action respectively, and both are given. We will denote ă' and d'g by z and x2 respectively in the following to simplify notation.
12

---

Under review as a conference paper at ICLR 2023
Table 3: Number of iterations to recover 50% of deterministic expert performance. Lower is better.
Task
Original
ERB
ERB+EQB
Hopper-v2
4
HalfCheetah-v2
190
183
80
Walker2d-v2
221
134
56
Ant-v2
201
114
75
Table 4: Number of iterations to recover 90% of deterministic expert performance. Lower is better.
Task
Original
ERB
ERB+EQB
Hopper-v2
>200
140
19
HalfCheetah-v2
331
299
180
Walker2d-v2
>438
207
98
Ant-v2
>284
165
121
Setting the derivative equal to O, we obtain
Qm (s, æ1) - a log (2ilss) a(log +1)
Q m (8, x2) + a log x (s)ga) + a(log(1 w) + 1) =
(20)
which gives us
W
(Qm(aaag) los (( (Qmc). a
(21)
+
Substituting this into the original formula gives us
VEgR = a log(e Qmllaa)) Iog (()
Qm(a.al)
(22)
At convergence, taking into account the continuous density, EQB adds a constant log term the v²*(s)) term in Equation|] | In practice, we found that removing the additional entropy term in the exponent was helpful to performance, resulting in Equation T0
B EXPERT STATE Q UPDATE ENTROPY WEIGHT IN EQB
As the modified EQB target value equation does factor in entropy from the continuous action selection, and treats the expert and policy selected actions as given, the scale of the discrete entropy on expert states different when performing EQB Q updates. Thus, additional tuning may be required to find the correct a entropy weight when applying EQB in expert state Q updates. We found that using an entropy weight of a in value estimation on expert state Q updates EQB (Fquation (6) worked better than the original a 0.2. However, using = 1 in expert state Q updates on ERB did not reach EQB level performance and in some cases even significantly hurt performance, shown in Figures 6 and 7
indicating that the resulting gain from EQB is not from hyperparameter tuning. All experiments use an entropy weight a = I for EQB on expert state Q updates unless specified otherwise.
C DIFFERENT ERB RATIOS
We experimented with various ratios between expert and learner samples in order to study the effect of different amounts of expert versus learner information on imitation performance. Our results indicate that using 0 expert samples and using 0 leamer samples perform worse than the default of using half expert, half learner samples. The latter likely due to the absence of a normalizing force that lowers the value learner states. As the policy only given access to high reward expert
13

---

Under review as a conference paper at ICLR 2023
Hopper-v2 Deterministic Return
HalfChestah-22 Deterministic
14000
4000
22000
10000
3000
8000
4000
1000
MaxEntIRL-ERR
MaxEnLIRL
2000
MaxEndRL--ERD
MaxEntIRL
MedErLIRL+ERR
MaxEntIRL-
MaxEntIRL SEEEEEnRRLLERR--- +508
MoxEntRRL -ERB RQB
200
150
250
350
Welker2d-v2 Deterministic Return
Ant-22 Deterministic Return
6000
5000
5000
4000
4000
3000
2000
MaxEntIRL MaxEntRL+ERS MaxEntRL: ERB ECB
MexEntiR
1000
MaxEntRL ERB
MaxEntiRL ERB-EQB
200
250
300
350
200
250
Figure 6: Deterministic returns for ERB with a compared to other methods.
Hopper-v2- Stochastic Return
HalfCheetah-v2 Stochastic Return
22000
3000
20000
8000
2000
4000
1000
MaxEntIRL
MaxEndRL
MasEntRRL-ERR
2000
MaxEntRL-EER
MaxEntIRL ERRB
MaxEntRL-+RB
MaxEntIRL MEUIRR-ERRR +800
MaxEndRL, ERB eQS
50
75
100
125
150
175
200
50
130
250
350
Walker2d-v2 Stochastic Return
Ant-v2 Stochastic Return
5000
4000
4000
3000
3000
2000
2000
MaxEnURL
MaxEnDRL+ERB MaxEntRRLEEna MaxEndRL+ERG ERB
1000
MaxEnBRL+ERB MuxEnURLL+RRS MavEntRL-ERG--EG
200
250
300
350
200
250
Figure 7: Stochastic returns for ERB with a compared to other methods.
14

---

Under review as a conference paper at ICLR 2023
Hopper-v2 Deterministic Return
HalfChestah-v2 Deterministic
14000
3500
12000
3000
10000
2500
2000
1900
4000
1000
EAS
500
ERB
2000
ERB 100
EAS 100
100
175
200
150
250
350
Welker2d-v2 Deterministic Return
Ant-42 Deterministic Return
6000
5000
4000
4000
3000
2000
2000
ERB
H
ERB 25
1000
ERB 50
88,
-2000
ERB 300
ERB 100
200
250
300
350
100
150
250
Figure 8: Deterministic returns for MaxEntRL++RB with different ratios of expert samples. In figure, ERB X denotes that X percent of update transitions in the expert replay buffer were expert transitions.
transitions, there is force decrease predicted value in non-expert areas, and all values will naturally float up. Learner samples serve the purpose of providing normalization force to make sure that non-expert areas do not have incorrectly high value. In addition, no particular non-extreme ratio of expert samples out of (25, 50, 75} is consistently better or worse than the others. Hence, we recommend using a default ratio of 50% expert samples in order to strike a balance between learner and expert information. Deterministic returns for this study shown in Figure and stochastic returns are shown in Figure
D f-RLL RESULTS
We evaluated ERB and EQB on three variants of an f-IRL baseline, minimizing Forward Kullback- Leibler, Reverse Dullback-Leiblles and Jensen Shannon Divergence. For the Forward KL variant, Deterministic Returns are given in Figure 10 and Stochastic Returns are given Figure [1 For the Reverse KL variant, Deterministic Returns given in Figure 12 and Stochastic Returns given Figure 13 For the Jensen-Shannon variant, Deterministic Returns are given in Figure 14 and Stochastic Returns given in Figure 15
Jensen-Shannon
E WALKER2D f f-IRL-FKL RESULTS
When evaluating methods ERB and EQB f-IRL-FKL baseline, found that both ERB and EQB provided consistent improvement on both Forward KL and real return over the f-IR baseline 3 of the environments. However, Forward KL surrogate objective but not on the return. We suspect this is due to the mismatch between the surrogate KL objective that f-IRL is optimizing and the return Walker2d, while on other environments the two are highly correlated. Deterministic returns are given in Figure 10 while Forward KI. is given in Figure 16
Walker2d, EQB provides improvement on the
15

---

Under review as a conference paper at ICLR 2023
Hopper-v2 Stochastic Return
HalfCheetah-v2 Stochastic Return
3500
12000
3000
2500
2000
8000
6000
1500
1000
ERB
ERB 0
ERB
2000
ERB 75
ERB 75
100
100
25
50
75
100
125
150
175
200
50
100
150
200
250
300
350
Walkerrd-v2 Stochastic Return
Ant-v2 Stochastic Return
4000
4000
2000
2000
ERB 25
MMM
ERB
ER.
MM
ER8.50
ERB 75
-200
ERB 75
EAB 100
ERB
100
130
200
250
300
350
400
100
150
200
250
Figure 9: Stochastic returns for MaxEndRL+ERB with different ratios of expert samples. In the figure, ERB X denotes that X percent of update transitions in the expert replay buffer were expert transitions.
Hopper-v2 Deterministic Return
HalfCheetah-v2 Deterministic Return
14000
12000
3000
20000
2000
6000
4000
1000
HRLLFK
2000
S-RL-FKL
FIRL-FKI
HIRL FIRL-PKL--ERB+EE
FIRL-FKL-RB++B0
25
50
75
100
125
150
175
200
100
150
200
250
300
Walker2d-42 Deterministic Return
Ant-v2 Deterministic Return
6000
6000
5000
4000
3000
3000
2000
2000
XIRL-FIL
SIRL-FIL
SIRL-FLL+ERS-ER
SARL-FKLEERS-EER
100
150
200
250
300
50
100
200
250
Figure 10: Deterministic returns on MuloCo tasks with a Forward KL. Divergence f-IRL baseline. Each unit on the x-axis represents one iteration, or 5000 policy update environment steps.
16

---

Under review as a conference paper at ICLR 2023
Hopper-v2 Stochastic Return
HalfCheetah-v2 Stochastic Return
4000
2500
12000
3000
10000
2500
2000
6000
FIRL-FKL
2000
FIRL-FLL-ER
FIRL-FKLL-ERR
FIRL-FLLEERB+ER
+IRL-RO--ERD-E00
Walker2d-v2 Stochastic Return
Ant-v2 - Stochastic Return
5000
5000
4000
4000
3000
2000
2000
1000
SIRL-FKL
1000
SIRL
+RR
SIRI-FKI +FAR
SIRL- FEL-ERBBE
Figure 11: Stochastic returns on 4 MuJoCo tasks with Forward KL Divergence f-RL baseline. Each unit on the x-axis represents one iteration, or 5000 policy update environment steps.
Hopper-v2 Deterministic Return
HalfCheetah-v2 Deterministic Return
24000
4000
12000
3000
20000
8000
2000
6000
4000
1000
NRRL-KK
FIRL RECL-ERB
NIRL-KKL BERK--ERBBHE EEB
FIRL- BIBL-AKL-EBB-EO
200
Welker2d-v2 Deterministic Return
Ant-v2 Deterministic Return
6000
5000
5000
4000
4000
3000
3000
2000
1000
FIRL-KL
FIRLIKK
FIRL-RKL+RRD
1000
FIRL RKL+ERB
FIRL-RKL + ERB
FIRL-RKL ERB+ EQB
0
Figure 12: Deterministic returns on 4 MuJoCo tasks with a Reverse Kullback-Leibler /RL base- line. Each unit on the x-axis represents one iteration, 5000 policy update environment steps.
17

---

Under review as a conference paper at ICLR 2023
Hopper-v2- Stochastic Return
HalfCheetah-v2 Stochastic Return
4000
3500
12000
3000
2500
8000
2000
6000
1500
1000
2000
500
ARR-
F-RL- FRL-AXL-ERR
GIRL-AK +RBB+00
Wa Walker2dd-2 Stochastic Return
Ant-v/ Stochastic Retum
6000
5000
4000
3000
3000
2000
2000
FIRL-RKL ERB
KIRLAKL+ER ARL-AKL-ERB-EEO
FIRL- ERB-EQQ
0
Figure 13: Stochastic returns on 4 MuloCo tasks with Reverse Kullback-Leible Divergence f- IRI. baseline. Fach unit on the x-axis represents one iteration, or 5000 policy update environment steps.
Hopper-v2 Deterministic Return
HalfChestahww2 Deterministic
24000
4000
12000
3000
8000
2000
6000
1000
HRLJJ
2000
NRL--S5ERB
H-RLJSSERR
HRL-JS-ERB+E00
SIRL #IRL/S-ERR+EE8
100
150
175
200
250
Walker2d-w2 Deterministic Return
Ant-v2 Deterministic Return
3000
2000
2000
1000
FIRL-JS
FIRL-SS
JS+LJS
1000
FIREJSHERB
FIRL-SS-ERB-E08
FiRL-JS+ENS-EOG
250
Figure 14: Deterministic returns on 4 MuloCo tasks with a Gensen-Shanmon Divergence f-RRL baseline. Each unit the x-axis represents iteration, 5000 policy update environment steps.
18

---

Under review as a conference paper at ICLR 2023
Hopper-v2- Stochastic Return
HalfCheetah-v2 Stochastic Return
12000
3500
10000
2500
8000
2000
8000
1500
4000
1000
2000
FIRL-S
500
NRRJSSERR
FIRL-SS-ERB
GRRL-S--RR+-ED
200
Wa Walker2d-v2 Stochastic Return
Ant-v/ Stochastic Retum
6000
6000
5000
6000
3000
3000
2000
2000
FIRL-JS
FIRL-JS
FIRL-JS
SIRL-JS+ ERB
+-RLL-55 1ILL-S-ERB--EO
0
+RRL-JSERB--EC
Figure 15: Stochastic returns on 4 MuJoCo tasks with a Jensen-Shannon Divergence f-RL baseline. Each unit on the x-axis represents one iteration, or 5000 policy update environment steps.
Hopper-v2 Approximate Forward Divergence
HalfCheetah-v2 Approximate Forward KL Divergence
35
30-
5-
WWW
10-
0-
FIRL-FIL
FIRL-FKL+ERA
5-
NIRL- FKLL-FLL-PR ERB
-5
FIRL-FLLEEBB-E0B
NKL ERB+EOB
50
75
100
125
150
175
200
50
100
150
200
250
300
Approximate Forward Divergence
Ant-v2 Approximate Forward KL Divergence
25
140
15
100
60
FIRL FKL
FIRL-FKL
FRL- FIRL-KKLERRHEOO
ARRL-FXL-ER
X.RL- ERB EQB
100
250
250
300
50
100
150
200
250
Figure 16: Approximate Forward KL Divergence over time on a Forward KL fIRL baseline. Each unit on the x-axis represents iteration, 5000 policy update environment steps.
19


Reviews:
[{"review_id": "yF8O8l7OuX", "reviewer": "Reviewer_K4n8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies an intriguing research question: utilizing the expert demonstration during the policy update step in an Inverse Reinforcement Learning (IRL) algorithm. This technique has been applied by many previous works, but none of them has carefully analyzed the impact of including expert demonstration. In order to utilize the demonstration data, the authors mainly focus on two methods: 1) adding expert transitions into the replay buffer of the inner RL algorithm. 2) using expert actions in Q value bootstrapping in order to improve the target Q value estimates and more accurately describe high-value expert states. The paper introduces the motivation and intuitively analyses the impact of each of these methods. In the experiment, the paper use MaxEnt-IRL and f-IRL as baselines for demonstrating the increase in training speed after including expert demonstration.", "summary_of_the_review": "This paper describes a very interesting research question by studying the approach of including expert data in the policy update of IRL algorithms. The proposed ERB and EQB algorithms are intuitive and easy to follow, but it requires a more careful understanding of the impact of using expert data, especially when they are significantly different from the current policy. The over-aggressive update can cause many issues (Check the motivation of the update constraining methods like TRPO and PPO). The comparison is incomplete without comparing the offline IRL methods. Overall, I am inclined to reject the paper based on its current version, but I agree the research question is important and I believe the authors can resolve these issues by continuing this research.", "strength_and_weaknesses": "**Strength**\n\n1. This paper focuses on an intriguing research question. To the best of my knowledge, while many previous IRL works did apply the expert demonstrate in policy update, but they have not carefully stuidied this method but rather treat it as an engineering trick that could potentially increase the training speed. I agree with the authors that this method requires a more carefully analyze by demonstrating its impact empirically and theoretically. While this paper does touch some of these aspects, I believe a more carefully study is required (please see my comment below).\n2. The paper is well-structured and easy to follow with sufficient details for supporting understanding.\n\n**Weaknesses**\nI suggest the author to improve their paper by considering the following ideas:\n\n1. **The Size of Policy Update**. In order to prevent destructively large policy updates, many RL algorithms, including the most popular PPO and TRPO algorithms, often include a constraint on the size of the policy update. They carefully tuned the objective to exclude state-action pairs from the policy that deviates from the current policy, but in this work, both ERB and EQB directly incorporate expert demonstration into the policy update and **the expert policy is significantly different from the imitation policy, especially in the beginning of training, which contradicts to the motivation of PPO or TRPO and is likely to drive a sub-optimal policy**. It is why MaxEnt IRL outperforms the proposed model in Figure 1(b) and Figure 2. I strongly recommend referring to the idea of constraining updates.   \n\n2. **Offline IRL baselines** Depending on whether IRL requires interaction with the environment, we can divide IRL into an online version and an offline version. For the Offline IRL algorithms (including IQ-Learn which has both the online and offline versions and AVRIL[1],), their policy update relies only on the expert demonstration data, so they require no more help from ERB and EQB mentioned in this paper. **I believe the author should include these offline IRL methods in their related works, and they could serve as strong baselines in this paper**. \n\n[1] Alex J. Chan and Mihaela van der Schaar. Scalable Bayesian Inverse Reinforcement Learning. ICLR 2021.\n\n3. **Testing Environment** The MoJoCo benchmark has 8 environments that are commonly applied for evaluation, but this paper studies only 4 of them. I am wondering how well ERB and EQB perform in the rest of the environments.  \n\n4. **Additional Concerns**\n\n- Please include labels into the x-y axis of all plots. The plots without labels are difficult to read.\n\n- \"*which does not recover a reward function that can be used for other purposes.*\"   \nI am confused by the term \"other purposes\". Please clarify.\n\n- \"*Formula (7)*\"  \nI am wondering how to measure if $s^{\\prime}$ is included in $B_{E}$. Will EQB examines all the samples and all the features to find the exact match? If yes, this step will be extremely time-consuming. I suggest training a density model for measuring the density of the input state in the expert distribution. If this density is large enough, we can use the EQB loss.\n\n- \"*Each meta policy is governed by a probability w such that the meta policy selects the current policy chosen action with probability w and the expert next action with probability 1 − w.*\"   \nI am confused by the definition of $\\omega$, especially when the authors substitute $log\\pi$ with $log\\omega$ in Appendix A. I think $\\omega$ is an important weight between the current and the expert policy, and intuitively, the agent should simply pick the expert action to maximize the Q values, which contradicts Formula (10).\n\n- MaxEnt IRL outperforms ERB and EQB in Figure 1 (b) and Figure 2, although it consumes more episodes or steps. I have explained this issue from the perspective of constraining updates.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZZZlX3ECQg", "reviewer": "Reviewer_kiih", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the high sample and computational complexity problem in IRL which\napplies RL in its inner-loop optimization. Therefore, the authors proposed two simple\nsolutions: 1). Putting expert transitions in the reply buffer of the inner RL algorithm to\nfamiliarize the learner policy with highly rewarded states, i.e., ERB, and 2). Using expert actions\nin Q value bootstrapping to improve target Q value estimation, i.e., EQB. Experiment results on\nthe MuJoCo tasks shows its superior performance when comparing with MaxEntIRL\napproaches in improving sample efficiency.", "summary_of_the_review": "In general, the paper proposes an interesting idea to speed up IRL training in the inner RL training loops with expert guidance. However, the proposed solutions seem incremental by simply incorporating expert demonstrations in reply buffer and target Q value estimation. In addition, it lacks experimental support on its performance over IL baselines, and has performance uncertainties in tasks like Walker.", "strength_and_weaknesses": "Strengths:\n\n1. The idea of incorporating expert guidance seem valid to speed up inner RL loop training.\n2. Experiment results show improved sample complexity over MaxEntIRL related baselines.\n\nWeaknesses:\n\n1. The approach of incorporating expert demonstrations into inner RL training seems to be intuitive. There is no show of applying expert demonstrations in the target Q value estimation will help recover a valid estimation similar to the ground truth Q value.\n2. The experiments do not show performance comparisons with imitation learning baselines such as GAIL.\n3. The performance of ERB and EQB is uncertain. Though in most experiments, ERB+EQB works, they fail in the Walker task. The explanation that the surrogate KL objective and return in the task does not correlate is questionable as it does not tell\nwhy f-IRL works in the Walker task but adding ERB+EQB worsens the performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KhzI41FcLn", "reviewer": "Reviewer_wpP8", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a method to accelerate the inverse reinforcement learning (IRL) process, by injecting knowledge about the expert's policy in two different ways. The first idea consists in introducing transitions from the expert's demonstrations in the replay buffer (expert replay bootstrapping, ERB). The second idea consists in employing the expert's actions as next actions for defining the target in the learning of the critic (expert Q bootstrapping, EQB). Both ideas are developed formally and tested. The evaluation is performed first on a toy example to illustrate the functioning of the methods and, then, on four Mujoco environments in combination with MaxEntIRL. ", "summary_of_the_review": "Overall, the paper proposes reasonable modifications to IRL algorithms coupled with actor-critic methods, in order to speed up the IRL process. Although the experimental evaluation succeeds in showing the advantages compared with the plain IRL, there is no comparison with BC. Moreover, the proposed modifications are, in my opinion, incremental. Therefore, at the moment I opt for a borderline negative evaluation.", "strength_and_weaknesses": "**Strengths**\n- The experimental evaluation shows the two approaches proposed in the paper are able to significantly outperform the plain MaxEntIRL in both the toy example and the Mujoco environments.\n- The approach seems to have the potential to be applied even for other learning algorithms besides SAC.\n\n**Weaknessees**\n- The approach is developed in detail assuming that the inner learning algorithm is SAC. It is not straightforward to generalize some aspects to general actor-critic.\n- It is not clear how these remarkable results compare with standard behavioral cloning.\n- The proposed approaches can be considered incremental w.r.t. the IRL algorithms to which they are applied. Indeed, although reasonable and easy to understand, no guarantee from a theoretical perspective is provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9OH8LNWbQs", "reviewer": "Reviewer_vmdh", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple set of modifications to two popular inverse-reinforcement learning (IRL) algorithms to improve their computational efficiencies, including MaxEntIRL and f-IRL. Note that a typical IRL algorithm contains two optimization loops: an outer loop that updates a reward function and an inner loop that runs reinforcement learning (RL), usually many steps of policy iteration. Algorithms like MaxEntIRL and f-IRL do not utilize expert data when solving the inner-loop RL problem and instead must act solely based on the black box reward. The authors propose two simple recipes to accelerate MaxEntIRL: (1) placing expert transitions into the replay buffer of the inner RL algorithm and (2) using expert actions in Q value bootstrapping. Simulation results show significant improvement over a MaxEntIRL baseline, which corroborates the authors' claim.", "summary_of_the_review": "This paper proposes two simple modifications which can significantly accelerate popular IRL algorithms including MaxEntIRL and f-IRL. Simulation results support the authors' conclusion. The simplicity of the solution and the degree of improvement are both appealing. However, most IRL algorithms do not restrict the type of data used in the inner loop RL, and can benefit from the expert's demonstrations using off-policy methods. In this sense, this paper's contribution might be limited. Still, the specific recipes proposed for a popular algorithm like MaxEntIRL could benefit other researchers in the field.", "strength_and_weaknesses": "This paper proposes two simple modifications which can significantly accelerate popular IRL algorithms including MaxEntIRL and f-IRL. The authors perform extensive simulations on standard benchmarks like MuJoCo suite of tasks. Simulation results support the authors' conclusion. The simplicity of the solution and the degree of improvement are both appealing.\n\nOn the other hand, most IRL algorithms do not restrict the type of data used in the inner loop RL, and can benefit from the expert's demonstrations. Therefore, one could apply off-policy Q-learning (or policy gradient) to obtain an optimal solution for the inner loop RL. For instance (Zhu, Lin, Dai & Zhou, NeurIPS 2020) combined off-policy RL with a popular IRL algorithm called GAIL. In other words, there exist IRL algorithms in the literature that could leverage expert data during the inner loop RL. In this sense, this paper's contribution might be limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject