Coherent Soft Imitation Learning
Authors: Joe Watson
Sandy Huang
Nicolas Heess
Venue: NeurIPS 2023 spotlight (2023)
Keywords: imitation learning
inverse reinforcement learning
behavioral cloning
learning from demonstration
TL;DR: Perform imitation learning by using a behavioural cloning policy to construct an estimate of a shaped reward function.

Abstract: Imitation learning methods seek to learn from an expert either through behavioral cloning (BC) for the policy or inverse reinforcement learning (IRL) for the reward.
Such methods enable agents to learn complex tasks from humans that are difficult to capture with hand-designed reward functions.
Choosing between BC or IRL for imitation depends on the quality and state-action coverage of the demonstrations, as well as additional access to the Markov decision process. 
Hybrid strategies that combine BC and IRL are rare, as initial policy optimization against inaccurate rewards diminishes the benefit of pretraining the policy with BC.
Our work derives an imitation method that captures the strengths of both BC and IRL.
In the entropy-regularized (`soft') reinforcement learning setting, we show that the behavioral-cloned policy can be used as both a shaped reward and a critic hypothesis space by inverting the regularized policy update. 
This coherency facilitates fine-tuning cloned policies using the reward estimate and additional interactions with the environment.
This approach conveniently achieves imitation learning through initial behavioral cloning and subsequent refinement via RL with online or offline data sources.
The simplicity of the approach enables graceful scaling to high-dimensional and vision-based tasks, with stable learning and minimal hyperparameter tuning, in contrast to adversarial approaches.
For the open-source implementation and simulation results, see https://joemwatson.github.io/csil/.

Coherent Soft Imitation Learning
Joe Watson fi
Sandy H. Huang
Nicolas Heess
Google DeepMind London, United Kingdom (shhuang, heess @google. com com
'T Darmstadt Darmstadt, Germany
Systems Al for Robot Learning German Research Center for Al dfki.de de
joe@robot-learning
Abstract
Imitation learning methods seek learn from expert either through behav- ioral cloning (BC) for the policy or inverse reinforcement learning (IRL.) for the reward. Such methods enable agents learn complex tasks from humans that are difficult to capture with hand-designed reward functions. Choosing between BC IRL for imitation depends on the quality and state-action coverage of the demonstrations, as well as additional access to the Markov decision process. Hy- brid strategies that combine BC and IRL are rare, as initial policy optimization against inaccurate rewards diminishes the benefit of pretraining the policy with BC. This work derives an imitation method that captures the strengths of both BC and IRL. In the entropy-regui regular-ee ('soft') reinforcement learning setting, we show that the behavioral-cloned policy can used both a shaped reward and critic hypothesis space by inverting the regularized policy update. This coherency facilitates fine-tuning cloned policies using the reward estimate and additional interactions with the environment. Our approach conveniently achieves imitation learning through initial behavioral cloning and subsequent refinement via RL with online offline data sources. The simplicity of the approach enables graceful scaling to high-dimensional and vision-based tasks, with stable learning and minimal hyperparameter tuning, in contrast to adversarial approaches. For the open-source implementation and simulation results, see joemwatson, goemwatson.githubiiuk github.io/csil.
Reward
Classifier
Ours
By using regression rather than classification, our approach infers shaped reward from expert data for this contextual bandit problem, whereas classifiers require additional non-expert data and may struggle resolve the difference between expert (X) and non-expert (X) samples.
Figure 1:
Introduction
Imitation learning (II.) methods [1] provide algorithms for learning skills from expert demonstrations in domains such robotics [2], either through direct mimicry (behavioral cloning, BC [3]) inferring the latent reward (inverse reinforcement learning, IRL [4-6]) assuming a Markov decision process susceptible covariate shift during
(MDP) [7, 8j. While BC straightforward to implement, evaluation [9]. IRL overcomes this problem but tackles the more complex task of jointly learning
"Corresponding author. Work done during internship at Google DeepMind.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

reward and policy from interactions. So far, these two approaches to imitation have been largely separated [10]. as optimizing the policy with an evolving reward estimate counteracts the benefit of pre-training the policy [11]. Combining the objectives would require careful weighting during learning and lacks convergence guarantees [12]. The benefit of combining BC and IRI. sample- efficient learning that enables any BC policy to be further improved using additional experience.
Contribution. Our method naturally combines BC and IRL by using the perspectives of entropy- regularized reinforcement learning (RL) [13-19] and gradient- based IRL [20, 21J. Our approach uses a BC policy to define an estimate of shaped reward for which it is optimal, which can then be used to finetune the policy with additional knowledge, such as online interactions, offline data, or dynamics model. Using the cloned policy to specify the reward avoids the need for careful regularization and hyperparameter tuning associated with adversarial imitation learning [11, 22]. and we estimate a shaped variant of the true reward, rather than use a heuristic proxy, e.g.. [23, 24). In summary,
1. We use entropy-regulanee regularized RL to obtain reward for which bchavioral-cloned policy is optimal, which we use to improve the policy and overcome the covariate shift problem.
2. We introduce approximate stationary stochastic process policies to implement this approach for continuous control by constructing specialized neural network architectures.
3. We show strong performance on online and offline imitation for high-dimensional and image-based continuous control tasks compared to current state-of-the-art methods.
2 Background & related work
A Markov decision process is a tuple (S.A.P,r, r.P. po). where S is the state space, A is the action space. P S x A x S +R+ the transition model, r S x A E is the reward function, is the discount factor, and Ao is the initial state distribution so ~ po(-). When evaluating a policy E II, we use the occupancy measure (s, a) - T(a ((a|s) s) (s),,, (s(s) & Et Y'M (s) where 1 = f horizon discounted expectations, E. Es.a-p, (f a)] E p( (-)=()))))) IE 7' fs at)l.
s, a, a(a 8) p) (s) da. This measure is used to compute infinite- ( P(s'
Densities du (8, a) = (1 7) Par (8, a) and 1x (8) = (1 7) p (8) are normalized measures. When the reward unknown, imitation can be performed from dataset D of transitions (s, a, s') as demonstrations, obtained from the discounted occupancy measure of the expert policy t. arg max, papp [r(s, a)]. The policy could be inferred directly or by inferring reward and policy jointly, referred to as behavioral cloning and inverse reinforcement learning respectively.
Behavioral cloning. The simplest approach to imitation is to directly mimic the behavior using regression [3], min 1(TT D) + X (TT), with some loss function I, hypothesis space II and regular- izer V. This method is effective with sufficient data coverage and an appropriate hypothesis space but suffers from compounding errors and cannot improve unless querying the expert [9]. The MIMIC-EXP algorithm of Rajaraman et al. [25] shows that the optimal no-interaction policy matches BC.
Inverse reinforcement learning. Rather than just inferring the policy, IRL infers the reward and policy jointly and relies on access to the underlying MDP [26, 5]. IRL iteratively finds a reward for which the expert policy is optimal compared to the set of possible policies while also finding the policy that maximizes this reward function. The learned policy seeks to match or even improve upon the expert using additional data beyond the demonstrations, such environment interactions [6]. To avoid repeatedly solving the inner RL problem, one can consider the -theretheoeetic approach [27], where the optimization problem is a two-player zero-sum game where the policy and reward converge to the saddle point minimizing the 'apprenticeship error' minaem max Ep [r]rr)E,,,.. [r]. This approach is attractive as it learns the policy and reward concurrently [28]. However, in practice, saddle-point optimization can be challenging when scaling to high dimensions and can exhibit instabilities and hyperparameter sensitivity [11]. Compared to BC, IRL methods in theory scale to longer task horizons by overcoming compounding errors through environment interactions [29].
Maximum entropy inverse reinforcement learning soft policy iteration. IRL is an under- specified problem [26]. in particular in regions outside of the demonstration distribution. An effective way to address this is causal entropy regularization [30], by applying the principle of maximum entropy [31] to TRL. (ME-IRI.). Moreover, the entropy- regulaiireellrrizeed formulation has an elegant closed- form solution [30]. The approach can be expressed as an equivalent constrained minimum relative entropy problem for policy g(a s) against uniform prior p(a s) Ua(a) using the Kullback-

---

Leibler (KL) divergence and a constraint matching expected features Φ, with Lagrangian
mill,ww Js (s) [9 s) || p(a s)] ds + w w (Es,a~D(((( a)] - Es as, (o(s,)))) (1)
The constraint term can be interpreted as the apprenticeship error, where w T 6(s, a) (s, a).
Solving Equation using dynamic programming yields the 'soft' Bellman equation [30],
((8, a) = [m, a) + + (s)),, (a()) = log / fexp (- (=0(s) (2) p(a 8) da, (2)
for temperature = 1. Using Jensen's inequality and importance sampling, this target is typically replaced with a lower-bound [32], which has the same optimum and samples from the optimized policy rather than the initial policy like many practical deep RT. algorithms [17]:
( a) = ( (s.
a) + ByBa'rn(-1a) P s..)) [Q(('' a') (log g(a') s') p(a) s'))].
(3)
The policy update blends the exponentiated advantage function 'pseudo- likelihood' with the prior, as a form of regularized Boltzmann policy [17] that resembles a Bayes posterior [33],
(a 8) exp(a (Q((s, a) - (x)) p(a s).
(4)
These regularized updates also be used for RI., where it is the solution to a KI.rregularized RI. objective, max, Rg Rx,aa,, (2(s, a)] a Dax[1(( [8) || p(a] s)]. where the temperature a now controls the strength of the regularization. This regularized policy update is known soft- [14, 17] or posterior policy iteration [34, 35] (SPI, PPT), as it resembles a Bayesian update. In the function approximation setting, the update is performed in a variational fashion by minimizing the reverse KL divergence between the parametric policy go and the critic-derived update at sampled states 8 [17].
0. arg ming Ea--1 (0ki /2o (a 8) || (a s)]] arg maxg (x), (0) E a [0), a-B [Q(s a) a (log s) logp(a 8 s))]
(5)
The above objective Jx can be maximized using reparameterized gradients and minibatches from replay buffer B [17]. A complete derivation of SPI and ME-IRL is provided in Appendix K.1.
Gradient-based inverse reinforcement learning. An alternative IRL strategy is a gradient- based approach (GIRL) that avoids saddle-por -oint optimization by learning a reward function such that the BC policy's 's policy gradient is zero [20], which satisfies first-order optimality. However, this approach does not remedy the ill-posed nature of TRI.. Moreover, the Hessian is required for the sufficient condition of optimality [36], which is undesirable for policies with many parameters.
Related work. Prior state-of-the-art methods have combined the game-theoretic IRL. objective [27] with antropy-regularizae RI. These methods be viewed minimizing divergence between the expert and policy, and include GAIL [37]. AIRL [38], f-MAX [39], DAC [40], ValueDICE [41], IQLeam [42] and proximal point imitation learning (PPIL,, [43]). These methods differ through their choice of on-policy policy gradient (GAIL) or off-policy actor-critic (DAC, IQLearn, PPIL), and also how the minimax optimization is implemented, e-g., using a classifier (GAIL, DAC, AIRL), implicit reward functions (ValueDICE, IQLearn) or Lagrangian dual objective (PPIL). Alternative approaches to rerrop-regullriiee use the Wasserstein metric (PWIL) [44], labelling of sparse proxy rewards (SQIL) [23], feature matching [30, 45-47], maximum likelihood [48] and matching state marginals [49] to specify the reward. Prior works at the intersection of IRL and BC include policy matching [50], policy-based GAII. classifiers [51], annealing between BC and GAIL policy [12] and discriminator- weighted BC [52]. Our approach inspired by gradient-based IRL. [20, 36, 21], which avoids the nume-theoretic objective and instead estimates the reward by analyzing the policy update steps with respect to the BC policy. An entropy-regularin
GIRL setting was investigated in the context of learning from policy updates [53]. For further discussion on related work, see Appendix A.
3 Coherent imitation learning
For efficient and effective imitation learning, we would like to combine the simplicity of behavioral cloning with the structure of the MDP, as this would provide a means to initialize and refine the imitating policy through interactions with the environment. In this section, we propose such method using GiRI--nsppred connection between BC and IRL in the entropy-regularzeed RT. setting. By inverting the entropy-regularivee policy update in Equation 4, we derive a reward for which the behavioral-cloning policy is optimal, quality refer to coherence.
3

---

Inverting soft policy iteration and reward shaping. The a a-regularized closed- form policy based on Equation 4 can be inverted to provide expressions for the critic and reward (Theorem 1).
Theorem 1. (KL-regularized policy improvement inversion). Let p and qa be the prior and pseudo- posterior policy given by posterior policy iteration (Equation 4). The critic can be expressed as
9a (a
8) + Va(s), Va(s) = =a /x exp (10(s.a) p(a)s) da.
((s. a) log
(6)
p(a s)
Substituting into the KL-regularized Bellman equation lower-bound from Equation 2,
r(s, a) log Ga (a 8) + Va(s) - -y -7Bgs)(ss.)))) (a
p(a 8)
The (s) term is the 'soft' value function. We assume (a (8) = O whenever p(a 8))
For the proof, see Appendix K.2. Jacq et al. [53] and Cao et al. [54] derived similar inversion but for a maximum entropy policy iteration between two policies, rather than PPI between the prior and posterior. These expressions for the reward and critic are useful as they can be viewed as a valid form 8)) is a shaped 'coherent' reward.
of shaping (Lemma 1) [54, 53], where alog q(a 8) - logp(a
Lemma 1. (Reward shaping. Ng et al. [55]). For a reward function r S x A + R. with optimal policy π E IL. for a bounded state-based potential function V/ S - R. a reward function shaped by the potential function satisfying r(s, a) = f(s, a) + (s) - PPlllla [ (s') has shaped critic Q satisfying ((s. a) = ②(s,a) a) (s) and the same optimal policy the original reward.
Definition 1.
The shaped 'coherent' reward and critic are derived from the log policy ratio by combining Lemma I and Theorem 1, with value function (s( as the potential (s). When policy (als) models the data D while matching its prior p otherwise, the density ratio should exhibit the following shaping
0.9
0.6
0.3
0.0
0.3
2 0ifs,
i(s. a) log aa s)
if D.
The coherent reward from Figure 1, made using a stationary Gaus- sian process policy, extended out-of- distribution. The reward approximates the shaping described in Definition 1.
Figure 2:
p(a 8) = 0 if D,Va
In a continuous selting, the learned policy should cap- ture this shaping approximately (Figure 2).
Coherent soft imitation learning (CSIL,, Algorithm 1) uses the BC policy to initialize the coherent reward and uses this reward improve the policy further with additional interactions outside of D. However, for the coherent reward to match Definition I, the policy g(a s) needs to match the policy prior p(a s) outside of the data distribution. To achieve this, it is useful to recognize the policy {(a) 8) as a "pseudo-posterior" of prior p(a 8) and incorporate this Bayesian view into the BC step.
Imitation learning with poauddopssteioos.. The "rceudo-potteriors formalism [35, 56-59] is a generalized way of viewing the policy derived for ME-IRL and KL-regularized RL in Equation 4. Definition 2. Eseudo-posterrin with additional scalar objective vector-valued constraint term and Lagrange multipliers X,
are solutions to KL-constrained or minimum divergence problems
max, (e - A (Dea (1(x)) || p(x)] c)
(x(x:) x exp(X-1 f(x)) p(x).
ming (ell((#) 11 p(x)] - X (f (")
(x(x) exp(A T f()))
The exp(AT f(x)) term is an unnormalized -eeudolikelihood"; it facilities Bayesian inference to solve a regularized optimization problem specified by f.
Optimizing the policy distribution, the top objective captures KL-regularized RL, where f is the return, and the regularization is implemented constraint with bound or soft penalty with constant X. The bottom objective is the form seen in ME-IRL (Equation 1), where f is the feature space that defines the reward model. Contrasting these e.g., [60, 61], we can compare the Bayesian likelihood used for regression with the critic- based pseudo-likelinn likeeooodd obtained from imitation learning. The pseudo-likelihood in entropy-regularize II. methods produce an effective imitating policy by incorporating the MDP and trajectory distribution because / captures the cumulative reward, compared to just the action prediction error typically captured by BC regression likelihoods. This point is expanded in Appendix B.
policies against BC with Gaussian processes,

---

Algorithm 1: Coherent soft imitation learning (CSIL)
Data: Expert demonstrations DD initial temperature a, refinement temperature, B, parametric policy class (a(a 8), prior policy p(a 8), regression regularizer V, iterations N Result: ςe (a s), matching or improving the initial policy qo, qo,
Train initial policy from demonstrations, 01 = arg maxo L6, a= [Log (a (()):
Define fixed shaped coherent reward, Te, (s, a) a<log qo, (a 8) log 8));
Choose reference policy (a s) to be p(a s) or qe, (a s) and initialize the shaped critic,
( ( = to, (s. a) + 7 (s', a²) (log qo, (a' |s*) log :Taa | a, ,s', a' ~ D; for n + N do
// finetune policy with reinforcement learning Compute 2. and qo, using soft policy iteration (Section 2), e.g. SAC, with temperature 3. end
Behavior cloning with three main implications when performing behavior cloning and designing the policy and prior:
Viewing the policy q(a 8) as a
has
1. We perform conditional density estimation to maximize the posterior predictive likelihood, rather than using a data likelihood, which would require assuming additive noise [62, 33].
2. We require hypothesis space p(a 8, w,,,), e.g., tabular policy or function approximator, that is used to define both the prior and posterior through weights w.
3. We need fixed weight prior p(w) that results in a predictive distribution that matches the desired policy prior p(a s) = p(a w w)p(w) dw Vs E S.
These desiderata are straightforward to achieve for maximum entropy priors in the tabular setting, where count-based conditional density estimation is combined with the prior, e.g., p(a s) Ua(a). Unfortunately, point 3 is challenging in the continuous setting when using function approximation, as shown in Figure 3. However, we can adopt ideas from Gaussian process theory to approximate such policies, which is discussed further in Section 4. The learning objective combines maximizing the likelihood of the demonstrations w.r.t. the predictive distribution, as well as KL regularization against the prior weight distribution, performing regularized heteroscedastic regression [63].
maxe
Ea..- (log ge s)] A Dx (xa(1((e)) (w)) || p(w)), (a(a 8) = p(a 88 w, go (ur) dw. (8)
This objective has been used to fit parametric Gaussian processes with a similar motivation [64].
We now describe how this BC approach and the coherent reward are used for imitation learning (Algorithm 1) when combined with soft policy iteration algorithms, such as SAC.
On a high level, Algorithm can be summarized by the following steps,
1. Perform regularized BC on the demonstrations with a parametric stochastic policy (a | s).
2. Define the coherent reward, ro 8; a.) = alog (a s)- log 8)), with temperature
3. With temperature 3 < a, perform SPT to improve on the BC policy with the coherent reward.
The reduced temperature 3 is required to improve the policy after inverting it with temperature a.
This coherent approach contrasts combining BC and prior actor-critic IRI. methods, where learning the reward and critic from scratch can lead to 'unlearning" the initial BC policy [11]. Moreover, while our method does not seek to learn the true underlying reward, but instead opts for one shaped by the environment used to generate demonstrations, we believe this is a reasonable compromise in practice. Firstly, CSIL uses IRL to tackle the covariate shift problem in BC, as the reward shaping (Definition 1) encourages returning to the demonstration distribution when outside it, which is also the goal of prior imitation methods, e.g., [23, 24]. Secondly, in the hattopy-regularizee setting, an IRL method has the drawback of requiring demonstrations generated from two different environments or discount factors to accurately infer the true reward [54], which is often not readily available in practice, e.g., when teaching a single robot a task. Finally, additional MIPS could be used to estimate the true (unshaped) reward from data [65, 53] if desired. We also provide divergence minimization perspective of CSIL in Section J of the Appendix. Another quality of CSIL is that conceptually simpler than alternative IRL approaches such as AIRL. As we simply use a shaped estimate of the true reward with entropy-regullrizee RL, CSIL inherits the theoretical properties of these regularized algorithms [17, 66] regarding performance. As a consequence, this means that analysis of the imitation quality requires analyzing the initial behavioral cloning procedure.
5

---

Desired
Heteroscedastic MLP
Stationary Heteroscedastic MLP
Figure 3: For the coherent reward to effective, the stochastic policy should return to the prior distribution outside of the data distribution (left). The typical heteroscedastic parametric policies have undefined of--siiiiibttti behavior and typically collapse to the action limits due to the network extrapolation and tanh transformation (middle). By approximating stationary Gaussian processes, we can design policies that exhibit the desired behavior with minimal network modifications (right).
Partial expert coverage, minimax optimization, and regularized regression. The reward shaping theory from Lemma shows that an advantage function can be used as a shaped reward function when the state potential is value function. By inverting the soft policy update, the log policy ratio acts form of its advantage function. However, in practice, we do not have access to the complete advantage function but rather an estimate due finite samples and partial state coverage. Suppose the expert demonstrations provide only partial coverage of the state-action space. In this case, the role of an inverse reinforcement learning algorithm to use additional knowledge of the MDP, e.g., online samples, dynamics model, or an offline dataset, to improve the reward estimate. As CSIL uses the log policy ratio learned only from demonstration data, how can be a useful shaped reward estimate? We show in Theorem 2 that using entropy regularization in the initial behavioral cloning step plays a similar role to the saddle-pon point optimization in game- theoretic IRL.
Theorem (Coherent inverse reinforcement learning as KL-regularized behavioral cloning). A KL- regularized game came-theoretic IRL objective, with policy go s) and coherent reward parameterization 8)) where 2 0, is lower bounded by a scaled KL-regularized behavioral cloning objective and a constant term when the optimal policy, posterior, and prior share a hypothesis space p(a| s, w) and finite parameters w.
ro(s, a) a<log go (a s)
E Ex [r)(s,a)] Ea-qe (-1s) a- (-) [ro(s,a) a) 3(log (a 8) log a 8))]
a aBD [log (a 8)] A Dx
p(w)] + Es.a-D [logp(a s)])-
where x x= (a and E, EsDaND [logp(a s)] is constant. The regression objective bounds the
IRL objective a worst-case on on-policy state distribution (s)ss,, which motivates its scaling through RL finetuning KL regularization is required, 3 and
3. D has sufficient coverage,
0 Df does not have sufficient coverage, then let B < a so > to regularize the BC fit and finetune the policy with RI. accordingly with additional soft policy iteration steps.
The proof is provided in Appendix K.3. In the tabular setting, it is straightforward for the cloned policy reflect the prior distribution outside of the expert's data distribution. However, this behavior is harder to capture in the continuous setting where function approximation is typically adopted.
4 Stationary processes for continuous control policies
Maximum entropy methods used in reinforcement learning can be recast as a minimum relative entropy problem against a regularizing prior policy with uniform action distribution, i.e., prior p(a s) = (a) V E S, where UA is the uniform distribution over A. Achieving such a policy in the tabular setting is straightforward, as the policy can be updated independently for each state. However, such a policy construction is far more difficult for continuous states due to function approximation capturing correlations between states. To achieve the desired behavior, we can use stationary process theory (Definition 3) to construct an appropriate function space (Figure 3).
Definition 3. (Stationary process, Cox and Miller (671). A process f : X + stationary if its joint distribution in y E V p(y1..... ...). is constant w.r.f. æ1..... In X and all n E N
To approximate a stationary policy using function approximation, Gaussian process (GP) theory provides a means using features in a Gaussian linear model that defines a stationary process [62]. To approximate such feature spaces using neural networks, this can be achieved through a relatively wide final layer with a periodic activation function (Sper,, which can be shown to satisfy the stationarity property [68]. Refer to Appendix C for technical details. To reconcile this approach with prior work such SAC, use the predictive distribution of our Gaussian process in lieu of network directly

---

predicting Gaussian moments. The policy is defined as = tanh(=(s)), where z(s) = W (()s).
The weights are factorized row-wise W = [.:...., wa,]" we = [ Σ,) to define a GP with independent actions. Using change-of-variables like SAC [17]. the policy is expressed per-action
das
[T.
{(a; s) -N (z: ((s), ((s) Σ, ((s)) det
<(s) Sper (W pmip( (s))-
dz;
W are weights drawn from a distribution, e.g., a Gaussian, that also characterizes the stochastic process and Qmlp is arbitrary MLP. We refer this heteroscedastic stationary model as HETSTAT.
Function approximation necessitates several additional practical implementation details of CSIL.
Approximating and regularizing the critic. Theorem and Lemma show that the log policy ratio is also shaped critic. However, we found this model was not expressive enough for further policy evaluation. Instead, the critic is approximated using as a feedforward network and pre-trained after the policy using SARSA [8] and the squared Bellman error. For coherency, a useful inductive bias is to minimize the critic Jacobian w.r.t. the expert actions to approximate first-order optimality, i... ming E lV 2o a)). as an auxiliary critic loss. For further discussion, see Section H in the Appendix. Pre-training and regularization are ablated in Figures 44 and 46 in the Appendix.
Using the cloned policy as prior. While uniform prior is used in the CSIL reward throughout, in practice, we found that finetuning the cloned policy with this prior in the soft Bellman equation (Equation 3) leads to divergent policies. Replacing the maximum entropy regularization with KL regularization against the cloned policy, i.c., DDx [go (a) s) qe, (a 8)]. mitigated divergent behavior. This regularization still retains maximum entropy effect if the BC policy behaves like stationary process. This approach matches prior work on KL-regularized RL from demonstrations, e.g., [69, 61].
'Faithful' heteroscedastic regression loss. To fit a parametric speudo-posterion the expert dataset, we use the predictive distribution for conditional density estimation [64] using heteroscedastic regression [63]. A practical issue with heteroscedastic regression with function approximators is the incorrect modeling of data as noise [70]. This can be overcome with a 'faithful' loss function and modelling construction
[71], which achieves the desired minimization of the squared error in the mean and fits the predictive variance to model the residual errors. For more details, see Appendix D.
Refining the coherent reward. The HETSTAT network use still approximates stationary process, as shown in Figure 3. To ensure spurious reward values from approximation errors are not exploited during learning, it can be beneficial to refine, in minimax fashion, the coherent reward with the additional data seen during learning. This minimax refinement both reduces the stationary approximation errors of the policy, while also improving the reward from the game-theoretic IRL perspective. For further details, intuition and ablations, see Appendix E, G and N.6 respectively.
Algorithm 1 with these additions be found summarized in Algorithm 2 in Appendix I. An extensive ablation study of these adjustments can be found in Appendix N.
5 Experimental results
We evaluate CSIL against baseline methods on tabular and continuous state-action environments. The baselines are popular entropy-regularized imitation learning methods discussed in Section Moreover, ablation studies are provided in Appendix N for the experiments in Section 5.2 and 5.3
MDP
Variant
Expert
BC
Classifier
ME-IRL
GAIL
IQLearn
PPIL
CSIL
Dense
Nominal
0.266
0.200
0.249
0.251
0.253
0.244
0.229
0.257
Windy
0.123
0.086
0.103
0.111
0.105
0.104
0.104
0.107
Sparse
Nominal
1.237
1.237
1.237
1.131
1.237
1.237
1.237
1.237
Windy 0.052
0.002
0.044
0.036
0.043
0.002
0.002
0.044
Table 1: Inverse optimal control, combining SPI and known dynamics, in tabular MDPS. The 'dense' MDP has uniform initial state distribution and four goal states. The 'sparse' MDP has one initial state, one goal state and one forbidden state. The agents are trained on the nominal MDP. The "windy' MDP has random disturbance across all states evaluate the robustness of the policy. CSIL performs well across all settings despite being the simpler algorithm relative to the IRI. baselines.
7

---

Ant-v2
Hal fhheethh-hhhhe
Hopper-Ve r-p
Walker2d-v2
"b
squi
Return
iqlearn
0.5
ppil
pwal
0.0
csil
10
30
30
30
30
Demonstrations
Demonstrations
Demonstrations
Demonstrations
Figure 4: Normalized performance of CSIL against baselines for online imitation learning for MuJoCo Gym tasks. Uncertainty intervals depict quartiles over ten seeds. To assess convergence across seeds, performance is chosen using the highest 25th percentile of the episode return during learning.
5.1 Tabular inverse optimal control
We consider inverse optimal control in two tabular environments to examine the performance of CSIL without approximations. One ('dense") has a uniform initial distribution across the state space and several goal states, while other ('sparse") has one initial state, one goal state, and one forbidden state with large negative reward. Table I and Appendix M.1 shows the results for nominal and a 'windy' environment, which has a fixed dynamics disturbance to test the policy robustness. The results show that CSIL is effective across all settings, especially the sparse environment where many baselines produce policies that are brittle to disturbances and, therefore, covariate shift. Moreover, CSI. produces value functions similar GAIT. (Figures 23 and 27) while being simpler algorithm.
5.2 Continuous control from agent demonstrations
standard benchmark of deep imitation learning is learning MuJoCo [72] Gym [73] and Adroit [74] tasks from agent demonstrations. We evaluate online and offline learning, where a fixed dataset is used in lieu of environment interactions [75]. Acme [76] used to implement CSIL and baselines, and expert data was obtained using rlds [77] from existing sources [11. 74, 75]. Returns are normalized with respect to the reported expert and random performance [11] (Table 2).
Online imitation learning. In this setting, we used DAC (actor-critic GAIL), IQLearn, PPIL, SQIL, and PWIL autropyrregularien imitation learning baselines. We evaluate on the standard benchmark of Iocomotion-based Gym tasks, using the SAC expert data generated by Orsini et al. [11]. In this setting, Figures 4 and 28 show CSIL closely matches the best baseline performance across environments and dataset sizes. We also evaluate on the Adroit environments, which involve manipulation tasks with complex 27-dimensional robot hand [74]. In this setting, Figures 5 and 29 show that saddle-point methods struggle due to the instability of the optimization high dimensions without careful regularization and hyperparameter selection [11]. contrast, CSIL very effective, matching surpassing BC, highlighting the benefit of coherency for both policy initialization and improvement. the Appendix, Figure 29 includes SAC from demonstrations (SACfD) [78, 74] as an oracle baseline with access to the true reward. CSIL exhibits greater sample efficiency than SACID, presumably due to the BC initialization and the shaped reward, and often matches final performance. Figures 38, 39 and 40 in the Appendix show ablation of IQLearn and PPIL with BC pre- pre-training. We observe fast unlearning of the BC policy due to the randomly initialized rewards, which was also observed by Orsini et al. 1], any initial performance improvement is negligible or temporary.
door-v8
hamme r- r-v0
sqill
Return
iqlearn
ppil
pwil
0.0
10
Demonstrations
Demonstrations
Figure 5: Normalized performance of CSIL against baselines for online imitation learning for Adroit tasks. Uncertainty intervals depict quartiles over ten seeds. To assess convergence across seeds, performance is chosen using the highest filtered 25th percentile of the return during learning.

---

Ant-v2 v2
Hal SalfCheetah-Ve
Hopper-Ve
Walker2d-Ve
be (demos)
iqlear
Return
ppil demodice
smodice
0.0
30
10
30
Demonstrations
Demonstrations
Demonstrations
Demonstrations
Figure 6: Normalized performance of CSIL against baselines for offline imitation learning for Gym tasks. Uncertainty intervals depict quartiles over ten seeds. To assess convergence across seeds, performance is chosen using the highest filtered 25th percentile of the episode return during learning.
Offiine imitation learning. Another challenging setting for deep is offline learning, where the online interactions are replaced with a static 'supplementary" dataset. Note that prior works have used 'offline" learning to deseribe the setting where only the demonstrations
used for learning [41, 79, 42, 43), like in BC, such that the II. method becomes a form of regularized BC [80]. Standard offline learning is challenging primarily because the critic approximation can favor unseen actions, motivating appropriate regularization (e.g., [81]). We use the full frelayy datasets from the d4rl benchmark [75] of the Gym locomotion tasks for the offline dataset, the offline and demonstration data is not the same. We also evaluate SMODICE [82] and DemoDICE [83], two offline imitation learning methods which use state- based value functions, weighted BC policy updates, and a discriminator-b beaseid reward. We opt not to reproduce them in acme and use the original implementations. As a consequence, the demonstrations are from a different dataset, but are normalized appropriately. For further details on SMODICE and DemoDICE, see Appendix A and L.
Figures 6 and 30 demonstrate that offline learning is significantly harder than online learning, with no method solving the tasks with few demonstrations. This can be attributed to the lack of overlap between the offline and demonstration data manifesting as a sparse reward signal. However, with more demonstrations, CSIL can achieve reasonable performance. This performance is partly due the strength of the initial BC policy, but CSIL can still demonstrate policy improvement in some cases (Figure 40). Figure 30 includes CQL [81] as an oracle baseline, which has access to the true reward function. For some environments, CSIL outperforms CQL with enough demonstrations. The DemoDICE and SMODICE baselines are both strong, especially for few demonstrations. However, performance does not increase so much with more demonstrations. Since CSII. could also be implemented with state-based value function and weighed BC policy update (e.g.. [16]), further work is needed to determine which components matter most for effective offline imitation learning.
5.3
Continuous control from human demonstrations from states and images
As a more realistic evaluation, we consider learning robot manipulation tasks such as picking (Lift), pick-and-place (PickPlaceCan) and insertion (ButAsssmmlySSuare from random initializations and mixed-quality human demonstrations using the robomimic datasets [84], which also include image observations. The investigation of Mandlekar al. [84] considered offline and various forms of BC with extensive model selection. Instead, we investigate the applicability of imitation learning, using online learning with CSIL an alternative tasks is that they have sparse reward based on success, which can be used to define an absorbing
model selection. One aspect of these
Lift
PickPlaceCan
NutAAsemmbySSuuaee
bc
Success
dac
0.5
iqlearn
esil
Z.h
50
100
200
50
100
200
100
200
Demonstrations
Demonstrations
Demonstrations
Figure 7: Average success rate over 50 evaluations for online imitation learning for robomimic tasks. Uncertainty intervals depict quartiles over ten seeds. To assess convergence across seeds, performance is chosen using the highest 25th percentile of the averaged success during learning.

---

Lift
PickPlaceCan
1,00
Success
0.75
0.50
0.25
0.5
0.75
1,0
0.25
0.5
0.75
1,0
Steps (1 x 10")
Steps x 10")
bc (Mandiekar
be-ma (Mandiekar
be (csil)
Figure 8: Average success rate over 50 evaluations for image-based online imitation learning for robomimic tasks. Uncertainty intervals depict quartiles over five seeds. Baseline results obtained from Mandlekar et al. [84]. BC (CSIL) denotes the performance of CSIL's 'SILss initial policy for comparison.
state, As observed in prior work [40], in practice there a synergy between absorbing states and the sign of the reward, where positive rewards encourage survival and negative rewards encourage minimum-time strategies. We found that CSIL. was initially ill-suited to these goal-oriented tasks as the rewards are typically positive. In the same way that the AIRL reward can be designed for given sign [1I], we can design negative CSIL rewards by deriving an upper-bound of the log policy ratio and subtracting it in the reward. For further details, see Appendix F. An ablation study in Appendix N.9 shows that the negative CSIL reward matches or outperforms a constant negative reward, especially when given fewer demonstrations, These tasks are also more difficult due to the variance in the initial state, requiring much larger function approximators than in Section 5.2.
Figures 7 and 36 show the performance of CST. and baselines, where up to 200 demonstrations are required to sufficiently solve the tasks. CSIL achieves effective performance and reasonable demonstration sample efficiency across all environments, while baselines such as DAC struggle to solve the harder tasks. Moreover, Figure 36 shows CSIL is again more sample efficient than SACED,
Figure 8 shows the performance on CSIL when using image-based observations. A shared CNN torso used between the policy, reward and critic, and the convolutional layers are frozen after the BC stage. CSIL was able to solve the simpler tasks in this setting, matching the model selection strategy of Mandlekar et al. [84], demonstrating its scalability. In the offline setting, Figure 37 shows state-based results with sub-optimal (" ('mixed human') )) demonstrations as supplementary dataset. While some improvement could made on simpler tasks, on the whole it appears much harder to learn from suboptimal human demonstrations offline. This supports previous observations that human behaviour can be significantly different that of RI. agents, such that imitation performance affected [11, 84).
Discussion
We have shown that 'coherency" is effective approach to IL, combining BC with IRL-based finetuning by using a shaped learned reward for which the BC policy is optimal. We have demonstrated the effectiveness of CSIL empirically across range of settings, particularly for high-dimensional tasks and offline learning, due to CSIL leveraging BC for initializing the policy and reward.
In Figure 35 of the Appendix, we investigate why baselines IQLearn and PPIL, both similar to CSIL, struggle the high-dimensional and offline settings. Firstly, the sensitivity of the saddle-point optimization can be observed in the stability of the expert reward and critic values during learning. Secondly, we observe that the policy optimization does not effectively minimize the BC objective by fitting the expert actions. This suggests that these methods do not necessarily converge to a BC solution and lack the 'coherence" quality that CSIL has that allows it to refine BC policies.
A current practical limitation of CSIL is understanding when the initial BC policy is viable as a coherent reward. Our empirical results show that CSIL can solve complex control tasks using only one demonstration, where its BC policy is ineffective. However, if demonstration-rich BC policy cannot solve the task (e.g., image-based MutAsseebbySSuurre) CSIL also appears to be unable to solve the task. This suggests there is scope to improve the BC models and learning to make performance more consistent across more complex environments. An avenue for future work is to investigate the performance of CSIL with richer policy classes beyond MLPS, such as recurrent architectures and multi- modll action distributions, to assess the implications of the richer coherent reward and better model sub-optimal human demonstrations.
10

---

Acknowledgments and Disclosure of Funding
We wish to thank Jost Tobias Springenberg, Markus Wulfmeier, Todor Davchev, Ksenia Konyushkova, Abbas Abdolmaleki and Matthew Hoffman for helpful discussions during the project. We would also like to thank Gabriel Dulac-Aroldd and Sabela Ramos for help with setting up datasets and experiments, Luca Viano for help reproducing PPIL, and Martin Riedmiller, Markus Wulfimeier, Oleg Arenz, Davide Tateo, Boris Belousov. Michael Lutter and Gokul Swarmy for feedback on previous drafts. We also thank the wider Google DeepMind research and engineering teams for the technical and intellectual infrastructure upon which this work is built, in particular the developers of acme. Some baseline experiments were run on the computer cluster of the Intelligence Autonomous Systems group at TU Darmstadt, which maintained by Daniel Palenicek and Tim Schneider. Joe Watson acknowledges the grant "Finrichtung eines Labors des Deutschen Forschungszentrum für Künstliche Intelligenz (DFKI) an der Technischen Universität Darmstadt" the Hessian Ministry of Science and Art.
References
[1] Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, et al. An algorithmic perspective on imitation learning. Foundations and Trends@ in Robotics, 2018.
[2] Stefan Schaal. imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 1999.
[3] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Computation, 1991.
[4] Rudolph E. Kalman. When is linear control system optimal? Journal of Basic Engineering, 1964.
[5] Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
[6] P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning, In International Conference on Machine Learning, 2004.
[7] Martin Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
[8] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[9] Stéphane Ross and Drew Bagnell. A reduction of imitation learning and structured prediction no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2011.
[10] Boyuan Zheng, Sunny Verma, Jianlong Zhou, Ivor W Tsang, and Fang Chen. Imitation learning: Progress, taxonomies and challenges. IEEE Transactions on Neural Networks and Learning Systems, 2022.
[11] Manu Orsini, Anton Raichuk, Léonard Hussenot, Damien Vincent, Robert Dadashi, Sertan Girgin, Matthieu Geist, Olivier Bachem, Olivier Pietquin, and Marcin Andrychowicz. What matters for adversarial imitation learning? In Advances in Neural Information Processing Systems, 2021.
[12] Rohit Jena, Changliu Liu, and Katia Sycara. Augmenting GAIL with BC for sample efficient imitation learning. In Conference Robot Learning, 2020.
[13] Jan Peters, Katharina Mülling, and Yasemin Altün. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, 2010.
[14] Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. In Conference Uncertainty in Artificial Intelligence, 2016.
11

---

[15] Herke Van Hoof, Gerhard Neumann, and Jan Peters. Non Non- parametric policy search with limited information loss. Journal of Machine Learning Research, 2017.
[16] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum Posteriori policy optimisation. In International Conference on Learning Representations, 2018.
[17] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, 2018.
[18] Matthew Fellows, Anuj Mahajan, Tim GJ Rudner, and Shimon Whiteson. VIREL: A variational inference framework for reinforcement learning, Advances Neural Information Processing Systems, 2019.
[19] Joan Bas-Serrang Sebastian Curi, Andreas Krause, and Gergely Neu. Logistic Q-learming, In International Conference on Artificial Intelligence and Statistics, 2021.
[20] Matteo Pirotta and Marcello Restelli, Inverse reinforcement learning through policy gradient minimization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2016.
[21] Giorgia Ramponi, Gianluca Drappo, and Marcello Restelli. Inverse reinforcement learning from a mundient-based learner. In Advances in Neural Information Processing Systems, 2020.
[22] Yi-Feng Zhang, Fan-Ming Luo, and Yang Yu. Improve generated adversarial imitation learning with reward variance regularization. Machine Learning, 2022.
[23] Siddharth Reddy, Anca Dragan, and Sergey Levine. SQIL: Imitation learning via reinforce- ment learning with sparse rewards. In International Conference on Learning Representations, 2019.
[24] Kiante Brantley, Wen Sun, and Mikael Henaff. benagreemmen-rrgguaaree imitation learning. In International Conference Learning Representations, 2020.
[25] Nived Rajaraman, Lin Yang. Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 2020.
[26] Stuart Russell. Learning agents for uncertain environments. In Conference on Computational Learning Theory, 1998.
[27] Umar Syed and Robert E. Schapire. A game-theoretic approach to apprenticeship learning. In Advances in Neural Information Processing Systems, 2007.
[28] Umar Syed, M. Bowling, and R.E. Schapire. Apprenticeship learning using linear program- ming. In International Conference on Machine Learning. 2008
[29] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 2020.
[30] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.
[31] Edwin Jaynes. On the rationale of maximum-entropp methods. Proceedings of the IEEE, 1982.
[32] Joseph Marino, Alexandre Piché, Alessandro Davide lalongo, and Yisong Yue. Iterative amortized policy optimization. In Advances in Neural Information Processing Systems, 2021.
[33] David Barber. Bayesian reasoning and machine learning. Cambridge University Press, 2012.
[34] Konrad Rawlik, Marc Toussaint, and Sethu Vijayakumar. On stochastic optimal control and reinforcement learning by approximate inference. In Robotics: Science and Systems, 2013.
[35] Joe Watson and Jan Peters. Inferring smooth control: Monte Carlo posterior policy iteration with Gaussian processes. In Conference on Robot Learning, 2022.
12

---

[36] Alberto Maria Metelli, Matteo Pirotta, and Marcello Restelli. Compatible reward inverse reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
[37] Jonathon Ho and Stefan Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
[38] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[39] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence min- imization perspective on imitation learning methods. In Conference on Robot Learning. 2019.
[40] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Diseriminatorr-ccoo--rttiie Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
[41] llya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020.
[42] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiamning Song, and Stefano Ermon. IQ- learn: Inverse soft-Q learning for imitation. In Advances in Neural Information Processing Systems, 2021.
[43] Luca Viano, Angeliki Kamoutsi, Gergely Neu, Igor Krawezuk, and Volkan Cevher, Proximal point imitation learning. In Advances in Neural Information Processing Systems, 2022.
[44] Robert Dadashi, Leonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal Wasserstein
imitation learning. In International Conference on Learning Representations, 2021.
[45] Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In International Conference on Artificial Intelligence and Statistics, 2011.
[46] Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv: 100..0007..888 2015.
[47] Oleg Arenz, Hany Abdulsamad, and Gerhard Neumann. Optimal control and inverse optimal control by distribution matching. In IEEE/RSJ International Conference Intelligent Robots and Systems, 2016.
[48] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, 2016.
[49] Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-RRL: Inverse reinforcement learning via state marginal matching. In Conference on Robot Leurning, 2020.
[50] Gergeley Neu and Csaba Szepesvári. Apprenticeship learning using inverse reinforcement learning and gradient methods. In Conference on Uncertainty Artificial Intelligence, 2007.
[51] Paul Barde, Julien Roy, Wonseok Jeon, Joelle Pineau, Chris Pal, and Derek Nowrouzezahrai. Adversarial soft advantage fitting: Imitation learning without policy optimization. In Advances Neural Information Processing Systems, 2020.
[52] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. ceneminnntto-weeihheee offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, 2022.
[53] Alexis Jacq, Matthieu Geist, Ana Paiva, and Olivier Pietquin. Learning from a learner. In International Conference on Machine Learning, 2019.
[54] Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. In Advances in Neural Information Processing Systems, 2021.
13

---

[55] Andrew Y. Ng. Daishi Harada, and Stuart J. Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999.
[56] Jan van Campenhout and Thomas Cover. Maximum entropy and conditional probability, IEEE Transactions on Information Theory, 1981.
[57] Benjamin Guedj. A primer on PAC-Bayesian learning, In Proceedings of the second congress of the French Mathematical Society, 2019.
[58] Pierre Alquier, James Ridgway, and Nicolas Chopin. On the properties of variational approxi- mations of Gibbs posteriors. Journal of Machine Learning Research, 2016.
[59] Jeremias Knoblauch, Jack Jewson, and Theodoros Damoulas. An optimization-centri view on Bayes' rule: Reviewing and generalizing variational inference. Journal of Machine Learning Research, 2022.
[60] Hanbit Oh, Hikaru Sasaki, Brendan Michael, and Takamitsu Matsubara. Bayesian disturbance injection: Robust imitation learning of flexible policies for robot manipulation. Neural Networks, 2023.
[61] Tim GJ Rudner, Cong Lu, Michael Osborne, Yarin Gal, and Yee Teh. On pathologies in KL-regularized reinforcement learning from expert demonstrations. Advances in Neural Information Processing Systems, 2021.
[62] Carl E. Rasmussen and Chris Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[63] Quoc V. Le, Alex J. Smola, and Stéphane Canu. Heteroscedastic Gaussian process regression. In International Conference Machine Leurning, 2005.
[64] Martin Jankowiak, Geoff Pleiss, and Jacob Gardner. Parametric Gaussian process regressors.
In International Conference on Machine Learning, 2020.
[65] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[66] Matthicu Geist, Bruno Scherrer, and Olivier Pictquin. A theory of regularized Markov decision processes. In International Conference on Machine Learning, 2019.
[67] David Roxbee Cox and Hilton David Miller. The theory of stochastic processes. CRC press, 1977.
[68] Lassi Meronen, Martin Trapp, and Arno Solin. Periodic activation functions induce stationarity. Advances in Neural Information Processing Systems, 2021.
[69] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp. Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations, 2020.
[70] Maximilian Seitzer, Arash Tavakoli, Dimitrije Antic, and Georg Martius. On the pitfalls of heteroscedastic uncertainty estimation with probabilistic neural networks. In International Conference on Learning Representations, 2022.
[71] Andrew Stirn, Hans-Hermann Wessels, Megan Schertzer, Laura Pereira, Neville E. San- jana, and David A. Knowles. Faithful heteroscedastic regression with neural networks. In International Conference on Artificial Intelligence and Statistics, 2023.
[72] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for model-based based control. In IEEE International Conference on Intelligent Robots and Systems, 2012.
[73] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. arXiv preprint arXiv:1606.015400 2016.
14

---

[74] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In Robotics: Science and Systems, 2018.
[75] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprini arXiv:2004.07219, 2020.
[76] Matthew W Hoffman, Bobak Shahriari, John Aslanides, Gabriel Barth-Maron, Nikola Mom- chev, Danila Sinopalnikov, Piotr Stañczyk, Sabela Ramos, Anton Raichuk, Damien Vincent, et al. Acme: A research framework for distributed reinforcement learning. arXiv preprint urXiv:2006.0099 00979, 2020.
[77] Sabela Ramos, Sertan Girgin, Léonard Hussenot, Damien Vincent, Hanna Yakubovich, Daniel Toyama, Anita Gergely, Piotr Stanczyk, Raphael Marinier, Jeremiah Harmsen, et al. Rlds: An ecosystem to generate, share and use datasets in reinforcement learning. arXiv preprint arXiv:2111.02767, 2021.
[78] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang. Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiv preprint arXiv:1707.08811 2017.
[79] Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods. arxiv preprint arxiv:2008. 03522, 2020.
[80] Li Ziniu, Xu Tian, Yu Yang. and Luo Zhi-Quan. Rethinking ValueDice - Does it really improve performance? ICLR Blog Track, rethinking-valuedice. 2022.
tracgg-trakk.... github. 1o/2022/03/25/
[81] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 2020.
[82] Yecheng Ma, Andrew Shen, Dinesh Jayaraman, and Osbert Bastani. Versatile offline imitation from observations and examples via regularized state-occupancy matching. In International Conference on Machine Learning, 2022.
[83] Geon-Hyeong Kim, Seokin Sco, Jongmin Lee, Wonseok Jeon, HycongJoo Hwang, Hongseok Yang. and Kee-Eung Kim. DemoDICE: Offline imitation learning with supplementary imper- fect demonstrations. In International Conference on Learning Representations, 2022.
[84] Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang. Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, 2021.
[85] Konrad Zoina, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. In Offline Reinforcement Learning Workshop. NeurIPS, 2020.
[86] Mare Peter Deisenroth, Gerhard Neumann, and Jan Peters. A survey on policy search for robotics. Foundations and Trends in Robotics, 2013.
[87] Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. LSSIQ: Implicit reward regularization for inverse reinforcement learning. In International Conference on Learning Representations, 2023.
[88] Andrew Szot, Amy Zhang, Dhruv Batra, 7solt Kira, and Franziska Meier. BC-IRL: Learning generalizable reward functions from demonstrations. In International Conference Learning Representations, 2023.
[89] Aleksandar Taranovic, Andras Gabor Kupesik, Niklas Freymuth, and Gerhard Neumann. Adversarial imitation learning with preferences. International Conference on Learning Representations, 2022.
15

---

[90] Gokul Swamy, Nived Rajaraman, Matt Peng, Sanjiban Choudhury, J Bagnell, Steven Z Wu, Jiantao Jiao, and Kannan Ramchandran. Minimax optimal online imitation learning via replay estimation. Advances in Neural Information Processing Systems, 2022.
[91] Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representattinss 2021.
[92] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Interna- tional Joint Conference on Artifical Intelligence, 2007.
[93] Jaedeug Choi and Kee-Eung Kim. MAP inference for Bayesian inverse reinforcement learning. Advances in Neural Information Processing Systems, 2011.
[94] Giuseppe Da Prato and Jerzy Zabezyk. Stochastic equations in infinite dimensions. Cambridge University Press, 2014.
[95] Chris Chatfield. The analysis of time series: An introduction. Chapman and Hall, 1989.
[96] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In Advances in Neural Information Processing Systems, 2020.
[97] John Schulman. Approximating KL divergence. html, 2020.
[98] Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational Bayesian neural networks. International Conference on Learning Representations, 2019.
[99] Joe Watson, Jihao Andreas Lin, Pascal Klink, and Jan Peters. Neural linear models with functional Gaussian process priors. In Third Symposium on Advances in Approximate Bayesian Inference, 2021.
[100] Div99. IQ-Learn implementation.
2021.
[101] Yihong Wu. Lecture notes for ECE598YW: Information-thhorrtii methods in high-dimensional statistics. Yale University, 2016.
[102] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Mille and Qiao Zhang. JAX: Composable transformations of Python+NumPy programs, 2018. URL bttp://githun com/ com//oooge/jaz.
[103] Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, lurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Laurent Sartran, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Miloš Stanojević, Wojeiech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http://gi http:///gitubt thtbbbtthhu.... com/ deepmind.
[104] lviano. Proximal point imitation learning implementation. http://tww...ttppp/ggggg.aoaa//a 12i1,2022
[105] geon-hyeong. DemoDICE implementation. imitation-dice, 2022.
httpps//ggihubb.cm//eeahhhpppppp
[106] JasonMa2016. SMODICE implementation. https://githb..coo///aaaae SMODICE, 2022.
[107] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed RL with importance weighted actor-learner architectures. International conference on machine learning, 2018.
16

---

A Extended related work
This section discusses the relevant prior literature in more breadth and depth.
Generative adversarial imitation learning (GAIL) [37] combines the game-theoretic IRL. objective with causal entropy regularization and convex regularization of the reward. This primal optimization problem can be transformed into a dual objective, which has the interpretation of minimizing a diver- gence between the expert and imitator's stationary distributions, where the divergence corresponds to the choice of convex reward regularization. Choosing regularization that reduced the reward to classification problem corresponds to the -ensen-Shannon (JS) divergence. While GAIL used on- policy RL, DAC [40] used off- off-policy actor-critic methods (e.g., SAC) in an ad hoc fashion for greater sample efficiency. ORIL [85] adopts a classification-beit based reward in the offline setting. Kostrikov et al. [40] also note that the GAIL classifier can be used in different forms to produce rewards with different forms. AIRL [65] highlight that the GAIL reward is shaped by the dynamics if the MDP following the shaping theory of Ng and Russell [5], and use additional approximators that disentangle the shaping terms. NAIL [79] replaces the minimax nature of adversarial learning with a more stable max-max approach using expectation maximization-Hi like optimization on lower-bound of the KL divergence between policies. Its reward function is defined, as with AIRL, as the log density ratio of the stationary distributions of the expert and policy, regularized by the causal entropy. Therefore, NAIL and CSIL have similar reward structures; however, while CSIL'S reward can be obtained from BC and prior design, NAIL requires iterative density ratio estimation during learning. Chasemipour et al. [39] investigate the performance of AIRL with different f-divergences and corresponding reward regularization beyond JS such as the forward and reverse KL divergence. Ghasemipour et al. [39] discuss the connection between the log density ratio, AIRL, GAIL and their proposed methods. Ghasemipour et al. [39] also connect BC IRL, but from the divergence perspective rather than the likelihood perspective, as BC can be cast as the conditional KL. divergence between the expert and agent policies.
separate line of study uses the convex duality in a slightly different fashion for 'discounted distribu- tion correction estimation' (DICE) methods. ValueDICE uses the Donsker- Varadhan representation to transform the KL minimization problem into an alternative form and a change of variables to represent the log ratio of the stationary distributions as with an implicit representation using the Bellman operator. However, this alternative form is still a saddlc-poin point problem like GAIL, but the final objective is more amenable off- off-policy optimization. DemODICE [83] considers the offline setting and incorporates KL. Kl-regularizztion into the DICE framework, simplififying optimization. Due to the offline setting, the reward is obtained using a pre-trained classifier. Policy improvement reduces to weighted BC, like in many KL-regularized RL methods [86, 16]. SMoDICE [82] is closely related to DemODICE, but focuses on matching the state distribution and, therefore, does not rely on expert actions, so the discriminator-basee reward depends only on the state.
Donsker-Varadhan
Inverse soft Q-learning (IQLeam) [42] combines the game- theoretic IRL objective, untropy-regularee actor-critic (SAC), convex reward regularization and implicit rewards to perform IRL using only the critic by maximizing the expert's 's implicit reward. While Garg et al. [42] claim their method is non-adversarial because it does not use a classifier, it is still derived from the game-thherede objective. For continuous control, the IQLearn implementation requires shaping the implicit reward using online samples, making learning not so different to fAIL-based algorithms. Moreover, the implementation replaces the minimization of the initial value with the value evaluated for the expert samples. For SAC, in which the value function is computed by evaluating the critic with the current policy, this shapes the critic have local maximimum around the expert samples. We found this implementation detail crucial to reproduce the results for a few demonstrations and discuss further in Appendix H.
Proximal point imitation learning (PPIL) [43] combines the game-theoreen and linear programming forms of IRL and entropy-regularize actor-critic RL to derive an effective method with convergence guarantees. Its construction yields single objective (the dual) for optimizing the reward and critic jointly in a more stable fashion, rather than alternating updates like in DAC, and unlike IQLearn it has an explicit reward model. For continuous control, its implementation builds on SAC and IQLearn, including the aforementioned tricks, but with two key differences: an explicit reward function and minimization of the 'logistic", rather than squared, Bellman error [19].
Compared to the works discussed above, CSTL has two main distinctions. One is the reward hypothesis space in the form of the log policy ratio, derived from KL-regularized RL, as opposed to a classifier-
17

---

based reward or pitic-as-classing implicit reward. Secondly, the consequence of this coherent reward that can be pre-trained using BC, and the BC policy can be used reference policy. While BC pre-training could be incorporated into prior methods in an ad-hoc fashion, their optimization objectives do not guarantee that coherency is maintained. This is demonstrated in Figure 35, where eQrearn and PPIL combined with BC pre-training and KI. regularization do not converge like CSII.
Concurrent work, least-squares inverse Q-learning (LS-IQ) [87]. addresses the undocumented regular- ization of IQLearn by minimizing a mixture divergence, which yields the desired regularization terms and better treatment of absorbing states. The treatment moves away from the entropy-regularrie regularized RL view and instead opts for explicit regularization critics.
Szot et al. [88], in concurrent work, use the bi-level optimization view of IRL to optimize the maximum entropy reward through an upper-level behavioral cloning loss using 'meta' gradients. This approach achieves coherency through the top-level loss while regularizing the learned reward. However, this approach is restricted to on-policy RI. algorithms that directly use the reward rather than actor-critics methods that use a Q function, making less sample efficient.
Brantley et al. [24] propose policies via BC and uses their predictive variance to define a proxy reward. This work shares the same motivation as CSIL in achieving coherency and tackling the covariate shift problem. Moreover, using ensembles is similar CSIL'S 's stationary policies, both models are inspired by Bayesian modeling and provide uncertainty quantification. Unlike CSIL, the method is not motivated by reward shaping, and the reward is obtained by thresholding the predictive variance to +1 rather than assume that the likelihood values are meaningful.
imitation learning, which trains an ensemble of
Taranovic et al. [89] derive ATRL formalation from the KI. divergence between expert and policy that results in a reward consisting of log policy ratios. However, they use a classifier to estimate these quantities and ignore some policy terms in the implementation.
Swamy et al. [90] use the BC policy for replay estimation [25] and a membership classifier augment the expert dataset used during IRL.
'Noisy' BC [91] is tangentially related to CSIL, as it uses the BC log-likelihood as proxy reward to finetune the BC policy using policy gradient updates when learning from demonstrations that are sub-optimal due additive noise.
Soft Q-imitation learning (SQIL) [23] uses binary rewards for the demonstration and interaction transitions, bypassing the complexity of standard IRL. The benefit of this approach comes from the critic learning to overcome the covariate shift problem through the credit assignment. This approach has no theoretical guarantees, and the simplistic nature of the reward does not encourage stable convergence in practice.
Barde et al. [51] proposed adversarial soft advantage fitting (ASAF), an adversarie approach that constructs the classifier using two policies over trajectories or transitions. This approach is attractive as the classification step performs the policy update, so no policy evaluation step is required. While this approach has clear connections regularized behavioral cloning, it's uncertain how the policy learns to overcome covariate shift without any credit assignment. Moreover, this method has no convergence guarantees.
CSIL adopts Bayesian-style approach to its initial BC policy by specifying a prior policy but uses the predictive distribution rather than likelihood to fit the demonstration data. This is in contrast to Bayesian IRL [92]. where the prior is placed over the reward function as a means of reasoning about its ambiguity. As the reward is an unobserved abstract quantity, the exponentiated critic is adopted as a
and approximate inference is required to estimate the posterior reward. Using a pseudo- seeuddo-lkkliine likelihood means that Bayesian IRL. and ME-IRL are not so different in practice, especially when using point estimates of the posterior [93].
18

---

B Likelihoods and priors in imitation learning
The pseudo-posterior perspective in this work was inspired by van Campenhout and Cover [56], who identify that effective likelihoods arise from relevant functions that describe a distribution. With this in mind, we believe it useful see IRL as extending BC from myopic, state- ieeee-nddeeeenntt likelihood to one that encodes the causal, state-dependent nature of the problem as done ME-IRL, essentially incorporating the structure of the MDP into the regression problem [50]. This perspective has two key consequences. One is the importance of the prior, which provides important regularization. For example, Ziniu al. [80] have previously shown that the ValueDICE does not necessarily outperform BC as previously reported but matches BC with appropriate regularization. Moreover, the MIMIC-EXP formulation of BC of Rajaraman et al. [25] proposes a BC policy with a stationary prior,
(p 8) if E D, where pp denotes conditional density estimation of D,
T a 8)
(a) otherwise.
This work shows how policies like these can be implemented for continuous control using stationary architectures.
The second consequence is the open question of how to define effective likelihoods for imitation learning. ME-IRL and other prior works use feature matching, which, while an appropriate constraint, raises the additional question of which features are needed. Using a classifier in AIRL has proved highly effective, but it not always straightforward to implement in practice without regularization and hyperparameter tuning. Much of IQLearn's empirical success for continuous tasks arises from shaping the likelihood via the critic through the expert and on-policy regularization terms.
This work addresses the likelihood design problem through coherence and the inversion analysis in Theorem 1, which informs the function-approximacte
critic regularization described in Section 4. An open question is how these design choices could be further improved, as the ablation study in Figure 46 shows that the benefit of the critic regularization is not evident for all tasks.
C Parametric stationary processes
The effectiveness of CSIL is down to its stationary policy design. This section briefly summarizes the definition of stationary processes and how they can be approximated with MLPS following the work of Meronen et al. [68].
While typically discussed for temporal stochastic processes whose input is time, stationary processes (Definition 3) are those whose statistics do not vary with their input. In machine learning, these processes are useful as they allow correlations (e.g., smoothness) across inputs, which can be used for regularization in regression and classification [62]. In the continuous setting, stationarity can be defined for linear models, i.c.. w Φ(z), w ~ Nu, Σ), through their kernel function (C(æ,, x') ((x) "(x'). the feature inner product. For stationary kernels, the kernel depends only on a relative shift r E IRd between inputs, i.e. K(r)= (xx)) o(x r). By combining the kernel with the weight prior of the linear model, we define the covariance function, e.g. C(r) o(x) Σ <(x + r). Boschner's theorem [94] states that the covariance function of a stationary process can be represented as the Fourier transform of positive finite measure. If the measure has a density, it is known as the spectral density S(w), and is a Fourier dual of the covariance function if the necessary conditions apply, known as the Wieener-Khinhhi theorem [95].
design priors that consistent per input but capture
C(r) = (2 [ S(w) exp(iw Tr) rr, dr, S(w) / J C( sisp(-iw> - Tr) dw.
(9)
To use this theory to design stationary function approximators, we can use the Wiener-Khintche theorem to design function approximators that produce Monte Carlo approximation of stationary kernels. Firstly, we consider only the last layer of the MLP, such that it can be viewed as a linear model. Secondly, Meronen et al. [68] describe several periodic activation functions that produce products of the form exp(iw) T r), such that the inner product the whole feature space performs a Monte Carlo approximation of the integral. We implemented sinusoidal, triangular, and periodic ReLUU activations (see Figure of Meronen et al. [68]) as a policy architecture hyperparameter. Thirdly, the input into the stationary kernel approximation should be small (e.g., < 20), as the stationary behavior
19

---

Heteroscedastic MLP
Heteroscedastic MLP (Faithful) Train MSE 0.05
Stationary Heteroscedastic MLP (NLLH)
Stationary Heteroscedastic MLP (Faithful) Train MSE 0.02
(NLLH)
Train MSE 0.11
Train MSE 0.04
Figure 9: We repeat the regression problem from Figure 3 to highlight the effect of 'faithful' heteroscedastic regression. The NLLH objective has tendency model 'signal" noise, which is most clearly seen in the heteroscedastic MI.. The faithful heteroscedastic loss reduces the mean prediction error significantly without affecting uncertainty quantification for both models.
dictated by r, and this distance becomes less meaningful in higher dimensions. This means the MLP feature space that feeds into the last layer should be small or have "bottleneck" architecture to compress the internal representation. Finally, we require a density to represent the spectral density, Gaussian, but other distributions
which also defines the nature of the stationary process. We such as a Student-t or Cauchy could be used if 'rougher' processes are desired. In summary, the last layer activations take the form f (W ((x)), where fp is the periodic activation function of choice, W are weights sampled from the distribution choice and (xxx) is an arbitrary (but bottlenecked) MI.P feature space of choice.
Related work combining stationary kernel approximations with MLPs have constrained the feature space with residual architecture and spectral normalization bound the upper and lower Lipschitz constant [96]. to prevent these features overlitting. We found we did not need this to achieve approxi- mate stationarity (e.g., Figure 3), which avoided the need for input-dependent policy architectures which may lead to risk of underfitting.
A crucial practical detail is that the sampled weights W should be trained and not kept constant, the theory suggests. practice, we did not observe the statistics of these weights changing significantly during training, while training this weight layer reduced underfitting enough for substantially more effective BC performance.
D Faithful heteroscedastic regression
This section summarizes the approach of Stirn et al. [71]. The faithful objective combines the mean squared error (MSE) and negative log-likelihood NELH loss with careful construction of the heteroscedastic model. The loss is
C(D,0) Ex,anD [(a = log q(a s)] ğ(a s) (N(sg(Hoss)) (s)),, (10)
where sg(-) denotes the stop gradient operator. Moreover, for the models 'shared torso' (e.g.. features), the gradient is stopped between the torso and the predictive variance so that the features are only trained to satisfy the MSE objective. Figure 9 illustrates the impact of the alternative loss function and model adjustments. Note that the faithful objective is not strictly necessary for BC or CSIL and is mitigating potential under-fitting fitting previously observed empirically. The issue of modeling "signal" as noise when using heteroscedastic function approximators poorly understood and depends on the model architecture, initialization, and the dataset considered. While motivated by ID toy examples, was observed during development that the faithful objective beneficial for CSIL's downstream performance.
20

---

E Reward refinement and function-space relative entropy estimation
In the function approximation setting, we refine the reward model with additional non-expert samples. Motivated by Theorem 2, we maximize the objective using samples from the replay buffers
go 8) p(a 8)
to (a p(a 8)
J, ( F2 aa log
Ex.earp log
(11)
If the behavior policy go(a 8), we can view the second term as a Monte Carlo estimation of the relative KL divergence between q and This estimator can suffer greatly from bias, as the KL should always be non-negative. To counteract this, we construct an unbiased, positively-constranned estimator following Schulman [97]. using ([(() -| p((x) dæ and log(ar) S
[x((((>) p(==)] ( log R(x)] Bx [R(x)) -1 log R(x)),
This estimator results in replacing the second term of Equation 11 with
Bg (so a)] - Es,avpa (ro a) a + (0)
This objective is maximized concurrently with policy evaluation, using samples from the replay buffer. This reward refinement is similar behavioral cloning via function-space variational inference (e.g.. [98, 99]), enforcing the prior using the prior predictive distribution rather than the prior weight distribution.
In practice, the samples are taken from replay buffer of online and offline data, so the action samples taken from different distribution ga. However, still found the adjusted objective effective for reward regularization. Without it, the saddle- saddle-point refinement was less stable. Appendix G illustrates this refinement in simple setting. Figure 45 ablates this component, where its effect is most beneficial when there are fewer demonstrations.
F Bounding the coherent reward
The coherent reward a a<log g(a s) log | 8)) can be upper bounded if p(a 8) 0 whenever q(a 8) > 0. The bound is policy dependent. We use the bound the continuous naahttransformed Gaussian policy. We add small bias term oiin to
setting, where use the predictive variance in order to define the Gaussian likelihood upper bound. Inconveniently.
the tanh change of variables term in the log-likelihood -ie log(1 - tanh?(u)) [17] has no upper bound in theory, as the 'latent' action u E [-o0, oo]. but in the Acme implementation this oogllikelhhood is bounded for numerical stability due to the inverse tanh operation. For action dimension d, and uniform prior across action range [-1, 1] and a d the upper bound is r(s, a)= < d. (-0.5d, log 2 min + c de log 2) = 0.5 log To min/ / + č, where č depends on the tanh clipping term.
G Visualizing reward refinement
To provide intuition on Theorem 2 and connect reward refinement (Appendix E), build on the regression example from Figure 3 by viewing it as a continuous contextual bandit problem, i.e., a single-state MDP with no dynamics. Outside of the demonstration state distribution, we desire that the reward is uniformly zero, as we have no data to inform action preference. Figure 10 shows the result of applying CSIL and reward refinement this MDP, where the state distribution uniform. The 'desired' BC fit exhibits coherent reward with the desired qualities, and the HETSTAT policy approximates this desired policy. The heteroscedastic MLP exhibits undesirable out-of-distribution (OOD) behavior, where arbitrary regions have strongly positive or negative rewards. Reward refinement improves coherent reward of the heteroscedastic MLP to more uniform OOD, which in turn makes regression fit look more like a stationary policy. This result reinforces Theorem 2, which shows that Klluregulr BC connected to lower bound on an entropy- regularized game- theoretic IRI. objective that uses the coherent reward.
21

---

Desired
Heteroscedastic MLP
Stationary Heterososdastic MLP
Initial regression
Initial regression
3.0
0.0
Initial coherent reward
Initial coherent reward
-1.5
Refined regression
Refined regression
4.5
6.0
Refined coherent reward
Refined coherent reward
-9.0
We extend the regression problem from Figure to highlight the effect of reward refinement detailed in Section E from the BC and IRL perspective. Refining the coherent reward with minimax objective results in a BC policy that appears more stationary, as seen in the heteroscedatic MLP. The stationary heteroscedastic policy is also refined, but to lesser degree, it's approximately stationary by construction. The colourmap is shared hetween contour plots, and rewards are clipped at -10 to improve the visual colour range. Regression uncertainty is standard deviation.
Figure 10:
H Critic regularization
For the function approximation setting, CSII. requires an auxiliary loss term during learning to incorporate coherency. The full critic optimization problem is min Jo()) for objective
Jo(Ø) = E., E. a) (* Es,a- VV (p(8
(12)
where Q* denotes the target Q values and B denotes the replay buffer, that combines demonstrations D and additional samples. The motivation is to shape the critic such that the demonstration actions first-order optimal by minimizing the squared Frobenius norm of the action Jacobian.
IQLearn and PPIL adopt similar, but less explicit, regularization. Equation 10 of Garg et al. [42] is
max Es f (Qp(s.
(
(1 (1-21 7) [T)),,
where (8) a(), a' T(- 1)) and f is the concave function defining the implicit reward regularization. In the implementation [100]. the 'vO' objective variant replaces the initial distribution Mo with the demonstration state distribution. This trick deployed to improve regularization without affecting optimality (Section 5.3, Kostrikov et al. [41]). If the policy matches the expert actions with its mode, this term shapes a local maximum by minimizing the critic action samples around the
Jacobian regularization
sample regularization
20/D0
Figure 11: Illustrating the two forms of critic regularization on toy ID problem for critic Q(a) with regression target X. Jacobian regularization minimizes the squared gradient and corresponds to CSIL. Sample regularization minimizes sampled nearby actions and corresponds to IQLearn and PPIL. Both approaches shape the critic to have zero gradient around the target action, denoted by the dotted lines. A standard two-layer MLP with 256 units and ELU actions is used for approximation.
22

---

Algorithm 2: Off- Off-po coherent soft imitation learning with function approximation
Data: Expert demonstrations D, initial temperature a, refinement temperature, 88, parametric policy class qo(a s), prior policy p(a s). regression regularizer V, total steps T
Result: qew (a s), matching or improving the initial policy qo, (a
// Pretrain coherent reward and critic Train initial policy from demonstrations, 0 = arg max Ex,a- [log qa 8) (())::
Define fixed shaped coherent reward, Fo, (8, a) = a<llog y0; (a 88 logp(a p(a 8));
Pretrain critic with SARSA on 'D using r. for t + T do
// finetune policy with reinforcement learning Interact with environment 81+1 Env(s,, az), ag ~ [. 8t), store in replay buffer B; Optimize reward and critic using (Ø) and (0) (Equations 12, 11) on minibatch from B; Update policy optimizing I= (0) (Equation 13) on minibatch from B;
end
expert. However, if the policy does not accurately match the expert actions, the effect of this term is harder to reason about. In Figure 11, we illustrate these two forms from critic regularization on a toy example, demonstrating that both can produce local maximums in the critic.
I Practical algorithm with function approximation
For deep imitation learning, CSII. adopts SAC for the SPI step in Algorithm 1, summarized in Algorithm The critic objective for both pre-training and policy evaluation is the squared Bellman error. The policy objective max, J (0) samples both the demonstrations and replay buffer,
(0) Fa (-1a), B.Bu BUBBUn [Q(s, a) Bloog (o (: () log qo, (- |))]]
(13)
The policy and demonstration data are combined equally. We did not experiment with tuning this ratio. Compared to SAC, we found we did not have to train multiple critics, but we did need target
networks for the crities to stabilize learning. The critic and reward objectives were also evaluated on the policy and expert data each update. Moreover, like IQLearn and PPIL, we kept 3 constant and did not use an adaptive strategy.
J A divergence minimization perspective
Many imitation learning algorithms, such GAIL and IQLearn, are derived from minimizing a diver- gence between the agent's state-action distribution and the expert's. While CSIL is derived through policy inversion, it also has divergence minimization perspective, albeit with two minimization steps rather than one.
The first step involves recognising that maximum likelihood BC fit minimizes the forward KI. divergence between the demonstration distribution (s, a) = PD s) PD(()) and the policy,
0 = arg ming (Epppo()) (PD s) || ga *)]] arg max Ex,a- (log (a s)]-
Using the forward KL means that the unknown log-likelihood of the data distribution is not required, and the policy can be trained using supervised learning and without environment interaction.
The second step involves rearranging the KL-regularized RL objective into two KL divergences,
go (als) log (a(a)s) p(a|s)
0. arg maxg Ba-ga(-le) s~p (-) log pla,s)
arg mingo Eu-r,p ( D1 (go 8) go, (a s)] (1 3/a) (xalgo(a s) p(a s)]].
Note that since 3 < then -1- 3/a) 2 O. The left-hand term minimizes the reverse KL between the agent and the expert, using the BC policy as a proxy. This beneficial because the reverse conditional KL requires evaluating policy go the environment. Moreover, the reverse KL requires ligeliillhoods of the target distribution, which is why the BC policy is used as a proxy of the demonstration distribution. The light term incorporates the coherent reward and encourages the policy to stay in states where the policy differs from the prior. As qa should be close to the BC policy due to the left-hand term, the coherent reward shaping from Definition should hold.
In this formulation of the objective, it is also clear to see that if 3 = a, no RI. finetuning will occur, and go will simply mimic the BC policy ge, due to the left-hand term.
23

---

K
Theoretical results
This section provides derivations and proofs for the theoretical results in the main text.
K.1 Relative entropy regularization for reinforcement learning
This section re-derives KL-rregularized RL and ME-IRL shared notation for the reader's reference.
Reinforcement learning. We consider the discounted, hefnite-horizon setting with a hard KLL constraint the policy update with bound c, max, Pes,srP((lee,er)) aer-g--les), (-) E"r"r(s az)] ( [ [4(a 8) p(a s)]l S c.
This constrained objective transcribed into the following Lagrangian objective following Van Hoof et al. [15]. using the discounted stationary joint distribution d(s, a) g(a s) v(s) = -1- (1-7))(s,a).
[ d(s.a))(s, + X (1-1111
[(d, X1, Az) (1 (1-
d(s, a) ds da
SXA
SSA
+ *S(DOes Ne(s') yP(s' s,a) a) + (1 7)o0(s') / d(s', a')',a')d'') ds
+ Xa (-- (--((()) v(s)
/ g(a log {a da ds
p(a 8)
where X is a function for an infinit--dimensional constraint.
consanini Solving for 0C(g, λ1, A2 0d)/Od= =0,
q(a 8) exp(a (r(s, (
( -V(((sss)(((())))))) V(s))) p(a s).
@(saa)
with (1-7) and A2(8) = (1 (s)),, where V can be interpreted as the soft value function,
/ exp
o(ss p(a s) da f g(a s)
((s) log
To obtain the more convenient lower bound of the soft Bellman equation, we combine importance sampling and Jensen's inequality to the soft Bellman equation
((s) log / exp (1 e(s.aa)
p(a 8) q(a 8) {(a p(a s)
q(a
8) da,
2 Ba))(()s) Q(s, a(s a log
This derivation is for the 'hard' KL constraint. In many implementations, including ours, the constraint is 'softened' where X3 and, therefore, is a fixed hyperparameter.
Inverse reinforcement learning. As mentioned in Section 2 and Definition 2, ME-IRL. is closely related to Kl.-regularize RT., but with the objective and a constraint reversed,
min, Essee (-) [Ka s) || p(a s)]] s.t. Eg.a~,, (Ø(s, a)] = E. E. pl(s, a)]. The constrained optimization is, therefore, transcribed into similar Lagrangian,
L(d, 22, As) = / u(s) f a(aa log p(a s)
{(a 8) dads A1 (1- SXA
f--//a4 d(s. a) dsda
1 >2(0') (16 7P(s' d( + - 7))) / (l'ss,a)d as'
d(s, a(s, Es,a-D(lb(s A1 (1--)-
SXA
The consequence this change that the likelihood temperature now implicit the reward model r(s, a) = (1 (1- - Φ(8 a) due to the Lagrange multiplier weights, this reward model is used in the soft Bellman equation instead of the true reward. The reward model jointly optimized to minimize the apprenticeship error by matching the feature expectation.
24

---

K.2 Proof for Theorem 1
Theorem 1. IKL-regularized policy improvement inversion). Let p and 9a be the prior and pseudo-
posterior policy given by posterior policy iteration (Equation 4). The critic can be expressed as
(a 8) + Va(s, (a(s) a log / exp ( p(a 8)
((s, a) a log
p(als) da. (6)
Substituting into the KL-regularized Bellman equation lower-bound from Equation 2.
(a 8) + Va(s) - yy -7 [V((')].
r(s, a) a log
p(a 8)
The (s( term is the 'soft' value function. We assume {a(a) = -0 whenever p(a () =)=0.
Proof. Equation is derived by rearranging the posterior policy update in Equation 4. Equation in derived by substituting the critic expression from Equation 6 into the KL-regularized Bellman equation (Equation 3),
log (a 8) (a)=
p(a 8)
(ss.a)
a alog lala'ts4 p(a' ss) + Va(s') - log
lald' pla' 88)
r(s, a) + +/Bawp(()a.)) al
2(x',a))
K.3 Proof for Theorem 2
Theorem relies the data processing inequality (Lemma 3) connect the function-space KL in CSIL with the weight-space KL used in regularized BC.
Assumption 1. (Realizability) The optimal policy be expressed by the parametric policy qo(a s). Lemma 2. (Coherent policy optimality). The optimal policy for a KL-regularized RL problem with coherent reward F(s, a) a(log
- logoggg p(a 8)) and temperature is q(a s).
Proof. This result arises from the origin of the coherent reward from policy inversion and re- ward shaping (Theorem 1, Lemma 1). Moreover, considering the entropy-augmented reward xo.x(8 (8, a) r(s, a) allog a(a 8) log p(a 8), when is the coherent reward and π q. To (8, a) = 0'' E S, a E A so q(a 8) is a solution to soft policy iteration.
Lemma (Data processing inequality,
[101], Theorem l). For two stochastic processes param- eterized with finite random variables w p(-), E W and shared hypothesis spaces p(y æ, w), the conditional f-divergence in function space at points X x², L E4, is upper bounded by the f-divergence in parameter space,
Dy(g(w) || p(w)) = D)l4((Y, w X) pY w X)] 2 Dyla(Y x) X)).
Theorem 2. (Coherent inverse reinforcement learning as KL KL-regularized behavioral cloning). A KL- regularized game-thhmmeeeee thhooreth IRL objective, with policy go (a s) and coherent reward parameterization
re(s, a) ax a<log ge (e(a) 1)-logppaa s)) where 2 0, is lower bounded by scaled KL-regularized behavioral cloning objective and a constant term when the optimal policy, posterior, and prior share hypothesis space p(a s, w) and finite parameters w,
E..a- (re a)] Eango (-1e), arHag [re(s, a) -3oog (a 8) logp(a 8))]
(Ba,anD [log (a s)] Dei. (go(w) Dxilgo p(w)) + Eaa-D Dogg((a x)])-
where = (a- and Eg pla a s)] is constant. The regression objective bounds the IRL. objective for a worst-case on-policy state distribution (s)s), which motivates its scaling through KL. regularization is required, so 3 and = D does not have sufficient coverage, then lei < a so > to regularize the BC fu and finetune the policy with RL accordingly with additional soft policy iteration steps.
3. Df has sufficient coverage, RL finetuning
25

---

Task
Random return
Expert return
HalfCheetah-v2
-282
8770
Hopper-v2
18
2798
Walker2d-v2
1.6
4118
Ant-v2
-59
5637
Humanoid-v2
123
9115
exoe-expo
-56
2882
hammer-expert-v0
-274
12794
Table 2: Expert and random policy returns used normalize the performance for Gym and Adroit tasks. Taken from Orsini et al. [11].
Proof. We begin with the KI.-regularized game-theoretic TRI. objective (c.f. Equation [17)),
JI, Ea,anD (rs,, a)] Ea.a-p [((s,a) a)
3log T(a 8) p(a 8)
Substituting the coherent reward from Theorem 1, the objective becomes
[(0, n. 3) Ea,anD log To
go 8) s)
3 log <(a 8) p(a 8)
E Eapap
aog
p(a 8)
p(a
Due to the realizablity assumption (Assumption 1), the optimal policy To. be obtained through maximum likelihood estimation 0, = arg max, E,,a~ [log (s,s, a)]. With this assumption, the left- hand term performs maximum likelihood estimation of the optimal policy. Therefore, combined with the coherent reward which is derived from policy inversion, we replace the inner policy optimization of ı with the maximum likelihood solution of qo. resulting in simpler objective J in terms of only 0, which matches the objective I exactly when 3 = a and क = qe (Lemma 2). We consider J with small negative perturbation
da, da 2 0 examine the effect of the right-term term,
F(0) J(0. qo. 5a) = Eg,aND log Go
ςo(a (a) s) p(a
ςo x) Ea-go (:1a), ampug da log
p(a s)
The right-hand term contains conditional KL divergence, which we can replace with an upper-bound using the data processing inequality from Lemma 3 and the weight-space KL. This weight-space KL is constant w.r.t. the state distribution expectation and we introduce the scale factor = da a. The resulting lower-bound objective is proportional to the regularized behavioral cloning objective proposed in Equation 8,
((0) 2 a(E,a-D [log (a s)] (
[[w)] + C)2 C= D= s)].
The intuition from Theorem is that the KI. regularization during BC achieves reward shaping similar to the IRL. saddle-point solution when using the coherent reward. The lower bound's 'tightness' depends on the data processing inequality and is difficult to reason about. Therefore, this result demonstrates that KL-regularized BC provides useful reward shaping from the game-theoretic per- spective, rather than provide a tight lower bound, To reinforce this result, we visualize the effect in Appendix G.
L Implementation and experimental details
Our codebase for this work is spurcee and is available at
The tabular experiments are implemented in jax automatic differentiation and linear algebra li- brary [102]. The continuous state-action experiments are implemented acme, again using jax and its ecosystem [103]. One feature of acme is distributed training, where we use four concurrent actors to speed up the wall clock learning time. eQre and PPII. were re-implemented based on the ooen-source implementations reference [100, 104]. CSIL, IQLearn, and PPIL shared a combined
26

---

implementation as 'soft' imitation learning algorithms due to their common structure and to facilitate ablation studies. Baselines DAC, SQIL, PWIL, SACfD, and CQL are implemented in acme. We used a stochastic actor for learning and deterministic actor (i.e., evaluating the mean action) for evaluation.
The DemoDICR and SMODICE offine imitation learning baselines were implemented using the original implementations [105, 106]. This meant that the expert demonstrations were obtained from d4r1 rather than Orsini et al. []]] but the returns were normalized to the d4r1 returns for consistency. We note that there is a non-negligible discrepancy between the d4rl and Orsini et al. [11] return values for the MuJoCo tasks, but in the imitation setting. comparing normalized values should be a meaningful metric.
Due to the common use of SAC the base RL algorithm, most implementations shared common standard hyperparameters. The policy and critic networks were comprised of two layers with 256 units and ELU activations. Learning rates were 3e-4, the batch size was 256, and the target network smoothing coefficient was 0.005. One difference to standard RI. implementations is the use of layernorm in the critic, which is adopted in some algorithm implementations in acme and was found to improve performance. For the inverse temperature a, for most methods, we used the adaptive strategy based on a lower bound on the policies entropy. IQLearn and PPIL both have fixed temperature strategy that we tuned in the range [1.0, 0.3, 0.1, 0.03, 0.01]. CSIL also has fixed temperature strategy but applied to the KL penalty. In practice, they are very similar, so we swept the same range. We also investigated a hard KL bound and adaptive temperature strategy, but there was no performance improvement warrant the extra complexity. CSIL's HETSTAT policies had an additional bottleneck layer of 12 units and final layer of 256 units with stationary activations. For the Gym tasks, triangular activations were used. For robomimic tasks, periodic ReLU activations were used. IQLearn, PPIL., and CSIL all have single critics rather than dual critics, while the other baselines had dual critics to match the standard SAC implementation. We tried dual crities to stabilize IQLearn and PPIL, but this did not help. CSIL's reward finetuning learning rate was le-3. As the reward model was only being finetuned with non-expert samples, we did not find this hyperparameter to be very sensitive. For robomimic, c, larger models with two layers of 1024 units were required. The HETSTAT policies increased accordingly with a 1024 unit final layer and 48 units for the bottleneck layer.
For DAC, we used the hyperparameters and regularizers recommended by Orsini et al. [11], including spectral normalization for the single hidden layer discriminator with 64 units. IQLearn has lower actor learning rate of 3e-5, and the open-source implementation has the option of different value function regularizers per task [100]. We used 'v0' regularization for all environments, which minimizes the averaged value function at the expert samples, In the original IQLearn experiments, this regularizer is tuned per environment [100]. PPIL shares the same smaller actor learning rate of 3e-5 and the open-source implementation also uses 20 critic updates for every actor update, which we maintained. The implementation of PPIL borrows the same value function regularization IQLearn, so we used 'vO' regularization for all environments [104]. For SACID, we swept the demo-to-replay ratio across [0.1, 0.01]. For Adroit, the ratio was 0.01, and for robomimic it was 0.1.
The major implementation details of CSIL. are discussed in Sections 4, C, and E. The pre-training hyperparameters are listed in Table 4. While the critic pre-training is not too important for perfor- mance (Figure 44), appropriate policy pre-training is crucial. Instead of training a regularization
Experiment
Algorithm
DAC
PWIL
SQIL
IQLear
PPIL
CSIL
Online Gym
H(T)
H((T)
(((T)
0.01
0.01
0.01
Offline Gym
H(T))
H(T
HC
0.03
0.03
0.1
Online Humanoid-v2
H(m)
H(T
H(
0.01
0.01
0.01
Online Adroit
H(T)
H(T))
(((T)
0.03
0.03
0.1
Online robomimic
H(T))
0.1
0.1
Offline robomimic
1.0
Image-based robomimic
0.3
Table 3: The entropy regularization hyperparameter value used across methods and environments. Values refer fixed temperature while H(m)) refers to the adaptive temperature scheme, which ensures the policy entropy is at least equal to - -d,, where d, is the dimension of the action space.
27

---

Experiment
Pre-training hyperparameters
n (Tr)
(T)
n (Q)
À (Q)
Online and offline Gym
25000
le-3
5000
lc-3
Online Humanoid-v2
500
1e-3
5000
le-3
Online Adroit
25000
le-4
5000
le-3
Online and offline robomimic
50000
le-4
2000
1e-3
Image-based robomimic
25000
le-4
2000
le-3
Table 4: CSIL's pre-training hyperparameters for the policy (л) and critic (Q), listing the learning rate (A) and number of iterations (n)).
hyperparameter, we opted for early stopping. As a result, there is a trade-off between BC accuracy and policy entropy. In the online setting, if the policy is trained too long, the entropy is reduced too far, and there is not enough exploration for RL. As a result, the policy pre-training hyperparameters need tuning when action dimension and dataset size change significantly ensure the fit sufficient for downstream performance. For robomimic, we used negative CSIL rewards scaled by 0.5 to ensure they lie around [-1, 0]. For vision-based robomimic, the CNN torso Acme ResNetTorso down to LayerNorm MLP with 48 output units, ELU activations, orthogonal weight initialization, and final tanh activation. Mandlekar et [84] used a larger ResNet18. As in Mandlekar et al. [84]. image augmentation was achieved with random crops from 84 down to 76. Pixel values were normalized [-0.5, 0.5]. For memory reasons, the batch size was reduced to 128. The implementations of IQLearn, PPIL and CSIL. combined equally-sized mini-batches from the expert and non-expert replay buffers for learning the reward, critic and policy. We did not investigate tuning this ratio.
impala-style [107]
Regarding computational resources, the main experiments distributed learning. As result, our 'learner' (policy evaluation and improvement) runs on a single TPU v2. We ran four actors interact with the environment. Depending the algorithm, there were also one or more evaluators. For vision-based tasks, we used A100 GPUs for the vision-base policies.
using acme, which implements
28

---

M Experimental results
This section includes additional experimental results from Section 5. Videos of the simulation studies are available at joemwatson, htt.://tsoonngttubioo/oo/ooo
M.1 Inverse optimal control
This section shows the converged value function, stationary distribution, and 'windy' stationary distribution for the tabular setting results from Table 'Oracle classifier' constructs an offline classifier-based reward separating the states and actions with- and without demonstration data, in the spirit of methods such as SQIL and ORIL [85]. No optimization was conducted on the windy environment, just evaluation assess the policy robustness to disturbance. We the Viridis colour map, so yellow regions have larger value than blue regions.
Rassioa
Nosinal sutioerry disibattos
Waddy darrattto
Noxinal Matkearr dveilaties
Windyasseny Incobosas
Figure 12: Expert on the dense MDP.
Figure 13: BC on the dense MDP.
Figure 14: Oracle classifier on the dense MDP.
Figure 15; GAIL on the dense MDP.
Viiue kaaatoo
Nondraal dicniratios
Wiadyyratlszzry datribatio
Value
Nenaza datiiasioo
Wiady audenary diariaation
+
Figure 16: IQLeam the dense MDP.
Figure 17: PPIL on the dense MDP.
Vaiue Isaatos
Rosinal matteary deciiaatic
Widyynatiesrrdarraaaut
Viles fseatis
Ne mationay dauforson
Winde mudccary datriation
Figure 18: ME-IRL on the dense MDP.
Figure 19; CSIL on the dense MDP.
29

---

Figure 20: Expert on the sparse MDP,
Figure 21: BC on the sparse MDP,
Nosini dimivaaioo
Vine Fanstion
Namiial euatosary Earisoson
Wileyy Murdonary Aurdeatio
Figure 22: Oracle classifier the sparse MDP.
Figure 23: GAIL on the sparse MDP.
sataeerr Graninatres
Wendystttenee Weddysttteeerraanbbre
Nemmai mutumry
Windy Mertenary
Figure 24: IQLearn on the sparse MDP.
Figure 25: PPIL on the sparse MDP.
Figure 26: ME-IRL on the sparse MDP,
Figure 27: CSIL on the sparse MDP.
30

---

Ant-v2 HalfCheetah-v2
Hopper-v2
Walker2d-Ve
Return
1)
Return
(n =n= 3)
Return (n= 10)
Return [n= = 30)
Steps (1 x 10%)
Steps (1 x 105)
Steps (1 x 10%)
Steps x 10%)
squi
dac
sgleam
ppu
pwil
exil
Figure 28: Normalized performance CSIL again baselines for online imitation learning on MuJoCo Gym tasks. Uncertainty intervals depict quartiles over ten seeds. CSIL exhibits stable convergence with both sample and demonstration efficiency. BC here applied to the demonstration dataset. refers to demonstration trajectories.
door-v0
hammer v
Return
(n= I))
Return
In In=33 3)
Return
(n= 10)
Return
(n=30) 30)
Steps (1 x 10%)
Steps (1 x 10%)
be
sacid
sgil
dac
iglearn
ppil
pwi
csil
Figure 29: Normalized performance of CSIL against baselines for online imitation learning for Adroit tasks. Uncertainty intervals depict quartiles over ten seeds. CSIL exhibits stable convergence with both sample and demonstration efficiency. Many baselines cannot achieve stable convergence due to the high-dimensional action space. SACfd is an oracle baseline that combines the demonstrations with the true (shaped) reward. n refers to demonstration trajectories.
31

---

Ant-v2
HalfCheetah-v2
Hopper-v2 Walker2d-v2
Return
(n= 1)
Return (n=33
Return ( 10)
Return (n 30)
Steps (1 x 10%)
Steps (1 10%)
Steps (1 x 10%)
Steps (1 x 10
og
bc
iqlearn
ppúl
cai
Figure 30: Normalized performance of CSIL against baselines for offfine imitation learning for Gym tasks.. Uncertainty intervals depict quartiles over ten seeds. While methods struggle this setting compared to online learning, CSII. manages convergence with enough demonstrations. CQL is an oracle baseline using the true rewards. The baseline trained on the whole offline dataset, not just demonstrations. refers demonstration trajectories.
Humanoid v2
1.0
Return
(x ===]] 1)
0.5
1.0
Return
(x (x 3)
0.0
Return
[n = 10)
0.5
0.0
1.0
Return
(n 00=30 30)
0.5
0.25
0.5
0.75
Steps (1 x 10%)
pwil
sgil
dac
iqlear
ppil
Figure 31: Normalized performance of CSIL against baselines for online imitation learning for Humanoid-v2. Uncertainty intervals depict quartiles over ten seeds. refers to demonstration trajectories.
32

---

M.2 Continuous control from agent demonstrations
Step-wise results. Figure 28, 29 and 30 show step-wise performance curves, as opposed to the demonstration-wint results in the main paper. For online experiments, steps refers to the actor and learner, while for offline experiments, steps correspond the learner.
Humanoid-v2 results. We evaluated CST and baselines on the bigh-dimensional Humanoid i locomotion task in Figure 31. We found that CSIL struggled solve this task due the difficulty for BC to perform well at this task, even when using all 200 of the available demonstrations in the dataset.
One aspect of Humanoid-y responds to the survival standing bonus, rather than actual locomotion (around 9000). While the positive nature of the coherent reward enabled learning the stabilize, this not consistent across seeds. We also found difficult to reproduce baseline results due to IQLearn's 's instability, which we also reproduced the author's 's implementation [100]. In theory, BC should be able to approximate the original optimal agent's 's policy with sufficient demonstrations. We believe the solution may be additional BC regularization, the environment has 376-dimensional state space.
note is that return of around 5000 (around 0.5 normalized) cor-
Performance improvement of coherent imitation learning A desired quality of CSIL. is its ability to match exceed its initial BC policy. Figures 32, 33 and 34 show the performance of CSIL relative to its initial BC policy. Empirical, CSIL matches or exceeds the initial BC policy across environments.
Ant v2
HalfCheetah-v
Hopper v2
Wa r2d
1.0
Return
(n ===1] 1)
0.5
0.0
Return
[m=3) 3)
0.5
1.0
Return
(x = 10)
0.5
0.0
1.0
Return
0.0
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
0.25
0.5
0.75
Steps
105)
Steps (1 x 109)
Steps (1 x 10*)
Steps
10
(cail)
Figure 32: Normalized performance of CSIL for online Gym tasks against its initialization. Uncertainty intervals depict quartiles over ten seeds. refers to demonstration trajectories.
33

---

door-vB
hammer r v8
Return
[m= 1)
0.3
0.0
Return
(x=3) 3)
1.0
0.0
Return
b/ 10)
1.0
0.5
Return
= 30)
1.0
0.5
0.0
0.25
0.5
0.75
0.25
0.5
0.75
Steps (1 x 100)
Steps (1 x 10%)
(csil)
Figure 33: Normalized performance of CSIL for online Adroit tasks against its BC initialization. Uncertainty intervals depict quartiles over ten seeds. n refers to demonstration trajectories.
Ant v2
HalfCheetah- -v
Hopper -22
Walker2d
1.0
Return
(n ===1 I)
0.5
Return
(n (n=3) 3)
0.5
0.0
1.0
Return
(n=10) 10)
0.0
1.0
Return
[m==00 30)
0.5
0.75
0.75
0.25
0.75
Steps x 10%)
Steps (1 x 10%)
Steps (1 x 10")
Steps (1 10
be (csil)
Figure 34: Normalized performance of CSIL for offline Gym tasks against its BC initialization. Uncertainty intervals depict quartiles ten seeds. refers to demonstration trajectories.
34

---

Ant v2
door vv
expert rewa rd
500
500
expert q
400
400
100
expert msse
0.2
0.0
0.25
0.5
0.75
0.25
0.5
0.75
Steps (1 10*)
Scops (1 x 10")
ppsl
Figure 35: We investigate the weakness of IQLearn and PPIL for offline learning (Ant-v2) and high-dimensional action spaces (door-vO). Compared to CSIL, IQLear and PPIL have less stable and effective values, in particular for the expert demonstrations, which results in the policy not inaccurately predicting the demonstration actions (expert mse).
Divergence in minimax soft imitation learning To investigate the performance issues in IQLearn and PPIL relative to CSIL, we monitor a few key diagnostics during training for online door-v0 and offline Ant-v2 Figure 35. One key observation is the drift in the BC accuracy. We attribute this to the instability in the reward or critic due to the saddle-point optimization.
M.3 Continuous control from human demonstrations
Figure 36 shows performance on the online setting w.r.t. learning steps, and Figure 37 shows the offline performance w.r.t. learning steps. For online learning, CSIL is consistently more sample efficient than the oracle baseline SACfD, presumably due to BC pre-training and CSIL 'ss shaped reward w.r.t. robomimic' ' sparse reward on success. CSIL was also able to surpass Mandlekar et al. [84]'s success rate on SettAsemblySSuare. For offline learning from suboptimal human demonstrations, it appears to be much harder setting for CSII. to improve on the initial BC policy, especially for harder tasks.
35

---

Lift
PickPlaceCan
vittAssemblySqual
Success
(n= 25)
Success
(m=50) 50)
Success
100)
Success
[n= 200)
0.25
0.55
0.75
0.25
0.5
0.75
0
0.25
0.75
Steps 10%)
Steps x 10%)
Steps x 109)
(Mandlekar al.)
be (Mandlekar et
dac
iglearn
csil
Figure 36: Average success rate 50 evaluations for online imitation learning for robomimic c tasks. Uncertainty intervals depict quartiles over ten seeds. The BC policy is trained on the demonstration data. refers demonstration trajectories.
Lift
PickPlaceCan
NutAssemb lySquare
1.0
Success
(n = 200)
0.5
0
0.25
0.5
0.75
1.0
0
0.25
0.5
0.75
1.0
0.25
0.5
0.75
1.0
Steps (1 x 10%)
Steps (1 x 10%)
Steps (1 x 10%)
(esil)
csil
Figure 37: Average success rate over 50 evaluations for offline imitation learning for robomimic tasks. Uncertainty intervals depict quartiles over ten seeds. This experiment shows that learning from suboptimal offline demonstrations is much harder than on-policy data. It is also harder the KL regularization to ensure improvement over the initial policy, n refers to demonstration trajectories.
36

---

N
Ablation studies
This section includes ablation studies for the CSIL algorithm and baselines.
N.1 Baselines with behavioral cloning pre-training
To assess the importance of BC pre-training, we evaluate the performance of IQLearn and PPIL with BC pre-training in Figure 38. While difficult performance at the beginning of training as the policy 'unlearns' as the critic and reward are randomly initialized since there is no notion of coherency.
across all experiments, there is small drop in
N.2 Baselines with behavioral cloning pre-training and KL regularization
To bring IQLeam and PPIL closer to CSIL, we implement the baselines with BC pre-training and KL regularization, but with SAC-style policies rather than HETSTAT. We evaluate the harder tasks, online Adroit, and offline Gym, to see if these features contribute sufficient stability to solve the task. Both Figure 39 and Figure 40 show increase in initial performance, is not enough completely stabilize the algorithms.
Ant-v v2
HalfCheetah
Hopper v2
Wal le r2d v2
(n 11=
(n 3)
Return
Oz 10)
Return (y = 30)
0.25
0.25
0.25
0.25
Steps (1 x 10%)
Steps (1 x 10%)
Steps (1 x 10%)
Steps x 10*)
sglear pretraining)
poll (w/ pretraining)
iglearn
ppil
Figure 38: Normalized performance IQLearn and PPIL for online Gym tasks with BC pre-training. Uncertainty intervals depict quartiles over ten seeds. refers to demonstration trajectories.
37

---

door-v0
hammer r-0
Return
0.0
Return
(m=3) 3)
Return
1.0
i==
0.5
Return
in =n=30 30)
0.5
0.0
0.35
0.75
1.0
0.25
0.5
0.75
1.0
Steps x 10%)
Steps (1x 10*)
iqlearn (w/ price)
ppil (w/ be prioe)
Figure 39: Normalized performance of IQLeam and PPIL for online Adroit tasks with BC pre- training. Uncertainty intervals depict quartiles over ten seeds. n refers to demonstration trajectories.
Ant v2
Halffheetah--e v2
Hopper v2
Walker20 -v2
Return
Return
In=3)
Return
(n= 10)
Return [n= 30)
0.25
1.0
0.75
0.25
0.5
0.75
0
Steps (1 x 10%)
Steps (1 x 10f)
Steps (1 x 10")
Steps (1 x 10%)
eare (w/ be prior)
ppil (w/ be prior)
iqlearn
ppil
Figure 40: Normalized performance of IQLearn and PPIL for offline Gym tasks with BC pre-training. Uncertainty intervals depict quartiles ten seeds. refers demonstration trajectories.
38

---

Ant-v2 HalfCheetah-v2
Hopper v2
Walker2d .v2
Return
1)
Return
(n= 3)
Beturn (n= 10)
Return (n= 30)
0.35
0.5
0.75
1.0
0.25
0.75
0.25
0.3
0.75
1.0
0.35
0.75
1.0
Steps (1 10%)
Steps (1 x 10%)
Steps (1 10%)
Steps
x 10")
iglearn (w/ hetstat policy)
ppll (w/ hetstat policy)
sgleurn
Figure 41: Normalized performance of IQLearn and PPIL for online Gym tasks with HETSTAT policies. Uncertainty intervals depict quartiles over ten seeds. n refers to demonstration trajectories.
N.3 Baselines with heteroscedastic stationary policies
To investigate the impact of HETSTAT policies alone, we evaluated PPIL and IQLearn with HETSTAT policies the online Gym tasks in Figure 41. There significant discrepancy between MLP and HETSTAT policies.
N.4 Baselines with pre-trained heteroscedastic stationary policies as priors
To bring IQL.aam and PPII. closer CSI,,, implement the baselines with pre-training, HETSTAT policies and KI. regularization with stationary policies. We evaluate on the harder tasks, online Adroit and offline Gym. to see if these features contribute sufficient stability to solve the task. Both Figure 42 and Figure 43 show an increase in performance, especially with increasing number of demonstrations, but ultimately enough to completely stabilize the algorithms the degree seen in CSIL.
39

---

door-v8
hamme r-v0
Return
1.0
0.0
Return
=3
1.0
0.5
Return
10)
0.5
0.0
Return
(n 30)
0.0
Steps (1 x 10%)
Steps (1x 105)
iqlearn (w/ pretrained hetraat prioe)
100. ppil (w/ pretrained hetstat price)
iglearn
ppil
Figure 42: Normalized performance of IQLearn and PPIL for online Adroit tasks with pre-trained HETSTAT policies and KI-rregularization Uncertainty intervals depict quartiles over ten seeds. refers to demonstration trajectories.
Ant v2
HalfCheetah- v2
Hopper v2
Wa Lalkerd 2d v2
Return
(n =
Retur (n= 3)
Return (n== 10))
Return
1==30 30)
0.25
0.5
0.75
1.0
0.75
0.25
0.5
0.75
1.0
0.55
0.75
1.0
Steps (1 x 10%)
Steps (1 x 10%)
Steps (1 x 10%)
Steps (1 x 10%)
Sgleam (w/ pretrained betstat prior)
ppil (w/ pretrained hetstat prior)
iqlear
ppil
Figure 43: Normalized performance of IQLeam and PPIL for offline Gym tasks with pre-trained HETSTAT policies and KL-regularization. Uncertainty intervals depict quartiles ten seeds. refers to demonstration trajectories.
40

---

Ant- v2
HalfCheetah- v2
Hopper- v2
Walker20l
1.0
Return
-1)
0.5
Return
(x=3)
0.5
Return
[n= 10)
0.0
1.0
Return
30)
0.0
0.25
0.25
0.25
0.25
Steps (1 x 109)
Steps x 10%)
Steps (1 x 109)
Steps (1 x 10%)
(w/o pretrsined critic)
Figure 44: Normalized performance of CSIL for online Gym tasks a critic pre-training ablation. Uncertainty intervals depiet quartiles over ten seeds. refers to demonstration trajectories.
Ant-v2
HalfCheetah v2
Hopper-v2
Walker2d-vz
Return
by 1)
0.5
1.0
Return
(n 3)
0.5
0.0
1.0
Return
in 10)
0.0
1.0
Return
(n 30)
0.5
0.25
0.75
0.25
0.5
0.75
1.0
0
0.25
0.5
0.75
1.0
0.25
0.75
Steps x 10f)
Steps (1 10%)
Steps
10%)
Steps 109)
csil (wfo reward fineruning)
Figure 45: Normalized performance CSIL for online Gym tasks with reward finetuning ablation. Uncertainty intervals depict quartiles over ten seeds. Refinement is most important in the low demonstration setting where the initial BC policy and coherent reward less defined, and therefore the initial policy is poor and the coherent reward is more susceptible to exploitation of the stationary approximation errors. n refers to demonstration trajectories.
41

---

N.5 Importance of critic pre-training
To ablate the importance of critic pre-training Figure shows the performance difference on online Gym tasks. Critic pre-training provides only minor performance improvement.
N.6 Importance of reward finetuning
To ablate the importance of reward finetuning, Figure 45. shows the performance difference online Gym tasks. Reward finetuning is crucial in the setting of few demonstration trajectories, as the imitation reward is sparse and therefore esil is more susceptible to approximation errors in stationarity, is also important for reducing performance variance and ensuring stable convergence.
N.7 Importance of critic regularization
To ablate the importance of critic Jacobian regularization, Figure 46. shows the performance difference on online Gym tasks. The regularization has several benefits. One faster convergence, as it encodes the coherency inductive bias into the critic. A second is stability, it encodes first-order optimality w.r.t. the expert demonstrations, mitigating errors in policy evaluation. The inductive bias also appears reduce performance variance. One observation that the performance of Ant-v2 improves without the regularization. This difference could be because the regularization encourages imitation rather than apprenticeship-style learning where the agent improves on the expert.
N.8 Stationary and non-stationary policies
One important ablation for CSIL evaluating the performance with standard MLP policies like those used by the baselines. We found that this setting not numerically stable enough complete ablation experiment. Due to the spurious behavior (e.g., as shown in Figure 3), the log-likelihowe values used in the reward varied significantly in magnitude, destabilizing learning significantly, leading to policies that could no longer be numerically evaluated. This result means that stationarity critical component of CSII..
N.9 Effectiveness of constant rewards
Many imitation rewards typically encode an inductive bias positivity or negativity, which encourage survival 'minimum time' strategies respectively [40]. We ablate CSIL with this inductive bias for survival tasks (Hopper-v2 and Walker-v2) and minimum-time tasks (robomimic). Figure 47 shows that positive-only rewards sufficient for Hopper-v2 but not for Walker-v2. Figure 48 shows that negative-only rewards is effective for the simpler tasks (Lift) but not the harder tasks ButtssseblySSuars
N.10 Importance of behavioral cloning pre-training
Figure 49 shows CSIL without any policy pre-training, so the reward learned only though the finetuning objective. CSIL without pre-training does not appear to work for any of the tasks. This poor performance could be attributed to the lack of hyperparameter tuning, since the reward learning hyperparameters
chosen assuming a good initialization. It could also due to the maximum likelihood fitting of the coherent reward, since the refinement does not use the 'faithful' objective (Section D) that improves the regression fit.
42

---

Ant HalfCheetah -v-
Hopper-v2
Walker2d v2
1,0
Return
(n 1
0.0
1.0
Return
in m) 3)
0.5
Return
(ns = 10)
0.5
0.0
Return
(n nm=33 30)
1.0
0.5
0.0
02255
0.5
0.75
1.0
0.25
0.5
0.75
1.0
0
0.25
0.5
0.75
1.0
0.25
0.5
0.75
1.0
Steps x 109)
Steps (1 10%)
Steps
x 10%)
Steps (1 x 109)
critic regularization)
Figure 46: Normalized performance of CSIL for online MuJoCo Gym without regularizing the critic Jacobian. Uncertainty intervals depict quartiles over ten seeds. The benefit of this regularization somewhat environment dependent, but clearly regularizes learning to the demonstrations effectively many cases. Performance reduction is likely due to underfiftting during policy evaluation due to the regularization. refers to demonstration trajectories.
Hopper- v2
Walker2d
1.0
Return
11
(n
0.0
Return
(nt m 3)
0.5
0.0
1.0
Return
in 10)
0.5
0.0
1.0
Return
in 30)
0.5
0.25
0.75
0.25
0.75
Steps (1 x 10%)
Steps (1 x1 10%)
csil (w/ reward 1)
esil
Figure 47: Online MuJoCo Gym tasks with absorbing states where CSIT. given constant +1 reward function. Uncertainty intervals depict quartiles over ten seeds. While this is sometimes enough to solve the task (due to the absorbing state), the CSIL reward performs equal or better. n refers to demonstration trajectories.
43

---

Lift
PickPlaceCan
NutAssemb LySquare
(nt = 2000
Success
Success
(n== 100)
0.5
0.0
10
Success
(n =n= (0)
0.5
0.0
(n= (n= 25)
0.5
0.25
0.5
0.75
1.0
0.25
0.5
0.75
1.0
0.35
0.5
0.75
1.0
Steps (1 10%)
Steps x 10*)
Steps x10%)
csil (-1 reward)
csi
Figure 48: Online robomimic tasks where CSIL is given constant -1 reward function. Uncertainty intervals depict quartiles over ten seeds. While a constant reward is sometimes enough to solve the task (due the absorbing state), the CSIL reward performs roughly equal better, with lower performance variance. n. refers to demonstration trajectories.
Ant-va v2
HalfCheetah v2
Hopper-v -vp
Walker2d (dterddvv
1.0
Return
0.5
in
0.0
Return
(m 3)
Return
Return
=0)
1.0
0.5
0.0
0.25
0.5
0.75
100
0.35
0.5
0.75
1.0
0.25
0.5
0.75
1.0 0
0.25
0.5
0.75
1.0
Steps (1 10%)
Steps (1x 108)
Steps (1 10f)
Steps (1 x 10%)
csil (w/o BC pretraining)
Figure 49: Online MuJoCo gym tasks where CSIL has no BC pre-training. Uncertainty intervals depict quartiles over 5 seeds. refers to demonstration trajectories.
44


Reviews:
[{"review_id": "qg68VKnQff", "reviewer": "Reviewer_tsdg", "summary": "The paper proposes an approach to inverse RL to fine-tune a behavior cloning policy using RL on online of offline data sources. Adopting the KL-regularized view to RL/IRL, CSIL expresses the reward in terms of the behavior cloned policy and connects the result with the well-known reward shaping results. This \"coherent\" reward is thereby used to improve the policy, resulting in a simple and performant inverse RL algorithm, compared to game-theoretic/adversarial approaches.", "questions": "How important is reward refinement? Do the final experiments use reward refinement? If so, can you report performance with and without reward refinement.\n\nThe paper mentions the issue of reward/critic causing the unlearning of the initialization. How does CSIL avoid this issue?", "limitations": "The paper provides a reasonable discussion of limitations, particularly how the success of CSIL can be sensitive to the success of the initial BC policy.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper provides a fairly comprehensive overview and background of inverse RL. The construction of the coherent reward using the entropy-regularized RL framework is interesting, and results in a simple algorithm with soft policy iteration. The experimental results are quite convincing and informative, and sufficiently compared against competitive methods.", "weaknesses": "In general, the presentation is a bit more jargon heavy than it needs to be, the writing can be greatly simplified to communicate key ideas more clearly, and make them more amenable for uptake.\n\nThe reward refinement seems to invoke a minimax optimization procedure. The paper claims to bypass adversarial IRL methods, but the reward refinement procedure seems to contradict that. Can you clarify the questions below?\n\nIt would be good to clarify where the uniform prior is used and where the reference policy is initialized to the BC cloned policy. More details for how the critic is initialized and trained would help address this question.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We have revised Section 3 to improve clarity and minimize jargon, and revised the end of Section 4 to add more technical details regarding the reference policies and critic pretraining. To answer the questions here: In the coherent reward, the prior policy is always uniform. For deep imitation learning, in the Bellman equation and policy update objective, we replace the prior policy with the BC policy, for extra regularization. Regarding critic pretraining, in the tabular case we use the log policy ratio like the reward. For the deep imitation learning case, we train an MLP with SARSA and the squared Bellman error objective.\n\nWe now answer the remaining questions:\n\n**Reward refinement.** Reward refinement is used in the deep imitation learning implementation. It is ablated in Appendix L.6, Figure 44 of the submission. The results show that it is mainly needed when there are few demonstrations (e.g. 1), and it also helps stabilize convergence in some cases. The reason why it is needed is to stop the RL agent from exploiting errors in the stationary approximation of the policy, which results in overestimation of the coherent reward being larger than desired. This issue is exacerbated in the single demonstration case since the initial BC is less defined and sub-optimal, and the coherent reward is defined in a smaller region of the state-action space, compared to when more demonstrations are available. \n\nOur claim that CSIL bypasses the minimax optimization of adversarial IRL is based on three factors:\n1. The exact algorithm (e.g. in the tabular setting) does not require reward refinement.\n2. The deep learning implementation can still work without reward refinement in some cases.\n3. We use reward refinement to reduce approximation error rather than solve the IRL problem. With a better approximation of a stationary process (e.g. using a non-parametric implementation with a stationary kernel), the refinement step becomes less necessary. \n\n**Policy unlearning.** A randomly-initialized reward and critic network (used in baseline methods such as DAC and PPIL) can have arbitrary optimal actions for a given state. As a result, training a BC-initialized policy with this critic leads to ‘unlearning’ when the optimal actions of the BC policy differ from the arbitrary optimal actions of the randomly-initialized critic. In contrast, for our coherent reward, the optimal action is the maximum likelihood action of the BC policy, which after behavioral cloning should be equal or close to the expert action in the demonstration data. SARSA pretraining of the critic transfers these optimal actions from the reward to the critic. As a result, training the BC policy with this critic should not change the policy’s actions in-distribution since the BC policy, reward and critic are all ‘coherent’ w.r.t. the expert demonstrations.\n\nOutside of the expert data distribution, the optimal action is unknown and the coherent reward should be approximately zero. However, the learned Q function will encourage actions that return to the demonstration distribution since the coherent reward is higher in this region.\n\nWe hope this addresses your questions and concerns. Please let us know if you have follow-up questions.\n"}, {"review_id": "31XUiQDlFm", "reviewer": "Reviewer_qELY", "summary": "The authors propose a hybrid BC and IRL method, which uses a maximum entropy KL-regularized BC policy to define a shaped reward which can be optimized with IRL.  The authors derive a “coherent” reward which is defined as a reward for which the BC policy is optimal, by inverting the soft policy iteration update.  The full method first trains a BC policy, defines the coherent reward based on the policy, then uses soft policy iteration to optimize this reward.  The authors also provide a version of the method for continuous control which requires several additional components.  They compare against other IRL algorithms and BC in a tabular and continuous environments and in online and offline RL settings.  Overall, the method performs well in continuous environments given enough demonstrations and performs comparably with other methods elsewhere.", "questions": "- Is $r_\\theta$ updated in each policy iteration with the new policy, $q_{\\theta_i}$?  If so, does this change the coherence property?\n- For continuous control policies, you state that you use the cloned policy as the prior in forming the reward.  Wouldn’t this lead to zero rewards everywhere because $r(s,a) = \\alpha (\\log q_\\theta(s,a) - \\log q_\\theta(s,a))$?\n", "limitations": "Yes, the note that the BC policy is not always viable as a coherent reward, especially if the task is not solvable with BC even given many demonstrations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors apply a novel perspective of coherence to formulating a hybrid BC and IRL method.\n- The proposed method is well-derived with theoretical backing.\n- Extensive connections to prior works throughout.\n- Experiments in multiple different settings and control tasks.\n", "weaknesses": "- While the tabular form of the method is simple and elegant, the continuous version is quite complex and requires many additional components to work.  This includes training the critic with an additional auxiliary loss, using heteroscedastic regression to fit the expert data, and finetuning the coherent reward with a new objective.  In comparison, most other IL methods can be applied out-of-the-box for both discrete and continuous tasks.  Given its complexity, the proposed method could be difficult to reproduce and tune in new environments.  The authors do demonstrate the continuous method on multiple tasks in different domains and include multiple ablations.  However, many details and all ablations are not included in the main paper.  It would be helpful to at least summarize the results to address the method complexity.\n- The derivation of the method can be difficult to follow and seems overly complicated in places.  For example, my understanding is that the initial policy is trained with KL-regularized BC but the authors use pseudo-likelihoods and stochastic process regression to derive it which seem like interesting connections but not necessary for the derivation.  Providing a simpler derivation or some intuition to guide readers could help make this part clearer.\n\nOverall, these issues are minor, could be addressed with a little re-writing, and do not outweigh the main strengths of the paper.  The authors succeed in deriving and empirically verifying a novel hybrid BC and IRL method with the idea of coherence.  While additional analysis on the robustness of the method and a simplified derivation could help, these are not major concerns that detract from the paper's main contributions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We wish to thank the reviewer for their comments and kind words about the submission.\nWe have revised Section 3 to make the derivation easier to follow and emphasize the key technical points and motivations of the KL-regularized BC.\nWe discuss more details of this text revision in our main rebuttal comment.\nWe have also added more references to the relevant ablation studies in the main text.\n\nWe now address specific concerns:\n\n**Implementation complexity.** In our experience, a deep imitation learning algorithm requires careful implementation details in practice. For example, GAIL / DAC implementations have additional regularization of the discriminator network using regularizers such as spectral norm or gradient penalties. \nIQLearn and PPIL used slightly different objectives (tuned per environment) to add additional regularization.\nWe wanted to be explicit with the implementation details in the main paper for full transparency and to aid the ease of reproduction, as these implementation details are often hidden in the appendix or open-sourced implementation. \nWe believe our extensive ablation experiments also help explain why each aspect is needed.\n\n**Updating reward parameters.** In the tabular setting, $r_{\\theta}$ is fixed. In the function approximation, $r_\\theta$ is refined due to non-stationary approximation error in the policy.\nThis refinement is done to correct any stationary process approximation error out-of-distribution (i.e. where the BC policy doesn't match the prior), rather than alter the BC fit, so the coherency motivation is not violated.\nSee Figure 9 in the Appendix to see a 1D example of how the refinement improves the stationary approximation of the policy.\n\n**Coherent reward prior.** We apologize that the ‘Using the cloned policy as prior.’ section is unclear.\nA uniform policy prior is used for the coherent reward in both the tabular and continuous implementations.\nThe BC policy is not used as the prior in the coherent reward for the reason you correctly identify.\nFor the deep imitation learning implementation, the BC policy is used as the prior in the KL-regularized Bellman equation (Equation 3) and policy objective to regularize the policy optimization. This is to encourage the policy to stay close to the BC policy, and has been used previously in online and offline RL to stabilize the policy updates.\nWe have updated the text in this section so this passage is more clear.\nWhile this switch changes the policy update from which the coherent reward was derived, since the BC policy should match the prior outside of the demonstration distribution due to the stationary property, and we don't desire the policy to change significantly in-distribution, this switch does not change the objective substantially and adds beneficial policy regularization. \n\nWe hope this answers your questions and addresses your concerns. Please let us know if you have additional issues. "}, {"review_id": "6IRaOwhuAA", "reviewer": "Reviewer_nMWn", "summary": "This study seeks to leverage the advantages of Behavioral Cloning (BC) and Inverse Reinforcement Learning (IRL) to develop a sample-efficient imitation learning algorithm. However, the integration of these two approaches is not straightforward, as optimizing the policy using a dynamic reward diminishes the benefits of BC pre-training. To address this challenge, this work derives a shaped reward based on pre-trained BC according to the entropy-regularized policy update. This shaped reward enables policy refinement without compromising the advantages gained from BC, as it remains consistent with BC. Experimental results demonstrate that the proposed Coherent Soft Imitation Learning (CSIL) effectively addresses both online and offline imitation learning tasks in high-dimensional continuous control and image-based scenarios.", "questions": "* How is p(a|s) in Eq 2 derived? It doesn’t seem to appear in [1].\n* How is p(a|s) in Eq 4 derived? It doesn’t seem to appear in [2].\n* Definition 1 appears to be somewhat redundant, given that the information is already provided in Eq 4 and KL-regularized RL objective in the Background section.\n* What does \"classifier\" refer to in Fig 1? Does it correspond to the discriminator in GAIL?\n* In Fig 1, it appears that the color intensity of Ours does not necessarily indicate a greater distance from the data. Does this imply that the agent may be unable to return to expert support in certain regions?\n* Algorithm 1 is vague, for example:\n    * Where is the fine tuning part using additional data(online/offline).\n    * The shape coherent reward uses $\\theta$ but there is no $\\theta$ at the right hand side of $=$.\n    * What’s the exact objective/equation for computing $\\tilde{Q}_n$.\n    * It would be better to include the tricks mentioned in Section 4 into Algorithm 1.\n* The placement of “|” in Lemma 2 seems to be wrong: $p(Y, w| X)$ -> $p(Y | w, X)$\n* What is the influence of the reference policy ($p(a | s)$ or $q_{\\theta_1}(a | s)$)?\n* What would happen if we only use Eq 10 or its improved version to train without BC pre-training?\n* In Fig 27, BC’s performance is better than the initial performance of CSIL, which is different from Fig 31, what’s the reason for this?\n* In Fig 31, there are some tasks where BC’s performance is better than CSIL, e.g HalfCheetah-v2(n=30) and Hopper-v2(n=3). This seems to contradict the feature of CSIL, what’s the reason for this?\n* The scores of the experts are somewhat low at HalfCheetah-v2(8770) and Hopper-v2(2798), it would be better to train on better experts, e.g HalfCheetah-v2($\\ge$12000) and Hopper-v2($\\ge$3500).\n* In Fig 3, why does CSIL exhibit a deteriorating performance in Hopper-v2 as the number of demonstrations increases?\n* Some typos:\n    * Fig 28, again -> against\n    * Fig 36, s hows -> shows\n    * Line 984, and and -> and\n \n1: Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.  \n2: Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, 2018.\n", "limitations": "As mentioned in the paper, the performance of CSIL is constrained by the capabilities of BC. If BC fails to successfully address the task with an adequate number of demonstrations, CSIL will encounter similar limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "* The experiments are extensive and comprehensive.\n* The proposed algorithm is able to solve complex control tasks using only a few demonstrations.\n* The novelty is commendable.\n", "weaknesses": "* The performance is influenced a lot by the efficacy of BC.\n* The algorithm requires some tricks to work well.\n* The readability is not good.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comprehensive review and feedback.\n\nWe were able to address all the issues and address the specific questions below. \n\n**Policy prior $p(a|s)$.** Maximum entropy RL is a specific case of KL-regularized RL when $p(a|s)$ is a uniform distribution.  We mention this in the passage in lines 72 - 78. \n\n**Definition 1.** The inclusion of pseudo-posteriors is to emphasize the connection between KL-regularized optimization and Bayesian inference methods, which was crucial in this work due to the necessity of a stationary posterior policy given the uniform action prior. Repeating similar mathematical results is somewhat unfortunate, but is used to provide a formal definition while also showing how the notion of a pseudo-posterior is general to any KL-regularized optimization problem beyond the RL discussion in Section 2, such as regression task that BC performs.\n\n**Figure 1.** The goal of Figure 1 was to illustrate, on a low-dimensional contextual bandit problem with one state and one action, why the coherent reward with a stationary prior might be preferable to the classifier-based reward used by methods such as GAIL. Because it's a contextual bandit problem, there is no dynamical system.\nThe agent will sample actions from the yellower regions.\nWe will make this clearer in the caption.\nThe coherent reward is constructed using a stationary nonparametric Gaussian process, so the color depends on the data points and the GP kernel.\nThe main takeaway we wanted to communicate is that the coherent reward captures the shape of the true reward faithfully by using regression and the log ratio.   \n\n**Reference policy.** This design decision is expanded in ‘Using the cloned policy as prior.’.\nIf the prior $p(a|s)$ is the uniform distribution, then using it results in the maximum entropy soft Bellman equation.\nIf instead the BC policy is used, like in the continuous setting, the soft Bellman equation and policy objective are now regularized explicitly against the initial BC policy.\nWhile this switch changes the policy update from which the coherent reward was derived, since the BC policy should match the prior outside of the demonstration distribution due to the stationary property, and we don't desire the policy to change significantly in-distribution, this switch does not change the objective too much and adds some very useful regularization. \n\n**BC vs CSIL performance.**\nThe BC policy used by CSIL is trained in a simpler way than the BC baseline.\nFor the BC baseline, we use a training protocol which learns for many iterations (e.g. 1M) and uses policy evaluation during training to pick the best performing policy, to mitigate the effects of under- and overfitting.\nFor CSIL, we thought this protocol was too elaborate, so we instead use fewer iterations and early stopping to obtain a reasonable initial BC policy, which simplifies the algorithm and implementation.\nThe BC baseline in Fig, 27 uses the training protocol described above, whereas the Fig. 31 ablation compares the initial CSIL BC policy. \n\n**Initial drop in CSIL performance.** We agree the initial performance below BC (CSIL) in Fig. 31 is unexpected.\nThis happens rarely and the policy surpasses BC quickly.\nThe best explanation we can give for the initial performance difference is that the initial $Q$ function may not be fully 'coherent' after SARSA pretraining, so the initial policy may 'unlearn' briefly during the initial updates.\nThis is a consequence of having to use black-box function approximation for the critic.\n\n**Online Hopper-v2 performance.** Looking at Figure 27 and 31, it seems that CSIL on Hopper has the highest performance variance across the gym tasks, and is also quite high variance in the initial BC (CSIL) performance.\nThis suggests that the Hopper data is harder to learn effective coherent rewards from; in other words, that it struggles to differentiate expert and non-expert actions and do good credit assignment.\nAlso note that for Figure 3 we are pessimistic and use the best 25th percentile return, to show a best worst-case performance. In Figure 3, the selected Hopper-v2 performance is right at the end of training where both median performance and the interquartile range happen to drop. If you instead compare the step-based curves in Figure 27, the performance actually looks similar across the different dataset sizes but is quite high variance.\n\n**Expert performance.** We built on the datasets shared from peer-reviewed prior work (Orsini et al. [11]), who also used Acme and open-sourced their expert demonstrations for researchers to build off. We cannot comment on the performance of these demonstrations.\n\n**Lemma 2.** We believe we have written the result down correctly in Lemma 2. The DPI arises from manipulating a joint probability distribution and applying Jensen’s inequality, and in our case we use $p(y, w | x) = p(y|w, x)p(w)$ as our joint of interest. \n\n**Algorithm 1.** We wanted to keep Algorithm 1 high level so it captures a general description of CSIL within the available space.\nWe have now added an additional algorithm in the Appendix that describes the SAC-based implementation and its additional tricks and highlighted it in the main text at the end of Section 4.\nTo answer your questions:\n* The soft policy iteration loop at the end of the algorithm performs RL\n* We have fixed the math typo in the reward\n* $Q_n$ is updated using the soft Bellman equation in Equation 3. For SAC, learning uses target networks and the squared Bellman error.\n\n**CSIL without BC.** This is an interesting question. In theory (and the tabular case) it should work.\nWe ran an ablation (BC iterations = 0) on the Gym tasks and added it to the rebuttal PDF.\nCSIL without BC it doesn't seem to work that well on Gym, probably because the reward learning is only tuned for refinement and the 'faithful' heteroscedastic regression fix is not done in the refinement loss. \n\nPlease let us know if you have additional questions."}, {"review_id": "RwCvTY8ZtD", "reviewer": "Reviewer_BA8W", "summary": "This paper studies the imitation learning (IL) problem. There are two main classes of IL methods: behavioral cloning (BC) and inverse reinforcement learning (IRL), each with its own advantages. This paper proposes an IL method Coherent Soft Imitation Learning (CSIL) which combines BC and IRL. CSIL first learns a reward function named “coherent reward” under which the BC policy is optimal, and then performs soft policy iteration on the learned coherent reward. The authors evaluate CSIL on a wide range of settings and tasks including online and offline IL for high-dimensional and image-based continuous control tasks.", "questions": "1. Was Figure 1 obtained from experimental results? If so, please provide a brief description of the experimental setup used to obtain the results shown in Figure 1.\n2. Algorithm 1 does not provide a clear description of the proposed CSIL algorithm. First, it does not explain how to initialize the shaped critic and only shows a fixed point equation where both sides depend on the critic to be learned. Additionally, Algorithm 1 does not clearly demonstrate how CSIL utilizes online interactions, offline samples, or the dynamic model.", "limitations": "Please see the detailed reviews in the Weakness part.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed algorithm CSIL is novel according to my knowledge.\n2. The authors evaluate CSIL on a wide range of settings and tasks including tabular tasks, online and offline IL from agent demonstrations, IL from human demonstrations from states and images.", "weaknesses": "1. The proposed method CSIL fails to combine the advantages of both BC and IRL, which is the main contribution claimed in this paper. In CSIL, the coherent reward function is learned under the principle that the BC policy is the (soft) optimal policy with that reward function. So the best policy we can expect to obtain from performing soft policy iteration on the learned coherent reward function is the BC policy. Besides, CSIL does not retain the main merit of IRL: addressing the compounding errors issue of BC. It is known that IRL leverages the state-action distribution matching principle to address the compounding errors issue in BC [1, 2, 3]. However, CSIL does not inherit this principle to learn the reward function. Therefore, the advantage of CSIL over BC and IRL is unclear.\n2. The theory about the coherent reward (Theorem 1) is not novel. In [4], they have derived the feasible reward formula in the maximum entropy RL setting, which is exactly the case when the prior policy is a uniform policy in Theorem 1 (this choice of prior policy is also used in experiments in this paper). However, the authors do not discuss this existing result.\n3. The writing in this paper needs improvement, and some parts are confusing. Even though I have read most of the IL papers in the reference, I still found it difficult to follow this paper. Please refer to the detailed comments in the Question section.\n4. In experiments, CSIL performs worse than the existing method DAC on the online IL in MuJoCo tasks. DAC outperforms CSIL in HalfCheetah-v2 and Hopper-v2 while having a competitive performance with CSIL in Ant-v2 and Walker2d-v2. Besides, the offline IL setting considered in this paper is actually the offline IL with a supplementary dataset setting instead of the pure offline IL setting. However, the authors does include the methods in the offline IL with a supplementary dataset setting [5, 6] for comparison.\n\nReferences:\n\n[1] Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. In Conference on Robot Learning, 2019.\n\n[2] Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 2020.\n\n[3] Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 2020.\n\n[4] Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. In Advances in Neural Information Processing Systems, 2021.\n\n[5] Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. In Proceedings of the 10th International Conference on Learning Representations, 2022.\n\n[6] Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In Prooceedings of the 39th International Conference on Machine Learning, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their comments and critical feedback. \n\n**Weakness 1.**  The reviewer highlights an important aspect of CSIL: If the demonstration data covers the entire state-action space and the BC fit is exact, then indeed CSIL will leverage and match this expert BC policy. But in this case, BC should be all you need, because there should be no compounding errors if the demonstration coverage is ideal.\nHowever, in the case of where the expert demonstrations do not cover the entire state-action space, then there will be regions where the BC policy is undefined and likely ineffective, so we perform RL to improve the policy in these gaps.\nThis is where CSIL comes in, by using the BC policy to construct the coherent reward and then using this reward to improve using additional environment interactions.\nThis property is in contrast to BC, which is a non-interactive algorithm and cannot interact with the environment to improve the policy.\n\nWe demonstrate this empirically in the tabular inverse optimal control setting (Table 1), where there are no approximations. In the sparse task, the BC policy has high performance, but is brittle to the wind disturbance due to compounding errors when out-of-distribution. CSIL is able to learn out-of-distribution actions that resolve this weakness to the same level as the best IRL baselines. The value functions in Appendix K.1 show that CSIL and GAIL also learn visually similar value functions. This validates our statement above that CSIL improves BC outside of the demonstration distribution.\n\nWe are also able to show this in the function approximation setting. For example, in Figures 32 CSIL is able to learn an effective high-dimensional manipulation policy on door-v0 from one demonstration, when its initial BC policy is non-performant. Figure 28 shows that DAC and SAC-from-demonstrations requires 3x the experience to reach similar performance on door-v0 with one demonstration. \nThis is evidence that CSIL learns an effective shaped reward from the demonstration data. \n\nIn summary: \n * CSIL improves on BC by being able to use additional interactions with a learned reward. \n* CSIL is an inverse RL method as it inverts the RL policy update and learns a shaped reward.\n* These properties are validated in our experimental results across a range of settings. \n\n**Weakness 2.** We wish to thank the reviewer for highlighting Theorem 1 and Remark 3 of Cao et al. (2021). We will mention this work alongside our acknowledgement of Jacq et al. in the main text, who also derive a similar maximum entropy policy inversion in 2019 and inspired our approach. We will also replace ‘derive’ with ‘use’ in our first contribution summary (line 36) to emphasize that our contribution of combining the policy inversion result with BC for IRL, rather than the mathematical result, although our relative entropy inversion is also a slightly more general derivation than the maximum entropy one. \n\n**Weakness 3.** We have restructured Section 3 to be easier to follow (see our main rebuttal response). If you have specific feedback for which passages were confusing or hard to follow, that would be appreciated and we would be happy to incorporate that feedback. \n\n**Weakness 4.**  In the paper we tried to show the performance of CSIL in many diverse domains, even when baselines may be stronger.\nWe believe that it is unrealistic to expect a new approach to beat all existing baselines on all tasks.\nWhile DAC is indeed better in some of the online MuJoCo locomotion tasks, CSIL still solves these tasks, whereas DAC performs less well on the more complex Adroit and Robomimic tasks that CSIL can solve. Moreover, our implementation of DAC was based on the extensive hyperparameter and regularizer investigation of Orsini et al. [11], which makes it a strong baseline. \n\nWe agree with the reviewer's suggestion for explicit offline imitation baselines for the offline locomotion tasks. \nWe have run DemoDICE [1] and SMODICE [2] in our offline setting.\nIn the interest of time, we used the author’s original open-source implementations of these algorithms rather than reimplement them in Jax and Acme in our codebase.\nWe discuss these algorithms and their performance in our main rebuttal comment and include the experimental results in the attached PDF. \n\n**Q1.** Figure 1 is a toy contextual bandit problem used for illustration purposes. The generating script is 'pull_figure.py' in the provided code in the supplementary material.\nThe classifier uses the same spectral-normalized network used in Acme's DAC implementation. The CSIL reward was computed using a non-parametric Gaussian process with a stationary squared exponential kernel. The aim of this figure was to illustrate why our log ratio using regression with a stationary process might be preferable to a classification approach. We have added more context to the figure caption.\n\n**Q2.** Algorithm 1 was written as a general summary of both the tabular and deep implementation of CSIL, which is why it lacks specificity. \nIn the tabular case, the critic is initialized with the log policy ratio, exactly like the coherent reward.\nIn the continuous case, the critic is pre-trained using SARSA and the squared Bellman error.\nThe soft policy iteration (SPI) loop at the end of the algorithm could be any oracle-based, model-free or model-based SPI routine that uses the coherent reward.\nFor the deep learning implementation, we use SAC.\nWe have updated Algorithm 1 to improve clarity and added a second algorithm to the Appendix that specifically describes the SAC-based online deep RL implementation.\n\nWe hope this rebuttal addresses your concerns. Please let us know if you have remaining concerns or further questions.\n\n[1] DemoDICE: Offline Imitation Learning with Supplementary Imperfect Demonstrations, Kim et al., ICLR 2022\n\n[2] Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matching, Ma et al., ICML 2022\n\n"}]

Author rebuttal:
We wish to thank all the reviewers for their comments and feedback. We believe we have been able to address all issues.

To summarize the four reviews:

**General Positives**
* CSIL is novel and interesting [tsdg, qely, nmwn, ba8w]
* The proposed method is sound theoretically [qely]
* Extensive connections to prior works [tsdg, qely]
* Comprehensive experiments, baselines, settings and / or ablations [tsdg, qely, nmwn, ba8w]

**General Weaknesses**
1. Clarity of Algorithm 1, Section 3 and the implementation details [tsdg, qely, nmwn, ba8w]
2. Missing an offline imitation learning baselines [ba8w]
3. Missing citation of Cao et al.’s similar policy inversion [ba8w]

**Our improvements**
1. Added missing technical details and improved clarity throughout the text, including Algorithm 1
    1. Restructured Section 3 to start with the policy inversion and then use the coherent reward to motivate pseudo-posteriors and regularized BC
    2. Added the coherent reward in an explicit and expanded definition block, and used an enumerated list to be more explicit about the regularized BC motivating points and details
    3. Added a second algorithm to the Appendix that describes the SAC-based implementation details and includes the additional implementation details from Section 4
2. We have run two additional offline imitation learning baselines: DemoDICE [1] and SmoDICE [2] and added them to Figure 5 and 29 (see PDF) 
3. Added discussion of Cao et al. w.r.t. our Theorem 1

While we have restructured and rewritten parts of Section 3, we ensured the underlying topics and details covered content remains unchanged.  

We are also preparing to open-source the code, so we can release our implementation if the submission is accepted.  

The attached PDF contains the offline results with DemoDICE and SMODICE baselines, and an additional ablation study requested by nMWn that runs CSIL without BC pretraining.

We hope this addresses the reviewers concerns. Please let us know if there are remaining issues. 

**A note on the new SMODICE and DemoDICE baselines.**
In the interest of time, we ran the author’s released code rather than implement the method in Jax and Acme like the other baselines.
The original papers used the D4RL datasets (expert and random) and hundreds (100-200) of demonstrations.
We ran the code for ten seeds with our offline setting (expert and full-replay datasets, with 1, 3, 10, 30 demonstrations).
The main difference is the use of the Orsini et al. expert datasets in our implementation and the use of the D4RL expert datasets in DemoDICE and SMODICE.
However, we have normalized to [0, 1] based on the expert performance in each algorithm’s corresponding expert dataset. 

The interesting design decisions between SMODICE and DemoDICE are a discriminator-based reward, state-based value function and weighted BC-based policy update. These aspects separate them from the current offline baselines. The last two aspects are attractive for offline learning, as they mitigate the issue of estimating high value in unobserved actions.
SMODICE and DemoDICE are very similar methods.
The biggest difference between the methods is that DemoDICE learns a state-action reward, while SMODICE learns a state-based reward.

Our experiments show these are both two very strong baselines, especially for the Ant environment.
In particular, they also perform well in the one-demonstration setting.
However, CSIL maintains competitive performance on HalfCheetah-v2, Hopper-v2 and Walker-v2 for 10 and 30 demonstrations.

We would also like to note that some implementation details, such as weighted BC-based policy update, could be incorporated into CSIL, since it is agnostic to the SPI implementation (e.g. use MPO rather than SAC). The primary goal with our offline experiment was to compare CSIL against its similar baselines IQLearn and PPIL, since they share many implementation details. 
 
[1] Demodice: Offline imitation learning with supplementary imperfect demonstrations, Kim et al., ICLR 2022

[2] Versatile offline imitation from observations and examples via regularized state-occupancy matching, Ma et al., ICML 2022


Meta-review: Dear Authors,

After an exhaustive review and contemplation of the reviewers' feedback, the decision to accept the paper for NeurIPS 2023 has been reached.  We acknowledge the strengths that the work brings to the field, particularly in the domain of reward refinement and its implications in deep imitation learning. The clarification on the usage of reward refinement, especially when fewer demonstrations are present, is appreciated. The points related to CSIL bypassing the minimax optimization of adversarial IRL, through the three mentioned factors, have been comprehensively understood. Furthermore, the reviewers recognize the value of the paper’s exploration of the implementation complexity, and appreciate the transparency maintained by highlighting these often overlooked details in the primary content, which indeed aids in the reproduction of experiments. 

After a comprehensive inspection, the conclusion has been made to accept the paper. We sincerely hope that this feedback serves as a useful guide and looks forward to witnessing the paper's contributions to the NeurIPS community.

Best regards,

Area Chair
Decision: Accept (spotlight)