LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learning
Authors: Firas Al-Hafez
Davide Tateo
Oleg Arenz
Guoping Zhao
Jan Peters
Venue: ICLR 2023 poster (2023)
Keywords: Inverse Reinforcement Learning
Imitation Learning
Reward Regularization
Deep Reinforcement Learning

Abstract: Recent methods for imitation learning directly learn a $Q$-function using an implicit reward formulation rather than an explicit reward function. However, these methods generally require implicit reward regularization to improve stability and often mistreat absorbing states. Previous works show that a squared norm regularization on the implicit reward function is effective, but do not provide a theoretical analysis of the resulting properties of the algorithms. In this work, we show that using this regularizer under a mixture distribution of the policy and the expert provides a particularly illuminating perspective: the original objective can be understood as squared Bellman error minimization, and the corresponding optimization problem minimizes a bounded $\chi^2$-Divergence between the expert and the mixture distribution. This perspective allows us to address instabilities and properly treat absorbing states. We show that our method, Least Squares Inverse Q-Learning (LS-IQ), outperforms state-of-the-art algorithms, particularly in environments with absorbing states. Finally, we propose to use an inverse dynamics model to learn from observations only. Using this approach, we retain performance in settings where no expert actions are available.

Published as a conference paper at ICLR 2023
LS-IQ: IMPLICIT REWARD REGULARIZATION FOR INVERSE REINFORCEMENT LEARNING
Firas Al-Hafez', Davide Tateo', Oleg Arenz', Guoping Z Jan Peters
Intelligent Autonomous Systems, LLocomotion Laboratory
3 German Research Center for AT (DFKT), Centre for Cognitive Science, Hessian.Al TU Darmstadt, Germany {name. Surneme-@tu-deee tddt. de
ABSTRACT
Recent methods for imitation learning directly learn a Q-function using an implicit reward formulation rather than an explicit reward function. However, these meth- ods generally require implicit reward regularization to improve stability and often mistreat absorbing states. Previous works show that squared norm regularization on the implicit reward function is effective, but do not provide a theoretical anal- ysis of the resulting properties of the algorithms. In this work, we show that using this regularizer under a mixture distribution of the policy and the expert provides particularly illuminating perspective: the original objective can be understood squared Bellman error minimization, and the corresponding optimization problem minimizes bounded D-veereeee between the expert and the mixture distribu- tion. This perspective allows us to address instabilities and properly treat absorb- ing states. We show that our method, Least Squares Inverse Q-Learning (LS-IQ), outperforms state-of-the-an algorithms, particularly in environments with absorb- ing states. Finally, we propose to use an inverse dynamics model to learn from observations only. Using this approach, we retain performance in settings where no expert actions are available
INTRODUCTION
Inverse Reinforcement Learning (IRL) techniques have been developed to robustly extract behaviors from expert demonstration and solve the problems of classical Imitation Learning (IL) methods (Ng et al. 1999 Ziebart et al.] 2008). Among the recent methods for IRI., the Adversarial Imitation Learning (All.) approach (Ho Ermon [0166] Fu et al., 2018 Peng et al.) (02T), which casts the optimization over rewards and policies into adversarial setting, have been proven particularly successful. These methods, inspired by Generative Adversarial Networks (GANs) (Goodfellow) cet al. 2014), alternate between learning discriminator, and improving the agent's policy w.r.t. a reward function, computed based on the discriminator's 's output. These explicit reward methods require many interactions with the environment as they learn both reward and value function.
Recently, implicit reward methods (Kosrrikov et al., 20200] Arenz & Neumann 2020; Garg et al.) 2021 have been proposed. These methods directly learn the Q-function, significantly accelerating the policy optimization. Among the implicit reward approaches. the Inverse soft Q-Learning (IQ- Learn) is the current state-of-the-art. This method modifies the distribution matching objective by including reward regularization on the expert distribution, which results in minimization of the x². divergence between the policy and the expert distribution. However, whereas their derivations only consider regularization on the expert distribution, their practical implementations on continuous control tasks have shown that regularizing the reward on both the expert and policy distribution achieves significantly better performance.
The contribution of this paper twofold: First, when using this regularizer, we show that the result- ing objective minimizes the x2 divergence between the expert and mixture distribution between the expert and the policy. We then investigate the effects regularizing w.r.t. the mixture distribution on the theoretical properties of IQ-Learn. We show that this divergence is bounded, which translates
'The code is available at http::::/githbbb.oooo//

---

Published as a conference paper at ICLR 2023
to bounds on the reward and Q-function, significantly improving learning stability. Indeed, the sulting objective corresponds to least-squares Bellman error minimization and is closely related to Soft Q-Imitation Learning (SQIL) (Reddy et al., 2020). Second, we formulate Least Squares Inverse Q-Learning (I.S-IQ), a novel IRI. algorithm. By following the theoretical insight coming from the analysis of the x2 regularizer, we tackle many sources of instabilities of the IQ-Leemn approach: the arbitrariness of the Q-function scales, exploding Q-functions targets, and reward bias Kostrikov ct al. (2019). i.e., assuming that absorbing states provide the null reward. We derive the LS-IQ algorithm by exploiting structural properties of the Q-function and heuristics based on expert opti- mality. This results in increased performance on many tasks and, in general, more stable learning and less variance in the Q- Q-function estimation. Finally, we extend the implicit reward methods to the IL. from observations setting by training an Inverse-Dynamies Model (IDM) to predict the expert actions, which are no longer assumed to be available. Even in this challenging setting, our approach retains performance similar to the one where expert actions are known.
Related Work. The vast majority of IRL and IL methods build upon the Maximum Entropy (MaxEnt) IRL framework (Ziebart 2010). In particular, HH & Ermon (2016) introduce Generative Adversarial Imitation Learning (GAIL), which applies GANs to the IL problem. While the original method minimizes the Sensen-Shannan divergence to the expert distribution, the approach is ex- tended to general f-divergences (Ghasemipour et al.] (019). building on the work of Nowozin al.. 2016). Among the f-divergences, the Pearson x" divergence improves the training stability for GANs (Mao al., (017) and for AIL (Peng et al. 2021). Kostrikov et al. (219)) introduce a replay buffer for off-policy updates of the policy and discriminator. The authors also point out the problem of reward bias, which is common in many imitation learning methods. Indeed, AIL methods implic- itly assign null reward to these states, leading to survival or termination biases, depending on the chosen divergence. Kostrikov et al. (2020) improve the previous work introducing recent advances from offline policy evaluation (Nachum et al. 2019). Their method, ValueDice, uses an inverse Bell- man operator that expresses the reward function terms of its Q-function, to minimize the reverse Kullback-Leible Divergence (KL) to the expert distribution. [Arenz & Neumann (2020) derive non-adversarial formulation based on trust-region updates on the policy. Their method, O-NAIL, uses standard Soft-Actor Critic (SAC) (Haarnoja et al.] 2018) update for policy improvement. O- NAIL can be understood as an instance of the more general IQ-Learn algorithm (Garg et al. 2021), which optimize different divergences depending on an implicit reward regularizer. Garg et al. 2021 also show that their algorithm achieves better performance using the x² divergence instead of the reverse KL. Reddy et al. 2020) propose method that uses SAC and assigns fixed binary rewards to the expert and the policy. Swamy et al. (2021) provide a unifying perspective on many of the methods mentioned above, explicitly showing that GAIL, ValueDice, MaxEnt-IRU and SQIL can be viewed as moment matching algorithms. Lastly, (Sikchi et al. (2023) propose a ranking loss for All., which trains a reward function using least-squares objective with ranked targets.
2 PRELIMINARIES
Notation. A Markov Decision Process (MDP) is a tuple AS Mo)) where S is the state space, A is the action space, P is the reward function, 7 is the discount factor, and μo S + R+ is the initial state distribu- tion. At each step, the agent observes a state s E S from the environment, samples an action E A using the policy π. S x A + IR, and transitions with probability P(s'|s, a) into the next state s' E S, where it receives the reward r(s,a). We define an occupancy measure (s, = Tta s) Et= 7'u7 (s). where H77 (s') = San M7 (s)s s, a,)dda ds is the state distribution for >> O, with μā (s) = po(s). The occupancy measure allows us to denote the ex- pected reward under policy π as Ep, [r(s, a)] & E 7r((s,, ):)]. where so ~ no. at ~ (.lss) and 8141 ~ P(.(ata.)) a.) for > 0. Furthermore, R = {x : x A + IR} denotes the set of functions in the state-action space and To denotes the extended real numbers IR. U (+00). We refer to the soft value functions as (ss) and @(s,,), a). while we use V(s) and Q(s, a) to denote the value functions without entropy bonus.
Mo), A S - R$ is the transition kemel, r S x A - R
Inverse Reinforcement Learning as an Occupancy Matching Problem. Given a set of demon- strations consisting of states and actions sampled from an expert policy næ. IRL aims at finding reward function r(s, a) from a family of reward functions R R assigning high reward to
2

---

Published as a conference paper at ICLR 2023
samples from the expert policy TE and low reward other policies. We consider the framework pre- sented in Ho & Ermon (0166) which derives the maximum entropy IRL objective with an additional convex reward regularizer 1yp RRSAA_ - I from an occupancy matching problem
max (p, x) max reR
min -HH,(T) - Ep. (n(s, menl
Ep-g (r(s, a)] - (r)),,
(I)
rER rell
with the space of policies II RSXAA, the discounted cumulative entropy bonus H₂(T)
1- logg [{aas))]] and a constant 3 controlling the entropy bonus. Note that the inner optimiza- tion maximum entropy Reinforcement Learning (RL) objective (Ziebart 2010), for which the optimal policy is given by
" (als) Z.
exp (Q)),))),
where Z, = exp (s,, a) dā is the partition function and (s,, a) is the soft action-value function, which is given for certain reward function by the soft Bellman operator (8* ()((s a) = rr(s,) +
yllg (s'), where (s') (.) (s, a) log [[ass)].
Garg et al. (2021) transform Equation | from reward-policy space to Q-policy space using the inverse soft Bellman operator (), a) = ( -Pp V= (s") to get one-to-one correspon-
dence between the reward and the Q-function. This operator allows to change the objective function Lp from reward- -ewarc-policy to Q-policy space, from now on denoted as Jp
max min (p T) = max min Jp(Q. m) FER mEll
QEΩ *ei
where = SxA is the space of Q-functions. Furthermore, they use Equation 2 to extract the optimal policy Th given a Q-function to drop the inner optimization loop in Equation such that
max JQ(Q, m = max Epgg [((s.a) - -B.. ((1na) V (s')] Qcn
(T))
(4)
Qen
p,, [(s..) a) - -E P(.ssa) [V (s') vi, (t Q).
Garg et al. (2021) derive a regularizer enforcing an L2 norm- penalty on the reward on state-action pairs from the expert, such that (r) = e pp [r(s, a)²] with e being regularizer constant. However, in continuous action spaces, this regularizer causes instabilities. In practice, Garg et al. (2021) address this instabilities by using the regularizer to the mixture
Practical Reward Regularization.
(a(r) a CBEr [r(s,a)"] (1 (1-oo) a) c.p,, [r(s,a))].
(5)
where is typically set to 0.5. It is important note that this change of regularizer does not allow the direct extraction of the policy from Equation [ anymoree. Indeed, the regularizer in Equation also depends the policy. Prior work did not address this issue. In the following sections, will provide an in-depth analysis of this regularizer, allowing us to address the aforementioned issues and derive the correct policy update. Before introduce our method, we use Proposition A. in Appendix A to change the objectives Lp and J, from expectations under occupancy measures to expectations under state-action distributions dig and dx, from now on denoted as L and J, respectively.
LEAST SQUARES INVERSE Q-LEARNING
In this section, we introduce our proposed imitation learning algorithm, which is based on the oc- cupancy matching problem presented in Equation | Jsiing the regularizer defined in Equation We start by giving an interpretation of the resulting objective as a x² divergence between the expert distribution and mixture distribution of the expert and the policy. We then show that the regular- izer allows us to cast the original objective into Bellman error minimization problem with fixed binary rewards for the expert and the policy. An RL problem with fixed rewards is a unique setting, which we can utilize to bound the Q-function target, provide fixed targets for the Q-function on expert states instead of doing bootstrapping, and adequately treat absorbing states. However, these techniques need to be applied on hard Q-functions. Therefore, we switch from soft action-value functions Q to hard Q- f-fuuctions, by introducing an additional entropy critic. We also present a reg- ularization critic allowing to recover the correct policy update corresponding to the regularizer in Equation Finally, we propose to use an IDM to solve the imitation learning from observations problem.
3

---

Published as a conference paper at ICLR 2023
3.1 INTERPRETATION AS A STATISTICAL DIVERGENCE
Ho Ermon ( showed that their regularizer results in Shassnnn divergence minimiza- ton between the expert's ' and the policy's 's state-action distribution. Similarily, Garg et al. (202T) showed that their regularizer =/ results in minimization of the x² divergence, However, the regularizer presented Equation not investigated yet. We show that this regularizer minimizes x2 divergence between the expert's state-action distribution and mixture distribution between the expert and the policy. Therefore, we start with the objective presented in Equation [ and note that strong duality - maX,er min L minxen m L - follows straightforwardly from the minimax theorem (Von Neumann 1928) as - (T), -E._ _Es_ a)] and (b(r) are convex in n, and -Ea, (r(s, a)], [ a)] and (r) are concave in r (Ho & Ermon 2016). We express the x" divergence between the expert's distribution and the mixture distribution using its variational form,
2x (d= go tod.
=sup2
(Fddg (Ba.gg [r(s,a)] (1s,aa]-T, - Fa (((s, =(a)"
=supBat+ [r(s E., [r(sa) -ca [r(s,a))] - -c(1-cc1100)0 a()-a)Ea, [r(s,a)]], (6)
with the regularizer constant = 1/2 and a 1/2. Now, if the optimal reward is in R. the original objective from Equation becomes an entropy-regullaiized x² divergence minimization problem
max min L = min 2x" (d=x || doggtde) -3H(T)) 2
rER mENl =Ell
Equation 7 tells us that the regularized IRL objective optimizes the reward to match a divergence while optimizing the policy to minimize the latter. When the divergence to be matched is unbounded, the optimal reward is also unbounded, causing instability during learning. Unlike the x² xivveeegee between the agent's and the expert's distribution, the x² diverregee to the mixture distribution is bounded [0, 1/4] as shown in Proposition [A.3] and its optimal reward
de (8. de
a*(,,a)
(8)
e d (s,a) + (s.
is also bounded in the interval [-1/s, 1/c as shown in Proposition A.2
3.2 REINFORCEMENT LEARNING PERSPECTIVE ON DISTRIBUTION MATCHING
In the following, we present a novel perspective Equation allowing to better understand the effect of the regularizer. Indeed, for the regularizer defined in Equation 5 we can interpret this objective an entropy-regularize least squares problem, as shown by the following proposition:
Proposition 3.1 Let (g(,)))) = (s* a) be the implicit reward function of Q-function, then for (((r)) = cBglrg(s, a) with di(s,a) a) = adng a) + (1 - a.) (s, a). the solution of Equa- tion i under state-action distributions equals the solution of an entropy-regularized least squares minimization problem such that arg min L(Q, RT) arg maxqeá [QQ, R8) with
C(Q, = &.E.ing (rg(()))) Tmmaa) )
+ (1 aaE [(ro(s.a) - -rmin)
3 (((T)).
(9)
where Tmax
and Tmin 2(1 23'
20c
The proof is provided in Appendix A.3
The resulting objective in Equation 9 is very similar to the one in the Least Squares Generative Adversarial Networks (LSGANs) (Mao et al. 2017) setting, where rg (s, a) can be interpreted as the discriminator, Ymax can be interpreted as the target for expert samples, and Miin can be interpreted as the target for samples under the policy и. For a 0.5 and c == 1. resulting in T'mx = 1 and T'in = 1, Equation differs from the discriminator's objective in the LSGANs setting only by the entropy term.
Now replacing the implicit reward function with the inverse soft Bellman operator and rearranging the terms yields
L(Q, m²) =alBdn [0((5.0) (Fima + +EE
(10)
+ (1 a)liing (Q(6.00) - (rmin + Eee (F + $ ((m)))
- =a (d fmax) ) + (1-a) 5d( rman) + 2 H(TT)
(11)
4

---

Published as a conference paper at ICLR 2023
where 8 is the squared soft Bellman error. We can deduce the following from Equation 11
regeuuaariee IRL under a mixture can be seen as an RL problem with fixed rewards Tmax and Tmin for the expert and the policy. This insight allows us to understand the importance of the regularizer constant c: it defines the target rewards and, therefore, the scale of the Q-function. The resulting objective shows strong relations to the SQIL algorithm, in which also fixed rewards used. However, SQIL uses mmax = and mmmn = O, which is infeasible in our setting for a < 1. While the entropy term appears be another difference, we note that does affect the critic update, where z is fixed. As in SQIL, the entropy is maximized by extracting the MaxEnt policy using Equation
Stabilizing the training in fixed reward setting is straightforward. We can have clean solution to the reward bias problem c.f., Section [.4 -, and we can provide fixed Q-target for the expert and clipped Q-function targets for the policy c.f., Section 3.5 3.7 to improve learning stability significantly. However, we must switch from soft to hard action-value functions by introducing an entropy critic to apply these techniques. Additionally, we show how recover the correct policy update corresponding to the regularizer in Equation by introducing regularization critic.
3.3 ENTROPY AND REGULARIZATION CRITIC
We express the Q-function implicitly using ((, Q(s,a) + Ht (s, decomposing it into hard Q-function and an entropy critic
H (s, Ep, Σ
B log o ((r++1/s444))
(12)
This procedure allows us to stay in the MaxEnt formulation while retaining the ability to operate on the hard Q-function. We replace the soft inverse Bellman operator with the hard optimal inverse Bellman operator (s.) = Q(s,a) 7B (s'), with the optimal value function V*(s)= = max, (s)
As mentioned before, the regularizer introduced in Equation 3 incorporates yet another term depend- ing on the policy. Indeed, the inner optimization problem in Equation[] the term in the brackets is
not purely the MaxEnt problem anymore, but includes the term -kE,, [r(s, a) with c(1 a). To incorporate this term into our final implicit action-value function Q (s, as, we learn additional regularization critic
oc
C( EP..
(13)
such that inner minimization problem in Equation 2
()" a) 1s CC(s, a). Using Q', we obtain the exact solution the In practice, we learn single critic G" combining H and C. We train the latter independently using the following objective
min & min Ed, (s, - (Is (o a)2 + Bypp
(14)
which entropy- regularize Bellman error minimization problem given the squared implicit reward To scaled by k.
3.4 TREATMENT OF ABSORBING STATES
Another technical aspect neglected by IQ-Learn is the proper treatment of absorbing states. Garg al. (2021) treat absorbing states by adding indicator v-where if is a terminal state front of the discounted value function in the inverse Bellman operator
(T ()(s a) Q(),a) - (1 -)).,E..-P((....) VT
(15)
This inverse Bellman operator is obtained by solving the forward Bellman operator for r(s, a) under
the assumption that the value of absorbing states is zero. However, as pointed out by Kostrikov
al. (2019), such an assumption may introduce termination or survival bias; the value of absorbing states also needs to be learned. Our perspective provides a clear understanding of the effect of the inverse Bellman operator in Equation [5| The objective in Equation 10 will regress the Q- function of transitions into absorbing states towards Fmax or T'inn respectively. However, based on Equation the implicit reward of absorbing states should be regressed toward Tmax or Pmin-
5

---

Published as a conference paper at ICLR 2023
Instead, we derive our inverse operator from the standard Bellman operator while exploiting that the value of the absorbing state 8 is independent of the policy क
(16) We further exploit that the value of the absorbing state can be com- puted in closed form (sA)) - where A equals mmax on ex- pert states and I'mi on policy states. Please note that the correspond- ing forward Bellman operator converges to the same Q-function, de- spite using the analytic value of absorbing states instead of bootstrap- ping, as we show in Appendix A.5 When applying our inverse opera-
1.0
tor in Equation to Equation 9 we correctly regress the Q-function
0.6
of transitions into absorbing states towards their discounted return. We show the resulting full objective in Appendix A.4
0.4
0.2
We show the effect of our modified operator the toy task depicted in Figure (top), where the black point mass is spawned in either of the four dark blue squares and has reach the green area in the mid- dle. Once the agent enters the red area, the episode terminates. The expert always takes the shortest path to the green area, never visiting the red area. The operator proposed by IQ-Learn does not sufficiently penalize the agent for reaching absorbing states, preventing the IQ- Learn agent from reaching the goal consistently, as can be seen from the orange graph in Figure [ (bottom). contrast, when using our
0.0
50
100
150
200
I0-Corrected
Figure 1: Point mass toy task (top) with success rate plot (bottom). Here, we compare the standard IQ-Learn operator to the modified operator.
operator Thiqg the agent solves the task successfully.
3.5 AN ALTERNATIVE FORMULATION FOR THE EXPERT RESIDUAL MINIMIZATION
The first term in Equation defines the squared Bellman error minimization problem on the distri- bution dag
6 rmax) = alEi [(ra(s, Tmaas)
(17)
Due to bootstrapping, this minimization become challenging, even for fixed expert policy, as it does not fix the scale of the Q-function unless the trajectory reaches an absorbing state. This problem arises particularly a fixed horizon. The lack of a fixed scale increases the variance of the algorithm, affecting the performance negatively.
expert data for cyclic tasks, where we generate trajectories up to
Therefore, we propose modified objective, analyzing Equation 17 achieved when ( a) = Tmax for all reachable (s, a) under drg- Thus, the objective of this term is to push the reward, on expert trajectories, towards Tmax If we consider this minimum, each transition in the expert's 's trajectory has the following Q-value:
The minimum of this term
TT as
Σ 7 mix 1=0
Tmax
Qmas:
with 8, deg (s.
(18)
As the objective of our maximization on expert distribution is equivalent to pushing the value of the expert's 's states and actions towards Qmax, we propose to replace the bootstrapping target with the fixed target Qmax resulting in the following new objective:
Casi =a)= [(Q(s.a)-( Qman (1-)) ((((s,a) ((riiaa)-miiittttt (1m (s'))))
(19)
Note that skip the terminal state treatment for clarity. The full objective is shown in Appendix A.4 Also, we omit the entropy term as we incorporate the latter now in (s a). This new objective incorporates bias toward expert data. Therefore, it not strictly equivalent to the original problem formulation. However, it updates the Q-function toward the same ideal target, while providing simpler and more stable optimization landscape. Empirically, we experienced that this modification, while only justified intuitively, has very positive impact on the algorithm's performance.

---

Published as a conference paper at ICLR 2023
3.6 LEARNING FROM OBSERVATIONS
In many real-world tasks, we do not have access to expert actions, but only to observations of expert's behavior (Torabi et al. 2019b). In this scenario, AlL methods, such as GAIfO (Torabi et al. 201 2019a), can be easily adapted by learning discriminator only depending on the current and the next state. Unfortunately. it is not straightforward to apply the same method to implicit rewards algorithms that learn Q-function. The IQ-Learn method (Garg et al. ) relies on simplification of the original objective to perform updates not using expert actions but rather actions sampled from the policy on expert states. However, this reformulation is not able to achieve good performance on standard benchmarks as shown in our experimental results.
A common practice used in the literature is to train an IDM. This approach has been previously used in behavioral cloning (Torabi et al.] 2018 Nair et al. (2017) and for reinforcement learning from demonstrations (Guo al. [019]] Pavse et al. 2020 Radosavovic et al. 2021). Following the same idea, we generate an observation-on! version of our method by training an IDM online on policy
data and using it for the prediction of unobserved actions of the expert. We modify the objective in Equation 19 to
Lisio (Qsig(())=( (Q)= ((((,TT((),'))))amaal + tame ((Q(s.)) 4-E. (V'))) (b))
(20)
with the dynamics model Ta (s, (').. its parameters w and â = (1 a). We omit the notation for absorbing states and refer to Appendix A A.nsstetdd. Notice that the IDM is only used to evaluate the expert actions, and is trained by solving the following optimization problem
min (r(w) min Es, PP (s,"= -11]] -
(21)
where the expectation is performed on the state distribution generated by the learner policy x. While the mismatch between the training distribution and the evaluation distribution could potentially cause problems, our empirical evaluation shows that on the benchmarks we achieve performance similar to the action-aware algorithm. We give more details on this approach in Appendix B
3.7 PRACTICAL ALGORITHM
Algorithm 1 LS--QQ
We now instantiate practical version of our algorithm in this section. An overview of our method is shown in Algorithm 1 In practice, we use parametric functions to approximate Q, π, G and r, and optimize the latter us- ing gradient ascent on surrogate objective functions that approximate the expectations under dx and d-g using the datasets Dx and Dzg. Further, we use target networks, as already suggested by the Garg et al. (2021). How- ever, while the objective in Equation 4 lacked intuition about the usage of target networks, the objective in Equa- tion I is equivalent to a reinforcement learning objec- tive, in which target networks are a well-known tool for stabilization. Further, we exploit our access to the hard Q-function as well as our fixed reward target setting to calculate the maximum and minimum Q-values possible, Qmin Emin and Qmax Fius, and clip I-
Initialize: Qo, To. 95 and optionally Io for step in {l.....N} do Sample mini-batches D,, and Drp (opt.) Predict actions for D=g using Fs
Deg f (fs, ((s,."..sss)) ((ss,""")'"))) ] [V E 2
Update the Q-function using Eq. 20
Oc+1 (+415-0++~q) V V (0,D, Dxg))
(opt.) Update G-function using Eq. (&
Kig V. G
Update Policy ma using the KL
Φi+1 <- -be Pa-Ks, Krz 1 (opt.) Update l'w using Eq. 21
W++1f-w - Kr V. (w,,
end for
the output of target network to that range. Note that this also holds for the absorbing states. In doing so, we ensure that the target Q always remains in the desired range, which was often not the case with IQ- Target clipping prevents the explosion of the Q-values that can occur due to the use of neural approximators. This technique allows the algorithm to recover from poor value function estimates and prevents the Q-function from leaving the set of admissible functions. Finally, found that training the policy on a small fixed expert dataset anneals the entropy bonus of expert trajectories, even if the policy never visits these states and actions. To address this problem, we clip the entropy bonus on expert states to a running average of the maximum entropy on policy states.
In continuous action spaces, Z, is intractable, which is why policy using Equation 2 As done in previous work (Haarnoja et al. 2018 Garg et al.) 2021). we use parametric policy अे to approximate gÕ by minimizing the KL DKI. (T TT) In our implementation, we found it unnecessary use a double-critic update. This choice reduces the
not directly extract the optimal
7

---

Published as a conference paper at ICLR 2023
Atlas
1.0
0.8
0.6
0.4
Ant-v3
Walker2d-v3
H3pper-v3
Humanoid-v2
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.00
0.00
LSIQ
LSIQ-H
LSIQ-HC
Expert
Figure 2: Comparison of different versions of LS-IQ. Abscissa shows the normalized discounted cumulative reward. Ordinate shows number training steps (x10*). The row shows the results and an exemplary trajectory - here the trained LS-IQ agent on locomotion task using an Atlas robot. The second row shows 4 MuloCo Gym tasks, for which the expert's cumulative rewards are - Hopper:3299.81. Walker2d:5841.72 Ant:6399.04 0449,, H2335oid:6233.45
computational and memory requirements of the algorithm, making comparable to SAC. Finally, replace (s) with v*()s the policy expectation, do not have access to the latter in continuous action spaces.
4 EXPERIMENTS
We evaluate our method on six MuJoCo environments: Ant-v3, Walker2d-v3, Hopper-v3, HalfCheetah-v3, Humanoid-v3, and Atlas. The latter is novel locomotion environment introduced by us and is further described in Appendix C.. We select the following baselines: GAIL (Ho & Ermon 2016). VAIL (Peng al.) 2019), IQ-Learn (Garg al., (02I] and SQIL (Reddy et al. 2020). For a fair comparison, all methods are implemented in the same framework, Mush- roomRL. ED'Eramo al., 202T). We verify that our implementations achieve comparable results to the original implementations by the authors. We use the hyperparameters proposed by the origi- nal authors for the respective environments and perform a grid search on novel environments. The original implementation of IQ-Leamn evaluates two different algorithm variants depending uses telescoping (Garg et al. (02T))
the
given environment. We refer to these variants as 1Qv0-whiee to evaluate the agent's expected return in Equation 4 and IQ IQ-which directly uses Equation and evaluate both variants on all environments. For our method, we use the same hyperparameters IQ-Learm, except for the regularizer coefficient and the entropy coefficient which each environment. We only consider equal mixing, i.e., a = 0.5.
In first experiment, we perform ablations the different design choices of LSIQ. We evaluate the following variants: LSIQ-HC uses a (combined) entropy critic and regularization critic, LSIQ-H only uses the entropy critic, and LSIQ does use ten seeds and five expert trajectories for these experiments. For the Atlas environment, we use 100 trajectories. We also consider IQ, IQvO, and SQIL as baselines and show the learning curves for four environments in Figure 2 The learning curves on the HalfCheetah environment can be found in Appendix [C.] It is interesting to note that IQ-Learn without telescoping does not perform well on Atlas, Walker, and Hopper, where absorbing states are more likely compared to Ant and HalfCheetah, which almost always terminate after fixed amount of steps. We hypothesize that the worse performance Walker and Hopper is caused by reward bias, as absorbing states not sufficiently penalized. IQvO would suffer less from this problem as it treats all states visited by the agent initial states, which results in stronger reward penalties for these states. We conduct further ablation studies showing the influence of the proposed techniques, including an ablation study on the effect of fixed targets, clipping the target Q-value, entropy clipping for the expert, well as
any additional critie, similar IQ-Learn. We
8

---

Published as a conference paper at ICLR 2023
Ant-v3
Walker2d-v3
Hopper-v3
Humanoid-v3
States and Actions
1.00
1.00
1.00
1.00
0.75
0.75
0.75
0.75
0.50
0.50
0.50
0.50
0.25
0.25
10
25
10
25
10
Ant-v3
Walker2d-v3
Hopper-v3
Humanoid-v3
Sates only
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.25
0.00
0.00
0.00
0.00
10
Gall
Vail
LSIQ-H
LSIQ-HC
Expert
Figure 3: Ablation study the effect of the number of expert trajectories different Mujoco environments.
Abscissa shows the normalized cumulative reward. Ordinate shows the number expert trajectories. The first row shows the performance when considering states and action, while the second row considers performance when using states only. Expert cumulative rewards identical to Figure 2
the treatment of absorbing states in Appendix C Our results show that the additional critics have little effect, while fixing the targets significantly increases the performance.
For our main experiments, we only evaluate LSIQ and LSIQ-H, which achieve the best performance in most environments. We compare method to all baselines for four different numbers of expert demonstrations, 1, 5, 10, and 25, and always use five seeds. We perform each experiment with and without expert action. When actions not available, we state transition discriminator (Torabi ett al. 2019a) for GAIL and VAIL, and IDMs for LSIQ (c.f., Section (.6). In contrast, IQ-Learn uses actions predicted expert states by the current policy when expert actions are available. In the dearning-from-onn -bom-observvttion setting, we do not evaluate SQIL, and we omit the plots for IQ, which does not converge in any environment and focus only on IQvO. Figure 3 shows the final expected return over different numbers of demonstrations for four of the environments. All learn- ing curves, including the HalfCheetah environment, can found in Appendix [C.6 for state-action setting and in Appendix C.5 for the learning fioiiinggrooooobevattiiiooo setting. Our experiments show that LSIQ achieves on-par or better performance compared to all baselines. In particular, in the learning- from-observation setting, LSIQ performs very well by achieving similar return compared to the setting where states and actions are observed.
CONCLUSION
Inspired by the practical implementation IQ-Learn, we derive a distribution matching algorithm using implicit reward function and squared L2 penalty the mixture distribution of the expert and the policy. We show that this regularizer minimizes a bounded x²-divergence to the mixture distribution and results in modified updates for the Q-function and policy. Our analysis reveals an interesting connection to SQII-which is not derived from an adversarial distribution matching objective-and and shows that IQ-Learn suffers from reward bias. We build on our insights to propose novel method, LS-IQ, which uses modified inverse Bellman operator to address reward bias, target clipping, fixed reward targets for policy samples, and fixed Q-function targets for expert samples. We also show that the policy optimization of IQ-Learn is not consistent with regularization on the mixture distribution and show how this can be addressed by learning an additional regularization critic. In our experiments, LS-IQ outperforms strong baseline methods, particularly when learning from observations, where we train an IDM for predicting expert actions. In future work, we will quantify the bias introduced by the fixed Q-function target and investigate why this heuristic is fundamental for stabilizing learning. We will also analyze the eттоr propagation in the Q-function target and derive theoretical guarantees on the Q-function approximation error.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
Calculations for this research were conducted on the Lichtenberg perehppeffrmaccc computer of the TU Darmstadt. This work was supported by the German Science Foundation (DFG) under grant number SE1042/41-1. Research presented in this paper has been partially supported by the German Federal Ministry of Education and Research (BMBF) within the subproject "Modeling and exploration of the operational area, design of the Al assistance as well as legal aspects of the use of technology" of the collaborative KIARA project (grant no. 13N16274).
REFERENCES
Alexander Amini, Wilko Schwarting, Ava Soleimany, and Daniela Rus. Deep evidential regression. In Proceeding of the Thirty-fourth Conference Neural Information Processing Systems, Virtual, December 2020.
Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to ad- versarial methods. arXiv, August 2020.
Carlo D' D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research, 22:1-5, 2021.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. Proceeding of the International Conference on Leurning Representations, Vancouver, Canada, April 2018.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Proceeding of the Thirty-fifth Conference Neural Information Processing Systems, Sydney, Australia, December 2021.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning methods. Proceeding of the Conference on Robot Learning, Osaka, Japan, November 2019.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Proceeding of the Twenty-eighth Conference Neural Information Processing Systems, Montreal, Canada, De- cember 2014.
Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid reinforce- ment learning with expert state sequences. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3739-3746, 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceeding of the International Conference on Machine Learning, Stockholm, Sweden, July 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In NIPS 2016
ICLR 2019. Proceeding of the International Conference on Learning Representations, New Orleans, United States, May 2019.
ICLR 2020. Proceeding of the International Conference on Learning Representations, Virtual, May 2020.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator--cttorrrrttt Addressing sample inefficiency and reward bias in adversarial imitation learning. In ICLR 2019
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off- ool-colic distrbution matching. In ICLR 2020
10

---

Published as a conference paper at ICLR 2023
Xudong Mao, Qing Li Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proceeding of the IEEE International Confer- ence on Computer Vision (ICCV), Venice, Italy, October 2017.
Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. In Proceeding of the Thirty-third Conference Neural Information Processing Systems, Vancouver, Canada, December 2019.
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2153. IEEE, 2017.
Andrew Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceeding of the International Conference Machine Learning, Bled, Slovenia, June 1999.
NIPS 2016. Proceeding of the Thirtieth Conference on Neural Information Processing Systems, Barcelona, Spain, December 2016.
Sebastian Nowozin, Botonf Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. NIPS 2016
Brahma Pavse, Faraz Torabi, Josiah Hanna, Garrett Warnell, and Peter Stone. Ridm: Reinforced
inverse dynamics modeling for learning from a single observed demonstration. IEEE Robotics and Automation Letters, 5(4):6262-6269, 2020.
Xue Bin Peng, Angjoo Kanazawa, Sam Toyer, Pieter Abbeel, and Sergey Levine. Variational dis- criminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining infor- mation flow. In ICLR 2019
Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics, 40:1 20, 2021.
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 7865-7871. IEEE, 2021.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In ICLR 2020
Harshit Sikchi, Akanksha Saran, Wonjoon Goo, and Scott Nickum. A ranking game for imitation learning. Transactions on Machine Learning Research, 2023.
Gokul Swamy, Sanjiban Choudhury, Andrew Bagnell, and Zhiwei Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In Proceeding of the Inter- national Conference on Machine Learning, Virtual, July 2021.
Faraz. Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceeding of the Seeenth International Joint Conference on Artificial Intelligence (IJCAI), pp. 4950- 4957, Stockholm, Sweden, July 2018.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. In Proceeding of the International Conference on Machine Learning, Long Beach, California, July 2019a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observa- tion. In Proceeding of the Twenty-Eight International Joint Conference on Artificial Intelligence (ICAI), Macao, China, August 2019
John Von Neumann. Zur theorie der gesellschaftssielle. Mathematische annalen, 100(1):295-33) 1928. Translated English by Sonya Bargman (Von Neumann 1959).
11

---

Published as a conference paper at ICLR 2023
John Von Neumann. On the theory of games of strategy. Contributions to the Theory of Games, 4: 13 42, 1959.
Brian D. Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, University of Washington, 2010.
Brian D. Ziebart, Andrew Maas, J.Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceeding of the Twenty-Third AAAI Conference on Artificial Intelli- gence, Chicago, United States, July 2008
12

---

Published as a conference paper at ICLR 2023
A PROOFS AND DERIVATIONS
In this section, we present proofs the propositions in the main paper. Furthermore, we provide two additional propositions on the optimal reward function and the bounds for the x² -ivdieegee when considering the mixture distribution.
A.1 FROM OCCUPANCY MEASURES DISTRIBUTIONS
Based on Proposition A.1 the solution arg qa Tp(Q. mq) under occupancy measures equals
the solution arg max I JQ. (T)) under state-action distributions. This result allows us use the following distribution matching problem from now on:
max
T(Q, mb) QEn
max
Bd.g [r a)] Edss [rq(s, -
(22)
QEn
where we introduce the implicit reward rg(s,a) a) -,( - [V*(s') for comprehen- sion, ,8 is constant controlling the entropy regularization, dxg is the state-action distribution of the expert, and dz is the state-action distribution under the policy.
-
Proposition A.1 Let maXrER mingen Lp(T,r) be the dual problem of regularized occupancy matching optimization problem and L(T, r) be the Lagrangian of the regularized distribution match- ing problem. Then it holds that LT, r) x L(Trr) and I,(Q.=o) & J(Q, =Q).
Proof of Proposition A.I.
Starting from the definition of the occupancy measure of an arbitrary policy 9, we compute the normalizing constant as an integral:
SxxA Pa (s, a)dsda
lim T-000
Σ
p (s)a(a)s) dsdo
SXA
1=
T-1
lim
(s)r(aas) dada dada
T-000
1=0
5xA
lim
T-000
(23)
Now compute the (discounted) state-action distribution
Px (8,
Pr(s, (1 (s,p(s,,))
(s,
(24)
ISKA p(s,a)dsdd
Thus, have:
pa p=(x, a)
(s.a)
(25)
Using equation 25 in the definition of the objective obtain:
1
[(x)
da (l(s,
Ea, Ir(s, a)]
(26)
1-7 AxA
1-7
A derivation similar equation 26 can be done for the entropy and the regularizer using equa-
tion 25 Substituting the derived formulas into equation and collecting the constant T- proves
the proposition.
A.2 THE BOUNDS OF THE X DIVERGENCE ON MIXTURE DISTRIBUTIONS
Proposition A.2 Given the Pearson diveereee between the distribution drs and the mixture distribution 0.5 dag + 0.5 dz in its variational form shown in Equation 6 the optimal reward is given by
(s,aa) = 1 d=g( da(a.a)
(27)
cd (s, + as
and is bounded such that r" (s, [-1/c, for all E S, A.
13

---

Published as a conference paper at ICLR 2023
Proof of Proposition A.2
This proof follows the proof of Proposition in Goodfellow et al. (2014).
We recall that the divergence on a mixture in Equation
2x" 2 [drg]
du til
max EEdy (r(s,a))-Bea Edha [r(s.)) ((((s,))r((sa))))
max
Edd, v [r(saa]] ((ssa)]-Bu,, Ea, [r(,,a)] kEd, (r(s,a))) kBn, [r(x,a)]]
neR
max [ de (5, kr(s,)) du (8.a) (r(*.a) kr(s, dads, (28)
X diverreene by setting k ./2. For any b R+ ((0), the function y + a(y / v²) bly f achieves its maximum la-b at catb
where k = 1/4 for the conventional x² divergeence. We generalize the
which belongs to the interval [-1/s, 1/c]-
To conclude the proof, we notice that the reward function can be arbitrarily defined outside of Supp(d,) U Supp(drx)) as it has effect the divergence.
Proposition A.3 The Pearson x² divergeenc between the distribution day and the mixture distri- bution 0.5 d + 0.5 d is bounded follows
0 v 2x (dTm
deg +
(29)
Anri
To increase the readability, we drop the explicit dependencies on state and action in the notation, and we write drg and dz for dig (8, (8,a) and (8(8,), a), respectively. The lower bound is trivially true for any divergence and is reached when d, = dmix = dxe To prove the upper bound, use the optimal reward function from Equation 27 and plug it into Equation 28 with k c/2
Proof of Proposition A.3
2x [dre
dry tul =
S
dre
de (r (s,a) +5r (s.a)²) (rr'"s,a)"
dads
2 daix
1 drn
1
d7E d +
2
dTe -r
dag dog +d
d7r dr Un
-dz
d7R dr dog + td + dr
dx {
t 1
dads
20 -d
dh
dx
da
TE
20 g dm
26
2d de 2dh dn +cl
dx
dads
2c(d, x + dz)²
+ d dr d3
dr dr
20 //
(drx da)² dads dag +dx
dods=
20(dag +2
1
de daptde
d7
2 da dads
2c J
dap de dry
doR +
dr
200
1
Bdng dag t
+Edn deg tda drg + dn
Eg
--
(30)
2c
20
Note that the bound tight, as the individual bounds of each expectation are only achieved in conjunction.
Proposition A.4 Let x2 (dRn || adxE + (1 a)dx) be the Pearson x² diveergge between the distribution dur and the mixture distribution ad + (1 - a.d,. Then it holds that:
x2 (drs
||
(ddg + (1-a)dd, I (1 a)x² a)x²(dn dz).
Proof of Proposition
A.4 The proof follows straightforwarddy from the joint convexity of f-
divergences:
(n + P2 xQ1 x) (DD 11 Q1) + (P= Q2),
14

---

Published as a conference paper at ICLR 2023
where [0, is the mixing coefficient. The x -ivdieeree is f-divergence with f(t) (t - 1)P. Now using the x² diverrgeene, and let P = P1 P2 = 22 D = Q1. and a = (1 - R):
{ (1) - (P D) + oxx P) x2 a aP a) x2 (P D) Setting P= d= and D d, concludes the proof.
A.3 FROM x2 REGULARIZED MAXENT-IRL LEAST-SQUARES REWARD REGRESSION
We recall that the entropy-reguulrized IRL under occupancy measures is given by
=p(r,)) (m) E,, [r(s + Ep. [[s - -,(().
(31)
We also recall that using soft inverse Bellman operator to do the change of variable yields:
Jn(Q. = Epgg (((s.a) -B. IV (8 (H
(32)
Bp [(ss,)) a) - -7E, (V (s') (r).
We can now use Proposition A.I to switch the objective from occupancy measures to distributions.
Proof of Proposition 3.7 Staring from the objective function in Equation 22
by expanding the
expectation and rearranging the terms, we obtain:
JQQ m =Bdng [ra(s,a)] a)] Bd, [r)(s,a)] cI3 -
_Rark [ro a)] - Rad, [r](s,a)
ca Ba Ban [ra c(1 a) Ra, -
= Edig [caro(s. a)² [)) - Ed, [d11 - a) + +ro(s,a)] (TH(TT)
=-c Ca [ra(s, a)² dr))(s,a)
a1 Bal, rr(ss,)² + ro + H(Ta))
Defining Fmax
and Mmin a)e and completing the squares we obtain:
I(Q,5g) = (a BdrR [ra(s,a)* de " a)] + (1 a) Es- [))(s, + a-aacra(s,a)
+ é H(())) + ac (rimx - rmm) + (1 (r rinn)
c aa [r)(s,0)²= re a) + r - a)) Est (rg(s,a²²
+( + +rin] + (((TTT)) + ac amr +(1 (1- (1-a)erim. 2
a E. [(rg(s,)) ) - En (rrossca) ZITaaz
+(H(TT)) + ac + (1 - a)crm
Finally, we obtain the following result:
J(Q, (g) -c -a (r)(() rmma) )²] + (1 a)) Es.
Es, [(r(ss,)) rmmn)" + + ? + K,
(33)
where K = ac 2 + (1 2 min
Jec + 4[1-aje
is a fixed constant.
Comparing Equation 9 with Equation 33 results in
J IQQ. + L(Ô, mQ)
(34)
Given that affine transformation (with positive multiplicative constants) preserves the optimum, arg max [QQ mT)) is the solution of the entropy-regulare least squares objective L(Q, TT).
15

---

Published as a conference paper at ICLR 2023
A.4 FULL LS-IQ OBJECTIVE WITH TERMINAL STATES HANDLING
Inserting our inverse Bellman operator derived in Section 3.4 into the least- east-squares objective defined in Equation 9 and rearranging the terms yields the following objective for hard Q-functions:
[L -1-v) v) (Q(s,a) a) - [Fmax + >V
[(Q, TQ) = =aE aaandugg
(35)
d'a P((sa))
(Q(s, -rrmx + 7 Fane
+ aE s,andoR s'mP(.lau)
+ a)B a,a-de
(1-1) (Q(3, (rmin + r (s(1))))
d'aP(las))
y (Q(s,a) -(rmin + ( )
+ L (TQ)
+ (1 - a)B sandte
d'uP(lasa))
= TQ)
sande
(1 v1- (Q(s, - (Fimx + +
(36)
v (Q(x, - [mx )
+ aE s.andag
+ (1 - a) aander
(1- (Q(s,a) (rmin + +V
s'ml(a)s)))
V (Q((.,) rmm)) )1 B ((()))
+ (1 a) a,add P(.laa))))
Now including the fixed target for the expert distribution introduced in Section 3.5 yields:
Lisi (Q, =Q)= = s.a-de paP(a)))))
(1 - (Q(s:a) imm)
(37)
(Q(s,a) I-7
a sanddr
d'm((alaa))
atB sandan
(1-v) (Q(s,a) (Tmin + +
d'MP(1a)))
+ (1 - R saandin
v (Q((,a))
Iuie I-
+ H(TQ)
3
[(((s))) - Iime
)
(Q, πQ) = al sardyy
(38)
p((a(a))
(1 a)B s.anda
[1 (1-v) (Q(s,a) (Mmin + y V> (8')))²
dNP(()))))
(Q(s.a) Aui ) (TQ)):
+ (1 a sandda dmP((1m))
where Equation 38 is the full LS-IQ objective for our hard Q-function. For the observations-only setting we predict the expert's 's actions using the IDM.
A.5 CONVERGENCE OF OUR FORWARD BACKUP OPERATOR
As described in Section 3.4 inverse operator,
(T ()) Q(s, - (Br-)()).)) (s))
(39)
is based on the standard Bellman backup operator, except that, instead bootstrapping, we use the known values for transitions into absorbing state. We will now show that repeatedly applying the corresponding forward Operator
(Bin (s, a) = r(s, a) + -Eg.e(()s.)) ((1 - (s'+
(40)
converges to the Q function. Our proof is based on the same technique that is commonly used prove convergence of the standard Bellmen operator, namely by showing that the Q function
16

---

Published as a conference paper at ICLR 2023
a') is fixed point of our operator, that is, (B bii (s, = Q* as) and by further showing that our operator is contraction,
@T(s,a) = r(s. a) + baa (s
(=) (n)
(s, Q=(s.a)]
(41)
were |l-lloe is the maximum norm; here, we assume finite states and actions for the sake of sim- plicity.
Proposition A.5 The function of policy is a fixed point of Bugq
(s, Q" (s)
(42)
Proof of Proposition A.5 operator performs the same update as the standard Bellman operator, if applied to the actual Q function of the policy, Q* (s, a), since then V(sA) r(SA) Q" Thus,
The proof follows straightforwardly from the fact that our forward
(Biig (s) r(s, a) + P ((1 v) (6') (V
(43)
( r(s, a) + 7R P(lea)) ((1 By Q (s', + ((u
(44)
r(s, a) + + Ba Q* = (s,
(45)
Proposition A.6 The forward operator Blg is contraction.
QA)(s, (1))))) S <n](Qa(s,a) -
(46)
Proof of Proposition A.6
(a)g (8),
(47)
max
Bat [(1 (1=))(Qu(s',d -
(48)
Sn max
E, Eu (a [(Qa(s',a') Qa((s,'')))
(49)
symax
-a(s),a)-Qn((s,)) 0',a'
(50)
=>/1ex(0,0)) -QE(s,))
(51)
17

---

Published as a conference paper at ICLR 2023
Run Policy
Append Replay Buffer
Update IDM
mio Bar,p (FF(),7)
x
Update Policy
Q
Update Q-function
Infer Actions
((s.)))) Pir
Expert Buffer
Figure 4: Training procedure of the IDM in I.S-QQ.
B LEARNING FROM OBSERVATIONS
This section describes the IDM in greater detail. Figure illustrates the training procedure of the IDM in LS-IQ. As can be seen, the IDM uses the replay buffer data generated by agent to infer the actions from state transitions. Therefore, the simple regression loss from Equation 21 is used. At the beginning of training, the IDM learns on transitions generated by (random) agent. Once the agent gets closer to the expert distribution, the IDM is trained on transitions closer to the ex- pert. The intuition behind the usage of an IDM arises from the idea that, while two agents might produce different trajectories and, consequently, state-action distributions, the underlying dynamics are shared. This allows the IDM infer more information about an action corresponding to a state transition than random action predicted by the policy includes, done by Garg et al. (202T). The experiments in Section 4 show that using an IDM yields superior or on-par performance w.r.t. the baselines in the state-only scenario.
While we only present a deterministic IDM, we also experimented with stochastic ones. For in- stance, modeled the IDM as Gaussian distribution and trained it using maximum likelihood loss. We also tried a fully Bayesian approach to impose prior, where we learned the parameters of a Normal-Inverse Gamma distribution and used Student's distribution for predicting actions of state transitions, as done by Ammin et al. (020)) However, stochastic approaches did not show any notable benefit, therefore we stick the simple deterministic approach.
C EXPERIMENTS
This section contains environment descriptions and additional results that have been omitted in the main paper due to space constraints.
C.I THE ATLAS LOCOMOTION ENVIRONMENT
The Atlas locomotion environment is novel locomotion environment introduced by us. This vironment aims to train agents on more realistic tasks, in contrast to the Mujoco Gym tasks, which generally have fine-tuned dynamies explicitly targeted towards reinforcement learning agents. The Atlas environment fixes the arms by default, resulting in 10 active joints. Each joint is torque- controlled by one motor. The state space includes all joint positions and velocities as well as 3D forces on the front and back foot, yielding state-diment titeeddimiiiinaltty of D. 20 + 2- = 32. The action space includes the desired torques for each joint motor, yielding an action dimensionality of D, = 10. Optionally, the upper body with the arms can be activated, extending the number of joints RR.'s and actuators 27. The Atlas environment is implemented using Mushroom RL's (D'Eramo al.) 2021 Mujoco interface.
For the sake completeness, we added the cumulative reward plots - in contrast to the discounted cumulative reward plots as in Figure with an additional VAIL agent in Figure The reward used as metric for the performance is defined as r = exp(-(u, Un )2), where Ux is the agent's s upper body velocity and Uxy is the expert's upper body velocity. The expert's 's walking velocity is 1.25 The code for the environment as well as the expert data is available at /htpp:: ://gthhubbt com/
robf i ras/ 1s-ig
18

---

Published as a conference paper at ICLR 2023
Atlas
1.00
0.75
0.50
0.25
0.00
500
1000
1500
LSIQ
LSIQ-H
LSIQ-HC
Vail
IQ
1Qv0
SQIL
Expert
Figure 5: Training results and an exemplary trajectory here the trained ISI agent of a locomotion task using simulated Atlas robot. Abscissa shows the normalized cumulative reward. Ordinate shows the number of training steps (x (x10%).
C.2 ABLATION STUDY: ABSORBING STATE TREATMENT AND TARGET Q CLIPPING
Figure 6 presents ablations on the effects of the proposed treatment of absorbing states and the clipping of the Q-function target on an LSIQ agent with bootstrapping. To see the pure effect of the absorbing state treatment and the clipping, we did not include fixed targets. Note that the fixed target implicitly treats absorbing states of the expert, it provides the same target for states and actions transitioning towards absorbing states. We have chosen LSIQ agent without an entropy and regularization critic. The experiments are conducted on the Humanoid-v3 task, the tasks HalfCheetah-v3. Ant-v3, either do not have or have very rare absorbing states, and Walker-v3 and easy
the effects. We parameter of a = 0.5 yielding a maximum reward of Timx = 2(1-0.5)0.5 = and a minimum This yields a maximum Q-value of Qmax = T'me I- 200 and Fime - 200. The rows show the different agent configurations: First, LSIQ with clipping and absorbing state treatment; second. LSIQ with clipping but no absorbing state treatment; and lastly, LSIQ without clipping and no treatment of absorbing states which equivalent to SQIL with symmetric reward targets. For a better understanding of the effect, the plots show the individual seeds of a configuration. learn the task and regresses the Q-value of the transitions towards absorbing states to the minimum Q-value of -000. The second configuration does not treat absorbing states and is not able to learn the task with all seeds. As can be seen, the average Q-value of absorbing states is between -2 and -6. Taking a closer look at the first two plots in the second row, one can see that those seeds did not learn, whose average Q Qavuue on non-absorbing transitions is close even below the average Q-values states and actions yielding to absorbing states. This strengthens the importance of our terminal state treatment, which pulls Q-values of states and action towards absorbing states to the lowest possible Q-value and, therefore, avoids a termination bias. Finally, one can see the problem of exploding Q-value in the last LSIQ configuration. This is evident by the scale of the abscissa, highlighted in the plots. Interestingly, while some seeds still perform reasonably well despite the enormously high Q-value, it clearly correlates to the high variance in the cumulative reward plot.
Hopper-v3
regularizer constant of 0.5 and mixing
reward of I'in = 21-0.550.3
minimum Q-value of Qmin - I-y
can be seen, the first LSIQ agent can successfully
C.3 ABLATION STUDY: INFLUENCE OF FIXED TARGETS AND ENTROPY CLIPPING
To show the effect of the fixed target (c.d., Section 3.5 and the entropy clipping (c.f., 3.7 conducted a range of ablation studies for different versions of LSIQ on all Mujoco task. The results shown in Figure / for the LSIQ version only with an entropy critic and in Figure for the LSIQ version with an entropy and a regularization critic. As can be seen from the Figures, the version with the fixed target and the entropy clipping performs best. It is especially noteworthy that the entropy clipping becomes of particular importance on tasks that require a high temperature parameter which the case for the Humanoid-v3 environment.
19

---

Published as a conference paper at ICLR 2023
O-Value Absorbing Policy
Q-Value Policy
Cumulative Reward
Clipping & Abs. Treatment
180
-190
4000
-200
-100
2000
-210
-200
200
400
600
800
200
400
600
800
No Abs. Treatment
Q-Value Absorbing Policy
Q-Value Pollcy
Cumulative Reward
100
6000
Clipping &
50
-
2000
200
400
600
800
200
400
600
800
200
400
600
800
Clipping & No Abs. Treatment
1.0
Q-Value Absorbing Policy
2.0
160
Q-Value Policy
6000
Cumulative Reward
0.5
0.0
4000
-0.5
-1.0
0.5
2000
-1.5
-2.0
200 400 600
800
0.0
200
400
600
800
200 400
600
800
Figure 6: Ablation study on the effect of the proposed treatment of absorbing states and the clipping of the Q-value target
LSIQ agent with bootstrapping (no fixed targets). The experiments conducted on the Humanoid-v3 task, with expert reaching cumulative reward of 6233.45. Multiple lines in each plot show the individual seeds. The first column presents the average Q-value states and actions yiclding to an absorbing state visited by the policy. The second column presents the average Q-value of all states and actions that do not yield in absorbing states visited by the policy. The third column presents the cumulative reward. The rows present the ablations done to the LSIQ agent. Ordinate shows the number of training steps (x 1x")..
Ant-v3
HalfCheetah-v3
Hopper-v3
0.4
0.2
0.2
0.2
0.0
0.0
0.0
500
1000
1500
500
1000
1500
500
1000
1500
Walker2d-v3
Humanoid-v3
1.0
LSIQ-HHEC
SQL
L5IG-H-F-EC
Expert
0.0
0.0
500
1000
1500
0
500
1000
1500
Figure 7: Comparison of different versions of LSIQ-H (Regularization Critic): First, the bootstrapping ver- sion - LSIQ-II; second, the bootstrapping version with entropy clipping -> LSIQ-II+EC; thirdly, the fixed target version with entropy clipping LSIQ-HI+CC+F.. IQ and SQIL are added for reference. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number training steps (x10*). Experi- ments are conducted with 5 expert trajectories and five seeds. Expert cumulative rewards + Hopper:3299.88 Walker2d:5841.7 Ant:6399.00 04, 123tec Humanoid:6233.45
20

---

Published as a conference paper at ICLR 2023
Ant-v3
HalfCheetah-v3
Hopper-v3
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
500
1000
1500
5000
1000
1500
500
1000
1500
Walker2d-v3
Humanoid-v3
1.0
1.0
0.8
0.8
L5IG-H
0.6
0.6
LSIG-ICHEC
5QL
LSRG-CHEC-FF
Expert
0.4
0.4
0.2
0.2
Figure 8: Comparison of different versions LSIQ-HC Catropy-Regnlarizzttoon Critie): First, the boot- strapping version + LSIQ-HC; second, the bootstrapping version with entropy clipping + LSIQ-HC+EC; thirdly, the fixed target version with entropy clipping - LSIQ-HC+EC+FT. IQ and SQIL are added for refer- ence. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x10"). Experiments conducted with 5 expert trajectories and five seeds. Expert cumulative rewards + Hopper:3299.88 Walker2d.5841.73. Ant:6399.04 ,04, MtMCCheethh:13355 Humanoid:6233.45
C.4 ALL EXPERIMENT RESULTS OF THE DIFFERENT VERSION OF LSIQ
Figure presents all the plots presented Figure with the additional Humanoid-v3 results. Fig- 10 and Figure I1 correspond to Figure but show all five environments. The corresponding
learning curves are shown in Appendix C.6 and
Ant-v3
HalfCheetah-v3
Hopper-v3
0.2
0.2
02
0.0
500
1000
1500
0.0
0
500
1000
1500
0.0
500
1000
1500
Walker2d-v3
Humanoid-va
1.0
SQIL
LSIG-
10v0
Expert
LSIO
0.0
500
1000
1500
0.0
0
500
1000
1500
Figure 9: Comparison of different versions of LS-IQ. Now also with the Humanoid-v3 environment. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x (x10*). Expert cumulative rewards - Hopper:3299.8.. Walker2d:5841.73., Ant:6399.04, MalIChheethh1122258 Hu-
manoid:6233.45
21

---

Published as a conference paper at ICLR 2023
3nt-v3
HalfCheetah-v3
Hopper-v3
1.00
1.00
1.00
0.25
0.25
0.25
0.00
0.00
0.00
Walker2d-v3
Humanoid-v3
1.00
1.00
USR0
Vail
LS10 H
Expert
0.25
SQIL
0.00
0.00
effect number expert trajectories on different Mujoco environments. States and actions from the expert are provided to the agent. All plots are added here for the sake of complete- ness. Abscissa shows the normalized cumulative reward. Ordinate shows the number of expert trajectories. The first shows the performance when considering states and action, while the second row considers the performance when using states only. Training results are averaged five seeds per agent. The shaded area constitutes 95% confidence interval. Expert cumulative rewards + Hopper:3299.88 81, Walker2d:5841.77 Ant:6399.04, HalfCheetah:12333 78, H2m33oid:6233.45
Figure 10: Comparison
Ant-v3
HalfCheetah-v3
Hopper-v3
1.00
1.00
0.50
0.50
0.50
0.25
0.25
0.25
0.00
0.00
0.00
Walker2d-v3
Humanoid-v3
1.00
1.00
0.75
0.75
Galf
LSIQ
Velfo
LSI0-H
10v0
Expert
0.00
0.00
5
10
10
25
Figure 11: Comparison the effect the number of expert trajectories on different Mujoco environments. Only expert states are provided to the expert. All plots are added here for the sake of completeness. Abscissa shows the normalized cumulative reward. Ordinate shows the number of expert trajectories. The first row shows the performance when considering when using states only. Training results are averaged over seeds per agent. The shaded area constitutes the 95% contidence interval. Expert cumulative rewards + Hopper:3299.8) 81, Walker2d:5841.77 Ant:6399.44 HalfCheetah: 12328.7 78, 43335
and action, while the second
considers the performance
22

---

Published as a conference paper at ICLR 2023
C.5
IMITATION LEARNING FROM STATES ONLY FULL TRAINING CURVES
The learning curves for the learning from observation experiments can be found in Figure 12 and 15 for 5, 10 and 25 expert demonstrations, respectively.
Ant-v3
HalfCheetah-v3
Hopper-v3
1.00
0.8
0.8
0.50
0.25
0.4
0.4
0.0
0.0
1000
1500
1000
1500
500
1500
Walker2d-v3
Humanoid-v3
1.0
0.8
0.8
Galfo
LSIO
0.6
Vallo
LS10-H
1Qv0
Export
0.4
0.2
0.2
0.0
0.0
1000
1500
500
1000
1500
Figure 12: Training performance different agents on Mujoco Tasks when using expert trajectory consisting of only states. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x10"). Training results averaged over five seeds per agent. The shaded area constitutes the 95% confidence interval. Expert cumulative rewards Hopper:3229.81 Walker2d:5841.73, Ant:6399.04, 12328. HalfChectah: 12328.78. 78, Humanoid:6233.45
Ant-v3
HalfCheetah-v3
Hopper-v3
1.00
1.0
1.0
0.75
0.8
0.8
0.50
0.6
0.6
0.25
0.4
0.4
0.2
0.2
0.0
0.0
500
1000
1000
1500
500
1000
1500
Walker2d-v3
Humanoid-v3
1.0
Calo
Valfo
LSIG-H
10+0
Expert
0.0
500
1000
1500
0.0
0
500
1000
1500
Figure 13: Training performance of different agents on Mujaco Tasks when using 5 expert trajectory consisting of only states. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x10"). Training results averaged seeds per agent. The shaded area constitutes the 95% confidence interval. Expert cumulative rewards Hopper:3299.8g 81, Walker2d:5841.73, Ant:6399.04, HalfCheetah: 12328.78. 78, 623334
23

---

Published as a conference paper at ICLR 2023
Ant-v3
HalfCheetah-v3
Hopper-v3
1.0
1.0
0.75
0.6
0.6
0.25
0.00
0.2
0.2
0
500
1000
1500
500
1000
1500
500
1000
1500
Walker2d-v3
Humanold-v3
1.0
1.0
0.8
Galfo
LSIO
0.6
0.6
Vallo
LS10-1
10v0
Expert
0.4
0.2
0.2
0.0
500
1000
1500
500
1000
1500
Figure 14: Training performance of different agents Mujoco Tasks when using 10 expert trajectory consist- ing of only states. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x10*). Training results are averaged over seeds per agent. shaded area constitutes the 95% confidence interval. Expert cumulative rewards - Hopper:3299.811 Walker2d:5841.77 Ant:6399.04, HalfCheetah: 1 12328.777 78, H2m33oid:6233.45
Ant-v3
MalfCheetah-v3
Hopper-v3
1.00
1.0
1.0
0.75
0.6
0.6
0.25
0.00
0.2
0.2
500
1000
1500
500
1000
1500
500
1000
1500
Walker2d-v3
Humanoid-v3
1.0
1.0
0.8
Galfo
LSIO
0.6
0.6
Vallo
LSIg-H
Export
0.2
0.2
500
1000
1500
500
1000
1500
Figure 15: Training performance of different agents Mujoco Tasks when using 25 expert trajectory consist- ing of only states. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x10*). Training results are averaged over seeds per agent. The shaded area constitutes the 95% confidence interval. Expert cumulative rewards - Hopper:3299.88 Walker2d:5841.77 Ant:6399.04 HalfChectah: 2328.77 78, 4333moid
24

---

Published as a conference paper at ICLR 2023
C.6
IMITATION LEARNING FROM STATES AND ACTIONS FULL TRAINING CURVES
The learning curves for the experiments where states and actions are observed can be found in Figure T6
17 18 and 19 for 5, 10 and 25 expert demonstrations, respectively.
Ant-v3
HalfCheetah-v3
Hopper-v3
1.00
0.8
0.8
0.50
0.25
0.4
0.4
0.0
0.0
1500
1000
1500
500
1500
Walker2d-v3
Humanoid-v3
1.0
1.0
0.8
0.8
Gall
LS10
0.6
Vell
150-1
0.4
Export
SOIL
0.2
0.2
0.0
500
1000
1500
500
1000
1500
Figure 16: Training performance different agents on Mujoco Tasks when using expert trajectory. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x10"). Training results averaged over five seeds per agent. The shaded area constitutes the 95% confidence inter- val. Expert cumulative rewards Hopper:3299.8/ Walker2d:5841.73, Ant:6399.04, HalfCheetah; 12328.78.
Hum3noid:6233.45
Ant-v3
HalfCheetah-v3
Hopper-v3
1.00
1.0
1.0
0.75
0.8
0.6
0.6
0.25
0.4
0.4
0.00
0.2
0.2
500
1000
1500
500
1000
1500
500
1000
1500
Walker2d-v3
Humanold-v3
1.0
0.8
0.6
0.6
LS1GH
10v0
L5/HHC
0.4
Expert
SQL
0.2
0.2
0.0
0.0
500
1000
1500
0
500
1000
1500
Figure 17: Training performance of different agents Mujoco Tasks when using expert trajectories. Abscissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x10"). Training results are averaged over ten seeds per agent. The shaded area constitutes the 95% confidence inter- val. Expert cumulative rewards + Hopper:3299.88 81, Walker2::5841.73 Ant:6399.04 04, HalfCheetah: 12328.78,
Humanoid:6233.45
25

---

Published as a conference paper at ICLR 2023
Ant-v3
HalfCheetah-v3
Hopper-v3
1.00
1.0
1.0
0.75
0.6
0.6
0.25
0.00
0.2
0.2
500
1000
1500
500
1000
1500
500
1000
1500
Wa
Humanoid-v3
1.0
1.0
0.8
0.6
0.6
Vell
15:-+1
10v0
LSIQ-HC
0.4
Expert
0.2
0.0
0.0
500
1000
1500
500
1000
1500
Figure 18: Training performance of different agents Mujoco Tasks when using 10 expert trajectories. Ab- scissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x (x10"). Training results are averaged over five seeds per agent. The shaded area constitutes the 95% con- fidence interval. Expert cumulative rewards - Hopper: Walker2d:5841.73, Ant:6399.04, HalfChee- tah: 12328.78. 78, H23350id:6233.45
Ant-v3
HalfCheetah-v3
Hopper-v3
1.00
1.0
1.0
0.75
0.8
0.6
0.6
0.25
0.00
0.2
0.2
500
1000
1500
1000
1500
500
1000
1500
Walker2d-v3
Humanoid-v3
1.0
1.0
0.8
0.6
0.6
LS1G-
10v0
LSIQ HC
0.4
Expert
0.2
0.0
500
1000
1500
0.0
500
1000
1500
Figure 19: Training performance of different agents Mujoco Tasks when using 25 expert trajectories. Ab- scissa shows the normalized discounted cumulative reward. Ordinate shows the number of training steps (x (x10"). Training results are averaged over five seeds per agent. The shaded area constitutes the 95% con- fidence interval. Expert cumulative rewards - Hopper:3299.811 Walker2d:5841.73, Ant:6399.04 HalfChee- tah:12328. 12328.78. 78, 62m3noid
26


Reviews:
[{"review_id": "UDAGwMDyx6", "reviewer": "Reviewer_GvAu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the problem of imitation learning, building on the recent IQ-learn framework. Instead of an adversarial reward-policy loss like GAIL, IQ-learn instead parameterizes the Q-function so that the policy can be directly extracted. While IQ-learn works fine, the paper notes that some of the practical tricks don't match the analysis, e.g. regularizing both the expert and the imitator (which also should prevent direct extraction of policy in theory).\n\nIn this paper, the authors present Least Squares Inverse Q Learning (LS-IQ). LS-IQ patches some of the gap described above with IQ-learn. \n\n- First, LS-IQ shows that the mixture regularizer is, naturally, regularizing a chi-squared divergence between the expert and the mixture of expert/policy occupancies. In practice this is good because the mixture ensures that the divergence is bounded.\n- Second, they use a least-squares RL perspective to figure out how to properly treat absorbing states.\n- Third, they propose using a regularization critic, to account for the additional regularizer term in the objective.\n- A few more tips and tricks, e.g., replacing bootstrapping target with fixed target, learning from observations (with no action information) by training an IDM.\n", "summary_of_the_review": "In summary the authors improve on IQ-learn, the current SOTA imitation learning algorithm on many tasks.\nThe paper presents a laundry list of tips and tricks, which was hard to read, but shows good practical improvements that should be very valuable to the community.\n\n--------\n\nUpdate: I have read the author response, and am keeping my evaluation.", "strength_and_weaknesses": "Strengths\n- the biggest strength of the paper is that it is very practical: it identifies many tips and tricks for making IQ-learn better. This is a great contribution, given the already strong results of the base IQ-learn algorithm.\n- some of these tricks are somewhat theoretically grounded. Though (as the paper notes itself) some parts are not super convincing, e.g. fixed Q target, it is still useful for future work.\n- the experiments have ablations of using different subsets of the proposed tricks.\n\nWeaknesses\nThe first few pages were find to read, but the rest of the paper (esp section 3) was poorly structured in my opinion. It reads like a laundry list A, B, C, and the reader has no idea what to anticipate next. Some of the insights feel like offhand remarks that have little relevance to the rest of the paper, and are hard to distinguish from the actual important insights. I would try to restructure the paper so that the most important parts are emphasized (even repeated), and move more minor details to the appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Fq2qDKEplTj", "reviewer": "Reviewer_BFcT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focus on addressing the reward bias issue of a SOTA imitation learning method called IQ-Learn that directly uses an implicit reward function. Specifically, this paper proposed an implicit reward regularization approach called Least Square Inverse Q-Learning (LS-IQ), which uses a modified inverse Bellman operator to address the reward bias issue of IQ-Learn. LS-IQ achieves this goal by using a squared L2 penalty on the mixture distribution of expert and policy. Empirically, LS-IQ outperforms some baseline methods on standard continuous control tasks.  ", "summary_of_the_review": "Overall, the proposed method is well motivated by theoretical analysis. I expect the authors to further improve the clarity of the paper and address my concerns above. \n\n\n======After rebuttal===\nThanks the authors for addressing my concerns. I agree to raise my rating to 6. Hope to see more details and open source code for better reproducibility. ", "strength_and_weaknesses": "## Strength\n\n+ This paper noticed that the squared norm regularization on the implicit reward function is effective in imitation learning, but lack of theoretical analysis. Interestingly, this paper uses the regularizer under a mixture distribution of the policy and the expert, and understand the learning procedure in a illuminating perspective. The original objective was understood as a least-squared Bellman error minimization, resulting an minimization of $\\chi^2$-divergence between the expert and the mixture distribution.   \n\n+ Given theoretical analysis of the regularizer, this paper indicates some sources of instabilities of the IQ-Learn approach: the arbitrariness of the Q-function scale, exploding Q-function targets and reward bias. Then, Least Square Inverse Q-Learning was proposed to addressed these issues. \n\n\n## Weakness \n\n+ The proposed method are only evaluated on online imitation learning tasks. To fairly and fullly compare the proposed method to IQ-Learn, I suggest the authors to also evaluate the methods on offline settings as in the IQ-Learn paper.\n\n+ The proposed method, LS-IQ, stabilize training by providing fixed targets for the Q-function on expert states, and properly treat absorbing states. This work also introduces an entropy-regulariation critic. However, there is no ablation to study the true effects of each component. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3jgZsUcRs2", "reviewer": "Reviewer_kpRZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "An online imitation learning algorithm with implicit reward is proposed, where the learning objective is to minimize $\\chi^2$-divergence between the expert distribution and the mixture of expert and policy distributions. The idea of this work is motivated by the practical implementation of IQ-Learn that violates their theoretical derivation from introducing reward regularizer in practice. Authors argue that the proposed learning objective with the mixture-based divergence is more stable since it properly bounds the range of Q (whereas the range is  unbounded when the mixture is not considered). Additional details on absorbing states, a shared critic for entropy and reward regularization, and LfO with Inverse Dynamics Model are considered to improve the performance of the proposed idea. ", "summary_of_the_review": "Although the motivation of this work is interesting, the paper should become clearer to be accepted. With the current version, the authors' intuitions behind are not straightforward. There's also a baseline reproducibility issue.", "strength_and_weaknesses": "Strengths\n- Authors look into the gap between practical algorithm and theoretical derivation of the baseline and try to minimize the gap.\n- Algorithm is shown to applicable to various IL scenarios (online IL, LfO) and seems stable for all MuJoCo tasks.\n- Algorithm used a lot existing techniques (dealing with absorbing states, IDM) to improve the performance. \n\nWeaknesses\n- The readability should be improved.\n- Whether the proposed learning objective is new or not is unclear. I'd like to argue that the objective (12) has a strong relation with SQIL's objective. \n- The baseline performance is much poorer than the performance reported in the paper. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-h16pteDcBX", "reviewer": "Reviewer_XCLn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides a theoretical analysis of Inverse soft Q-Learn (IQ-Learn) and proposes a novel algorithm named Least Squares Inverse Q-learning (LS-IQ), which outperforms state-of-the-art algorithms such as GAIL, VAIL, IQ, and SQIL. At first, the authors show that the maximum entropy IR objective with the regularizer proposed by Garg et al. (2021) is identical to a minimization of a $\\chi^2$ divergence between the expert state-action distribution and a mixture of expert and learner. Then, the authors formulate the IRL objective using the regularizer as the entropy-regularized least-squares problem, which is related to LSGAN. To derive the practical algorithm, the authors modify the loss function for the Q-function to incorporate the term added by the regularizer. ", "summary_of_the_review": "I think this paper studies an important problem and has some interesting theoretical results. Moreover, this paper is well-written, and the contribution is quite impressive. ", "strength_and_weaknesses": "Strength\n1. The paper is well-written, and I found the main contributions of this study to be novel.\n2. The proposed method provides the correct way to deal with absorbing states in inverse reinforcement learning.\n3. Show the relation between the proposed method, IQ-Learn, and SQIL. \n\nWeakness\n1. The proposed method is evaluated on the MuJoCo benchmarks, but they are relatively easy for modern IRL methods. It is unclear whether the proposed method is robust to distribution shifts between the expert and agent distributions. \n2. As described below, some technical details are not shown in the manuscript. In particular, $r_{\\min}(s)$ in Equation (20) should be explained. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster