IQ-Learn: Inverse soft-Q Learning for Imitation
Authors: Divyansh Garg
Shuvam Chakraborty
Chris Cundy
Jiaming Song
Stefano Ermon
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: reinforcement learning
imitation learning
inverse reinforcement learning
statistical learning
energy-based models

Abstract: In many sequential decision-making problems (e.g., robotics control, game playing, sequential prediction), human or expert data is available containing useful information about the task. However, imitation learning (IL) from a small amount of expert data can be challenging in high-dimensional environments with complex dynamics. Behavioral cloning is a simple method that is widely used due to its simplicity of implementation and stable convergence but doesn't utilize any information involving the environment’s dynamics. Many existing methods that exploit dynamics information are difficult to train in practice due to an adversarial optimization process over reward and policy approximators or biased, high variance gradient estimators. We introduce a method for dynamics-aware IL which avoids adversarial training by learning a single Q-function, implicitly representing both reward and policy. On standard benchmarks, the implicitly learned rewards show a high positive correlation with the ground-truth rewards, illustrating our method can also be used for inverse reinforcement learning (IRL). Our method, Inverse soft-Q learning (IQ-Learn) obtains state-of-the-art results in offline and online imitation learning settings, significantly outperforming existing methods both in the number of required environment interactions and scalability in high-dimensional spaces, often by more than 3x.

IQ-Learn: Inverse soft-Q Learning for Imitation
Divyansh Garg Jiaming Song Stefano Ermon
Shuvam Chakraborty Chris Cundy
Stanford University
{divgarg, shuvamc, cundy, taong, ermon]@@taanfrd.comm
Abstract
In many sequential decision-making problems (e.g.. robotics control, game play- ing, sequential prediction), human or expert data is available containing useful information about the task. However, imitation learning (IL) from a small amount of expert data can be challenging high-dimensional environments with com- plex dynamics. Behavioral cloning is a simple method that is widely used due to simplicity of implementation and stable convergence but doesn't utilize any information involving the environment's dynamics. Many existing methods that exploit dynamics information are difficult to train in practice due to an adversarial optimization process over reward and policy approximators biased, high variance which avoids
gradient estimators. We introduce a method for dynamics-aware adversarial training by learning single p-function, implicitly representing both reward and policy. On standard benchmarks, the implicitly learned rewards show a high positive correlation with the ground-trin truth rewards, illustrating also be used for inverse reinforcement learning (IRI). Our method, Inverse soft-Q learning (IQ-Learn) obtains state-of-the-ar results in offline and online imitation learning settings, significantly outperforming existing methods both the number of required environment interactions and scalability in high-dimensional spaces, often by more than 3x.
method
Introduction
Imitation of an expert has long been recognized as a powerful approach for sequential decision- making [29, 1], with applications as diverse as healthcare [39], autonomous driving [41]. and playing complex strategic games [8]. In the imitation learning (IL) setting. we are given a set of expert trajectories, with the goal of learning policy which induces behavior similar to the expert's. The learner has no access to the reward, and no explicit knowledge of the dynamics.
The simple behavioural cloning [34] approach simply maximizes the probability of the expert's actions under the learned policy, approaching the II problem While this can work well in simple environments and with large quantities of data, it ignores the sequential nature of the prcisionnmakingg problem, and small errors can quickly compound when the learned policy departs from the states observed under the expert. A natural way of introducing the environment dynamics is by framing the problem as an Inverse RL (IRL) problem, aiming to learn reward function under which the expert's trajectory is optimal, and from which the learned imitation policy can be trained [1]. This framing has inspired several approaches which use rewards either explicitly or implicitly to incorporate dynamics while learning an imitation policy [17, 10, 33, 22]. However, these dynamics-aware methods are typically hard to put into practice due to unstable learning which can be sensitive to hyperparameter choice minor implementation details [21].
supervised learning problem.
In this work, we introduce dynamics-awar imitation learning method which has stable, non- adversarial training, allowing us to achieve state-of-the-art performance on imitation learning bench-
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table 1:
A comparison of various algorithms for imitation learning. "Convergence Guarantees" refers to a proof is given that the algorithm converges to the correct policy with sufficient data. We consider an algorithm "directly optimized" if it consists of optimization algorithm (such as gradient descent) applied to the parameters of a single function
Non- Adversarial Training
Dynamics Aware
Convergence Guarantees
Non-restrictive Reward
Direct
Method
Reference
Optimization
Max Margin IRI.
[29, 11
Max Entropy IRL
[43]
Online
GAIL/AIRL
[17. 10]
ASAF
SQIL
[33]
Ours (Online)
Max Margin IRL
[24,20]
Max Likelihood IRL
[18]
Max Entropy IRL
[16]
Offline
ValueDICE
[22]
Behavioral Cloning
[34]
Regularized BC
[30]
EDM
[19]
Ours (Oflline)
marks. Our key insight is that much of the difficulty with previous IL methods arises from the IRL-motivated representation of the IL problem min-max problem over reward and policy [17, 1].
This introduces a requirement to separately model the reward and policy, and train these two functions jointly, often in adversarial fashion. Drawing connections between RL and energy -assed models [13, 14], we propose learning a single model for the Q-value. The Q-value then implicitly defines both reward and policy function. This turns difficult min-max problem over policy and reward functions into a simpler minimization problem over a single function, the Q-value. Since our problem has one-to-one correspondence with the min-max problem studied in adversarial IL [17], we maintain the generality and guarantees of these previous approaches, resulting in a meaningful reward that may be used for inverse reinforcement learning. Furthermore, our method may be used to minimize a variety of statistical divergences between the expert and learned policy. We show that we approaches special cases of particular divergences, such as
recover several previously-descric the regularized behavioural cloning of [30], and the conservative Q-learning of [23].
In our experiments, find that method is performant even with very sparse data surpassing prior methods using one expert demonstration the completely offline setting and can scale to complex image-based tasks like Atari reaching expert performance. Moreover, our learnt rewards highly predictive of the original environment rewards.
Concretely, our contributions are as follows:
We present a modified Q-learning update rule for imitation learning that can be implemented top of soft-Q learning soft actor-critic (SAC) algorithms in fewer than 15 lines of code.
We introduce simple framework to minimize wide range of statistical distances: Integral Probability Metries (IPMs) and f-divergences, between the expert and learned distributions.
We empirically show sfate-of-art results in a variety of imitation learning settings: online and offline II. On the complex Atari suite, we outperform prior methods by 3-7x while requiring 3x less environment steps.
We characterize our learnt rewards and show a high positive correlation with the ground-trut truth rewards, justifying the use of our method for Inverse Reinforcement Learning.
2 Background
Preliminaries We consider environments represented as a Markov decision process (MDP), which defined by a tuple (S, A. Po. P,r.7). S, A represent state and action spaces, Po and P(s'|s, ( a) represent the initial state distribution and the dynamics, r(s, a) represents the reward function, and

---

(0, 1) represents the discount factor. RSxAA = {o S x - will denote the set of all functions in the state-action space and R will denote the extended real numbers R J {oc}. Section 3 and 4 will work with finite state and action spaces and A, but our algorithms and experiments later in the paper use continuous environments, IT is the set of all stationary stochastic policies that take actions in A given states in S. We work in the y-discounted infinite horizon setting, and we will use expectation with respect to policy E II to denote an expectation with respect to the trajectory it generates: [x[r((s a]] _ E E t= 7'r(*t; dt)], where 8p ~ po.
IT ((11s)), and A TR. as Pa (s, a) π (als)) Et=o 7'P (s, s|r). We refer to the expert policy as gg and its occupancy measure as Pg. In practice, TE is unknown and we have access to a sampled dataset of demonstrations. For brevity, we refer to Px for learnt policy in the paper.
8141 P( St. at) for 1 0. For policy x E II, we define its occupancy measure Pr
Soft Q-functions For a reward r E RSX and a E II, the soft Bellman operator 8" RSSA + RSXA defined as (B*Q) = r(s,a) + -B.'')(s..) V*(')))) with (*(s) = Ea-x(-la) (Q(s,a) a) log (a]s)). The soft Bellman operator is contractive [13] and defines a unique soft Q-function for r, given as = B
Max Entropy Reinforcement Learning For given reward function r E RSXA, maximum entropy RL [14,5] aims to learn a policy that maximizes the expected cumulative discounted reward along with the entropy in each state: max+en E, (r(s, a)] + H (T). Where H()) - E, log n (a]s)] is the discounted causal entropy of the policy æ. The optimal policy satisfies [42, 5]:
1 Z.
(als)
exp (Q(s.a)).
(1)
where Q is the soft Q-function and Z, is the normalization factor given as En. exp (Q (s,a')).
Q satisfies the soft-Bellman equation:
(s r(s, a) + P log Σ exp(Q(s', a'))
(2)
In continuous action spaces, Z, becomes intractable and soft actor-critic methods like SAC [13] can be used to learn an explicit policy.
Max Entropy Inverse Reinforcement Learning Given demonstrations sampled using the policy TE. maximum entropy Inverse RL aims to recover the reward function in a fam- ily of functions R that rationalizes the expert behavior by solving the optimization problem: max,R min Ex [r(s, a)] - [r(s,a) + (T)), where the expected reward of TE is em- pirically approximated. It looks for a reward function that assigns high reward to the expert policy and a low reward to other policies, while searching for the best policy for the reward function in an inner loop.
The Inverse RL objective can be reformulated in terms of its occupancy measure, and with a convex reward regularizer v R.S.A E [17]
max min L(m, r) Ep - (T) ((r) nell
In general, exchange the max-min resulting in an objective that minimizes the statistical distance parameterized by v. between the expert and the policy [17]
min max L(T, min de (p. PE) (T).
(4)
Hell PER
#ell
with dx e (pE - p), where si* is the convex conjugate of v.
3 Inverse soft Q-learning (IQ-Learn) Framework
A naive solution to the IRL problem in (Eq. 3) involves (1) an outer loop learning rewards and (2) executing RI. an inner loop to find an optimal policy for them. However, we know that this optimal policy can be obtained analytically in terms of soft Q-uuctions (Eq. 1). Interestingly, as will show
3

---

later, the rewards can also be represented in terms of Q (Eq. 2). Together, these observations suggest it might be possible to directly solve the IRL problem by optimizing only over the Q-function.
To motivate the search of an imitation learning algorithm that depends only on the f-function, we characterize the space of Q-functions and policies obtained using Inverse RI.. We will study IT, r E R and Q-functions Q E Ω where R = Ω = IR SxA, We assume 1I is convex, compact and that TE II' We define V (s) (-lx( [Q(s, a) log क (a]s)]
We start with analysis developed in [17]: The regularized IRL objective L(T,, r)) given by Eq. 3, is concave in the policy and convex in rewards. And has a unique saddle point where is optimized.
To characterize the Q- Q-functions it is useful to transform the optimization problem over rewards to a problem over Q-functions. We can get one-to-one correspondence between r' and Q:
Define the inverse soft bellman operator T SxA + R such that
(T" a) Q a) P(ss.)(s.a) V (s'),
Lemma 3.1. The inverse soft bellman operator T is bijective, and for any r, (T*)-1r is the unique contraction of 1"
The proof of this lemma is in Appendix A. For a policy n, we are thus justified in changing between rewards and their corresponding soft-Q functions. We freely transform functions from the reward-pee reward-policy space: II x R to the Q-policy space: II x S2, giving us the lemma:
Lemma 3.2. If [(T.r) Epg [r(s, a)] - Ep(r(s. a)] - (x)) (b(r and
J(TT, Q) = Epp [ (T)))ss a)] - (T=())) a) (()) ) - 2b (T (T*Q). then for all policies π E II,
(m,r) J (T") (T") for all r E R, and I (T, Q) = [(T, T Q), for all Q E f2
Lemma 3.1 and 3.2 allow us to adapt the Inverse RL objective L(T,, r) to learning Q through J (TT, Q).
Simplifying our new objective (using Lemma A.3 in Appendix):
J(x, Q) E. sapp IQ -7BB''P((le.a) (7((s)) (1 =o (so)) (T")), (5)
We are now ready to study (x, Q). the Inverse RL optimization problem in the Q-policy space. As the regularizer s depends on both Q and n, a general analysis over all functions in SxA becomes difficult. We restrict ourselves to regularizers induced by convex function g : - E such that (r(r) = Epe (r(r(s, a))]
(6)
This allows us to simplify our analysis to the set of all real functions while retaining generality?. We further motivate this choice in Section
Proposition 3.3. In the Q-policy space, there exists unique saddle point (='2"") that optimizes J. i.e. Q' argmaxqen min I (π, Q) and πt argmin maxqes I (T, Q). Furthermore, n² and p" = TT" Q" are the solution to the Inverse RI. RI. objective [(TTrr).
Thus we have, maxqen min [TT, Q) max,eR mingen L(Trr).
This tells us, even after transforming to Q f-functions have retained the saddle point property of the original IRL objective and optimizing J (T, Q) recovers this saddle point. In the Q-policy space, we get an additional property:
Proposition 3.4. For a fixed Q argmin [(T, Q) is the solution to max entropy RL with rewards r r= TTQ Thus, this forms manifold in the Q-policy space, that satisfies
(<(aass=
exp(Q(s,a)),
with normalization factor Z, = E. exp Q(s, ()s,,) and ÆQ defined as the ा corresponding to Q.
Proposition 3.3 and 3.4 are telling us that if we know Q, then the inner optimization problem in terms of policy is trivial, and obtained in closed form! Thus, we recover objective that only requires learning Q:
max min QEa well
(T, Q)
max (mQ Q)
QEn
Furthermore, we have:
'The full policy class satisfies all these assumptions
Averaggng the expert occupancy allows v to adjust to arbitrary experts and accommodate multimodality

---

Proposition 3.5. Let (*() = (TQ Q). Then J is concave in Q.
Thus, this new optimization objective is well-behaved and is maximized only at the saddle point. In Appendix C, expand on our analysis and charac- terize the behavior for dif- ferent choices of regularizer vi, while giving proofs of all our propositions. Figure I summarizes the properties for the IRL. objective: there exists a optimal policy man- ifold depending on Q, al- lowing optimization along it (using ) to converge to the saddle point. We further present analysis of IL methods that learn Q-functions like SQIL [33] and ValueDICE [22] and find subtle fallacies affecting their learning.
Saddlo point
Sadole point
Concesa
(x""."")
Jor Correee
Policy Manifold
G. G-fundtio
I (Q)
exp(Q)
rewang
[t.r). Nonver
T(.Q) Guasi-convex
polay
polay
Figure 1: Properties of IRL objective in reward-pollce space and Q-policy space.
Note that although the same analysis holds in the reward-policy space, the optimal policy manifold depends on Q, which isn't ' trivially known unlike when in the Q-policy space.
Approach
In this section, we develop our inverse soft-Q learning (IQ-Learn) algorithm, such that recovers the optimal soft Q-function for MDP from given expert distribution. We start by learning energy-basee models for the policy similar to soft Q-learning and later learn an explicit policy similar to actor-critic methods.
4.1 General Inverse RI. Objective
For designing practical algorithm using regularizers the form vy (from Eq. 6), we define g using
I <(x) if E Rs otherwise
a concave function é Rs + R, such that g(x) = +00
with the rewards constrained in Rgp-
For this choice of th, the Inverse RL objective [( r) takes the form Eq. with a distance measure:
(p = max Epg (d(rr(, (ef(r((sa))) a))] (,)r(s,)),
rER,
This forms a general learning objective that allows the use wide-range of statistical distances including Integral Probability Metrics (IPMs) and f-divergences (see Appendix B).
4.2 Choice of Statistical Distances
While choosing a practical regularizer, can be useful to obtain certain properties on the reward functions we recover. Some (natural) nice properties are: having rewards bounded in a range, learning smooth functions or enforcing a norm-penalty.
In fact, we find these properties correspond to the Total Variation distance, the Wasserstein- 1 dis- tance and the -dvveregee respectively. The regularizers and the induced statistical distances are summarized in Table 2:
Table Enforced reward property, corresponding regularizer and statistical distance (Raax, K E R
Reward Property
y
do
Bound range
+ 0f Raux and too otherwise
2.max TV((p,p)
Smoothness
V 0 if Irllup < K and too otherwise
K Wi(p,P))
1.2 Penalization
(r) ar?
(p,
We recover IPMs when using identity ó and restricted reward family R
5

---

We find that these choice of regularizers' work very well in our experiments. In Appendix B, we further give table for the well known f-divergences, the corresponding and the learnt reward estimators, along with a result ablation using different divergences. Compared to x², we find other f-divergences like Censen-Shannon result in similar performances but are not as readily interpretable.
4.3 Inverse soft-Q update (Discrete control)
Optimization along the optimal policy manifold gives the concave objective (Prop 3.5): (* (1 B0 [V (so)].
max (Q) = Epg (Q( a) PBe
(9)
QEO
with V*(s) log E. ((s
For each Q, we get corresponding reward r(s, = - log E.r exp Q (5', a²)] This correspondence is unique (Lemma A.1 in Appendix), and every update step be as finding a better reward for IRL.
Note that estimating (s) exactly is only possible in discrete action spaces. Our objective forms variant soft-Q learning: to learn the optimal Q-function given an expert distribution.
4.4 Inverse soft actor-critic update (Continuous control)
In continuous action spaces, it might not be possible to exactly obtain the optimal policy TQ. which forms an energy-based model of the Q-function, and
explicit policy approximate Tq-
For any policy ı, we have objective (from Eq. 5): J(T, Q) lop - - V (5))) (1 [V*
(10)
For a fixed Q, soft actor-critic (SAC) update: min E (an - log ग [a]s)], brings '
closer to गh while always minimizing Eq. 10 (Lemma A.4 in Appendix). Here D is the distribution of previously sampled states, or replay buffer.
Thus, we obtain the modified actor-critic update rule learn Q-functions from the expert distribution:
1. For a fixed म, optimize Q by maximizing (TT, Q)-
2. For a fixed Q, apply SAC update to optimize æ towards TQ-
This differs from ValueDICE [22], where the actor is updated adverserially and the objective may not always converge (Appendix C).
5 Practical Algorithm
Pseudocode in Algorithm 1, shows our Q-learning and actor-critic variants, with differences with conventional RL algorithms in red (we optimize -J to use gradient descent). We can implement our algorithm IQ-Learn in 15 lines of code on top standard implementations (soft) DQN [14] for discrete control or soft actor-critic (SAC) [13] for continuous control, with a change on the objective for the Q-function. Default hyperparameters from [14, 13] work well, except for tuning the entropy regularization. Target networks were helpful for continuous control. We elaborate details in Appendix
D.
5.1 Training methodology
Corollary 2.1 in Appendix A states E, E(a.a)- -7.,.--......... (*)(' = (1 (s)),
where μ is any policy's occupancy. We use this to stabilize training instead of using Eq. 9 directly.
Online: Instead of directly estimating Epp (V*(so) in our algorithm, we can sample (s, a, s') from replay buffer and get a single-sample estimate )" (V* (V*(ss >V)) This removes the issue where we are only optimizing Q in the inital states resulting in overfitting of V (su), and improves the stability for convergence in our experiments. We find sampling half from the policy buffer and half from the expert distribution gives the best performances. Note that this is makes our learning online, requiring environment interactions.
'The additional scalar terms scale the entropy regularization strength and can be ignored in practice

---

Algorithm Inverse soft Q-Learning (both variants)
1: Initialize Q-function Qo, and optionally policy Re
Offline: Although Ep, (so))
2: for step l in [I...N] do
can be estimated offline we still observe an overfitting issue. In- stead of requiring policy samples
3: Train Q-function using objective from Equation 9:
t- 0e+1
aq Vol-J(())
(Use V for Q- Q-learning and VT for actor-critic)
we use only expert samples to estimate B(ssee="")~espenl (V*(s)
4:
(only with actor-critic) Improve policy Ta with SAC style actor update:
Dc+1 < or Or V. De ((s, log no [als)]
(V"(s') to sufficiently approxi-
5: end for
mate the term. This methodology
gives us state-of-art results for of-
fline II.
Algorithm 2 Recover policy and reward
1: Given trained Q- Q-function Qo, and optionally a trained policy
5.2 Recovering rewards
2: Recover policy m:
(Q-leaming)
2 exp Q3
Instead of the conventional reward function r(s, a) on state and action pairs, our algorithm allows recov- ering rewards for each transition (s, a, s') using the learnt Q-values as follows:
(actor-critic) x := ns 3: For state s, action a and s' ~ P( a)
4: Recover reward r(s, a, Qa(s, a) -V (s')
((s,0,8) = -(s,
(1I)
Now, ( (Q(s,)) ~V" (s')] = -(Es,a)-EEBnP((bb... -E [V* = (TQs,a) This is just the reward function r(s, a) we want. So by marginalizing over next-states, our expression correctly recovers the reward over acteons Thus, Eq. 11 gives the reward over transitions.
Our rewards require s' which can be sampled from the environment, or by using a dynamics model.
5.3 Implementation of Statistical Distances
Implementing TV and W1 distances is fairly trivial and we give details in Appendix B. For the diveerggeee,, we note that it corresponds to x(x)
ia x². On substituting in Eq. 9, we get
max Eps [(Q(s, a) (B (*()')) - (1 - E, [V* QEB
a.. Eps [[Q(s,)) a) ~pria))))) (*'))
In fully offline setting, this can be further simplified as (using the offline methodology in See 5.1):
1 E ((Q(s, a) -B V (8())
EEp (+)) QER
min
(12)
4a
This is interestingly the same as the Q-learning objective in CQL [23], an state-of-art method for offline RI (using 0 rewards), and shares similarities with regularized behavior cloning [33]
5.4 Learning state-only reward functions
Previous works like AIRL [10] propose learning rewards that are only function of the state, and claim that these form of reward functions generalize between different MDPs. We find our method can predict state-only rewards by using the policy and expert state-marginals with a modification to Eq. 9:
max (Q) = Es>og(s) [aa (á(Q(s, a) -.E V* (s I] - 1010-(11101000 ppp (V* (So) QER
Interestingly, our objective no longer depends on the the expert actions Tg and can be used for IL using only observations. For the sake of brevity, we expand on this in Section 1 in Appendix A.
6 Related Work
Classical II: Imitation learning has a long history, with early works using supervised learning to match a policy's actions those of the expert [15, 35]. A significant advance was made with the formulation IL as the composition of RL. and IRL [29, 1, 43), recovering the expert's policy
'The simplification get Eq. (12) is not applicable in the online IL setting where method differs
7

---

by inferring the expert's reward function, then finding the policy which maximizes reward under this reward function. These early approaches required uund-designed featurization of the MDP, limiting their applicability to complex MDPs. In this setting, early approaches [9, 31] noted formal equivalence between IRI. and II. using an inverse Bellman operator similar to our own.
Online IL: More recent work aims to leverage the power of modern machine learning approaches to learn good featurizations and extend IL complex settings. Recent work generally falls into one of two settings: online or offline. In the online setting, the IL algorithm is able to interact with the environment to obtain dynamics information. GAIL [17] takes the nested RL/IRL formulation of earlier work, optimizing over all reward functions with a convex regularizer. This results in the objective in Eq. (3). with a max max-min adversarial problem similar
GAN [11]. A variety of further work has built on this ad adversarial approach [21, , 10, 3]. A separate line of work aims to simplify the problem in Eq. (3) by using a fixed r or n. In SQIL [33], r is chosen to the 1-0 indicator on the expert demonstrations, while ASAF [4] takes the GAN approach and uses discriminator (with role similar to r) of fixed form, consisting of ratio of expert and learner densities. AdRIL [38] is a recent extension of SQIL, additionally assigning decaying negative reward to previous policy rollouts.
Offiie IL: In the offline setting, the learner has no access to the environment. The simple behavioural cloning (BC) [34] approach is offline, but doesn't 't use any dynamics information. ValueDICE [22] a dynamics-aware offline approach with an objective somewhat similar to ours, motivated from minimization of a variational representation of the KL-divergence between expert and learner policies. ValueDICE requires adversarial optimization to learn the policy and Q-functions, with a biased gradient estimator for training. We show way to recover a unbiased gradient estimate for the KL-divergence in Appendix C. The O-NAIL algorithm [2] builds on ValueDICE and combines with an SAC update to obtain method that is similar to our algorithm described in section 4.4, with the specific choice of reverse KL-divergence as the relevant statistical distance. The EDM method [19] incorporates dynamics via learning an explicit energy based model for the expert state occupancy, although some theoretical details have been called into question (see [37] for details). The recent AVRIL approach [6] uses variational method to solve probabilistic formulation of IL, finding posterior distribution over r and т. Illustrating the potential benefits of alternative distances for IL, the PWIL [7] algorithm gives non-adversarial procedure to minimize the Wasserstein distance between expert and learned occupancies. The approach is specific to the primal form of the Wi-distance, while method (when used with the Wasserstein distance) targets the dual form.
7 Experiments
7.1 Experimental Setup
We compare IQ-Learn ("TQ") to prior work on diverse collection of RL tasks and environments ranging from owwddimessional control tasks: CartPole, Acrobot, LunarLander - to more challenging continuous control MuJoCo tasks: HalfCheetah, Hopper, Walker and Ant. Furthermore, we test on the visually challenging Atari Suite with high-dimensional image inputs. We compare on offline IL. with no access the the environment while training, and online IL- with environment access. We show results on W1 and x² as our statistical distances, as we found them more effective than TV distance. In all cases, we train until convergence and average over multiple seeds. Hyperparameter settings and training details are detailed in Appendix D.
7.2 Benchmarks
Offline IL We compare to the state-of-art IL methods EDM and AVRIL, following the same experimental setting as [6]. Furthermore, we compare with ValueDICE which also learns Q-functions, albeit with drawbacks such as adversarial optimization. We also experimented with SQIL, but found that it was not competitive in the offline setting. Finally, we utilize BC as an additional IL baseline.
Online IL We use MuJoCo and Atari environments and compare against state-of-art online IL methods: ValueDICE, SQIL and GAIL. We only show results on X2 as W1 was harder to stabilize on complex environments Using target updates stabilizes the Q-learning on MuJoCo. For brevity, further online IL. results are shown in the Appendix D.
xx2 and W1 be used together to still have convex regularization and more stable
8

---

7.3 Results
Acrobot
Cartpole
LunarLander
500
Expert
Expert
Expert
Reward
(Gurs)
503
Randam
12 Mi (Gurs)
Bewsard
302
Random
(Curs)
-002
AvHUL
(Our)
200
AVRIL
AVRIL
-00
BC
-103
BC
EDM
EDH
-203
-503
15
Number of Expert Pajectories
Number Expert Pajectories
Number Expert Trajectories
Figure 2: Offline IL results. We plot the average environment returns the number of expert trajectories.
Offiine IL We present results on the three offline control tasks Figure On all tasks, IQ strongly outperforms prior works compare to in performance and sample efficiency. Using just expert trajectory, we achieve expert performance on Acrobot and reach near expert on Cartpole.
Mujoco Control We present re- sults on the MuJoCo tasks using a single expert demo in Table 3. IQ achieves expert-level performance in all the tasks while outperforming prior methods like ValueDICE and GAIL. We did not find SQIL competitive in this setting, and skip it for brevity.
Table Mujoco Results. We show our performance on MuloCo control tasks using a single expert trajectory.
Task
GAIL
DAC
ValueDICE
IQ (Ours)
Expert
Hopper
3252.5
3305.1
3312.1
3546.4
3532.7
Half-Cheetall
3080.0
4080.6
3835.6
5076.6
5098.3
Walker
4013.7
4107.9
3842.6
5134.0
5274.5
Ant
2299.1
1437.5
1806.3
4362.9
4700.0
Humanoid
232.6
380.5
644.5
5227.1
5312.8
Atari We present our results on Atari using 20 expert demos in Figure 3. We reach expert performance on Space Invaders while being near expert on Pong and Breakout. Compared prior methods like SQIL, IQ obtains 3-7x normalized score' and converges in ~300k steps, being 3x faster compared to Q-learning based RL. methods that take more than 1M steps to converge. Other popular methods like GAIL. and ValueDICE perform near random even with 1M env steps.
Pong
Breakout
Space Invaders
10
002-
Reward
Bawward
200-
000-
250
-20
200-
202
50-
-20
0.0
0.4
64
0.0
68
Env steps (in million))
Eny steps din millions
Enw steps lin miltions)
Figure 3: Atari Results. We show the returns vs the number of env steps. (Averaged over seeds)
7.4 Recovered Rewards
IQ has the added benefit of recovering rewards and can be used for IRL. On Hopper task, our learned rewards have a Pearson correlation of 0.99 with the true rewards. In Figure 4, we visualize our recovered rewards in a simple grid environment. We elaborate details in Appendix D.
8 Discussion and Outlook
We present new principled framework for learning soft-Q functions for II. and recovering the optimal policy and the reward, building on past works in IRL. [43]. Our algorithm IQ-LLearn outperforms prior methods with very sparse expert data and scales to complex image-based environments, We also recover rewards highly correlated with actual rewards. It has applications autonomous driving and complex decision-making but proper considerations need to be taken into account to ensure safety
normalizing rewards obtained from random behavior 0 and expert

---

Newards Map Ground
Reward Map Recovered
Value Map Recovvred
A2
90
51
55
60
51
11
03
03
24
23 64
14
44
13
04
01
60
00
34
02
63
6.
60
22
53
114
Figure 4: Reward Visualization. We use discrete Grid environment with 5 possible actions: up. down, left, right, stay. Agent starts random state. (With 30 expert demos)
and reduce uncertainty, before any deployment. Finally, human expert data can have errors that can propagate. A limitation of our method that our recovered rewards depend on the environment dynamics, preventing trivial use on reward transfer settings. One direction of future work could be to learn reward model from the trained soft-Q model make the rewards explicit.
Acknowledgements
We thank Kuno Kim and John Schulman for helpful discussions. We also thank Ian Goodfellow as some initial motivations for this work were developed under an internship with him.
10 Funding Transparency
This research was supported in part by NSF (#1651565, #1522054, #1 #1733686), ONR (N00014-19-1- 2145), AFOSR PPA9550-19-1-000 and FLI.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. International conference Machine learning (ICML), 2004. 1,
[2] Oleg Arenz and Gerhard Neumann. Non-adversarial imitation learning and its connections to adversarial methods. arXiv preprint arXiv:2008.03525, 2020.
[3] Nir Baram, Oron Anschel, and Shie Mannor. Model-based adversarial imitation learning. stat, 1050:7, 2016. 8
[4] Paul Barde, Julien Roy, Wonseok Jeon, Joelle Pineau, Christopher Pal, and Derek Nowrouzezahrai. Adversarial soft advantage fitting: Imitation learning without policy op- timization. Advances in neural information processing systems (NeurIPS), 2020.
[5] M. Bloem and Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. 53rd IEEE Conference on Decision and Control, pages 491 4916, 2014. 3
[6] Alex J. Chan and Mihacla van der Schaar. Scalable bayesian inverse reinforcement learning, 2021. 8,
[7] Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pictquin. Primal wasserstein imitation learning. In ICLR 2021-Ninth International Conference on Learning Representations, 2021.
[8] G Alphastar DeepMind. Mastering the real-time strategy game starcraft ii, 2019.
[9] Krishnamurthy Dvijotham and Emanuel Todorov. Inverse optimal control with lincarly-solvable mdps. In ICML, 2010. 8
[10] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018. URL
https://openreviev.eet ptses//ooummidddrdlyllllll
1,
[11] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, 2014.
10

---

[12] Ishaan Guirajani, Faruk Ahmed, Martín Arjovsky, Vincent Dumoulin, and Aaron C. Courville. Improved training of wasserstein gans. In NIPS, 2017. 16
[13] T. Haarnoja, Aurick Zhou, P. Abbeel, and S. Levine. Soft actor-critic; Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018. 2, 3, 6, 15, 22
[14] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. 2017. 2, 14
[15] G HAYES. A robot controller using learning by imitation. In Proc. 2nd Int. Symposium on Intelligent Robotic Systems, LIFTA-IMAG, Grenoble, France, 1994.
[16] Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard. Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. International conference on artificial intelligence and statistics (AISTATS), 2016. 2
[17] Jonathan Ho and S. Ermon. Generative adversarial imitation learning. In NIPS, 2016. 1, 2, 4. 14
[18] Vinamra Jain, Prashant Doshi, and Bikramjii Banerjee. Model-free irl using maximum likelihood estimation. AAAI Conference Artificial Intelligence (AAAI). 2019. 2
[19] Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energy-based distribution matching. Advances in neural information processing systems (NeurIPS), 2020. 8, 21
[20] Edouard Klein, Matthieu Geist, and Olivier Pietquin. Batch, off-policy and model-free appren- ticeship learning. European Workshop on Reinforcement Learning (EWRL), 2011. 2
[21] Iya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actorcritti Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference Learning Representations, 2018.
1,8
[22] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy dis- tribution matching. In International Conference on Learning Representations, 2020. URL https://openreviie
foeeet/frrmmiddiyy-eee. JCDFF. 2, 8, 20
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative g-learning for /wwpiv.xiivv.////////a/////aaa orgo aa//ab////0000 04779. 2,
offline reinforcement learning. 2020. URL https: /
[24] Donghun Lee, Srivatsan Srinivasan, and Finale Doshi- Truly batch apprenticeship learning with deep successor features. International Joint Conference on Artificial Intelligence (IJCAI). 2019. 2
[25] Mario Lucic, Karol Kurach, Marcin Michalski, S. Gelly, and O. Bousquet. Are gans created equal? large-scale study. In NeurIPS, 2018. 16
[26] Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. ArXiv, abs/1 802.05957, 2018. 16
[27] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602. 2013. 21, 22
[28] Olir Nachum, Yinlam Chow, B. Dai, and L. Li. Dualdice: Efficient estimation of off-policy stationary distribution corrections. 2019. 15
[29] Andrew Y Ng. Stuart Russell, al. Algorithms for inverse reinforcement learning. Interna- tional conference on Machine learning (ICML), 2000. 2,
[30] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for apprenticeship learning. International conference on Autonomous agents and multi-agent systems (AAMAS), 2014. 2
11

---

[31] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Bridging the gap between imitation learning and inverse reinforcement learning. IEEE transactions on neural networks and learning systems, 28(8):1814-1826, 2016. 8
[32] Antonin Raffin. RI baselines3 zoo. http:p//ggtttt.o.om/m////////a// 2020. 21
[33] Siddharth Reddy, A. Dragan, and Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards, arXiv: Learning. 2020. 2, 8, 20, 25
[34] Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. International conference on artificial intelligence and statistics (AISTATS), 2010. 1, 2, 8, 24
[35] Claude Sammut, Scott Hurst, Dana Kedzier, and Donald Michie. Learning to fly, In Proceedings of the Ninth Conference on Machine Learning, pages 385-393. Elsevier, 1992.
[36] Maurice Sion. On general minimax theorems, Pacific Journal of Mathematics, 8(1):171 176,
1958. doi: ppm/ 103040253. URL https://doo org/ / 17
[37] Gokul Swamy, Sanjiban Choudhury, J. Andrew Bagnell, and Zhiwei Steven Wu. A Critique of Strictly Batch Imitation Learning. arXiv:2 10.02063 [cs], October 2021.
[38] Gokul Swamy, Sanjiban Choudhury, Zhiwei Steven Wu, and Andrew Bagnell, Of moments and matching: Trade-off's and treatments in imitation learning, arXiv preprint drXiv:2103.03236, 2021. 8
[39] Lu Wang, Wenchao Yu, Xiaofeng He, Wei Cheng, Martin Renqiang Ren, Wei Wang, Bo Zong, Haifeng Chen, and Hongyuan Zha. Adversarial cooperative imitation learning for dynamic treatment regimes. In Proceedings The Web Conference 2020, pages 1785-1795, 2020. 1
[40] Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. In NeurlPS, 2019. 26
[41] Jinyun Zhou, Rui Wang, Xu Liu, Yifei Jiang, Shu Jiang, Jiaming Tao, Jinghao Miao, and Shiyu Song. Exploring imitation learning for autonomous driving with feedback synthesizer and differentiable rasterization. arXiv preprint arXiv:2103.01882 2021.
[42] Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010. 3
[43] Brian Ziebart, Andrew L. Maas, Bagnell, and Dey. Maximum entropy inverse reinforce- ment learning, In AAAI, 2008. 2, 7, 9, 23
12


Reviews:
[{"review_id": "s79uNiUeA8", "reviewer": "Reviewer_3TJb", "summary": "This paper suggests an alternative approach, equivalent to adversarial IRL (that learns both the policy and the reward in an adversarial fashion), but in a direct way (learning the Q-function that encompasses both the reward and the policy). As in adversarial methods (GAIL, AIRL etc) they use the maximum entropy assumption and more explicitly, they assume that the optimal policy is greedy wrt an optimal soft-Q-function. \n\nUsing this relation between the optimal policy and the Q-function, they restrict the set of solutions to a manifold in which the objective only depends on the Q-function. They just have to minimize this objective (ensuring on the side that the policy stays in the manifold, i.e. it stays greedy) to obtain an optimal policy and Q-function. Their solution, in theory equivalent to the ones from an adversarial setting, is expected to outperform them as they avoid the learning difficulties of adversarial learning. Optionally, they can recover a reward function from the Q-function by simply inverting the Bellman operator. \n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel IL/IRL algorithm, that can be applied both in online and offline settings, that is easy to adapt for continuous states/action spaces (approximating the value function within a separate critic network). The method is simpler to implement (close to SAC) and proved to converge. \n\nI would argue to accept it, my only concern being the lack of experiments: \nMujoco: Did the authors try Humanoid? \nAtari: only 3 games, I understand that these environments are longer to train and require more engineering, but presenting just 3 envs looks like they tried more but had less good results on the other. I believe this is not the case, but having more games would really improve the paper (even if the results are not that good in other games, the paper is already great).\nReward reconversion: having results from more environments (at least the other mujoco) would be great to convince that the reward is correlated. \n\nHowever, the theory part is nice enough and the already given results are promising, so I will go for an accept. But if the authors can provide more results in the paper revision I will even increase my score.\n\nDetails:\nA closely related IRL work inverses soft-policy improvement also to recover the soft-Q-function, but assuming the observed trajectories are generated by a soft-Q learner and not an expert:\nLearning from a Learner (ICML 2019) by Jacq, Geist, and Pietquin.\n\nIn Eq.1, the notation $\\pi^*$ is confusing, as it should be used for the policy that is greedy wrt the optimal soft-Q function ($\\pi^*(a\\vert s) \\propto \\exp(Q^*(s,a))$).\n\nSame for $J^*(Q)$, that would be the optimized objective after convergence.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper has no societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "noKc_A6hlf_", "reviewer": "Reviewer_rfsf", "summary": "This work aims at improving the sample efficiency in terms of expert demonstrations in IL. In order to do so, it transforms the usual max-min objective of IRL into a single maximization objective by expressing both the reward and the policy in terms of the Q-function. The methods are supported with proofs and tested on both discrete and continuous action environments for the online setting and on simpler environments for the offline setting.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "2. Strong points\n\nThe method seems well supported by the theoretical derivations and the experiments. It is tested on many different settings (offline/online, continuous/discrete actions, large/small observation space) and outperforms the baselines. It can also be used for IRL as it retrieves rewards close to the ground truth ones (at least in simple environments). The method seems to be implemented easily which would make it practical.\n\n\n3. Weak points\n\n   1- The related work section could be improved. To my knowledge, Klein et al. (2012) were the first to propose to represent the policy in terms of the Q-value to directly learn a reward function in Off-line IRL. Also, it would be interesting to discuss more in-depth the relationship of the proposed method with ASAF which also proposed an analytical solution to the maximization step of the min-max IRL objective enabling to directly learn the Q-values parametrizing the policy.  \n\n   2- The derivation of the method could be made clearer, in particular how to get to Eq. (5) and how it yields the objectives of the method Eqs. (9) and (10).\n\n\n4. Recommendation\n\nI recommend accept because the method seems efficient, practical, and well supported both theoretically and experimentally but I think the paper can be improved in terms of clarity and discussion with related methods. Additionally, some points remain to be clarified (see 5. Additional Remarks)\n\n------------------------------\n ####  UPDATE\nThe author's response addressed my concerns and provided additional experiments so I decided to raise my score. \n\n------------------------------\n\n5. Additional Remarks \n\n   1- In the continuous case, when SAC is used to improve the policy, the method falls back to a min-max objective. How do you explain that it does not experiment with the same drawbacks as adversarial methods such as ValueDICE and GAIL?\n\n   2- I would be interested in seeing how the performance on Atari varies in terms of the number of expert demonstrations (like Fig 2.).  \n\n   3- I would be interested in seeing the performance of agents trained on the learned IRL reward as correlation is not always a good indicator of the quality of the reward. For example, the hardcoded reward of SQIL will train good agents and the one of GAIL can't (it is an apprenticeship algorithm and not an IRL one) yet SQIL has a lower correlation than GAIL (line 723 Appendix D).\n\n   4- The methods you compare against on Atari were proposed for low dimension observations (MuJoCo), are there other IL/IRL methods you could compare against that were proposed for high dimensional envs?\n\n\n6. Minor Remarks\n\n   1- I am confused at line 490 and line 82: if $Q=\\mathcal{B}Q$ and $Q=\\mathcal{B}r$ then we would have $Q=r$ which does not make sense. Moreover $\\mathcal{B}$ operates on $Q$ not $r$\n\n   2- line 81-82, $Q$ is unique provided that $r$ is given but also $\\pi$. It might be nice to differentiate $Q$ and optimal $Q$ (using for example $Q^*$ in Eqs. (1), (2), etc.) \n\n   3- line 271-272 \"with role similar to $r$\" is confusing in the case of ASAF as the discriminator is not used to learn a reward in order to train the imitation policy like GAIL or AIRL (or the hard-coded reward of SQIL)\n\n\nReferences\n\nKlein, Edouard, et al. \"Inverse reinforcement learning through structured classification.\" NIPS 2012. 2012.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4,5 hours"}, "extra_text": {}}, {"review_id": "bCY9U_usDXQ", "reviewer": "Reviewer_XfMp", "summary": "This paper proposes to get around the typical two-stage inverse RL pipeline of (1) learn a reward model and (2) optimize it, by encapsulating both rewards and dynamics in a single Q function, learned by imitation. This reframing of IRL preserves the standard properties of IRL but is simpler to optimize and avoids the need for adversarial training used by other methods. This results in a simple imitation learning method that is competitive with or outperforms reasonable baselines on a few standard benchmarks. The proposed algorithm has other nice features, including the ability to easily recover the learned reward and to be trained both offline and online, and it can be easily rephrased to obtain a method for imitation from observations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n- See my summary. I found the core technical contributions of this work to be novel, and potentially high-impact, as they simplify and reframe IRL to remove a common annoyance: the need for two-stage (or simultaneous two-part) training.\n- The theoretical analysis is correct as far as I can tell. Although certain details were overemphasized to my taste (e.g. the choice of regularizers, which didn't make a huge difference empirically), the exposition was quite clean and easy to follow.\n- The experiments compare against a representative set of reasonably strong baselines, and include enough domains to suggest that this method will work generally.\n\nNegatives:\n- As far as I can tell, none of the experimental domains here require generalization. These domains include very stereotyped initial state distributions, which means that an imitation agent that exactly replays the expert demonstration will solve the task. This likely plays a role in the proposed method's ability to perform well given a single demonstration. And in my opinion it is misleading, because it sidesteps one of the principle challenges in RL and IL, namely learning policies that can be used in the presence of at least mild occupancy drift (see [1] and [2] for discussions). Ideally, imitation learning methods should be evaluated in settings where the expert initial state distribution does not completely cover the states the agent can start in, such as the DeepMind Control Suite (as discussed in [2]), procedural tasks like CoinRun (as used in [3]) or tasks with more challenging perception (such as the door-opening task in [4]).  \n- The proposed baselines are not as strong as they could be. In particular, Discriminator-actor-critic [5] or strongly regularized GAIL variants (such as [6]) are closer to what are used in practice than vanilla GAIL and typically give much better results. And another recently published method [4] produces similar results on Gym with single demonstrations, but is not mentioned or compared to.\n\nMinor:\n- I found the point made in footnote 2 to be obscure as phrased. Can you rephrase this to explain exactly how this relates to (6) and what this has to do with arbitrary experts?\n\nDespite the limitations of the experimental evaluation, I believe the proposed algorithm is a solid and potentially quite impactful contribution to the IRL and imitation learning literature. I found the exposition insightful and I admire the simplicity of the proposed result. \n\nI am willing to raise my score if the limitations of the experimental evaluation are addressed.\n\n[1] Mania et al - NeurIPS 2018 - Simple random search of static linear policies is competitive for reinforcement learning\n\n[2] Jaegle et al - ICML 2021 - Imitation by Predicting Observations\n\n[3] Edwards et al - ICML 2019 - Imitating Latent Policies from Observation\n\n[4] Dadashi et al - ICLR 2021 - Primal Wasserstein Imitation Learning\n\n[5] Kostrikov et al - ICLR 2019 - Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning\n\n[6] Peng et al - ICLR 2019 - Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "4rQzNccBZO", "reviewer": "Reviewer_HuRr", "summary": "The paper presents an IL/IRL method that avoids estimating the reward during optimization by alternating between updates of the policy and Q-function instead, which builds on recent insights from ValueDice. Indeed, the optimization problem for learning the Q-function is very similar to the one used by ValueDice, when optimizing the reverse KL. Notable differences compared to ValueDice are the generalization to a large class of divergences (including f-divergences), non-adversarial formulation, and a procedure to recover rewards.\nIn particular, I think that the modification for recovering a state-only reward function is very interesting.\nThe algorithm only requires mild modifications of SAC (mainly by using a different loss for the Q-function) and shows promising results in experiment evaluations.", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality\n======\nOverall, the approach seems technically sound and promising. Section 3 establishes the foundations for framing inverse reinforcement learning in terms of the Q function.  Section 4 exploits the insight that we do not need to reverse the order of $\\text{IRL} \\circ \\text{RL}$ (which was proposed by GAIL) to make the optimization tractable, if we directly learn the Q-function instead of the reward function, since the Q-function is already an implicit representation of the optimal policy, making the inner reinforcement learning trivial. \n\nThe second important important insight is that the Q-function for a state-only reward function can be learned, simply by replacing the expectation over expert actions with an expectation over agent actions, which can be shown using importance sampling.\n\nHowever, I have a question regarding the state-only rewards. It is not clear how a reward function of the form $r(s)$ is derived from the learned Q-function. From what I understand the learned models are all function of states and actions, and while the inverse Bellman update should result in a state-only reward, it is not clear to me how this step can be performed in practice.\n\nNovelty / Significance\n===================\n* The presented derivations and algorithm are novel and very interesting. In particular, the derivation of the loss for the Q-function for general divergences and the modification for state-only rewards seem significant. Furthermore, the experimental results are promising.\n\n* I think that Section 3 in itself does not contain that many new insights because it well known that the advantage function is a shaped [1] reward function, which also holds for MaxEnt RL, where maximizing the expected advantage + entropy regularization directly corresponds to minimizing the RKL to the distribution specified by A (or Q when disregarding normalization). Also, the transformation from the reward objective to Q-objective via the \"inverse\" Bellman operator strongly resembles the change-of-variables applied by DualDice/ValueDice. However, applying these ideas to Eq. 3 is an important contribution of the current submission. \n\n* The idea to simplify IRL/IL by directly learning the Q-function is not novel, and was introduced already in the early times of IRL (e.g. Opt-Q [2]) and became more traction with recent deep learning methods such as ValueDice.\n\n* The Q-update for the RKL looks very similar to the ValueDice objective, which I think should be stated more clearly. From what I can tell, the differences to ValueDice are 1) The use of the f-gan-formulation of the RKL instead of Donsker-Varadhan (e.g. using the expectation instead of the log-expectation in the first term), 2) Learning the soft-Q-function rather than the Q-function, 3) the actor update. Here 1) is rather insignificant as this option was also stated by Kostrikov et al. 2020. More similar is the connection to O-NAIL [3], which also uses the (DV or F-Gan) objective for the soft-Q-function in combination with the soft-actor-critic update.  Here the only difference seems to be that they provide an additional reward based on the last policy to incur a KL penalty between updates.\n\n\nClarity\n======\nThe presentation of theory and algorithm is excellent. \n\n\nReferences\n==========\n[1] Ng, Andrew Y., Daishi Harada, and Stuart Russell. \"Policy invariance under reward transformations: Theory and application to reward shaping.\" Icml. Vol. 99. 1999.\n\n[2] Dvijotham, Krishnamurthy, and Emanuel Todorov. \"Inverse optimal control with linearly-solvable MDPs.\" ICML. 2010.\n\n[3] Arenz, Oleg, and Gerhard Neumann. \"Non-Adversarial Imitation Learning and its Connections to Adversarial Methods.\" arXiv preprint arXiv:2008.03525 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "limitations and societal impact have been sufficiently addressed.", "review_text": "", "extra_scores": {"ethical_concerns": "The paper does not raise ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received very positive reviews initially. The reviewers liked the originality of the work, it's theoretical soundness and the good performance of the algorithm. They had some concerns and wanted to see more about generalization capacities, stronger baselines and a larger set of test environments. The discussion was rich enough to convince the reviewers to raise their scores. The authors made additional experiments and addressed all the concerns of the reviewers. 
Decision: Accept (Spotlight)