Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees
Authors: Siliang Zeng
Chenliang Li
Alfredo Garcia
Mingyi Hong
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Inverse Reinforcement Learning

Abstract: Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert. Many algorithms for IRL have an inherent nested structure: the inner loop finds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of fit. For high dimensional environments such nested-loop structure entails a significant computational burden. To reduce the computational burden of a nested loop, novel methods such as SQIL \cite{reddy2019sqil} and IQ-Learn \cite{garg2021iq} emphasize policy estimation at the expense of reward estimation accuracy. However, without accurate estimated rewards, it is not possible to do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks. In this paper we develop a novel {\em single-loop} algorithm for IRL that does not compromise reward estimation accuracy. In the proposed algorithm, each policy improvement step is followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm provably converges to a stationary solution with a finite-time guarantee. If the reward is parameterized linearly we show the identified solution corresponds to the solution of the maximum entropy IRL problem. Finally, by using robotics control problems in Mujoco and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.

Maximum-Likeliood Inverse Reinforcement Learning with Finite-Time Guarantees
Siliang Zeng
Chenliang Li The Chinese University of Hong Kong, Shenzhen, China chenliangli@limg cuhk edu. cn
University of Minnesota, Twin Cities Minneapolis, MN, USA
zeng0176@umn edu
Alfredo Garcia Texas A&M University College Station, TX, USA alfredo. garcia@tamu. edu
Mingyi Hong
University of Minnesota, Twin Cities Minneapolis, MN, USA mhong@umn edu
Abstract
Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy that best fits observed sequences of states and actions implemented by an expert. Many algorithms for IRL have an inherently nested structure: the inner loop finds the optimal policy given parametrized rewards while the outer loop updates the estimates towards optimizing a measure of fit. For high dimensional environments such nested-loop structure entails a significant computational burden. To reduce the computational burden of a nested loop, novel methods such SQIL. M and IQ-Leamn 2 emphasize policy estimation at the expense of reward estimation accuracy. However, without accurate estimated rewards, is not possible do counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks. In this paper we develop novel single-loop algorithm for IRL that does not compromise reward estimation accuracy. In the proposed algorithm, each policy improvement step followed by a stochastic gradient step for likelihood maximization. We show that the proposed algorithm provably converges stationary solution with a finite-time guarantee. the reward parameterized linearly, we show the identified solution corresponds to the solution of the maximum entropy IRL problem. Finally, by using roboties control problems in MuloCo and their transfer settings, we show that the proposed algorithm achieves superior performance compared with other IRL and imitation learning benchmarks.
1 Introduction
Given observed trajectories of states and actions implemented by an expert, we consider the problem of estimating the reinforcement learning environment which the expert was trained. This problem is generally referred to as inverse reinforcement learning (IRL) (see [3] for a recent survey). Assuming the environment dynamics are known (or available online), the IRL problem consists of estimating the reward function and the expert's 's policy (optimizing such rewards) that best fits the data. While there are limitations on the identifiability rewards 44) the estimation of rewards based upon expert trajectories enables important counterfactual analysis such as the estimation of optimal policies under different environment dynamics and/or reinforcement learning of new tasks.
In the seminal work [5), the authors developed an IRL formulation, in which the model for the expert's behavior is the policy that maximizes entropy subject to a constraint requiring that the expected
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

features under such policy match the empirical averages in the expert's observation dataset. The algorithms developed for MaxEnt-IRL [5 have a nested loop structure, alternating between an outer loop with a reward update step, and an inner loop that calculates the explicit policy estimates. The computational burden of this nested structure is manageable in tabular environments, but becomes significant in high dimensional settings requiring function approximation.
Towards developing more efficient IRL algorithms, a number of works B propose to leverage the idea of adversarial training [3] These algorithms learn a non-stationary reward function through training discriminator, which is then used to guide the policy to match the behavior trajectories from the expert dataset. However, [14] pointed out that the resulting discriminator (hence the reward function) typically cannot be used in new learning tasks, since is highly dependent on the corresponding policy and current environment dynamics. Moreover, due to the brittle approximation techniques and sensitive hyperparameter choice in the adversarial training, these IRI. algorithms can [15 16].
be unstable.
have developed algorithms to alleviate the computational burden of the nested-loop training procedures. In (.. the authors propose to model the IRL using certain maximum entropy RL problem with specific reward function (which assigns r +1 for matching expert demonstrations and r = O for all other behaviors). Then a soft Q imitation learning (SQIL) algorithm developed. In 2 the authors propose to transform the standard formulation of IRL (discussed above) into a single-level problem, through learning a soft O-function to implicitly represent the reward function and the policy. An inverse soft-Q Q learning (IQ-Learn) algorithm then developed, which is shown to be effective in estimating the policy for the environment that it is trained on. Despite being computationally efficient, IQ-Learn sacrifices the accuracy in estimating the rewards since it indirectly recovers rewards from a soft Q-function approximator which is highly dependent upon the environment dynamics and does not strictly satisfy the soft-Bellman equation. Therefore is not well-suited for counterfactual prediction transfer learning setting.
More recent works
Finally, in f-IRL 14 the authors consider an approach for estimating rewards based on the minimization of several measures of divergence with respect to the expert's 's state visitation measure. The approach is limited to estimating rewards that only depend on state. Moreover, while the results reported are based upon a single-loop implementation, the paper does not provide convergence guarantee to support performance. We refer the readers to Appendix A for other related works.
Our Contributions. The goal of this work is to develop an algorithm for IRL which is capable of producing high-quality estimates of both rewards and behavior policies with finite-time guarantees. The major contributions of this work are listed below.
We consider a formulation of IRL based maximum likelihood (ML) estimation over optimal (entropy-regularriee) policies, and prove that a strong duality relationship with maximum entropy IRL holds if rewards are represented by linear combination of features. D The ML formulation is a bi-level optimization problem, where the upper-level problem maximizes the likelihood function, while the lower-level finds the optimal policy under the current reward parameterizatton. Such bi-level structure is not only instrumental to the subsequent algorithm design, but is also flexible to incorporate the use of state-only, as well the regular reward function (which depends the state and action pair). The former is suitable for transfer learning since it is insensitive to the changes of the environment dynamics, while the latter can be used efficiently imitate the expert policy.
Based on the ML-IRL formulation, we develop an efficient algorithm. To avoid the computational burden of repeatedly solving the lower-level policy optimization problem, the proposed algorithm has a single-loop structure where the policy improvement step and reward optimization step are performed alternatingly so that each step can be performed relatively cheaply. Further, show that the algorithm has strong theoretical guarantees: to achieve certain e-approximate stationary solution for non-linearly parameterized problem, it requires (c-2) steps of policy and reward updates each. To our knowledge, it is the first algorithm which has finite-time guarantee for the IRL problem under nonlinear parameterization of reward functions.
We conduct extensive experiments to demonstrate that the proposed algorithm outperforms many state-of-the-art IRL algorithms in both policy estimation and reward recovery. In particular, when
Heuristic arguments for this duality result are discussed in 5 wherein the distribution of state-action paths is approximated (see equation (4) in (5) and the equivalence between maximum entropy estimation and maximum likelihood (over the class of exponential distributions) [17 is invoked.

---

transferring to a new environment, RL algorithms using rewards recovered by the proposed algorithm outperform those that use rewards recovered from existing IRL and imitation learning benchmarks.
Preliminaries
In this section, we review the fundamentals of the maximum entropy inverse reinforcement learning (MaxEnt-IRL.)) We consider an MDP defined by the tuple (S, A, , 7, r): S and denote state space and the action space respectively; Pss'|s, a) S x A 8 [0, denotes the transition probability; 17()) denotes the distribution for the initial state; r(s, a) function and is discount factor.
A R is the reward
The MaxEnt- IRL formulation 6 18 consists of finding a policy maximizing entropy subject to the expected features under such policy matching the empirical averages in the expert's observation dataset. Specifically, the MaxEnt-IRL. formulation is given by:
max H(T) = EYNT
-2 log (aglstt)
(MaxEnt-IRL)
20
s.t.
Exnn
" at)
= E
Σ
where - _ ( [= (s, a) and n= denotes the expert policy. Let 0 denote the dual variable for the linear constraint, then the Lagrangian of (MaxEnt-IRa) is given by
denotes a trajectory, (ss, a) the feature vector of the state-action pair
oe Σ + "((ose ai) aol))
= +())+ 0, E,Ey
Σ (o(() ai)
ET
(1)
In 6 18 19, the authors proposed "dual descent" algorithm, which alternates between i) solving max, [(T, 0) for fixed and ii) a gradient descent step to optimize the dual variable It is shown that the optimizer Tă in step i) can be recursively defined as TT (aclst)) Zaglen. where
Zu.o
log -0, o @(st T + - [log .e....e] and log Ze, log (EEEA Zals.,))-
From a computational perspective, the above algorithm which repeatedly computes the optimal policy лे under each variable A. It is known that when the underlying MDP is of high-dimension, such an algorithm can be computationally prohibitive
efficient: has nested-loop structure,
Recent work [2] proposed an algorithm called IQ-Learn to improve upon the MaxEnt-IRL. by considering a saddle-point formulation:
min max H(x)) + +
Σ xf r(s,, at)] ETagE Σ n2 r(s,, r(s,, a.]} 0=
(2)
t=
where r(st, oz) is the reward associated with state-action pair (st, oz). The authors show that this problem can be transformed into optimization problem only defined in terms of the soft Q-function, which implicitly represents both reward and policy. IQ-Learn is shown to be effective in imitating the expert behavior while only relying on the estimation of the soft Q-function. However, the implicit reward estimate obtained is not necessarily accurate since its soft Q-function estimate depends on the environment dynamics and does not strictly satisfy the soft-Bellman equation. Hence, is difficult transfer the recovered reward function to new environments.
3 Problem Formulation
In this section, we consider a ML formulation of the IRL problem and formalize duality relationship with maximum entropy- antropy-based formulation (MaxEnt-IRL).
Maximum Lok--Likelihood IRL (ML-IRL)
model of the expert's behavior randomized policy no(+) where my is a specific policy corresponding to the reward parameter 0. With the state dynamics P( 1|s, az), the discounted
3

---

log-likelihood of observing the expert trajectory J under model To can be written follows:
E E R
log II ( = B
rd log [a)
120
120
By
Σ
log P ((
120
Then we consider the following maximum log-likelihood IRL. formulation:
max L(O) = E
r log To (ar((ss))]
(ML-IRL)
Σ o ((81),0100) + ((m((11)))
8.1 ==
arg max E
(3a)
1=0
where r(8, a; 0) is the reward function and H(T((11)) -.EA ([ass) log ((lss).
We now make some remarks about ML-IRL. First, the problem takes the form of bi-level optimization problem, where the upper-level problem ML-IRLL optimizes the reward parameter 0, while the lower-level problem describes the corresponding policy To as the solution to an entropy-reguuarizee MDP (21 22 In what follows we will leverage recently developed (stochastic) algorithms for bi-level optimization 25), that avoid the high complexity resulted from nested loop algorithms. Second, it is reasonable to use the ML function as the loss, because it searches for a reward function which generates a behavior policy that can best fit the expert demonstrations. While the ML function has been considered in 26 27 nested-loop computations to solve their IRI. formulations, and the theoretical properties are not studied. Finally, the lower-level problem has been well-studied in the literature (2 22
they rely on heuristic algorithms with
23 30 The entropy regularization in (3a) ensures the uniqueness of the optimal policy under the fixed reward function r(s, a; 0) [2]
Even when the underlying MDP high-dimensional and/or complex, the optimal policy could still be obtained; see recent developments in [21 22 We close this section by formally establishing a connection between (MaxEnt IRL) and (L-
Theorem 1. (Strong Duality) Suppose that the reward function is given as: r(s, a; 0) == ó(s, 0. for all s E and a E A. Then (RL- is the Lagrangian dual of (MaxEnt-IRL) Furthermore, (xt) where 0* and π² are the global optimal solutions for problems (ML-IRL) and MaxxEnt IRL respectively.
strong duality holds, that is: L(0")= H
The proof of Theorem T is relegated to Appendix G To our knowledge this result which specifically addresses the (MaxEnt-IRL) formulation is novel. Under finite horizon, duality between ML estimation and maximum causal entropy is obtained in [8 Theorem 3J. However, the problem considered in that paper is not in RL nor IRL setting, therefore they cannot be directly used in the context of the present paper.
The above duality result reveals strong connection between the two formulations under linear reward parameterization. Due to the duality result, we know that (LL-IRL) is a concave problem under linear reward parameterization. In this case, any stationary solution (L- is global optimal estimator of the reward parameter.
The Proposed Algorithm
In this section, we design algorithms for (L- Recall that one major drawback of algorithms for (axXEntIRRL that, they repeatedly solve certain policy optimization problem the inner loop. Even though the recently proposed algorithm IQ-Learn [2] tries to improve the computational efficiency through implicitly representing the reward function and the policy by a Q-function approximator, it has sacrificed the estimation accuracy of the recovered reward. Therefore, one important goal of our design is to find provably efficient algorithms that can avoid nigh-complexity operations and accurately recover the reward function. Specifically, it is desirable that the resulting algorithm only uses a finite number of reward and policy updates to reach certain high-quality solutions.
To proceed, we will leverage the special bi-level structure of the ML-IRL problem. The idea is to alternate between one step of policy update to improve the solution of the lower-level problem, and

---

Algorithm 1 Maximum Likelihood Inverse Reinforcement Learning (ML-IRL)
Input: Initialize reward parameter B, and policy To. Set the reward parameter's is stepsize a.
for = 0, 0.1..... K -1 do
Policy Evaluation: Compute @rag (t..) under reward function r(;:;0,)
Policy Improvement: Tx+1 (-1s) x exp(Qra, (s, -). Vs E S.
Data Sampling I: Sampling an expert trajectory _. == {st. arte20
Data Sampling II: Sampling a trajectory T = {st, at h20 from the current policy Th+1
Estimating Gradient: 9k == h(0g; Tk) 11(00::: T where h(0; = Σ120 7 'Vor(st, (1;
Reward Parameter Update: 0k+1 = 0g + agk
end for
one step the parameter update which improves the upper-level loss function. At each iteration given the current policy Ti and the reward parameter On. new policy T&+1 is generated from the policy improvement step, and 0k+1 generated by the reward optimization step.
This kind of alternating update is efficient, because there is no need to completely solve the policy optimization subproblem, before updating the reward parameters. It has been used in many other RL related settings as well. For example, the well-known actor-critic (AC) algorithm for policy optimization [T 32 23] alternates between one step of policy update, and one step of critic parameter update. Below we present the details of our algorithm at given iteration k.
Policy Improvement Step. Let consider optimizing the lower-level problem, when the reward parameter 0g is held fixed. Towards this end, define the so-called soft Q and soft value functions for given policy we and reward parameter 0g:
n2 (r(s), ar (((T(((15))) su s)
soft Th (s) = E
Σ 1=
(4a)
Qroft (s,a) = r(s, a; 02) +7B.s-((1s..)
V (s)
(4b)
We will adopt the well-known soft policy iteration [21] to optimize the lower-level problem Baa. Under the current reward parameter 8₁ and the policy Th. the soft policy iteration generates new policy k as follows
(a|s) x exp (Qeeft (m (s) As S
(5)
Under a fixed reward function, it can be shown that the new policy T monotonically improves Ti. and it converges linearly to the optimal policy; see 21 Theorem 4] and 28 Thoerem 1].
Note that in practice, we usually do not have direct access to the exact soft Q-function in 4b In
order to perform the policy improvement, a few stochastic update steps in soft Q-learning 21 or
In the appendix, we present Alg. to demonstrate such practical implementation of our proposed algorithm.
soft Actor-Critic (SAC) [22 could used to replace the one-step soft policy iteration
Reward Optimization Step. We propose use stochastic gradient-type algorithm optimize Towards this end, let us first derive the exact gradient V (()). See Appendix D for detailed proof.
Lemma 1. The gradient of the likelihood function L(0) can be expressed as follows:
L = E,
Σ r' ''Vorsss, (s;)) t20
''Vor(s,, (;;
(6)
120
To obtain stochastic estimators of the exact gradient VL(Os), we take two approximation steps: 1) approximate the optimal policy Te, by Tk+1 in Gp. since the optimal policy Te. is not available throughout the algorithm: 2) sample one expert trajectory Th which is already generated by the expert policy F;; 3) sample one trajectory T from the current policy T&+1-
Following the approximation steps mentioned above, we construct a stochastic estimator 9& to approximate the exact gradient (((+)) in O6 as follows:
gk := h(0k; TF) h(0x; TA,, where h(0;T):= Σ a)
5

---

With the stochastic gradient estimator 9k, the reward parameter 2g is updated as:
0k+1 02 + agk-
(8)
where is the stepsize in updating the reward parameter.
In summary, the proposed algorithm for solving the MI.-RRL problem (M..IRI)) is given in Alg.
Theoretical Analysis
In this section, we present finite-time guarantees for the proposed algorithm.
To begin with, first recall that in Sec. have mentioned that (ML-IRL) is bi-level problem, where the upper level (resp. the lower level) problem optimizes the reward parameter (resp. the policy). In order solve (ML-IRL), our algorithm has singe-loop structure, which alternates between one step of policy update and one step of the reward parameter update. Such a single-loop structure indeed has computational benefit, but also leads potential unstableness, since the lower level problem can stay far away from its true solutions. Specifically, at each iteration k. the potential unstableness is induced by the distribution mismatch between the policy TT++1 and Te. when we use estimator Ok to approximate the exact gradient (L(e)) in updating the reward parameter 0g.
Towards stabilizing the algorithm, we adopt the so-called iwo-timescale stochastic approximation (TTSA) approach 33 23 where the lower-level problem updates in a faster time-scale (i.e., converges faster) compared with its upper-level counterpart. Intuitively, the TTSA enables the Th+1 tracks the optimal Te,, leading to stable algorithm. In the proposed Algorithm I the policy (lower-level variable) is continuously updated by the soft policy iteration (5) and it is 'fast" because converges linearly to the optimal policy under fixed reward function 28 Theorem IJ. On the other hand, the reward parameter update (8) does not have such linear convergence property, therefore works in a 'slow' timescale. To begin our analysis, let us first present a few technical assumptions.
Assumption (Ergodicity). For any policy æ,, assume the Markov chain with transition kernel P irreducible and aperiodic under policy т. Then there exist constants K >> 0 and p E (0, 1) such that
sup [[P(st -|s0 s, s, (x(-)l S kpp',
Vt
#ES
where || ||rv is the total variation (TV) norm; P is the stationary state distribution under π.
Assumption T assumes the Markov chain mixes at geometric rate. It is common assumption in the iterature RL 35 32), which holds for any tme-hhomgenneoss Markov chain with finite-state space any uniformly ergodic Markov chain with general state space.
Assumption 2. For any s E S. E and any reward parameter 9, the following holds:
Virr(s, 0) < Lr,
(9a)
Vor(s, a; 01) - Vor(s, 02) - 1.9101 021l
(9b)
where L, and Lg positive constants.
Assumption 2 assumes that the parameterized reward function has bounded gradient and is Lipschitz
smooth. Such assumption in Lipschitz property are common in the literature of min-max / bi-level
optimization 36 2X
K77
Based on Assumptions
we next provide the following Lipschitz properties:
Lemma 2. Suppose Assumptions
2 hold. For any reward parameter 0₃ and 02 the following
results hold:
1 To, (s,a)
Croup Cray may (s, L, 0211.
Vs S
(10a)
[[V](()) - VL S Lel10, 02ll
(10b)
where Quoft (..)) denotes the soft Q-function under the reward function r(:);0) and the policy πe- The positive constants Ly and L. are defined in Appendix B
The Lipschitz properties identified in Lemma are vital for the convergence analysis. Then we present the main results, which show the convergence speed of the policy {ΛT }a20 and the reward parameter (0x 1200 in the Alg. Please see Appendix E for the detailed proof.

---

Theorem 2. Suppose Assumptions y hold. Selecting stepsize := oo for the reward update step K
where og > 0 and a E (0. 1) are some fixed constants, and K is the total number of iterations to be run by the algorithm. Then the following result holds:
K
E
log Th+1 - log mol =
(11a)
K
E
[IVV = ) + +O(K-)
(11b)
where we denote |o xk+1 log Te. Ils == max log ( - log ne, (a)s)]- In particular, setting = 1/2, then both quantities in (Iaa) and l converge with the rate
In Theorem we present the finite-time guarantee for the convergence of the Alg Moreover, as a special case, when the reward is parameterized as a linear function, we know that (ML-IRL) concave and Theorem provides a stronger guarantee which identify the global optimal reward estimator in finite time.
We provide proof sketch below to present the key steps. The detailed proof is in Appendix H
Proof sketch. We outline our main steps in analyzing (Ia and QTP
respectively.
In order to show the convergence of policy estimates in 11 there are several key steps. First, we note that both policies 7k+1 and no, are in the softmax parameterization, where 1++1(-1s) xx
exp (Qra,... (s)-)) and TTg (-Is) x exp (Qth) so, (s,-)). Then, we can show a Lipschitz continuity property between the policy and the soft Q-function:
||oogT*+1 - logte, 10 S 211Qr00
Tág .50, 1000
where the infinty norm || bl is defined over the state-action space S A. Moreover, by analyzing the contraction property the soft policy iteration ( we bound | ro, Quoft Toy Ilso as:
gooft 1 - Qray 1l S x11 rea-1 -oa-
al + 2Lg 8--
To ensure that the error term ||0g 0 is small, we select the stepsize reward parameters as = KF, where K is the total number of iterations and a > 0. Then, by combining previous two steps, we could further show the convergence rate of the policy estimates (a.).
dg
To prove the convergence of the reward parameters (116) we first leverage the Lipschitz smooth property of L(0) in IOOb. However, one technical challenge the convergence analysis is how to handle the bias between the gradient estimator Jb defined in 7 and the exact gradient V (Ox).. When we construct the gradient estimator Ok in 7.. need sample trajectories from the current policy T&+1 and the expert dataset D. However, according to the expression of VL in (6). the trajectories are sampled from the optimal policy тæ, and the dataset D. Hence, there distribution mismatch between T and TØ.. Our key idea is to leverage (a to handle this distribution mismatch error, and thus show that the bias between OA and V (L.)0)) could be controlled.
To the best of our knowledge, Theorem 2 is the first non-asymptotic convergence result for IRI. with nonlinear reward parameterization.
6 A Discussion over State-Only Reward
In this section we consider the IRL problems modeled by using rewards that are only a function of the state. lower dimensional representation of the agent's preferences (i.e. in terms only of states as opposed to states and actions) is more likely to facilitate counterfactual analysis such as predicting the optimal policy under different environment dynamics and/or learning new tasks. This is because the estimation of preferences which are only defined in terms of states is less sensitive to the specific environment dynamics the expert's demonstration dataset. Moreover, in application such as healthcare 39 and autonomous driving 40 where simply imitating the expert policy can potentially result in poor performance, since the learner and the expert may have different transition dynamics. Similar points have also been argued in recent works 14 41
7

---

Next, 1et us briefly discuss how we can understand IL- and Alg when the reward is parameterized as a state-only function. First, it turns out that there is an equivalent formulation of (M.-IRL), when the expert trajectories only contain the visited states.
Lemma 3. Suppose the expert trajectories T is sampled from a policy nB and the reward is parameterized as a state-only function r(8; 0). Then ML.-IRI is equivalent to the following:
min Es
Tu. (5o) - -B (xp-~(t)) (o.n=((0))
(12a)
80
s.t. To == arg max E
Σ r2
(n(s::)+ ((T((115))
(12b)
Please see Appendix F for the detailed derivation. Intuitively, the above lemma says that, when dealing with the state-only IRI., (ML-IRD minimizes the gap between the soft value functions of the optimal policy me and that of the expert policy n². Moreover, [ also be easily implemented with the state-only reward. In fact, the entire algorithm essentially stays the same, and the only change is that r(s, a; 0) will be replaced by r(:; 0). In this way, by only using the visited states the trajectories, one can still compute the stochastic gradient estimator in 7 Therefore, even under the state-only IRI. setting where the expert dataset only contains visited states, our formulation and the proposed algorithm still work if we parameterize the reward as state-only function.
7 Numerical Results
In this section, we test the performance of our algorithm on diverse collection of RL tasks and environments. In each experiment set, we train algorithms until convergence and average the scores of the trajectories over multiple random seeds. The hyperparameter settings and simulation details are provided in Appendix
MuJoCo Tasks For Inverse Reinforcement Learning. In this experiment set, we test the performance of our algorithm on imitating the expert behavior. We consider several high-dimensional roboties control tasks in MuJoCo 44
Two class of existing algorithms are considered as the comparison baselines: 1) imitation learning algorithms that only learn the policy to imitate the expert, including Behavior Cloning (BC) 45 and Generative Adversarial Imitation Learning (GAIL) [[0: 2) IRL algorithms which learn a reward function and a policy simultancously, including Adversarial Inverse Reinforcement Learning (AIRL) (M. f-IRL [4 and IQ-Learn (2]. To ensure fair comparison, all imitation learning / IRL. algorithms use soft Actor-Critic (22) as the base RL algorithm. For the expert dataset, we use the data provided in the official implementation of f-IRL.
In this experiment, we implement two versions of our proposed algorithm: (LtttR((State-AAttion) function of state and action; ML-IRL
where the reward is parameterized utilizes the state-only reward function. In Table [ we present the simulation results under a limited data regime where the expert dataset only contains single expert trajectory. The scores (cumulative rewards) reported in the table is averaged over 6 random seeds. In each random seed, we train algorithm from initialization and collect 20 trajectories to average their cumulative rewards after the algorithms converge. The results reported in Table | show that our proposed algorithms outperform the baselines. The numerical results with confidence intervals are Table 3 (See Appendix).
which
We observe that BC fails to imitate the expert's behavior. It is due to the fact that BC is based on supervised learning and thus could not learn a good policy under such limited data regime. Moreover, we notice the training of IQ-Learn is unstable, which may be due to its inaccurate approximation to the soft Q-function. Therefore, in the MuJoCo tasks where IQ-Learn does not perform well, so that we cannot match the results presented in the original paper 22, we directly report results from there (and mark them by in Table D. The results of AIRL are not presented in Table since it performs poorly even after spending significant efforts in parameter tuning (similar observations have been
made in 46 14)..
Transfer Learning Across Changing Dynamics. We further evaluate IRL algorithms on the transfer learning setting. We follows the environment setup in [II] where two environments with different dynamics are considered: Cus tomttoommaan Ant vs Disabled-Ane Ant. We compare ML-IRLLSttaee-Onyy with
several existing IRL methods: 1) AIRL III, 2) IRL |E] 3) IQ-Learn
8

---

Task
BC
GAIL
IQ-Learn
f-IRL
ML-IRL
ML-IRL
Expert
(State-Only)
(State-Action)
Hopper
20.49
2813.59
2981.01
3074.55
3089.79 79
3121.68
3592.63
Half-Cheetah
-1.87
3301.52
4175.88
4375.88
4472.85
4086.92
5098.3
Walker
-14.01
1112.79
3961.42
4464.20
4380.17
4504.88
5344.21
Ant 760.46
1154.27
4362.90"
4571.71
4675.34
4984.34
5926.18
Humanoid 78.48
3016.40
5227.10*
5243.90
5390.31
5240.57
5351.08
Table
MuJoCo Results. The performance benchmark algorithms under a single expert trajectory.
We consider two transfer learning settings: 1) data transfer; 2) reward transfer. For both settings, the expert dataset trajectories are generated Custom-Ant In the data transfer setting, we train IRL agents in Disabled Ant by using the expert trajectories, which are generated in Custom Ant. In the reward transfer setting, we first use IRI. algorithms to infer the reward functions in Custom-Ant and then transfer these recovered reward functions to Disabled Ant for further evaluation. In both settings, we also train SAC with the ground-truth reward in Disabled-Ar and report the scores.
The numerical results are reoprted in Table 2 the proposed ML-IRL[State-Onj] achieves superior performance compared with the existing IRL benchmarks in both settings. We notice that IQ-Learn fails in both settings since it indirectly recovers the reward function from soft Q-function approximator, which could be inaccurate and is highly dependent upon the environment dynamics. Therefore, the reward function recovered by IQ-Learn can not be disentangled from the expert actions and environment dynamics, which leads to its failures the transfer learning tasks.
Setting
IQ-Learn
AIRL
fIRL ML-IRL(State-Onj) Groud- Truth
Data Transter
-1.788
-5.39
188.85
221.51
320.15
Reward Transfer
-1.04
130.3
156.45
187.69
320.15
Table Transfer Learning, The performance of benchmark algorithms under single expert trajectory. The scores in the table are obtained similarly as in Table
8 Conclusion
In this paper, we present a maximum likelihood IRL formulation and propose a provably efficient algorithm with a single-loop structure. To our knowledge, provide the first non-asymptotic analysis for IRL algorithm under nonlinear reward parameterization. As by-product, when we parameterize the reward state-only function, our algorithm could work in state-only IRL setting and enable reward transfer to new environments with different dynamics. Our algorithm outperforms existing IRL methods on nigh-dimensional robotics control tasks and corresponding transfer learning settings. A limitation of our method is the requirement for online training, one future direction of this work further extend our algorithm and the theoretical analysis to the offline IRL. setting.
Potential Negative Social Impacts
Since IRL methods aim recover the reward function and the associated optimal policy from the observed expert dataset, potential negative social impacts may occur if there are bad demonstrations included in the expert dataset. Thus, for sensitive applications such autonomous driving and clinical decision support, additional care should be taken to avoid negative biases from the expert demonstrations and ensure safe adaptation.
Acknowledgments
We thank the anonymous reviewers for their valuable comments. M. Hong and S. Zeng are partially supported by NSF grants CIF- 1910385, CMMI-1727757 and AFOSR grant 19RT0424. A. Garcia would like to acknowledge partial support from grant FA9550-19-1-000 by AFOSR.

---

References
[1] S. Reddy, A. D. Dragan, and Levine, "SQIL: Imitation learning via reinforcement learning with sparse rewards," in International Conference Learning Representations, 2020. [Online]. Available:
[2] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon, "Iq-learm: Inverse soft-q learning for imitation," Advances in Neural Information Processing Systems, vol. 34, 2021.
[3] T. Osa, J. Pajarinen, G. Neumann, A. Bagnell, P. Abbeel, and J. Peters, An Algorithmic Perspective Imitation Learning, ser. Foundations and Trends in Robotics, 2018, vol. 7.
[4] K. Kim, S. Garg, K. Shiragur, and S. Ermon, "Reward identification in inverse reinforcement learning," in International Conference on Machine Learning. PMLR, 2021, pp. 5496-5505.
[5] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey et al., "Maximum entropy inverse reinforcement learning." " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 8. Chicago, IL, USA, 2008, pp. 1433-1438.
[6] B. D. Ziebart, A. Bagnell, and A. K. Dey, "Modeling interaction via the principle of maximum causal entropy," in International Conference on Machine Learning, 2010.
[7] M. Wulfmeier, P. Ondruska, and I. Posner, "Maximum entropy deep inverse reinforcement learning," " arXiv preprint 11Xi::5507.04886 2015.
[8] C. Finn, P. Christiano, P. Abbeel, and S. Levine, "A connection between generative adversarial networks, inverse reinforcement learning, and energy- models," arXiv preprint 16 1.03852, 2016.
[9] C. Finn, S. Levine, and P. Abbeel, "Guided cost learning: Deep inverse optimal control via policy optimization," in International Conference on Machine Learning. PMLR, 2016, pp. 49-58.
[10] J. Ho and S. Ermon, "Generative adversarial imitation learning." Advances in Neural Information Processing Systems, vol. 29, 2016.
[11] J. Fu, K. Luo, and S. Levine, "Learning robust rewards with adversarial inverse reinforcement learning," " arXiv preprint arXiv:1710.11222 2017.
[12] M. Chen, Y. Wang, T. Liu, Z. Yang, X. Li, Z. Wang, and T. Zhao, "On computation and generalization of generative adversarial imitation learning," arXiv preprint 200122200..02792. 2020.
[13] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Bengio, "Generative adversarial nets," Advances in Neural Information Processing Systems, vol. 27, 2014.
[14] Ni, H. Sikchi, Y. Wang. T. Gupta, L. Lee, and B. Eysenbach, "f-irl: Inverse reinforcement learning via state marginal matching," arXiv preprint arXiv:2011.04779, 2020.
[15] K. Kurach, M. Lucic, X. Zhai, M. Michalski, and S. Gelly, "The gan landscape: Losses, architectures, regularization, and normalization." 2018.
[16] Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and Tompson, "Diseriminator-acttrriiti:: Addressing sample inefficiency and reward bias in adversarial imitation learning," in International Conference on Learning Representations, 2019. [Online]. Available: http://rwww.ieteeeooooooaaaa/a/d
[17] E. T. Jaynes, "Information theory and statistical mechanics," Physical review, vol. 106, p. 620, 1957.
[18] B. D. Ziebart, J. A. Bagnell, and A. K. Dey, "The principle of maximum causal entropy for estimating interacting processes," IEEE Transactions on Information Theory, vol. 59, no. 4, pp. 1966-1980, 2013.
10

---

[19] M. Bloem and N. Bambos, "Infinite time horizon maximum causal entropy inverse reinforcement learning, " in 53rd IEEE conference on decision and control. IEEE, 2014, pp. 491 I 1-4916.
[20] Z. Zhou, M. Bloem, and N. Bambos, "Infinite time horizon maximum causal entropy inverse reinforcement learning," JEEE Transactions on Automatic Control, vol. 63, no. 9, pp. 2787-2802, 2017.
[21] T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, "Reinforcement learning with deep energy- based policies," International Conference on Machine Learning. PMLR, 2017, pp. 1352-1361.
[22] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off- Off-policy maximum entropy deep reinforcement learning with a stochastic actor," International Conference on Machine Learning. PMLR, 2018, pp. 1861-1870.
[23] M. Hong, H.-I. Wai, Z. Wang, and Z. Yang, "A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic," arXiv preprint arXiv:2007.05170, 2020.
[24] K. Ji, J. Yang, and Liang. "Bilevel optimization: Convergence analysis and enhanced design." "
in International Conference on Machine Learning. PMLR, 2021, pp. 4882-4892.
[25] Khanduri, Zeng, M. Hong, H.-T. Wai, Z. Wang, and Z. Yang, "A near-optimal algorithm for stochastic bilevel optimization via double-momentum." Advances in Neural Information Processing Systems, vol. 34, 2021.
[26] V. Jain, P. Doshi, and B. Banerjee, "Model-free irl using maximum likelihood estimation," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp- 3951-3958.
[27] N. Sanghvi, S. Usami, M. Sharma, J. Groeger, and K. Kitani, "Inverse reinforcement learning with explicit policy estimates," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 11, 2021, pp. 9472-9480.
[28] S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi, "Fast global convergence of natural policy gradient methods with entropy regularization." Operations Research, 2021.
[29] S. Cayei, N. He, and Srikant, "Linear convergence of entropy-regullriieed natural policy gradient with linear function approximation." arXiv preprint arXiv:2106.04090 2021.
[30] O. Nachum, M. Norouzi, K. Xu, and D. Schuurmans, "Bridging the gap between value and policy based reinforcement learning," Advances in Neural Information Processing Systems, vol. 30, 2017.
[31] V. Konda and J. Tsitsiklis, "Actor-critic algorithms," Advances in Neural Information Processing Systems, vol. 12, 1999.
[32] Y. F. Wu, W. Zhang, P. Xu, and Q. Gu, "A finite-time analysis of two time-scale actor-critic methods," Advances in Neural Information Processing Systems, vol. 33, pp. 17 67611717766 628, 2020.
[33] V. S. Borkar, "Stochastic approximation with two time scales," Systems & Control Letters, vol. 29, no. 5, pp. 291-294, 1997.
[34] J. Bhandari, D. Russo, and R. Singal, "A finite time analysis of temporal difference learning with linear function approximation." in Conference on learning theory. PMLR, 2018, pp. 1691-1692.
[35] S. Zou, T. Xu, and Y. Liang, "Finite- sample analysis for sarsa with linear function approximation," Advances in Neural Information Processing Systems, vol. 32, 2019.
[36] C. Jin, P. Netrapalli, and M. Jordan, "What is local optimality in nonconvex-nonconces minimax optimization?" International Conference on Machine Learning. PMLR 2020, pp- 4880-4889.
11

---

[37] Z. Guan, T. Xu, and Y. Liang, "When will generative adversarial imitation learning algorithms attain global convergence." in International Conference on Artificial Intelligence and Statistics. PMLR, 2021, pp. 1117-1125.
[38] T. Chen, Y. Sun, and W. Yin, "Closing the gap: Tighter analysis of alternating stochastic gradient methods for bilevel problems," Advances in Neural Information Processing Systems, vol. 34, pp. 25 212-99-22 25 307, 2021.
[39] C. Yu, J. Liu, S. Nemati, and G. Yin, "Reinforcement learning in healthcare: A survey," ACM Computing Surveys (CSUR), vol. 55, no. pp. 1-36, 2021.
[40] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez, "Deep reinforcement learning for autonomous driving: A survey," IEEE Transactions on Intelligent Transportation Systems, 2021.
[41] T. Gangwani and J. Peng, "State-only imitation with transition dynamics mismatch," in International Conference on Learning Representations, 2020. [Online]. Available:
[42] L. Viano, Y.-T. Huang, P. Kamalaruban, A. Weller, and V. Cevher, "Robust inverse reinforcement learning under transition dynamics mismatch," Advances in Neural Information Processing Systems, vol. 34, 2021.
[43] F. Torabi, G. Warnell, and P. Stone, "Generative adversarial imitation from observation," arXiv preprint arXiv: 2018.
[44] E. Todorov, T. Erez, and Y. Tassa, "Mujoco: physics engine for model-based control," in 2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5033.
[45] D. A. Pomerleau, "ALVINN: An autonomous land vehicle neural network," Advances in Neural Information Processing Systems, vol. 1, 1988.
[46] F. Liu, Z. Ling, T. Mu, and H. Su, "State aligmment-based imitation learning." in International Conference on Learning Representations, 2020. [Online]. Available: https:/opennevvew.ettffommm www..////foumm/dddryrrrd.iir
[47] A. Kamoutsi, G. Banjac, and J. Lygeros, "Efficient performance bounds for primal-dum reinforcement learning from demonstrations," in International Conference on Machine Learning. PMLR, 2021, pp. 5257-5268.
[48] H. Cao, S. Cohen, and L. Szpruch, "Identifiability in inverse reinforcement learning," Advances in Neural Information Processing Systems, vol. 34, 2021.
[49] R. Wang, C. Ciliberto, V. Amadori, and Y. Demiris, "Random expert distillation: Imitation learning via expert policy support estimation," in International Conference on Machine Learning. PMLR, 2019, pp. 6536-6544.
[50] D. S. Brown, W. Goo, and S. Niekum, "Better-than-demmonrrraa imitation learning via automatically-ranked demonstrations," in Conference on robot learning. PMLR, 2020, pp. 330-359.
[51] P. Barde, J. Roy, Jeon, J. Pineau, C. Pal, and D. Nowrouzezahrai, "Adversarial soft advantage fitting: Imitation learning without policy optimization," Advances in Neural Information Processing Systems, vol. 33, pp. 12334-12 344, 2020.
[52] H. Kretzschmar, M. Spies, C. Sprunk, and W. Burgard, "Socially compliant mobile robot navigation via inverse reinforcement learning," The International Journal of Robotics Research, vol. 35, no. 11, pp. 1289-1300 2016.
[53] C. Xia and A. El Kamel, "Neural inverse reinforcement learning in autonomous navigation." Robotics and Autonomous Systems, vol. 84, pp. 1-14, 2016.
[54] Rust and C. Phelan, "How social security and medicare affect retirement behavior in world of incomplete markets." Econometrica: Journal of the Econometric Society, pp. 781-831, 1997.
12

---

[55] Z. Eckstein and K. I. Wolpin, "Why youths drop out of high school: The impact of preferences, opportunities, and abilities," Econometrica, vol. 67, no. pp. 1295-1339, 1999.
[56] E. Duflo, R. Hanna, and S. P. Ryan, "Incentives work: Getting teachers to come to school," American Economic Review, vol. 102, no. 4, pp. 1241-78, 2012.
[57] H. Fang and Y. Wang, "Estimating dynamic discrete choice models with hyperbolic discounting, with an application mammography decisions," International Economic Review, vol. 56, no. pp. 1655596,99 2015.
[58] L. Caliendo, M. Dvorkin, and F. Parro, "Trade and labor market dynamics: General equilibrium analysis of the china trade shock," Econometrica, vol. 87, no. 3. pp. 741-835, 2019.
[59] C. Cirillo, R. Xu, and F. Bastin, "A dynamic formulation for car ownership modeling," Transportation Science, vol. 50, no. pp. 322-335, 2016.
[60] T. Xu, Z. Wang, and Y. Liang, "Improving sample complexity bounds for (natural) actor-critic algorithms," Advances in Neural Information Processing Systems, vol. 33, pp. 4358-4369, 2020.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes]. [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curatinn contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "h_dtOBufpk", "reviewer": "Reviewer_6Mga", "summary": "The paper formalizes IRL as a entropy-regularized maximum likelihood problem. The authors show that this problem is dual to maximum entropy IRL, and then formulates this problem as a bi-level optimization. They propose an algorithm that iterates between single steps of evaluation and policy improvement (in contrast to most max entropy IRL methods that have to solve RL problems in an inner loop). For this algorithm, the authors provide a _finite time_ convergence guarantee under some regularity conditions. Finally, they provide an empirical evaluation on standard MuJoCo task, demonstrating that their algorithm outperforms alternative IRL approaches.", "questions": "Can you provide standard errors for the results in Table 1?\n\nWhy does the state-action version of the algorithm perform worse than the state-only algorithm in the half-cheetah environment?\n\n\n**Typos:**\n - \"inherent nested\" -> \"inherently nested\" (line 4)\n- \"Mujoco\" -> \"MuJoCo\" (line 19)\n- \"computational efficient\" -> \"computationally efficient\" (line 53)", "limitations": "I did not find any discussion of potential broader impact of this work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper provides the first finite-time convergence analysis without assuming linear rewards, as far as I can tell. The assumptions of the main theoretical results seem pretty reasonable; this is a solid theoretical contribution.\n- The duality between MaxEnt IRL and the proposed maximum likelihood formulation might be interesting to the IRL community and could motivate interesting follow-up work.\n- The authors provide a good experimental evaluation that covers most relevant baselines and shows that their algorithm can perform well empirically.\n- The authors discuss the novelty of their work and provide particularly appropriate comparisons to results in prior work at multiple point, e.g., for their ML formulation, the proposed algorithm, and the main theoretical result.\n\n\n**Weaknesses**\n- The duality with MaxEnt IRL is not particularly novel and has been observed before in other context (as noted by the authors).\n- The results in Table 1 don't show very large differences, and it is hard to evaluate if these are significant differences, given that only 3 random seeds were run.\n- The experimental evaluation is limited to MuJoCo locomotion tasks which can sometimes be too simple to draw strong conclusions.\n- Overall, the writing could be clearer at times, and the structure of the paper could be improved. For example, the Introduction spends a lot of time discussion prior work before stating the contributions of this paper, which is sub-optimal.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KCS-jU1EG8G", "reviewer": "Reviewer_fBA5", "summary": "The authors propose a maximum likelihood approach for IRL. Theoretical results show that the algorithm converges in finite time and that for linear rewards that the ML-IRL problem is dual with MaxEnt-IRL and that there is strong duality. Empirical results show that ML-IRL outperforms state-of-the-art IRL algorithms across several standard benchmarks.", "questions": "The authors are missing related work that also seeks to make IRL more tractable.\n\nWang, Ruohan, et al. \"Random expert distillation: Imitation learning via expert policy support estimation.\" International Conference on Machine Learning. PMLR, 2019.\nand\nBrown, Daniel S., Wonjoon Goo, and Scott Niekum. \"Better-than-demonstrator imitation learning via automatically-ranked demonstrations.\" Conference on robot learning. PMLR, 2020.\nPropose approaches that do not require interleaving reward learning and policy learning.\nRecent work\nBarde, Paul, et al. \"Adversarial soft advantage fitting: Imitation learning without policy optimization.\" Advances in Neural Information Processing Systems 33 (2020): 12334-12344.\nproposes an adversarial IL approach that does not require any RL.\n\nOne of the stated contributions is to implement  single-loop approach. This doesn't seem like a contribution since most Adversarial IRL methods do the same thing. This idea is well studied, going back to GAIL and Guided cost learning, and probably earlier. I do not think the authors can claim this as novel. Furthermore, the two-timescale approach is common in GANs and GAIL-like algorithms. I don't think this can be claimed as a contribution. \n\nEquation 3: The theta parameterizes r which informs argmax to get pi. This is confusing with line 119 where the policy is parameterized by theta. Policies are not usually parameterized by rewards.\n\nLine 155: is supposed to be an example of solving a hard optimization problem in the inner loop but it refers to IQ-Learn which does was previously noted to be computationally efficient because it avoids this inner loop mdp solver.\n\nLemma 1: How is his different from the guided cost learning update. In guided cost learning the partition function is estimated with onpolicy rollouts so it appears quite similar.\n\nLine 188: Why only sample one demo and one policy rollout? Doesn't this have very high variance?\n\nAllowing the algorithm to work with state only demonstrations is nice, but many other algorithms already do this. E.g. \nBrown, Daniel S., Wonjoon Goo, and Scott Niekum. \"Better-than-demonstrator imitation learning via automatically-ranked demonstrations.\" Conference on robot learning. PMLR, 2020.\nTorabi, Faraz, Garrett Warnell, and Peter Stone. \"Generative adversarial imitation from observation.\" arXiv preprint arXiv:1807.06158 (2018).\nAs well as all of the classical IRL algorithms. \nIt would be good to motivate more why this is a contribution.\n", "limitations": "A discussion of limitations are lacking in the main body of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+The strong duality theorem appears novel and draws an interesting connection between ML-IRL and MaxEnt-IRL\n\n+The authors prove the first finite guarantees for IRL with nonlinear reward functions\n\n+The empirical results are promising and improve upon SOA\n\n-Some of the claims of novelty seem to be standard practice and are used in prior work.\n\n-It is unclear why an ML approach to IRL should perform better than other approaches. The authors have nice theory, but in practice the implemented algorithm seems very similar to prior work.\n\n-It is unclear how statistically significant the empirical results are with only 3 seeds and no confidence intervals.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EmashNvo1_X", "reviewer": "Reviewer_wFjp", "summary": "In this paper, the authors proposed a new formulation of IRL based on maximum likelihood, which is equivalent to MaxEnt when the reward function is linear.  By leveraging this formulation, the authors propose a new computationally efficient gradient-based iterative algorithm that does not require solving MDP in each iteration. The authors further prove a non-asymptotically rate of how fast the algorithm converges to a stationary point, which is the first non-asymptotic convergence result for ILR with nonlinear reward parameterization. Finally, the authors conduct extensive experiments that show that the proposed algorithm outperforms several state-of-the-art IRL algorithms. ", "questions": "In the algorithm, it’s assumed that the soft-Q function can be accurately estimated. Can the approximation  in soft-Q function be considered in the theorem as well?\n\nInterestingly, the gradient in Eq.(6) is the same as the gradient to r with fixed pi in Eq. (2) (MaxEnt). And the inner optimization in Eq. (2) is finding the optimal policy given r which is the same as solving Eq. (3a). Therefore, the proposed algorithm can be considered as solving Eq. (2) directly without the ML-IRL formulation. Would the authors provide some explanation on why formulating IRL in ML-IRL is fundamental to the convergence proof?\n\nIn the text the authors try to distinguish between MaxEnt IRL and adversarial IRL (e.g. Line 30 - 44). I wonder why that's the case as these two have been shown to be closely related [6, 7].\n\n[6] Finn, Chelsea, et al. \"A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models.\" arXiv preprint arXiv:1611.03852 (2016).\n\n[7] Swamy, Gokul, et al. \"Of moments and matching: A game-theoretic framework for closing the imitation gap.\" International Conference on Machine Learning. PMLR, 2021.\n", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength: \n\nThe authors claim that Thm 2 is the first non-asymptotic convergence result for ILR with nonlinear reward parameterization. I believe this is true as a recent paper [1] from ICML 2021 proves a non-asymptotic rate for a gradient-based method with linear reward parameterization (although that paper’s proof does not seem to have the assumption the value function can be accurately estimated).\n \n### Weakness: \n I think the contribution above is important to the IRL community.  However, other contributions of this paper may not be as significant as this one. \n\n1. The new formulation of IRL. This maximum likelihood formulation is known [2]. The proof seems to be similar to Thm 3 in [3]. \n2. The algorithm. The authors mentioned that the algorithm enjoys computational efficiency and is capable of reward transferring. However, GAIL-type of algorithms that use state-dependent rewards in the discriminator [4] and scalable MaxEnt IRL algorithms [5] are able to achieve those benefits as well. \n3. The experiments. I like the experiments on reward transferring. However, the main goal of proposing the new algorithm and theory is to reduce computation burden, but the experiments focus on the quality of the learned policy / reward after convergence. It’s unclear from the experiments whether the new algorithm is indeed faster. For example, an interesting experiment would be a comparison with MaxEnt IRL as in theory the proposed method is able to converge to a stationary point that’s the same as MaxEnt IRL, but faster.\n\nFurthermore, the clarity of the paper can be improved. Line 118 to 135 is rather confusing and only until the second reading I was able to understand the math. In Line 119,\\pi_\\theta is parameterized by \\theta. But in Line 123, \\theta is used to parameterize the reward function. \n\n[1] Kamoutsi, Angeliki, Goran Banjac, and John Lygeros. \"Efficient Performance Bounds for Primal-Dual Reinforcement Learning from Demonstrations.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Jain, Vinamra, Prashant Doshi, and Bikramjit Banerjee. \"Model-free IRL using maximum likelihood estimation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.\n\n[3] Ziebart, Brian D., J. Andrew Bagnell, and Anind K. Dey. \"The principle of maximum causal entropy for estimating interacting processes.\" IEEE Transactions on Information Theory 59.4 (2013): 1966-1980.\n\n[4] Torabi, Faraz, Garrett Warnell, and Peter Stone. \"Generative adversarial imitation from observation.\" arXiv preprint arXiv:1807.06158 (2018).\n\n[5] Finn, Chelsea, Sergey Levine, and Pieter Abbeel. \"Guided cost learning: Deep inverse optimal control via policy optimization.\" International conference on machine learning. PMLR, 2016.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1LSosV6dtK", "reviewer": "Reviewer_pJ4m", "summary": "This paper proposes a novel single-loop IRL algorithm to reduce the computational burden of the previous IRL framework. The authors prove that the solution of the proposed framework is equivalent to the maximum entropy IRL framework with a  linearly parameterized reward function. They also provide a convergence guarantee of the proposed algorithm. The experiment result seems promising. ", "questions": "- Could the authors discuss the MaxEnt-IRL literature?\n- Could the authors explain why the ML-IRL framework works for the transfer learning tasks?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper is well written. \n- The problem studied in this paper is important and the proposed method is novel.\n- The authors provide theoretical guarantees for the proposed framework.\n\nWeaknesses:\n\n- There may need more literature review on the MaxEnt-IRL.\n- Although the ML-IRL framework could reduce the computational complexity, it is not obvious why the proposed ML-IRL framework works for the transfer learning tasks.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept