Title: Inverse Reinforcement Learning with the Average Reward Criterion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Machine Learning

Reinforcement Learning

Inverse Reinforcement Learning

Markov Decision Process

stochastic optimization

complexity analysis
Abstract: We study the problem of Inverse Reinforcement Learning (IRL) with an average-reward criterion. The goal is to recover an unknown policy and a reward function when the agent only has samples of states and actions from an experienced agent. Previous IRL methods assume that the expert is trained in a discounted environment, and the discount factor is known. This work alleviates this assumption by proposing an average-reward framework with efficient learning algorithms. We develop novel stochastic first-order methods to solve the IRL problem under the average-reward setting, which requires solving an Average-reward Markov Decision Process (AMDP) as a subproblem. To solve the subproblem, we develop a Stochastic Policy Mirror Descent (SPMD) method under general state and action spaces that needs $\mathcal{O}(1/\varepsilon)$ steps of gradient computation. Equipped with SPMD, we propose the Inverse Policy Mirror Descent (IPMD) method for solving the IRL problem with a $\mathcal{O}(1/\varepsilon^2)$ complexity. To the best of our knowledge, the aforementioned complexity results are new in IRL with the average reward criterion. Finally, we corroborate our analysis with numerical experiments using the MuJoCo benchmark and additional control tasks.

Inverse Reinforcement Learning with the Average Reward Criterion

Abstract

We study the problem of Inverse Reinforcement Learning (IRL) with an averagereward criterion. The goal is to recover an unknown policy and a reward function when the agent only has samples of states and actions from an experienced agent. Previous IRL methods assume that the expert is trained in a discounted environment, and the discount factor is known. This work alleviates this assumption by proposing an average-reward framework with efficient learning algorithms. We develop novel stochastic first-order methods to solve the IRL problem under the averagereward setting, which requires solving an Average-reward Markov Decision Process (AMDP) as a subproblem. To solve the subproblem, we develop a Stochastic Policy Mirror Descent (SPMD) method under general state and action spaces that needs O (1 / ε ) steps of gradient computation. Equipped with SPMD, we propose the Inverse Policy Mirror Descent (IPMD) method for solving the IRL problem with a O (1 / ε 2 ) complexity. To the best of our knowledge, the aforementioned complexity results are new in IRL literature with the average reward criterion. Finally, we corroborate our analysis with numerical experiments using the MuJoCo benchmark and additional control tasks.

1 Introduction

Reinforcement Learning (RL) problems are frequently formulated as Markov Decision Processes (MDPs). The agent learns a policy to maximize the reward gained over time. However, in numerous engineering challenges, we are typically presented with a set of state-action samples from experienced agents, or experts , without an explicit reward signal. Inverse Reinforcement Learning (IRL) aspires to recover the expert's policy and reward function from these collected samples.

Methods like Imitation Learning [29, 15, 10] reduces the IRL problem to predicting an expert's behavior without estimating the reward signal. Such formulation is undesirable in some scenarios when we either wish to continue training or analyze the structure of the agent's reward signal, e.g., in reward discovery for animal behavior study [31, 34, 26, 14]. Furthermore, as a popular methodology in Imitation Learning, Generative Adversarial Network (GAN) [15, 42] suffers from unstable training due to mode collapse [32]. The sampling-based Bayesian IRL (BIRL) approach [8, 9, 6] treats the reward function as a posterior distribution from the observation. This approach is conceptually straightforward but suffers from slow convergence of sampling. Another line of research reformulates

the IRL as RL with general utility functions [36, 39, 12, 1]. While theoretically attractive, such a formulation does not learn reward functions.

Furthermore, inferring rewards poses an additional challenge: without any additional assumption, the same expert behavior can be explained by multiple reward signals [40]. To resolve this ambiguity, the authors propose adding the principle of Maximum Entropy that favors a solution with a higher likelihood of the trajectory. In the following work [41], the objective is replaced by maximizing the entropy of the policy, which better characterizes the nature of stochastic sequential decision-making. A body of work focuses on alleviating the computational burden of the nested optimization structure of under the Maximum Entropy framework. In [11], the authors skipped the reward function update stage by representing rewards using action-value functions. f -IRL [23] minimize the divergence of state visitation distribution for finite-horizon MDPs. More recently, in [37], the authors propose a dual formulation of the original Maximum Entropy IRL and a corresponding algorithm to solve IRL in the discounted setting.

Despite the success in practice, theoretical understanding of RL/IRL methods is lacking. For RL, specifically for Discounted Markov Decision Processes (DMDPs), analysis of policy gradient methods for solving DMDPs is catching up only until recently [2, 16, 5, 19, 18]. Meanwhile, understanding of the Average-reward Markov Decision Processes (AMDPs) remains very limited, where the policy aims to maximize the long-run average reward gained. Li et al.[20] propose stochastic first-order methods for solving AMDPs with a rate of convergence ˜ O (log( ε -1 )) for policy optimization, but it is constrained to finite states and actions with policy explicitly represented in tabular forms. Moreover, for IRL, the only known finite time analysis are [35, 37, 38, 21] for either solving DMDPs, or finite-horizon MDPs. There are no well-established convergence analyses for IRL with AMDPs. Moreover, prior works only focus on MDPs with finite state and action spaces, which do not fully address many real-life problems that inherently feature continuous elements (e.g., robotics [17]). Such settings further require general function approximations, the usage of which lacks theoretical understanding in recent work as well.

In summary, all previous works either do not fundamentally address our problem, which necessitates the learning of reward functions, or they miss key elements such as general state and action spaces, general function approximation, and the analysis incorporating approximation errors. Each of these facets demands substantial effort. In this paper, we focus on convergence analyses for average-award MDPs. We extend the RL algorithm [20] for AMDPs to general state and action spaces with general function approximation and develop an algorithm to solve the IRL problem with the average-reward criterion. Analyses of both algorithms are new in the literature to the best of our knowledge.

1.1 Main contributions

Stochastic Policy Mirror Descent for solving AMDPs : Our research introduces the Stochastic Policy Mirror Descent (SPMD) algorithm, a novel approach designed to tackle Average-reward Markov Decision Processes (AMDPs) that involve general state and action spaces. Leveraging a performance difference lemma, we demonstrate that the SPMD algorithm converges within O ( ε -1 ) steps for a specific nonlinear function approximation class, and in no more than O ( ε -2 ) steps for general function approximation.

Inverse Policy Mirror Descent for solving Maximum Entropy IRL : We expound a dual form of the Maximum Entropy IRL problem under the average-reward framework with the principle of Maximum Entropy. This dual problem explicitly targets the minimization of discrepancies between the expert's and agent's expected average reward. Consequently, we propose a first-order method named Inverse Policy Mirror Descent (IPMD) for effectively addressing the dual problem. This algorithm operates by partially resolving an Entropy Regularized RL problem at each iteration, which is systematically solved by employing SPMD. Drawing upon the two-timescale stochastic approximation analysis framework [3], we present the convergence result and establish a O ( ε -2 ) rate of convergence.

Numerical experiments : Our RL and IRL methodologies have been tested against the well-known robotics manipulation benchmark, MuJoCo, as a means to substantiate our theoretical analysis. The results indicate that the proposed SPMD and IPMD algorithms generally outperform state-of-the-art algorithms. In addition, we found that the IPMD algorithm notably reduces the error in recovering the reward function.

2 Background and problem setting

2.1 Average reward Markov decision processes

An Average-reward Markov decision process is described by a tuple M := ( S , A , P , c ) , where S denotes the state space, A denotes the action space, P is the transition kernel, and c is the cost function. At each time step, the agent takes action a ∈ A at the current state s ∈ S according to its policy π : S → A . We use Π as the space of all feasible policies. Then the agent moves to the next state s ′ ∈ S with probability P ( s ′ | s, a ) , while the agent receives an instantaneous cost c ( s, a ) (or a reward r ( s, a ) = -c ( s, a ) ). The agent's goal is to determine a policy that minimizes the long-term cost where h π is a closed convex function with respect to the policy π , i.e., there exists µ h ≥ 0 such that

where ∇ h π ′ denotes the subgradient of h at π ′ and D ( · , · ) is the Bregman distance, i.e.,

Here ω : A → R is a strongly convex function with the associated norm ‖ · ‖ in the action space A and let us denote ‖ · ‖ ∗ as its dual norm. In this work we also utilize a span semi-norm [27]: ‖ v ‖ sp, ∞ := max v -min v, ∀ v ∈ R n . If h π = 0 , Eq. 1 reduces to the classical unregularized AMDP. If h π ( s ) = E π [ -log π ] =: H ( π ( s )) , i.e., the (differential) entropy, Eq. 1 defines the average reward of the so-called entropy-regularized MDPs.

In this work, we consider the ergodic setting, for which we make the following assumption formally:

Assumption 2.1. For any feasible policy π , the Markov chain induced by policy π is ergodic. The Markov chain is Harris ergodic in general state and action spaces (see [22]). The stationary distribution κ π induced by any feasible policy exists and is unique. There is some constant number 0 < Γ < 1 such that κ π ( s ) ≥ 1 -Γ .

As a result of Assumption 2.1, for any feasible policy π , the average-reward function does not depend on the initial state (see Section 8 of [27]). Given that, one can view ρ π as a function of π . For a given policy we also define the basic differential value function (also called bias function ; see, e.g., [27])

and the basic differential action-value function (or basic differential Q-function) is defined as

Moreover, we define the sets of differential value functions and differential action-value functions (differential Q -functions) as the solution sets of the following Bellman equations, respectively,

Under Assumption 2.1, the solution of Eq. (6) (resp., Eq. (7)) is unique up to an additive constant. Finally, our goal in solving an AMDP is to find an optimal policy π ∗ that minimizes the average cost:

2.2 Inverse Reinforcement Learning

Suppose there is a near-optimal policy, and we are given its demonstrations ζ := { ( s i , a i ) } i ≥ 1 . IRL aims to recover a reward function such that the estimated reward best explains the demonstrations. We consider solving the IRL problem under the Maximum Entropy framework (MaxEnt-IRL), which

aims to find a reward representation that maximizes the entropy of the corresponding policy and incorporates feature matching as a constraint. Formally, the MaxEnt-IRL problem is described as

where d E and d π denote the state-action distribution induced by the expert and current policy π E and π respectively, and ϕ ( s, a ) ∈ R n denotes the feature of a given ( s, a ) pair. If we assume that for a given state-action pair, the cost is a linear function to its feature, i.e., c ( s, a ; θ ) = θ T ϕ ( s, a ) for some parameter θ with the same dimension of the feature ϕ ( s, a ) , we can show that the parameter θ is the dual multiplier of the above optimization problem, as the Lagrangian function can be written as

Therefore, the dual problem is formulated as

Notice that to find π , we need to solve an Entropy Regularized Reinforcement Learning problem, i.e., solving AMDP with the regularizer set to be the negative entropy, h π = -H ( π ) . To this end, we first propose a Stochastic Policy Mirror Descent (SPMD) method for solving AMDP. The solution methods are presented in section 3. Then we introduce an Inverse Policy Mirror Descent (IPMD) algorithm based on SPMD for the inverse RL problem (Dual IRL), introduced in section 4. We will see that we only need to solve the subproblem AMDP partially.

3 Stochastic Policy Mirror Descent for AMDPs

This section proposes an SPMD method to solve Regularized Reinforcement Learning problems for AMDPs. SPMD operates in an actor-critic fashion. In each step, the agent evaluates its current policy (critic step) and performs a policy optimization step (actor step). In this work, we assume there is a way to perform the critic step using standard methods, e.g., Temporal Difference learning using neural networks. Further discussion on implementation is included in section 3.1. We will focus on designing a novel actor step and providing its complexity analysis.

Our policy optimization algorithm is motivated by the following performance difference lemma , which characterizes the difference in objective values of two policies π , π ′ .

Lemma 3.1. (Performance Difference) Assume that Assumption 2.1 holds. For any π , π ′ ∈ Π

where

Proof can be found in A.1. The above lemma shows the gradient of the objective function with respect to action a is ¯ Q π ( s, π ( s )) + h π ( s ) , i.e.,

The existence of the above equation requires the differentiability of ¯ Q , h and locally Lipschitz continuity of κ π [see 24, 25]. Note that ψ can be seen as some generalized advantage function. Lemma 3.1 shows that the gradient of the objective relates to differential Q-functions . This inspires us to update the policy in the following mirror-descent style:

where η k is some predefined step size. The complexity analysis of this type of algorithm has been thoroughly studied in [20] in tabular forms or linear function approximation settings, i.e., a lookup

table represents the policy, and the differential Q-functions are approximated as linear functions with respect to some feature space. In practice, especially in the Deep Reinforcement Learning community, both policy and the Q -functions are represented by neural networks. Thus, novel analysis is required for general function approximation. Additionally, the exact value of ¯ Q π and ∇ ω ( π ) in Eq. 19 can only be estimated through approximation. Note that ∇ ω ( π ) rises inside the Bregman distance D ( π k ( s ) , a ) , referring to Eq. 13. We consider their stochastic estimators calculated from state-action sample pairs ζ , denoted as ¯ Q π , ζ ( s, a ; φ ) , ˜ ∇ ω ( π ( s ); φ ) respectively, where φ denotes the parameters of the method of choice, for example, weights and biases in a neural network. For the rest of the paper, we abbreviate ¯ Q π , ζ ( s, a ; φ ) as ¯ Q ( s, a ; φ ) when the context is clear. We described the Stochastic Policy Mirror Descent (SPMD) algorithm in Algorithm 1.

Algorithm 1: The Stochastic Policy Mirror Descent (SPMD) algorithm for AMDPs

1: Input : Initialize random policy π 0 and step size sequence { η k }

3: Sample collection: ζ k = { ( s t , a t , c t ) } t ≥ 1

2: for k = 0 , 1 , · · · , K do

4: Critic step : Approximate ¯ Q π k , ∇ ω ( π k ( s )) with

5: Actor step : Update the policy

6: end for

The following paragraphs present the complexity analysis for Algorithm 1. Note that we assume the approximated Q function might not be convex but weakly convex, i.e.,

is convex w.r.t. a ∈ A for some µ ¯ Q ≥ 0 and π 0 is the initialized policy. This assumption is general as any differential function with Lipschitz continuous gradients is weakly convex. Moreover, we assume that h · ( s ) , ¯ Q ( s, · ; θ ) and ¯ Q π ( s, · ) are Lipschitz continuous with respect to the action.

Assumption 3.2. For all a 1 , a 2 ∈ A there exist some constants M h , M ¯ Q , M ¯ Q such that

and ¯ Q ( s, a ; φ ) is µ ¯ Q -weakly convex, i.e., Eq. 16 is convex.

Note that the strong convexity modulus of the objective function in Eq. 15 can be very large since η k can be small, in which case the subproblem Eq. 15 is strongly convex, thus the solution of Eq. 15 is unique due to strong convexity [18].

Our complexity analysis follows the style of convex optimization. We begin the analysis by decomposing the approximation error in the following way

We refer to the first two and last two approximation error terms in δ ω as δ ω ,det k and δ ω ,sto k , respectively, as they represent a deterministic approximation error which we cannot reduce and a stochastic error related to the variance of the estimator. They represent the noise introduced in the critic step. To make

the noise tractable and the entire algorithm convergent, we must assume they are not arbitrarily large. But before making more assumptions about these error terms, we must distinguish the case when ¯ Q + h is convex and when it's not, i.e., when µ ≥ 0 or µ < 0 , where µ = µ h -µ ¯ Q for simplicity. We can obtain globally optimal solutions if µ ≥ 0 . While in the other case, only stationary points can be obtained, e.g., both ψ π ( s, π ′ ( s )) and D ( π k , π k +1 ) approach 0 . Note that ψ relates to the progress we make in each iteration. See A.6 for more detail. These two cases require different assumptions on the error terms, so they must be treated differently. We start with the optimistic case when µ ≥ 0 , under which we make the following assumption about the approximation error δ ¯ Q k ( s, a ) and δ ω k ( s ) : Assumption 3.3. When µ ≥ 0 , the critic step has bounded errors, i.e., there exist some constants ς ¯ Q , ς ω , σ ω such that

where ζ k is the samples we collect in each iteration.

With the above assumption, we present the complexity result of SPMD when µ ≥ 0 .

Theorem 3.4. Suppose 2.1, 3.2, 3.3 hold and the step size η k satisfies

for some β k ≥ 0 . Then after running the SPMD algorithm for K iterations, we have

where ¯ D A := max a 1 ,a 2 ∈ A D ( a 1 , a 2 ) .

Proof can be found in A.4. Equation 25 encapsulates a convergence overview without specifying explicit step size choices. To be specific, α k , β k are both step sizes, and the central term ∑ ρ k -ρ ∗ (distance to the optimal value) plus D ( π k , π ∗ ) (the distance of the current policy to the optimal policy) will shrink as k grows. The sum is bounded by some combination of α k and β k plus an irreducible function approximation error ς . The left-hand side is a weighted average of function value errors plus an average error from step 1 to K . The following result summarizes the convergence rate using particular step size choices. We obtain the first part of Corollary 3.5 by fixing the number of iteration K and optimizing the right-hand side of Eq. 25. We get the rest by straightforward computation.

then the average reward will converge to the global optimal

[ ] , ρ ∗

b) If µ > 0 , and η k = 1 µk and β k = 1 , then the average reward will converge to the global optimal

c) If µ > 0 , η k = 2 µk and β k = k +1 , then the average reward will converge to the global optimal

Remark The above result shows that if we carefully choose the function approximation ¯ Q and the regularizer h so that µ > 0 , we need O ( ε -1 ) number of SPMD iterations to obtain ε precision solution, i.e., 1 K ∑ K -1 k =0 ρ π k -ρ ∗ + E [ D ( π K , π ∗ )] ≤ ε . The complexity deteriorates to O ( ε -2 ) if µ = 0 . Notice that ς ¯ Q and ς ω are irreducible solely by policy optimization, as they arise due to function approximation and stochastic estimation errors.

When µ < 0 , global optimal solutions are unobtainable. The best we can do is to find a stationary point, that is when ψ and D ( π k , π k +1 ) tend to 0 . In this case, we only need to assume that the estimation error terms δ ω ,det k ( s ) and δ ω ,sto k ( s ) are bounded, i.e.,

Assumption 3.6. When µ < 0 , the critic step has bounded errors for any s ∈ S , i.e., there exist some constants ¯ ς ω , ¯ σ ω such that

Theorem 3.7. Let the number of iterations K be fixed. Suppose that 2.1, 3.2, and 3.6 hold. Also assume that µ < 0 and that η k = η = min {| µ | / 2 , 1 / √ K } , k = 0 , . . . , K -1 . Then for any s ∈ S , there exist iteration indices k ( s ) found by running K iterations of the SPMD method such that both the generalized advantage function and the distance between the iterates will converge to 0 , i.e.,

Remark This is the most general case, as we don't assume any restrictions on the function approximation class. As expected, the complexity bound becomes worse compared to previous results. To reach a ε -precision stationary point, we require at least O ( ε -2 ) iterations. Note that ψ ≈ 0 indicates that ¯ Q π k ( s, π k +1 ( s )) has virtually no difference than ¯ V π k ( s ) . This has a similar implication of D ( π k +1 ( s ) , π k ( s )) approaching 0 which implies that we are reaching a stationary point.

3.1 Practical algorithm for Entropy Regularized AMDP

The above SPMD algorithm computes π ( s ) for every state. In each iteration, we need to solve a subproblem Eq. 15 for every state (or for any state encountered). In practice, we prefer performing each SPMD iteration in a mini-batch style, i.e., approximately solving Eq. 15 from some trajectories collected. If the regularizer h and ω are the negative entropy, we can approximately solve the SPMD actor step Eq. 15 by

where Z ( φ k ) is some normalization constant and ˜ Q π k ( s, a ; φ k ) := -η k 1+ η k ¯ Q π k ( s, a ; φ k ) -1 1+ η k log π k ( s ) . In practice, we apply the so-called reparameterization trick [13] to obtain an unbiased gradient estimator. Let the policy be parameterized by

where /epsilon1 is an input noise sampled from a fixed distribution; ξ represents the parameters of the policy. Then we can approximately perform the actor step by solving the following problem:

For large action spaces, the reparameterization trick efficiently approximates the solution of Eq. 15. As for the critic step, we can approximate ¯ Q by minimizing the Temporal Difference (TD) error, i.e., solving the following problem:

where ˜ ρ π is the estimated average cost, e.g., by taking ˜ ρ π = 1 | ζ | ∑ c ( s, a ) + h a ( s ) .

4 Inverse Policy Mirror Descent for IRL

Equipped with an efficient solver for AMDPs presented in section 3, we can now solve the IRL problem in the form of (Dual IRL). We suppose θ parameterizes the reward function with arbitrary methods, e.g., neural networks. To solve the dual problem, we update the parameter θ by performing a gradient descent step θ k +1 = θ k -α k ∇ θ L ( θ ) where α k is some predefined step size. The gradient computation of the dual objective function is

Algorithm 2: The Inverse Policy Mirror Descent (IPMD) algorithm

1: Input : Initialize random policy π 0 and step size sequence { α k }

3: Collect samples { ( s t , a t ) } t ≥ 1 and compute c ( s t , a t ; θ k ) based on current reward estimation.

2: for k = 0 , 1 , · · · , K do

4: Critic step : Implement a policy evaluation algorithm to evaluation ¯ Q π k ,

5: Actor step : Update the policy by solving the following for every s ∈ S

6: Dual Update step : Perform the stochastic update

7: end for

Without accessing the transition kernel P , the true gradient is unavailable. So we use a stochastic approximation instead, denoted as g k := g ( θ k ; ζ E k ) -g ( θ k ; ζ π k ) where g ( θ ; ζ ) := 1 N ∑ N t =1 ∇ c ( s t , a t ; θ ) is the stochastic estimator of the gradient of the average reward. We describe the proposed Inverse Policy Mirror Descent (IPMD) algorithm in Algorithm 2. In this section, we provide our analysis that captures the algorithm behaviors across iterations since, in each iteration, ¯ Q is only an approximation of the true differential Q -function, which itself alters due to the change of the reward estimation. The idea of the proof is based on the Lipschitz continuity of the iterates, as it controls the difference of these functions across iterations. We make the following formal assumptions for such a purpose.

Assumption 4.1. For any s ∈ S , a ∈ A , the gradient of the reward function is bounded and Lipschitz continuous, i.e., there exist some constant real numbers L r , L g so that the following holds:

Note that ¯ Q is also a function of θ . As shown in section 3, ¯ Q can be parameterized but we only denote the estimator as ¯ Q π ( s, a ; θ k ) since its parameters do not contribute to the analysis.

Assumption 4.2. Suppose that at least one of S , A is continuous, we assume that where L q is some positive constant. For convenience we denote ¯ Q π ( s, a ; θ k ) also as ¯ Q π θ k ( s, a ) .

We further assume that the estimation error from using ¯ Q π θ is bounded.

Assumption 4.3. For any s ∈ S , a ∈ A , there exist some constants ς k , ς , ν k , ν such that the following inequalities hold:

Notice that this error bound subsumes both the estimation errors and approximation errors. In general, the Bellman operator is nonexpansive in the average-reward setting, so analysis based on the contraction property of the Bellman operator with the infinity norm, as what is used in [37], fails in this case. To this end, we assume that the operator is span contractive, i.e.,

Assumption 4.4. In the critic step, there's a way to construct a span contractive Bellman operator T such that there exists 0 < γ < 1 for any differential Q -functions ¯ Q 1 , ¯ Q 2 ,

In A.6, we provide conditions when the Bellman operator has a J-step span contraction. Now we can discuss the convergence results in the following theorems. The convergence of the differential Q-function is characterized in the following result.

Theorem 4.5. Suppose that assumptions 2.1, 4.1-4.4 hold. If α = α 0 √ K , the differential Q-function will converge to the optimal solution for a given reward function parameterized by θ k , i.e.,

Proof can be found in A.7. The above theorem shows that the differential Q-function will approach the optimal differential Q-function ¯ Q π θ k θ k with respect to the current reward estimation θ k . If the reward estimation is accurate, so is the differential Q-function . Finally, in the next theorem, we show that the policy converges to the optimal policy for a given reward function, and the reward function approximation converges to a stationary point. Proof can be found in A.8.

Theorem 4.6. Suppose that assumptions 2.1, 4.1-4.4 hold. If α = α 0 √ K and run the proposed IPMD algorithm K iterations, the algorithm will produce near stationary solutions for the reward function and the optimal policy for such reward function. Specifically, the following holds,

where α 0 , ν , L c , L r , C d are some constants and |S| , |A| some measure of the state and action space.

Remark First, the above theorem shows that the policy will converge to the optimal policy given the reward parameterization, although each iteration's reward parameter differs. Second, the reward parameter θ will converge to a stationary point of (Dual IRL), as the objective can be highly nonconvex under general function approximation. Third, for general AMDPs without 1 -step span contraction, we can use a J -step Bellman operator (see Appendix A.7) for policy evaluation to maintain a O ( ε -2 ) complexity for the entire algorithm.

5 Numerical experiments

In this section, we showcase the performance of the proposed SPMD and IPMD algorithms. Our code can be found at https://anonymous.4open.science/r/IPMD-9D60 . See more detail about all our implementation in the Appendix A.9.

MuJoCo Robotics Manipulation Tasks for RL This experiment tests our RL agent's performance on robotics manipulation tasks. Our SPMD algorithm is based on the stable-baselines3 [28]. We compare the performance of our algorithm with Soft Acrot-Critic (SAC) [13] implemented in [28]. The policy network employs two fully connected hidden layers of dimension 256 each, taking actions as input and outputting a distribution. Both the Q network and reward function share the same architecture, with ReLU activation used in hidden layers. A double Q -learning technique is used to minimize overestimation [13]. During training, we found that setting the entropy coefficient term to 0.01 makes training stable and efficient. The learning rate is 3 e -4 . Each step of the algorithm samples 512 state-action sample pairs. Table 1 reports the numerical results of each model. Our proposed SPMD achieves on-par performance with SAC and exceptionally better performance in the Humanoid environment.

MuJoCo benchmark for IRL In this experiment, we compare the proposed IPMD method with IQ-Learn [11] and f -IRL [23]. The authors of ML-IRL have not released its implementation at the time we experimented. Nevertheless, the performance of ML-IRL is comparable to f -IRL. It is observed in the literature that imitation learning algorithms have inferior performance [37]. Hence we omit these methods. Table 2 reports the numerical results of each model. Note that we cannot record a competitive result for IQ-Learn with Humanoid as claimed in the original paper, which we highlight using ∗ . The result shows that IPMD outperforms in a variety of environments. One possible reason Ant is falling behind is that Ant has more ground contact since it has more legs. This will impact the mixing time of the MDP and our assumption on the 1 -step contraction. Compared to Half-Cheetah, or Humanoid, it is harder for the Ant to transition from an arbitrary state to another arbitrary state as it involves multiple legs working together. One remedy is that we construct a J -step contractive operator for policy evaluation, as done in [7]. The success of humanoid is possibly due to a slightly different policy evaluation scheme compared to the discounted setting, where the entropy

Task

SAC

SPMD (Ours)

Hopper

3876

±

78

3619

±

5

Half-Cheetah

13300

±

46

13025

±

37

Walker

6115

±

41

4454

±

26

Ant

5118

±

57

5230

±

118

Humanoid

6923

±

11

10249

±

7

Table 2. MuJoCo Results for IRL.

The aver-

age performance of IRL agents across five runs.

IQL

f

-IRL

IPMD (ours)

Expert

1909

3083

3564

4115

9132

11259

12634

15467

5155

5378

5423

5323

3486

5460

4053

5783

661

∗

3580

7379

7137

Pendulum-v1

LunarLanderContinuous-v2

IQ-Learn

IQ-Learn

IPMD

IPMD

10²

103

Errorlsp.0

101

100

102

2

Episode

3

2

Episode

3

4

term from the policy no longer plays a part, as the term c -ρ cancels out the additional regularization and entropy of the policy. We suspect this brings more stable training and thus higher performance.

Reward Recovery Finally, we compare the proposed IPMD method against IQ-Learn on recovering the expert's reward function in two environments: Pendulum and LunarLanderContinuous, both implemented in [4]. The result of the experiment is shown in Figure 1. Using state-action pairs from 4 different episodes, we compare the span seminorm of the predicted reward and the true reward (the reward the expert uses). The experts are trained with a discount factor 0 . 99 . IQ-Learn's discount factor is set to 0 . 99 . The result shows IPMD's superiority in reward recovery.

6 Discussion

In this paper, we formulate the Maximum Entropy IRL problem with the average-reward criterion and propose efficient algorithms to solve it. We devise the SPMD method for solving RL problems in general state and action spaces beyond the scope of linear function approximation with strong theoretical guarantees. We integrate this method to develop the IPMD method for solving the entire IRL problem and provide convergence analysis. To reach a ε precision stationary point, IPMD requires O ( ε -2 ) policy optimization steps.

We also notice there are possible improvements from this work. For example, we impose various assumptions both the function approximation classes and strong reliance on the accuracy of the critic step. It is natural to seek more robust methods when facing inaccurate estimations. Additionally, we impose continuity assumption in various Assumptions (e.g. Assump 4.2). Such assumptions can be violated if neural networks are used as function approximations. In practice, it is possible to alleviate the issue by "clipping" the gradient norm when performing gradient descent, which proves effective in related works [30]. Furthermore, we only consider uniform ergodic chains. It is interesting to see how our method would hold under the unichain or multi-chain setting. Lastly, the analysis based on span contraction might be improved for general MDPs. This requires nontrivial work and thus we leave it for future development.

References

Anas Barakat, Ilyas Fatkhullin, and Niao He. Reinforcement learning with general utilities: Simpler variance reduction and large state-action space. arXiv preprint arXiv:2306.01854 , 2023.

Jalaj Bhandari and Daniel Russo. A note on the linear convergence of policy gradient methods. arXiv preprint arXiv:2007.11120 , 2020.

Vivek S Borkar and Vijaymohan R Konda. The actor-critic algorithm as multi-time-scale stochastic approximation. Sadhana , 22:525-543, 1997.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 2021.

Alex J Chan and Mihaela van der Schaar. Scalable bayesian inverse reinforcement learning. arXiv preprint arXiv:2102.06483 , 2021.

Zaiwei Chen, Sajad Khodadadian, and Siva Theja Maguluri. Finite-sample analysis of off-policy natural actor-critic with linear function approximation. arXiv preprint arXiv:2105.12540 , 2021.

Jaedeug Choi and Kee-Eung Kim. Nonparametric bayesian inverse reinforcement learning for multiple reward functions. Advances in neural information processing systems , 25, 2012.

Jaedeug Choi and Kee-Eung Kim. Hierarchical bayesian inverse reinforcement learning. IEEE transactions on cybernetics , 45(4):793-805, 2014.

Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAI Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in neural information processing systems , 30, 2017.

Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems , 34: 4028-4039, 2021.

Matthieu Geist, Julien Pérolat, Mathieu Lauriere, Romuald Elie, Sarah Perrin, Olivier Bachem, Rémi Munos, and Olivier Pietquin. Concave utility reinforcement learning: the mean-field game viewpoint. arXiv preprint arXiv:2106.03787 , 2021.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Tsubasa Hirakawa, Takayoshi Yamashita, Toru Tamaki, Hironobu Fujiyoshi, Yuta Umezu, Ichiro Takeuchi, Sakiko Matsumoto, and Ken Yoda. Can ai predict animal movements? filling gaps in animal trajectories using inverse reinforcement learning. Ecosphere , 9(10):e02447, 2018.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems , 29, 2016.

Sajad Khodadadian, Zaiwei Chen, and Siva Theja Maguluri. Finite-sample analysis of offpolicy natural actor-critic algorithm. In International Conference on Machine Learning , pages 5420-5431. PMLR, 2021.

Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32(11):1238-1274, 2013.

Guanghui Lan. Policy optimization over general state and action spaces. arXiv preprint arXiv:2211.16715 , 2022.

Guanghui Lan. Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming , 198(1): 1059-1106, 2023.

Tianjiao Li, Feiyang Wu, and Guanghui Lan. Stochastic first-order methods for average-reward markov decision processes. arXiv preprint arXiv:2205.05800 , 2022.

Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. arXiv preprint arXiv:2304.12966 , 2023.

Sean P Meyn and Richard L Tweedie. Markov chains and stochastic stability . Springer Science &Business Media, 2012.

Tianwei Ni, Harshit Sikchi, Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning , pages 529-551. PMLR, 2021.

GChPflug. On-line optimization of simulated markovian processes. Mathematics of Operations Research , 15(3):381-395, 1990.

Georg Ch Pflug. Derivatives of probability measures-concepts and applications to the optimization of stochastic systems. In Discrete Event Systems: Models and Applications: IIASA Conference Sopron, Hungary, August 3-7, 1987 , pages 252-274. Springer, 2006.

Robert Pinsler, Max Maag, Oleg Arenz, and Gerhard Neumann. Inverse reinforcement learning of bird flocking behavior. In ICRA Swarms Workshop , 2018.

Martin L Puterman. Markov decision processes: Discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1-8, 2021.

Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences , 3(6):233-242, 1999.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Can Eren Sezener, Eiji Uchibe, and Kenji Doya. Obtaining reward functions of rats using inverse reinforcement learning. 2014.

Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. Advances in Neural Information Processing Systems , 30, 2017.

Tengyu Xu, Zhe Wang, and Yingbin Liang. Improving sample complexity bounds for (natural) actor-critic algorithms. Advances in Neural Information Processing Systems , 33:4358-4369, 2020.

Shoichiro Yamaguchi, Honda Naoki, Muneki Ikeda, Yuki Tsukada, Shunji Nakano, Ikue Mori, and Shin Ishii. Identification of animal behavioral strategies by inverse reinforcement learning. PLoS computational biology , 14(5):e1006122, 2018.

Tom Zahavy, Alon Cohen, Haim Kaplan, and Yishay Mansour. Average reward reinforcement learning with unknown mixing times. In The Conference on Uncertainty in Artificial Intelligence (UAI) , 2020.

Tom Zahavy, Brendan O'Donoghue, Guillaume Desjardins, and Satinder Singh. Reward is enough for convex mdps. Advances in Neural Information Processing Systems , 34:25746-25759, 2021.

Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Maximum-likelihood inverse reinforcement learning with finite-time guarantees. arXiv preprint arXiv:2210.01808 , 2022.

Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Understanding expertise through demonstrations: A maximum likelihood framework for offline inverse reinforcement learning. arXiv preprint arXiv:2302.07457 , 2023.

Junyu Zhang, Alec Koppel, Amrit Singh Bedi, Csaba Szepesvari, and Mengdi Wang. Variational policy gradient method for reinforcement learning with general utilities. Advances in Neural Information Processing Systems , 33:4572-4583, 2020.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence , 2008.

Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In ICML , 2010.

Konrad Zolna, Scott Reed, Alexander Novikov, Sergio Gomez Colmenarejo, David Budden, Serkan Cabi, Misha Denil, Nando de Freitas, and Ziyu Wang. Task-relevant adversarial imitation learning. In Conference on Robot Learning , pages 247-263. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses Inverse Reinforcement Learning (IRL) in the average-reward setting. The authors propose a Stochastic Policy Mirror Descent (SPMD) method to solve the Average Reward Markov Decision Process subproblem and use SPMD to propose the Inverse Policy Mirror Descent method to solve the IRL problem. The authors provide complexity results for these methods and experimentally validate them using the MuJoCo benchmark.\n\nSTRENGTHS:\n- The authors develop a novel method and show its merits both theoretically and experimentally compared to existing methods.\n- The theoretical analysis is very thorough and the authors clearly outline the assumptions that are made.\n- The proposed IPMD method achieves good results on a variety of environments.\n\nWEAKNESSES:\n- The paper is a bit light on experiment details (e.g. hyperparameters). More detail here would be appreciated (e.g. see Questions).\n- It’d be nice if the authors could provide an intuitive explanation for the result of Theorem 3.4. Equation 25 is a bit difficult to understand.\n- The authors make a fair number of assumptions in their theoretical analysis. It'd be nice if the authors added some discussion on which assumptions hold in practice and/or how they enforce those assumptions, as well as how restrictive these assumptions are.\n\nQUESTIONS:\n- Does Assumption 3.2 hold in practice? Do the authors do anything to enforce these constraints?\n- What is $\\omega$ in Equation (3) and Algorithm 1 chosen to be in practice?\n- Can the authors provide architecture details for the policy, Q network, and learned reward function in the appendix?\n- Equation 24 in Theorem 3.4 makes an assumption about step size. Is this assumption implemented in practice (i.e. in Algorithm 1 what is $\\eta_k$ chosen to be?\n- It seems the authors assume access to a large number of expert trajectories (Appendix A.9 says expert demonstrations are collected for five million steps). I’m curious how IPMD would perform with a more limited number of expert trajectories (for instance, the f-IRL[1] paper has experiments where they just use 1 expert demonstration). Have the authors considered how number of expert trajectories affects performance of IPMD?\n\n[1] f-IRL: Inverse Reinforcement Learning via State Marginal Matching (Ni et al.)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper inverse reinforcement learning is studied when the teacher was using an average-reward criterion rather than discounted rewards with known discount factor. The paper proposes a stochastic first-order method starting from stochastic policy mirror descent for MDPs and continuing towards inverse policy mirror descent for solving the IRL problem. The paper contains also some numerical experiments based on MuJoCo benchmarks.\n\nSTRENGTHS:\nThe paper is very clear and elegant in its use of average reward method in RL and IRL. There are many papers on duality of this type although not yet specifically on the average reward criterion, although in NAC papers similar considerations were made.\n\nThe supplementary material is strictly supplementary and very clear and comprehensive.\n\nIt's great the hints for practical use are included.\n\nWEAKNESSES:\n“To the best of our knowledge” is not a useful sentence in an abstract, in particular in an anonymous manuscript. It is more useful to discuss such opinions later in the paper in the context of related results (see e.g. previous NeurIPS conferences).\n\nDuality has been used in many similar contexts, but it could still be said more clearly why it is a good idea here. It is acceptable, if you want to avoid discussing natural AC, but then there is more to explain.\n\nExperiments are relatively few with just final results given, so no understanding why the proposed method is useful can be gained. The is no discussion of the experimental results, therefore, although one may guess, it is not clear why the proposed method struggles on Ant or whether there is any relation of the performances comparing tables 1 and 2. There is no attempt included to check whether the complexity results are tight or whether global optima are indeed found.\n\nCheck: “Acrot”\n\nAssumptions 4.2 and 4.3 are formally stated in the main text, but do not seem to be used. In the appendix only 4.2 is mentioned.\nRef. 28 is not mentioned in the text, perhaps a separate bibliography for the appendix would make sense.\n\nI find it a bit difficult to distinguish between all the Qs. The distinction between math-Q and cal-Q is probably necessary, For example, tilde-cal-Q (32) could as well be substituted by its definition. It occurs again only in (34) and (38), where at least a ref to the definition needs to be included. In “differential Q-function” (in italics in the main text vs. theorem environment) different fonts are used, which should be avoid this, ideally by using the math-Q font that seems to occur also in the theorem environment.\n\nQUESTIONS:\nCan you explain more about the improvements (as in Humanoid RL) and deficiencies (as in Ant IRL or walker RL)? What feature of the algorithm contributes to the success or suboptimality of the performance in each problem? Are the assumption not strictly satisfied, or is the sampling the reason for any suboptimality?\n\nWhich theorems have appeared in earlier work or have strong similarities to theorems in earlier work?\n\nI would assume that 1/(1-gamma) not much smaller than K, and the difference between them is related to the number of states and actions. In other words, is there a practical need for the O-symbols or is the complexity already implied by the known factor proportional to 1/(1-gamma)?\n\nWhy does the performance show variablity even for simple problems (see Fig. 1)?\n\nLIMITATIONS:\nThe authors state in the supplementary material: \"We anticipate no potential negative societal impacts concerning this research.\" which could seem questionable in the context of IRL, because IRL has the potential to uncover hidden motives in legal human behavior without the consent of a person. The research seems fine, but some discussion, beyond the mere statement would be useful.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an inverse reinforcement learning (IRL) algorithm for infinite horizon average reward Markov decision processes (AMDPs). At first, the authors show the stochastic policy mirror descent (SPMD) algorithm that achieves $\\mathcal{O}(\\varepsilon^{-1})$ rate of convergence. Then, the authors propose the inverse policy mirror descent (IPMD) algorithm that achieves $\\mathcal{O}(\\varepsilon^{-2})$ rate of convergence. The SPMD algorithm is compared with SAC on the MuJoCo benchmarks, and the SPMD achieves on-par performance with SAC. The IPMD algorithm is compared with f-IRL and IQL, and the experimental results show that IPMD is slightly better than f-IRL. Interestingly, the proposed algorithms perform much better than the baselines for the Humanoid environment that has many degrees of freedom.\n\nSTRENGTHS:\n- Originality: RL and IRL algorithms for AMDP with strong theoretical basis are novel, although Dvijotham and Todorov proposed IRL under the linearly solvable Markov decision processes (LMDP). \n- Quality: The paper provides some strong theoretical analysis on AMDP.\n- Clarity: The authors show detailed steps to prove the theorems in Appendix. \n- Significance: The standard IRL algorithms deal with infinite horizon discounted reward problems under the assumption that the discount factor is known in advance. It is problematic because the expert's discount factor is usually unknown. The proposed method is promising because it can avoid tuning the discount factor.\n\nWEAKNESSES:\n- Some recent studies regarding AMDPs are not mentioned in the manuscript.\n\nQUESTIONS:\nMajor comments\n- Is the SPDM algorithm for AMDPs (Algorithm 1) an on-policy method? The objective function for training $\\bar{\\mathcal{Q}}^{\\pi, \\zeta}$ (35) suggests an on-policy algorithm, but I did not find the explanation. In addition, (35) does not use a target network. Is it a benefit of the property of the on-policy algorithm? \n- I do not fully understand why $\\nabla \\omega (\\pi)$ is usually unavailable, as described in Lines 143-144, because $\\omega$ is selected by ourselves. Furthermore, it is unclear why its estimator is parameterized by $\\phi$ because $\\pi$ is parameterized by $\\xi$ (33). \n- Please show $g(\\theta; \\zeta)$ in detail because it is used to update $\\theta$. For example, is $g(\\theta_k; \\zeta^\\pi_k)$ the estimator of $\\mathbb{E}_{(s, a)~d^\\pi} [\\nabla_\\theta c(s, a; \\theta)]$? \n- The experimental results are promising, but one potential reason for successful results is that the reward function of the MuJoCo benchmark is well-shaped. For example, the reward of Walker2D is given by healthy_reward + forward_reward - control_cost: https://gymnasium.farama.org/environments/mujoco/walker2d/ . It may imply that the average reward formulation is more appropriate than the discount reward one in the MuJoCo benchmark. My interest is the performance of the proposed method when it is evaluated in sparse reward settings such as navigation tasks. \n\nMinor comments\n- The following paper proposes a framework based on stochastic mirror descent for AMDP. Please discuss the relationship between these papers and the proposed method. \n  - Y. Jin and A. Sidford. (2020). Efficiently Solving MDPs with Stochastic Mirror Descent. In Proc. of ICML. G. \n  - Neu et al. (2017). A unified view of entropy-regularized Markov decision processes. http://arxiv.org/abs/1705.07798 . \n  - C.-Y. Wei, et al. (2020). Model-free Reinforcement Learning in Infinite-horizon Average-reward Markov Decision Processes. In Proc. of ICML.\n- As mentioned in Strength, the following paper proposes a method to estimate the reward function under LMDP: K. Dvijotham and E. Torodov. (2010). Inverse Optimal Control with Linearly Solvable MDPs. In Proc. of ICML. It would be better to discuss their paper.\n\nLIMITATIONS:\nMinor comment: The authors briefly mention issues on potential societal impacts in A.10.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the problem of inverse reinforcement learning (IRL) under the maximum entropy framework (MaxEnt-IRL) and an average reward criterion. The MaxEnt-IRL problem is formulated as a combination of an average reward Markov decision process (AMDP) and a dual IRL problem. The proposed algorithms, namely the SPMD algorithm and the IPMD algorithm, are utilized to solve the AMDP and dual IRL problem, respectively. The SPMD algorithm achieves a gradient computation step complexity of $\\mathcal{O}(1/\\epsilon)$ under general state and action spaces, while the IPMD algorithm has a complexity of $\\mathcal{O}(1/\\epsilon^2)$. The empirical studies conducted on the MuJoCo benchmark and various control tasks confirm the theoretical findings.\n\nSTRENGTHS:\n1. The idea of formulating the IRL problem as a dual IRL problem, incorporating an AMDP as a subproblem, is interesting.\n2. The paper extends a previous RL algorithm [1] to general state and action spaces with a general function approximation class for AMDPs. The paper is clear and easy to follow.\n3. The proposed algorithms are well-founded and supported by theoretical analyses of average reward Markov decision processes. The convergence analysis of the algorithms is provided under certain assumptions on the approximation function classes.\n\nOverall, the paper is well-written and easy to follow. The proposed algorithms are well-motivated and supported by theoretical foundations. However, certain assumptions regarding function approximation classes and the critic step may be too restrictive, potentially resulting in the failure of convergence in scenarios involving deep neural networks. Despite these concerns, the theoretical contributions of the paper could be of interest to the community, and I recommend the acceptance of the paper.\n\n[1] Tianjiao Li, Feiyang Wu, and Guanghui Lan. Stochastic first-order methods for average-reward markov decision processes.arXiv preprint arXiv: 2205.05800, 2022.\n\nWEAKNESSES:\n1/ The advantages of employing the average reward criterion over discounted IRL are not adequately explained in the paper.\n2/ Assumption 3.2 assumes weak convexity and Lipschitz continuity for the approximated Q-functions. These assumptions may not hold in practice, particularly when neural networks are used for function approximation. As the global convergence result of SPMD relies on the convexity of the approximated Q-function and the regularizer h, it may not hold in practical scenarios.\n3/ Assumption 3.6 imposes a restriction that the errors of the critic step, typically represented by neural networks, are bounded. This assumption may be overly restrictive, as estimations can be highly inaccurate in many cases. The convergence result for general function approximation classes may also fail to hold in practice.\n4/ When the reward function could be parameterized by neural networks, which is normally the case in practice, Assumption 4.1 would be too strong.\n5/ In the reward recovery experiment, it would be beneficial to compare the proposed algorithms with discounted IRL algorithms since the expert is trained in a discounted environment (the author trained the expert agent using SAC with a discount factor of 0.99).\n6/ Including the running time of the proposed algorithms and the baselines would provide valuable insights.\n\nQUESTIONS:\nSeen in the weaknesses section.\n\nLIMITATIONS:\nSeen in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}