Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
Authors: Yang Chen
Menglin Zou
Jiaqi Zhang
Yitan Zhang
Junyi Yang
Gael Gendron
Libo Zhang
Jiamou Liu
Michael J. Witbrock
Venue: NeurIPS 2025 poster ()
Keywords: Inverse Reinforcement Learning
Imitation Learning
Stablization
Trust Region
Majorization-Minimization
Primary area: reinforcement_learning
TL;DR: Introduce the Trust Region Reward Optimization theory and Proximal Inverse Reward Optimization algorithm that stabilize Inverse RL

Abstract: Inverse Reinforcement Learning (IRL) learns a reward function to explain  expert demonstrations. Modern IRL methods often use the adversarial (minimax) formulation that alternates between reward and policy optimization, which often lead to {\em unstable} training. Recent non-adversarial IRL approaches improve stability by jointly learning reward and policy via energy-based formulations but lack formal guarantees. This work bridges this gap. We first present a *unified* view showing canonical non-adversarial methods explicitly or implicitly maximize the likelihood of expert behavior, which is equivalent to minimizing the expected return gap. This insight leads to our main contribution: *Trust Region Reward Optimization* (TRRO), a framework that guarantees *monotonic* improvement in this likelihood via a Minorization-Maximization process. We instantiate TRRO into *Proximal Inverse Reward Optimization* (PIRO), a practical and stable IRL algorithm. Theoretically, TRRO provides the IRL counterpart to the stability guarantees of Trust Region Policy Optimization (TRPO) in forward RL. Empirically, PIRO matches or surpasses state-of-the-art baselines in reward recovery, policy imitation with high sample efficiency on MuJoCo and Gym-Robotics benchmarks and a real-world animal behavior modeling task.

Trust Region Reward Optimization and Proximal Inverse Reward Optimization Algorithm
Yang Chen I
Menglin Zou
Jiaqi Zhang
Yitan Zhang?
Junyi Yang?
Gacl Gendron Libo Zhang"
Jiamou Liu
Michael J. Witbrock2
1 Shanghai Artificial Intelligence Laboratory 2 University of Auckland Chongging University
chenyang4@pjlab.co org.
Abstract
Inverse Reinforcement Learning (IRL) learns reward function to explain expert demonstrations. Modern IRL methods often use the adversarial (minimax) formu- lation that alternates between reward and policy optimization, which often lead to unstable training. Recent non-adversarial IRI. approaches improve stability by jointly learning reward and policy via energy-based formulations but lack formal guarantees. This work bridges this gap. We first present unified view showing canonical non-adversarial methods explicitly or implicitly maximize the likelihood of expert behavior, which is equivalent minimizing the expected return gap. This insight leads to our main contribution: Trust Region Reward Optimization (TRRO), a framework that guarantees monotonic improvement in this likelihood via a Minorization -aaiiiizatiiiiiiiooonnn process. We instantiate TRRO into Proximal Inverse Reward Optimization (PIRO), practical and stable IRL algorithm. Theo- retically, TRRO provides the IRL counterpart to the stability guarantees of Trust Region Policy Optimization (TRPO) in forward RI. Empirically, PIRO matches surpasses state-of-the-arr baselines in reward recovery. policy imitation with high sample efficiency MuJoCo and Gym-Robotics benchmarks and real-world animal behavior modeling task.
Introduction
Learning optimal policies from fixed reward functions reinforcement learning (RL); learning rewards from fixed expert policies is inverse reinforcement learning (IRL) [28]. Modern IRL methods [12, 40, 33] often take a minimax game formulation and a bi-level optimization procedure, where reward function (min player) is adversarially optimized differentiate between a best-response policy (max player, an RL subroutine) and the expert policy via their expected return gap (a.k.a. the imitation gap [39]). Due to the advantages of interpretabilitty robustness dynamics shifts [1], and out-of-distribution generalization [6], these methods have been effectively applied in autonomous driving [21], robotics [7], and reward modeling in language models [38]. However, despite its theoretical grounding and practical appeal, adversarial training introduces optimization instability due brittle approximations and high sensitivity hyperparameters, hindering reliable reward recovery.
"Title used at submission and review: PIRO: Toward Stable Reward Learning for Inverse RI. via Monotonic Policy Divergence Reduction.
Main contributors. Yang Chen developed the theorems, completed the proofs, wrote the paper, and implemented the initial version of algorithm. Menglin
the experimental evaluation. Jiaqi Zhang and Junyi Yang validated the algorithm using toy models. Yitan Zhang conducted the experiments on robotics and animal behavior modeling tasks. The remaining authors contributed through critical discussions and feedback.
1 spondddg author.
'The implementation is available
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Policy Optimization
PPO
Pratical Stablity
PIRO
HE
*
33
32
R1
b
T3
22
"dualism"
10
Thecretical
TRPO
Reward Optimization
TRRO
70
33
imitation performance
sample efficiency
E
monotonic 122 MT
policy space di reward space
expected return gap
PIRO
TO
reward-induced policy
learning stability (the first formal guarantee)
reward update dependency
policy update trajectory Irust region reward updates
Figure I: Theoretical (top) and practical (bottom) contributions. Top: PPO - rooted in TRPO's theory of monotonic policy improvement- has been (one of) the most successful RL algorithm(s). This work motivated by a dualism: the mathematical beauty of TRPO should not exist in isolation, but in conju- gation with its inverse problem space. We identify and formalize this inverse counterpart, completing the "right half"" of this "symmetric picture". We believe this contribution advances RL theory and opens new avenues for designing robust IRI. algorithms. See Sec. 4 for theoretical justifications. Bottom: PIRO, our practical algorithm, achieves three-way balance among learning stability, imitation performance, and sample efficiency. To our knowledge, PIRO is the first IRL method that achieves state-of-the-ar performance in imitation performance and learning stability with high sample efficiency. See See. 5 for the practical algorithm design and Sec. 7 for experiments.
Figure Comparing Adversarial IRL, Non- adversarial IRL and our Trust Region Reward Optimization (TRRO). (a) Adversarial IRL meth- ods frame reward learning a game against (nearly) best-response policy, often resulting in unstable train- ing dynamics due to the inherent minimax structure. (b) Non adversarial IRL methods bypass this game setup by coupling reward and policy via energy- -asee formulations and jointly update them by minimiz- ing the expected return gap ka. the imitation gap). However, lacking principled control over re- ward update makes them sensitive to optimization emors. (c) TRRO reformulates non-adversarial IRL majorization-nin minitttiiiimiiiiiiiiiin (MM) process that identi- fies a trusted reward update in cach step. This ensures monotonic reduction in imitation gap and providing. to our knowledge, the first formal stability guarantee in IRI.. (Note: This is theoretical comparison assum- ing exact policy computation.)
Recent aon-adversarial IRL approaches [32, 15, 30, 50, 51, 44] revive a line of early apprenticeship learning methods [27, 31]; they bypass the nested adversarial training by coupling the reward and policy via an energy-based model [17]. jointly updating them to optimize some measure of fit to expert behavior. While improving empirical stability, they still lack principled control over reward updates. As result, provably stable IRL mechanism,
that ensures consistent progress toward expert imitation, remains elusive. This work aims to address this gap.
By leveraging the fact that the expected return gap between two policies equals the expected advantage value of one under the other [35, 25, 50], we develop unified view of canonical non-adversarial IRL methods. We show that they all, explicitly or implicitly, optimize the likelihood of expert behuvior, which is equivalent to minimizing the imitation gap (Sec. 3). This leads to our key insight: IRL, stability can be achieved by provably increasing the likelihood of expert demonstrations at every update step. We realize this insight in a principled non-ad adversarill IRL framework and a practical algorithm that together offer stable alternative to existing approaches.
Concretely, our contributions are summarized as follows, which are illustrated in Fig.
We propose Trust Region Reward Optimization (TRRO), a principled non-adversarial IRL framework that, to our knowledge, for the first time provides formal guarantee stability. As depicted in Fig. 2, it provides principled control on reward update via a Minorization-Maxiiiaaae (MM) process, which iteratively optimizes a surrogate objective function to identify trusted reward update that ensures a monotonic improvement in the likelihood of expert behavior (equivalent to reducing the imitation gap). (Sec. 4)
We develop Proximal Inverse Reward Optimization (PIRO), a practical IRL algorithm that approximates the theoretical guarantee of TRRO through adaptive step sizes in place of the theory- informed small updates. PIRO achieves a balance among learning stability, imitation performance and sample efficiency. It can be easily implemented on top of Soft Acitr-Critic [17] by adding few stochastic gradient steps for the controlled reward update. (Sec. 5)
We empirically demonstrate the strong performance of PIRO. Across MuloCo and Gym Robotics tasks, PIRO offers substantially improved stability and high sample efficiency, while matches or exceeds sfate-of-the-art IRL methods in reward recovery and policy imitation. (Sec. 7)

---

TRRO/PIRO mirrors the success of Trust Region Policy Optimization (TRPO) [35] and its successor Proximal Policy Optimization (PPO) [36]. TRPO guarantees monotonic policy improvement in expected return with respect to a fixed reward function, while TRRO ensures monotonic reduction in the expected return gap with respect to the expert behavior. In this sense, TRRO/PIRO serves as the inverse RL counterpart to TRPO/PPO in forward RI.
2
Preliminaries
Consider a Markov decision process (MDP) defined by n, where S and A are the state and action spaces, 17(-) is the initial state distribution, P x A x S - [0, 1] is the transition function, S A + R is the reward function, and y (0, the discount factor. stochastic policy S x A 7 [0, 1] defines a probabilistic action selection at each state. We denote the occupancy measure as p² (s. a) Ei=o" Pr(s: s, a, also ~ 17, n, P). Note that we will
omit the normalizing constant 1 for p" (s,
I-7
2.1 Maximum Entropy RL
MaxEnt RL characterizes the optimal behavior a policy n² that maximizes the policy entropy- augmented rewards:
J(,,r) Ep [r(s, a)] + H(T)), H(T) == Ep- [- [og(((((aa))))
(MaxEnt-RL)
Here, H(TT) is the discounted causal entropy [53] of a policy In MaxEnt RL, optimal policy follows an energy-based model:
" r"(als) exp(QF (s,a) V,," (s)),
where O the optimal soft Q-function and V the optimal soft value function satisfying:
V (s) = log EaEA exp(Q7 a)), OF (s, a) r(s,a) (l-ss=((11s.a)) [V."' (s')]-
(2)
Eq. (2) is the so-called Soft Bellman Equation. Given a reward function E C RSXA and policy x E II C ISSA the soft Q-value can be computed by iteratively applying the soft Bellman
operator B RSXA + R defined as:
(BTQ) a) - r(s, a) "/Bags" V(s) = E () a) log lals). (3)
The operator B is contractive [18] and defines the soft Q-function Q as unique fixed point solution, ie. OF = B O. An improved policy can be derived from O, through
Ta(ass) x (xp(Q?
which guarantees OF (als) > (als) for all (s, E S x A. Starting from arbitray policy т, repeated application of Eq. (3) and Eq. (4) gives the so-called soft policy iteration [18], which converges to the optimal policy x² that maximizes (T in (MaxEnt-RL).
2.2 Maximum Entropy IRL
Suppose we do not know the reward function but have of demonstrations DE = (SS, a ao....) } sampled from an expert policy TE. MaxEnt IRL aims to recover the reward function that explains demonstrations by minimizing the expected return gap (a.k.a. imitation gap [39]) through solving the following optimization problem: 2
min max J(TE, J( r) = Ex- [r(s, a)] - (Ep- [r(s, a)] + +())).
(MaxEnt-IRI.)
*ETI FER
In practice, EE,= (rss, a)] is emprically estimated expert demonstrations Dz. The minimax formulation of (MaxEnt-IRL) suggests an adversarial solution structure: outter loop optimizes the reward function by differentiating expert and learned policies through maximizing the imitation gap (Line 4, Alg. 1) and an inner loop trains optimal policy via MaxEnt RL process (Line Alg. 1). MaxEnt IRL has been well studied theoretically [53, 4] and has been practically applied [45, 13J. However, its nested structure can introduce significant training instability and computational burden, especially when state-action spaces are high-dimensional or continuous.
We hereafter omit the constant expert policy entropy H(Tæ) J
See Sec. 6 for the discussion on adversarial IRL methods.
3

---

Algorithm 1 Adversarial IRL
Algorithm 2 Non-Adversarial IRI.
1: Provided: Expert demonstration Dg, Reward
Provided: Expert demonstration Dg, Reward
parameter Po.
parameter 0u Policy To-
2: for in I,..., N do
2: for i in I,..., .N do
, // One round of soft policy iteration.
// full RL process
3: T. +- MaxEntRL 1).
3: (aaas)
exp(Qra a)).
4: 0, 4 arg max ro)
ro) - ro),
4: 0, t 0 + Vo([ ro))
5: end for
5: end for
2.3 Maximum Likelihood IRL
ML-IRL bypasses the nested loop in MaxEnt IRL by jointly updating the reward and policy via the energy-based model (Eq. (1)), thereby improving stability. Let He denote the optimal policy induced by a (0-parameterized reward function ra with 0 E Id. ML-IRL aims maximize the likelihood of expert behavior under To (equivalent to minimizing the KL divergence DKi((Te) (a|s) [ (als)) = Epex [log x (als)) log No (a)s))):
max (()) == Ep [log To (als))
(ML-IRL)
An important property of (()) is that it can be equivalently expressed as the imitation gap. 4 Proposition 1 (Lemma 1 in [50].. The log-likelihood objective ((0) in (ML-IRL) has the following equivalent form that implies the expression of its gradient:
(()) = E fe (s a)] - Ex (V (8o)] = J ro) - (T)
(5a)
Vol(0) E Voro a)] E, (s)
(6a)
Indeed, Proposition 1 is not so surprising, as it reflects a standard identity in RL theory: the expected return gap between two policies equals the expected advantage value (Q(s, a) - V(s) of one policy under the occupancy measure of the other [23, 35, 25]; in MaxEnt RL, the advantage value corresponds to log π (see Eq. (1)). However, its implication for MF-IRL is noteworthy: effectively bypasses the inner RI. loop typically required in MaxEnt IRI. As result, the nested-loop optimization reduced a single-loop structure: alternating between one round of soft policy iteration for policy improvement (Line 3, Alg. 2) and one gradient step for reward update (Line 4, Alg. 2).
To further mitigates instability, [50] employ a decaying gradient step size a₁ on for reward updates, where N is the total number of iterations and E (0. 1) is a constant. Under the assumption of exact policy computation for n; [50, Theorem 2] show that with Alg. (()) converges rate O(N-1)+ + ONN "), and converges to the optimal value under linear reward functions. However, this setup still lacks a formal stability guarantee, as gradient- reward updates with heuristic step sizes cannot ensure improvement in {(0) at each step. Our key contribution fills this gap: novel aon-adversarial IRL. framework that, under the similar assumption of exact policy computation, guarantees monotonic improvement in {(0) through a carefully designed non- non-gradient reward update mechanism (Sec. 4).
No
3 A Unified View of Non-Adversarial IRL: IR, ER and Beyond
In this section, we show an interesting yet natural fact that a range of canonical non-adversarial IRL methods - both implicit reward (IR) methods that learn soft Q-functions (e.g., Soft Q Imitation Learning (SQIL) [32], Inverse Q Learning (IQ-Learn) [15]) and explicit reward (ER) methods that directly learn reward functions (e.g., f-IRL [30] and ML-IRL) can be unified under the objective of maximizing the likelihood of expert behavior. As discussed further in Sec. 6, this unified view extends to broader class of non-adversarial IRI. methods that go beyond the settings of these canonical methods. This allows for unifying aoveraveraliaal IRL methods under a general optimization procedure (Alg. 2). highlighting the generality of maximizing the likelihood as principled objective and situates our framework (next section) within a broader methodological landscape.
Wee provide the proof of Proposition 1 in Appendix A.1 using the notations in this paper.

---

For IR, we already know that the objectives of SQIL and IQ- IQ-Learn are regularized versions of
(Q(w) = p (s. - B (Vo (so)),
(5b)
where V*(s) log EaEA (w, a)). Eq. (5b) can be derived by transforming (()) (Eq. (5a))
via replacing ne with rQw (s. a) = (w. a) PPCys aasaaal((V [V'( - the implicit reward defined
as the differences of w-parameterized soft Q-values via the soft Bellman equation (Eq. (2)).
For ER, we show that the objective of a basic form of f-IRL assuming state-only rewards and minimizing the KL divergence between expert and learner state marginals is equivalent to (()). up to constant. That is (proof of Eq. (6b) in Appendix B),
(s)
VoDKI. (p (s)))) x Vol(())
(6b)
where (s) p= a)/m(als) denotes the state marginal of the occupancy measure.
Pros and cons of IR/ER methods are well-documented [33]. IR offers higher computational efficiency, as Eq. (5b) depends solely on estimating the soft Q-function, which encodes both reward and policy. However, this coupling of reward and environment dynamics can lead to inaccuracies under dynamics shift, thereby limiting the reward transferability to new dynamics. In contrast, ER methods learn reward functions directly and avoid this entanglement, offering better robustness to dynamics shift. In light of this, our framework will adopt the ER formulation.
Trust Region Reward Optimization
In this section, introduce Trust Region Reward Optimization (TRRO), theoretical IRL framework that enforces stability by producing a guaranteed increase on the likelihood of expert behavior. To our knowledge, it provides the first formal theoretical stability guarantee for IRL.
To proceed, let Oota denote the current reward parameter and assume we have the corresponding optimal policy Told. As argued in Sec. 2.3, gradient- based reward updates cannot rigorously ensure an improvement in ((O). We thus consider a con--radient--asee bbased approach. Our key idea is to restrict the search for Inew within region centered around Oni such that all 0 in that region admit increase on E()). To do so, we introduce the following local approximation to {()):
Faad ta (0) = = (ro (s, a)] - EBo- (VTo (so)] = JTT re) (T).
(5c)
Proposition 2. Suppose is differentiable. The surrogate function la (0) in Eq. (5c) matches the original objective {(0) in Eq. (5a) to first order, for any value Oola:
Ce (foaa) = ((Oola)
and
V (0) VoL
(6c)
EE,7 Iro (s.a)) wnl (eraadd (sv))
=E, (Vrro(s.a)) -E,]]]EEE, Eooold [Vor((,ni))o--
Proof. See annotated equivalence relationships above.
Proposition implies that sufficiently small step Pala Inews which increases la (0), will also increases {(O). However, it still does not provide guidance on the suitable step size for this update. Our theorem below addresses this by deriving explicit lower bound on {(0new) in terms of
Ce (Onew) and the difference between Yagia and Tinew.
Theorem 3. Let Caara ( (enw)) = maxs.a |re.. (s) - Tema (s, a)}. Assume [A
< x and
|ro... (s, a) R, Vs E S, E A. Then, the following inequality holds:
((O ((0aww) 2 Lo (Onew) Oco.sa (Onew), where the constant
2|A1
- (5- (7 72 + 2)| log [A]
(7)
C (1 7
7)
Proof. In Appendix A.2.
SSee [32, Sec. 3.3] for SQIL and [15, Sec. 4] for LeaLeam..
5

---

R
((0)
Figure 3: Illustration the mechanism of Trust Region Reward Optimization (TRRO). The reward optimization follows Minorization process, iteratively optimizing surrogate func- tion that minorizes the original likelihood objective, thereby guaranteeing monotonic improvement in the likelihood of expert demonstrations (assuming exact policy optimization).
cD (Onew)
intorizaatoo-Maxiizzzttiinn Maximization
SxA.A.
(0) ( Cco.
0g G2 U 00
R
Since E0 (ota) 0. by continuity, there exists Onew in the neighborhood of Oold such that ((Onew) 2 u neew)) Cee ( (Onew). This implies that maximizing the lower bound in Theorem guarantees an increase (or at least no decrease) on {(O), which leads to the following procedure that alternates between policy and reward update:
π arg max, [TT, rema
new
arg max Lear (0) Cco.sa (0).
(TRRO)
Oold
Ouew
This implies the following theoretical guarantee on stability.
Corollary 4. Assume exact policy optimization. Staring from arbitrary reward parameter Oo. (TRRO) will yield a sequence of reward functions To:: : ... such that the corresponding likelihoud of expert demonstrations monotonically increases: E(Ou) S {{01) S ((02) S
As illustated in Fig. 3, TRRO is type of -itir/attion-maimiizzziiionnn (MM) algorithms [20]. where ( Cce. (0) is surrogate that minorizes E(O) and matches it at O A Ooa-- Maximizing the surrogate ensures progress on the original objective. light of this, TRRO plays role in inverse RL analogous to Trust Region Policy Optimization (TRPO) [35] in forward RI: while TRPO's theoretical framework uses the MM algorithm ensure monotonic policy improvement in expeted return with respect to fixed reward function, our TRRO ensures monotonic expected return gap (equivalent to the likelihood) reduction with respect to the given the expert behavior.
Proximal Inverse Reward Optimization Algorithm
In this section, we develop practical algorithm, Proximal Inverse Reward Optimization (PIRO). It approximates the theoretical guarantee of TRRO, enabling adpatively larger reward update steps, efficient policy optimization and accommodating continuous state-action spaces. It operates under realistic constraint of finite expert demonstrations Dg = {(50, ag. ..)
Adaptive reward update. The original scale factor C is often too large, leading to excessively small reward updates." To mitigate this, we introduce customizable coefficient μ >> relax the scale. Another issue is that eamu (0) is indifferentiable due to its definition as the maximum norm. To address this, we replace e (0) with the differentiable 2 norm of reward differences and calculate
on the state-action space for the tabular cases or, more generally, estimate on subset D DE and set of rollouts Ds sampled from Vold for continuous control:
Eouia (0) :0 E -E (ro.. a) ro(ss a))² 1/2
(8)
Note that through Ea. (0), we also implicitly penalize the magnitude of the reward function (the L norm (s, a)ll2), similar to the reward sparsity regularization in SQIL [32], which discourages assigning high rewards to state-action pairs absent in demonstrations.
The above approximations yield the following objective for each reward update step:
new arg max Loaia (0) le (0) p pēo (0).
We minimize Lo (0) using gradient descent by estimating
V (0) = E.g Vere(s, a)] - Eps a)) - VV (0).
(10)
'If La (0) Cce (0) reaches local maximum O.id. a wider search range needed known limitation of MM algorithms. This, however, is out of the scope of this paper. See Appendix D.2 for experiment for the performance comparison between theoretical and adaptive C.

---

We adaptively adjust the coefficient fr as follows:
If EBaia (0) > glrget X I, then μ + P x y: If Eo (0) < Eunget/ /x,, then μ + p/v,
(11)
where glurge 0, x. y > 1 are predefined hyperparameters. The updated j is used for the next reward update step. Sensitivity tests for æ, y, Etunee are in Sec. 7.6.
Practical policy optimization. In practice, we cannot expect exact policy optimization. For efficiency, similar to the setting in ML-IRL [50, 51], we calculate Told by performing several rounds of soft policy iterations through Soft Actor-Critic [18] under Teaia and Told-
Final practical algorithm. Finally, we obtain the following practical iterative procedure for PIRO:
π k SAC rounds with Tooia Vold
Rold
Ouew f-
grad. steps with V Vo (0). (PIRO)
Oold
Onew
Note that (PIRO) degrades into Alg. 2 (the general procedure of non- adversarial IRI.) k = n and 0. This indicates that, in theory, PIRO improves stability at the cost of more frequent updates. However, our empirical evaluation in the next sec-
Algorithm 3 Proximal Inverse Reward Optimization (PIRO)
1: Input: Expert demostrations Dy; Initialized reward parameter
Ooiá. policy Vold: Targets e coefficient μu and scalars 2;, >
Loop control parameters m, k, >
2:
for = 1 to m do
3: Vold t- k rounds of SAC based on read and Told-
4: for = 1 to do
tion (Tab. 1 and Fig. 4) reveals that this added computational effort does not compromise time efficiency, the improved stability leads to faster con- vergence, effectively offsetting the ad-
5: Sample batch Deg C Deg
6: Rollout Vold to sample a set of transitions Ds.
7: Estimate Lo (0) on De and Ds.
D Eq. (10)
8:
Update O to increase Le (0) via V VoLg. am end for
(0).
9:
ditional update overhead.
10: Adjust μ and Set Oold +- 0.
D> Eq. (11)
end for
To summarize, we show the training procedure of PIRO in Alg.
12: Output: reward reaia and policy Told.
Related Work
Adversarial IRL. Predominant IRL methods follow adversarial learning paradigm (see GAIL [19] and dissrimiinator-ctoor-iiiiit actor-critic (DAC) [24]), with AIRL variants [10-12] and extensions [48, 47, 8, 9]
as key representatives. As argued in [33], this also includes methods that do not explicitly adopt a min-max game formulation but implicitly learn from its adversarial dynamics, such as classic approaches like Apprenticeship Learning [1. 2] and Max-Ent IRL [52, 533. Recent work [39] unifies these adversarial methods through the concept of Moment Matching (a.k.a. Integral Probability Metric) [26], offering a broader perspective on their underlying principles. Building on this, recent methods further improve ad adversarial IRL by providing sample-efficient policy update mechnisms such as FILTER [40] (resets the learner to expert states) and HyPE [33] (a hybrid-RL based IRL algorithm that trains on a mixture of online and expert data to curtail unnecessary exploration in policy updates). In contrast all these methods, our approach is adversaaaai and features principled stable reward learning.
Non-adversarial IRL. We expand the discussion on non-adversarial IRL methods in the introduction and Sec. 3. Coherent Soft Imitation Learning (CSIL) [44] simplifies the idea of non-adversarial IRL with two-stage procedure: it first extracts a reward function from a max-likelihood policy with reference policy and then trains a policy based on this reward. BC-IRL [41] minimizes the mean squared loss rather than maximizing the likelihood, but with no guarantee on stability. Least-squares inverse Q-learning (LSIQ) [3] penalizes the reward function magnititude and give its theoretical support; PIRO does so implicitly in its practical implementation of reward update constraints. To handle distributional shift due to limited state-action coverage, some methods adopt the model-based paradigm and conservative updates - either on the policy (Offline ML-IRL [51]) or on the reward function (CLARE [49]). In contrast, our PIRO is model-free and leverages online rollouts. Another recent method, SFM [22], minimizes the imitation gap by matching expert Successor Features (i.e., predictions of future state occupancies under a policy). A technically related method is P2IL [43], which applies the proximal point method to stabilize soft Q-function learning under linear MDP assumptions. Our method addresses general MDPs with explicit rewards.
7

---

Table I: Averaged Rewards (five independent runs) on five MuloCo and four Gym Robotics tasks.
Tock
Expert
II
Adv, IRL (Ouline)
AIREL
FILTER
H,PE
Adv, (Oftine)
Non-Adx. Online
ML-IRL
]-RR
Nan-Adv.
Offine
P[.
PIRO
Grin
Aut-va
5926.2
9914
-376.3
28.0.5
923.8
420.7
976.5
3467.2
+354.7
Robveles MaloCo
Hu
418.1
251.4
201.7
717.5
76.3
1947.5
5573.4
+381.5
Walker2d
5524.5
394,4
72.5
3023.0
6794.7
696.1
1054.0
+849.0
Heppa-od
57.8
2593.7
13216
3434.5
3316.4
67
25.8
-173.7
Halichestbbod
12666
221.2
1298.3
20.3
2251.4
0.3
6473.4
9645.
3825.5
11873.2
-0.7
-107.2
-0.3
+714.2
35.6
8.8
51
4.5
5.1
11.9
39
3.0
3.4
25.7
+13.
AutMune MediumDeuse vi
25.9
1.1
1.3
3.4
2.6
19
3.0
34
0.9
1.1
2.9
9.4
+5.0
AutMauee LargeDensee vá
11.5
1.1
1.
34
0.6
1.3
0.5
0.3
0.5
0.2
8.8
+5.8
1062.5
64.1
-8.7
-344.3
-598.0
-65.4
-8667
-751.0
-251.2
-5.3
-6.2
254.0
+200.0
runtime per tieesion
5-146
8-19%
5-16
9-41s
11-76:
13-142
7-376
90-1566
16-83s 68-90s
20-111s
(9611/2s
Note: DAC, CSIL, and PLIL are not evaliaaedd on certain lasks due to computibilit issues cause by version conflicts. Specifically, the current
implementations of DAC, and PºII. are incomparible with the current Gymnasium Roboties suite, while POTI. and CST. are incompatible with the Humanoid version used in testing other algorithms.
Stable Inverse Optimal Control. line of work in inverse optimal control uses trust-region or system dynamics and second-order optimization, limiting scalability, PIRO, in contrast, model-free and relies only on first-order optimization, making it more practical for real-world applications.
Lyapunov-based methods [37, 5, 42] to ensure stability but requires knowledge of
7 Experiments
We focus on the following key performance indicators in the empirical evaluation: (1) reward recovery and policy imitation, (2) learning stability, (3) sample efficiency. We also test PIRO's capability of reward transfer to new environment dynamics and learning state-only rewards We evaluate alogrithms on five MuJoCo locomotion and four Gym--Rboticcs tasks (see Tab. 1). To examine PIRO's capability of real-world problem solving, we additionally provide a real-world case study on an animal behavior modeling task in Appendix E,, where PIRO shows superior performance compared baselines.
Experimental Setup. For MuJoCo tasks, we use the same demonstrations as f-IRL [30] and ML-IRL [50], keeping original hyperparameters except for standardized batch sizes and training steps to ensure fair comparison under identical computational budgets. Robotic tasks use expert trajectories from Minari Offline RL datasets [46]. We use a single expert trajectory per task in order to examine their imitation capability; the only exception AdroitHandPen, where we use 10 expert trajectories instead of one to ensure convergence. Full implementation details, including hyperparameters, network architectures and trajectory lengths, are in Appendix C.
7.1 Reward Recovery and Policy Imitation
The reward performance is shown in Tab. 1. PIRO consistently outperforms matches all baselines across nearly all tasks. The performance gains are especially pronounced in harder domains such as Humanoid, AntMaze, and AdroitHand, where PIRO shows substantial improvements over the best baseline. On average, PIRO demonstrates strong reward recovery and policy imitation. Although PIRO incurs moderately higher computation time per iteration, this reflects its principled stable reward optimization mechanism: the increased runtime stems from controlled updates that ensure stable policy improvement (justified in the next experiment).
7.2 Learning Stability
We investigate learning stability by analyzing the learning curves across all experimental tasks, which are shown in Fig. 4. PIRO consistently outperforms ML-IRL and demonstrates significantly higher stability compared to other baselines throughout the learning process (except slightly weaker performance AntMaz--MediumDenn MediumDense--va. In challenging AntMaze environments, while PIRO exhibits fluctuation, it remains the only method capable of successfully imitating expert behavior, likely due to the complex environment dynamics that cause the failures of other algorithms.
7.3 Sample Efficiency
We assess sample efficiency by analyzing the convergence speed with respect to the environment steps, which can be observed in Fig. 4. PIRO consistently delivers competitive faster convergence speed. Although certain environments our method exhibits lower sample efficiency than some baselines (c.g., DAC on HallCheetah-v4), PIRO ultimately achieves higher final rewards after convergence
8

---

Ant-v4
HaltCheetah-v4
Walker2d-v4
Rewords (x100)
60
120
60
PIRO
80
40
ML-IRL
40
0-
20
HRL
-30
0
80
160
240
320
400
40
80
160
240
320
400
0
0
70
140
210
280
350
AIRL
Hopper-v4
Humanaid-vv
AdroilHand- Pen-v-
GALL
Rewards (x100)
40-
60,
10
FILTER
30
40
5
DAC
20
CSIL
-10
P
40
80
120
160
200
40
80
120
160
200
200
400
600
800
1000
HyPE
AntMaze- Mearree--
MediumDense--vd
AniMaaze- Larieezee
Rewards (c100)
0.38
0.28
0.12
IQ-Leamn
0.24
0.21
0.09
MM
0.14
0.06
BC
0.12-
0.07
0.03
Expert
0.00
0.00
40
160
200
Environment steps (x5000)
Environment steps (x5000)
Environment steps (x50000
Figure Reward curves of algorithms on MuJoCo Incomation tasks and Gym Robotics tasks.
and approaches expert-level performancem, while most baselines far from expert performance after convergence. Moreover, in these environments PIRO demonstrates more stable improvements throughout training.
7.4 Learning State-only Rewards
Grid world
Ground Inth
0.04
As explored in [12], restricting rewards to be solely state-dependent mitigates ambiguity from reward shaping [29], that is, class of reward transformations that yield the same optimal poli- cies, making it impossible for IRL algorithm to identify true reward without prior knowl- edge of the environment. This also improves generalization across MDPs with different dy- namics. Thanks to explicit reward learning, PIRO naturally supports state-only rewards by directly parameterizing re(s), without the ad- ditional modifications required by implicit re- demon-
2-
0.22
0.1
0
2
6
Figure 5: Experiments reward recovery in tasks with state-only rewards. Left: The task is x grid world, where the agent starts from a random initial position (blue circles) with the objective of reaching the target position (red star) via the shortest possible path. Right: The ground truth reward at each position defined as the negative Euclidean distance to terminal state. Middle: The reward recovered by PIRO and the ground truth reward function is highly consistent with the ground truth reward. Cumulative rewards: -9.24 (expert) vs. -8.48 (PIRO).
ward methods [15]. Empirically, strate PIRO's s effectiveness in recovering state- only ground-truth rewards in Fig. 5.
7.5 Reward Transfer
To assess the transferability of the learned reward function, we evaluate whether reward learned under the original environment dynamics can induce effective policy when the dynamics change. LunarLander provides a testbed for this as we can alter its dynamics by 'adding winds" in the simulated physical conditions. As shown Fig. 6, the resulting policy performs well under the modified dynamics, demonstrating that PIRO recovers robust reward functions capable of generalizing across environmental changes.
7.6 Sensitivity Tests
To assess the robustness of PIRO with respect hyperparameters controlling reward update magni- tude, we conduct sensitivity tests on three key parameters: ztarget and its associated scaling factors and y. which govern the adaptive adjustment of the regularization coefficient in Eq. (11). Specifi-

---

Figure 6: Results for reward transfer to environments with altered dynamics. Left panels: Policy behavior learned by PIRO the original LunarLander environment. PIRO succeeds in most cases. Right panels: Policy behavior under PIRO's learned reward function LunarL.andrr with altered dynamics (stochastic wind added). The policy is robust in general, despite some failure cases, c.g., row 3.
cally, we vary one parameter at time while keeping all others fixed. Results are reported in Fig. 7, which suggest that the algorithm is not highly sensitive to the hyperparameters x, y: both can set within the range (I, 2) without significant impact. We also observe that setting the target value glrget within the range (0.1,1 1) generally does not significantly affect the reward performance.
Scaling factor for (fold, ()
1.5, target 0.5)
Bewents 1a330)
20
125
35-
ED
0.24
0.16
125
0.22
0.14
0.00
0.20
0.18
0.12
0.06
0.10
0.10
0.04
Scaling for ((old, (ar plarget
0.5)
128
35-
6o,
60
0.39
0.10
125
0.30
0.14
126
122
0.20
0.10
0.06
Threshold target
1.5, y 1.5)
125
6o
39.5
59.0
126
125-
34.5
34.3
0.30
0.12
0.06
0.32
38.0
33.0
0.88
0.08
0.06
& ddeet
Anbwi
Hospareel
Humanotivve
Waber2ses
UManeDense v
MadunDersent
LargeDeseeet
Figure 7: Sensitivity test for the parameter glarges and its scaling factors x,
8 Conclusion
We propose Proximal Inverse Reward Optimization (PIRO), novel non-adversarial, practical IRL algorithm that stabilizes reward learning by approximating Trust Region Reward Optimization (TRRO) novel theoretical framework guaranteeing monotonic improvement in the likelihood of expert behavior. Experiments MuJoCo and Gym Robotics benchmarks show that PIRO achieves stable training, accurate and robust reward recovery. high sample efficiency, and good reward transfer capability. This work provides theoretical foundation for stabilizing IRL, and we hope it provides a new perspective for designing more robust IRL. algorithms.
Limitations. Despite its advantages, PIRO has limitations. First, while stabilizes reward learning, the overall training stability also depends stable policy optimizer, especially in high-dimensional and
settings. Second, the dependency on on on-policy sampling may reduce sample efficiency in environment interactions, potentially limiting scalability sample-expensive tasks.
Future work. First, improving the efficiency of policy optimization by incorporating resets expert states [40, 33] may substantially reduce computational cost. Second, exploring alternative policy alignment measures beyond likelihood (e.g., statistical divergences other than KL) may open new paradigms for stable IRI. Finally, on the application side, extending PIRO to real-world scenarios such as learning reward models and policies for aligning large language models with human feedback offers promising path to improving agent performance in practice.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by a locally commissioned task from the Shanghai Municipal Govermment.
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticshhip learning via inverse reinforcement learning. In Proceed- ings of the lwenty-first international conference on Muchine learning, page 1, 2004. 7,
[2] Pieter Abbeel and Andrew Y Ng. Exploration and apprenticeship learning in reinforeement learning. In Proceedings of the 22nd international conference Machine learning, pages 1-8, 2005. 7
[3] Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping 7hao, and Jan Peters. Least squares inverse q-lcarning.
In Sixteenth European Workshop on Reinforcement Learning, 2023. 7
Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. 53rd IEEE conference on decision and control, pages 491 1-4916. IEÉE, 2014.
[5] Kun Cao and Lihua Xie. Trust-region inverse reinforcement learning. IEEE Transactions on Automatic Control, 69())1037710444, 2023. 8
[6] Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing
Systems, 34:965 979, 2021.
[7] Jiayu Chen, Tian Lan, and Vancet Aggarwal. Option-aware adversarial inverse reinforcement learning for robotic control. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5902-5908. IEEE, 2023.
[8] Yang Chen, Libo Zhang, Jiamou Liu, and Michael Witbrock. Adversarial inverse reinforcement learning for mean field games. In Proceedings of the 22nd International Conference on Autonomous Agents and Multi-agent Systems, 2023. 7
[9] Yang Chen, Xiao Lin, Bo Yan, Libo Zhang, Jiamou Liu, Neset Özkan Tan, and Michael Witbrock. Meta- inverse reinforeement learning for mean field games via probabilistic context variables. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pages 1407-11445, 2024.
[10] Chelsea Finn, Paul Christiano, Pieter Abheel, and Sergey Levine. connection between generative adversar- ial networks, inverse reinforcement learning, and energy-hased models. arXiv preprint 161 2016. 7
[11] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
[12] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference Learning Representations, 2018. 7,9
[13] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. In International Conference Learning Representations, 2023. 3
[14] Bolin Gao and Lacra Pavel. On the properties of the softmax function with application game theory and reinforcement learning. arXiv preprint 1XX4:1700.0000: 2017. 22
[15] Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq le-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028 144028-4099 2021. 2,
4, 5.9
[16] Gaël Gendron, Yang Chen, Mitchell Rogers, Yiping Liu, Mihailo Azhar, Shahrokh Heidari, David Arturo Soriano Valdez, Kobe Knowles, Padriac O'Leary, Simon Eyre, et al. Behaviour modelling of social animals via causal structure discovery and graph neural networks. arXiv preprint 2rXiv:2312.14333, 2023. 32
[17] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy based policies. In International conference machine learning, pages PMLR, 2017.

---

[18] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018. 3,7
[19] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[20] David R Hunter and Kenneth Lange. A tutorial on mm algorithms. The American Statistician, 58(1):30-37,
2004. 6
[21] Maximilian Igl. Daewoo Kim, Alex Kuefler, Paul Mougin, Punit Shah, Kyriacos Shiarlis, Dragomir Anguelov, Mark Palatucci, Brandyn White, and Shimon Whiteson. Symphony: Learning realistic and diverse agents for autonomous driving simulation. 2022 International Conference on Robotics and Automation (ICRA), pages 2445-2451. IEEE, 2022.
[22] Amav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Irina Rish, Glen Berseth, and Sanjiban Choudhury. Non-Adversarial Inverse Reinforeement Learning via Successor Feature Matching, 2025. 7
[23] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pages 267 2244, 2002. 4
[24] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations. 2018. 7
[25] Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020. 2,4
[26] Yujia Li, Kevin Swersky, and Rich Zemel. Generative moment matching networks. In International conference on machine learning, pages 1718-1727. PMIR, 2015. 7
[27] Gergely Neu and Csaba Szepesvári. Apprenticeship learning using inverse reinforcement learning and gradient methods. In Proceedings of the -hind Conference on Uncertainty in Artificial Intelligence, pages 295-302, 2007. 2
[28] Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 663-670, 2000. 1
[29] Andrew Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning, page 278-281 287, 1999. 9
[30] Tianwei Ni, Harshit Sikchi. Yufei Wang, Tejus Gupta, Lisa Lee, and Ben Eysenbach. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pages 529-551. PMLR, 2021. 2, 4, 8, 28, 29
[31] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for ap- prenticeship learning. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems, pages
2014. 2
[32] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020. 2, 4, 5,
[33] Juntao Ren, Gokul Swamy, Steven Wu, Drew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforce- ment learning. International Conference on Muchine Learning, pages 22422-42444 PMLR, 2024. 5,
7, 10
[34] Mitchell Rogers, Gaël Gendron, David Arturo Soriano Valdez, Mihailo Azhar, Yang Chen, Shahrokh Heidari, Caleb Perelini, Padriac O'Leary, Kohe Knowles, Izak Tait, Simon Eyre, Michael Withrock, and Patrice Delmas. Meerkat behaviour recognition dataset, 2023. URL https: www.aawwww......aaaao..ooeeooore aba//306. 11326.
[35] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015. 2, 3, 4,
[36] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
12

---

[37] Yu Shen. Weizi Li, and Ming C Lin. Inverse reinforcement learning with hybrid-weight trust-region optimization and curriculum learning for autonomous maneuvering. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7421-7428. IEEE, 2022. 8
[38] Hao Sun and Mihaela van der Schaar. Inverse reinforcement learning meets large language model post- training: Basics, advances, and opportunities. arXiv preprint 13Xi::5577.13588 2025.
[39] Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and matching: game-thor theoretic framework for closing the imitation gap. International Conference on Machine Learning.
pages 10022- 10032. PMLR, 2021. 1, 3,7
[40] Gokul Swamy, David Wu, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Inverse reinforcement learning without reinforcement learning. In International Conference on Machine Learning, pages 33299-
33318. PMLR, 2023, 1, 10
[41] Andrew Szot, Amy Zhang, Dhruv Batra, Zsolt Kira, and Franziska Meier. Be-irl: Learning general- izable reward functions from demonstrations. In The Eleventh International Conference on Learning Representations, 2023. 7
[42] Samuel Tesfazgi, Leonhard Sprandl, Armin Lederer, and Sandra Hirche. Stable inverse reinforcement learning: Policies from control іуаршnоv landscapes. IEEE Open Journal of Control Systems, 2024. 8
[43] Luca Viano, Angeliki Kamoutsi, Gergely Neu. Igor Krawczuk, and Volkan Cevher. Proximal point imitation lcarning. Advances in Neural Information Processing Systems, 35:24309-2444 2022. 7
[44] Joe Watson, Sandy Huang. and Nicolas Heess. Coherent soft imitation learning. Advances in Neural Information Processing Systems, 36:14540 14583, 2023. 2.7
[45] Zheng Wu, Liting Sun, Wei Zhan, Chenyu Yang, and Masayoshi Tomizuka. Efficient sampling-based maximum entropy inverse reinforcement learning with application to autonomous driving. IEEE Robotics and Automation Letters, 5(4):5355 5362, 2020. 3
[46] Omar G. Younis, Rodrigo Vicee-Vieette, John U. Balis. Will Dudley, Alex Davey, and Jordan K. Terry. Minari, September 2024. URL https:///ooi http://ooooorr////// 10. $281/zenoda 13766625. 29
[47] Lantao Yu, Jiaming Song, and Stefano Ermon. Multi- agent adversarial inverse reinforcement learning. In International Conference on Machine Learning, pages
PMLR, 2019. 7
[48] Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. Advances neural information processing systems, 32, 2019. 7
[49] Sheng Yue. Guanbo Wang. Wei Shao, Zhaofeng Zhang, Sen Lin, Ju Ren, and Junshan Zhang. Clare: Conservative model- based reward learning for offline inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, 20233. 7
[50] Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. Haximum-likelihoee inverse reinforcement learning with finite time guarantees. Advances in Neural Information Processing Systems, 12210022-10033 2022. 2, 4, 7, 8, 29
[51] Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong. When demonstrations meet generative world models: A maximum likelihood framework for offline inverse reinforcement learning. Advances in Neural Information Processing Systems, 36:65531 -6665551-66555,,, 2023. 2,7
[52] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433-1438. Chicago, IL, USA, 2008. 7
[53] Brian D. Ziebart, J. Andrew Bagnell, and Anind K. Dey. Modeling interaction via the principle of maximum causal entropy. In Proceedings of the 27th International Conference on International Conference on Ma- chine Learning, ICML' 10, page 1255 1262, Madison, WI, USA, 2010. Omnipress. ISBN 1781605589077.
3,
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We end the introduction section (Sec. 1) with summary main contributions. These contributions are introduced in more intuitive and condensed manner in the abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitations of our proposal approach in the conclusion section (See. 8).
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model cell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: We explain the ideas of proofs in the main text of the paper (see text right below Theorem 3) and provide assumptions and full proofs in Apendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The contribution of this paper is a new algorithm with empirical evaluation. The code is submitted supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We submit the code the supplemental material with readme file that indicates the detailed instructions for running the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpccccppccccoecccc.c.ccc..cttt gugucdg Cooeeuumiii wwabbbbmsssoon wwwiisssssssieooooeooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We briefly introduce the experimental settings in Sec. 7 in the main paper and provide detailed settings (network architecture, hyperparameters, data collection, and pre pre-processing and training procedures) in Appendices and E.1. The code submitted as supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct multiple independent runs for each experiment (with different seeds) and report the mean value of standard deviations in figures and tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide this information in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https::/neut /ttpp////////...////// cc cc/public/ httpa////hhisshiellnaaaaee
Answer: [Yes]
Justification: Ethics.
The research conducted in the paper conforms with the NeurIPS Code of
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consider- ation due laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents a novel inverse reinforcement learning algorithm that advances the research in machine learning, which we feel has no negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example, by requiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the original paper for the real-world meerkat behavior dataset and give a URL in Sec.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is submitted supplemental material with readme field for detailed instructions to run the code.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20

---

Appendices
A Proofs
A.1 Proof of Proposition 1
Let us first show that (()) = E [log _e = e (re Basnsn = J re)
( Ta) (Eq. (5a)). Let de (s) denote the state distribution under policy æ. Note that dj = 1)
where 1) the fixed initial state distribution.
(()) E [log To (a|s)
E 127 (s, a) Vi (s)]
EpT [ro a) + BB (s,a) (a. [V - (s)
00
Ep' (ro a)]
Σ
7 /Band (awT((((ss))) VTTo (s) - p(laa) VT (s)]
(12)
2 Σ
1 [ro a)]
> 'I Baa
x (V
2+ Band [V (()(
t=
E [ro a)] E B = [V7 (s)l
EPT [ro a)] - Eg IiTe (s)] J() J(mo: ro). Note that in Eq. (12), we omit the constant policy entropy H(Tx) in (oT
We next show
Vol(O) E (err a)] To [Vero(s, a)]
in Eq. (6a). Let begin with investigating the gradient of Q (st. a:):
VoQ (Sc, as)
Vero(s,, ar) + Σ P(sc+1||c, az) Vol (s(+1)
(s(+1)
(6) Vora(si,ay ax) + Σ P(Sr+ at)
Σ
exp(( (S at+1)) (8:41, (13)
exp(Q" (8)+1,
8141
Vero(s., ar) + Σ P(sc+1|se az)
Σ To S541 QT
Equality (a) uses the soft Bellman equation, while Equality (b) follows the energy-based formulation of the policy. Notably, both VoQue (St, ax) and Vo (sx) exhibit recursive forms, where the
gradient (org a.) accumulates as an expectation alongside the expansion of Q and V. Continuing this recursive expansion, we derive:
V T (Sc, a.) = E 2:11 P()))
Vero((),
(14)
a141 ~ no (-144)))
V VoV = E
k- (
at [1(())) 8 P(- ac)
(15)
Then, have
Ve 1 (so) = Eg (e)))
(or
(16)
84+1 d0
21

---

Finally, according to Eq. (12), we have
Vol(O) p (oro a)] Bo VoV. (so)]
8
Epr (s, a)] a ~
2 (s)
(17)
0
p (oro a)] Ep (oro(ss,))
A.2 Proof of Theorem 3
We begin by presenting some useful lemmas that tell us how much the policy discrepancy (Lemma 5), state margin discrepancy (Lemma 6), log policy discrepancy (Lemma 7). Q and V functions (Lemma 8), log policy (Lemma 9), and the expected entropy discrepancy (Lemma 10) grows based on the reward difference. In all these lemmas, we use the following notations:
Ti (s, a) and (s, a) are two reward functions,
Ti (-ls) and (-(ss) are optimal policies under r1 and 72 under the MaxEnt RL framework, respectively.
= max(s.a) (s, a) (2 a) denotes the reward difference.
[s) R, E {1,2}. s S, E A.
Lemma 5, The total variation distance between Ti (-1s) and (-(s))) is upper-bounded as follows:
1 Σ
(mlss) m2(a|s)] S |Ale
Im (-1s) x2(+1s)]1 2
2(1 (1) (18)
Proof. We start by analyzing the sensitivity of the policy to changes in soft Q-function. The difference in क (als) and (a)s)) arises from the difference in their respective soft Q-functions, Q1 a) and Q2 a). Expanding the policies gives:
Ti (als) - x2(ass) = exp (s, a)) En exp (Q1 (s. a'))
exp (Q2(s, a))
(19)
E. exp (s,aa))
This softmax-like function is 1 L-Lipschitz continuous [14] with a being the temperature in the energy-based model (w.l.o.g., assume
in this paper). This means small changes in Q lead to proportionally small changes in the softmax output. This allows us to approximate the policy difference for small deviations in Q. Thus, the policy difference be bounded as:
| (als) T2 s (Q, a) Q2 a))
(20)
Summing over actions, the (doubled) total variation distance becomes:
|| - (2(-s)] = Σ (als) - (2(a)))| <
Σ
(Qi((s, Q2((s,
(21)
aEA
aEA
We bound (Q)(s,a) (s. a) by:
max 9.a
[Q (s. a) Q2(s, a)l
< max (s.a)
Eg ((-1(aa)) (V')s - V2(('))
Se max
log
exp (s', - -log Σ exp (Q2(s')
(22)
of
(0)
6 + max
Q1 (s. a) Q2 a)l,
(s,a)
where inequality (a) uses the fact that for any two sets of values {x;} and {vi).
log Σ exp(x;) - log Σ exp( S max (x; Val.
(23)
22

---

Rearranging Eq. (22) and performing some algebra yields:
max [Qi a) - Q2(s, a) S 1-
(s.a)
Finally, according Eq. (21) summing over action space introduces scaling:
|Ale
| (+ T2(+1s)ll [A] a) Q2(s,a) - 1-7
(24)
Lemma 6. Let d, (s) and ds² denote the state marginal distributions at time under each policy. starting from the same initial distribution 17. Then, for any 2 0.
1 =2lld dz 1|1 < LDTV (TI,
Drv(d ,4²)) =
(25)
where
D (T),T2) = max Dww(Dri(--Ssss) T2(+1s)) = 1 1-7 (Lemma 5)
(26)
the worst-case total variation distance between σ2 and n over all states.
Proof. We proceed by induction on t.
Base case (t 0): At O, du² di (the initial distribution),
| do 1l = 0.
(27)
which satisfies the bound.
Inductive step: Suppose that at time 1,
Drv(d S LVD (TI,
(28)
We now show that the same holds at time t +
The state marginals evolve according to
ds') = Σ (s) Σ ((((ss)
Thus,
d() (s') (=)==>>> -
((( (al)) ((s))) a()
(29)
Taking the L norm and using the triangle inequality,
1 diill
Σ (s) (s)m
(30)
Now expand the difference inside:
d (s)ss(aas) di (s)m (a|s) = (d72 (s) - df (s)) + de (s) (TT - (als)). (31)
Using triangle inequality again:
(s) di (s)tT(-1s)]] < (d) - de' (s) + di (S) T1(1s)]l1
(32)
Thus,
| - d S Σ (d - ds) + Σ (s) -
(33)
The first term of the right-hand side is simply
| d. ||1 = 2Dv ,d
(34)
23

---

and the second term is at most
TD (22
(35)
since of is a distribution and || (-1s))] S TV (T2, T1) for all s.
Therefore.
1
DTv (Ptt1: Gif
- Drv(d,, + DTV (T2, TT).
(36)
Applying the inductive hypothesis:
Drv(d, ,d,)) S IDTV (TI, T2),
(37)
we conclude
Drv Drv G21 & (t + (m)
(38)
Thus, the claim holds for t + 1, completing the induction.
Lemma 7. Under MaxEnt RL, ler æ (als) and πT2 (a(s)) be two policies defined over a finite action set A, induced by reward functions Tı (8, a) and (2 a) respectively. Assume that for all s, a. the rewards are uniformly bounded by constant R > ie., (s, S for 1,2. Let maX (s) - a2). Then, the log-policy difference is bounded as:
2e
log m (-|s) log [2(+)) S 1-9
(39)
Proof. We start from the softmax policy expression:
log n. (als) Qi(s, a) log Σ
exp(Qi(s,a')), i=1,2
(40)
a'EA
So the difference is:
log Tz (als) - log n2 (a|s) = Qi(s,)) Q2(s,a)
(41)
log exp(Q1(s,a') - log Σ exp(Q2(s.a'))
n'
Following from the Lipschitz continuity of the log Σ exp(-) function with Lipschitz constant under I -norm, have
log Σ exp((i(s,a)) log Σ exp(Q2(s, a)) s - -12llQ = 1 --7
(Lemma 5).
a
(42)
Combining everything:
log ti (als) log (a(ss)
(s, a) Q2(ss a)l + log Σ (s, - log Σ exp(Q2(s,a))
(43)
< 2|[Q1 - 2221oo
26
1-?
Lemma 8. Under MaxEnt RL, we have
R + ? log AA
1101 <
(44)
1 1- ?
log [A]
| <
(45)
1 7
24

---

Proof.
V(s) I log Σ (xp(Q((s,)) - 110 + log |A].
(46)
Since Q(s, a) = r(ss, a) + PB we have
< s 20 + log [A]).
(47)
Rearranging, we obtain Eq. (44) and hence Eq. (45).
L
Lemma 9. Under MaxFnt RI, we have
2R + (1 +) log A
|| log 11 S
(48)
1 1-7
Proof. This directly follows Lemma 8 because
log (aass)] (Q(s, a) V(s)) & (Q(ss a) + [V(s)).
Lemma 10. The discounted entropy difference is bounded by
E
7 log 1T (ar(s:)
16,
7a log T (arls:)
L==
t=
(49)
2|
(2R + (1+1) 7) log |Al
(2R + (1 + 7) log [A4) Alyr
7)
(1-7)3
(1-7)4
Proof. We express the expected discounted sum as
0c
log क (arls))
Σ
x Fxxd
Σ ( log ((as)
Now consider the difference:
Σ m (als) log कं (a|s)) 1
Σ (als) m2(als)
Bana?
Band
1=
y
E E
Σ mm (m)s) log (2(aa))a)))))))))))))(())) - mi (als) log क. (als))
0
(first term)
[Beaat= - -Band")
Σ n. (als) log (als)
(50)
(second term)
We bound each term:
The first term is bounded by
(first term) < E Ex 2 [11s) x1(+ss)] log T2 loll
|| log Ty (a|s) - log ग (als) [a]s)]]]] [|]]
AAle 2R + (1 77 (+77)oog log |A|
2 A|e
(Lemmas 5,
(51)
1-7
7
(2R (1 7) log AA)]
A|e 1-2 7
(1 71-
25

---

The second term is bounded by
(second term
- d, d, |11 |||ll| log 10
te |A|
2R + (1 (1+7) log |A]
$ 1-9
(Lemmas 6, 9)
1-7
(52)
(2R + (1 + 7) log (A)]AEe (1 7)2
t.
Summing over i and applying Et 70
1 and Σt 7't (1-23² completes the proof. 1-7
We next prove Theorem 3.
Proof. Substracting ((Ouew) from to (fuew) gives
Ce (Onew) - E((0 = Ex'
V8oow (s) V: A0old
(s) +
Qracee (s, a)
Qrawew new (s, a
(53)
20 bocause Tenow is optimal rowww
S R
(s)
Vironaad (s)
To bound Eag (Onew) ((Onew), it suffices to bound V 70rmw (s) Vi (s). To do so, us first investigate the definition of V (s) with moe optimal re:
20
V
E
Σ 7 (r(s,,a)) log (a,ls:))
which indicates that the value function Va (s) can be split into two terms:
1. Reward term: Eng IEro ('r az)]-
2. Entropy term:
-Ero EEe 7¹ log To
Thus, we can decompose
V
(s)
ooold (s) into two terms: renwww
Vreuew
Runow renow
(s)
V
Roold Frewww
(s)
rowardd (s) + Dennopy (s),
(54)
where
Areward(s) = Eroeyer
7 Tow.. (sc, 8o = 8
ET
o rrow (St, an) =
(55)
x
entrryy (s) Emogs
Σ
d log To (2/ls)
E Brewew
Σ
7 log To (a. So
1=0
(56)
We first bound the term Arwaards):
26

---

A Arewadd (8)
7p
Σ dieuow (s)
Tno (als)r nnow as)
Σ
dr Told (s)
Taan (a (a(s)ro... a)
1=0
Σ
dp
Bunn (8)
Σ
(Newe (als) Toal (a|s)) rn (s.
Σ
Toew (s)
ood (s) Σ Toai (als)rommm (s)
(triangle inequality)
7
de Bnne (s) .2R2DVV (T (-|s), Toaia (-1s) + Σ 7 Rllld To -d, 1|2
(Lemma 6)
(2RDDV TTre. Toma + 21RDTV (Te T
(2((1 11RDDV max
(TT TOss))
RILAKEO (Onow)
Σ 1=
7
(t 1 1)
7-7
Al Goaa
0 new)
Σ 2 Σ gd
1=0
1=0
R R|Aleaaa (Onww
7
+
1
(1 7)-
1-7
A. (m
(57)
71
The bound of the term entroy (s) directly follows Lemma 10:
2|A
(2R + (1+ log [A)A
A entropy (8) v
(1 7)2
(1-7)3
(58)
(2R + (1 + 7) log [A)] (1 7)
COaia ( (new)).
Finally, combining Eq. (57) and Eq. (58), we complete the proof by Cowi (Onew)
e(0new)
< EgNPTT
V.Tooww (s)
V- (s)
(59)
< A roward (s)
+ entropy (8) (Eq. (58)+Eqq (57))
2|A1
(5 A| + (7 - 2) log |A|
cousa (Onew) )-
7)2
(1 74
B A Unified View of Non-adversarial IRL
Let Cov (Ng (2)))))))) = (x) x2(x)] - Epce) [Mz (x)] Eppxo) [x2()) denote the co- variance of two functions Ki (x), (x) under the distribution p(x). We first show an equivalent expression of (()).
27

---

Lemma 11. The likelihood objective has the following equivalent expression:
{()) = Eppe ( a)] - To (ra (s, a)]
pT* (s.
a
p (s, a) pro (s,
= E
ro (8, a) - E
x Ep [ro(s,
(5)(())
(60)
- (s, a)
po
Voro(s.a))
pT (s. a)
We next show that the KL--based f-IRL [30] essentially maximizes the likelihood of expert demon- strations (minimize the imitation gap). Recall from the main text that f-IRL assumes a state-only reward function, (s),,, and seeks to match the expert's 's state marginal distribution by minimizing an f-divergence objective:
Ly( ==
(61)
where p² (=((s denotes the state marginal the occupancy measure such that ps a) = (s))sa(()sss) It has been shown in [30, Appendix A2] that if Dy is taken as the KL divergence, then ( can be reduced the following analytical form:
(0) = (+)
Exer
p
Pawr(ro(s))
pro (s)
(62)
p (s)
(x)
E E
(r) (Rwy [r)(s)]]
(s)
where pa* denotes the trajectory distribution under the reward rg and Eg [.] denotes the expecta- tion w.r.t states over the cumulative state visitation frequency determined by given trajectory. To show that KL-based f-IRL essentially maximizes E(O), it suffices to show that Eq. (62) is propotional
Eq. (60). To proceed,
first notice that or>p"o(r) [[]] = (1 7 (eil as both
represent the state marginal of the occupancy measure. Given this equivalence relationship, we can reduce the second term the right-hand side of Eq. (62) the following term:
(s)
E BxmpTo(T)
B
x E Ee (ror((o(s))
pa (s)
(5()s
(63)
= (1 - (1->)EampT0)) T0
x (s) [ro(s)]
pT (s)
= (1 - (1-7) - (e) (r)(s)]
We next investigate the first term in the right-hand side of Eq. (62):
p (s)
(r)
E
E E [ra(s)]
(s)
= (r)
- Cov-
1 (s)
Voro(s))
+ Bxur
(F((s) (s)
V Voro(s)
(s)
(64)
(T
Cov+
p (s)
Voro(s))
+ E To)
ExNu
p (**(s) (s)
V Voro
(s)
(s)
Vore(s) + (1 - (1-7)) (x Vere(s)).
(s)
Combining Eq. (63) and Eq. (64), we have
p
voL,(0) = Cov ((()
Vora(s)
+ (1 - -1) - (err(s)) - Bo) (ro(s)
(2*){(s (s)
Cov (s)
Vore(s)
+ (1 - (1-7)Cov,
pr(s)) (T)(s)
Vere(s)
p
(s)
pTTR
(s)
--7Covppore
Voro(s)
(s)
(65)
28

---

Therefore, if the reward is state-only, i.e., ro(s), we have VoL,(0) x ----- Vol((). This completes the proof for Eq. (6b) in the main text.
C Detailed Experimental Setup
C.1 Experimental Setup for PIRO
Training procedure is given in Alg. 3. Network architecture and hyperparameter setup for each task are listed in Tab. and Tab. 3.
Table 2: Network architecture and hyperparameter setup for MuJoCo tasks.
Hopper
Walker2D
Ant
Humanoid
Cheetah
Expert demo. (s-a pairs)
1000
1000
1000
1000
1000
Reward network (hidden layers)
128
128. 128
128,
128,
128.
Batch size (s-a pairs)
256
256
256
256
256
Reward learning rate
lc-4
lc-4
Ic-4
lc-4
lc-4
SAC epochs per iteration
5
5
Entropy coefficient
0.2
0.2
0.2
0.2
0.2
Threshold slarget
0.5
0.5
0.5
0.5
0.5
Scaling factor z, for غ
1.5
1.5
1.5
1.5
1.5
Scaling factor Ye for é
1.5
1.5
1.5
1.5
1.5
SAC rounds per iteration
Reward gradient steps per iteration (m)
Table 3: Network architecture and hyperparameter setup for AntMaze and Adroit tasks.
AntMaze-U
AntMaze-M
AntMaze-L.
HandPen
Expert demo. (s-a pairs)
700
1000
1000
2000
Reward network (hidden layers)
128, 128
128, 128
128, 128
256, 256
Batch size (s-a pairs)
256
256
256
256
Reward learning rate
cc-4
lc-4
lc-4
3e-5
SAC epochs per ileration
Entropy coefficient a
0.2
0.2
0.2
0,2
Threshold starget
0.5
0.5
0.5
0.5
Scaling factor x, for
1.5
1.5
Scaling factor ye for ē
1.5
1.5
SAC rounds per iteration (k:)
Reward gradient steps per iteration (n)
C.2 Pre-trained Expert Policy Model and Expert Demonstrations
The sources of pre-trained policy models or offline trajectory datasets for experts are provided in Tab. 4. MuJoCo tasks, we use these high-quality pre-trained policy models sample expert demonstrations. In Robotic tasks, we directly use the expert trajectories from the Minari Offline Reinforeement Learning datasets [46].
Table 4: The sources expert policies or demonstrations,
Task
Source
MuJoCo Tasks
Same expert policies used in f-IRI. [30] and ML-IRL [50]
UMazeDense
https://ninari. . farama. http//attssttt//////aaaaae/e/eee
MediumDense
http://mmmaraaaaaaaaaa https://minari.ff arama. www.///aaaaee//////aaaaaa///////
LergeDense
Adndien
farama. httpa///aaaaaaaees/////e//////ee
29

---

D
Additional Experimental Results
D.1 Hardware Information
Hardware specifications are provided in Tab. 5.
Table 5: Hardware configuration used in experiments.
Hardware
Specifications
CPU GPU
AMID EPYC 7713 64-Core Processor @ 2 GHz
NVIDIA A100-SXM4-80GIB @ 1215 MHz
Memory
2 TB
D.2 Comparison Between Theoretical and Adaptive on CartPole
To further validate our theoretical analysis, we conduct an additional experiment on CartPole, where |A| = 2, = 1, and 0.9. According to Eq. (7), we have that the exact theoretical value C' Rs 111,373.55.
We compare this theoretical C against the adaptive C method (bounded [0.001, 10). As shown in Figure 8, the adaptive method substantially reduces KL divergence throughout training (mean 226.3 vs. 648.9) while also achieving significantly higher final rewards, both undiscounted (313.6 vs. 19.1) and discounted (10.0 vs. 7.1). When using theoretical C', the reward performance does improve within the acceptable training range, but the progress is neither as fast nor as stable as with the adaptive C.
These results highlight the practical benefit of adaptively adjusting C during training, despite the theoretical guarantees provided by the closed-form expression. In particular, adaptive C' allows stable and sample-efficient learning while avoiding the instability caused by the overly large theoretical constant.
(A) Divergence Over Training
(B) Undiscounted
Adaptive C
500
Adaptive C
Divergence
800
Thcorstoal
Undiscourtedd Reward
400
Thoorctical
Expert
400
200
100
2
Environment Steps (x10')
Enwironment Steps (x10')
(C) Discounted Reward
(D) Performance Summary
848.8
Theordool
Disooutted Reward
600
500 D
Velse
400
6-
Adeptive
200
Theorssoal
Expert
10.0 7.1 10.0 Discounted Reward (Finel)
0
2
3
KL Divargnnce (Ween)
Undiscounted Reward (Final)
Environment Steps (x10")
Figure 8: Comparison between theoretical and adaptive C' on CartPole.
E
A Real-World Case Study: Meerkat Behavior Modeling
E.1
Dataset Details
As real-world case study, we apply PIRO an animal behavior modeling task using a dataset of twenty 12-minute annotated videos capturing the spatial-temporal actions of meerkat mob in zoo
30

---

Camera view the entrance and forag-
Camera view the mound and backside of the enclosure
ing area
Figure Example images of camera views.
habitat [34]. To obtain the meerkat behavior, Rogers et al. [34] used two GoPro Max cameras set on the back wall of the meerkat enclosure, focusing two hubs of activity (Fig. 9). The current zone, coordinates, and behavior of every visible meerkat are labeled for every timestep. Fig. 10 illustrates the full set of behaviors. In addition, each meerkat is identified by unique identifier during a sequence, keeping track of the same individuals. The heatmap of meerkat's 's activity is shown in Fig. 11 and the region division for each camera is shown in Fig. 12.
Allogrooming.
Carrying pup.
Digging.
Foraging.
Grooming.
High sitting/standing.
Interact with object.
Interact with human.
Interact with pup.
Low sittin/sstanding.
Lying.
Moving.
Playfighting.
Raised guarding.
Sunbathing.
Figure 10: Fifteen types of the meerkat behaviors.
Entrance & foraging Mound & backside
Entrance & foraging
Mound & backside
Figure 11: The frequency of meerkat activity in vari- ous regions corresponds the heatmap from the cam- perspective. The areas where meerkats fre- zones. quently active are highlighted.
Figure 12: Different colors are labelled for each area to visually illustrate the division of meerkat activity
31

---

E.2 Experimental Results for Policy Divergence Reduction
The dataset includes 25 discrete actions (15 behaviors + 10 actions the represent moving between zones in the habitat) and state representations based on zones (10 total) and social context (counts of close and distant neighbors). The goal is to learn behavior model that predicts the actions of an individual meerkat, assuming a shared policy across individuals [16]. We extract independent demonstration trajectories of 30 consecutive transitions per individual.
Since ground-truth rewards are unavailable in this real-world setting, we evaluate policy imitation using frequencies of transition across habitat zones. Visualizations of the expert's frequencies, PIRO's outputs and baseline results are provided in Fig. 13.
PIRO consistently outperforms baselines in learning stability, reflected in its lowest error rate. AIRL and IQ-Learn also demonstrate low errors, but these errors remain noticeably higher compared to PIRO. This highlights PIRO's capability to reproduce meerkat trajectories with high similarity.
14%
Door
Lan Steas
I. 1.56%
11.6
Runt
0
Door 0,90
Lant' Slats 0.93%
1,00
0.30%
1,34%
16.24%
8, 1.14%
0,83%
18.45%
(a) Expert
(b) PIRO (weighted mean error: 5.5 %
Anskgroun
Mooegrous
2dh 11.38. CIS
.00%
10.1 14%
Boor
Left Stors
1,0%%
Mound
Rigit Stick
Dner
Lan' Sticks
8.97%
0, 0,79
Vound
Right 50ick
0.50
1.413
27
WOR-
13:9
12.45%
9.3%
Sand
8.55%
28. 28.52%
19.90%
(c) AIRI. (weighted mean error: 8.7%)
(d) BC (weighted mean error: 9.3%)
no
Bechgroun
1.59%
Groping
0.31%
Sookgroum
1.932
555
Kank Shiks
0.55%
0.59
Kiak
1.792
Moking
3,36%
1.5%%
0,52
Motror
64.8%%
Sand Area
39. 1%
(e) GAIL (weighted mean error: 11.8%)
(I) f-IRL (weighted mean error: 16. 1%)
nondonn 5.30%
5.90%
16.79%
3,80
Len Siete 3.52%
11-0%%
Aoonn
17.66%
3.3
Door 8,867
Lann Shees
7.93%
0.90%
5.83%
11.07%
Sand Ares
(g) FILTER (weighted mean error: 10.0%)
(h) MM (weighted mean error: 10.5%
500k-roum
14.48%
14.2 1%
Boar 0,59%
62
LLil Slick
0.50%
0.59
Runi Styko
150000
Boor 0,34%
LLit' Sticks Vound
3
Raik 30uka
Mon 3,86%
35.9 9 3%
a. 9
Mai
3.50%
Sand
6.3%%
(i IQ- Leam (weighted mean error: 9.8%)
(i) ML-IRL (weighted mean error: 11.3%)
Bazkgroun
10%
Foroing
20.97%
Ronegrrum
3023
38%
Stess
0.59%
Figkt S0oks
0,535
Ri Sbaks
80%
Mohing Armo 23.9
0.2%
0,70%
Mciing 13.97%
0.60%
15.07%
Sand Anna
Sand Arno
10.07%
0.70%
(k) HyPE (weighted mean error: 14.6%)
(l) P2II. (weighted mean error: 13.0%)
Figure 13: Regional visitation frequency map generated by analyzing real meerkat trajectories alongside those produced by algorithms. PIRO achieves the lowest weighted mean error.
32


Reviews:
[{"review_id": "jnZTfm7jAk", "reviewer": "Reviewer_zT3i", "summary": "PIRO performs IRL in the spirit of conservative policy iteration (CPI), where the reward update is bounded to ensure the policy update is bounded like in other CPI methods like NG, NAC, TRPO and PPO. The reward objective is a somewhat hacky saddle point objective that combines the agent's return, a proximal term, and approximate function-space regularization.", "questions": "[Q1] How sample-efficient (w.r.t. environment samples) is PIRO compared to SOTA off-policy methods, such as DAC, P2IL and CSIL?\n\n[Q2] Taking expert performance from Table 3 of What Matters for Adversarial Imitation Learning? Orsini et al, \n\n| Environment | Random | Expert|\n|-------------|---------|---------|\n| HalfCheetah-v2 | -282 | 8770 |\n| Hopper-v2 | 18 | 2798 |\n| Walker2d-v2 | 1.6 | 4118 |\n| Ant-v2 | -59 | 5637 |\n| Humanoid-v2 | 123 | 9115 |\n\nThe results in Figure 3 look nowhere near optimal, since they are in the hundreds not the thousands. Am I missing something?\n\n[Q3] Following from Q2 somewhat, why in Figure 3 does PIRO improve a lot in the first 1-2 updates and then remain largely flat? Shouldn't there be monotonic improvement according to the theory?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**[S1] Novel look at proximal reward updates.** To my knowledge, performing CPI with IRL in this way is novel and makes sense. \n\n**[S2] Beautiful figures.** Figures 1 and 2 are very informative and nicely made. Good job!\n\n**[W1] Ignores off-policy performance methods.** I am not familiar with HyPE, but I am aware of Discriminator Actor Critic (DAC), Proximal Point Imitation Learning (P2IL), Coherent Soft Imitation Learning (CSIL) as very effective off-policy IRL methods. It feels like these methods have been ignored as baselines because PIRO is on-policy?\n\n**[W2] Implements off-policy baselines with on-policy variants?** The authors say 'PPO is also used for policy optimization in all IRL baselines except HyPE, which uses soft actor-critic (SAC)'. IQLearn is also based on SAC. Was a PPO version of IQLearn implemented instead?\n\n**[W3] Results reported in \"updates\" rather than well-established environment steps.**\nFigure 3 reports performance in terms of updates rather than environment interactions. This is quite implementation-specific and means sample efficiency might vary massively across algorithms from the update definition.\n\n**[W4] Results don't report expert performance.** Figure 3 is missing expert performance; currently, you cannot assess the performance of the algorithm properly.\n\n**W[5] Missing references.** There are a few recent maxent IRL methods missing, such as CSIL as mentioned earlier, and also least-squares inverse Q-learning (LSIQ), an IQLearn variant, and BC-IRL (BC-IRL: Learning generalizable reward functions from demonstrations, Szot et al). CSIL is relevant because it avoids reward learning at all via coherence (in theory). BC-IRL is similar to this work in that it backprops through a policy matching objective and uses PPO.", "was_revised": true, "final_justification": "I advocate for rejection purely due to the quality of the experimental section\n* The authors ignored several recent state-of-the-art methods\n* The authors reproduce baselines in their own codebase, and therefore report performance in terms of ambiguous 'updates' rather than environment steps like standard practice. It is also not clear if they have fairly reproduced the performance of their baselines.\n* The quality of their expert demonstrations for the MuJoCo locomotion tasks are far below expert \n\nDuring the rebuttal period, the authors devised an off-policy variant that will greatly improve their next submission. However, even with this more sample-efficient variant, baselines converge while the performance curve of their method plateaus. This paints a completely different picture from their Figure 3 in terms of optimization efficiency, which is why I don't believe the paper can be accepted in the current iteration.", "rebuttal": "We thank the reviewer for the helpful and constructive feedback. We are glad they found our approach a “**novel look at proximal reward updates**” and appreciated the \"**beautiful and informative figures**”.Below we provide our responses, and we’re happy to follow up on any additional questions.\n\n\n**[Q1,W1,W2] How sample-efficient is PIRO compared to SOTA off-policy methods, such as DAC, P2IL and CSIL?**\n\nAs correctly noted in **W1**, PIRO is an on-policy method, and our evaluation thus focused on comparisons with other on-policy baselines. Meanwhile, to address the concern in **W2**, we thank the reviewer for pointing out our wording issue. While PPO was used to train expert policies across most of the baselines (for consistency), IQ-Learn was implemented with SAC as its policy optimizer, as in the original paper. We will revise the manuscript to clarify this distinction.\n\nGiven the reviewer’s valuable suggestion, we conducted additional experiments to evaluate PIRO’s sample efficiency against DAC, P2IL, and CSIL. All methods use the same high-performing expert models trained with SAC to ensure fair comparison. We will present more detailed results in our response to Q2.\n\nFollowing standard IRL benchmarking protocols, we assess sample efficiency by measuring the final performance achieved within a fixed number of environment steps (**1M**, relatively small in the practice of IRL approaches). Results (averaged over three runs) on representative MuJoCo tasks are presented below:\n\n| Task            | DAC     | P2IL     | CSIL    | PIRO    |\n|-----------------|---------|----------|---------|---------|\n| Ant-v2          | 907.6   | 976.6   | 391.8   | **3504.2** |\n| Hopper-v3       | **3309.6** | 25.8    | 6.6     | 1820.1  |\n| Walker2d-v3     | -3.07   | 1054.0 | 215.1   | **2052.8**  |\n| HalfCheetah-v2  | **8930.9** | -0.1   | -115.7  | 6600.2  |\n\nRelative Computation Cost per Batch (Normalized)\n\n| Environment     | DAC   | P2IL  | CSIL  | **PIRO** |\n|-----------------|-------|-------|-------|----------|\n| Ant-v2          | 1.00× | 0.81× | 0.66× | 1.00× |\n| Walker2d-v3     | 1.30× | 1.30× | 0.38× | 1.00× |\n| Hopper-v3       | 1.32× | 0.19× | 0.01× | 1.00× |\n| HalfCheetah-v2  | 1.16× | 0.88× | 0.56× | 1.00× |\n> _Table: Average per-batch computation time normalized by PIRO = 1.00×. Measured over the same training rounds and batch structure across methods._\n\nThese results demonstrate that PIRO consistently achieves competitive or superior final performance across MuJoCo tasks. Notably, while DAC performs well in Hopper and HalfCheetah, PIRO shows substantial gains in more challenging environments like Ant and Walker2d, indicating overall better sample efficiency across diverse settings. Although PIRO incurs slightly higher per-batch computation time in some tasks, we observe that DAC is often slower overall. \nImportantly, PIRO’s optimization framework guarantees monotonic policy improvement at each reward update, enabling more stable training and faster convergence in practice. This tradeoff between efficiency and stability ultimately leads to superior overall performance.\n\n\n**[Q2,W4] The results in Figure 3 look nowhere near optimal**\n\nThe key reason is that we use PPO-trained experts to match the learner’s optimization and ensure consistency. Prior benchmarks (e.g., Tianshou, SpinningUp) have consistently shown that PPO-trained experts tend to achieve lower episodic rewards than SAC-trained ones, particularly in MuJoCo tasks such as Hopper, Walker, and Ant.\n\nMoreover, we use the latest version of the MuJoCo physics engine (v4), which adopts stricter physical dynamics and reward definitions. This results in significantly lower raw reward values compared to older versions of the environment, even for well-trained expert policies.\n\nAs a result, the reward ceilings in our Figure 3 plots are naturally lower—not because PIRO performs poorly, but because the reward scale is inherently limited by the PPO expert ceiling. \n\nThis also addresses the concern in **W4** regarding the missing expert reference: we now report expert performance explicitly. Below we report the approximate expert rewards and standard deviations used in our study:\n\n|Environment|ExpertReward(Mean±Std)|\n|---|---|\n|Acrobot-v1|-80.8±7.6|\n|Ant-v4|1083.2±6.2|\n|BipedalWalker-v3|101.2±9.3|\n|Humanoid|300.9±18.2|\n|CartPole-v1|500.0±0.0|\n|LunarLander|252.0±13.7|\n|Hopper|428.5±113.2|\n|Walker2d|1093.8±19.6|\n|Pong|21.0±0.0|\n|Q*bert|15685.0±115.2|\n\n\n\nIn light of the reviewer's insightful comments, we recognize the limitations of relying solely on PPO-trained experts and on-policy pipelines in complex environments. As a result, we have extended PIRO to support SAC-based policy optimization and conducted additional experiments on representative MuJoCo locomotion and Gym Robotics tasks.\n\nBelow are the updated reward metrics averaged over five independent runs:\n\n> _Table: Performance across 5 independent runs on MuJoCo control tasks using high-quality expert demonstrations trained with SAC._\n\n| Task            | Expert   | **PIRO** | BC     | GAIL   | AIRL   | IQ     | HyPER  | FILTER | MM     | ML-IRL | DAC     | P2IL    | CSIL   |\n|-----------------|----------|----------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Ant-v2          | 5572.2   | **5967.2** | 1631.5 | 996.9  | 991.4  | 3589.8 | 2800.5 | -376.3 | -304.0 | 5382.5 | 923.8   | 976.6   | 420.7  |\n| Humanoid-v3     | 5468.8   | **5954.9** | 418.1  | 508.4  | 281.4  | 1847.5 | 717.5  | 291.7  | 367.2  | 5573.4 | 76.3    | —       | —      |\n| Walker2d-v3     | 5453.4   | **5643.7** | 384.4  | 4158.1 | 72.8   | 3023.0 | 1478.7 | 77.7   | 70.4   | 4794.7 | -3.0    | 1054.0  | 686.1  |\n| Hopper-v3       | 3559.6   | 3362.0 | 1034.4 | **3675.9** | 13.5   | 3424.5 | 2593.6 | 37.3   | 57.8   | 3316.4 | 3321.6  | 25.8    | 6.7    |\n| HalfCheetah-v2  | 12174.6  | **12587.4** | 221.2  | 1298.8 | 2251.4 | 3825.5 | 6473.4 | 0.3    | 20.3   | 11873.2| 9645.0  | -0.1    | -107.2 |\n\n> _Table : Performance across 5 independent runs on Gym Robotics tasks._\n\n| Task                      | Expert | **PIRO** | BC   | GAIL | AIRL | MM     | FILTER | HyPER  | IQ     | ML-IRL | P2IL   |\n|---------------------------|--------|----------|------|------|------|--------|--------|--------|--------|--------|--------|\n| AntMaze-UMazeDense-v4     | 35.6   | **25.7** | 8.8  | 5.2  | 4.5  | 5.1    | 6.1    | 11.9   | 3.9    | 4.2    | 3.4  |\n| AntMaze-MediumDense-v4    | 26.9   | **9.4**  | 1.1  | 1.3  | 2.6  | 3.4    | 1.9    | 3.0    | 3.4    | 0.9    | 2.9 |\n| AntMaze-LargeDense-v4     | 11.5   | **8.8**  | 1.1  | 0.9  | 3.4  | 1.7    | 0.6    | 1.5    | 0.8    | 0.3    | 0.2   |\n| AdroitHandPen-Human-v1    | 1062.5 | **254.0**| 44.1 | -8.7 | -593.9 | -344.3 | -685.4 | -866.7 | -751.9 | -251.2 | -61.2   |\n\n\n\n**[Q3,W3]: On early convergence and update granularity**\n\nAs noted in our response to Q2, we use PPO-trained expert policies. These tend to converge quickly but may not reach the highest possible reward ceilings achievable by more exploratory algorithms such as SAC. As a result, PIRO often aligns with these experts in just a few global rounds, leading to flattened performance curves after early updates.\n\nImportantly, this behavior reflects PIRO’s high sample efficiency rather than stagnation. Unlike many baselines that may require extended exploration or suffer from unstable updates, PIRO benefits from a theoretically grounded guarantee of monotonic policy improvement. Each reward update provably brings the learner policy closer to the expert, avoiding regressions and unnecessary detours. As a result, PIRO not only ensures stable convergence but also reaches expert-level performance more directly and with fewer interactions—especially in simpler environments such as CartPole, where optimal behavior is quickly attainable.\n\nWe report learning progress using global rounds, where each round comprises multiple environment steps followed by one full policy and reward update. This unit more accurately reflects algorithmic progress than raw environment steps.This may also clarify the concern raised in **W3**: although update-based metrics are implementation-dependent, all baselines in our study are evaluated under the same global-round setup with consistent hyperparameters. None of the results are borrowed from prior work — all experiments are conducted within our unified evaluation framework.\n\nIn short, PIRO achieves monotonic improvement under this regime while also demonstrating faster convergence than many baselines in practice.\n\n**W[5]: Missing references**\nThe comparison with CSIL can be found above. In practice, BC-IRL can be seen as a special variant of ours by removing constraint on the reward update magnititute, except for the way of calculating loss function: BC-IRL minimizes the mean squared loss $\\mathbb{E}_{(s,a)\\sim \\pi_E}(\\pi_\\theta(s) - a)^2$ while our method minimizes the KL divergence. Least-squares inverse Q-learning (LSIQ) penalizes the reward function magnitude and gives the theoretical support for doing so; we also make a similar penalization through a regularization term (see Eq.(10) in Section 5.1).\n\nOur method stands out as all these prior methods take **no** constraint on the reward update magnitude, making them vulnerable to training instability. The performance degradation caused by removing the control over reward update magnitude can be found in our ablation study in Appendix G."}, {"review_id": "8D44cfFQLN", "reviewer": "Reviewer_55QK", "summary": "The paper introduces a new non-adversarial inverse RL method based on updating the reward to decrease the KL divergence between the expert and learner policies. The authors prove that the idealized version of this algorithm guarantees policy improvement and implement an approximation based on the idea of surrogate objectives. The experiments show that the method has better stability, sample efficiency, and better performance than baselines. The method is evaluated on diverse tasks include continuous control and animal behavior modeling. The paper also shows that the learned reward function accurately models the ground truth reward, improving interpretability.", "questions": "1. Did you ever try the naive gradient-based approaches to optimize the KL against the expert? If so, is it unstable in practice? \n2. How sensitive is PIRO to the inner policy optimizer? Did you try other methods beside PPO? Is it correct that any method which learns the EBM (and can sample from it) would work? \n3. Is there any kind of clipping used to control the importance ratios in the gradient updates? Empirically, do these not blow up? \n4. For Q and V gradient updates, could you maintain a buffer of all off policy rollouts and probabilities seen up to that point? Do you have thoughts on whether PIRO would stably support this kind of change to improve sample efficiency? \n5. Is there an explanation why on-policy robustness of PIRO seems to differ per task? Hopper performance seems unaffected despite it having complex dynamics.\n6. For reward recovery in appendix E, does the PIRO-learned policy match the expert performance? \n7. Why is it reasonable to assume there should be a shared policy across meerkat individuals?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The presentation of the work is extremely clear. The clarity of explanation, grounding of related work, motivation for surrogates / connection to CPI / TRPO, and presentation of diagrams and plots are all very good. I really enjoyed reading the paper because of the fantastic diagrams. The core part of the method is pretty intuitive. The idea of surrogates is pretty classic in RL but the authors apply it in a new way for inverse RL (which needs stable, non-adversarial methods). I think the main algorithmic idea will be of broader interest to the community.\n\nThe main set of results are pretty convincing and cover a wide variety of settings and eval measurements: the authors measure rewards, stability, and sample efficiency for  various control, Atari, and real-world tasks. These are nicely supplemented with very thorough ablations. The experimental results positively answer all of the questions introduced in Section 6. Although the final algorithm is complex, the empirical changes are well-motivated and the results are positive. I have various questions about some details of the empirical results.", "was_revised": true, "final_justification": "I am maintaining my score of 5 after rebuttal. I had some minor questions about the empirical results which the authors resolved during rebuttal: sensitivity to PPO vs other optimizers, stability of importance sampling weights, and other baselines like direct KL minimization. There was an extended discussion by Reviewer zT3i on the online sample efficiency of the baselines the authors use. Although the requested baseline demonstrates strong sample efficiency, I believe other contributions of the proposed method (stability, sample efficiency re. expert demonstrations) are still valuable, and the authors did a faithful job of reproducing baselines.", "rebuttal": "We thank the reviewer for the thoughtful feedback. We're glad you found the **\"presentation extremely clear\"**, the diagrams **\"fantastic\"**, and the method **\"intuitive\"**. We're especially encouraged by your belief that the **\"main algorithmic idea will be of broader interest to the community\"**. We address your questions below.\n\n**[Q1]: Did you try directly minimizing the KL to the expert, and was it unstable?**\n\nYes. We experimented with directly minimizing the policy KL divergence $\\mathrm{KL}(\\pi_E | \\pi_\\theta)$ via gradient descent. However, as shown in the table below, this naive approach exhibits significant instability across tasks. In environments like CartPole and Q*bert, performance dropped sharply and fluctuated erratically. Even in more stable tasks like Humanoid, it failed to consistently improve the policy.\n\nThese results reflect common issues with directly optimizing KL: the gradients can be high-variance and poorly aligned with actual policy improvement, especially when $\\pi_\\theta$ is far from $\\pi_E$. This was a key motivation behind our design of PIRO. By using a majorization-minimization (MM) framework to upper-bound the KL surrogate at each iteration, PIRO ensures stable updates with guaranteed monotonic improvement—effectively avoiding the oscillations and regressions observed with naive KL minimization.\n\nTo illustrate this behavior concisely, we select one representative environment from each category (Gym, MuJoCo, Atari, Box2D), due to words constraints. \n\n> _Table: Return comparison between PIRO and naive KL minimization over iterations. Directly optimizing KL divergence results in instability (high fluctuation)._\n\n|Env|Iter|Original|Naive KL|\n|----|----|--------|---------|\n|CartPole|0|281.47|46.37|\n| |25|384.18(+36%)|9.41(-80%)|\n| |50|351.37(-9%)|9.48(+0.8%)|\n| |75|363.39(+3%)|9.40(-0.8%)|\n| |100|380.89(+5%)|9.36(-0.5%)|\n|BipedalWalker|0|-88.57|-92.49|\n| |25|-78.96(+11%)|-103.25(-12%)|\n| |50|-65.85(+17%)|-100.68(+2%)|\n| |75|-62.78(+5%)|-99.89(+0.8%)|\n| |100|-64.58(-3%)|-91.41(+8%)|\n|Humanoid|0|278.93|296.61|\n| |10|256.37(-8%)|216.17(-27%)|\n| |20|275.36(+7%)|207.89(-4%)|\n| |30|279.59(+2%)|302.82(+46%)|\n| |40|276.98(-0.9%)|307.65(+2%)|\n| |50|316.56(+14%)|246.95(-20%)|\n|Qbert|0|138.53|69.63|\n| |25|149.92(+8%)|42.42(-39%)|\n| |50|166.87(+11%)|39.24(-8%)|\n| |75|144.21(-14%)|10.95(-72%)|\n| |100|165.76(+15%)|46.79(+327%)|\n\n\n**[Q2-1]: How sensitive is PIRO to the inner policy optimizer? Did you try other methods beside PPO?**\n\nPIRO’s core theoretical framework does not rely on a specific policy optimizer. It only assumes access to a differentiable policy that supports sampling and advantage estimation. Therefore, in principle, any optimizer—on-policy or off-policy—can be used, as long as it enables stable policy updates and maintains sufficient overlap with expert trajectories.\n\nIn response to this question, we conducted additional experiments using **Soft Actor-Critic (SAC)** in place of PPO for policy optimization.\n\n> _Table: Performance across 5 independent runs on MuJoCo control tasks using high-quality expert demonstrations trained with SAC._\n\n| Task            | Expert   | **PIRO** | BC     | GAIL   | AIRL   | IQ     | HyPER  | FILTER | MM     | ML-IRL | DAC     | P2IL    | CSIL   |\n|-----------------|----------|----------|--------|--------|--------|--------|--------|--------|--------|--------|---------|---------|--------|\n| Ant-v2          | 5572.2   | **5967.2** | 1631.5 | 996.9  | 991.4  | 3589.8 | 2800.5 | -376.3 | -304.0 | 5382.5 | 923.8   | 976.6   | 420.7  |\n| Humanoid-v3     | 5468.8   | **5954.9** | 418.1  | 508.4  | 281.4  | 1847.5 | 717.5  | 291.7  | 367.2  | 5573.4 | 76.3    | —       | —      |\n| Walker2d-v3     | 5453.4   | **5643.7** | 384.4  | 4158.1 | 72.8   | 3023.0 | 1478.7 | 77.7   | 70.4   | 4794.7 | -3.0    | 1054.0  | 686.1  |\n| Hopper-v3       | 3559.6   | 3362.0 | 1034.4 | **3675.9** | 13.5   | 3424.5 | 2593.6 | 37.3   | 57.8   | 3316.4 | 3321.6  | 25.8    | 6.7    |\n| HalfCheetah-v2  | 12174.6  | **12587.4** | 221.2  | 1298.8 | 2251.4 | 3825.5 | 6473.4 | 0.3    | 20.3   | 11873.2| 9645.0  | -0.1    | -107.2 |\n\n> _Table : Performance across 5 independent runs on Gym Robotics tasks._\n\n|Task|Expert|**PIRO**|BC|GAIL|AIRL|MM|FILTER|HyPER|IQ|ML-IRL|P2IL|\n|---|---|---|---|---|---|---|---|---|---|---|---|\n|AntMaze-UMazeDense-v4|35.6|**25.7**|8.8|5.2|4.5|5.1|6.1|11.9|3.9|4.2|3.4|\n|AntMaze-MediumDense-v4|26.9|**9.4**|1.1|1.3|2.6|3.4|1.9|3.0|3.4|0.9|2.9|\n|AntMaze-LargeDense-v4|11.5|**8.8**|1.1|0.9|3.4|1.7|0.6|1.5|0.8|0.3|0.2|\n|AdroitHandPen-Human-v1|1062.5|**254.0**|44.1|-8.7|-593.9|-344.3|-685.4|-866.7|-751.9|-251.2|-61.2|\n\nThese results demonstrate that PIRO is not sensitive to the choice of policy optimizer, and performs robustly under both PPO and SAC when compatible conditions are met.\n\n\n**[Q2-2] Is it correct that any method which learns the EBM (and can sample from it) would work**\n\nIn theory, any method which learns the EBM (and can sample from it) is compatible with our framework. But this is not to say that they will work well because the sampling mechanism should be well designed to make it able to cope with high-dimensional and continuous states or actions. Given that, SAC would be a suitable alternative policy optimizer and its peformacme can be found above.\n\n**[Q3]: Is there any kind of clipping used to control the importance ratios in the gradient updates? Empirically, do these not blow up?**\n\nWe do not apply explicit clipping to the importance ratios in PIRO. Instead, we reset the behavior policy used for advantage computation to the current policy after a **small** fixed interval. This guarantees the current policy will not deviate from the old policy too much, ensuring that importance weights remain well-behaved.\n\nFurthermore, our implementation buffers and reuses trajectories based on specific (state, action) pairs. This design amortizes advantage estimation across updates and avoids repeatedly sampling from stale or highly divergent policies. Empirically, this mechanism helps contain the variance of importance weights. Across all tasks, we observe no gradient explosions related to importance sampling—learning. \n\n**[Q4]: For Q and V gradient updates, could you maintain a buffer of all off-policy rollouts and probabilities seen up to that point? Do you have thoughts on whether PIRO would stably support this kind of change to improve sample efficiency?**\n\nWhile PIRO is primarily on-policy in its reward update logic, our implementation does maintain trajectory buffers for off-policy reuse—specifically for estimating Q and V values under the old policy and current reward. These buffers are keyed by individual $(s, a)$ or $s$ pairs and store full rollouts sampled from the behavior policy. The associated log-probabilities from the behavior policy are retained to enable importance sampling (IS) during advantage estimation.\n\nThis design is crucial for our IS-based reward updates, as reusing cached rollouts allows us to amortize gradient estimates without repeatedly querying the environment. To avoid accumulating divergence, we do not maintain an ever-growing replay buffer. Instead, we periodically clear and refresh the buffers via Monte Carlo re-sampling (see Table 2), which resets the behavior policy to the current policy snapshot. This bounds policy shift and ensures that the IS weights remain well-behaved, as discussed in **[Q3]**.\n\n**[Q5]: why on-policy robustness of PIRO seems to differ per task?**\n\nWhile PIRO theoretically ensures monotonic alignment at the reward level, its practical robustness depends on how reliably policy updates respond to reward signals. Hopper, despite having nonlinear dynamics, tends to exhibit smooth and dense reward landscapes and stable PPO training behavior. This makes it less sensitive to minor variations in reward updates, hence showing strong performance and stability under PIRO.\n\nIn contrast, tasks like Ant or HalfCheetah often involve higher-dimensional control and more complex reward trade-offs (e.g., speed vs stability), where small reward shifts can induce divergent behaviors. Since PIRO does not directly constrain policy updates but rather modulates them via learned rewards, the resulting robustness can vary across environments depending on policy sensitivity to the shaped rewards.\n\n**[Q6]: For reward recovery in Appendix E, does the PIRO-learned policy match the expert performance?**\n\nYes. The experiment in Appendix E is conducted in a 6×6 GridWorld environment with a manually designed ground-truth reward that depends only on the agent’s position. The PIRO agent is trained without access to the true reward and learns solely from expert demonstrations. The reward heatmaps confirm that the recovered reward closely matches the ground-truth.\n\nTo verify behavioral alignment, we measured the average return of the learned policy over the last 5 evaluation rounds and compared it to the expert return:\n\n|Policy|AvgReturn|\n|------|---------|\n|Expert|-9.24|\n|PIRO|-8.48|\n\nThis indicates that the PIRO-learned policy not only matches but slightly exceeds the expert's performance in terms of accumulated reward.\n\n\n**[Q7]: Why is it reasonable to assume a shared policy across individual meerkats in the dataset?**\nThe meerkat behavivor dataset only records meerkats' physical behavior, like transiting among locations in their habitat, foraging and play-fight. These physical behaviors do not include social, sychological or ethical behavior, and thus they are largely same among adult meerkats. This assertion is supported by the records in the dataset, where each individual meerkat has a unique track number in a video clip, one can find no outstanding differnece over behavior of different individual meerkats. For example, raising guard would be a supporting evidence, as different adults may act as guards in turn."}, {"review_id": "yn4s51IDYQ", "reviewer": "Reviewer_oq8i", "summary": "This paper aims to address the problem of reward learning instability that is prevalent in inverse reinforcement learning (IRL), especially in the adversarial optimization framework. The authors put forward a core idea: if each update of the reward function ensures that the divergence between the learned policy and the expert policy decreases monotonically, then the stability of learning can be guaranteed in principle. Based on this, the paper first proposes a theoretical framework called \"Proximal Reward Update\" (PRU), which theoretically guarantees the monotonic decrease of the policy KL divergence through a majorization-minimization procedure. Subsequently, the authors instantiate this theoretical framework into a practical algorithm called \"Proximal Inverse Reward Optimization\" (PIRO). PIRO is an IRL algorithm that non-adversarially and explicitly learns reward functions, which relies on on-policy trajectory sampling for reward updates. Experiments on multiple simulated control tasks and a real-world animal behavior modeling task show that PIRO has significantly improved training stability while maintaining or exceeding the performance of existing advanced methods.", "questions": "On the gap between theory and practice: The PIRO algorithm replaces the constant C in the theoretical derivation with an adaptive coefficient $\\mu$. To what extent does this heuristic design affect the stability of the algorithm? Does it break the theoretical monotonic convergence guarantee in practice? If so, can you quantify how often or how much the KL divergence actually \"rises\" in iterations in your experiments?\n\nOn the trade-off in sample efficiency: The paper correctly points out the trade-off between \"stability\" and \"sample efficiency of online interactions\". Can you further elaborate on in which types of tasks this trade-off is acceptable (e.g., having a cheap and efficient simulator) and in which tasks it may be a bottleneck for its application (e.g., real-world physical interactions)? In addition, is there potential to partially replace online trajectory sampling with more advanced off-policy correction techniques (such as importance sampling with truncation) to alleviate this problem?\n\nComparison with non-adversarial value matching methods: The paper classifies PIRO as a \"policy alignment\" method. \"Value matching\" methods such as IQ-Learn also achieve non-adversarial learning. Although IQ-Learn has been used as a baseline, can you provide deeper insights into the fundamental pros and cons of these two non-adversarial paths: i.e., what are the core advantages and limitations of ensuring “policy divergence decreases monotonically” (PIRO’s path) versus retaining “value matching goal but avoiding adversarial training” (IQ-Learn’s path)?", "limitations": "Yes. The authors clearly discuss the two main limitations of PIRO in the conclusion: 1. The overall stability still depends on a stable policy optimizer; 2. The reliance on online policy sampling may reduce the sample efficiency of environment interaction and limit its scalability. This part of the discussion is sufficient and relevant.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe paper proposes a novel theoretical perspective to solve the stability problem of IRL, which shifts the core of the problem from the traditional \"reward matching\" or \"adversarial game\" to ensuring \"monotonic convergence of policy divergence\". This is an insightful and principled contribution.\n\nThe paper contains solid theoretical work, and provides detailed derivations and proofs for key theorems in the appendix. The experimental evaluation is extensive, covering a variety of tasks from classical control to high-dimensional MuJoCo and even real-world datasets, and is comprehensively compared with 8 baselines including several state-of-the-art methods, which strongly supports its effectiveness.\n\nWeaknesses:\n\nThere is a gap from theory to practice. The practical PIRO algorithm makes several heuristic approximations to the theoretical framework PRU, for example, replacing the theoretically derived (usually difficult to calculate or too large) penalty term coefficient C with adjustable coefficients $\\mu and \\nu$, and replacing the maximum norm with the L2 norm. These approximations make it impossible to strictly guarantee the theoretical monotonicity in practice.\n\nThe idea of ​​“policy alignment” for non-adversarial IRL is not new, and other works have adopted similar concepts. The originality of PIRO lies in how to “guarantee” the monotonicity of the alignment process through the MM framework.\n\nA core trade-off of the method is its reliance on on-policy trajectory sampling. As the authors acknowledge in the paper, this reduces its sample efficiency in environmental interaction, which may make it difficult to apply to tasks with high cost of interacting with the real world, such as robotics.\n\nThe overall process of the algorithm is relatively complex, and online sampling is required to estimate the gradient of the advantage function with respect to the reward parameter, which may become an obstacle to its generalization and application.", "was_revised": true, "final_justification": "Some questions have been solved, and I think the experimental part can still be improved.", "rebuttal": "We sincerely thank the reviewers for their constructive and thoughtful feedback. We are encouraged that the reviewers found our theoretical contribution principled and insightful, appreciated the rigorous derivations, and recognized the breadth of our empirical evaluations across both simulated and real-world domains. Below, we address reviewer’s comments in turn.\n\n---\n\n**[W1,W2,Q1]: In practice, does using heuristic approximations (e.g., adaptive coefficient $C$) break the theoretical monotonicity guarantees of PIRO? Can you quantify how often KL divergence increases during training?**\n\nIndeed, using a relaxed adaptive coefficient $C$ cannot rigorously guarantee the monotonicity in theory. Theoretically, with an adaptive coefficient and assuming we have a space of reward parameter that makes the KL divergence Lipschitz continuous, we can then quantify that KL divergence increase is upper bounded by $\\mathcal{O}(\\frac{C\\epsilon}{1-\\gamma})$. This is because $Q$ is $\\frac{1}{1-\\gamma}$-Lipschitz in $r$ and the energy-based policy $\\pi$ is $1$-Lipschitz in $Q$ (assuming the temperature = 1). Empirically, we found that the rate of KL divergence increases is lower than baselines in general. On the other hand, approximating the theory for better feasibility and efficiency is common in reinforcement learning approaches. Closely related to our method is TRPO, which also employs a heuristic KL constraint to replace the theory-informed one that guarantees the monotonic policy improvement. At its reputation-gain successor PPO, the KL constraint is further simplified as a penalty term with an adaptive coefficient (or clipping). This simplification makes the great success of PPO, which is now the gold-standard of RL algorithms. In light of this, we believe our methods -- inspired by theory but not rigorously following the theory -- are in line with the mainstream design and practice of RL algorithms. It is also worth noting that although some prior methods also adopt the concept of policy alignment, no one, at least in theory, analyzes how to achieve monotonic alignment, but we did in this work. This sets our work apart from the literature. \n\n---\n\n\n\n**[Q2,W3,W4]: Can you elaborate on when the trade-off between stability and sample efficiency is acceptable, and whether off-policy correction techniques (e.g., truncated IS) could be used to alleviate this?**\n\nThis trade-off is acceptable in settings where interaction is cheap (e.g., simulated control), or where expert trajectories are fixed and reward learning is purely offline (e.g., behavioral modeling). For instance, in MuJoCo, stable reward updates are more beneficial than marginal gains from reuse. In real-world applications like sim2real robotics or clinical modeling from logs, safety and interpretability matter more than sample count—making stability a higher priority than efficiency.\n\nBy contrast, in real-time robotics or interactive recommendation systems—where every environment step is expensive or delayed—this trade-off becomes limiting, and improving sample efficiency becomes essential.\n\nAs for off-policy correction, we already employ importance sampling in our reward updates, using behavior-policy log-probabilities stored in the trajectory buffer. Instead of explicit truncation, we control variance via periodic Monte Carlo resampling, which resets the behavior policy and keeps IS weights bounded. This approach achieves similar stability benefits without requiring manual clipping thresholds.\n\nIn addressing this question, we also aim to clarify two concerns raised in the weaknesses: **the reliance on on-policy sampling (on-policy cost)** and **the complexity of gradient estimation** .\n\nWithin our framework, when demonstrations are rich, we can adopt an equivalent but potentially more efficient formulation of reward gradient estimation that eliminates the need for estimating $A(s) = Q(s,a) - V(s)$ on trajectory rollouts. Instead of relying on advantage estimates derived from sampled returns—which often involve high-variance Monte Carlo evaluations—we directly optimize the expected reward difference between expert and policy transitions according to the following equivalence relationship: \n\n$$\n\\mathbb{E}\\_{\\rho^{\\pi\\_E}} [\\log \\pi\\_{\\theta}(a|s)] = \n\\mathbb{E}\\_{\\rho^{\\pi\\_E}} [ Q\\_{r\\_{\\theta}}^{\\pi\\_{\\theta}}(s, a) - V\\_{r\\{\\theta}}^{\\pi\\_{\\theta}}(s) ] = \n\\mathbb{E}\\_{\\rho^{\\pi\\_E}} [ r\\_{\\theta}(s, a) + \\gamma \\mathbb{E}\\_{s\\'\\sim P(\\cdot|s,a)} [V\\_{r\\_{\\theta}}^{\\pi\\_{\\theta}}(s\\')] - V\\_{r\\_{\\theta}}^{\\pi\\_{\\theta}}(s) ]\n$$\n\n$$\n= \\mathbb{E}\\_{\\rho^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \n\\sum\\_ {t=0}^\\infty \\gamma^t \\mathbb{E}\\_{s \\sim d\\_t^{\\pi\\_E}, a \\sim \\pi\\_E(\\cdot | s)}\n\\big[V\\_{r\\_{\\theta}}^{\\pi\\_{\\theta}}(s) - \\gamma \\mathbb{E}\\_{s\\'\\sim P(\\cdot|s,a)} [V\\_{r\\_{\\theta}}^{\\pi\\_{\\theta}}(s\\')] \\big]\n$$\n\n$$\n= \\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \\left( \\sum\\_{t=0}\\^\\infty \\gamma\\^t \\mathbb{E}\\_{s \\sim d\\^{\\pi\\_E}\\_t}[V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s)] - \\sum\\_{t=0}\\^\\infty \\gamma\\^{t+1} \\mathbb{E}\\_{s \\sim d\\^{\\pi\\_E}\\_{t+1}}[V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s)] \\right)\n$$\n\n\n$$\n= \\mathbb{E}\\_{\\rho^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \n\\mathbb{E}\\_{s \\sim d^{\\pi\\_E}\\_0}[V\\_{r\\_{\\theta}}^{\\pi\\_{\\theta}}(s)] \n= \\mathbb{E}\\_{\\rho^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \n\\mathbb{E}\\_{s \\sim \\eta}[V\\_{r\\_{\\theta}}^{\\pi\\_{\\theta}}(s)]\n$$\n\nHere, $d^\\pi_t(s)$ denotes the state distribution induced by $\\pi$ at $t$ and $\\eta$ is the (known) initial state distribution. This is a known result in offline IRL and non-adversarial IRL (e.g., ValueDICE, SQIL, IQ-Learn), but relying on rich demonstrations. It is theoretically equivalent to maximizing the expected reward difference $\\mathbb{E}\\_{\\pi_E}[r_\\theta] - \\mathbb{E}\\_{\\pi}[r_\\theta]$, and may offer improved efficiency by avoiding advantage estimation and long-horizon rollouts. \n\nThis sampling strategy serves as a practical implementation choice within our broader framework. Empirically, it improves sample efficiency and reduces per-update training cost, while maintaining stable convergence (**see Table below**). This addresses both the sampling cost and implementation complexity concerns raised in the review.\n\nTo further demonstrate PIRO’s sample efficiency, we report the number of iterations required for each algorithm to reach convergence across a range of MuJoCo tasks. The table below summarizes the average convergence steps of five independent runs, providing a direct comparison of training speed under consistent evaluation protocols.\n \n>_Table: Mean convergence (**the point where significant increases stop and (small or big) fluctuations happen afterwards**) steps across five independent runs on MuJoCo tasks._\n>_Format: Convergence / Rewards_\n\n|Environment|PIRO|AIRL|BC|FILTER|GAIL|HyPE|IQ-Learn|ML-IRL|MM|f-IRL|\n|-----------|----|----|--|------|----|----|--------|------|--|-----|\n|Ant|251/5967.2|160/991.4|60/1631.5|278/-376.3|46/996.9|217/2800.5|32/3589.8|278/5382.5|400/-304.0|29/-304.0|\n|HalfCheetah|205/12587.4|287/2251.4|338/221.2|150/0.3|61/1298.8|154/6473.4|271/3825.5|213/11873.2|136/20.3|365/20.3|\n|Hopper|55/3362.0|32/13.5|38/1034.4|67/37.3|105/3675.9|296/2593.6|408/3424.5|50/3316.4|103/57.8|127/57.8|\n|Humanoid|223/5954.9|50/281.4|29/418.1|25/291.7|74/508.4|54/717.5|59/1847.5|161/5573.4|35/367.2|416/367.2|\n|Walker2d|69/5643.7|51/72.8|29/384.4|81/77.7|40/4158.1|127/1478.7|306/3023.0|83/4794.7|39/70.4|202/70.4|\n\nThese results demonstrate that although PIRO may not exhibit the fastest raw convergence in terms of iteration count, it consistently achieves stable and reliable progress across tasks. Unlike methods that converge quickly but suffer from regressions or collapse, PIRO guarantees monotonic policy improvement (in theory)—ensuring that each update brings the learner closer to the expert. As a result, PIRO avoids unnecessary detours and converges toward expert behavior in a more direct and (empirically) sample-efficient manner. This highlights the practical benefit of our framework: trading off aggressive updates for principled, steady advancement leads to faster and more stable convergence overall.\n\n\n\n**[Q3]: Can you compare the core strengths and limitations of PIRO’s monotonic policy alignment versus value-matching non-adversarial methods like IQ-Learn?**\n\nBoth PIRO and IQ-Learn avoid adversarial training by replacing discriminator-style reward objectives with more stable surrogate losses. The key difference lies in what is aligned with the expert: IQ-Learn focuses on value matching—training a value function such that its gradient implicitly guides policy learning—whereas PIRO explicitly ensures monotonic reduction in policy divergence from the expert via a proximal reward update mechanism.\n\nThe main advantage of PIRO is that it directly optimizes for behavioral alignment between the learned and expert policies. This leads to predictable, stable policy improvement steps with a clear theoretical guarantee (via majorization-minimization). In contrast, IQ-Learn's value-matching objective does not guarantee that policy divergence decreases, and its performance can depend heavily on the quality of value estimates, which are sensitive to bootstrapping errors.\n\nThat said, IQ-Learn can be more sample-efficient in certain settings due to its compatibility with off-policy value-based learning, while PIRO is more conservative by design. Our approach prioritizes stability and monotonic policy improvement, which is particularly beneficial in settings where safe or interpretable reward learning is critical."}, {"review_id": "dp0BNIHp6Q", "reviewer": "Reviewer_QmGm", "summary": "The paper titled \"PIRO: Toward Stable Reward Learning for Inverse RL via Monotonic Policy Divergence Reduction\" introduces a novel approach to inverse reinforcement learning (IRL) that focuses on stabilizing reward learning by ensuring monotonic reduction in policy divergence. The authors propose Proximal Inverse Reward Optimization (PIRO), a non-adversarial IRL algorithm that leverages a theoretical framework to guarantee stable reward updates. PIRO explicitly learns reward functions and scales to high-dimensional tasks while maintaining strong sample efficiency. Empirical evaluations across various OpenAI Gym tasks and a real-world animal behavior modeling task demonstrate that PIRO significantly improves training stability and matches or exceeds the performance of state-of-the-art IRL methods. The paper concludes by discussing the limitations of PIRO, such as its dependency on on-policy trajectory sampling, and suggests future work to address these limitations and further enhance the robustness and efficiency of IRL algorithms.", "questions": "1. **How does PIRO handle the trade-off between stability and sample efficiency in on-policy rollouts?**  \n   The paper mentions that PIRO trades off on-policy sample efficiency for stability. Can the authors elaborate on potential strategies to mitigate this trade-off, such as using techniques like resets to expert states or more efficient sampling methods, to improve the overall efficiency of the algorithm?\n\n2. **What are the implications of PIRO's dependency on a stable policy optimizer in high-dimensional settings?**  \n   Given that PIRO's overall training stability depends on a stable policy optimizer, especially in high-dimensional settings, how do the authors suggest addressing potential instabilities in policy optimization? Are there specific policy optimization algorithms or techniques that are particularly well-suited for use with PIRO in such settings?\n\n3. **Can the authors provide insights into the generalizability of PIRO to other types of reward functions beyond state-only rewards?**  \n   The paper demonstrates PIRO's effectiveness in recovering state-only reward functions. How well does PIRO generalize to other types of reward functions, such as those involving state-action pairs or more complex reward structures? Are there any modifications or extensions needed to handle these cases effectively?", "limitations": "1. **Dependency on On-Policy Rollouts**: PIRO relies on on-policy rollouts for reward updates, which can significantly reduce sample efficiency in environment interactions. This dependency might limit its scalability to sample-expensive tasks, as generating on-policy trajectories can be time-consuming and computationally expensive.\n\n2. **Overall Training Stability**: While PIRO stabilizes reward learning, overall training stability also depends on a stable policy optimizer, especially in high-dimensional settings. The paper acknowledges that the stability of the entire training process is not solely determined by the reward learning mechanism but also by the policy optimization component. This implies that PIRO's effectiveness might be compromised if the policy optimizer is not robust.\n\n3. **Complexity of Implementation**: The implementation of PIRO involves several components, including reward updates, policy optimization, and the management of on-policy rollouts. This complexity might make the algorithm more challenging to implement and tune compared to simpler IRL methods. Additionally, the need for careful hyperparameter tuning and the dynamic adjustment of coefficients (μ and ν) could be a barrier for practitioners.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. **Theoretical Guarantees for Stability**: The paper introduces Proximal Inverse Reward Optimization (PIRO), which provides a principled framework for stable reward learning in IRL. This framework ensures monotonic reduction in policy divergence through a Majorization-Minimization (MM) procedure, offering a theoretical guarantee for stable training. This is a significant advancement over existing adversarial IRL methods that often suffer from unstable reward updates.\n\n2. **Empirical Performance**: PIRO demonstrates strong empirical performance across a diverse set of tasks, including classic control, Box2D, Atari, MuJoCo, and real-world animal behavior modeling. It matches or exceeds the performance of state-of-the-art IRL methods while significantly improving training stability. This robustness and effectiveness make PIRO a practical and reliable choice for IRL applications.\n\n3. **Scalability and Sample Efficiency**: PIRO is designed to scale to high-dimensional continuous tasks, making it suitable for complex environments. Additionally, it exhibits strong sample efficiency, requiring fewer expert demonstrations to achieve good performance. This is particularly important in real-world applications where obtaining large amounts of expert data can be challenging or costly.\n\nWeaknesses:\n1. **Dependency on On-Policy Rollouts**: PIRO relies on on-policy rollouts for reward updates, which can reduce sample efficiency in environment interactions. This dependency might limit its scalability to sample-expensive tasks, as generating on-policy trajectories can be time-consuming and computationally expensive.\n\n2. **Overall Training Stability**: While PIRO stabilizes reward learning, overall training stability also depends on a stable policy optimizer, especially in high-dimensional settings. The paper acknowledges that the stability of the entire training process is not solely determined by the reward learning mechanism but also by the policy optimization component.\n\n3. **Complexity of Implementation**: The implementation of PIRO involves several components, including reward updates, policy optimization, and the management of on-policy rollouts. This complexity might make the algorithm more challenging to implement and tune compared to simpler IRL methods. Additionally, the need for careful hyperparameter tuning and the dynamic adjustment of coefficients (μ and ν) could be a barrier for practitioners.", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer for the thoughtful and constructive feedback. We appreciate the recognition of PIRO's theoretical contributions, strong empirical performance, and practical scalability. Below, we address the reviewer’s concerns regarding on-policy sample efficiency, policy optimizer stability, and generalization to more complex reward structures.\n\n**[Q1,W1,W3]: How does PIRO handle the trade-off between stability and sample efficiency in on-policy rollouts?**\n\nOur framework includes strategies specifically designed to address this trade-off. In the original implementation, we used importance sampling (IS) to reuse previously collected rollouts in reward updates. By logging behavior policy probabilities, we reweighted each trajectory’s contribution to the reward gradient. To avoid high variance, we periodically reset the behavior policy using Monte Carlo sampling, which effectively bounds the IS weights and improves sample efficiency without manual clipping.\n\nUnder the assumption of sufficiently informative demonstrations, our framework admits an alternative yet theoretically equivalent approach to reward gradient estimation that circumvents the need for computing advantage values \\(A(s, a) = Q(s, a) - V(s)\\) through full trajectory rollouts. Rather than estimating advantages via high-variance Monte Carlo returns, we exploit a closed-form identity that connects the log-likelihood of expert actions with the expected reward difference between expert and learner policies:\n\n$$\n\\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} [\\log \\pi\\_{\\theta}(a|s)] = \\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} [ Q\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s, a) - V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s) ] = \\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} \\left[ r\\_{\\theta}(s, a) + \\gamma \\mathbb{E}\\_{s'\\sim P(\\cdot|s,a)} [V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s')] - V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s) \\right]\n$$\n$$\n= \\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \\sum\\_{t=0}\\^\\infty \\gamma\\^t \\mathbb{E}\\_{s \\sim d\\^{\\pi\\_E}\\_t, a \\sim \\pi\\_E(\\cdot | s)}\\left[V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s) -\\gamma \\mathbb{E}\\_{s'\\sim P(\\cdot|s,a)} [V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s)] \\right]\n$$\n$$\n= \\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \\left( \\sum\\_{t=0}\\^\\infty \\gamma\\^t \\mathbb{E}\\_{s \\sim d\\^{\\pi\\_E}\\_t}[V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s)] - \\sum\\_{t=0}\\^\\infty \\gamma\\^{t+1} \\mathbb{E}\\_{s \\sim d\\^{\\pi\\_E}\\_{t+1}}[V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s)] \\right)\n$$\n$$\n= \\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \\mathbb{E}\\_{s\\sim d\\^{\\pi\\_E}\\_0}[V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s)] = \\mathbb{E}\\_{\\rho\\^{\\pi\\_E}} [ r\\_{\\theta}(s, a)] - \\mathbb{E}\\_{s\\sim\\eta}[V\\_{r\\_{\\theta}}\\^{\\pi\\_{\\theta}}(s)]\n$$\n\nThis is a known result in offline IRL and non-adversarial IRL (e.g., ValueDICE, SQIL, IQ-Learn), but relying on rich demonstrations. It is theoretically equivalent to maximizing the expected reward difference $\\mathbb{E}\\_{\\pi_E}[r_\\theta] - \\mathbb{E}\\_{\\pi}[r_\\theta]$, and may offer improved efficiency by avoiding advantage estimation and long-horizon rollouts. \n\nThis sampling strategy serves as a practical implementation choice within our broader framework. Empirically, it improves sample efficiency and reduces per-update training cost, while maintaining stable convergence—addressing both the sampling cost and implementation complexity concerns raised in the review.\n\n\n*Table: Training efficiency (time) normalized by PIRO = 1.00×, across MuJoCo and Gym Robotics tasks.*\n\n|Environment|PIRO|ML‑IRL|GAIL|AIRL|MM|FILTER|IQ‑Learn|HyPE|f‑IRL|\n|---|---|---|---|---|---|---|---|---|---|\n|Ant|1.00×|0.93×|0.10×|0.05×|0.57×|0.20×|0.42×|0.43×|0.19×|\n|Humanoid|1.00×|0.93×|0.05×|0.04×|0.14×|0.10×|0.04×|0.29×|0.19×|\n|Walker|1.00×|0.93×|0.06×|0.05×|0.21×|0.15×|0.12×|0.55×|0.18×|\n|Hopper|1.00×|0.98×|0.06×|0.04×|0.22×|0.16×|0.06×|0.56×|0.15×|\n|Cheetah|1.00×|0.97×|0.07×|0.06×|0.33×|0.34×|0.46×|0.58×|0.25×|\n|Umaze|1.00×|0.97×|0.05×|0.06×|0.40×|0.20×|0.50×|0.27×|0.65×|\n|Medium|1.00×|0.74×|0.03×|0.04×|0.17×|0.17×|0.57×|0.21×|0.46×|\n|Large|1.00×|0.95×|0.03×|0.04×|0.18×|0.18×|0.59×|0.24×|0.59×|\n|Pen|1.00×|0.97×|0.03×|0.04×|0.07×|0.08×|0.06×|0.10×|0.19×|\n\nThis demonstrates that PIRO's theoretical framework naturally accommodates efficient sampling strategies, reducing dependence on costly on-policy rollouts without compromising training stability.\n\nRegarding the reviewer’s concern on Dependency on On-Policy Rollouts(W1), we agree that standard PIRO updates require on-policy trajectories, which may reduce sample efficiency in interaction-limited settings. However, our framework is flexible and admits off-policy-compatible variants. In particular, the pairwise classification objective introduced above enables fully off-policy reward updates using single-step transitions from a shared reference distribution. This eliminates the need for fresh rollouts during reward optimization and significantly reduces the cost of environment interaction. Our experiments show that this variant achieves comparable performance while improving per-update efficiency, making PIRO more suitable for sample-expensive or offline imitation scenarios.\n\n>_Table: Mean convergence (**the point where significant increases stop and fluctuations happen afterwards**) steps across five independent runs on MuJoCo tasks._\n\n**Convergence / Rewards**\n\n|Environment|PIRO|AIRL|BC|FILTER|GAIL|HyPE|IQ-Learn|ML-IRL|MM|f-IRL|\n|-----------|----|----|--|------|----|----|--------|------|--|-----|\n|Ant|251/5967.2|160/991.4|60/1631.5|278/-376.3|46/996.9|217/2800.5|32/3589.8|278/5382.5|400/-304.0|29/-304.0|\n|HalfCheetah|205/12587.4|287/2251.4|338/221.2|150/0.3|61/1298.8|154/6473.4|271/3825.5|213/11873.2|136/20.3|365/20.3|\n|Hopper|55/3362.0|32/13.5|38/1034.4|67/37.3|105/3675.9|296/2593.6|408/3424.5|50/3316.4|103/57.8|127/57.8|\n|Humanoid|223/5954.9|50/281.4|29/418.1|25/291.7|74/508.4|54/717.5|59/1847.5|161/5573.4|35/367.2|416/367.2|\n|Walker2d|69/5643.7|51/72.8|29/384.4|81/77.7|40/4158.1|127/1478.7|306/3023.0|83/4794.7|39/70.4|202/70.4|\n\n\nWe also take this opportunity to address a related concern from **W3** regarding hyperparameter tuning. Specifically, we tested the impact of reward-related coefficients such as $\\mu$ and $\\nu$ in Appendix H. The sensitivity plots show that PIRO remains stable across a broad range of values, and performance degrades gracefully outside the recommended intervals. This suggests that careful tuning is helpful but not critical, and that the method is reasonably robust for practitioners.\n\n\n**[Q2] What are the implications of PIRO's dependency on a stable policy optimizer in high-dimensional settings?**\n\nPIRO is designed as a modular reward learning framework that delegates policy optimization to an external RL algorithm. This dependency enables flexibility rather than limitation: PIRO does not require any specific optimizer and can be paired with both on-policy methods like PPO and off-policy methods like SAC or TD3. In high-dimensional environments, where policy optimization becomes more sensitive to exploration and variance issues, this modularity allows PIRO to leverage state-of-the-art policy optimizers to improve scalability and stability.\n\nAs our formulation imposes no structural constraint on the optimizer, the reward learning mechanism remains agnostic to the policy update dynamics, enabling seamless integration with more advanced or domain-specific optimizers. This compatibility is particularly beneficial in high-dimensional settings, where the choice of a robust optimizer can significantly impact performance. In this sense, PIRO's reliance on a stable optimizer is not a weakness, but a strength that supports generalization and extensibility across domains.\n\n\n\n---\n\n\n\n**[Q3]Can the authors provide insights into the generalizability of PIRO to other types of reward functions beyond state-only rewards?**\n\nWe discuss two more types of reward functions:\n* Action-only reward $r(a)$, which is common in bandit settings. Our method naturally supports this reward setting by setting an action-only reward model $r_\\theta(a)$.\n* Transition-dependent reward $r(s,a,s')$. In IRL, a common technique to cope with this kind of rewards is to assume a potential-based function $h_\\phi(s)$ and use a particular form to model $r(s,a,s')$:\n$$r_{\\theta,\\phi}(s,a,s') = r_\\theta(s,a) + \\gamma h_\\phi(s') - h_\\phi(s).$$\n\nBased on this structure, we have \n$$Q_{r_{\\theta,\\phi}}^{\\pi_{\\theta,\\phi}}(s,a) =  \\mathbb{E}[r_\\theta(s,a) + \\gamma h_\\phi(s') - h_\\phi(s) + \\gamma (r_\\theta(s',a')+\\gamma h_\\phi(s'') - h_\\phi(s')) + \\cdots] = $$\n$$\\mathbb{E}\\left[\\sum_{t=0}^\\infty \\gamma^t r_\\theta(s,a) | s_0=s, a_0=s\\right]$$\n\nThus, the Q-function gradient under this shaped reward formulation is mathematically equivalent to that used in our original derivation, resulting in complete compatibility with the PIRO update mechanism."}]

Author rebuttal:

Meta-review: This is a nice IRL paper with some theoretical motivation. The reviewers generally have a positive consensus about the theoretical and practical contributions. The negative review is because there is an apparent difference in baseline performance. I personally am satisfied with the authors' response, and if they did use the same data for all methods, their not replicating exactly previous results is not an issue.
Decision: Accept (poster)