RIZE: Adaptive Regularization for Imitation Learning
Authors: Adib Karimi
Mohammad Mehdi Ebadzadeh
Venue: Accepted by TMLR ()

Abstract: We propose a novel Inverse Reinforcement Learning (IRL) method that mitigates the rigidity of fixed reward structures and the limited flexibility of implicit reward regularization. Building on the Maximum Entropy IRL framework, our approach incorporates a squared temporal-difference (TD) regularizer with adaptive targets that evolve dynamically during training, thereby imposing adaptive bounds on recovered rewards and promoting robust decision-making. To capture richer return information, we integrate distributional RL into the learning process. Empirically, our method achieves expert-level performance on complex MuJoCo and Adroit environments, surpassing baseline methods on the Humanoid-v2 task with limited expert demonstrations. Extensive experiments and ablation studies further validate the effectiveness of the approach and provide insights into reward dynamics in imitation learning. Our source code is available at https://github.com/adibka/RIZE.

Published in Transactions on Machine Learning Research (11/2025)
RIZE: Adaptive Regularization for Imitation Learning
Adib Karimit
adibkarimi22onut..c..
Department of Computer Engineering Amirkabir University of Technology
Mohammad Mchdi Ebadzadch
ebadzadeh Baut.ac.
Department of Computer Engineering Amárkabir University of Technology
Reviewed on OpenReview: /etps://openrreet net/ forum? id= a6DWgXJ22Z
Abstract
We propose novel Inverse Reinforcement Learning (IRL) method that mitigates the rigid- ity of fixed reward structures and the limited flexibility of implicit reward regularization. Building on the Maximum Entropy IRL framework, our approach incorporates squared temporal-difference (TD) regularizer with adaptíve targets that evolve dynamically during training, thereby imposing adaptive bounds on recovered rewards and promoting robust decision-making. To capture richer return information, we integrate distributional RL into the learning process. Empirically, our method achieves expert-level performance on com- plex MuJoCo and Adroit enviromments, surpassing baseline methods on the Humanoid-v2 task with limited expert demonstrations. Extensive experiments and ablation studies fur- ther validate the effectiveness of the approach and provide insights into reward dynamics in imitation learning. Our source code is available at https::/// /trps::/github com/ aat/bkkkkkaaizEE..
1 Introduction
Designing effective reward functions remains a fundamental challenge Reinforcement Learning (RL). While dense, well-shaped rewards can facilitate learning, they often require laborious task-specific tuning and domain expertise, limiting their scalability (Amodei et al., 2016 Peng et al.) 2020). To circumvent these
limitations, researchers have explored alternative approaches, including sparse rewards upon task completion
(Silver et al. 2016), lcarning from expert trajectories (Schaal
1996
Ng & Russell 2000 Osa et al. 2018
hurnan preference-bassee based reward modeling (Christiano et al.] 2017 Lee et al. 2021 Hejna & Sadigh 2023
and intrinsically motivated RT. (Oudeyer et al. 2007 Schmidhuber
2010 Colas et al. 2022). Among these,
Inverse Reinforcement Learning (IRL) (Abbeel & Ng. 2004) offers a compelling alternative by inferring
reward functions directly from expert demonstrations, bypassing manual reward engincering. IRL has driven breakthroughs in roboties (Osa et al. 2018), autonomous driving (Knox el. al. 2023 and drug discovery
(Ai et al. 2024).
A prominent framework in IRL is Maximum Entropy (MaxEnt) IRL (Ziebart 2010), which underpins many state-of-the-art (SOTA) IL methods. Prior works have combined MaxEnt IRL with adversarial training (Ho & Ermon 2016 Fu et al. 2018) to minimize divergences between agent and expert distributions. However, these adversarial methods often suffer from instability during training. To address this, recent research has introduced implicit reward regularization, which indirectly represents rewards via Q-values by inverting the Bellman equation. For instance, TQ-Learn (Garg et al. 2021 unifies reward and policy representations using Q-functions with an Ly-norm regularization on rewards, while LSIQ (Al-Hafez et al. 2023) minimizes the chi- squared divergence between expert and mixture distributions, resulting in a squared temporal difference (TD) error objective analogous to SQIL (Reddy et al. 2020). Despite its effectiveness, this method has limitations:
*Corresponding author.

---

Published in Transactions on Machine Learning Research (11/2025)
LSIQ assigns fixed targets for implicit rewards (e.g., +1 for expert samples and -1 for agent samples), which constrains flexibility by treating all tasks and state-action pairs uniformly, limiting performance and requiring additional gradient steps for convergence.
We propose an extension of implicit reward regularization under the MaxEnt IRL framework, introducing two key advancements. Adaptive Targets: We enhance prior TD-error regularization by introducing learnable targets A and X that dynamically adjust during training. These targets replace static constraints with context-sensitive reward alignment, preventing rewards from over-increasing over-decreasing. Crucially, our theoretical analysis reveals that these adaptive bounds constrain implicit rewards to well-defined ranges 2x + max (A*E, } where e is the regularization coefficient. Since implicit rewards derive from Q-values, this ionularization indirectly stabilizes policy training. Distributional RL Integra- tion: We incorporate return distributions
[- 26 + min { }"} 1
a) [Bllemare et al., 2017) to capture richer uncertainty information in returns, while using their expectations for policy optimization. Though distributional RI. has shown success in adversarial IRL (Zhou et al. 2023), it remains unexplored in non-adversarial scttings. Our work bridges this gap; demonstrating its efficacy in MaxEnt. IRL while preserving theoretical guaran- tees. Unifying these advances,
framework outperforms IL baselines on MuJoCo (Todorov et al. (012) and Adroit (Rajeswaran et al. 2018) benchmarks. Notably, our approach shows clear benefits on complex tasks like Humanoid-ve and Hammer-v1, where adaptive targets and distributional learning improve training stability, as confirmed by our ablations.
Our contributions are threefold. First, we introduce adaptive targets for implicit reward regularization, en- abling dynamic reward bounds that enhance stability and aligmment during training. Second, we integrate return distributions into implicit reward frameworks, capturing richer uncertainty information while preserv- ing theoretical consistency. Third, we empirically validate our approach through extensive experiments on MuJoCo and Adroit tasks, demonstrating superior performance in complex environments.
2 Related Work
Imitation learning (TL) and inverse reinforcement learning (IRL.) (Watson et al. 2023 are foundational paradigms for training agents to mimic expert behavior from demonstrations. Behavioral Cloning (BC) [Pomerleau] 1991), the simplest IL approach, treats imitation as a supervised learning problem by directly mapping states to expert actions. While computationally efficient, BC is prone to compounding errors (Ross) & BBagnel 2011) due covariate shift during deployment. The Maximum Entropy IRL framework (Ziebart, 2010) addresses this limitation by probabilistically modeling expert behavior as reward maxinization under an entropy regularization constraint, establishing a theoretical foundation for modern IRL methods. The advent of adversarial training marked a pivotal shift in IL methodologies. [H & Ermon (0166) introduced Generative Adversarial Imitation Learning (GAIL), which formulates imitation learning as a generative ad- versarial game (Goodfellow et. al.] 2014) where an agent learns a policy indistinguishabbe from the expert's 's by minimizing the Jensen Shannon divergence between their state action distributions. This framework was generalized by Ghascmipour et al. (2019) in f-GAIL, which replacos the -hasenn-hannon divergence with arbitrary (f)-divergences to broaden applicability. Concurrently, Kostrikov et al. (2019) proposed Discrim- inator Actor-Critie (DAC), improving sample efficiency via. off-policy updates and terminal-state reward modeling while mitigating reward bias. Most recently. Chang et al. (2024) cast adversarial imitation as a boosting procedure: AlLBoost maintains an ensemble of weighted policies and trains the discriminator against a weighted replay buffer approximating the ensemble's occupancy, enabling fully off-policy training and reporting gains over DAC on DeepMind Control tasks (Tassa et al. 2018).
Recent advances have shifted toward methods that bypass explicit reward function estimation. Kostrikov et al. 2 introduced ValueDICE, offline IL method that leverages an inverse Bellman operator to avoid adversarial optimization. Similarly, Garg ct al. (2021) developed IQ-Learn, which circumvents the challenges of MaxEnt IRL by optimizing implicit rewards derived directly from expert. Q-values. A parallel research direction simplifies reward engineering by assigning fixed rewards to expert and agent samples. Reddy et al.) (2020) pioncered this approach with Soft Q Imitation Learning (SQIL), which assigns binary rewards to transitions from expert and agent trajectories. Most recently, Al-HHafz et al. (2023) proposed Least Squares Inverse Q-Learning (LSTQ), enhancing regularization by minimizing chi-squared divergence between expert
2

---

Published in Transactions on Machine Learning Research (11/2025)
and mixture distributions while explicitly managing absorbing states through critic regularization. In the
same spirit of avoiding adversarial training, Coherent Soft Imitation Learning (CSIT.) (Watson et al.
2023
inverts the soft policy update to derive an explicit shaped "coherent" reward from behavior-cloned
policy
and then fine-tunes the policy with standard RL using online or offline data. Orthogonally, Jain et al. introduce Successor Feature Matching, non-adversarial IRL method that forgoes explicit reward learning by directly matching expert and learner successor features, and notably supports state-only demonstrations. Complementing these, Wu al. (2025) propose baafuusioo-based framework that learns score functions on expert and learner states and optimizes score-difference cost (a diffusion score divergence), offering non- adversarial alternative that also works with state-only demos. Our work builds on TQ-Learn (Garg et al. 2021) and LSIQ (AI-Hafez et al. 2023) by applying a squared TD regularizer with adaptive targets and by employing an Implicit Quantile Network (IQN) critic (Dabney et al. 2018a).
2025)
3 Background
3.1 Preliminary
We consider Markov Decision Process (MDP) Puteeran 2014) to model policy learning in Reinforcement Learning (RL). The MDP framework is defined by the tuple AS,A, po, P. R, 7), where S denotes the state space, A the action space, Po the initial state distribution, P S x Ax8- + [0, 1] the transition kernel with P(- s.)) specifying the likelihood of transitioning from state s given action a, R : S A + R the reward function, and 7 E [o, 1] the discount factor which tempers future rewards. stationary policy w E is characterized as a mapping from states S to distributions over actions a E A.. The primary objective in RL (Sutton & Barto 2018) is to maximize the expected sum of discounted rewards, expressed E,, EE= R a))). Furthermore, the occupancy measure px(s,a) a) for policy E II is given by r). The corresponding measure for an expert policy, TE; is similarly denoted by Pg. In Imitation Learning (IL), the expert policy ne is typically unknown, and only a finite set of expert demonstrations is available, rather than explicit reward feedback from the environment.
(1 (a 8) E P'P(ss
3.2 Distributional Reinforcement Learning
Maximum Entropy (MaxEnt) RL (Haarnoja et al. 2018) focuses on addressing the stochastic nature of action selection by maximizing the entropy of the policy, while Distributional RI. (Bellemare et al. 2017) emphasizes capturing the inherent randomness in returns. Combining these perspectives, the distributional soft value function Z S A Z (Ma et al., (020) for policy II encapsulates uncertainty in both rewards and actions, with Z representing the space of return distributions. It is formally defined as:
Z(s,a) = Σ Ri(s,, ar) + aH(T(. ss)],
(1)
where H(T)= = Ex,-- [aa) s)] denotes the entropy of the policy, and a > 0 balances entropy with reward.
The distributional soft Bellman operator B7,- : Z + Z for a given policy a is introduced as (B) (s 2
R(s,a) + >ZZ(s's a log (aa' s')], where s' ~ P(- 8, a), a' ~ x(- | 8'), and 2 signifies equality distribution. Notably, this operator exhibits contraction properties under the p- Wasserstein metric, ensuring convergence to unique fixed point, the distributional soft return function for the given policy.
A practical approach to approximating the return distribution Z involves modeling its quantile function Fz (r), evaluated at specific quantile levels T [0.1] (Dabnyy et al. 2018b). The quantile function is defined as (r) = inf(= E R : < Fz(z)), where F2(2)= = PPZ S 2) is the cumulative distribution function of Z. For simplicity, we denote the quantile- based representation as Z, == Fz' To discrctize this representation, we define sequence of quantile levels, denoted as (ri):.....NNi. where
To <... < TN-1 1. These quantiles partition the unit interval into N fractions. For uniformly sampled quantiles UO0 1), Z (s, a) denotes the r-quantile (scalar) of the return distribution.
3

---

Published in Transactions on Machine Learning Research (11/2025)
3.3 Inverse Reinforcement Learning
Given expert trajectory data, Maximum Entropy (MaxEnt) Inverse RL (Ziebarl 2010) aims to infer reward function R(s, a) from the family R = RSxA, Instead of assuming a deterministic expert policy, this method optimizes for stochastic policies π E II that maximize R while matching expert behavior. GAIL (Ho & Ermon 2016) extends this framework by introducing a convex reward regularizer & :R:S5x + R, leading to the adversarial objective:
max min [(T, R) Eps (R(s, Ep, [R(s, - H(T) - (B)) RER nell
(2)
IQ-Learn GGarg et al. Q-functions E Ω (Piol el al. 2014). It leverages the inverse soft Bellman operator T", defined as:
2021
departs from adversarial training by implicitly representing rewards through
(T*Q)
as) P(ss) (Q((s''')) aog ('a'ss))
(3)
For fixed policy æ; T bijective, ensuring a one-to-one correspondence between Q-values and rewards: T"Q R and Q = ( This allows reframing the MaxEnt IRL objective (2) in Q-policy space maxqen mingen [TT, Q). IQ-Learn simplifies the problem by defining the implicit reward RQ(s, T"Q(s, a) and applying an L2 regularizer ((Rq). The final objective becomes:
max I [[T, Q) Eps (RQ a)] (R. a²((z) Qef =ell
(4)
c [p [R E,, E. (RQ(sa)))))
4
Methodology
This section introduces a framework that integrates Distributional Reinforcement Learning with Inverse RL. We first show how return distributions can replace point-estimate crities. We then propose adaptive regularization technique for implicit rewards and analyze its properties. Finally, we derive RIZE, which combines distributional critics with bounded-reward imitation learning.
4.1 Distributional Value Integration
Our approach departs from traditional imitation learning by explicitly using return distributions as critics within an actor-critic framework (Zhou et al. 2023). We argue that learning the soft return distribution Z(s,0) a) in Equation (1. rather than relying solcly on point estimates like Q(s, a), enhances docision-making by capturing uncertainty in complex environments. This view is consistent with recent neuroscience findings suggesting that docision-making the prefrontal cortex relies on learning distributions over outcomes rather than only their expectations (Muller al. 2024).
Moreover, access to the full return distribution enables the computation of statistical moments most notably the expectation- which optimize both policy and critic using the expectation of Z (Bellemare: et al. 2017
Dabney et al. 2018b a), yielding more robust learning signal while remaining compatible with IQ-Learn.
We compute Q as the expectation of the soft return distribution:
N-1
((s,a) = Σ
(T41 - Ti) Ln, (s,a),
(5)
1=0
where (T.) are quantile fractions and Zr, (s, a) denotes the corresponding quantile values (see Lemma A.1
4.2 Implicit Reward Regularization
In this section, propose regularizer for inverse RL that refines existing implicit reward formulations (Garg et al.] 2021. The implicit reward is defined as:
R)(s,a) = Q(s,a) a) >Er (Q(s', ( a') log (a'(s'))
(6)
4

---

Published in Transactions on Machine Learning Research (11/2025)
Previous works typically regularize implicit rewards either using Lg-norms (Garg al. 2021 or by treating them as squared-TD) errors between rewards and fixed targets (Reddy et al. 2020 Al-Halez et al. 2023
While we adopt similar squared- IDD setting, we introduce adaptive targets = (for the expert TÆ) and X (for the imitation policy π) to construct our convex regularizer T RSS + R:
A)) Eps [(o[s, )² + Ep [[ a) - a) -
(7)
These targets self-update through a feedback loop where reward estimates continuously adapt to match moving targets:
min Ep (RQ(s, - min Ep, (Ro(s.a)-1*)] A
(8)
Substituting T(Re, A) for the L2 term in Equation and using Equation 5 to compute Q, we obtain:
C(T, Q) = Epe (RQ(s, a)] - Ep. [RQ a)] - aT((T)
- c BAE (R)( )² + Ep, [Ro(s -
Bc
where e is the regularization coefficient.
As in IQ-Learn and LSIQ, method seeks behavior indistinguushabbe from expert demonstrations in a non-adversarial implicit-reward sctting. Prior work analyzes optimal implicit rewards, drawing on Max-Min analyses from GANs (AI-Hafez et al. 2023 Goodfellow et al., 2014). Because we bind rewards to adaptive targets via [(Rq, >) (Equation (7). we carry out an analogous analysis, stated below.
Proposition 4.1. Let Ro(s, a) (T*Q)(s,a) denole the implicil reward derived from point-eslimale Q- values, where as) = [[)(s, Let pε(s,a) and pn(s,a) denote occupancy measures under gg and respectively. For fixed π, the optimal TD-regularized reward satisfies:
p (=(sa)) + pE((, (=(s Ro(s,a)= (pEls,a) p=
(10)
PE(S, + + (s,a)
Proof. Differentiating [(T, Q) in Equation result to zero yields
with respect to Ro(s,a) (holding fixed) and setting the
P= P= -0--200 2clpe(RR - XTE) + P - X²)],
(11)
from which Equation
10 follows.
Corollary 4.2. The optimal implicit reward satisfies:
1 + Amin, + Amax
Rg(s,a) E
(12)
2c
where Amin == min [ and Amax = max {A=E {²}. rewards within well-defined range.
The coefficient and adoplive targels bound
1 since PE, Pr 2 0 Proof. Considering the optimal reward Equation 10 the first term lies in [- 2c 3e (achieved when either Pr - 0 or Pg + 0). The second term is a convex combination of == and X" thus in [Ammn Amax]- Combining intervals yields the result. In practice, we use À E [5, 10] and E [0.1, 0.5), which we found to promote stable training (see Appendix C.7
Moreover, when the occupancy measures match, the optimal reward simplifies significantly, stated in Corollary A.2
Rs a) =
(13)
Recalling that we represent rewards through Q-values, the boundedness of the optimal reward ensures that critic updates remain rained and because the policy directly depends on these critic values promotes stable policy optimization. However, as previously noted (Al-Hafez et al., 2023 Viano et al. 2022). the convergence guarantee originally stated for IQ-Lcarn (Garg et al.] 2021)) does not extend to the reguularie used in practice. Consequently, formal proof for the resulting alternating SAC updates remains an open question and is left. for future work.
5

---

Published in Transactions on Machine Learning Research (11/2025)
Median
IQM
Mean
Optimality Gap
BC LSIO SOIL
Demo 3
CSIL RIZE
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.2
0.4
0.6
0.8
0.2
0.4
0.6
0.8
Median
IQM
Mean
Optimality Gap
LSIO
BC
Demo 10
SOIL
CSIL
RIZE
0.4
0.6
0.8
1.0
0.25
0.50
0.75
1.00
0.4
0.6
0.8
1.0
0.2
0.4
0.6
Figure 1: RLiable (Agarwal et al. 2021 plots for RIZE vs, BC, LSIQ, SQIL, CSLL, and IQ- on six MuJoCo/ Adroit tasks. For cach setting (3 demos; demos), we report aggregate Median, IQM, Mean, and Optimality Gap with 95% confidence intervals computed via percentile bootstrap stratified over tasks and five seeds. Scores are normalized to expert performance. Higher is better for Median, IQM, and Mean; lower is better for Optimality Gap.
Adroit
4.3 Practical Algorithm
We now introduce RIZE:
Adaptive Regularization for
Algorithm 1 RIZE
Imitation Learning (Algorithm [] We approximate and æ with neural networks, optimizing via Q = B[Z]. Following Distributional SAC (Ma al., 2020
1: Initialize Zø, Tg, AMM, and X=
2: for step t in f1...., N) do
3: Calculate (s,,a) E [s using Eq.
employ target policies for next-action sampling and double-critic architecture with target networks for sta- bility. Lower learning rates proved critical for robust policy updates, with a four-layer MLP policy network necessary for complex tasks.
4: Update Ze using Eq. 9
5:
Φi+1 dr - Bz V - [)]
6: Update (like SAC)
0.11 + 02 + 3r VoE
min (s a) k=1.2
7:
Targets x and \ are optimized to match expected rewards. We initialize x higher than X and update x with lower learning rates due to its sensitivity.
log no (a|s)
8:
Update d and AE using Eq
8
9:
X + AĀ -A I
(Ro X)
10:
AR + A, BATE V (RQ
11: end for
5 Experiments
We study contimuous-controll imitation learning from state action expert samples, evaluating our algo- rithm on five MuJoCo (Todorov al., 2012) benchmarks (HalfCheetah-v2, Walker2d-v2, Ant-v2, Humanoid-v2, Hopper-v2) and Adroit Hand task (Hammer-vi). We compare against state-of-the-art basclines IQ-Lcarn (Garg et al. 2021). LSIQ (AI-Hafez et al. 2023), SQIL (Reddy et al., 2020), CSIL (Wat- son et. al. 2023 and Behavior Cloning (BC) (Pomerleau 1991). All experiments use live random seeds [Henderson et al. 2018). We assess each method with three and ten expert trajectories. We report mean 95% confidence intervals (CI) across five seeds. Episode returns are normalized by expert performance. For implementation details, additional experimental results, and visualizations, see Appendix and C
Main Results. Figure reportts aggregale metrics computed with RLiable (Agarwal et al. 2021 b, which provides statistically principled evaluation via Median, Interquartile Mean (IQM), Mean, and Optimality Gap with stratified bootstrap confidence intervals; we adopt this protocol throughout, consistent with recent II. practice (e.g., AlLBoost (Chang 2024): SFM (Jain et al. (0255). Across both 3- and 10-demonstration settings, RIZE achieves higher Median, TQM, and Mean and
lower Optimality Gap than the baselines; the

---

Published in Transactions on Machine Learning Research (11/2025)
HalfChectah--2
Walker2d-v2
Hopper-v2
Ant-v2
Humanoid-v2
Hammer-v]
Return
10
10
Trajectories
Trajectories
Trijectories
Trajectories
Trajectories
Trajectories
BC LSIQ
squ.
IQLeum
CSIL
RIZE
Expert
Figure 2: Normalized returns MuJoCo and Adroit tasks for RIZE and baselines. We first compute, per seed, the average opisodic return over the final third of training steps; bars show the mean across five seeds and error bars denote the 95% confidence interval. Returns are normalized to expert performance and reported for both 3 and 10 expert demonstrations.
HallCheetah--2
Walker2d-v2
Hupper-v2
Retum
Ant-v2
Humanoid-v2 v2
Hammer-vl
Retuma
100
Steps (x10²)
(x10=)
(x10*)
BC
LSIQ
SQIT
IQuaarn
CSL.
RIZE
Expert
Figure 3: Learning curves MuJoCo and Adroit tasks with 10 expert demonstrations. Lines show the mean normalized return across five seeds; shaded regions denote 95% confidence intervals.
only competitive methods overall are CSIL and IQ-Learn. Increasing the number of expert trajectories from 3 to 10 consistently improves RIZE on all RLiable aggregates. Task-wise, RIZE is the only method that solves Humanoid-v2 (all baselines fail), while on Hammer-vi RIZE, CSIL, and BC show higher returns than the remaining methods. In terms of sample efficiency, CSIL often attains strong returns early benefiting from behavior-cloning initialization but when BC ineffective (e.g.. Humanoid-v2). CSIL underperforms whereas RIZE ultimately succeeds. These trends are corroborated by the bar plot in Figure 2 (average the final third of training) and the learning curves in Figure? For learning curves with three demonstrations, see Figure f
Recovered Rewards. In this section, we assess whether our choice of regularizer I(Ro, A) in Equa- tion with adaptive targets can effectively bound the implicit rewards. By Corollary 4.2 togother with Equation (12). the optimal implicit reward confined to the interval [- 2c Amins 2c >max Since ex- pert rewards supposed to be higher than those of the learner during training (based IRI. objective

---

Published in Transactions on Machine Learning Research (11/2025)
HalfCheetah-v2
Walker2d-v2
Hopper-v2
15
Reward
10-
5-
0-
200
300
100
200
Ant-v2
Humanoid-v2
Hammer-vl
60
20
Roward
10
20
5
-20
-5
0-
200
300
100
200
Steps (x1O²)
Steps (x10²)
Steps (x10')
Expert reward
Policy
Expen
Policy
Figure 4: Implicit reward curves for expert and policy samples MuJoCo and Adroit tasks with 10 expert demonstrations. Each subplot reports the mean across five seeds, with shaded regions showing the 95% confidence interval. Theoretical upper and lower bounds derived in this work are overlaid as separate curves in each subplot.
loss 4 and the adaptive targets track these levels, we empirically observe A 2 X"; hence Amax X and Amin x= In practice, we should see rewards confined to | + A*, +
Figure f depicts the recovered reward trajectories together with the theoretical upper and lower bounds. Across tasks, the curves remain within the predicted band, indicating that regularizer with adaptive largets effectively bounds both expert and policy rewards. We also observe that the position and width of the band vary with the task and data regime, reflecting the flexibility of the adaptive targets to adjust to different dynamics. In contrast, fized (non-adaptive) targets lack this ability, leading either to overly loose bounds or to target-reward mismatch. Overall, these results support the role I(Rq, X) and the adaplive largets in producing stable, well-calibrated implicit rewards that align with Corollary 4.2 For the 3-demonstration setting, Figure shows reward trajectories with their theoretical bounds; for gross-method comparisons of reward trajectories, see Figures and 10a
Ablation on Critic Architecture. To assess the effect of modeling the return distribution, we replace the IQN-based critic Z(s,a) a) (Dabncy al.) 2018a in RIZE with a point-estimate Q(s. a) critic and compare against. the original implementation. Here, 7(s,a) denotes the full distribution of discounted returns whose expectation yields Q(s,a). As shown in Figure the IQN critic consistently outperforms the Q-network across all tasks, exhibiting lower variance and greater sample efficiency. Notably, on Humanoid-ye and Hammer-vi, the Q-network fails to match expert performance, whereas the IQN critic maintains expert-level returns.
See Figure 13 for reward trajectories with theoretical bounds when using Q-network in place of the IQN critic, and Figures and 10b for critic value-estinee estimation curves MuloCo and Adroit tasks.
6 Conclusion
We propose a novel IRL framework that overcomes the limitations of fixed reward mechanisms through dynamic reward adaptation and context-sensitive regularization. Our approach ensures bounded implicit re- wards and stable value function updates, leading robust policy optimization. By integrating dist distributional
8

---

Published in Transactions on Machine Learning Research (11/2025)
HalfCheetah-v2
Walker2d-v2
Hopper-v2
Retum
Ant-v2
Humanoid-v2
Hammer-vl
Retum
100
Steps (x10²)
Steps (x10*)
Steps (x10')
RIZE (with Zie all
RIZE (with @ wi)
Expert
Figure 5: Ablation critic architecture: a( via Implicit Quantile Networks (IQN) Dabney et al, 2018a) versus classic Q(s.a). We report apper-nnrmalizee returns across all MuJoCo and Adroit tasks using three expert demonstrations; metrics show the mean over five seeds with 95% confidence intervals.
RL with implicit reward learning, capture richer return dynamics while preserving theoretical guaran- tees. Empirical results on MuJoCo and Adroit benchmarks show expert-like proficiency on the Humanoid-v2 and Hammer-v1 tasks with three expert demonstrations. Ablation studies confirm the important. role of our rogularization mochanism. This work unifies implicit reward regularization with distributional return repre- sentations, offering scalable and sample-efficent solution for complex decision-making. Future directions include extending this framework offline IL and risk-sensitive robotics, where TD regularizer with adaptive targets and uncertainty-awaro return distributions can further improve robustness and generalization.
References
P. Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning. 2004.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Mare G Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL https: //openreview http://nee//ourtte//eeee wtwp:ummmi=uuuuuuuummm U44KBB
Cheng Ai, Hong Yang, Xinyu Liu, Ruijie Dong, Ying Ding, and Fei Guo. Munol-gpt: De novo multi-target molecular generation with transformer-based generative adversarial imitation learning. PLoS Computa- tional Biology, 2016)ee1,, 2024. doi: /0.1371/jourmall.bb..000002 101.2..2....1.. URL https://doi.org org/10...11. 10.
1371/journal. pcbii.cbi...... 1012229
Firas Al-Hafez, Davide Tateo, Oleg Arenz, Guoping Zhao, and Jan Peters. LS-IQ: Implicit reward reg- ularization for inverse reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul F. Christiano, John Schulman, and Dan Mané. Concrete 06565
problems in AT safety. CoRR, abss/1606.0656 2016. URL http:://aaas...og//aaaaaaa
Alex Ayoub, Kaiwen Wang, Vincent Liu, Samuel Robertson, James Meinerncy, Dawen Liang, Nathan Kallus, and Csaba Szepesvari. Switching the loss reduces the cost. in batch reinforcement learning. Tn Ruslan

---

Published in Transactions on Machine Learning Research (11/2025)
Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 2135 2158. PMLR, 21 27 Jul 2024. URL https:
7/proceedings mlr. httsss//255//youb22225 html
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, 2017. URL https://apl stt../ppp..mmmticccccooooa...... org/
CorpusID:
: 966543
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAI Gym. arXiv preprint Xiv: 01540, 2016.
Jonathan Daniel Chang, Dhruv Sreenivas, Yingbing Huang, Kianté Brantley, and Wen Sun. Adversarial imitation learning via boosting. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://pperrvvier net/ http//rrrrrddDDussssSSSe
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. In 1. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and R. Garnett (eds.), Advances in Newral Information Processing Systems, vol- wttpppcee.tllleee/e/e/eeeeaapsee
ume 30. Curran Associates, Inc., 2017. URL httpss://rroceddnggs.oo www.gs....uipp.... httppppp.cpppppppppppppppppppeee Suaperrffle/// filles/paper/ pdf
Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Yves Oudeyer. Autotelic agents with intrinsically motivated goal-conditioned reinforcement learning: A short survey. Journal of Artificial Intelligence Research, 74, July 2022. doi: 10.1613/jairr..1355.. URL
Will Dabney, Georg Ostrovski, David Silver, and Remi Munos. Implicit quantile networks for distribu- tional reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 1096-1105. PMLR, 2018a.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Remi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, SLATI8/TAAT18/EANT AAAI Press, 2018b. ISBN
978-1-57735-800-8.
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adricn Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RI. In Forty-first International Conference on Machine Learning, 2024. URL https:///openreviekk http:://ppppareeeeee.eeeeooororo Pi
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning. In International Conference on Learning Representations, 2018
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning, 2021. URL btttge//aaraa..ogg/a////0...0..0
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. In Advances in Neural Information Processing Systems, volume 34, 2021.
Seyed Kamyar Seyed Ghasemipour, Richard Zemel, and Shixiang Gu. A divergence minimization perspective on imitation learning mothods. In Conference on Robot Learning, 2019.
lan J Goodfellow, Jean Pouget-Abaas Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances Neural Information Processing Systems, volune 27, 2014.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In International Conference on Machine Learning, 2018.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Jocy Hejna and Dorsa Sadigh. Inverse preference learning: Preference-basee RL without a reward function. In Therty-seuenth Conference on Neural Information Processing Systems, 2023. URL. https://oppnrrevier. net//borumm1d--ga52222aa
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforeement learning that matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirlieth Innovalice Applications of Artificial Intelligence Conference and Eighth AAAT Symposium on Educational Advances in Artificial Intelligence, AAAI AAAII8/LAAPI//AAATU AAAI Press, 2018. ISBN 978-1-57735-800-8
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Infor- mation Processing Systems, volume 29, 2016.
Arnav Kumar Jain, Harley Wiltzer, Jesse Farebrother, Trina Rish, Glen Berseth, and Sanjiban Choudhury. Non-adversarial inverse reinforcement learning via. successor feature matching. In The Thirteenth In- ternational Conference Learning Representations, 2025. URL http:p/://pparreeee.eeeettfooros LvRQgsvd5V
W. Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis)design for autonomous driving. Artíf. Intell., 316(C), March 2023. ISSN 0004-3702. doi: 10.1016/j.artint.2222 103829. URL https://doi. 10110/001///11111// artint. wtw.iit...2...... 103829
llya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminatoo--ctor--ciiii Addressing sample inefficiency and reward bias in adversarial imitation learn- ing. In International Conference on Learning Representations, 2019.
Tya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Lee, Kimún and Smäth, Laura and Abbeel, Pieter, 2021.
Xiaoteng Ma, Qiyuan Zhang, Li Xia, Zhengyuan Zhou, Jun Yang, and Qianchuan Zhao. Dsac: Distributional soft actor critic for risk-sonsitive reinforcement lcarning. arXiv, 2020.
Timothy H. Muller, James L. Butler, Sebastijan Veselic, Bruno Miranda, Joni D. Wallis, Peter Dayan, Timothy E. J. Behrens, Zeb Kurth-Nelson, and Steven W. Kennerley. Distributional reinforcement learning in prefrontal cortex. Nature Neuroscience, 27(3):403 408, mar 2024. ISSN 1546-1726. doi: 10.1038/ 441598-023-01535-- URL httt:p::/ddo.org//// 111.://11...0/11111101---1----11 023-01535-w
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, 2000.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. Foundations and Trends in Robotics, 2018.
Pierre Yves Oudeyer, Frdric Kaplan, and Verena V. Hafner. Intrinsic motivation systems for autonomous mental development. IEFE Transactions on Evolutionary Computation, 11(2):265-288 2007. doi: 10.
(119/TEVC.2000.500077
Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. CoRR, abs/2004.00784, 2020. URL https:
//axxiv.org wrw. wwwpo//////.../0////0/ 00784
B. Piot, M. Geist, and O. Pictquin. Boosted and Reward-regularized Classification for Apprenticshhip Learning. In Proceedings of the International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2014.
11

---

Published in Transactions on Machine Learning Research (11/2025)
D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural Compu- tation, 3(1):88-97 97, 1991.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations, 2018. URL
wwtp:////////a////o////o//s http://bs/1700...11 1709.10087
Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020.
Stéphane Ross and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, 2011.
Stefan Schaal. Learning from demonstration. In M.C. Mozer, M. Jordan, and Petsche (eds.), Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996. URL https://proceed..gg neurips.cc/paper.
pdf
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Trans- actions on Autonomous Mental Development, 2(3):230 247, 2010. doi: 10.1109/TAMD.200 201.A.....00.2566665
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioamnis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game go with deep neural networks and search. Nature, (7887)5)) 489, 2016. doi: 10.1038/ 10.1038/natures 16666e1666..
URL https::/doi...rr http///ti.oor///////
Richard S. Sutton and Andrew G. Barlo. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, Massachusetts; London, England, second edition, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018. URL https::/arxxv..or//asssse
E. Todorov, T. Erez, and Y. Tassa, MuloCo: A physics engine for model-based control. In 2012 GEE//S. International Conference Intelligent Robots and Systems, pp. 5026-5033, 2012.
Luca Viano, Angeliki Kamoutsi, Gergely Neu, Tgor Krawczuk, and Volkan Cevher. Proximal point imitation learning. In Advances in Neural Information Processing Systems 35 (NewrIPS 2022), 2022.
Joe Watson, Sandy Huang, and Nicolas Heess. Coherent soft imitation learning. In Thirth- Conference on Newral Information Processing Systems, 2023. URL
Runzhe Wu, Yiding Chen, Gokul Swamy, Kianté Brantley, and Wen Sun. Diffusing states and matching scores: A new framework for imitation learning. In The Thirteenth International Conference on Learning Representations, 2025. URL htttps:///peeneviie.. wtt..///ooruuriiiw-iiIIuSS.
Y. Zhou, M. Lu, X. Liu, Z. Che, Z. Xu, J. Tang, and Y Zhang. Distributional generative adversarial imitation learning with reproducing kernel generalization. Neural Networks, 165, 2023.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of marimum causal entropy. PhD thesis, University of Washington, 2010.
12

---

Published in Transactions on Machine Learning Research (11/2025)
A Supporting Proofs
A.1 Expectation of the Return Distribution
Lemma A.1. The expectation of the distributional return satisfies
(iZ [)] Q(x,a) a).
Proof. From the soft return distribution definition
7(s = Σ 7
[&))]]
(14)
taking expectations yields:
E [[) = E
(R(s), a.) aH((T sx))
t=0
BB [R(s, ar) + a |+]] sx)]
= ((s a)
(by soft Q-value definition).
A.2 Convergence of the Optimal Reward
Corollary A.2. For Pr PE, the optimal reward satisfies (s) x A7
Proof. When Prr = PE, Proposition 4.1 simplifies to
(s) = T +
(15)
2
Substituting this expression into the loss functions for AH and X in Equation g
yiclds
min Epg 1 (A- - XmmO3 min Ep, 1 (A ---- xT)² A
(16)
Differentiating each objective with respect to its corresponding target and setting the derivatives to zero gives A A This result implics that the optimal targets coincide at convergence. Therefore, the optimal reward reduces to
Ro(s,o) x=E ====- =
(17)
B Implementation Details
MuJoCo Suite. We evaluate RIZE five Gym (Brockman et al. 2016) MuJoCo (Todorov al., 2012) locomotion tasks: HalfCheetah-v2. Walker2d-v2, Ant Ant-v2, Humanoid-v2 and Hopper-v2. Expert trajecto- ries for these tasks are taken from TQ-Learn (Gar et al. 2021) and were generated with Soft. Actor-Critie HHarnoja et al. 2018); each trajectory contains 1,000 state action transitions. Episode returns are normal- ized by expert performance with the following expert evaluation returns: HalfCheetah (5, (5,) Walker2d (5,200), Ant (4,700), Humanoid (5,300), and Hopper (3,500).
Adroit. For Hammer-v1 from the Adroit suite (Rajeswaran 2018). we use the D4RL dataset (uu al. 2021) and filter the top 100 episodes from the original 5,000. The resulting expert subset. has average
13

---

Published in Transactions on Machine Learning Research (11/2025)
return of 16,800, and each episode comprises 200 state-action pairs. We normalize returns in this domain by the average expert return of the selected subset.
Baselines. We evaluate five baselines: IQ-Learn [Garg et al. 2021 LSIQ (Al-Hafez et al. 2023), SQIL
[Reddy et al.) 2020). CSIL [Watson et al. 2023 and Behavior Cloning (BC) (Pomerleau, 1991h. h. For MuJoCo tasks, we use the authors' original code and configurations for TQ-Learn, LSTQ, and CSTL; SQIL is implemented using the LSIQ codebase, and BC follows the CSIL implementation. For Hammer-v1, we use the original CSIL code and configs. For IQ-Learn on Hammer, we observe that it does not solve the task even after small sweep we report a=0.03 and loss=value. For LSTQ on Hammer, we search over E {0.01, 0.05, 0.1} and loss E {value, vO} and select a=0.1, loss-value as the better-performing variant. For SQIL, we set a-0.2 consistent with our other tasks.
the entropy coefficient in {0.01, 0.03, 0.1} and the loss mode in {value, v0];
Our architecture integrates components from Distributional SAC (DSAC) Ma et al. 2020g and IQ-
Learn (Garg et al., 2021), , with hyperparameters tuned through search and ablation studies. Key conligura- tions for experiments involving three and ten demonstrations summarized in Table I All implementation details used in our experiments are publicly available at https: htwcchuub.comaaaaaaaaa adi Oka/RIZZ
Distributional SAC Components. The critic network is implemented as three-layer multilayer per- ceptron (MLP) with 256 units per layer, trained using a learning rate of 3 x 10-1 The policy network is a four-layer MLP, also with 256 units per layer. To enhance training stability, we employ a target policy a delayed version of the online policy-and and sample next-state actions from this module. For return distribution training "g we adopt the Implicit. Quantile Networks (TQN) (Dabney et al. 2018a) approach by sampling quantile fractions uniformly from (00 1). Additionally, dual critic networks with delayed updates are used, which empirically improve training stability. We use the following settings across tasks: replay buffer size 108, batch size 256, 24 quantile levels, and 10,000 pretraining steps. We evaluate every 10 steps, which takes 3.5 minutes.
IQ-Learn Adaptations. Key adaplations from IQ-Learn include adjustments to the regularizer coellicient and entropy coefficient a. Specifically, for the regularizer coefficient c, we find that 0.5 yields robust performance on the Humanoid task, while 0.1 works better for other tasks. For the entropy cocfficient a, smaller values lead to more stable training. Unlike RL, where exploration is crucial, imitation learning relies less on entropy due to the availability of expert data. Across all tasks, we set initial target reward parameters as x = 10 and \ = 5. Furthermore, we observe that lower learning rates for target rewards improve overall learning performance.
Previous implicit reward methods such as TQLearn, ValueDICE, and L.TC have employed distinct modifi- cations to the loss function. In our setup, two main loss variants are defined:
value loss:
C(T, Q)- = Epp (005a)-~V(()))) -((0)))))(((())))M))))))-- -V)))
v0 loss:
C(T, Q) Exx (Q(s,)(())))))))))))){)))) -V(()) (p (Vo - -CI -RTTR)))))
Here, p is a mixture distribution, Po denotes the initial distribution, Ro(s, a) is the implicit reward defined as as( = Q(s, a) - (s'), the state-value function is given by (ss)) Q(s', - a log (a'|s")) and lastly, convex regularizer is expressed I(Rq, X) Eps - Ep. (RQ -
)
The choice between v0 or value loss variants depends on environment complexity: we find that for a complex task like Humanoid-v2, the v0 variant demonstrates greater robustness. And, for 2 Walker2d-v2, Hvp Ant-v2, and Hammer-v1, the value variant performs better.
htaps://ggthuub.co///tm...
http:://ggttub..cmm/////////
https///gthab.ccc//rrffrr///////
14

---

Published in Transactions on Machine Learning Research (11/2025)
Table Hyperparameters for 3 and 10 Demonstrations (merged where identical)
Environment
a (3 10)
Ir π
Ir A
Ir A" (3 10)
Ant-v2
0.05 / 0.10
0.1
5 10-5
x 10
1x 10-3 x 10
HalfCheetab-v2
0.05 / 0.10
0.1
5X 10-5
1 10
1x 10
10
Walker2d-v2
0.05 / 0.10
0.1
5 10-5
1 10
1x 10-5 X 10-4
Hopper-v2
0.20 / 0.20
0.1
5x 10-5
1 10
1x 10
X 10
Humanoid-v2
0.05 / 0.10
0.5
1 1x 10-5
x 10-4
5 10
X 10-5
AdrottHandHHmmeerr
0.30 / 0.30
0.1
3 X 10
X 10
X 10
5 x 10-5
HaliCheetah-v2
Walker2d-v2
Hopper-v2
Retum
100
200
300
100
200
300
100
200
300
Ant-v2
Humannid-v2
Hammer-vl
Return
100
200
300
100
200
300
100
300
500
Steps (x10")
Steps (x10?)
Steps (x10?)
BC
LSIQ
SQUI
IQLemr
Expert
Figure 6: Learning curves on MuJoCo and Adroit tasks with expert demonstrations. Lines show the mean normalized return across five seeds; shaded regions denote 95% confidence intervals.
C
Additional Experiments
This section augments the main experimental results with additional experiments, including targeted abla- tions (critic architecture, regularization design, and loss choice) and hyperparameter sensitivity analysis.
C.1 Main Results (Extended)
Figure complements the main-text results by presenting learning curves for the throo-demonstration regime. Learning curves mirror the 10-demo setting (ef. Figure 3): RIZE and CSIL performs betler than the baselines with more stable learning across tasks; CSIL shows strong early progress due to sehavior-clloing initializa- tion; IQ-Learn is competitive yet below RIZE; and LSIQ/SQIL, trail behind. Task-wise, Humanoid-v2 remains solved only by RIZE, while on Hammer-vi RIZE and CSIL lead. As expected with fewer demon- strations, normalized returns are lower and variance higher, but the relative ordering of methods and the sample-efficiency patterns are consistent with the RLiable aggregates reported in the main body (Figure)
C.2 Recovered Reward (Extended)
We complement the main-text analysis with additional plots in the 3 demonstration regime and cross-method comparisons. First, Figure 7 mirrors the main-body analysis for the 10 10-demo setting: it overlays recovered
15

---

Published in Transactions on Machine Learning Research (11/2025)
HaliCheetah-v2
Walker2d-v2
Hopper-v2 v2
25
Reward
15
10
10-
0-
Ant-v2
Humanoid-v2
Hammer-vl
25
20
15
Roward
15
10-
Steps (x10") Expert reward
Steps Ox10") Expen
Steps (x10')
Policy
Policy
Figure 7: Implicit reward curves for expert and policy samples MuJoCo and Adroit tasks with 3 expert demonstrations. Each subplot reports the mean across five seeds, with shaded regions showing the 95% confidence interval. Theoretical upper and lower bounds derived in this work are overlaid as separate curves in each subplot.
reward trajectories (expert and policy) with the thcoretical bounds implied by our regularizer I(Rq, A) and adaptive targets (Corollary 4.2 As in the main text, the curves remain within the predicted band, and the band's position/ width adapts by task, reflecting how the adaptive targets track and limit expert/polde rewards in practice.
Across all three figures-Fiuurtt Figures and 10a RIZE and LSIQ keep implicit rewards bounded, whereas SQIL and TQ-Learn exhibit drifting or unbounded rewards on the challenging Humanoid-v2 and Hammer-vi tasks. In LSIQ, boundedness is likely due to its target clipping; notably, expert-sample rewards concentrate near zero. By contrast, SQIL fails to control reward scales nearly all tasks, and IQ-Learn is particularly unstable on Humanoid-v2 and Hammer-v1. These observations support the view that adaptive, task-sensitive regularization is effective in maintaining calibrated implicit rewards.
C.3 Critic Value-estimation
We examine how critic estimates evolve in RIZE versus baselines. Unlike methods that train point-estimate Q-networks, RIZE employs an TQN critic that models the full return distribution 7(x, a) and optimizes losses using expectation (s, (Dabncy al. (2188)). Bocause all policies are updated by maximizing state action values, unbounded estimates can destabilive training and undermine robustness.
Figures 10b and 12 show that RIZE and LSIQ maintain bounded values on the challenging Humanoid-v2 and Hammer-vi tasks, whereas SQIL and IQ-Learn exhibit large, drifting estimates. For LSIQ, boundedness primarily arises from larget clipping to [-200, 200], which also explains why expert-sample values cluster near +200. Importantly, bounded critics alone do not guarantee expert-level control within our training budgets (3c5 steps for MuJoCo; 5c5 for Adroit): LSIQ remains below RIZE Figures 3 and D indicating that value stability achieved by adaptive reward regularization can translate into better performance than rigid critic clipping with fixed reward targets.
16

---

Published in Transactions on Machine Learning Research (11/2025)
HalfChcetah-vy v2
Walker2d-v2
Hopper-v2
Ant-v2
Humanoid-V
0
300
Expen Polkcy
LSIQ Reward
0
-15
8
-8
200
100
-30
16
-16
0
-24
x10*
150
25
2
SQIL Reward
1.2
Polic
0
0.6
0
150
-25
0.0
-300
-50
X107
5.0
3
Expett
0.8
LeLm Reward
2
Pilicy
2.5
0.0
0.0
-0.8
24
RIZE Reward
16
20
16
Policy
Expot
10
0
0
-10
100
200
300
100
200
300
100
200
300
100
200
300
100
200
300
Steps (x10²)
Steps (x10²)
Steps (x10²)
Steps (x10²)
Steps (x100)
Figure 8: Implicit reward curves MuJoCo tasks comparing RIZE with SQIL, IQarennn, and LSIQ using 10 expert demonstrations. Lines show the mcan across five sceds; shaded regions denote 95% confidence intervals.
C.4 Ablation on Critic Architecture (Extended)
complements the main-body ablation by examining recovered rewards with theoretical bounds when RIZE uses a point-estimate Q(s, a) critic in place of the IQN return-distribntion critic Z(s, (cf. Figure 5 for returns, and Figure 7 for bounded rewards under IQN). With the Q critic, implicit rewards are less well constrained in Humanoid-v2 and Hammer-vi environments: we observe more excursions beyond the predicted band and larger oscillations, in contrast to the IQN setting where rewards remain lightly within the adaptive interval. This comparison suggests that both components contribute to reward stability: the squared-TD regularizer with adaptive targets provides principled bounds, and modeling the return distribution with IQN supplies steadier targets targets/updates /uddates that help keep rewards within those bounds. Together, these effects yield more reliable policy learning and align with the performance gap observed in the return curves.
Figure 13
C.5 Ablation on Loss Choice (HL-Gauss in LSIQ)
Recent work argues that some the gains attributed to distributional RL may stem from the loss rather than from modeling the return distribution itself (Farebrother el al. 2024 Ayoub et al. (0224.. Following this "loss swapping" view, we replace the mean-squared error (MSE) used in LSIQ's critic with the HL-Gauss classification loss (Farebrother al. 2024)), which discretizes the value range into bins and trains with haussian-smoother target over bins (turning value regression into calibrated classification).
Setup. Among our basclines, LSIQ (and SQIL) are natural candidates because their crities use MSE-like
objectives, whereas TQ-Learn and RIZE optimize MaxEnt-style IRL objectives. We therefore apply the swap
17

---

Published in Transactions on Machine Learning Research (11/2025)
HalfCheetah--2
Walker2d-v2
Hopper-v2
Ant-v2
Humanoid-v2
Expen Policy
LSIQ
Reward
0
20
-15
-8
-40
-10
-30
X10
x10
5.0
1.5
1.0
150
Expert
SQIL Reward
2.5
1.0
Polkcy
0.0
0.0
-2.
-150
0.0
-0.5
-200
X10'
5.0
3.0
Expen
IQLearn Reward
0.8
Polkyy
1.5
0.0
0.0
0.0
-2.
1.5
30
RIZE Reward
24
20
20
16
24
Expert
8
10
100
200
300
100
200
300
100
200
300
100
200
300
100
200
300
Steps (x10²)
Steps (x10²)
Steps (x10²)
Steps (x10²)
Steps (x100))
Figure 9: Implicit reward curves MuJoCo tasks comparing RIZE with SQIL, IQ-Learn, and LSIQ using three expert demonstrations. Lines show the mean across five sceds; shaded regions denote 95% confidence intervals.
LSIQ and keep all other components unchanged. Following LSIQ's critic clipping in [-200, 200), we set Umin -200, Vmax 200, use num bins=101, and Gaussian width 8. We evaluate Walker2d-v2, Ant-v2, and Hammer-vi with three demonstrations.
Results. As shown in Figure 14 HL-Gauss underperforms the original MSE-based LSIQ on all three tasks: it yields no improvement on Walker2d-v2 or Ant-v2, and remains near zero on Hammer-v1. likely cause is structural: LSTQ's critic objective is the sum two squared-error terms (expert and policy), which is integral to its beast-squares/mixtut formulation; swapping those squared losses for categorical (multi-bin)) classification surrogate alters the optimization geometry and weakens the intended expert-policy mixture shaping. In contrast, standard Q-learning wwhee loss swaps have shown benefits-regressee regresses to single TD target, making the classification surrogate a closer drop-in for MSE.
C.6 Ablations on Regularization Strategies
We study how the regularizer design affects performance Walker2d-v2, Ant-v2, and Hammer-vi with three demonstrations. Our baseline uses a squared TD-error regularizer I in Equation 7 with soparately optimized adaptive targets for expert and policy samples. We compare this to two alternatives: (i) a coupled target, where we replace the separate targets with a single shared (initialized to or 10), and (ii) a plain L2-regularizer on rewards (no targets), akin to IQ-Learn's constraint.
Figure shows that squared TD with seporate, adaptive targets yields the most robust and highest returns across tasks. Using coupled target is brittle: X=10 attains expert-like performance Ant-v2, remains
18

---

Published in Transactions on Machine Learning Research (11/2025)
Hammer-v n=3
Hammer-vl 10
Hammer-vl n=3
Hammoe-v - n=10
Zxpe Paks
1.50
150
Boen Pdly
LS1O
Roward
-90
LSIQ
0
-120
1.0
-1.90
-150
x107
X10
X1I
3.0
3.0
Raper
SQIL Roward
1.5
SQIL
(0..0)
1.5
Poller
0.0
50
X10P
X10
15
IQLewn
1.2
IQL Roward
2.5
(u5,0)
-25
-15
-0.6
x10
X10'
Zapert Packs
4.5
RIZE Rowar
RIZE Bl
3.0
500
100
300
500
Stepe Oxtof)
Stepe (K307)
Steps (x10")
Steps
(a)
(b)
Figure 10: Adroit Hammer-vl results: (a) implicit reward curves and (b) value estimation curves, each reported for 3 and 10 expert demonstrations (n denotes number of trajectories). Lines show the mean across five seeds; shaded regions denote 95% confidence intervals.
below the original result on Walker2d-v2, and fails on Hammer-v1, while λ=0 collapses across all tasks. Replacing squared TD with a plain L22 penalty also fails all three tasks in our setup. We attribute this difference from IQ-Learn's reports to implementation and hyperparameter choices tailored to our setting: RIZE employs an IQN critic, target policy networks, different learning rates (notably for the policy), a higher entropy coefficient (0.05 vs. 0.01), and a smaller regularizer coefficient (0.1 vs. 0.5). These choices were tuned for squared TD with adaptive targets; swapping to an L2 penalty disrupts learning. Overall, the results indicate that adoptive, decoupled targets are crucial for stabilizing reward across tasks.
C.7 Hyperparameter Tuning
We present our analysis and comparison important hyperparameters utilized in our algorithm. Plots depict mean over five seeds with 95% confidence intervals.
Adaptive Targets. Selecting appropriate initial values and learning rates for the automatic fino-tuning of A and X is critical in our approach. First, we observe that suitable learning rate essential for the stable training of our imitation learning agent, illustrated in Figure 16a Our findings indicate that x must be optimized very slowly; using larger learning rates destabilize training and hinder progress. In contrast, T demonstrates greater resilience when optimized with higher learning rates. Additionally, = remains robust even with varying initial values. However, as shown in Figure T6b failing to select an appropriate initial value for A can negatively impact Icarning. Overall, Figures 16a and 16b highlight the need for careful selection of both the learning rate and initial value when optimizing A", while A exhibits considerable robustness in this regard.
Rogularization Coefficient. Our experiments on the regularizer coefficient e reveal that smaller values of encourage expert-like performance, while larger values overly constrain rewards and targets, limiting learning. This finding highlights the critical role of selecting appropriate e, as it directly impacts the
19

---

Published in Transactions on Machine Learning Research (11/2025)
HalfChectah-v2
Walker2d-v2
Hopper-v2
Ant-v2
Humanoid-v2
200
180
160
Experi Pulicy
LSIQ
Q(s.a)
100
120
150
100
100
60
0
50
100
x10
x10
x10'
7.5
1.6
750
Expert
90
1.6
Policy
SQIL Q(x,0)
5.0
0.8
500
2.5
250
0.8
30
0.0
0.0
x10
30
60
Expest
240
Leam
(s,aa)
30
2
Policy
400
160
-30
200
80
-30
60
X10
x10
X10
1.5
900
900
Expert
L2 [((s
Policy
RIZE
600
1.0
300
300
0.4
0.4
0.5
0.0
0.0
0.0
100
200
300
100
200
300
100
200
300
100
200
300
100
200
300
Steps (x10²)
Steps (x10²)
Steps (x10²)
Steps (x10?)
Steps (x110)
Figure 11: Valhee-stimaoion curves MuJoCo tasks comparing RIZE (IQN critic (Dbnney et al. with SQIL, IQ-Learn, and LSIQ (point-ostimate Q) using 10 expert demonstrations. Lines are means over five seeds; shaded regions denote 95% confidence intervals.
2018a
balance between learning from expert data and regularization: higher values prioritize regularization at the cost of learning, whereas smaller values favor learning but reduce regularization (see Figure 17a).
Entropy Coefficient. We observe that the entropy coefficient a crucial hyperparameter inverse reinforcement learning (IRL) problems. As shown in Figure 17b] IRL methods typically require small values for a, point previously noted (Garg et al. 2021). With expert demonstrations available, an imitation learning (II.) policy does not need to explore for optimal actions, as these are provided by the demonstrations. Consequently, higher values of a can lead to training instability, ultimately resulting in policy collapse.
20

---

Published in Transactions on Machine Learning Research (11/2025)
HalfChectah-ve v2
Walker2d-v2
Hopper-v2
Ant- v2
Humanoid-y -2
200
150
180
180
160
Experi Policy
100
LSIQ
((s,a)
120
120
60
-150
x107
x10
x10
x10*
1.6
7.5
5.0
Exper Policy
90
3.0
SQIL
((s,a)
0.8
5.0
2.5
60
0.0
2.5
0.0
1.5
30
0.0
-2.5
0.0
x10
x10"
LQLearn
30
2
300
40
6
Policy
Expert
((..a)
4
-30
0
100
2
-60
-40
x10
x10
x10
x10
1.5
1.2
1.5
Expert
RIZE
E|2(s, a)
0.8
1.0
800
Policy
0.5
400
0.5
0.0
0.0
0.0
0.0
100
Steps (x10*)
Steps (x10*)
Steps (x10+)
Steps (x10²)
Steps (x100)
Figure 12: Valuo-estimation curves on MuJoCo tasks comparing RIZE (IQN critic DDbnnyy et al. 2018 (188a)) with SQIL, IQ-Learn, and LSIQ (point- ppoint-estimate Q) using three expert demonstrations. Lines are means over five seeds; shaded regions denote 95% confidence intervals.
21

---

Published in Transactions on Machine Learning Research (11/2025)
HaltChectah-v2
Walker2d-v2
Hopper-v2
20
Reward
10
10-
5-
0-
-5
100
200
300
100
200
300
100
200
300
Ant-v2
Humanoid-v2
167
Hammer-vl
300
20
200
Reward
100
10-
0.0
100
-0.5
-200
100
200
300
100
200
300
100
300
50U
Steps (x10')
Steps (x10")
Steps Ox10")
reward
Policy reward
Expen 2+
Policy
Figure 13: Implicit reward curves for expert and policy samples on MuJoCo and Adroit tasks using RIZE with a classic Q(s, a) critic. Each subplot shows the mcan over five sceds with shaded 95% confidence intervals, using three expert remonstrations. Theoretical upper and lower bounds derived in this work are overlaid as separate curves in each subplot.
Walker2d-v2
Ant-v2
Hammer-vl
Retum
Stops (x10P)
Steps (x10?)
Steps (x10))
LSIQ (MSB)
LSIQ (HI--Gunas)
Expert
Figure 14: Loss swap in LSIQ: MSE HL-Gauss Walker2d-v,, Ant-v2, and Hammer-vi with three demonstrations. We keep Miin 200, Vmax 200, num bins=101, and a = 8. HL-Gauss does not improve over MSE and remains near zero Hammer-v1.
Walker2d-v2
Ant-v2
Hammer-vl
Retumn
100
200
300
100
200
300
100
300
500
Steps (x10")
Steps (x10*)
Steps (x10²)
(main)
(coupled, O=
(cospled,
(mo TD)
Ixppet
Figure 15: Ablation of regularization strategies on selected tasks (Walker2d, Ant, Hammer) using three expert demonstrations. Results are expert-normalized and reported as the mean over five seeds with 95% confidence intervals.
22

---

Published in Transactions on Machine Learning Research (11/2025)
Ant-v2
Ant-v2
Return
Return
RIZE =0==0====--1 Ir == 10-))
RIZE (2 = 10, 2 5)
RIZE ======1: 10-1 2 10-1)
RIZE (2 = 10, = 0)
RIZE =0= Ir 2 10-))
RIZE =1 ==
100
200
300
100
200
300
Steps (x10²
Steps (x10²)
(a)
(b)
Figure 16: tuningg analysis of adaptive targets. (a) Learning rates: Turquoise represents our method's primary result with learning rates of le-4 for x and 1e-5 for X= Orange and blue lines indicate higher learning rates (e.g., le-3) for x and X", respectively. (b) Starting values: Turquoise shows the main result with initial values of 10 for x and for A", while other lines explore different starting values. Three trajectories are used throughout the analysis.
Ant-v2
Ant-v2
Return
Return
RIZE (w/ a=0.55)
RiZE a00.)
RIZE (w/ C=1..)
RIZE (w/ C=0.6)
100
200
300
100
200
300
Steps (x10²)
Steps (x10²)
(a)
(b)
Figure 17: (a) Effeet of the regularizer coefficient c. Turquoise shows the primary result of our method with 0.1, while gray represents larger value (c= 0.6). (b) Effect of the temperature parameter Turquoise shows the result with a 0.05, and gray corresponds to larger value (a = 0.5). Three trajectories are used throughout the analysis.
23


Reviews:
[{"review_id": "NwpnBKkBuW", "reviewer": "Reviewer_uutq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces RIZE, a novel imitation learning method that extends the Maximum Entropy IRL framework with two key innovations:\n- Adaptive reward targets for squared TD regularization: Unlike prior implicit reward methods (e.g., SQIL, LSIQ) that fix reward targets to $\\pm 1$, RIZE learns adaptive targets $\\lambda^{\\pi_E}, \\lambda^{\\pi}$ that evolve during training. This provides flexible, context-sensitive reward alignment and provably bounds the implicit reward within a well-defined interval.\n- Integration of distributional reinforcement learning: The authors incorporate distributional critics (via quantile-based value networks) into IRL. They optimize policies with respect to the expectation of the learned return distribution, capturing richer uncertainty while stabilizing learning.\n\nThe authors present a theoretical analysis of reward boundedness (Proposition 4.1, Corollary A.2--A.3), a practical algorithm built on distributional SAC, and extensive experiments on four MuJoCo benchmarks. RIZE consistently outperforms SQIL, LSIQ, and IQ-Learn, and uniquely achieves expert-level performance on the challenging Humanoid-v2 task with only three expert demonstrations. Ablations examine the role of adaptive regularization, distributional critics, and hyperparameter sensitivity.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- Theoretical claims: The paper provides clear derivations showing that the adaptive TD regularizer constrains rewards within bounded intervals. The proofs in Appendix A are correct and easy to follow. The authors are careful to state that convergence guarantees for alternating updates remain open, which appropriately limits their theoretical claims.\n\n- Empirical claims: The main claim — that RIZE improves robustness and achieves expert-level Humanoid performance with few demonstrations — is supported by strong results. Ablations show that both adaptive regularization and distributional critics matter. However:\n    - The evaluation metric “top 25% returns” is non-standard. While it highlights convergence, conventional practice (mean or median returns with confidence intervals, AUC, final performance) would be a stronger basis for claims of efficiency and stability.\n    - Hyperparameter tuning fairness across baselines, particularly in the 3-demo setting, is not fully transparent. RIZE appears to receive environment-specific tuning and alternative losses (value vs.\\ v0), whereas it is unclear if baselines were given comparable treatment.\n\nOverall, the evidence is convincing for RIZE’s effectiveness, but additional results and clarifications would further strengthen the claims.", "audience_yn": "Yes", "audience_explanation": "Yes. The work addresses fundamental challenges in imitation learning — robust reward recovery and stability under limited expert data. Both adaptive regularization and distributional critics are conceptually novel in the context of non-adversarial IRL, and the empirical demonstration on Humanoid with only three demonstrations is particularly appealing. TMLR’s audience, which includes researchers in reinforcement learning, imitation learning, and robotics, would be interested in these findings.", "requested_changes": "- Provide conventional learning curves (mean $\\pm$ CI over seeds/episodes), final-100-episode means, and AUC, alongside the “top-25%” metric currently emphasized.\n- Clarify how tuning budgets and search spaces were matched across RIZE and baselines, especially in the 3-demo regime.\n- Isolate the contributions of (a) quantile-based critics vs.\\ point-estimate critics, (b) number of quantiles, and (c) stabilizers like target-policy and double critic.\n- Provide time-series plots of $\\lambda^{\\pi}, \\lambda^{\\pi_E}$ and histograms of recovered rewards to confirm they respect the theoretical bounds.\n- Add Behavioral Cloning (with and without fine-tuning) as a low-data reference, and optionally include an adversarial baseline (e.g., GAIL or DAC) on at least one environment.\n- Expand the appendix with implementation details such as replay-buffer size, batch size, target-network update rate $\\tau$, quantile samples per update, warm-up steps, evaluation frequency, and wall-clock compute.", "broader_impact_concerns": "The broader impact is generally positive. RIZE aims to improve robustness and efficiency of imitation learning, which is directly relevant to robotics and autonomous systems. Potential concerns include:\n- Reward mis-specification risks: While adaptive bounds improve stability, learned rewards could still encode biases in expert demonstrations.\n- Safety-critical domains: Using adaptive implicit rewards without strong convergence guarantees could pose risks in safety-sensitive applications (e.g., healthcare, autonomous driving).\n\nThese concerns should be acknowledged, but none appear to be critical blockers.", "additional_comments": "- The writing is clear, and the introduction and related work position the contribution well.\n- The theoretical analysis is concise and correctly caveated.\n- Figures are generally informative, though including raw curves and confidence intervals would improve interpretability.\n- Table 1 is helpful, but reproducibility would benefit greatly from code release.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3kkHdcxZTq", "reviewer": "Reviewer_h3JC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n\nThis paper introduces an online imitation learning method, called RIZE,\nwhich extends the IQLearn method by introducing the following\nimprovements:\n\n1.  Substituting the Q-function parameterization with a distributional\n    variant, and using the induced mean in place of the Q-function; and\n2.  An adaptive reward regularization scheme, which regularizes rewards\n    toward learned \"centers\" over the course of training.\n\nThrough experiments in Mujoco tasks, RIZE considerably outperforms the\ntested baselines (including IQLearn). Notably, in the Humanoid task,\nRIZE approximately recovers expert performance, while the baseline\nmethods fail entirely.\n\n## Strengths\n\nThe proposed method is simple and easy to implement, while achieving\nstrong performance. The adaptive regularization scheme is novel and\ninteresting. The authors carry out a few ablations to justify the\nnecessity of the novel pieces of RIZE for achieving strong performance.\n\n## Weaknesses\n\nThe modifications made to IQLearn are not very well motivated.\nParticularly:\n\n- **Distributional RL**. The only motivation really appears to be that\n  modeling return distributions is successful in RL, so it should be\n  successful in imitation learning. But in principle, there is no clear\n  reason for this to be the case, and as I discuss below, there are\n  confounding factors that might explain the performance achieved by\n  incoroporating distributional RL in this framework.\n- **Adaptive reward regularization**. This is a seemingly novel\n  technique, but again it is not clear why it should be expected to\n  work. Any theoretical results about this scheme would hold for the\n  case where the regularization parameters are zero (which is the case\n  of IQLearn)—thus, these results do not actually suggest that learning\n  the regularization is beneficial in principle. In fact, it is not\n  clear to me why this doesn't lead to divergence.\n\nMoreover, as I explain below, I have some concerns with the experimental\nprocedure, as well as the breadth of the experiments, baselines, and\nablations. Consequently, it is difficult for me to fully appreciate both\nthe performance of the proposed method relative to the current state of\nthe field and the effects of the individual components that were\nintroduced.", "claims_and_evidence_yn": "No", "claims_explanation": "The theoretical results appear to be correct. However, the claims about\nthe empirical performance are too strong from my perspective. This is\ndue to the relatively small collection of environments tested relative\nto competing works, the lack of recent strong baselines, and the lack of\nimportant ablations (each discussed below).", "audience_yn": "Yes", "audience_explanation": "Online imitation learning is a very active area of research in machine\nlearning. Despite my concerns about the empirical results, the proposed\nmethod does achieve strong performance in the difficult Humanoid task,\nand especially due to its simplicity, I believe the imitation learning\ncommunity will find this useful.", "requested_changes": "## Critical changes\n\n1.  The literature review omits several works on non-adversarial\n    approaches to IRL / imitation learning. Off the top of my head,\n    [1] and [3-4] from the references I listed in **Additional Comments**\n    appear quite relevant.\n2.  Regarding the performance of RIZE, while it appears quite good, it\n    is tested on relatively few environments/tasks. Additionally, it is\n    not compared to some recent baselines which I suspect would perform\n    very competitively (see e.g. [2-4] in the references I listed under\n    **Additional Comments**).\n3.  While I appreciate the ablation in Section 5.3, some questions\n    still remain. For instance, [5-6] in the references I listed in\n    **Additional Comments** argue that the performance improvements brought\n    along by distributional RL should be attributed to the loss\n    function, and not the fact that they're modeling the return\n    distribution. Thus, perhaps by substituting the Q function MSE in\n    any of the tested methods with something like HL-Gauss [6], we\n    could see performance enhancements rivaling RIZE.\n4.  Figure 1 is difficult to interpret. It says it's plotting the\n    return of the \"top 25% episodic returns across 5 seeds\". How\n    exactly is this computed? How many rollouts do you estimate for\n    each seed? Is the top 25% taken seed-wise or over all returns\n    generated by a given method? Based on the information given, it\n    sounds like you're running one rollout per seed and plotting the\n    top 25%, which is (I'm guessing?) simply the best return. Also,\n    there seems to be a discrepancy between Figure 1 and 2; somehow the\n    baselines look much weaker in Figure 1. The results should be\n    explained more precisely.\n5.  In section 5.4, the ablation should be conducted on the full suite\n    of tasks&#x2014;it is not clear that any conclusions drawn from Ant\n    generalize.\n6.  In section 5.4, I believe there is a crucial ablation missing:\n    namely, the setting where $\\lambda^\\pi = \\lambda^{\\pi_E}$. The\n    \"expert-focused TD\" and \"policy-focused-TD\" configurations that you\n    test are quite unnatural to me: I don't know why there would be\n    such an asymmetry to the regularization method for the reward\n    function under the policy's vs the expert's state visitation\n    distributions. However, intuitively I would have thought the method\n    would make the most sense with $\\lambda^\\pi = \\lambda^{\\pi_E}$,\n    which still employs a degree of adaptive regularization that you argue for.\n\n\n## Strengthening changes\n\n1.  The term \"value distribution\" isn't entirely accurate, \"return\n    distribution\" is preferred. This is because \"value\" refers to the\n    value function which measures expected returns by definition,\n    whereas the \"return\" is random (and its distribution is what is\n    modeled by distributional RL).\n2.  Towards the end of section 1, it says \"complex tasks like *Humanoid*,\n    where both adaptive rewards and distributional learning are\n    essential\". Can you provide a citation for this? Surely\n    distributional RL is not essential for humanoid, since many methods\n    can solve this task without distributional RL. Also, it is not\n    clear yet what you mean by \"adaptive rewards\".\n3.  There are inaccuracies in the discussion of the distributional soft\n    Bellman operator. Particularly, under equation (1), you claim that\n    the operator exhibits contractions properties and that this yields\n    convergence to the optimal distributional value function. The\n    discrepancy is the following:\n    1.  The operator $\\mathcal{B}^\\pi_D$ that you define is a policy\n        evaluation operator, its iterates do not yield an optimal\n        distributional value function.\n    2.  The closely-related distributional soft optimality operator is\n        not a contraction (so, contraction properties are not\n        responsible for convergence to an optimal distributional value function).\n    3.  Despite the fact that the distributional soft optimality\n        operator has convergent iterates, its iterates do not converge\n        to a classically optimal distributional value function; it is\n        only optimal for the entropy-regularized RL objective (but this\n        is more of a nit).\n4.  Just above section 3.3, you refer to \"the value distribution\n    $Z_\\tau(s, a)$\", but this is not a distribution&#x2014;for any \n    $\\tau$, $Z_\\tau(s, a)$ is a scalar.\n5.  Since Corollary A.2 is framed (at least from my perspective) as a\n    key result, it should be in the main text.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "1.  The motivation for the adaptive reward targets is not at all clear\n    to me. Particularly, under this parameterization, is it not\n    possible (perhaps even likely) for the $\\lambda^{\\pi_E}$ and\n    $\\lambda^\\pi$ terms to diverge? It's especially odd to me that\n    $\\lambda^\\pi, \\lambda^{\\pi_E}$ are \"decoupled\", did you try\n    enforcing $\\lambda^\\pi \\equiv \\lambda^{\\pi_E}$?\n2.  Can you explain why the IQLearn implicit rewards diverge in\n    Humanoid? From what I can tell, the objective of IQLearn is\n    equivalent to that of RIZE, but with $\\lambda^\\pi = \\lambda^{\\pi_E}\n       = 0$ (equivalently, $\\lambda_{\\min} = \\lambda_{\\max} = 0$).\n    So by Corollary A.2, the implicit rewards of IQLearn should also be\n    bounded (and even to a smaller range than those of RIZE). So by\n    your own argument for RIZE, shouldn't IQLearn at least produce\n    conservatively bounded implicit rewards?\n3.  In Figure 6, I do not see the curve for RIZE with baseline $L_2$\n    regularization. Is it lying on the $x$-axis? This would surprise\n    me, given that this formulation appears almost equivalent to\n    IQLearn, which performed well in Ant.\n4.  I believe the paper places too much attribution to distributional\n    RL. The RIZE method effectively only uses distributional RL as an\n    empirical trick; the return distributions are used *only* to extract\n    expected values, so the distributions themselves are not\n    contributing anything explicitly. Thus, it feels like a stretch to\n    title the paper \"Regularized Imitation Learning via Distributional\n    RL\". It also removes emphasis from the adaptive regularization\n    scheme, which seemingly also plays a large role, and which\n    actually changes the learning objective.\n\n\n## References\n\n1.  \"Of Moments and Matching\". Swamy et al., 2021. *International\n    Conference on Machine Learning.*\n2.  \"Adversarial Imitation Learning via Boosting\". Chang et\n    al., 2024. *International Conference on Learning Representations*.\n3.  \"Non-Adversarial Inverse Reinforcement Learning via Successor\n    Feature Matching\". Jain et al., 2025. *International Conference\n    on Learning Representations*.\n4.  \"Diffusing States and Matching Scores: A New Framework for\n    Imitation Learning\". Wu et al., 2025. *International Conference\n    on Learning Representations*.\n5.  \"Switching the Loss Reduces the Cost in Batch Reinforcement\n    Learning\". Ayoub et al., 2024. *International Conference on Machine\n    Learning*.\n6.  \"Stop Regressing: Training Value Functions via Classification for\n    Scalable Deep RL\". Farebrother et al., 2024. *International\n    Conference on Machine Learning*.", "extra_scores": {}, "extra_text": {}}, {"review_id": "whrCka41e2", "reviewer": "Reviewer_q66R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper tackles the problem of inverse reinforcement learning (RL), i.e. learning a policy given only trajectories and their returns. \nThe authors argue that previous methods in this area, which only considered point-estimate Q-value functions, do not capture the full information and are thus slow to converge.\n\nIn this work, the authors leverage the distributional Q-value functions, which capture the inherent aleatoric uncertainty in returns under stochastic policies. They theoretically showed that the reward functions learned by such functions are bounded and thus promote stable optimization. The resulting practical algorithm, RIZE, is shown to be able to learn well-performing policies (in terms of returns) with only a few trajectories to learn from.\n\nOverall, the manuscript is well-written. The presented results are compelling, albeit hard to judge their significance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The main claim of this paper is that using value functions that take (aleatoric) uncertainty into account is beneficial for inverse RL. The authors showed compelling evidence:\n\n1. In Prop. 4.1, the authors analyze the impact of such a decision on the (implicit) reward function learned from expert demonstrations/trajectories. They showed that this reward function is bounded, and thus makes optimization stable\n2. In the empirical analysis (Sec. 5), the authors provided empirical validation for the previous point, along with an ablation study.", "audience_yn": "Yes", "audience_explanation": "The general insight of incorporating various sources (in this case, aleatoric) uncertainty for improving decision-making and optimization is useful.", "requested_changes": "Some comments to strengthen the submission:\n\n1. I believe it would be very useful if the authors discuss at a high level the intuition on why considering aleatoric uncertainty in the Q-value function leads to a bounded reward function. While I could see that considering uncertainty, i.e., having more information, is generally useful for various tasks like optimization and learning, the connection between uncertainty and boundedness is unintuitive to me.\n2. I would suggest the authors provide a significance test on top of the empirical plots. Eyeballing plots is unreliable for judging significance.\n3. I suggest the authors open-source their implementation.", "broader_impact_concerns": "Doesn't seem to be applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)