Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learning
Authors: Shangzhe Li
Dongruo Zhou
Weitong Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: imitation learning
model-based learning
Primary area: reinforcement learning

Abstract: We study online adversarial imitation learning (AIL), where an agent learns from offline expert demonstrations and interacts with the environment online without access to rewards. Despite strong empirical results, the benefits of online interaction and the impact of stochasticity remain poorly understood. We address these gaps by introducing a model-based AIL algorithm (MB-AIL) and establish its horizon-free, second-order sample-complexity guarantees under general function approximations for both expert data and reward-free interactions. These second-order bounds provide an instance-dependent result that can scale with the variance of returns under the relevant policies and therefore tighten as the system approaches determinism. Together with second-order, information-theoretic lower bounds on a newly constructed hard-instance family, we show that MB-AIL attains minimax-optimal sample complexity for online interaction (up to logarithmic factors) with limited expert demonstrations and matches the lower bound for expert demonstrations in terms of the dependence on horizon $H$, precision $\epsilon$ and the policy variance $\sigma^2$. Experiments further validate our theoretical findings and demonstrate that a practical implementation of MB-AIL matches or surpasses the sample efficiency of existing methods.

Published as a conference paper at ICLR 2026
NEAR-OPTIMAR SECOND-ORDER GUARANTEES FOR MODEL-BASED ADVERSARIAL IMITATION LEARNING
Shangzhe Li
Dongruo Zhou Indiana University Bloomington Bloomington, IN, USA dz 10 id.edu
Weitong Zhang UNC Chapel Hill Chapel Hill, NC, USA we weitonge weitoongz@unc.een edu
UNC Chapel Hill Chapel Hill, NC, USA Shangzhe@f uun..unc.cc.
ABSTRACT
We study online adversarial imitation learning (AIL), where an agent learns from offline expert demonstrations and interacts with the environment online without access to rewards. Despite strong empirical results, the benefits online inter- action and the impact of stochasticity remain poorly understood. We address these gaps by introducing a model-based AlL algorithm (MB-AIL) and estab- lish its horizon-fres second-order sample-complexity guarantees under general function approximations for both expert data and reward-free interactions. These second-order bounds provide an instance-dependent result that can scale with the variance of returns under the relevant policies and therefore tighten as the sys- tem approaches determinism. Together with second-ordes information-ther theoretic lower bounds on newly constructed bard-instance family, we show that MB-AIL attains minimax-optimal sample complexity for online interaction (up to logarith- mic factors) with limited expert demonstrations and matches the lower bound for expert demonstrations in terms the dependence on horizon H, precision and the policy variance o². Experiments further validate our theoretical findings and demonstrate that a practical implementation of MB-AIL matches or surpasses the sample efficiency of existing methods.
INTRODUCTION
Imitation learning (IL) focuses on learning a policy that mimics expert behavior for sequential decision-making using demonstrations. Unlike reinforcement learning (RL), IL learns from expert trajectories without the reward signals. This advantage has drawn substantial attention to IL, with demonstrated successes across diverse real-world applications, including robot learning (Chi al. 2024 Shi et al. 2023) and autonomous driving (Couto & Antonelo, 2021 Cheng et al. 2024) where the reward signal is hard to design implicit to learn.
Among the many variants imitation learning, methods largely fall into two families: behavioral cloning (BC) and adversarial imitation learning (AIL). In particular, BC applies supervised learn- ing to fit the expert policy directly from demonstrations (Florence et al. 2022) CCi et al., 2024), while AIL employs an adversarial framework to align the state-action distributions of expert and learner policies (Ho & Ermon 2016 Garg et al.) 2. Importantly, AIL can also leverage online, reward-free interactions during training. This online interaction will allow the agent to collect the environment dynamics information in order to better align with the behavioral policy. To explain empirical findings that AIL often outperforms BC in low expert demonstration regimes (Ho & Er- mon. 2016 Garg et al.] 2021), Xu et al. (2022) analyzed this phenomenon under a specific MDP structure, relating it to the expert sample complexity results of Rajaraman et al. (2020). Naturally, have the following question:
What is the tight characterization of the benefits of online interaction in imitation learning?
In order to obtain tight characterization, two aspects need to be addressed. First, a line of research
has extended the model- -aased framework in RL (Sun et al. 2019a Wang et al.] 2024) to AIL, both theoretically (Xu et al. 2023) and empirically (Baram et al. 2016 Kolev et al. [2024] Yin et al. 2022 Li et al.) 2025), partially demonstrating that model--asee approaches can yield superior sample efficiency. However, existing theoretical analyses for model-based II. have not yet clarified why such

---

Published as a conference paper at ICLR 2026
approaches are beneficial to AIL under the general setting. Second, it has been observed that the stochasticity of the experts plays an important role in achieving tight sample complexity. Existing works analyze BC and AIL under deterministic versus stochastic experts and have established gap: for deterministic experts. Rajaraman et al. (202)) Foster et al. (2024) obtain an ÖEE expert sample complexity, whereas for stochastic experts, existing results typically yield Ō 2) (Foster ct al., 2024; Xu et al., 2024). Nevertheless, a complete understanding of how stochasticity affects
the sample complexity of AlL remains open.
In this paper, we answer this question affirmingly by presenting model- AIL algorithm (MB-AIL) with a corresponding horizon-free, mation understand the benefits of online interaction and the impact of stochasticity the sample complexity to both expert demonstration and online interaction. A second-order bound scales with the variance of the total return and can be small as the system approaches determinism, providing a powerful tool for us to demonstrate the effect of stochasticity. Combining with a lower bound on the minimax sample complexity for both expert and online interaction, we understand the benefit and barrier of the online interaction statically. In particular, our contributions can be listed as follows:
seceede,cccoooree second-order analysis in general function approxi-
We present a model-based adversarial imitation learning algorithm (MB-AII inspired by the fact that the policy class II can be split into the reward class R and the model class P. -IB decomposes this reward learning and model learning procedure by estimating the reward adver- sarially using expert demonstration using no-regret algorithm and learn the model using simple MLE estimator for the model and an optimistic exploration strategy for online interaction.
We present a second-order, horizon-free bound for MB AIL under general function approximation on sample complexity for both of the expert demonstration and online interactions. This result re- veals the benefit of the online interaction and the impact of the variance to the sample complexity. At the core of our analysis is to decompose the regret into the error of reward and the error of the model estimation. Then two fine-grained variance-depend analysis is conducted in both of the part for horizon-free second-order regret analysis in model-based RL.
pntommation-theoretii lower bound for both of the expert demonstration and online interaction on a newly designed hard instance for imitation learning. The lower bound reveals that the expert demonstration can help with estimating the reward while the online interaction is tied with learning the transition kernel. This lower bound also suggests that the MB-AIL is minimax-optimal in terms of the online interaction up to log factors when given limited expert demonstrations. While MB-AIL leaves only a log [R] gap in terms of the expert demonstrations.
We present
Besides the theoretical advancements in understanding the online interaction in imitation learning. we also present a practical implementation of the proposed M3-AII algorithm to validate our theo- retical findings. MB-AII L also reported with better sample complexity in a various of environment.
Notation. In this paper, we use plain letters such as x to denote scalars, lowercase bold letters such as x to denote vectors, and uppercase bold letters such as to denote matrices. Functions are denoted by bold symbols such as f. Sets and classes are denoted by the calligraphic font such as F. For vector x, ||x||2 denotes its norm fg. For a positive integer N, we use [N] to denote {1, 2,..., N). We use standard asymptotic notations, including (c-), 2()), 0(.), and O+),, (()) which will hide logarithmic factors. For two distribution p and g we define the Hellinger distance as H (p, q)
dp//dw dg/dw dw. We denote the d-dimensional binary set by Bd = {+1]d that is, the set of all d-dimensional vectors whose entries are either +1 or -1
2 RELATED WORKS
Below we provide comprehensive review of the related work on theoretical understanding of imi- tation learning. We defer more detailed discussion on the empirical implementation on imitation learning, theoretical background of reinforcement learning into Appendix
Theoretical Understanding in Imitation Learning. Imitation learning can be interpreted as variant of RL in which the agent learns from expert demonstrations instead of reward signals. Early
works (Abbeel & Ng 2004. Syed & Schapire 2007 Sun et al.) 2019b; Rajaraman et al.) 2020 Chen cet al. 2020) assumes either known transition dynamics or access to an exploratory data distribution. More recent studies consider the unknown transitions and provide theoretical guarantees the
2

---

Published as a conference paper at ICLR 2026
Table I: Summary of Existing Results. We summarize existing upper and lower bound results on imitation learning and compare them with our results. Stoch. Expt? indicates that if the presented analysis can deal with the stochastic expert or is for deterministic expert only. Demonstration indicates the number of expert demonstration trajectories required for achieving c-optimal policy. Interaction indicates the rounds of online interaction which is not required for the algorithms marked with "Offline BC Algorithms that may require an online interaction without specifying a bound is left with '-'. The sample complexity is evaluated under the bounded cumulative reward Σ Σi=1 Th I 1 and the results with bounded reward assumption rs. S are translated by letting <- He in the original results and marked with (*). Dimension dæ and model class log [[I], log (R], log [2, log [P] in general function approximations inherits their original definitions in the paper, and it usually corresponds to the dimension of linear function when reduced to linear function approximations. Variance o² refers to the variance of the collected return, as defined in detail in the paper, and is always bounded by H² In contrast, oe denotes the variance of the collected return under the expert policy. Our results are highlights in boldface and in cyan background.
Setting
Algorithm
Stoch. Expt?
Demonstration
Interaction
Rajaraman et al.
2020
O(H(S
Offline BC
TV-AIL Xu
O(SS) 2) ounvass by
Tabular MDPs
MB MB-TAILL Xu el al.
2023
O Py" Oflline BC
Lower Bound Rajaraman et ml 2020
Lower Bound Xu et al. 2022
Linear MDPs
BRIG Viano et
2024
Ölde-y
O(H"""-=")) O(H""''-
Lincar Mix. MDPs
OGAIL
[Liu ct al
2021
O(HdPe"))
Foster et al.
2024
Olo log [[] ?)
Offline BC
General
Foster el al.
2024
Oflog [i]le 9)
Oflline BC
OPT-AIL Xu all.
2024
Ötlog [R]
O(fAds [R][[]]
Function
MB-AIL (Ours)
V
Olok (R))))))
Olo"dg log( Pl)e
Lower Bound (ooster et al
2024
Dl 1
Offline BC
Approximation
Lower Bound
(Foster et al 2024
V
Blofe 9) Mlake")
Offline BC
Lower Bound (Ours)
[Pog]]]]) (ex) exp[-N]ee
sample complexity of expert data under various structural assumptions, For example. Shani et al. 2022) provide an O(VK) regret upper bound for on-policy AlL, while Chen et al. (024)) provide a convergence guarantee for off-policy AlL in tabular MDPs. Moulin et al. (2025b) investigates the state-only imitation learning setting under discounted linear MDPs and provides an expert
sampling complexity upper bound O (1x ?) ??- and interaction complexity upper bound (d8 - -1) Xu et al. (2023) study the tabular MDP setting with a deterministic expert policy, obtaining an expert sample complexity of ((H5/2|S)) and interaction complexity of ((F3 Viano et al. (0244) obtain d/H2d/e2) expert sample complexity with the same O(H*d /e²) interaction complexity for linear MDPs. Liu al. 2021) establish Ö(HF) 2 expert sample complexity and *(H+dd/ /e2) interaction complexity for linear mixture MDPs. For general function approximation, Xu et al. 2024) provide bounds of (HF (o)()( on expert sampling complexity and Ö((I1* d log (og(N(R)N(()) + //3)/e) on interaction complexity.
Information-Theoree Lower Bounds for Imitation Learning. Several information-theoreds minimax lower bounds have been established for the number of expert policies. In the tabular MDP with deterministic expert, Rajaraman al. (0020) prove a lower bound of D(2SSHH2/ Foster et al. 2024) further derive lower bounds of (HH/e) )) for deterministic experts and D( (2) for stochastic experts. Similar lower bounds has been studied in in[Xu et al.) (2022) with minimax- optimal algorithm under a special class of MDPs. However, none of these infformation-thhoreee lower bound targets for the sample complexity of online interaction. Moulin et al. (2025b) provides a lower bound for expert sampling complexity 2(W2 - 7) -2))2))) and a lower bound for inter- action complexity [dd(1 - 7) -2) for all imitation learning algorithms under discounted linear
MDPs. We record part of these existing results in Table
"TV-AIL considers a special class of tabular MDP referred RBAS-MDP.
"The lower bound in Xu et al. (2022) is only for the TV-AIL. algorithm thus not minimax lower bound. f . is the number of expert demonstrations and can be small.
3

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
sime-homogeneous Episodic MDPs. We consider the time-homogeneouss episodic Markov deci- sion processes (MDPs) denoted by M = A, H. P*, by convention. Here, S and A are state and action spaces, H is the length of each episode, P S A + A(S) is the transition probability from state to 8 with action R 3 pr:: 8 x A + 00 1] is the reward function. For any policy and reward r, the state-action value function hh (x, a) on stage h defined as:
(T. IE E a8), 8n a, = 8c+1 (* ae), d++1 m(S+1))
and the value function is V :r) Popp. Ta (s)). The optimal values are defined by:
(. (s): pr (s, a) max, PhP (s.a). pr (s) max; V Pr
For any value function V : S + and stage h E [H], we define the first-order Bellman operator associated with policy n, reward function r, and transition model P* as
(TrVi ()) r(s, a) + Bg (-1.,)) [V+L = r(,, a) + [P PP' (VI..e...
The conditional variance of the value function under state-action pair (s, a) is defined as
P ([ a) = [P' (P= a) - (IP"V](s,a))"
The variance of the cumulative reward collected by policy л under reward function r is defined by:
VaR," = E [ (EX r(sn, an) ) - (Vi....(s)))
the expectation is taken with respect to the trajectory distribution induced by the policy æ and transi- tion kernel P*, i.e., ag ~ (. sx) and 5h+1 ~ (* (h, state is fixed sj. This can generalized
Sh, az). In this paper, we assume the initial distribution of the initial state w.l.o.g..
Online Adversarial Imitation Learning, In this paper we consider the online adversarial imitation learning with offline demonstration. To be specific, the agent is given offline dataser Dg 1 containing N trajectories rolled out by the expert policy F The agent is allowed to online interact with the environment. Throughout the offline dataset and online interactions, the agent performing in a reward-free paradigm such that it has no access the collected reward in either offline demonstrations and online explorations.
{sh, aj hE[H]
[E[N]
The goal of the adversarial imitation learning (AIL) is to output a policy л such that the policy x and T have similar behavior over the reward functions, usually defined by the typical AIL objective by
min max E. EVIV R
(s),,
(3.1)
FER
where the expectation is taken over the initial state distribution. In the online setting, we further define the regret of this All. objective over K rounds by
Regret( K) max Σ K Ek=1 E. (s) vT)) FER
(3.2)
Model-based reinforcement learning with general function approximation. We consider the model-based RL with general function approximation. In particular, assume the function class of the transition kernel as P P and define the L, Eluder dimension of a function class
Definition 3.1 (Cp Eluder dimension, Russo & Roy 2013). DE, X,,)) c) is the Eluder dimension for sequence æ1,-.. Fr C X' enjoys
X' with function class & when the longest epsilonn the length less than (G, x). In other words, there exists function g E G such that for all t << (E, X, e) we have Σ (a(x))) < e" and [g(x:)) 1 2 c². Through out this paper, we work with the & Eluder dimension defined by dg DE, (G.S x A, defined over the Hellinger distance class 9 {(s, a) (P* (s, || P(s, a)): P E P) and 1)
For the analysis under general function approximation, we leverage the following definitions on covering and bracketing numbers
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 Model-based Adversarial Imitation Learning (MB -IB-
Input: Number iterations K, confidence radius 3, expert dataset DE. [DEl = N
Input: Policy class II, reward class R. model class P
1: Let π (-1s) = Unif.(AA, r (s. a) = 0 for all (s, a) E A, Du
2: for k = ........ K do
k- k- 'n using policy mk-1, update D3 DR- U (Tk-1} "h 5: Obtain reward function R by running a no-regret algorithm to solve the online optimization problem with observed loss {C((r))=0 up to an optimization error tolerance Cop
3: Collect 7k-1 (
4: Compose the value difference loss (r-1 Σ. r(s 0g 1 N En, Tsk
6:
Construct a version space pk:
Pk PE PEP::: log P(s'ls,a) 2 maxper E(ea,)) log P(s'|s, (s((s a) |8}
7: Set (x* Pk) t- argmax Pept V 1:P
8: end for Output: (nif((TT))) (R=1) 1
Definition 3.2 (e-covering number). For a function class S (X' + R), the e-covering number of F, denoted (F). is the smallest integer E N such that there exists subset J C F of size |F' = n such that for all f F. there exists f' E F' with supzex If(x)) - f'(x) S.
Definition 3.3 (Bracketing number, .van de Geer 2000). Let be function class X' TR. Given two functions I,u such that Mxx) S u(xx for all x E X, the bracket [] is defined as the set of functions g E G satisfying ((x) S g(x) I uxx) for all E X. We call [L, ul an e-bracket ||u - ||] I e. The e-bracketing number of G with respect to a norm || denoted by (f G, II), the minimum number of c-brackets needed cover g.
We build our analysis based on the bounded total reward assumption.
Assumption 3.4 (Bounded cumulative rewards). For any reward function r E R, the cumulative reward collected by any possible trajectory ((n,,a))) bounded by S En r(su. an) S
Up rescaling factor H. Assumption 3.4 generalizes the standard reward scale assumption where rh E [0, 1] for all h E [H]. This assumption also ensures that the value function VT. (8) and action-value function PT, (s, a) belong to the interval [0, 1].
We also make the realizability assumption for the reward, policy and the transition kernel. Assumption 3.5. The ground-truth reward function, transition dynamics, and optimal policy are realizable, i.e., r* R, P* P, n* E II.
These assumptions are common in the literature on model-based reinforcement learning and imita- tion learning, they ensure that the learning problem is well-specified.
4 MODEL-BASED ADVERSARIAL IMITATION LEARNING
In this section, we introduce our proposed algorithm Model Based Adversarial Imitation Learning (MB-AIL). The full procedure is presented in Algorithm I The algorithm takes as input an expert dataset DE consisting of multiple expert trajectories; three function classes II, R, and P for policy, reward, and transition model function approximations, respectively; and various hyperparameters, including the confidence radius for the version space, and the number of iterations. The algorithm begins by initializing the policy with uniform distribution and setting the reward function to zero. In each iteration, first collects a new trajectory Tk-1 using the current policy π described in Line Based be decomposed into two procedure in learning the reward r E R and learning the model P E P separately, which is deseribed as follows.
hypothesis that the learning the expert policy E within the policy class II can
Procedure A. Adversarial Reward Learning for r E R. According to the adversarial online imitation learning objective in Eq. 3.1 the reward function must be optimized each iteration
5

---

Published as a conference paper at ICLR 2026
to maximize the value gap between the expert policy x² and the behavioral policies n' generated in previous iterations < k. Therefore, the reward optimization aims to maximize the adversarial objective max, Pt, - VT.P., ,r by minimizing the following empirical loss ( with respect to the reward function E
fx-1(r)= = Σ rs 1 hh 1-11
N
3/ Σ aR) = VI. Vr 15
(4.1)
where we denote (.) as the empirical estimation of V(.). Since the series of loss function { k-1 collected by adversarial policy {= that is trying to maximize VT.P. (81), regrett online optimization must be called in Line to output the reward rk- Over the rounds, the optimal
solution for this reward learning is denoted by maxrER Σx (VT.)..(())) V.. (s(1) and thus the optimization error is denoted by
cggt K max Σ (VT..r (81) (o))))(1)))
(8)))
(4.2)
We note that Follow-the tethe--Rgularzed-Leeee practical example of the no-regret online optimization algorithm and can obtain the optimization error 6pp O(VK) optimization error in this adversarial reward learning procedure. Such an optimization- beiimization-based reward learning method is also leveraged in Xu et al. (2024) where they assume the optimization based on the individual reward function each step Ru. Ours by contrast assumes the reward function is hime-hooogeneous and therefore the optimization is based on all reward function r E R. Despite this difference, we note that the major difference of our approach is that MB AIL is model-based approach and is provably more efficient to capture the online interactions with carefully designed learning and analysis, as we will describe in detail below.
(Shalev-Shward & Singer 2007 can be used as
Procedure B. Model and Policy Learning for P P. In parallel with learning the reward, in Line we rely on an simple maximum likelihood estimator to learn the transition kernel information P within version space controlled by parameter 3 described by
pk = - E(e log P(s'Is,a) Z max E(Sa. log P(s'|s, ()) -33
where the candidate transition kernel is of at most 3 smaller than the 'optimal' version P in terms of their log likelihood. In practice, this procedure described in Line o can be mimic by training series of world models by maximizing their likelihood.
Finally, with the obtained reward r E R and the model Pk C P, the updated policy is obtained by the policy that obtains the maximum cumulative reward with the optimistic estimation over P 5k as described in Line7Jin [ We note that similar optimistic- pplinistic-based MLE approach has been widely applied in the theoretical literature on model-based [ et al. (2023); /hhn et al. (2022): Wang et al. (2024) but with fixed reward function r. However, in MB-AII AIBAII,, I., the ground truth reward unknown so that the policy optimization should be based on the current reward Ti: estimated
in Line 5 We also note that this procedure can be efficiently implemented by ensembling the estimation of several models in practice Ye et al. (203)) Zhang et al. (2024a); Janner et al. (2019)
5 THEORETICAL ANALYSIS
We present detailed theoretical analysis of Algorithm 1 In particular, we establish a horizon- free, second order bound under general function approximation for both expert demonstrations and the online interactions. We further present information-ther thhooretic minimax lower bound to statistically understand the boundary of the benefit for online interactions and the expert dataset, Based on the intuition that the policy class II can be decomposed into the reward R and the model P.
5.1 UPPER BOUNDS FOR MODEL-BASER ADVERSARIAL IMITATION LEARNING (MB-AIL)
In this subsection, we establish an upper bound on the average regret of our proposed algorithm. We first start with the regret analysis for the online adversarial imitation learning.
Theorem 5.1. For any å E (0, let if FTRL (Shalev-Shwartz & Singer 2007 is employed the no-regret algorithm in Line
7og (KNp//), Under Assumptions 3.4 and 3.5

---

Published as a conference paper at ICLR 2026
Algorithm
I
the averaged adversarial imitation learning regret defined in (32 for MB-AIL satisfies:
Regret( K)
Vi. (x)
K
K
PER
K No. +
K VaRg log
Nv.
dæ log Aiz + log Nz
VaRadg log
+
K
K
K
k=1
NR log No. 1 VK
VaRg log
where (.)) hides the polynominal logarithmic factors on K, H. dg is the cluder dimen- sion defined based on Hellinger distance as described in Definition 3.1 We denote VaR, max,eR VaR.
the maximum variance of the cumulative return for policy n², VaR maxrER VaR, is the short hand for the maximum variance collected by the expert policy T. Np is the bracketing number for the model class defined in Definition 3.3 while NR the covering number for the reward class in Definition 3.2
The regret result in Theorem 5.1 can be immediate translated to the sample complexity bound for mixed policy as stated in the following corollary.
Corollary 5.2. Let o² == m&X YER, Varā. where II is the policy class induced by the plan- For the mixed policy π = Unif (**) k==) output by Algorithm] ] for any E [0, 1], with probability at least 1 - 6, Al-
ning oracle (T,, P) = arg max+, Vr,,,,(6 (s1) used Line 7 of Algorithm [
gorithm returns c-optimal imitator, i.e., maxtER (s1) - VT.p.,,
with expert
and interaction sample complexity as:
1 dgo log No 8 + log No. de log
No o +log No
No VaR log No log
22
Remark 5.3 (Expert Sample Complexity). Corollary 5.2 reveals an Ô [VaRglog(App
(Nx) -) sample complexity for expert demonstrations when VaR U. When the reward is dg- as assumed
in Viano et al. 2024). this matches the Viano et al. 2024;;
:; Xu et al. (024) as Öd 22 with the second-order information VaRg < 1. We also improve the Hd factor compared with Liu et al. 2222 results for model-basee AIL on linear mixture MDPs and makes this horizon-free.
Remark 5.4 (Online Interaction Complexity). When > O, Corollary 5.2 suggests that MB-AIL enjoys an Ö (o² (dg loglogy + log(NR)) e ?) interaction complexity. When reducing linear mixture MDPs with Nr O(d) and NR Od). this O(d-oe--2) sample complexity im- proves Liu et al. 2021 by an Hdd factor and obtains the horizon-free result with second order information. We note that the additional 1/e2 sample complexity comes from the adaption of the no-regret algorithm (FTRL) with Epp Ö(1/VK) and dominated when No or NR is large.
Besides directly compare our work with existing results, the second-order analysis reveals some interesting results regarding the deterministic expert deterministic model.
Remark 5.5 (Deterministic Model and Policy). When both the model class P and the policy class II only contain deterministic transition kernel and deterministic policies, it's easy to ver- ify that o² = and VaRg = 0. In such a case, the online interaction sample complexity is significantly reduced to O + (dy log + N1))--1). If we relax the AIL gap to be maxreR (T.) - Vi. ((i)) < e + E to accommodate the optimization error E an !-rrder sample complexity is built, which suggests that a deterministic model will benefit online interaction. Similarly, an O log Nx sample complexity is obtained.
Remark 5.5 indicates an -1 sample complexity in deterministic systems (disregarding the <opt term) and an rate in stochastic systems, consistent with Foster et al. (2024). Crucially, however, the meaning of "stochasticity" and the mechanism behind the = dependence differ fundamentally between behavioral cloning and adversarial imitation learning would like elaborate more.

---

Published as a conference paper at ICLR 2026
Remark 5.6 (Behavioral Cloning (BC) v.s. Adversarial Imitation Learning (AIL)). One fundamen- tal difference between (offline) behavioral cloning (BC) and (online) adversarial imitation learning (AIL) is the target of estimation. BC fits the expert policy x E II directly from demonstrations, whereas MB-AII searches over rewards r E R and learns the transition kernel P E ? before plan- ning. Consequently, when additional structure constrains P or R. AlL's effective hypothesis space can be much smaller than II, making AlL statistically easier than BC. This distinction is reflected in the rates: for stochastic policies, our result depends a sample complexity of Ö(a² 1og, )
whereas Foster et al. (2024) obtain Ö(o² log ) Moreover, Moulin et al. (222aa) provide an up- per bound that depends only the function class for Q-functions
the offiine imitation setting, obtaining (c((1 -) - for linear Q-functions and O (logNo (1 - -7)---- for general func- tion approximation under discounted MDPs. However, in general Nq NR. since multiple distinct Q-functions can correspond to the same reward function. Therefore, in particular, when Wr is small, our analysis indicates that AIL needs fewer expert demonstrations with the help of online interaction.
Remark 5.7 (Stochasticity in BC vs. AIL.. As summarized in Table 2 BC and AIL react differently to stochas- expii-especially when the expert policy x is deterministic while the dynamics are stochastic (high- lighted in red in Table 2) In this regime, Foster et al. (2024) obtain a sample complexity of Alog [T] for BC with deterministic experts, whereas MB-AIL yields (log.NRe~3) We interpret this as fundamental gap between BC and AlL: BC only needs to match actions and is largely insensitive to transition stochasticity, whereas AIL must match the ex- pert's occupancy measure, whose estimation necessarily reflects the stochasticity of P. Additionally,
Table 2: Expert sample complexity under different kinds of stochasticity between BC (Foster el al. 2024) and MB AII (ours).
Policy n
Stochastic.
Deterministic.
Model
Stochastic.
BC: a² log 1
BC: log [[|c-
AlL: log Nge=
AlL: log
Deterministic.
BC: o² log [[]c
BC: log [I]e
AlL: NRc
AIL: log.Rcc
Rajaraman al. (2000) report an (11/)) expert-sample complexity in tabular MDPs as a tabular- specific artifact noted by Foster et al. (2024). Outside this specific tabular result, we found that the (deterministic) vs. e (stochastic) rates remain consistent across imitation leattinn analyses.
Taken together, Remark [5.6] and Remark 5.7 described clear theoretical separation between BC and AIL in terms of expert sample complexity: AIL is preferable when the reward class R is highly structured (Remark (.6): in the extreme case with [R]] = 1, AlL reduces to standard RL and no expert data are needed. Conversely, BC is preferable when the expert policy class IT is simple and deterministic while the transition dynamics are highly stochastic and complex (Remark) 5.7
5.2
MINIMAX LOWER BOUNDS FOR IMITATION LEARNING
In this subsection we present minimax lower bound on the sample complexity of the offline demon- strations N = [Drz and the online interactions K. We start from constructing hard instance as described below.
Hard instance. As illustrated in Figure I we consider an (H + 1)-step time-homee home-homogcneous MDP. For a reward dimen- sion dn. kernel dimension dr, and horizon H, The state has two components S = Š {0, 1}. where the first component = =B U {oo} and the second component is 0 if and only if the state is the initial state, write the state as s, 0 for these
1/2 + cp app
s
3.0
1/2 (M, ar)
So.
two components. The action space AA = d x (11, +1} con- centration concatenation of two action where ap E BBdr the first part and an E {+1} as the second part. The agent always starts from state s, 0 where ~ Unif. (IB)) For the first step h 1, the agent will either proceed to the same but absorbing state s, 1 or fall into the special state su For the rest of the steps h E (1, H]. the agent will stay in their state ; 1 The reward is only accessed these regular absorbing state [s, The MDP is parameterized with parameter 0 E Md C BdR for reward and μ E Bdr for model, where Mdd is special almost orthogonal class of binary vectors defined in Appendix D.2 For detailed
Figure 1: Structure of the Hard
Instance. The reward can only be observed in Green states. The ab- sorbing fail state is shown in gray.
8

---

Published as a conference paper at ICLR 2026
Table 3: MuJoCo Results on Cumulative Rewards. We conduct experiments on the Hopper, Walker2d, and Humanoid environments, eeaaauutnnneealluatnnn cumulative rewards and comparing our approach with BC, GAIL, and OPT-AIL baselines. Our method achieves comparable performance on Hopper and Walker2d, while significantly outperforming all baselines on Humanoid. Results are averaged over three random seeds.
Environment Expert
BC
GAIL
OPT-AIL
MB-AIL (Ours)
Hopper
3609.2
2856.7+42.5
3211.9438.3
3438.6 + 21.1
3451.3 + 15,5
Walker2d
4636.5
2697.3 = 97.2
3776.5 + 64.3
4238.4 39.2
4169.7 48.3
Humanoid
5884.6
342.5 = 23.8
1614.4 = 118.1
2014.4 + 342.2
5816.4 + 15.2
formulation of the transition dynamies P(s' [s, a: (t), the reward function r(s, a; 0), and the policy class (a)s;; ; fi, 0), we refer the reader to Appendix D.1
Based this construction, we ready present the lower bound for sample complexity.
Theorem 5.8. Given the hard instance described in Figure for all 0 < < min 4dp 4dg } dR>48,, 48, and o= rRRR Var,,r < 1, any (online offline) imitation learning algorithm that guarantees producing c-suboptimal policy with probability at least under the averaged form of the regret defined in Eq. 3.2 by using N expert demonstrations and K iterations requires at least:
N Q (max {o² /e2/e2, 1/22,,0 o² log" (R(})
K- = Ω (max/ (o² log (PI
log? [R] log* expp NN/lo IP1), (PI),1)) 13),
with variance of the expert policy o = MAN,ER Varzg.r bounded by J0²²² S o S o². This suggests that N = -0 and K = [2 log" (Pl (Plexp(-N)e
The detailed proof of Theorem 5.8 is deferred into Appendix D
Remark 5.9. In Theorem 5.8 K" indexes the policies, which indicates that the number of online interactions K 1 since πK learned from the offline dataset together with the previous
rounds of interaction. Specially, the purely offline setting corresponds to K 1. More discussions about this interaction complexity in the offline setting is deferred to Appendix B For stochastic policies,(Foster
Foster et al. (2024) suggests M( expert demonstration. In contrast, Theorem 5.8 presents fine- fine-grained analysis on the sample complexity for both expert demonstration and online interaction: When online interaction is allowed,
results matches Foster et al.] (2024) with an lower bound
(2) expert sample complicity with an additional 22olg"" 'Plo² exp(-N)e for online interaction. This result implies that in the practical AlL case where the expert demonstra- tion is limited and is much smaller than the model class with N < log² [P], the online interaction conducted in AIL can effectively assist the policy learning K 2(log |plo2 -2) with an minimax optimal sample complexity.
Remark 5.10. Together with Theorem 5.1 Theorem 5.8 shows that MB-AIL is minimax-optima
w.r.t. online interaction K with only logarithmic gap when given small number of expert demon- strations N, and is within a log [R] factor its dependence on N. We hypothesize that removing this
log [R] gap may be intrinsically difficult as similar gaps (Ö II/e) vs. (2(=²2 persist in Foster al. 2024). Specifically, Xu et al. (2022) establish an (2SS -2] lower bound for their TV-AIL algorithm in tabular MDPs, yet a minimax lower bound that explicitly exhibits a log [II] or log [R]
dependence remains open even in the tabular setting.
6 EMPIRICAL RESULTS
We implement MB-AIL with deep neural networks and evaluate its performance on three stan- dard MuJoCo benchmarks (Brockman et al. 2016). covering three environments (Hopper, Walker2d and Humanoid). We compare its performance against existing offline and online imitation learning baselines in terms of episode rewards and sample efficiency, showing that our practical algorithm matches or even surpasses existing approaches while highlighting the superior sample efficiency of model-basee approach. Details of the practical implementation are provided in Appendix E the MuJoCo experimental results are presented in Section 6. and the ablation studies on network sizes discussed in Section 6.2

---

Published as a conference paper at ICLR 2026
6.1 MUJoCo EXPERIMENTS
We evaluate the proposed practical algorithm on several MuJoCo environments (Brockman et al. 2016), including Hopper, Walker2d, and Hu- manoid. For each task, we use 64 expert trajecto- ries as demonstrations for training. We compare our method against Behavioral Cloning (BC), GAIL (Ho & Ermon| 2016), and OPT-AIL (Xu et al. 2024). The cumulative reward results are reported in Table 3 Our method achieves per- formance comparable to the best baselines on Hopper and Walker2d, while significantly outper- forming all compared methods on the more challenging Humanoid task, demonstrating its stronger capability in complex, high-dimensional control settings. All results are averaged over three random seeds. In addition, we compare the interaction complexity of our method with the model-free base- line OPT-AIL (Xu et al. (2024) in Table 4 which highlights the improved interaction efficiency of our model-based approach.
Table 4: MuJoCo Results on Interaction Complexity. We compare our interaction com- plexity with OPT-AIL and show that our method requires significantly fewer steps to reach opti- mal performance.
Environments
OPT-AIL
MB-AIL (Ours)
Hopper Walker2d Humanoid
~210
~60K
~320K
~220
~220K
~90K
6.2 ABLATION STUDIES ON THE NETWORK SIZES
In this section, we conduct an ablation study on network capacity empirically examine the claim in Remark 5.6 that log [R] < log [II]. Specifically, we vary both the depth and the hidden-layer dimensions of the reward and pol- icy networks while keeping all other compo- nents fixed. In particular, we fix the policy net- work to a 2-layer MLP with 1024 hidden units and vary the reward network in Table 5 (a) and fix the reward network to a 2-layer MLP with 256 hidden units and vary the policy network in (b). The results indicate that the reward function can be learned effectively with rela- tively small network, whereas reducing the ca- pacity of the policy network leads to a substan- tial degradation in performance. These findings provide empirical support for the claim that, in typical settings, the complexity of the reward class is significantly smaller than that of the policy class. The detailed ablation results are summarized in Table
Table 5: Network architecture ablation. in changing the network architecture for the policy network and reward network.
(a). Changing the reward network
Reward Network
Episode Rewards
2-layer, 256 hidden units
4169.7 + 48.3 4109.5 + 39.6
2-layer, 128 hidden units
I-layer. 256 hidden units
4125.1 = 28.2
(b). Changing the policy network
Policy Network
Episode Rewards
2-layer, 1024 hidden units 2-1ayer, 256 hidden units
4169.7 + 48.3
704.9 + 112.5
6.3 ADDITIONAL EXPERIMENTS
In addition to the experiments on MuJoCo, we conduct empirical analyses in GridWorld environ- ments to examine the effects of different reward classes R, environment stochasticity, impact of the model misspecification, and the benefits of online exploration. The corresponding results are presented in Appendix [F] showing that smaller reward class can lead to improved performance for online imitation learning algorithm, thereby supporting our theoretical analysis in Theorem 5.1
7 CONCLUSION
We introduced MB-AIL, a model-based adversarial imitation learning algorithm, and established sharp statistical guarantees. We proved horizon-free, free, second-order upper bounds under general func- tion approximation and complementary information-theored is minimax-optimal in its use of online interaction when the expert demo is limited. We also show that MB MB-AIL is optimal within log |R] factor in its dependence periments have been conducted to validate our theoretical claims and demonstrate that the MB-AIL matches or exceeds the sample efficiency of strong baselines across diverse environments. Our anal- ysis clarifies how online interaction and system stochasticity impact the sample complexity of AIL and delineates regimes where AlL can outperform behavioral cloning.
lower bounds, showing that MB-AII L
expert demonstrations. Ex-
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of our results. The experiment de- tails are documented in Appendix G 6.1 and F Our source code provided to facilitate faithful reproduction of our experiments in the supplementary materials.
ETHICS STATEMENT
We have carefully reviewed the Code of Ethics and find that our work does not raise any significant ethical concerns. Our research does not involve human subjects, sensitive data, or potentially harm- ful applications. We believe our methodology and contributions align with principles of faimess, transparency, and research integrity.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their helpful comments. This research was supported by WZ's startup funding provided by the School of Data Science and Society UNC Chapel Hill.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the Twenty-Firs, International Conference on Machine Learning. ICML '04, pp. 1, New York. NY, USA, 2004. Association for Computing Machinery. ISBN 1581 138385. doi:
01114//1015330.01555300
Alekh Agarwal, Yujia Jin, and Tong Zhang. Vo q I: Towards optimal regret in model-free rl with nonlinear function approximation. In The Thirty Sixth Annual Conference on Learning Theory, pp. 987-1063. PMLR, 2023.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Kazuoki Azuma. Weighted sums certain dependent random variables. Tohoku Mathematical
Journal, Second Series, 19(3):357-367, 1967.
Nir Baram, Oron Anschel, and Shie Mannor. Model-based adversarial imitation learning. arXiv preprint arXiv:1612.02119, 2016.
N. Bernstein. On a modification of chebyshev's inequality and of the error formula of laplace. Annales de la Faculté des Sciences de Toulouse, 3:1-12, 1924.
Jean Bretagnolle and Carmen Huber. Estimation des densités: risque minimax. Zeitschrift für Wachsscheinicchkeitstheeee und Verwandte Gebiete, 47 1979.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.011400. 2016.
Minshuo Chen, Yizhou Wang, Tianyi Liu, Zhuoran Yang. Xingguo Li, Zhaoran Wang, and Tuo Zhao. On computation and generalization of generative adversarial imitation learning, In Inter- national Conference on Learning Representations, 2020.
Yilei Chen, Vittorio Giammarino, James Queeney, and Ioannis Ch Paschalidis. Provably ef- ficient off-policy adversarial imitation learning with convergence guarantees. arXiv preprint arXiv:2405. 16668, 2024.
Zixiang Chen, Chris Junchi Li, Angela Yuan, Quanquan Gu, and Michael Jordan. general framework for sample-efficient function approximation in reinforcement learning, arXiv preprint arXiv:2209.15634 2022.
Jie Cheng, Yingbing Chen, and Qifeng Chen. Pluto: Pushing the limit of imitation learning-bast planning for autonomous driving. arXiv preprint 2xXv:2444.143277 2024.
Cheng Chi, Zhenjia Xu, Siyuan Feng. Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024.

---

Published as a conference paper at ICLR 2026
Gustavo Claudio Karl Couto and Eric Aislan Antonelo. Generative adversarial imitation learning for end-to-end autonomous driving on urban environments. In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1-7. IEEE, 2021.
Robert M Fano. Transmission of information: A statistical theory of communications, American Journal of Physics, 29(11):793-73 1961.
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph Gonzalez, and Sergey Levine.
Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint
arXiv:1803.00101, 2018.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conference on robot learning, pp. 158-168. PMLR, 2022.
Dylan J Foster, Sham M Kakade, Jian Qian, and Alexander Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv22112.13442 2021.
Dylan J Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon imitation learning. Advances in Neural Information Processing Systems, 37:120602- 120666, 2024.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. Pmir, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Michael Janner, Justin Fu. Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in neural information processing systems, 32, 2019.
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire. Con- textual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning. pp. 1704-1713. PMLR, 2017.
Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of problems, and sample-efficient algorithms. Advances in neural information processing systems, 34:13406-13418, 2021.
Victor Kolev, Rafael Rafailov, Kyle Hatch, Jiajun Wu, and Chelsea Finn. Efficient imitation learning with conservative world models. In 6th Annual Learning for Dynamics & Control Conference, pp. 1777-1790. PMLR, 2024.
Shangzhe Li. Zhiao Huang, and Hao Su. Reward-frec world models for online imitation learning. In International Conference on Machine Learning (ICML), 2025.
Qinghua Liu, Praneeth Netrapalli, Csaba Szepesvari, and Chi Jin. Optimistic mle: A generic model- based algorithm for partially observable sequential decision making. Proceedings of the 55th Annual ACM Symposium on Theory of Computing, 2023.
Zhihan Liu, Yufeng Zhang, Zuyue Fu, Zhuoran Yang, and Zhaoran Wang, Provably efficient gener- ative adversarial imitation learning for online and offline setting with linear function approxima- tion. arXiv preprini arXiv:2108 08765, 2021.
Antoine Moulin, Gergely Neu, and Luca Viano. Inverse gelarnning done right: Offline imitation learning in q-realizable mdps. arXiv preprint arXiv:2505. /9946, 2025a.
Antoine Moulin, Gergely Neu, and Luca Viano. Optimistically optimistic exploration for provably efficient infinite-horizee reinforcement and imitation learning. arXiv preprint arXiv:2502.13900 2025b.
12

---

Published as a conference paper at ICLR 2026
Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ranchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:2914-2924, 2020.
Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Proceedings of the 27th International Conference on Neural Information Process- ing Systems - Volume 2, NIPS' 13, pp. 2256-2264, Red Hook, NY, USA, 2013. Curran Associates Inc.
Shai Shalev-Shward and Yoram Singer. Convex repeated games and fenchel duality, In Advances in Neural Information Processing Systems, volume 20, 2007.
Lior Shani, Tom Zahavy, and Shie Mannor. Online apprenticeship learning. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 8240-8248, 2022.
Lucy Xiaoyang Shi, Archit Sharma, Tony 7. Zhao, and Chelsea Finn. Waypoint-based imitation learning for robotic manipulation. In Conference on Robot Learning, pp. 2195-2209. PMLR, 2023.
Yuda Song and Wen Sun. Pe-mip: Model-based reinforcement learning with policy cover guided exploration. In International Conference on Machine Learning, pp. 9801-9811. PMLR, 2021.
Wen Sun, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, and John Langford. Model-based rl in contextual decision processes: Pac bounds and exponential improvements over model-free approaches. Conference on learning theory, pp. 2993-2933. PMLR, 2019a.
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient imitation learning from observation alone. In International conference on machine learning. pp. 6036-6045 113666045.PMLR, 2019b.
Umar Syed and Robert E Schapire. A game-theoretic approach to apprenticeship learning. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007.
Mohammad Sadegh Talebi and Odalric-Ambrym Maillard. Variance- aware regret bounds for undis- counted reinforcement learning in mdps. In Firdaus Janoos, Mehryar Mohri, and Karthik Sridha- ran (eds.), Proceedings of Algorithmic Learning Theory, volume 83 of Proceedings of Machine Learning Research, pp. 770-805. PMLR, 07-09 Apr 2018
Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference Learning Representations, 2021.
Sara van de Geer. Empirical Processes in M-estimation, volume 6 of Cambridge Series in Statis- tical and Probabilistic Mathematics. Cambridge University Press, 2000.
Luca Viano, Stratis Skoulakis, and Volkan Cevher. Imitation learning in discounted linear mdps without exploration assumptions. In Forty-first International Conference on Machine Learning, 2024.
Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, and Wen Sun. The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in neural information processing systems, 05:2275-23122 2023.
Zhiyong Wang, Dongruo Zhou, John Lui, and Wen Sun. Model-based rl as minimalist approach to horizon-free and second-order bounds. arXiv preprint arXiv:2408. 08994, 2024.
Tian Xu, Ziniu Li, Yang Yu, and Zhimin Luo. Understanding adversarial imitation learning in small sample regime: stage-coupled analysis. ArXiv, wwwp/220..0099999 2022.
Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. Provably efficient adversarial imitation learning with unknown transitions. In Uncertainty in Artificial Intelligence, pp. 2367-2378. PMLR, 2023.
Tian Xu, Zhilong Zhang, Ruishuo Chen, Yihao Sun, and Yang Yu. Provably and practically efficient adversarial imitation learning with general function approximation. In Proceedings of the 38th Conference on Newral Information Processing Systems (NeurtPS), 2024.
13

---

Published as a conference paper at ICLR 2026
Chenlu Ye, Rui Yang, Quanquan Gu, and Tong Zhang. Corruption-robuss offline reinforcement learning with general function approximation. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp- pp.36208-366 36208-36221. Curran Associates, Inc., 2023.
Zhao-Heng Yin, Weirui Ye, Qifeng Chen, and Yang Gao. Planning for sample efficient imitation learning. Advances Neural Information Processing Systems, 35:2577-2589, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14199-14144 2020.
Andrea Zanette and Emma Brunskill. Tighter problem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds, In International Conference Machine Learning, pp. 7304-7312. PMLR, 2019.
Wenhao Zhan, Masatoshi Uchara, Wen Sun, and Jason D Lee. Pac reinforcement learning for predictive state representations. arXiv preprint arXiv:2207.05735 2022.
Junkai Zhang, Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Uncertainty-awant reward- -ree exploration with general function approximation. In International Conference Machine Learn- ing. pp. 60414-60445. PMLR, 2024a.
Renhao Zhang, Haotian Fu, Yilin Miao, and George Konidaris. Model-based reinforcement learning for parameterized action spaces. In International Conference Machine Learning, pp. 58935- 58954. PMLR, 2024b.
Zihan Zhang. Yuxin Chen, Jason D Lee, and Simon S Du. Settling the sample complexity of online reinforcement learning. In The Thirty Seventh Annual Conference on Learning Theory, pp. 5213- 5219. PMLR, 2024c.
Heyang Zhao, Jiafan He, Dongruo Zhou, Tong Zhang, and Quanquan Gu. Variance- deeeedet regret bounds for linear bandits and reinforcement learning: Adaptivity and computational efficiency. In The Thirty Sixth Annual Conference Learning Theory, pp. 4977-5020. PMLR, 2023.
Han Zhong, Wei Xiong, Sirui Zheng, Liwei Wang, Zhaoran Wang, Zhuoran Yang, and Tong Zhang. Gec: A unified framework for interactive decision making in mdp, pomdp, and beyond. arXiv preprint arXiv:2211.011 01.9962, 2022.
Runlong Zhou, Zhang Zihan, and Simon Shaolei Du. Sharp variance-dependent bounds in reinforce- ment learning: Best of both worlds in stochastic and deterministic environments. In International Conference on Machine Learning, pp. 42878 42914. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
CONTENTS OF APPENDICES
Additional Related Works
16
Additional Discussions for Theorem 5.8
16
Proof of the Upper Bounds
17
Upper Bound with Finite Function Class
17
Proof of Theorem C.I
18
Upper Bound for the Reward Error
19
C.3.
Technical Lemmas
19
C,3.2
Proof of Lemma C.2
20
C,4
Upper Bound for the Policy Error
22
C.4.
Technical Lemmas
22
1422
Proof of Lemma
25
C.5
Upper Bounds under General Function Approximation
26
C.5.1
Upper Bound for the Reward Error
27
C.5.2 Upper Bound for the Policy Error
30
D Proof of the Lower Bounds
30
D.I Details of the Hard Instance Structure and Proof Sketch
................................
30
D.2 Technical Lemmas
31
D.3 AlL Gap Lower Bound
33
D.4
Lower Bound for Reward Accuracy
34
D.5 Lower Bound for Model Accuracy
34
D.6
Proof of Theorem
35
Practical Implementation
37
GridWorld Experiments
37
Details of the MuJoCo Experiments
38
15

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORKS
Variance-aware Second-order Analysis in Reinforcement Learning. Variance-dependent (or second-order) bounds are instance-dependen guarantees that scale with the variance of the re- They have been studied in various settings, including tabular MDPs (Zanette & Brunskill] Zhou et al.) 2023 Zhang et al., 2024c; Talebi & Maillard 2018), linear mixture MDPs (Zhao 2023). low-rank MDPs (Wang et al. 2023), and general function approximation (Wang al.,
turn.
2019
ct al.
2024
In imitation learning, Foster et al. (2024) also established variance-dependent upper and
lower bounds for stochastic experts.
RL with General Function Approximation A large body of theoretical work has investigated re- inforcement learning and imitation learning under general function approximation. To characterize the theoretical limits and better understand practical RL/IL algorithms, researchers have proposed a variety of statistical complexity measures, including Bellman Rank (Jiang et al.] (2017). Witness
Rank (Sun et al. 2019a), Eluder Dimension (Russo & Roy 2013). Bellman Eluder Dimension (Jin
iet al. 2021). Decision Estimation Coefficient (DEC) (Foster et al. (022), Admissible Bellman Char-
acterization (Chen al. 2022). Generalized Eluder dimension (Agarwal al. 2023). and Gener-
alized Eluder Coefficient (GEC) (Zhong et al.) (022), among others. For imitation learning, related
works have considered general function approximation in the context of behavior cloning (Foster
et al. 2024 and optimization-ben based adversarial imitation learning (Xu et al. 2024
Model-based Reinforcement Learning Many works have conducted theoretical analyses of model- based reinforcement learning. Prior studies have examined model-based frameworks with rich func- tion approximation in both online RI. (Sun et al. 2019a] Foster et al. al. 2021 [Snn Sun 2021| /hhan et al. 2022; Wang et al. (024) and offline RI. (Wang et al. 2024] Uehara & Sun 2021] p. In partic- ular, Wang et al. (2024) derived a nearly horizon
second-order bound for model-
RL in
both online and offline settings, where the upper bound depends only logarithmically on H. In addi- tion to theoretical analysis, numerous studies have investigated practical algorithms for model-based
RL (Janner et al. 2019 Yu al. 2020 Feinberg al. 2018 Zhang al. 2024b).
B ADDITIONAL DISCUSSIONS FOR THEOREM 5.8
Extension to Purely Offline Behavioral Cloning Our lower bound is originally established for the online setting. When extending it to the purely offline seeting, this corresponds to the regret with K since
SubOpt(**) Regret(K 1) max E, [V (s) (s),
FR
since n² only depends on the offline demonstrations. As a results, the online interactions K by definition suffers from a trivial lower bound K = (2) for purely offline setting. The full version of the lower bound for online interaction presented in Theorem 5.8 is therefore 2(max(1, log² [P] (P)cxp(-N)<=))
As result, when the offline interaction N is significantly large (which corresponds to the behavior cloning setting, precisely N = Mllog o² log" IPI) = Dioog a log (P)) )), the lower bound becomes trivial since
e
K = 4(max (1, log" |P] eexpp- 2]) = 2(1).
We can further highlight that when the reward class is limited, Theorem 5.8 together with Theo- rem [.1 suggests that would be possible to obtain a smaller and complete the estimation on model through online interactions, which goes beyond Foster al. (2024).
Expert Policy Variance vs. Maximum Policy Variance Foster et al. (024) derive a variance- aware lower bound that depends only the expert policy's variance. In contrast, our lower bound generally depends on the maximum variance over all policies in the policy class. In the construction used for Theorem 5.8 this implies that the expert variance of must be of the same order as this maximum variance, i.c., a lower bound on the expert variance is required, as stated in the following lemma:
Lemma B.1. By the construction of the hard instance for Theorem 5.8 the variance of the expert policy oz max VaR_E, lower bounded by the MDP parameter 7:
27
16

---

Published as a conference paper at ICLR 2026
Proof. Given the definition of the expert policy variance:
H
(sh an) ) - (E(VE
2 max VaR Mr, max E
Σ
B
A=1
by Eq. D.3 and the definition of the reward in the constructed hard instance in Appendix D.I let the parameterization of the reward be and the expert policy be π (als;; ju", 0*), we can have:
H
) = ( + exedd ,1²) B i
Σ r(s, an))
1 + 2dg
(0,s) S) + 4d
h=1
E| or (s) = [[ - Ex ExedR( n²)) (ž + 1/4 (0 5) 1
By simple algebra, since we have S cp $ 1/ddp) and 0 S Ex I 1/dg. we can derive the lower bound for c.
oB max E
Σ r(sh, an)
-IE(VEE-,
FER
2 ( + Ene , ))²
2 1/"²
where the first inequality is given by:
max- -(13) (1E[ViTP-.. ViTp-
E Vi (s)))
min rER
min (2 + czcrdn(m n²)) ( zo (0,s) (0 .) ]²
0
min
(2 2dndn
dulpt
p²)) (1amtpp)))
2 di
00 O 8) ]
and
11
2
max E
1 1=1
r(sh an)
max (1 + exepdn( , T²B (+ (+ 2dR (Ō,s) + 1,7 (0,
0
( eneddn p²)) 72
which concludes the proof.
Since we show in the proof of Theorem 5.8 that Var I 72 for all x E and r E R, the expert variance of cannot be significantly smaller than x² max reR Var In particular, o must be of the same order as o². This ensures that construction is consistent with the result in Foster et al. 2024).
C PROOF OF THE UPPER BOUNDS
C.1 UPPER BOUND WITH FINITE FUNCTION CLASS
We first aim to upper bound the average regret of our proposed algorithm under finite function classes, i.e., when both P and are finite. Formally, this upper bound can be stated in the following theorem:
Theorem C.I. For any 5 E (0. 1), let = 4 log (K)P)/5) Under Assumptions 3.4 and 3.5 FTRL (Shalev-Shwart) & Singer 2007) is employed the no-regret algorithm, and R and P
finite function classes, the average adversarial imitation learning regret defined in 3.1 satisfies:
K
(1/ Rigr)tegre(KK) (K) = K
max YER
V Vi. (s) - (r)
k=
17

---

Published as a conference paper at ICLR 2026
K
PP]
1 K
Pl
log( RR//6)
VaR A de log
+
dr log
VK +
K
8
8
K
log([R] /d)
K
VaR log( R|/6) +
N VaR,, log (Rg(())/))
N
K
k=1
where (() hides the polynominal logarithmic factors on K, H, dg is the eluder dimen-
sion defined based on Hellinger distance as described in Definition 3.1 We denote VaR, max,eR VaR." the maximum variance of the cumulative return for policy x², VaRg max,eR Val, is the short hand for the maximum variance collected by the expert policy T..
C.2 PROOF OF THEOREM C.1
According to the definition of average regret in Eq. 3.2 we can decompose it into two compo- nents-reward learning and policy learning-ant and analyze them separately:
1
Regret(K)
max FER
ViPp Vi(pa))
K
K
Vi ViP.
ViP - (VT.
V Vi
max PER
K
T1: reward error
K
Vxz VisPaggh))
T2: policy
This regret decomposition standard in AlL analysis. Since the learned reward is required to exactly match the ground-truth reward function r", it is natural to define the reward error as the difference in value gaps between the expert and behavioral distributions. We provide upper bounds for the reward and policy errors under finite function classes in the following lemmas:
Lemma C.2 (Reward Error Upper Bound). For any E (0, 1), using FTRL (Shalev-Shee Shanrrtz & Singer 2007) the no-regret algorithm, given finite reward function class R, with probability 0, the average regret for reward optimization is bounded:
K
v
E VIP=
Vi (VT.P V (p+
max FER
K
Am
(
1 max VaR log(1R]/6) + PER
log( (l/()
VK
N
log( RR(/5)
max VaR log([R)/6))
K
where VaR denotes the variance of accumulative rewards over horizon H.
Lemma C.3 (Policy Error Upper Bound). For any e (0, 1), with finite model class P, let 4 log (I/)))) With probability 1-ô, the average regret the policy optimization is bounded:
K
K
1
Σ
VaR de log (H((K/))))) log( H))
K
rER
18

---

Published as a conference paper at ICLR 2026
d log (KH)P16))) log// log((H))
where dE ( A, 1 K)) is the Eluder dimension and VaR,, max VaR=* is the maximum variance for x².
In the proof steps of Lemma C.2 the reward error T1 is decomposed into an optimization error and a value estimation error. The optimization error, stemming from the no-regret algorithm in Line of Algorithm| 1 and related to cpp in Eq. 4.2 is bounded by OLLVVK) when using FTRL (Shalev- Shwartz & Singer 2007). The value estimation error, which represents the difference between the empirical value Vir and the true value function V.P is bounded using standard Bernstein- style techniques and union bounds for a finite reward class. For an infinite reward class, as detailed in Appendix C.5] we establish the bound by constructing a p-cover using the covering number from Definition 3.2 which enables the use of union bounds followed by Bernstein-style analysis. We present the analysis for the finite reward class in the proof of Lemma C.2 and the analysis for the infinite reward function class in Lemma
Regarding the policy error T2, we generally follow the proof steps in Wang et al. (0224) but with changing rx during iterations instead a fixed r*. Overall, the bound constructed by first applying the standard MLE analysis the estimation of transition model and careful analysis
training-
to-testing distribution transfer via Eluder dimension adopted in prior work (Wang et al. (024). After
this analysis, dependency on the horizon H. We provide the detailed analysis in the proof Lemma for the finite model class, and in Lemma C.15 for the infinite model class.
obtain variance-awar
upper bound for the policy error with logarithmic
Finally, by combining Lemma C.2 and Lemma C.3 we complete the proof of Theorem C.I. which the upper bound with finite function classes. For the results under general function approximation in Theorem 5.1 we provide the detailed analysis in Appendix C.5
C.3
UPPER BOUND FOR THE REWARD ERROR
C.3.1 TECHNICAL. LEMMAS
In this subsection, we present several technical lemmas that are instrumental in deriving the bound on the reward error.
Xx, Lemma C.4 (Bernstein Ihequality,BBemsteinn /.BBsttennnnn 1924). Let X1, X2,........ .2,....,,,,, be ii.d. random variables, with mean = E[X,], empirical mean μ Σi= X₁, and variance o² = Var(X,). Then, for any d E (0, 1), with probability at least 8, the following inequality holds:
20 l2g()/6)
log (2/6)
7 Al
3n
Lemma C.5 Beuua-Bernsttin Inequality, sequence with respect to filtration (Fi) Lee., increments satisfy
Azuma 1967
Let (X+) be a martingale difference almost surely, and suppose the
Xx
Fa-1l 1]
Xul < almost surely for all k;
and the conditional variances satisfy
0. = = FR
Define the total conditional variance
V, kk
2
k=1
Then, for any E (0, 1), with probability least 1
8,
2
+ b
2 8
Σ
Xx
2V, log
log
8
3
k=
19

---

Published as a conference paper at ICLR 2026
C.3.2 PROOF OF LEMMA C.2
The reward error can be decomposed into two components: the optimization error the no-regret algorithm and the estimation error of the value function. We begin by providing formal description on optimization eтror introduced in Eq. 4.2
Definition C.6 (uu et al. (2024)). Given sequence policies {n * k=1 suppose a no-regret reward optimization algorithm sequentially outputs reward functions r' r" ,..., , The reward optimiza- tion emror Cop is defined as:
Σ k=1
(Vr
(Drt 1:P-rr VTPp:))
Cope K
This eтror can be bounded by 0(1vv K) when using the Dollow-the-Regulariie----ee algorithm (Shalev-Swwarz Singer 2007) value estimates under both the learned and expert policies to facilitate bounding the value estimation error:
(FTRL)
the no-regret learner. We then define the empirical
Definition C.7 (Xu )) The empirical value of policy z under reward function R is defined as:
#1
Vi 1.P
E
( and)
where [( ) is a trajectory sampled from policy x². For the expert policy Th, given an expert dataset DE = {TI, T2........ .N.. J.J. the empirical value is defined as:
Σ
Σ
r(sx(T),
N
TEDg h=1
With Lemma
C.4
C.5
and Definition
C.6
C.7
we're now ready to demonstrate the proof for
Lemma C
Proof of Lemma
We first apply the following decomposition separate the reward error into reward optimization component and value estimation component, as described in Appendix C.2
C.2
K
va
(VTT
ViT Vigpaggn))
max FER
K
1.P:F
VIPP
K
P Dat ViP - I:Prppt (pp))
(DrE
max FER
K
T3: optimization
K
Fa Virt. + 1.P
Vak L.Pgg
D Ppprgpe
+ max FER
1.P
+
1rt.p-
ViP +
K
k=
estimation
Term T3 corresponds directly to the optimization error of the no-regret algorithm, as defined in Definition C.6
R 1;Pr,5 - Dzp 1. - (D
Dat (ep
T3
max
1:Prrp
PER
K
K
1
(Vr.p. (i.p.
max K PER
k=1
20

---

Published as a conference paper at ICLR 2026
Having bounded the optimization error, turn to bounding the estimation error:
K
T4 = max FER
Va B
1:P
Vi. :) +
V
K
1=1
Vi 1:P
Vor 1:Pr:pt
+
We aim to derive a variance-aware bound by providing an analysis using a Bernstein-style inequal- ity. Following the definition of variances in Section we first focus on the estimation emoo in- curred when approximating Vi: with its empirical counterpart inequality, we obtain that, for any fixed reward function the estimation error is bounded with probability least 1 8:
Vi Vi. By applying Bernstein's
H Σ
Vi 1:p-3r
Σ
r(s((T), Ex
Σ A=1
rss an)
TED g h=1
2 DreDg Va 12 11
2i (T))
2 log(22/8)
N
3NV
2
2 220g(2/0)
N Va log(2/0)
(C.1)
3N
Since here only consider finite reward class R, by stardard union bound:
N VaR=e,, log(2 + (2g(2
Vi Vi:p-r - 1.
Vi:P- S
3NV
VaR_ log(2(R()/8) + log(() (
N
3N
Therefore, we can bound the first two terms in the estimation error:
max
Vi.P
(Vire (VI.P-ge
V VI.P..r.)
PER
2 N
4 4log(2(R1/8)
<
2
max
vaR=",,) PER
log(2/Rl/5) +
(C.2)
3N
We then focus on the value difference with respect to n² for any E R using Azuma-Bernstein's inequality, with probability least d:
K
H rShh, an) E,
H Σ h=1
V
rsh an)
K
K
k=
h=1
2
K
2 2log(2/8)
r(sh. logg2/)(2
K
3K
h=1
2
K
2 2ogg(2/6)
VaR by log
K
3K
Following the analysis of the terms involving " we apply the union bound:
1
2
2 log 2og (
VaR p log(2)R1/6) +
K
K k=
3K
k=1
21

---

Published as a conference paper at ICLR 2026
In this case, the terms associated with T² are bounded by Bernstein-style variance- aware upper bounds, with probability at least 8:
K
V Vi + (vrr
Vint
Di P.p.p
max FER
1:Pryu
K
km
2
4 1ogg(2) 10g(211R1/0)
2
(max ga log(2/R1/8) + rER
(C.3)
K
3K
Then finally by union bound, using two high-probability inequalities: Eq. probability least 1 8:
and Eq.
C.3
with
K
max FER
ViPPo
Vipp. + (Vr.P V. 11PP
K
k=1
2
4 1log(4/R1/0)
2
max PER
VaR 7 log(4/R\/0) +
N
3N
Σ
V.P Vitp- + V 1:
Vist
max PER
K
k=1
2
K
VaR log(4/R)/5) + 1lg(44/2//0)
S 2
(max PER
K
3K
Combining two high-probabilltt inequalities, with probability 1 1 5:
max VaR, log(4[R1/6)) + 4 Hog(4/R1/3
T1 S opt + 2v
N
PER
3N
2
K
(max VaR (og(4(R1/0)) + log (|/5)
+ 2
K
FER
3K
k=1
1
log(RR) /0)
max VaR log(1R]/6) + FER
VK
N
K
(max VaR Log(( log( (R)() PE
K
when incorporating FTRL the no-regret algorithm for reward optimization, providing an O(1/vK) bound for e
C.4 UPPER BOUND FOR THE POLICY ERROR
C.4.1 TECHNICAL LEMMAS
In this subsection, we present several technical lemmas that are essential for analyzing the policy error bound. Our analysis conducted under the following function class:
{s,,a) H2 (P*(s, || P(s,a)) P P
where (-I:)) denotes the squared Hellinger distance and P the model function class. We now state series of key lemmas used in the subsequent analysis.
Lemma C.8 (New Eluder Pigeon Lemma, Wang et al. 2024). Let the event & be:
k-1
E:
vk
[K]
ΣΣ
HP (sh. (h) S n.
Then under event E, there exists set K. [K] such that:
22

---

Published as a conference paper at ICLR 2026
We have L S (3 DEIGG, A.1/(8nKH))
We have
(H P* ()) & (G, (,1/KH) 7n (H))) + 1.
kE[KJyCh=]
Lemma C.9 (Bounded Hellinger Distance, Wang et al. (2024) By standard MLE generalization bound and the realizability assumption, by running Algorithm it exists:
P E Po..
With probability 1 1- 8, Σm En (P P az,) < 22 log (|//)).
Lemma C.10 (Bounded Sum of Mean Value Differences, Wang al. (024). Given Lemma C.9 happens under high probability, we have:
V Egg Pt V A (5') BerPPr att
Va (s))))
VE[K](K A=0
(Vp- Var
(et at) ( (11_)//))AA/)))0001101)0000000
KEJATVC
(G, x logK log(K()))
Lemma C.11 (Variance Conversion Lemma, Wang et al. 2024). Let de 5x If the following events happen with high probability:
vk E [K]: Pa E and k-1 E
Σ
(P PP at)) s 22 log (6))
Vm E [0, [ogg( KH defined as:
)1| Cm 2 2n + (Tog(1/0) Cm+1 + log(1/0). G and C.m are
H
G
Σ kE(KIV h=O
(Vp.Vell hhip (ohh (8) de log(K PP/0) log (KH)
+ dg log K (P)//8) log(KH),
"
Cm = Σ Σ
(V (V (oh
kE[KJuK A=D
we have:
H-1
Σ
Σ
(
V Viri vet. (sh of
KE|KTVC hmo
#-1 1
50 Σ
(V V (oh att) dr log( log K log(k()) KF))
KE|KJKK
R=0
Lemma C.12 (Generalization Bound of MLE for Infinite Model Classes, Wang et al. 2024). Let X be the context (feature) space and y be the label space. Suppose we are given a dataset {x ((mi,;)) he)n generated from martingale process, where for each = 2......, n, the input a; is drawn from distribution ((F1i=1,,91:-1 and the output is sampled from the conditional distribution p(. x;))
Let the true data-generating distribution be denoted as f* y) = p(u Ix). and assume the model class F X y + ((R) realizable, i.e. f F. Let potentially infinite. Fix any 8 E (0, 1), and define
= 3llog ((mD)) 1lx))
5
23

---

Published as a conference paper at ICLR 2026
where ND ((n)))))),FF1 l112)) Let the version space be defined as
denotes the bracketing number defined in Definition 3.3
|EF: log (ax Ui) 2 max
log [ar Un) 7B
fer
Then, with probability least 5, the following statements hold:
The true distribution lies in the version space, i.e., f F.
Every function in the version space close to the true distribution in Hellinger distance:
BanD [H² (((x, 283, VfEF.
Lemma C.13. For any 8 (0, 1). by applying Bellman equation recursively and with Lemma C.10 with probability
Σ r (ohh
VE[RIVA
H-1
Σ
Σ
(Vp. Vat
pe (s at) log(1/6) + log(1/6)
KE h=
Σ
(Vp- Vot
, ( at)] DE, x (1/// log( H)
KE|KIVC hm0
+ (G, x A, 1,1/KH) log (K(P)/)) HH)
where (Vp- VV 1 ) (sh a is the variance of the value function with respect the true tran- sition model p*, and DE (9,S x 1.1/HH) is the Eluder dimension.
Proof of Lemma C.13
H H-1
Σ
Σ
x (sh.
1:pkypt
VE
0=
H
Σ
1 BeNP
Vot pt - yor
pa (oh+t2)
KE[K]VC
h=0
#-1
Σ
Σ
B B (M at)
Vor I (s') - E Egr A+1: (s')
KIVC h=0
2
Σ
Σ
H
pv Vit ( at) log(1/0) + 3
2 log log(1/0)
KE[KIVK h=o
Σ
E pa (s (s
V t (s') - E Es Pr Vor / (s')
RE KI A=0
#-1
Σ
pv yat
Vith ((s at) log(1/6) + log(1/5)
KE|KJVE A=0
#-1
Σ
(Vp Vrr
op ) (s at)]
(E, x ((g(11/8))))) log(/ H)
VE
h=0
24

---

Published as a conference paper at ICLR 2026
DE x A/// H))
C.4.2 PROOF OF LEMMA C.3
Our proof for bounding the policy emror generally follows the proof steps in (Wang et al. 2024 but with r instead of fixed r*. By Line 6 of Algorithm the policy emro can be bounded by the optimism guarantee, and with S [K] and Lemma C.8:
Proof of Lemma C.3
T2
1
Σ
V
1:Proght)
K
k=1
K
K
H
1 K
1
Va
K
1pprip
K
(sh an)
+
K
h/ 01)
K
11P
h=1
Σ
(VrL
Σ
pk (s att +1
> K
VaR k log(1/0)
2 llog(1/0)
K
K
1:Pk
K
3K
RE(KJI
h=0
11 Σ
log (4nKH) x A
+
1
Σ
(vat (Vipkeyn
r² (s% (1h)aat)))
K
K
kE(RiVK
h=0
kk= VaR_*.r r (1g()/0) 2 2lg((1/))
K
3K
By Lemma C.9 and probability least 1 6, set 17 22 12ggg we have:
T2 <
log (lo((K)P1/0)))))) ( A, 1/100((KP1//0))
K
Σ KE(KTTA
V+ 1:Plepp
H- Σ h=0
(sh, (t) +
Σ K
A
VaR xk log(1/6)
log(1/8) K
K
K
(C.4)
We now bound the second term on the RHS of C.4 By Lemma CI13 this term admits a variance- aware upper bound that depends on the Eluder dimension. Substituting this result, together with Lemma back into Eq. C.4 yields:
T2 S K
log (log K KP//))HH)) (E, x A1/(bb((1P//)))X))
1= VaR, , log(1/6)
#-1
Σ
Σ
(V Vot pu at) log(1/8)
K
K
RE(KTTK A=D
+
1
Σ
H-1
pV Vi
h (sh a
logg + log(1/8)
K
K
VE h=0
$
(log( K (P//))KH) (9, (1100//66x11(00((111/)0)))
K
+
Σ
Σ
V WP york
V (sh ott)
+ dr log((g(())) log(KH) (()) log(1/0)
K
KE (E] h=0
#
+ 1
Σ
pv vt
p (s ot)]
dE (o) H)
K
KE[K]VK VC K=D
log(1/0) 160)
VaR l1g61/6)
-k=1
+
K
K
25

---

Published as a conference paper at ICLR 2026
1
$
log (log (log(K D DE10.55xA1//lg(////000X))
K
4-1
1
Σ
[ W
p.p.Vii yor
p (oh att] + dæ (o0(0 log(1/0)
K
kELKTHK h=0
+
1
Vpp. po (0 a
+ dæ (1g00(((11///)))g((())) log(1/0)
K
V
KE(KTVK h=0
x V dg log( K\P1/6) (H)) + log(1/0)
Σx VaR log(1/6)
K +
K
K
a de Log(k|P) (H)
1 K
+
dg
llog(A (K (1/8)) log(KH)
K
S
Σ
max VaR=*, de log( ((PP)))) log( K KH) PER
K
1
+ ds logg (P(/)) log(K)))))))))))
where dg (E, x A,)/KH) Replacing with H/(5KH) concludes the proof.
C.5 UPPER BOUNDS UNDER GENERAL FUNCTION APPROXIMATION
In this section, we extend the our upper bound result to the general function approximation, provid- ing the proof for Theorem 5.T The proof steps generally follow the proof steps for Theorem C.I but with covering number analysis for the reward function class R and bracketing number analysis for the model function class P. The covering number is defined in Definition 3.2 and the bracketing number is defined in Definition 3.3 We present the lemmas for the upper bounds of reward and policy error under general function approximation as follows:
Lemma C.14 (Reward Error Upper Bound with Infinite Reward Class). For any 5 E (0,1), using FTRL (Shalev- Sheeettz Singer (007) as the no-regret algorithm, given an infinite reward function class R. with probability 1 ò, the average regret for reward optimization is bounded:
K
max FER
VIP. Vi - (VI.P.
Var ()
K
1=1
log(NR/0)
log(NR//)
1
max Va log(NN log(NR/ FER
VK
N
K
max VaR log(NR
N
PER
where VaR m. denotes the variance of accumulative rewards horizon H under n and reward function r. NR (RX is the covering number for reward function class R.
Lemma C.15 (Policy Error Upper Bound). For any E (0, 1), with an infinite model class P. let B = 7log (K Np) With probability 8, the average regret of the policy optimization is bounded:
K
V Vi
V V....ght)
K
26

---

Published as a conference paper at ICLR 2026
K
max VaRx*, de log (8.( KH) rER
1 + K dE dE log (HH log (K()K)))))
where dg ( A, 1 is the Eluder dimension, VaR m * the maximum variance for xx and Np = VN ((KH)S))-1, P. llx)) the bracketing number.
By combining Lemmas approximation.
C.14 and C.15
complete the proof of Theorem 5.1 Jnder general function
C.5.1 UPPER BOUND FOR THE REWARD ERROR
Proof of Lemma C.14 We first revisit the bound on the reward estimation enror under a general function approximation setting. Our goal to refine the upper bound established in Lemma C.2 by expressing it in terms of the covering number.N, (R. rather than the cardinality (R]. From the regret decomposition used in the proof of Lemma C.2] the reward optimization error remains unchanged, it depends solely on eyp and is independent of the reward function class. Therefore, the focus shifts to revisiting the estimation error:
K
T4 max Σ K Armm
Vi
( +
( p.p.age - V. P:
FER
Vizl VITP-))) + (Vi.P:.. - Datt 1:P
By applying Bernstein's 's inequality, we obtain that, for any fixed reward function r, the estimation enror is bounded with probability at least 5:
H Σ
H r an)
VIPP. - Virptor
E,n
N
DEDE 4=1
1=1
2 Σ rEDg V E n-
log///)
2 log(22/8)
3N
N VaR, log(2/8) 22gg(2/0)
(C.5)
3N
Unlike the analysis for finite function classes, we cannot directly apply union bound over the cardinality [R] since the reward class R now infinite. To address this, we construct p-cover of the reward space, denoted (R)p. Then, for all f (R)p. we can apply the union bound:
2
(R))
Va 11
$
N VaR e, (1) +
3NV
According to the definition of p-cover, for any E R, there exists F (R)p that satisfies max (F(8,a) -((8, a)l S p. Therefore, it exists:
H
Pi -
Σ
Σ
((s)))))))))(((()).. S Hp.
N
TEDE h=1
H
E VEPP
V Vi
E
Σ
Hp.
Therefore, the value estimation difference of any reward function can be upper bounded by that of its representative F within the p-cover of the reward function class:
D 1:Pr:
1/7
11.P <
P..P.
VT 1.P + 2Hp
27

---

Published as a conference paper at ICLR 2026
v 2 N
2 2oog (R))(2(()),11))
VaR sy (og())))o
+ 2HH
(C.6)
3N
For VaR is we have:
, EDNDE
f(sa(F), -EzE
Σ F(sh, an)
H
r(sh 0n) 2
< RRNDE
r(sh(T), ((8n(T),nn(T)) - Bgg
Σ
+ +p P2
h=1 VaR R + p²H²
4=1
We can further bound Eq. C.6 with VaR,
Va 1:pr. "I:P.;r
2 N
(VaR= p2 (2 + (l)
S
+ +HH
3N
(C.7)
By plugging in F and n and selecting p == 1/NH:
( Vippore Vi vispaark))
max FER
V.PF:F-
K
k=1
(2
2
max VaR, + 22 log(2((R),11/8)) +
+ 4HP
N
FER
3NV
max VaR, + N (2)(((()) 4log(2
4
2
+
(C.8)
N
FER
3N
N
We then focus the value difference with respect to n² for any R using Azuma-Bernstein's inequality, with probability at least 6:
K
H
H Σ
rsh an)
Viip-rr
(8h,
an)
E.
K
K
h=1
h=1
2
log((1/8)
r(s an))
K
3K
2
2 2log(1/6)
VaR log(1/0)
K
3K
Following the analysis for terms related to æ F. given a p-cover (R)ps for all 7 E (R)p- by union bound:
K
K
(2 (R))
1.P'
ViT V1:p.
S
K
VaR (og(2
+
3K
By the definition of p-cover, exists:
H
VT.Pe - V.Ppopl $ E,
Σ
r(s a/h) -F(s at)
$ Hpp
Vipp VI.P.sp < Eyd
r(5 a = anh
Hp.
28

---

Published as a conference paper at ICLR 2026
Therefore, the average value estimation difference of any reward function r can be upper bounded by that its representative f within the p-cover of the reward function class:
K
K
V VIP
2Hp
K
2oog (R)pl/0) (R)))))))))
VaR log (og(2(())))1)))
+ 2H
K
3K
Similar to the analysis for the terms related to _², we also have the variance bound:
+ I + p² H
Therefore, we can finally upper bound the rest of the terms in the estimation error:
K
1:P'
K k=1
2
(2
(aR + (R)( +
+ 2Hp
K
3K
Thus, the last two terms related x² is upper bounded, selecting L/KH, with probability least - 5:
Vi
(VT
Virk pippagr
1.P
FER
K
2
K
4 (l0g(2
S 2
(max VaR + (11) + PER
+ 4HH
(C.9)
K
3K
2
K
(og(2(()))/KH110)) + ())
= 2
mmxx PER
VaR +
(C.10)
K
3K
K
Then finally by union bound, using two high-probability inequalities: Eq. probability least 1 8:
C.8
and Eq.
C.9
with
max FER
ViPpn
:E + (Vi.
Va ppppgpkk
K
2
+ N2 log (R8(4(()))// + ()) /6)
2
max VaR
N
HER
3N
N
K
Vi Vark Vi + yonk Vire 1.pk))k))
1:ppype
K
km
2
K
4 (lo((4(R))/xee 1/8)
4
2
max FER
VaRg*
K² log(
3K
K
Combining two high-probability inequalities, let PK == 1/KH and PN = 1/N H. with probability
8:
2
No 12) (R) (log(4((R)p.. 1/5))
4
T4 < 2
max VaR, Eir +
N
3N
N
2
4 log( (4(())
px 1/8)
4
2
(max VaR + K2 PER
log(41(R)pp 1/8) +
K
3K
K
29

---

Published as a conference paper at ICLR 2026
log(1()) 1/0)
(oy
1
K
1 K2
S O
+
max VaR VaR + EER
log(1(R) 1/6)
K
N
K
k==1
max VaR "yr +
log( ())((())xx11))
N
FER
N²
log(N PK (R)/8)
log(Npx (R)/8)
2 O
(max FER
VaR, + +
log(NVx (R)/5)
K
N
K
K²
A=
max VaR, fir + N FER
log(WNN PN (R)/6)
N
When we incorporate FTRL as the no-regret algorithm for reward optimization, providing an O(1/VK) bound for cop and keeping only the dominating terms, we can finally upper bound the reward error as:
log(N (R)/5) +
log(N (R)/5)
K
TI +
max VaR log (R)/6)
VK
K
N
K
ER
R=1
VaR, log(NN
(R)) (R))
(C.11)
N
reR
Further upper bounding the estimation error with NR max(Nox Now (R) concludes the proof.
C.5.2 UPPER BOUND FOR THE POLICY ERROR
For the policy error, by following the approach in Wang et al. (2024), and together with the proof steps in Appendix C.4 obtain policy bound under general function approximation using bracketing number for model class covering.
Proof of Lemma C.15
applying Lemma C.12
Specifically, with
KN (1H ll=c)
== 7
5
the bound holds with probability least probability 8:
1
T2 0
(1)) VaR ( -1 log( (H)) rER
K
+ de log(
K
where dg (G, A,1/KH).
D PROOF OF THE LOWER BOUNDS
D. DETAILS OF THE HARD INSTANCE STRUCTURE AND PROOF SKETCH
In this subsection, we define the transition dynamics, the reward function of the constructed hard instance (Figure [ and the policy class used in the lower bound analysis. Formally speaking, the transition dynamics and reward function is written by:
P(s' s,
0 a) + + (p ap),
P(s' Sg. 1 ss 0 a)
(M, ar),
P(s' = s, 1
s.1,a) = PPss' = So. 1 ||s Su. 1 1.aa) = 1, Va E A Bdr+1
r(s s, a)
2 (0,s)) P(s'ls,a) (s, a) otherwise if not stated.
30

---

Published as a conference paper at ICLR 2026
It's easy verify that the reward class log R ((dx) and the model class log [P] ((dp) in this setting, while log S = ((dR) and log |A| = (ddp).
Policy class. We define a stable stochastic policy class parameterized by (ji 0). The policy outputs action E Bdr+1, sampled from Rademacher distribution biased by ju and A:
(a s: fi. 0) Rademacher ( ldp Cxdnū
+
G (s.
(D.1)
where denotes concatenation, jú Bdr and 0 MdR, For action a generated by the policy (- s: fu. 0). we can find that the first component ap is governed by fi, while the second component an is determined by both the state s and the parameter 0. For any instance defined by parameters (f²", )) the offline expert demonstrations generated by the expert policy x rta s: pt 0*).
The high-level idea behind the proof of Theorem 5.8 is summarized as follows:
Proof Sketch of Theorem 5.8 We first relates the variance o² with parameter T in the hard in- stance and the Fano's inequality. We would like to highlight that the lower bound is built two cases from the hard instances where (Ep, = (e, (AddR)))) when the policy is hard-to-learn and (x) (1/(4dp), e) when the model is hard-to-learn. Then for all algorithm that seeks uni- form performance on all hard instances, the final lower bound is obtained by taking the maximum of these two cases.
D.2 TECHNICAL LEMMAS
To construct the proof of the lower bounds, first present few technical lemmas that will be used during the proof. We first provide a lemma for the existence and property regarding the reward parameter space Md BdR;
Lemma D.I. Given 7 (0, 1). there exists a M C BdR such that for any two different tors x, x E Mi and (x, xx) S dRY, and the log-cardinality of the proposed set is bounded as log [Mda| dR?2/44
Proof of Lemma D. To begin with, we assume that Unif(BBdR), ), i.c. [x]; [-1, Thus given any x, x' ~ RB we have
dx
x((x,x) 2 dRT) -P P
Σ
2, 2 dRT
dR
-P,-Unif((L)))
24 2
dR
S (exp(_dR7//0 /2),
The last inequality follows from the Ezuma-Hoeffdiin inequality, noting that 2; ~ Unif(-1,1) is bounded random variable. Consider set df of cardinality [M[d]; there are most [M pairs (x, x'). Applying a union bound over all such vector pairs, we obtain
P(5x, x' E Md x x', (x,x') duy) |² exp(-dnn2212
thus
P(Vx,x' E MdR, x x', (x, x') day) 2 MM cex(( lapp--dpp 2
Once we have that [MdR exp(-edppp /2) < 1, there exists a set M such that for any two different vector x' E Mdd (x,x') I day..
1x-x'||1 By Lemma D.1 for any x, x' E M we have ||x- x'||1 1 2 dn [x * x'), since at least one coordi- nate differs between the two vectors, contributing at least dg to the l distance.By Lemma D.I we can construct a parameter set Mi with [Md]] = [exp(dx>> (//pd)/2//4] set indeed satisfies Lemma D. and thus obtain the following lower bound log [M[a]]
and f. The constructed
31

---

Published as a conference paper at ICLR 2026
Lemma D.2. Given y 2. for dæ 48, we have the following lower bound log |M where | [exp(dR>2/4) 1:
dR7²
log [ 2 4
-
Proof of Lemma D.2
log [ | 2 log(exp(d=>) -
-1)
dun² + log 1 1- exp ( dur
dn? -3,
where the last inequality holds by applying = 1/2 and dR > 48.
We then provide useful lemma for Mda to assist the proof of Lemma D.7 Lemma D.3. Consider y. E M given
1 we have the following lower bound:
xT y)) ス dR 1ly tyl
XEM*R
Proof of Lemma D.3 Since we can write the LHS of the inequality
max xJ (y - y') = y (y y' = lly y'll1-
XEMOR
Consider two cases: (i): y y' and (ii): + y'. For the first case, it's easy to verify that lly y'll 1ly A y" which makes LHS RHS. For the second case, since we have:
Ily y' dR
dR - 7) 1ly + y'1 = = 1(y + Hy],
where the inequality follows from y') S dRY and [y # y = 1, with = applied in the final step of the proof. Combining cases, conclude the proof.
We finally present some inequalities that are instrumental in our lower bound analysis:
Lemma D.4 (KI. divergence Rademacher distribution). Let P be two Rademacher distribu- tions with probability + and for I 1/4, the KI. divergence of P and Q can be upper bounded as:
KL(P,Q) Q) < 1662
Proof of Lemma D.4
KL(P,Q) = Ep log(P/(Q)
2 E) log
2
+e
+ (1
1
log
( -6
7
-
3 +
(1+
= 2e log
1 2e
< 16
where the last inequality leverages the property x log(1 x)/(1 x) S 4x when S
n
32

---

Published as a conference paper at ICLR 2026
Lemma D.5 (Fano's inequality, Fano 1961) Consider finite set of probability measures {Pu U on a measurable space Q. Let ti be any estimator based on samples drawn from Pu- Then, for any reference distribution Po on ? the average probability of error satisfies:
log 2 + [2 E. (L Po)
Σ Ful uEZ
uJ _
U
log [M
Lemma D.6 (Bretagnolle- BrectanityBBectagnolle Huber 1979). Let P Pnd Q be two prob- ability measures on the same measurable space, and let A be any measurable event. Then
P(A) + Q(4)) 2 1 exp (-KI. Q)),
where KL(P|I denotes the KL divergence between P and Q.
D.3 AIL GAP LOWER BOUND
In this section, provide lemma that draws the connection between the performance loss and the estimation error of the parameter μ and
Lemma D.7. Consider the model parameter (f" and the corresponding expert policy x de- scribed by z(a ss; O*),, for any II with parameter (fa, 0). the AIL risk is bounded by:
max (r (s) (s) 2 rept-da || - أاl1 + 2 G 1 * 0).
rER where the expectation taken over the uniform distribution over the state space.
Proof of Lemma D.7 To begin with, for any reward function parameter @ E M and policy ा with parameter fú, 0, the value function yr (s) can be calculated by
(s) Ea Σm= 2H T+ 2Hda (0.s) s) + 2da (0, s.sss s) [as] = f de (0, (0, s),
AAT
whee the second last equation is due to the implementation of stable policy and the last equation from the definition of the policy class (D.I). The transition probability from the initial state s, the state s, is:
P(s
= 0,0 ग ( p))
Ea } + tepl"" ap)] eRe,dR(mi, (J),
(D.2)
where the expectation is taken over the action defined the policy class. Since V.J,pe,r So, 1 always zero, applying Bellman's equation yields:
Vi. or (s) (5 ExEddR(m" (.)
(5T (1.
= = + recenda(p" 7 II
(+r TepeadR( pi) (0,))(0s))
(D.3)
2
da
Repeating the calculation presented in (D3)) for Th parameterized by n², 0", the performance gap between any policy π and the expert policy a can be written by:
V.. (s) V. (s) repe,dnli (1r .14 -
SxT (f,s) (0) 0,s)
+ EEPT (4I 1 11 (0, s) 10" 8) ( (0 (0,s)
rdgepe, (1 11- P SI (.s) (0 0.s) (e (0 0,s), (D.4)
where the last inequality uses the fact that (μ² ( S (n" fx)) dp. Therefore, taking the adver- sarial reward by maximizing this gap over all possible / Md (D.4) and taking an expectation over the state space yields
max Ey E
I,P', (s) VIT :
PER
max Eg Tdurpe, ip- 41" (2)
C.I TTOTs ss (0' -0) Te: ss T 0)
2
rdgepis 114" full1 + 1 (TEx + TE & 11 0
+
rdnent=
I| full1 + 1 7 1 0
(D.5) n
where the property E.ssT Lemma
D.1 and
D.3 the second inequality.
33

---

Published as a conference paper at ICLR 2026
D.4 LOWER BOUND FOR REWARD ACCURACY
In order to lower bound the second term in Lemma D.7 related to the reward parameter 0, we introduce the following lemma:
Lemma D.8 (Lower Bound for Reward Accuracy). Given the defined MDP structure, reward func- tion, and policy class, for dn > 48, any (offline or online) imitation learning algorithm must suffer from the reward estimation error by:
E Po + 0] 21 21-(1+
16
)
log 16N 57
dn
48
where the expectation is taken over the uniform distribution in E Mdd and Po measures the probabilities when the reward model is 0.
For the proof of Lemma D.8 we start from leveraging the Fano's inequality as Lemma D.5 We then ready for the proof.
Proof of Lemma D.8 We start from the Fano's inequality with <- O thus log [4] = log |M < dRy² /4. The reference distribution defined by Po [eń Ev Pv. Since the agent can only observe
the reward step H. denote {aw over E [N] offline demonstrations. Then for any v E U. the KL divergence (P,
L.FuuPP Po) is bounded by:
(a(x)
KL, Eg-P,
log 2 log |M
Eveu Pv(
P.(
$ log | log + ExwP,
log (x(x))
log [ log 2 (L,P,,,PP).
(D.6)
Then consider < Oo, v - 01 E M as vector that only differs in one coordinate, according
the definition of the policy class, over the action sequence {a Ji:N collected over the offline demonstration with it's length as N, for 6 I 1/(4dn) and d, 2 1, we have:
(L(P(fa@h..Nee (((a J.N 01)) = N E (L Pann 01))
LRddL((Rademccher + 6x), Rademacher($ Ex))
26 1 - 2c,
2Nd log
< 16Ndd
(D.7)
where I 1, and the last inequality is because of Lemma D.4 Plugging D.7 and D.6 into the
statement of Lemma D.55 yields:
E E
Pol + 0]
> log
|
log 2 + [77 Σ, (L un Po))
2 - log
|
(log 2 + log |M
log 2 (L Pa.))
2 1 - (1 + log -1 | log 2 16.
16
2 1 1 + dR 48
log 2 - 16N
(D.8)
where the last inequality leverages Lemma D.2 with
according to the construction of M
for dR > 48.
D.5 LOWER BOUND FOR MODEL ACCURACY
In order to lower bound the first term in Lemma D.7 related to the model parameter μa E Bde, we introduce the following lemma:
Lemma D.9 (Lower Bound for Model Accuracy). Given the defined MDP structure, reward func- tion, and policy class, any (offline online) imitation learning algorithm must suffer from the model estimation error by
dr
K
2 dr
Σ PPP.
Σ
1
exp (-16 N(16c + 12))
2
4
1=1
34

---

Published as a conference paper at ICLR 2026
The proof of the estimation enror for i follows from analysis based on the Bretagnolle Huber inequality, as detailed in Lemma D.6
Proof of Lemma D.9 We start from the Bretagnolle-Huber inequality with u μu thus [2] 2dr, The agent can infer the model le from two streams: first, the expert demonstration {a]'} con- tains the model information as ap; second, from both the expert demonstration and online ex- ploration, the agent tribution of {ap, 8, 1, [NE[N] U p s, ] Consider u < and v < l4 E Bdr as vectors that differ only one dimension. According to the chain rule of the KI. divergence, let (M( = P(s.1 s, {a²)) Jas (1) and (a( = P( s, {a²) Jn. 40), we have that KL u, Pv) N (L((P
observe { So, So,1) Therefore,
denote the P(xr) the joint dis-
Jn (.))
+ KL(Pu (f
=N (KL (T ((a)))) (f ((a))))) KL KL (n
+K (F + KL (Mo), Ph(
= (KL (T ((a)),), ).), ((a') J..) (FP, Px + K KL (PL(()))
where the last equation drops the term KL (x 12), x** (fa }x) since the online policy is not affected by the offline demonstration, in terms of the model-related part. According to the definition of the expert policy and model, by Lemma D.4 for e, I 1/(4dn). Ep I 1/(dr) and dn, dr 1, we have:
KL (T (Hap) (ap)))) Jai ((pp))) S 16d
KL P
s, {a Ins mo), Pl s, s,
{a Ins u1) S 1667.
Therefore, we can bound the KL divergence:
(L(Fu, I K. 1667. + N (16d7 + 16E7.).
(D.9)
For i [dp]. we define:
K Σ
K
Pu,i =
1lū + mil
2
With Lemma D.6lad Eq. D.9 letting E7. 1/K, we can obtain:
2 exp -16 N (16d7:22 +
16
Pu.: + Pv,
2
K
By averaging over the parameter space U. it implies:
dp
dp
-16 - 11(16N22 16) K ))
Σ
Pu.i
2
exp
4
1=1
Therefore, we can obtain:
dr
K
K
dp
(-16 (-16- (16d7, 16) )
1 1
+ m1)
>
exp
2
4
which concludes the proof.
D.6 PROOF OF THEOREM 5.8
Proof of Theorem 5.8 Let the probability of learning an incorrect reward be P(O" + 0) = -1/2,
from Eq. D.8 for dR
48, yields:
16
2
log2 ~ 2(1/(2)
16c
2
dR
48
35

---

Published as a conference paper at ICLR 2026
The suboptimality gap is lower bounded:
max Ex (ITPP:;)s) - V VI..P... (8)] 2 PCoTP(O + O),
rER
which suggest that /((7/e)) is the lower bound for N.
By Lemma D.7 Lemma D.9 and summing the suboptimality over the iteration, letting ep = 1/ VK the regret of the imitation learning algorithm can be lower bounded as:
Regret K) = max Ek= Rog (I (8) VITP-., (s)]
rER
Te TRldRl -1ol11 -f
2
k=1
K
dp
TendR 1 + p4,1
2
K
i=1i=1
K
dp
K E 1 + mil
dEN
2
dpdRTed
K
16 - N (1667122 + K
16
exp
16
Let (Ep,ex) be a problem instance parameterized by two perturbations. For é > 0 and max,eR Eg [ (.. (s) S e, for two hard instances parameterized by e: (1/(Adp). and (e, 1/(4d))) the lower bound should be able to apply both instances. For instance (e, 1/,d/))))) the lower bound for N and K is as follows:
N 2
K L(r exp(x-))))))
For instance (1/(4dp), <) the lower bound for N and K is follows:
N 2 (7%/))
K z 2(7 d exp(-N/df.))
Combining two lower bounds above, the lower bound must hold for both hard instances:
N 2 Ω (max
K ž Ω (max ( exppxxp(NN)/6 200///2,2200 d exp( exp(-N/d3)))
Consider the variance of the cumulative rewards:
) -(E[ViPP:,,)))
(sh (1).)aa)))
VaR
E
Eh=1 2H 2dRH (0.s) )
< P 1
8 0
en
G + G.epda (1)) T
1 + an
2dn (0.s) ad (0,s) s)))
where we use a = in the last line. Let ep < 11/(2lp) and & S 1 fd we have:
VaR_,r < 2 + 2dg
O s) sd.
0 م²
Therefore, the lower bound for N and K can be reformulated using the variance of the accumulated rewards:
N 2 Ω (max VVR va
K > 9 (max (VaR,, de. (oxppNN/// VaR,, 2ddd
Substituting VaR with o together with dR = (log RR)) and dp = O(log (Pl) completes the proof.
36

---

Published as a conference paper at ICLR 2026
Algorithm 2 Model-based AIL (Practical)
Input: Number of iterations K, discount factor 7, expert dataset Dg, policy ne, model ensemble {Pe, /:::0- with ensemble size D and reward model Tv.
1: for k ..2....KK do
2: Collect trajectories using current policy and populate behavioral buffer B".
3: Sample behavioral and expert batches B", B . B
4: Update ro using Eq. E.
5: Update model ensemble (P.. using Bd U B using Eq. E.2
6:
Collect D model rollouts {Ta [=d using (P& h=::--1 and No.
7:
Estimate the value of each rollout V(Ta) (
8: Select the rollout with 7* = argmax ((Ta) and store it into the model buffer BM
9: Train the RL algorithm on the model buffer Bd using the reward model Tv to update the
policy Te.
10: end for
E PRACTICAL IMPLEMENTATION
In this section, we present practical implementation of MB-AIL, optimization-basel approach with neural network parameterizations for the policy тg, reward Twh, and transition model ensemble (P=DD------ For reward learning, the reward model is optimized using the following training objective:
min CR(() = (s..a))BB as) - (.,.)) (s, + Φ(r),
(E.1)
where 1 and E denote the replay buffers for the policy and the expert, respectively. The term ((r) is regularization function, for which we adopt the gradient penalty (Arjovsky et al. 2017).
Then for each model Pe: in the ensemble {P&: D-1 ) we update it using the maximum likelihood
objective conducted on data sampled jointly from the expert and behavioral replay buffer:
min (E)))) = -B log Pg. ())
(E.2)
The practical implementation the algorithm summarized in Algorithm In Line 9, we employ Soft Actor-Critic (SAC) Haarnoja et al. (2018) the RL algorithm.
F GRIDWORLDD EXPERIMENTS
Environment Setup We conduct toy experiment 9 9 GridWorld environment, illustrated in Figure 3 Rewards of value are available only in the top-right corner of the grid. The state space corresponds to the dynamics are stochastic: when an action is taken (c.g., up), the agent transitions to the intended neighboring cell with probability pt each of the other three neighboring cells selected with prob- ability {1 - p). Each episode lasts 20 steps, and performance is measured by the accumulated reward. We use single expert trajectory with a total reward 8.0 across all experiments. We initialize the agent in the area (0, (0, 1), (1, 0), 1)).
grid, and the action space A {up, down, left, right). The environment
We evaluate behavioral cloning (offline) and online imitation learning in this environment in order to analyze the benefits of online exploration in the context of imitation learning. Both BC and online IL agents do not observe rewards directly but have access to the expert trajectory. For BC, the agent mimics expert actions in states encountered in the trajectory. and acts randomly elsewhere. For online IL. we maintain both a Q-table and a reward table, initialized with zeros. The Q-learning agent explores using e-greedy with actions chosen as arg max Q(s, a). Whenever the agent encounters a state-action pair from the expert trajectory, it assigns r(8, a) = 1 in its reward table; otherwise, the reward is set to O. The Q-table is updated from visited experiences using this reward table.
Experiment Settings We evaluate two settings in the GridWorld experiment. The first examines the effect of reward space size. Specifically, we partition the reward ta- ble into n n regions, assigning the same reward value within each region. Thus, ([N/n])2. where N 9. We fix p 0.65 and test n = 1,2,3,4 to study the impact of varying reward space sizes.
the reward space size scales as [R]
37

---

Published as a conference paper at ICLR 2026
Different the Reward Space
Different Environment Stochastidty
Deet
Eollnds
Figure 2: Results for Grid Experiments. We present results analyzing the impact of varying reward space sizes and different levels of environment stochasticity on adversarial imitation learning and behavioral cloning, reported over 5 random seeds.
The second setting investigates the effect of environment stochasticity on imitation learn- ing performance. We fix n 4 and control stochasticity by adjusting the transition proba- bility p. a larger corresponds to more de- terministic environment. We experiment with p = 0.45, 0.55, 0.65, 0.75 to analyze this effect.
Main Results We present the empirical results of the two experiment settings in Figure 2 For each configuration,
evaluate 10 episodes for each the 5 random seeds and report the mean and standard deviation. In the reward space ex- periment, when the reward space is large, on- line IL performs similarly BC. However, the reward space becomes smaller (larger n), online IL achieves significantly better perfor- mance, approaching the optimal value, which is consistent with our theoretical upper bound in Theorem [.1 In the stochasticity experiment, both online IL and BC improve as the environment becomes more deterministic. Across stochas- ticity levels, online IL consistently outperforms BC.
Senue Rayon
Figure 3: GridWorld Illustration. An illustra- tion of the 9 x GridWorld with the agent initial- ized at (0, 0) shown.
Results on the Model Misspecification order to evaluate the behavior of online IL when Assump- tion 3.5 is mildly violated, conduct experiments evaluate the impact of model misspeci- fication. Specifically, we randomly sample m states in GridWorld as misspecified states. During Bellman updates, whenever transition leads write the next state as (0, 0), thereby introducing controlled transition model misspecification. The results show that the algorithm maintains strong performance when the proportion of misspecified states is small, but its performance degrades as the degree of misspecification increases. The detailed results are shown in Figure 4
of these misspecified states, manually over-
G DETAILS OF THE MUJoCo EXPERIMENTS
Hyperparameter Setting We summarize the hyperparameter settings used in our experiments be-
low:
Model ensemble size is for tasks.
Model optimization: learning rate 3 x 10-4, weight decay 5 10-5 batch size 256.
Policy parameterization: stochastie Gaussian policy.
Discriminator: ensemble size 7, trained with batch size 4096 and learning rate 8 x 10
SAC training: learning rate x 10 batch size 256, entropy coefficient a = 0.2 for all tasks.
38

---

Published as a conference paper at ICLR 2026
Different Levels of Model Misspectfication
Entodd Rowards
Number Misspecified
Figure 4: Results on the Model Misspecification We show the performance of online imitation learning when some of the states in the GridWorld will lead to misspecification. Results show that the algorithm maintains strong performance when the proportion of misspecified states is small, but its performance degrades as the degree misspecification increases. The results are reported using 5 random seeds.
Network architecture: hidden size 400 for the model network, and 512 for both the dis- criminator and policy networks.
Environment Details The specifications of the environments are summarized in Table 6
Enviromment
Observation Dimension
Action Dimension
Hopper
11
Walker2d
17
6
Humanoid
45
17
Table 6: Environment Details. Observation and action dimensions for each environment. The Humanoid task features higheerdimeenioonl state and action space, making it significantly more challenging for imitation learning compared to the lower-dimensional Hopper and Walker2d tasks.
Baseline Methods For the OPT-AIL baseline (Xu et al. 2024), adopt the official implementation provided in their repository For the GAIL baseline (Ho & Ermon 2016), we use the open-source implementation available at this repository) For the BC baseline, train two-layer MLP with hidden dimension 256 on the expert dataset for direct behavioral cloning.
USE OF LARGE LANGUAGE MODELS
We used LLMs solely as a writing assistant for minor grammar and phrasing corrections during manuscript preparation. LLLMs were not involved in research ideation, experiment design, data analysis, or result interpretation.
39


Reviews:
[{"review_id": "7u8GfVLiZl", "reviewer": "Reviewer_yf8s", "summary": "This paper has proposed MB-AIL, a model-based adversarial imitation learning algorithm (Algorithm 1). Second-order regret/sample complexity upper bounds are provided in Section 5.1 (see Theorem 5.1 and Corollary 5.2), and minmax lower bounds are provided in Section 5.2 (Theorem 5.9). Preliminary experimental results are provided in the appendices.", "questions": "Please try to address the weaknesses listed above.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- Overall, this is a very strong theoretical machine learning paper. To the best of my knowledge, the proposed MB-AIL algorithm is novel. This paper has established both upper regret/sample complexity bounds for MB-AIL and minmax lower bounds. They together justify that MB-AIL is near-optimal in both its use of online interaction and its dependence on expert demonstrations.\n\n- This paper has done a very good job of literature review, as summarized in Table 1.\n\n- Extensive and rigorous discussions on the theoretical results are provided in Section 5. Many of such discussions are thought-provoking.\n\nOverall, I recommend accepting this paper.", "weaknesses": "- It is not clear to me why the authors do not include **any** experimental results in the main body of the paper. Of course, there is a page limit, but I think the authors can rewrite the paper to include at least one experimental result. I think this will make the paper more \"balanced\" between the theoretical results and the experimental results.\n\n- My understanding is that the realizability assumption (Assumption 3.5) is a major weakness of this paper. In particular, with function approximation, this assumption rarely holds in practical problems. I fully understand that this is a standard assumption made in theoretical papers to ensure the analyses are tractable, and analyzing the proposed algorithm without this assumption can be very difficult. However, I recommend that the authors add some experimental results when Assumption 3.5 is (mildly) violated to justify that the proposed algorithm is robust to (mild) model misspecification. (It is not clear to me if such experimental results already exist in the appendices; if so, please explain.)\n\n- In Theorem 5.1, the regret bound depends on three notions of complexities: the eluder dimension $d_E$, the bracketing number for the model class $\\mathcal{N}_{\\mathcal{P}}$, and the covering number for the reward class $\\mathcal{N}_{\\mathcal{R}}$. Please better explain why we need three different notions of complexities to establish this regret bound. The readers should be able to understand the intuitions without reading the proof.\n\n- I recommend accepting this paper, but I do feel that this paper is too \"dense\" for a conference paper. I recommend that the authors further polish the writing to make it more readable.\n\n- There are still minor typos in the paper. Please make another pass and fix them. Examples:\n  - In the first equation on Page 4, $a_{h'}$ should be $a_t$\n  - In equations 3.1 and 3.2, the expectation over the initial state is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bXfL9WBF5Y", "reviewer": "Reviewer_7CkW", "summary": "The paper studies how an agent can learn to imitate expert behavior using both expert demonstrations and online interactions, without access to reward signals. The authors propose a new algorithm called Model-Based Adversarial Imitation Learning (MB-AIL). It combines model-based reinforcement learning with adversarial imitation learning by separately learning a reward function from expert data and a world model from online interactions.\n\nThe paper provides strong theoretical results. It proves that MB-AIL achieves horizon-free, second-order sample complexity bounds, meaning its efficiency depends on the variance of returns and improves in more deterministic settings. It also establishes information-theoretic lower bounds showing that MB-AIL is nearly minimax-optimal—it uses about as few samples as theoretically possible. Experiments on MuJoCo and GridWorld tasks confirm the theory, showing that MB-AIL matches or surpasses some imitation learning methods in sample efficiency.", "questions": "1. The analysis assumes that the true reward function, transition dynamics, and optimal policy are contained within the function classes (Assumption 3.5). How sensitive is MB-AIL to mild violations of this assumption?\n2. Remark 5.6 states that AIL is better than BC when $\\log (N_{R}) < \\log (|\\Pi|)$. Could the authors empirically validate this claim by controlling the complexity of the policy networks and reward networks? \n3. The established theory shows that the variance term plays a key role in AIL. However, the experiments do not validate the role of variance in theory. Could the authors show experiments where the environment or policy stochasticity is systematically varied?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper derives horizon-free, second-order upper bounds for adversarial imitation learning under general function approximation. This is technically meaningful and advances the theoretical understanding of how stochasticity affects AIL.\n2. The idea of learning transition models in AIL is natural and well-motivated. Splitting the learning into adversarial reward estimation + model estimation + optimistic planning is elegant and leverages the strengths of model-based RL in a principled way.", "weaknesses": "1. As stated in the introduction, the paper's main goal is to provide a tight characterization of the benefits of online interaction in imitation learning. However, the result is unsatisfying and does not clarify when adversarial imitation learning outperforms BC. The paper claims that online interactions are helpful only when $\\log (N_{R}) < \\log (|\\Pi|)$. In practice, however, it is difficult to compare the complexity of the policy class and the reward class, which are typically neural networks.\n2. The lower bound result (Theorem 5.9) appears to be incorrect. If I understand correctly, Theorem 5.9 claims that achieving an approximately optimal policy requires both $\\Omega (\\sigma^2 \\epsilon^{-2})$ expert demonstrations and $\\Omega\\left(\\sigma^2 \\cdot \\log ^2|\\mathcal{P}| \\exp (-N) \\epsilon^{-2}\\right)$ online interactions. However, as shown in Table 1, [Foster et al., 2024] proved that BC requires $\\Omega (\\sigma^2 \\epsilon^{-2})$ expert demonstrations but zero online interactions, which conflicts with Theorem 5.9.\n3. The paper provides a practical version of MB-AIL, but this version differs significantly from the original algorithm, creating a notable gap between theory and practice. The original algorithm constructs a version space over transition models and solves a joint optimization problem over both the policy and transition model. The practical version ignores this design entirely—it simply learns transition models using maximum likelihood estimation and then runs RL on the learned model. From this view, such a model-based adversarial imitation learning has been proposed in [1]. \n4. The experimental validation is insufficient to substantiate the paper’s claims that MB-AIL matches or surpasses the sample efficiency of existing methods. The empirical study is limited to a single toy GridWorld environment and three MuJoCo tasks, which do not provide enough diversity or complexity to convincingly demonstrate the generality of the proposed approach. Moreover, the comparisons are restricted to only two adversarial imitation learning baselines, omitting several important and state-of-the-art methods (e.g., [1, 2]). Including these stronger baselines is essential for a fair and comprehensive evaluation. Without broader empirical coverage and more competitive baselines, the experimental evidence remains too weak to support the strong performance claims made in the paper.\n\nReferences:\n\n[1] Juntao Ren, Gokul Swamy, Zhiwei Steven Wu, J Andrew Bagnell, and Sanjiban Choudhury. Hybrid inverse reinforcement learning.\n\n[2] Gokul Swamy, David Wu, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Inverse reinforcement learning without reinforcement learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7mo1pMiJem", "reviewer": "Reviewer_J4QT", "summary": "This work takes a theoretical perspective on adversarial imitation learning (AIL), providing results that clarify the benefits of leveraging online interactions and the role of stochasticity in both the expert policy and the environment’s transition dynamics.", "questions": "1. What new practical insights do the theoretical results provide for the design of MB-AIL? Are there concrete guidelines that practitioners can draw from the analysis?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This is a well-presented and solid theoretical contribution to AIL. The paper formulates an interesting in-depth analysis, leading to theoretical results that, to my knowledge, were not previously established in the literature.", "weaknesses": "1. It is not fully clear what new practical insights the theoretical analysis provides for the design of MB-AIL. While the bounds are technically interesting, the paper does not convincingly articulate how these results translate into concrete algorithmic guidance or practical improvements.\n\n2. The presented algorithm MB-AIL is derived by decomposing the regret into the error of reward and error of the model. Other studies, not mentioned in the Related Work, have previously leveraged this division in their analysis. Refer to:\n\n[1] Chen Y, Giammarino V, Queeney J, Paschalidis IC. Provably efficient off-policy adversarial imitation learning with convergence guarantees. arXiv preprint arXiv:2405.16668. 2024 May 26.\n\n[2] Shani L, Zahavy T, Mannor S. Online apprenticeship learning. InProceedings of the AAAI conference on artificial intelligence 2022 Jun 28 (Vol. 36, No. 8, pp. 8240-8248).\t\n\nMinor: \n\nA few typos and grammar errors: “stragety” in Line 71, “as an practical example” Line 277, “Despite of this difference” in Line 282.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HPulgSMdtf", "reviewer": "Reviewer_PeTQ", "summary": "The authors derive second order sample complexity bound in the context of imitation learning where the learner can collect reward free trajectories.\nAs a result the expert samples and MDP interaction complexity only scales with the maximum reward variance where the maximum is taken over both the reward and policy class and not on the horizon. These are very interesting results.\n\nMy low score at the moment is because there seems to me that there is a contradiction between the lower bound in theorem 5.9 and the BC upper bound, which scales only with the expert variance and not with the maximum variance over the policy and reward class.", "questions": "The lower bound seems related to the construction used in [2] Section 5.3. Could you elaborate on the differences?\n\nRelated to weakness 2. Does MB-AIL need to observe the actions or only the values of the reward functions in $\\mathcal{R}$ evaluated at the state actions in the expert dataset?\n\n In the algorithm, how to solve efficiently the problem at Step 7 where the maximization is over both policies and transition dynamics in the confidence set. Are you planning to use extended value iteration? Can this be done in a more computationally efficient way using corresponding exploration bonuses?\n\nPlease answer also to my questions raised in the Weaknesses Section.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "The analysis relies on elegant techniques and it looks correct to me.\n\nThe results are important for the imitation learning literature.\n\nThe experiments are convincing.", "weaknesses": "Here are my main suggestions on how to improve the presentation of your work. I think that the most important one is weakness 5 since it identifies a potential technical flow. I am willing to raise my score significantly once I am sure that there is no technical issue there.\n\n1) In Remark 5.6, the authors discuss that in environments where the expert is complex ( ie Pi contains many functions) than the BC bound in Foster et al 2024 becomes suboptimal compared to MB-AIL, claiming in this way that online interactions are beneficial. However, even in the offline case, it is possible to obtain bounds that are independent of $\\Pi$ but that depend on the covering number of the $Q$ value class (see [1]). I suggest that the authors add a comparison with this result, and they suggest that it is reasonable to expect that $\\abs{\\mathcal{R}} \\leq \\abs{\\mathcal{Q}}$ because for a fixed reward function, I can obtain many different $Q$ functions changing policy and dynamics.\n\n2) In the context of Linear MDP, algorithms such as BRIG or FRA-IL[2] can work when observing only the reward features visited by the expert and not the chosen actions. This setting captures as an important example of state-only imitation learning. Similarly, in the context of general function approximation considered by your work, I think that it's enough to observe all the reward functions in the class $\\mathcal{R}$ evaluated at the expert state action pairs. Observing the actions seems unnecessary. Pointing this out makes the work even stronger because it would make your algorithm applicable when BC is not (because BC needs to observe the actions).\n\n3) I found Remark 5.5 a bit odd because if expert and dynamics are deterministic, then trivially only one trajectory is needed. Therefore, $1/\\epsilon$ bound in this setting is disappointing. Noticed that Foster et al. 2024 attained $\\mathcal{O}(1/\\epsilon)$ assuming only a deterministic expert but not transitions. Also in Table 2, for the case deterministic expert and deterministic transitions (right bottom cell ), I think that one trajectory is enough for BC because simply copying the actions can guarantee remaining with probability one on the support of states visited by the expert. Can the same result be proven for MB-AIL ?\n\n4) The expert sample complexity of MB-TAIL scales with the maximum variance over the policy class while the BC bound scales with the maximum variance in class. Can you improve the bound to match the variance dependent bound in BC ? Please fix this in Table 1 where this difference is not visible. \n\n5) This also seems at odd with the lower bound in Theorem 5.9. The authors there state that any online or offline algorithm needs to pay the maximum variance in class. How is this possible considering that the bound for BC scales only with the expert variance. Moreover, how is it possible to prove a lower bound on $N$ for any offline algorithm since by definition an offline imitation learning algorithm uses $N=0$?\n\n[1] Moulin et al. 2025, Inverse Q Learning Done Right: Offline Imitation Learning in $Q^\\pi$ realizable MDPs\n[2] Moulin et al. 2025, Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learning\n\nMinor\n\nAt line 358, I think it should $\\Pi$ and not $\\mathcal{R}$.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)