IL-SOAR : Imitation Learning with Soft Optimistic  Actor cRitic
Authors: Stefano Viel
Luca Viano
Volkan Cevher
Venue: ICML 2025 poster (2025)
Keywords: provably efficient IL
SAC based IL
Deep IL
Primary area: reinforcement_learning->inverse

Abstract: This paper introduces the SOAR framework for imitation learning. SOAR is an algorithmic template 
which learns a policy from expert demonstrations with a primal-dual style algorithm which alternates cost and policy updates. Within the policy updates the SOAR framework prescribe to use an actor critic method with multiple critics to estimate the critic uncertainty and therefore build an optimistic critic fundamental to drive exploration.
When instantiated to the tabular setting, we get a provable algorithms dubbed FRA with guarantees matching the best known results in $\epsilon$.
Practically, the SOAR template is shown to boost consistently the performance of primal dual IL algorithms building on actor critic routines for the policy updates. Approximately, thanks to SOAR, the required number of episodes to achieve the same performance is reduced by a half.

IL-SOAR Imitation Learning with Soft Optimistic Actor cRitic
Stefano Viel
Luca Viano *1
Volkan Cevher
Abstract
Indeed, several recent theoretical imitation learning achieve performance guarantees by adding exploration bonuses on top of the regularized policy updates, which encourage the learner to visit state-action pairs that have not been visited previously. Unfortunately, such works are only available in the tabular setting (Shani et al., 2021; ; Xu et al., 2023) and in the linear setting ( (Viano et al., 2024). The design of the exploration bonuses in these works is strictly tight to the tabular or linear structure of the transition dynamics, therefore, these analyses offer little insight on how to design efficient exploration mechanism using neural network function approximation.
This paper introduces the SOAR framework for imitation learning. SOAR an algorithmic tem- plate that learns a policy from expert demonstra- tions with a primal dual style algorithm that alter- nates cost and policy updates. Within the policy updates, the SOAR framework uses actor critic method with multiple critics to estimate the critic uncertainty and build an optimistic critic funda- mental to drive exploration.
When instantiated in the tabular setting, we get provable algorithm with guarantees that matches the best known results in the desired accuracy parameter c.
There is, therefore, lack of a technique that satisfies the following two requirements.
Practically, the SOAR template can boost the performance of any imitation learning algorithm based on Soft Actror Critic (SAC). As an exam- ple, we show that SOAR can boost consistently the performance of the following SAC-based imi- tation learning algorithms: f-IRL, ML-IRL and CSIL. Overall, thanks to SOAR, the required num- ber of episodes to achieve the same performance is reduced by half."
It is statistically and computationally efficient in the tabular setting.
It can be implemented easily continuous states and actions problems requiring neural networks function approximation.
In this paper, we present a general template, dubbed Soft Optimistic Actor cRitic Imitation Learning (SOAR-IL) sat- isfying these requirements.
The main idea is to act according to an optimistic critic within the SAC block on which many IL algorithms rely. Here, optimism means appropriately underestimating the expected cunulative cost incurred by playing policy the environment. This principle known as optimism in the face of uncertainty has led to several successful algorithms in the bandits community.
1. Introduction
Several recent state of the art imitation learning (IL) algo- rithms (Ni et al., 2021; Zeng et al., 2022; Garg et al., 2021: Watson et al., 2023; Viano al., 2022b) are built on Soft Actor Critic (SAC) (Haarnoja et al., 2018) to perform the policy updates. SAC uses entropy regularized policy up- dates to maintain a strictly positive probability of taking each action. However, this is known to be an inefficient exploration strategy if deployed alone DessaBBianchi et al., 2017).
While optimism is often achieved using the structure of the problem (tabular, linear, etc.), in this work, we build optimistic estimators using an ensemble technique. That is, multiple estimators for the same quantity are maintained and aggregated obtain an optimistic estimator. This technique scales well with deep imitation learning. To summarize, have the following contributions.
'Equal contribution dence to: Stefano Viel <stefano.viel @eh Luca Viano <luca.viano@ eptn.o.pfl..h.....
EPFL, Lausanne.
Correspon-
Theoretical contribution We show that there exists computationally efficient algorithm that uses an ensemble based exploration technique that gives access to O(c expert trajectories and Ole interactions in a tabular MDP outputs a policy such that its cumulative expected cost at
Proceedings of the 4nnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 hy the author(s).
Project code available https://ght httgggttuubb..oo com/ stefanoviel /SOAR- Teteer-i.ttteeeeeeaaaee

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Average Performance Across Environments
ML-IRI
ML-IRL
Average Normalzee Retum
1.0
0.2
0.0
0.2
0.1
0.6
0.5
1.0
0.0
0.2
0.4
0.6
0.5
100
0.0
0.2
0.1
0.6
0.8
1.0
0.0
0.2
0.1
0.6
0.0
1.0
Episode
Egisode
1e5
Episode
1e5
Episode
106
CSIL (Base)
ML-IRL (SA) (Bose)
MI-II (Base)
RXC (Base)
Expert
CSIL SOAR
ML-RRL (SA) + SOAR
ML-IL- + SOAR
RXL SOAR
Figure 1: Summary of experimental results. Each plot compares the average normalized return across MuloCo environments with 16 expert trajectories for a base algorithm and its SOAR-enhanced version. SOAR replaces the single critic in SAC-based methods with multiple crities to compute an optimistic estimate. Across all algorithms, incorporating SOAR consistently improves performance. ML-IRL (SA) stands for ML-IRL (Zeng et al., 2022) from expert stale-action demonstrations.
most e higher than the expert cumulative expected cost with high probability.
as dt E AsxA and defined as follows d' a) & (1
7) = [s, a is visited after h steps acting with rr).
We can also define the state occupancy measure as d d'((s) -
^AI [s is visited after h steps acting with T]. (1 11-7) En=o
Practical Contribution We apply an ensemble-based ex- ploration technique, SOAR, to boost the performance of deep imitation learning algorithms built on SAC, demon- strating its effectivenesss on MuJoCo environments. Specifi- cally, we show that incorporating SOAR consistently boosts the performance of base methods such as Coherent Soft Imitation Learning (CStL))WWtson et al., 2023), Maximum Likelihood IRL (ML-IRL)(Zeng et al., 2022) and RKL (Ni et al., 2021). As shown in Figure 1, our approach consis- tently outperforms the base algorithms across all MuJoCo environments. Notably, SOAR achieves the best perfor- mance of the baselines requiring only approximately half the number of learning episodes.
Imitation Learning In imitation learning, the learner is given dataset Dng of expert trajectories collected by an unknown expert policy mg. By trajectory 7², mean the sequence of states and actions sampled rolling out the policy mr for a number of steps sampled from the distribution Geometric(1 - -)). Given Drc the learner adopts an algorithm to learn a policy nou such that Vo, Veruue
-VE Oreue
with high probability.
We use the notation c, to denote vector in RISI zero ev- erywhere but in the coordinate corresponding the state s for an arbitrary ordering of the states). Analogously, use es,a to denote vector in R|S|| zero everywhere but in the (s, a) entry which equals one.
2. Preliminaries and Notation
The environment abstracted as Markov Decision Pro- cess (MDP) (Puterman, 1994) which consists of tuple (, P. Po: 7) where is the state space, A is the tion space, P P:SxA A + As is the transition kernel, that is, P(s'|s, a) denotes the probability of landing in state s' after choosing action in state Moreover, Po is a distribution over states from which the initial state is sampled. Finally, S A + [0, 1] is the cost function, and y E [0, 1) is called the discount factor.
3. The Algorithm
In this Section, we describe Algorithm 1. A meta-algorithm that encompasses several existing imitation learning algo- rithms. Inside each iteration of the main for loop, the learner collects new trajectory sampling actions from the policy h (Line 4 in Algorithm 1) and then performs the following steps.
Value functions and occupancy measures We define the state value function at state s E S for the policy π under the cost function c as s]. The expec- tation over both the randomness of the transition dynamics and the one of the learner's s policy. Another convenient quantity is the occupancy measure of a policy æ denoted
The Cost update. At Line 6 of Algorithm 1, the learner updates an estimate of the true unknown cost function
(s) A E T as)ls1
"In order, to accommodate state-only and state-action with a unified analysis we overload the notation for the expert dataset. Dr, denotes collection of samples from the expert state occu- pancy measure in the former case and collection of state-actions sampled from the state-action occupancy measure the latter case.
2

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Algorithm 1 SOAR-Imitation Learning
In other words, it underestimates the update that could have been performed by value iteration if the transi- tion matrix P was known to the learner. We provide aggregation routines that satisfy this requirement if L is large enough.
Require: Reward step size a, Expert dataset D.,, mp Discount
factor y, Policy step size 77.
1: Initialize x' as uniform distribution over A.
2: Initialize empty replay b.ffer,i.e. D = -
3: for k = to K do
The policy update As the last step of each inner loop, the learner updates the policy using the optimistic state-action value function estimate. In the tabular case, we will instantiate the update using an online mirror descent (OMD) step (Beck & Tehoulle, 2003; Nemirovskij & Yudin, 1983) (also known as the multi- plicative weights update (Warmuth et al., 1997; Auer al., 1995)). As it will be evident from Section 4, this update we can ensure that the KL divergence between consecutive policies is upper bounded in terms of the policy step size 17. For the continuous state-action ex- periments, the online mirror descent is approximated via a gradient descent step on the SAC loss.
4: - <- COLLECTTRAFCCTRR((fT
5: Add 7* to replay buffer.i.e. DP -
6: c* +- UPDATECOST(( DaE D)
7:
for ( = 1 to L do
8: Compute estimator Q
9:
end for
10: Q* = OFTMMSSTICQ(()))
11: T (als) POLICYUPDATE(Y, (Q (8, ())
12:
end for
with the algorithm-dependent routine UPDATECOST. For instance, Generative Adversarial Imitation Learn- ing (GAIL), Adversarial Inverse Reinforcement Leam- ing (AIRL), and Discriminator Actor Critic (DAC) (Ho & Ermon, 2016; Fu et al., 2018; Kostrikov et al., 2019) use a reward derived from a discriminator neural net- work trained to distinguish state-action pairs visited by the expert from those visited by the learner.
Remark 3.1. Notice that only one pair of critics is used in the implementation of SAC (L 1) that serves as base RL algorithm for several commonly used II. algorithms ( (GAIL (Ho & Ermon, 2016), AIRL (Fu et al., 2018 8), IQ-Learn (Garg et al., 2021), PPIL (Viano et al., 2022b), RKL (Ni et al., 2021) and ML-IRL (Zeng et al., 2022)). As proven in Corollary 4.11, L, = 1 is not enough to ensure optimism, not even in the tabular case. In our experiments, we show that a value of L larger than 1 is beneficial in all the MuJoCo environments we tested on.
Using a fixed cost function obtained from a behavioral cloning warm up is, instead, the approach taken in CSIL (Watson et al., 2023). Moreover, updating the re- ward to minimize an information theoretic divergence between expert and learner state occupancy measure is the approach taken in RKL (Ni et al., 2021). Fi- nally, (Zeng et al., 2022) updates the cost using online gradient descent (OGD) (Zinkevich, 2003).
3.1. Algorithm with guarantees in the tabular case
We consider an instance of Algorithm in the tabular case for which we will prove theoretical sample efficiency guar- antees. We present the pseudocode in Algorithm 2.
The state-action value function update. In the for loop at Lines 7-9 of Algorithm 1, the learner updates L different critics trained on different subsets of the data sampled from the replay buffer D* denoted as {Dt)/_] |e=1" For a fixed state-action pair each dataset contains independent samples from P(-|s, a). This allows creating L jointly independent random variables {ot) L 1=1 that estimate the ideal value iteration update (i.e. c* +>PV* which cannot be implemented exactly due the lack of knowledge P.
For what concerns the analysis, the first step is to extract the policy achieving the sample complexity guarantees above via an online-to-batch conversion. That is, the output pol- icy is sampled uniformly from a collection of K policies { 'A=1' The sample complexity result follows from prov- ing that the policies { { K
j=m produced by Algorithm 2 is sequence with sublinear regret in high probability. More formally, we define the regret as follows.
Definition 3.2. Regret The regret is defined as follows
In Section 3.1, we provide an explicit way to compute L slightly optimistic estimates for the tabular setting. Moreover, in the deep imitation learning experiments, we train L different critics via temporal difference, as it is commonly done in Soft Actor Critic implementation see (Haamooa et al., 2018) ).
K
Regret (K) A
Cine; d=* dd
k=
Remark 3.3. Notice that the regret defined in this way sat-
isfies Regret(K) ΣK K ( (vo, V** V Cueue ) For this reason, we require the factor (1 -1--)-1 in the definition.
Finally, in Line 10 of Algorithm 1, the L critics are ag- gregated to generate an estimate Q² which is, with
high probability, optimistic i.e. Q++1 1 s ch PVV.
Omitting dependencies on the horizon and the state action
3

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
spaces cardinality, we will guarantee that
These cases differ only in the stochastic loss for the cost update. Notice that overload the notation to address both state-only and state-action imitation learning with a unified analysis. In particular, d=* is an unbiased estimate of the learner occupancy measure. For state-only imita- tion learning we use d=* Cat. and estimated expert occupancy measure equals to d = [Dra 1 "EvED,, e, while for state-action imitation learning d== e 22
Regret(K) I O(K)/2 + K [PAEl )/
with high probability. Notice that this bound is sublinear in K, for ne O(K). To obtain such bound, we adopt the following decomposition for (1 Regrett (K) adapted from (Shani et al., 2021) accommodate the infinite hori- zon setting.
d
TH
Cerue &t, d=t d
and dhs = (Daml I "E..ED es,a- The formal bound on Regret, is given in Theorem 4.3. The rest of the algorithm
==1 Heggett, (K,
== Hlegettt (KK
Algorithm 3 COSTUPDAIIETABULAR
Algorithm 2 Tabular SOAR-IL
Require: Current cost vector ct1,, trajectory 7*, expert
Require: Step size 1), Expert dataset Drg. Discount factor (s, y, Reward step size a, ( a) = 0 for all number
dataset DTT.
1: if STATE-ONLY = TRUE then
of estimators L = 36 log (IS) [A|] K/5).
2: tt
= =0, i,k
1: Initialize ग as uniform distribution over A
3: d= (DiEll -1 "E C2
2: for k ==11 K do
3: Sample trajectory length L* Geometric(1 - 7)) L*
4: else
4: r² = ((s ( } rolling out ** for LK steps.
d=
= = + i
5: Update counts for all st of E Tk:
6: dev. (Dorel E LS es,a
end if
(5 at) = (st at) + 1.
8: Return: eh Ie [ot ald=e a(d=t- dra))
6: Add ott at s/-1 1 to the datasets with index L t= No ot) mod L.
aims to provide a sublinear bound on Regret,. In particular, the updates for the estimated transition kernels { [xx) J Jem
U fot, at) R = R U fst stt...
D = DJ-
in Lines 8-12 of Algorithm 2 serves to build L slightly optimistic 3 estimate of the ideal value function update.
7:
* = COSTTUDATTTAABUARR 1,,, Dee).
8:
forl= mmr=1to do
In the routine APTIMISTICCTAABLLA aggregation rules to generate the optimistic Q value estimate to be used in the policy update step. The first one, takes the minimum of the estimators as in Equation (Min), while the second option Equation (Mean-Std considers the mean of the I. estimators minus a factor proportional to the empirical standard deviation. By Samuelson's 's inequality (Samuelson, 1968), we prove that the second option is more optimistic.
we propose two
9:
(H = ERN (80,8==00))))
10:
(5(s,0) EESaED; 1 [[,,==a)).
()()),,))= (maa))
11:
NE(s.a)+2
12:
end for
13:
Qu+l= OPTQTAB(VV> { {Pe]e
f=1 ct).
14:
T (als) x tt (als) exp (-nQ (s,
15:
(++ = ( (.1s), 1 (s.
Finally, an iteration of the tabular case algorithm is con- cluded by the policy update implemented via OMD.
16: end for
17: Return The mixture policy a
Having described our main techniques are in the position of stating our main theoretical results hereafter.
The algorithmic design for the tabular setting aims at up- dating the cost variable so that the term Regret, grows sublinearly (see Line 7 in Algorithm 2).
Theorem 3.4. Main Result For any MDP, let us consider either the update Equation (Min) or Equation (Mean-Std), it holds that with probability 1 56 that Regret(K) of Tabular
We consider both cases of imitation from state-action ex- pert data (Lines 4-6 COStUPDATETABULAA ) and state- only expert data (Lines 2-3 of CASTUPDATETABBULARR)
K
'hhe optimism is achieved by adding 2 in the denominator of the estimated transition kernels.
4

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Algorithm 4 OOPTMISTICQTARUULAR OPTQTAB)
proof of Corollary 4.11, consider intermediate step in the proof MDPs where from each state action pairs is possible to observe transition to only two other possible states. While this restriction on the dynamics appears to be limiting any MDP can be cast into this form at the cost of a quadratic blow up in the number of states, from [S] to | To see this, for a general MDP where from given state action pair a transition to all possible |S] states can observed is equivalent to a binarized MDP where this one layer transition is represented with tree of depth at most log2 ([Sl) with binary transitions only. Moreover, the discount factor in the binarized MDP should be set to Ybin 7 logz|S] to maintain the return unchanged. We consider in this section a binarized MDP with |S] states in this section and we squared the number of states in stating Theorem 3.4 which holds for general MDPs. Moreover, in stating the result for general MDP we also inflated the effective horizon by factor log2 S| as shown in Lemma C.6.
Require: current state value function estimate Vš, ensem-
ble of estimated transitions
{ p t=1
cost ch.
1: // Option 1
return Qh+1= = = + min plyk [E(L]
(Min)
2: // Option 2
return Qu+l = c +
1
PAVk 0
max
L
Mean-Std)
3: with o² Viin (PAVk - - + Pg
SOAR-IL (Algorithm 2) is upper bounded by
As mentioned, the proof decomposed into two main parts: (i) bounding the policy regret Regret, and (ii) bounding the cost updates regret Regret,- In particular, we can prove the two following results.
δ S1 |A| 10g(1/6) (1 7)K
S| |Al log (18) |A|| /8) (log([S)) + 2)2 (1- 7)² (Daul
Theorem 4.2. Policy Regret In a binarized MDP with [S] states and discount factor ?,, it holds that with probability 1- -- any policy n², Regret, (K, x²) is upper bounded by
Therefore, choosing K K= 5 |51 log(118)) and 'D
(1-7)²²*
1s AA (1
<1(1-))
holds that the mixture pol-
log [A|
+ (1 71- +
nK
K | |A log(1/0) (15) (1 (1-7)2
icy म satisfies Vo:
V#* Cerue
Vare
S with probability
111 77)
at least - 56.
Remark 3.5. The bound on |D is the bound on the num- ber of either state-only or state-action expert trajectories depending on the setting considered.
and for v
/ K
[oo] A[I)A|(1-2)) it holds that using the update
in (Min) or in (Mean-Std) it holds that Regret (K, x²) is upper bounded by δ
(
Remark 3.6. The gurantees are stated for the mixture pol- icy * i.e. the policy which has an occupancy measure equal to the average occupancy measure of the policies in the no-regret sequence. That is, it holds that d K- EX d=d The policy @ cannot be computed with- out knowledge of P but sampling trajectory from it can be done by choosing an index k ~ [[K]] at the beginning of each new episode and continuing rolling out the policy R for a number of steps sampled from Gcom(1- 7).
(1-7))
Theorem 4.3. Cost Regret In binarized MDP with |S] states and discount factor 7; it holds that with probability 26, (1 Regret, (K; Ctrue) is upper bounded by
|||A log [S)|A|//
4V K log(1/0) K
2 1Drr
Remark 4.4. Once Theorems 4.2 and 4.3 are proven the bound Theorem 3.4 follows trivially by union bound and bounding Regret, and Regret, with Theorem 4.2 and Theorem 4.3 respectively and dividing everything by K (because in Theorem 3.4 we consider the quantity (eggre(KK//a Finally, we also divide by 1 - 7, to match the definition of Regret( in Definition 3.2.
Remark 3.7. In the case of state-only expert dataset the provided upper bounds for K and |D=,l are optimal up to log factors in the precision parameters e. Indeed, these upper bounds match the lower bounds in (Moulin et al., 2025).
4. Theoretical analysis
4.1. Proof Sketch of Theorem 4.2
We need to start with an important remark the structure of the MDP considered in the proof. Remark 4.1. For technical reasons, in particular for the
The regret decomposition towards the proof of Theorem 4.2 leverages the following Lemma.
5

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Lemma 4.5. Consider the MDP M AS,,,,.c. vo:?) and two policies n, S A4- Then consider for any Q E RISIIAA and (s) = (T())s) (((.))..((s.)) and
(1)) s O(n) for all S. Thanks to this slow changing property, we can prove the following bound on (Shift).
Q²,, V= be respectively the state-action and state value function of the policy ı in MDP M. Then, it holds that
Lemma 4.8. For the sequence of policies {I² [=²] gener- ated by Algorithm 2, for all 1j > 0, it holds that (Shift) < nk
(1-7) (po V== -o-V==vr
V=l)
equals
(1-)8
Remark 4.9. The step size choice for in Theorem 4.2 is made to trade off optimally the bounds in Lemmas 4.7 and 4.8.
(1 1-C---c--ppp + (((6) n' 2111))
Remark 4.6. This Lemma is generalization of the well- known performance difference Lemma (Kakade, 2001) to the case of inexact value functions. Indeed, notice that if Q = Q", then the first term in the decomposition equals zero and the result boils down to the standard performance difference Lemma. For arbitrary Q, the first term a tem- poral difference error averaged by the occupancy measure d=
Finally, the most technical part of the proof aims at bounding the term (Optimism).
Lemma 4.10. Let us consider an MDP where max supp(P(,Is.a)) = 2 For each k E [K], if the Q++1 in Algorithm 2, are updated according to (Min) or (Mean-Std), the iterates produced by Algorithmm 2 satisfy with probability 1 36 that
We can apply two times Lemma 4.5 on each of the sum- mands of the sum from k 1 to K, to obtain convenient decomposition of Regret, Denoting (s,(s)) a as)
V K | |A| log(1/6)
(Optimism) Sō
PVV((s, a) Q (s, a) and g" (s, a) & (staa)
@* (8.a), we have that
Proof sketch of Lemma 4.10 The proof of this Lemma, leverages that the temporal difference errors 6s a) pro- duced by Algorithm 2 are positive with high probability as shown by the next result*.
(1 7) Regret, (
(1 7)
Vo, V= - V* + pk Vx¹)
4.11. Consider an MDP where max,,0eSxA supp(P((|s,00
Corollary
= 2. then for it holds that with probability
K
E Bendor (s, - x -())
(BTRL)
L 2 36 log
|S||
at least 1 1- 8
k=1
d=a (8
dd (s) (s) [s,
min (s,a)s s (s) v s,ae SxA, wkk [K]. EE[L]
(Optimism)
Corollary 4.11 implies that (d='8")) < 0
0 for all & e [K]
K
+
E a [g" as]
E amd [gt (s, a)
and therefore that (Optimism) S > ( ,5+)
Remark 4.12. The above inequality, it is crucial for obtain- ing the result. Indeed, it upper bounds (Optimism) with the on-policy temporal difference errors which are small enough ensure sublinear regret. To see this (informally) consider two cases. First, let us assume that d= is rela- tively large for some action pair. Then, that action pair expected to be visited often in the rollouts and therefore 8 is expected to be small. Vice versa, if 5 for certain state-action pair is large, this means that for that state-action pair d== relatively small. Overall, we always expect the product ( (a"", 8th to be small quantity. Notice that the
(Shift)
Next, we bound each of these terms individually. Starting from the first term, the next Lemma shows that our policy update (Line 14 of Algorithm 2) can be seen as an instance of Be the regularized leader (BTRL) see e.g. (Orabona, 2023) ). Therefore, guarantees that for any sequence {Q*} JK=1 the term (BTRL) is bounded follows.
Lemma 4.7. Let us consider the sequence of policies K
{T*)] k=1 generated by Algorithm 2 for all > 0 then holds that (BTRL) < log]AA
'In the main text, we present the proof for the update in (Min). The case of update as in (Mean-Stte Std) is deferred to the Appendix. "That the temporal difference errors 8 averaged by the learner occupancy measures d=
Next, we show that thanks to the multiplicative weights update for the policy the KL divergence between consecu- tive policies is upper bounded by the policy step size 17, i.e.
6

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
same arguments could not have been carried out replacing d= with dt because the rollouts used in Algorithm are not sampled with n².
in the proof is to decompose (1 - 7) Regret, as follows
c. ,d -d
Crrue
To formalize the above intuition, we upper bound the tempo- ral difference errors with the inverse of the number of times each state-action pair is visited.
K
Crme - ch. d d=
Lemma 4.13. (Simplified Version of Lemma B.5 ) Let us consider a binarized MDP with (S] states and discount factor y With probability 1 - 6, it holds that for all a E S x A and for all k E [K].
Otrue ch dd d
The first term in the decomposition is upper bounded by O(KK) via a standard online gradient descent analysis (Zinkevich, 2003). Since d=* is an unbiased estimate of the learner occupancy measure, the second term in the de- composition is the sum of a martingale difference sequence. Therefore, an application of the Azuma-Hoeffding inequal- ity ensures that this term grows Ö (log(1/0)v/) with probability at least - 5.
(V
(5 a) 20
L (1g(1/)) (N) 1)(1 - 7
Therefore, by concentration inequalities and noticing that & $L,, at ~ d= holds that with high probability
K
K
0t) =
Σ
(s o
Finally, the last term is bounded as 1/2 1-13) with probability least
Ô ( [ (og((1))) 1- 6. This is done, proving that for the empirical average estimators for the expert occupancy measure it holds
I. S| log( (1/5)
0
(N (s atte) +1)(1 7 7)2
that d d S (D 1 log(1/5) with probability at least
K
L log(1 (1/5) (N LA Lx) + +)((1 - 7
union bound concludes the proof of Theorem 4.3. The formal proof is deferred to the Appendix.
K
At this point, the proof is concluded by bounding the last sum over K with a standard numerical sequences argument (see Lemma C.2).
5. SOAR for continuous state and actions problems.
In this section, we explain how Algorithm I is instantiated in imitation learning problems with continuous states and action spaces, which therefore requires neural networks to approximate the value function and policy updates. Since in our analysis for the tabular case, we need to use multi- plicative weights/softmax updates, we decided to use SAC, which is an approximation of such updates in the continuous state- action setting.
Optimal choice of the number of critics network L It important to notice that Corollary 4.11 and Lemma 4. 13
creates a tradeoff for what concerns the optimal choice of the number of critics. In particular, from Corollary 4.11, 11, L should be chosen large enough to ensure that optimism holds with high enough probability. On the other hand, one can notice that Lemma 1.13 upper bounds the expected on policy temporal difference error as (LL) therefore smaller number of critics ensures a tighter bound. All in all, the best choice is the smallest L that ensures optimism with probability at least - 8, that is L. = 36 log [S||A|))) The tradcoff with respect to the number of crities is also observed in practical ablation study (see Figures and 4).
However, the standard SAC keeps only one network, often called the critic network, to estimate the Q values. On the other hand, we use a pair of them to avoid the excessive overestimation noticed in Double DQN (van Hasselt et al., 2015). Since it uses only one pair of critics, SAC cannot achieve optimism reliably with high probability.
To fix this issue, we consider multiple crities and we used an optimistic estimate the mean minus the standard de- viation of the ensemble as explained in Algorithm 5. In addition, the standard deviation needs to be truncated at a threshold, as was done in the tabular analysis, to avoid the value function estimators growing out of the attainable range. For any state s, the estimated value functions are
4.2. Proof Sketch of Theorem 4.3
The proof of this term is considerably easier than the bound of the regret for the policy player because we have exact knowledge of the decision variables domain 6, The first step
"C is taken to he the f radius 1
7

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
truncated in the interval [0, -)
Each of the esti-
For the state-action experiments, plug in the SOAR template CSIL,, the state-action version of ML-IRL and HYPE (Ren et al., 2024). We coined the derived versions CSIL+SOAR and ML-IRL+SOAR (see Appendix F for detailed pseudocodes of these algorithms). We also com- pare with GAIL (Ho & Ermon, 2016), SQIL (Reddy et al., 2019b), and OPT-AIL. We observe that the exploration mechanism injected by the SOAR principle allows us to achieve reliably superior results (see Figure 3).
Algorithm 5 OPTIMISTICQ-NN
Require: Replay buffer D, Estimators ( maximum standard deviation d.
1: {s:)/=1 f sample observations from D
2: 0, < -s()))
3: ((); = Ei= ar(si,a))
4: std-Q(se,, at) = V Et=n (Qe(s;, (8s a:))
Further details about the hyperparameters are provided in the Appendix D. Moreover, we notice that for all the algo- rithms in the higher-dimensionn and thus more challenging environments (Ant-v5 and Humanoid-v5), the advantage of the SOAR exploration technique becomes more evident.
5: std-Q(ss; (:) f- Clip((tdd-Q(i 0.).
6: a(s) = Q(sss 0;) std-Q 8/ ai)
7: Return: ((si. a,) for all I....,N.
mators (critics) {Qe)ém trained in the same way (mini- mizing the squared Bellman error in standard SAC) on different dataset collected by the same actor. That is, on independent identically distributed datasets. For complete- ness, the SAC critic training is included in Algorithm 7 in Appendix F. In the continuous setting, it is clearly not pos- sible to compute the optimistic state-action value at every state-action pair. Thankfully, it suffices to compute the op- timistic state action value function Q, invoking the routine OPTIMISTICQ-NN, only for the state-actions in mini- batch D = {ss, a, ) Indeed, the policy network weights does not require perfect knowledge of Q over S x A but only an Adam (Kingma& & Ba, 2015) update step on the loss log (a; + Q(s;, a.)).
6.1. Experiment on a hard exploration task
An anonymous reviewer pointed out that the MuJoCo bench- mark not the hardest for what concern exploration. This is very valid suggestion that we address here. This will also allow to understand better the role of the number of critics L. Therefore, to highlight even more the importance of exploration especially in imitation learning from states only we run SOAR-IL in the worst case construction used in the lower bound for the number of environment interaction in (Moulin al., 2025, Theorem 19).
Cumulative Reward
72
L=1
C =
L=
In the next section, we show that for multiple choices of UPDATECOST (ML-IRL, CSIL and RKL) replacing the standard SAC critic update routine with OPTIMISTICQ-NN leads to improved performance.
64
L=3
L= 5
L= 10
56
L= 50
L= 100
48
Expert
6. Experiments
We perform experiments for both state only and state ac- tion IL on the following MuJoCo (Todorov et al., 2012) environments: Ant, Hopper, Walker2d, and Humanoid.
1.2 10
MDP Samples
For the state-only IL setting, showcase the improvement on RKL (Ni et al., 2021) and ML-IRL (State-Only) (Zeng et al., 2022). In both cases, we found that using L = 4 critic networks and an appropriately chosen value for the standard deviation clipping threshold a consistently improves upon the baseline. In the Appendix D, we conduct an ablation study for L and a.
Figure 4: Ablation for L on hard exploration task. State only imitation experiment a hard exploration environment (used in the lower bound from (Moulin et al., 2025, Theorem 19)). Results averaged over 5 seeds, for dataset of 100 states sampled from the expert occupancy measure.
We can see that with only network, the mean of the learner does not reach the expert performance and the variance is very high meaning that some seeds are successful and others fail. This is in perfect agreement with Corollary 4.1 which predicts that for low values of the number of critics L, the optimistic properties of the critic estimators can not be guar- anteed with high probability. For 2, the environment
We denote derived algorithms RKL+SOAR and ML- IRL+SOAR. In addition to observing improvement over standard RKL and ML-IRL, we outperform the state-only version of the recently introduced OPT-AIL algorithm (Xu et al., 2024) (see Figure 2) which incorporates an alternative, more complicated, deep exploration technique.
8

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Ant-vs
HopperyV
Wallierddiv55
Humanoid vs
1.0
0.0
0.0
Episode
166
Eplsode
Episode
Episode
Expert
ML-IL SOAR (Duns)
[Ouns]
Figure 2: Experiments from State-Only Expert Trajectories. 16 expert trajectories, average 5 seeds, 4 Clipping values
ML-IRL: |Ant 10.0, Hopper: 50.0, Walker2d: 0.5, Humanoid: 5.0), rkl: [Ant: 0.8, Hopper: 50.0, Walker2d: 30.0, Humanoid: 100.0]
Ant-u5
Hopper-vS
Walkerrd-v5
Humannod--45
1.0
0.6
0.0
Episode
Episode
Episode
Episode
gei sqii
SONR (Uurs)
KL-RRL (SA) SUAR (Curs)
IMPL SONR (Durs)
Expert
Figure 3: Experiments from State-Action Expert Trajectories. expert trajectories, average over 5 seeds, L 4. Clipping values CSIL: [Ant: 10.0, Hopper: 5.0, Walker2d: 0.5, Humanoid: 0.1], ML-IRL(SA): [Ant: 5.0, Hopper: 10.0, Walker2d: 0.5, Humanoid: 50.0]
solved successfully albeit with higher variance than the case L = 3.
networks. Similarly, the recent work (Xu et al., 2024) uses exploration technique in Deep IL but requires solving a com- plicated non-concave maximization problem. Our approach is remarkably easier to implement. achieves convincing empirical results results and enjoys theoretical guarantees.
Increasing further leads to worst results in terms of MDP samples needed solve the task. This is because according Lemma 4.13 the upper bound on the expected on policy temporal difference large L should be avoided. We used 0.5, 4, and we scaled the standard deviation bonus by 0.001.
scales with I. so excessively
Moreover, framework be applied to and is expected to be beneficial for other existing IL. algorithms based on SAC (such as AdRIL (Swamy et al., 2021) and SMILING (Wu et al., 2024)). Moreover, the same hope for any future deep IL algorithm using SAC for policy updates.
7. Conclusions and Open Questions
While there has been interest in developing heuristically effective exploration techniques in deep RL, the same is not true for deep IL. For example, even in the detailed study What matters in Adversarial Imitation Learning ? (Orsini et al., 2021) the effectiveness of deep exploration techniques is not investigated.
Open Questions On the theoretical side, we plan to ana- lyze the ensemble exploration technique with function ap- proximation under more general structural assumption on the environment. From the practical one, very relevant question is investigate if the exploration enhanced ver- sions of DQN (Osband et al., 2016a; 2018) can speed up imitation learning from visual input. Finally, the same idea might find application in the LLM finetuning given the re- cently highlighted potential of IL for this task (Wulfmeier et al., 2024; Foster et al., 2024).
Prior to our work, only few studied the benefits of explo- ration in imitation learning, mostly in the state-only regime (Kidambi al., 2021). However, their theoretical algo- rithm uses bonuses that cannot be implemented with neural

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Acknowledgments
Cao, Z. and Sadigh, D. Learning from imperfect demonstra- tions from agents with varying dynamics. IEEF Robotics and Automation Letters, 6(3):5231-52233 2021.
This work funded (in part) through a PhD fellowship of the Swiss Data Science Center, a joint venture between EPFL and ETH Zurich. This work was supported by Hasler Foundation Program: Hasler Responsible AI (project num- ber 21043). Research was sponsored by the Army Re- search Office and was accomplished under Grant Number This work was supported by the Swiss National Science Foundation (SNSF) under grant number 200021 205011.
Cassel, A., Levy, O., and Mansour, Y. Batch ensemble for variance dependent regret in stochastic bandits. arXiv preprini 24kiv::200.08570, 2024.
Cesa-Bianchi, N., Gentile, C., Lugosi, G., and Neu, G. Boltzmann exploration done right. Advances in neural information processing systems, 30, 2017.
Chang, J. D., Sreenivas, D., Huang, Y., Brantley, K., and Sun, W. Adversarial imitation learning via boosting. arXiv preprint WxXiv:2404005111 2024.
Impact Statement
The current submission is expected to have an impact in the imitation learning community particularly because it highlights the benefits of exploration, not only theoretically but also in simulated robotics experiments. Impact can be expected also in the broader machine learning community and in related disciplines such as roboties and control the- ory. Beyond that, we do not expect direct impact the society (i.e. outside the machine learning community and IT industry). ).
Chang, W.-D., Fujimoto, S., Meger, D., and Dudek, G. Im- itation learning from observation through optimal trans- port. arXiv preprint 2r.11:2310.01632, 2023.
Chen, R. Y., Sidor, S., Abbeel, P., and Schulman, J. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502, 2017.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials using proba- bilistic dynamics models. Advances in neural information processing systems, 31, 2018.
References
Abbasi-Yadee Yadkori, Y., Pál, D., and Szepesvári, C. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems (NeurIPS), 2011.
Ciosek, K., Vuong, Q., Loftin, R., and Hofmann, K. Bet- ter exploration with optimistic actor critic. Advances in Neural Information Processing Systems, 32, 2019.
Abbeel, P. and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning (ICML), 2004.
Curi, S., Berkenkamp, F., and Krause. A. Efficient model- based reinforcement learning through optimistic policy search and planning. Advances in Neural Information 2020.
Anschel, O., Baram, N., and Shimkin, N. Averaged-dqn: Variance reduction and stabilization for deep reinforce- ment learning. In International conference on machine learning, pp. 176-185, PMLR, 2017.
Processing Systems,
Dadashi, R., Hussenot, L., Geist, M., and Pietquin, O. Pri- mal Wasserstein imitation learning. In International Con- ference on Learning Representations (ICLR), 2021.
Arora, S., Du, S., Kakade, S., Luo, Y., and Saunshi, N. Provable representation learning for imitation learning via bi-level optimization. In International Conference on Machine Learning, pp. 367-376. PMLR, 2020.
Edwards, A., Sahni, H., Schroecker, Y., and Isbell, C. Imi- tating latent policies from observation. In International conference on machine learning, pp. 1755-1763. PMLR, 2019.
Auer, P.. Cesa- DasacBBinnch,,, N., Freund, Y., and Schapire, R. E. Gambling in a rigged casino: The adversarial multi-armed bandit problem. In Proceedings of IEEE 36th annual foundations of computer science, pp. 322-331. IEFF, 1995.
Foster, D. J., Block, A., and Misra, D. behavior cloning all you need? understanding horizon in imitation learning. arXiv preprint arXiv:2407. 15007, 2024.
Fu, J., Luo, K., and Levine, S. Learning robust rewards with adverserial inverse reinforcement learning. In Interna- tional Conference on Learning Representations (ICLR), 2018.
Beck, A. and Teboulle, M. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3):167-175, 2003.
Fujimoto, S., Hoof, H., and Meger, D. Addressing function approximation error actor-critic methods. In Interna- tional conference on machine learning, pp. 1587-1599 PMLR, 2018
Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T., and Efros, A. A. Lcrge-scale study of curiosity-drin driven learning. arXiv preprint arXiv: /808,00080000 00355, 2018.
10

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Ganai, M., Hirayama, C., Chang, Y.-.., and Gao, S. Learn ing stabilization control from observations by learning lyapunov-like proxy models. In 2023 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 11.2993-2222 IEEE, 2023.
Ishfaq, H., Wang, G., Islam, S. N., and Precup, D. Langevin soft actor-critie: Efficient exploration through uncertainty- driven critic learning. arXiv preprint 27X11:2501.17827, 2025.
Kakade, S. A natural policy gradient. Advances in Neural Information Processing Systems (NeurIPS), 2001.
Gangwani, T. and Peng, J. with transition dynamics mismatch. arXiv preprint arXiv:2002.1/879, 2020.
State-only imitation
Kamoutsi, A., Banjac, G., and Lygeros, J. Efficient per- formance bounds for primal-dual reinforcement learning from demonstrations. In International Conference on Machine Learning (ICML), 2021.
Gangwani, T., Zhou, Y., and Peng, J. Imitation learning from observations under transition model disparity, arXiv preprint arXiv:2204. 1146, 2022.
Kidambi, R., Chang, J., and Sun, W. Mobile: Model-based imitation learning from observation alone. Advances in Neural Information Processing Systems, 34:28598- 28611, 1, 2021.
Garg, D., Chakraborty, S., Cundy, C., Song, J., and Er- mon, IQ-learn: Inverse soft-Q learning for imitation. In Advances in Neural Information Processing Systems (NeuRIPS), 2021.
Kim. G.-H., Lee, J.. Jang, Y.. Yang. H., and Kim, K.-E. Lob- sdice: Offline learning from observation via stationary distribution correction estimation. Advances in Neural Information Processing Systems, 35:8252- 2282222-2264, 2022a.
Gupta, A., Devin, C., Liu, Y., Abbeel, P., and Levine, S. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv:100..0294, 2017.
Kim, G.-H., Seo, S., Lee, J., Jeon, W., Hwang, H., Yang, H., and Kim, K.-E. Demodice: Offline imitation learn- ing with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022b.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with stochastic actor. In International Conference on Machine Learning (ICML), 2018
Kingma, D. P. and Ba, Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Henaff, M., Raileanu, R., Jiang, M., and Rocktäschel, T. Exploration via elliptical episodic bonuses. Advances in Neural Information Processing Systems, 35:37631- 37646, 2022.
Kostrikov, I,, Agrawal, K. K., Dwibedi, D., Levine, S., and Addressing sam- ple inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Rep- resentations (ICLR), 2019.
Tompson, Discriminator-actoo--iiii
Ho, J. and Ermon, S. Generative adversarial imitation learn- ing. In Advances in Neural Information Processing Sys- tems (NeurIPS), 2016.
Ho, J., Gupta, J. K., and Ermon, S. Model-free imitation learning with policy optimization. In International Con- ference on Machine Learning (ICML), 2016.
Kurutach, T., Clavera, I., Duan, Y., Tamar, A., and Abbeel, P. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 1802. 10592, 2018.
Ishfaq, H., Cui, Q., Nguyen, V., Ayoub, A., Yang. Z., Wang, Z., Precup, D., and Yang, L. Randomized exploration in reinforcement learning with general value function approximation. In International Conference on Machine Learning, pp. 4607 4607-4616 4616. PMLR, 2021.
Lattimore, T. and Szepesvári, C. Bandit algorithms. Cam- bridge University Press, 2020.
Liu, F., Ling, Z., Mu, T., and Su, H. State alignment-based imitation learning. arXiv preprint 19X1::911.119941 2019.
Ishfaq, H., Lan, Q. Xu, P., Mahmood, A. R., Precup, D., Anandkumar, A., and Azizzadenesheli, K. Prov- able and practical: Efficient exploration in reinforce- ment learning via langevin monte carlo. arXiv preprint arXiv:2305. 10.305..82466. 18246, 2023.
Luis, C. E., Bottero, A. G., Vinogradska, J., Berkenkamp, F., and Peters, J. Model-base uncertainty in value functions. In International Conference on Artificial Intelligence and Statistics, pp. 8029-80008088 8052. PMLR, 2023.
Ishfaq, H., Tan, Y., Yang, Y., Lan, Q., Lu, J., Mahmood, A. R., Precup, D., and Xu, More efficient randomized exploration for reinforcement learning via approximate sampling. arXiv preprint arXiv:2406. 12241, 2024.
Lyu, J.. Ma. X., Yan, J., and Li, X. Efficient continuous control with double actors and regularized critics. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 7655-7663, 2022.

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Ma, Y., Shen, A., Jayaraman, D., and Bastani, O. Versatile offline imitation from observations and examples via reg- ularized state-occupancy matching. In International Con- ference on Machine Learning, pp. 14639-14663, PMLR, 2022.
Osband, I. and Van Roy, B. Near-optimal reinforcement learning in factored mdps. Advances in Neural Informa- tion Processing Systems, 27, 2014.
Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016a.
Mnih, V., Kavukeuoglu, K., Silver, D., Rusu, A. A., Ve-
ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovkki, G., Petersen, S., Beattie, C.,, Sadik, A., Antonoglou, I., King, H.,, Kumaran, D., Wier- stra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, 2015.
Osband, I,, Van Roy, B., and Wen, Z. Generalization and exploration via randomized value functions. In Interna- tional Conference Machine Learming, pp. 2377-2386. PMLR, 2016h.
Osband, I., Aslanides, J., and Cassirer, A. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Moskovitz, T., Parker-Holder, J., Pacchiano, A., Arbel, M.,, and Jordan, M. Tactical optimism and pessimism for deep reinforcement learning. Advances in Neural Information Processing Systems,
Osband, I., Wen, Z., Asghari, S. M., Dwaracherla, V.,
12863, 2021.
Ibrahimi, M., Lu, X., and Van Roy, B. Approximate thompson sampling via epistemic neural networks. In Un- certainty in Artificial Intelligence, pp. 1586-1595, PMLR, 2023a.
Moulin, A. and Neu, G. Optimistic planning by regularized dynamic programming. arXiv preprint arXiv:2302. 14004, 2023.
Moulin, A., Neu, G., and Viano, L. Optimistically opti- mistic exploration for provably efficient infinite-horizon reinforcement and imitation learning. arXiv preprint arXiv:2502. 13900, 2025.
Osband, L. Wen, Z., Asghari, S. M., Dwaracherla, V., Ibrahimi, M., Lu, X., and Van Roy, B. Epistemic neural networks. Advances in Neural Information Processing Systems, 36:2795-288 2023b.
Nair, A., Chen, D., Agrawal, P., Isola, P., Abbeel, P., Malik, J., and Levine, S. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE international conference on robotics and automa- fion (ICRA), pp- 2156-2153. IERR, 2017.
O' O'Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty bellman equation and exploration. In Interna- tional conference on machine learning. pp. 3836-3844 2018
Parder-Holder, J., Pacchiano, A., Choromanski, K. M., and Roberts, S. J. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33: 13:18850-1806., 2020.
Nemirovskij. A. S. and Yudin, B. Problem complexity and method efficiency in optimization. 1983.
Ni, T., Sikchi, H., Wang, Y., Gupta, T., Lee, L., and Eysen- bach, B. f-irl: Inverse reinforcement learning via state marginal matching. In Conference on Robot Learning, pp. 529-551. PMLR, 2021.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised predic- tion. In International conference on machine learning. pp. 2778-2787. PMLR, 2017.
O' Donooguue, B. Variational bayesian reinforcement learn- ing with regret bounds. Advances in Neural Information Processing Systems, 01222088-8222. 2021.
Pathak, D., Mahmoudich, P., Luo, G., Agrawal, P., Chen, D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., and Darrell, T. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 2050-2053, 2018.
O' Donoghue, B. Efficient exploration via epistemic- risk-secking policy optimization. arXiv preprint arXiv:2302.00933, 093330...999....... 2023.
Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1st edition, 1994.
Orabona, F. A modern introduction to online learning, 2023.
Orsini, M., Raichuk, A., Hussenot, I.,, Vincent, D., Dadashi, R., Girgin, S., Geist, M., Bachem, O., Pictquin, O., and Andrychowicz, M. What matters for adversarial imitation learning? Advances in Neural Information Processing Systems, 34:1 1656-14666666 2021.
Qin, A.. Gao, F., Li, Q., Zhu, S.-.,, and Xie, S. Learn- ing non-markovian decision-man -akinng from state-only se- quences. Advances in Neural Information Processing Systems, 36, 2024.
12

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Radosavovic, I., Wang, X., Pinto, L.., and Malik, J. State- only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 7865-7871. IEEE, 2021.
networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and automa- tion (ICRA), pp. 1134-1141. IEEE, 2018.
Shani, L., Zahavy, T., and Mannor, S. Online apprenticeship learning. arXiv:2102.06924 2021.
Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. baslle- Reliable rein- forcement learning implementations. Journal of Machine Learning Research, 22(268):1 2021. URL http: //jmlr. org/ hvtp:///////////// 2001222222116222
Sikchi, H., Saran, A., Goo, W., and Niekum, S. A ranking game for imitation learning. arXiv preprint arXiv:2202.03448 2022.
Rajaraman, N., Yang. L.., Jiao, J., and Ramchandran, K. Toward the fundamental limits of imitation learning. Ad- vances in Neural Information Processing Systems, 33: 2914-2924, 2020.
Sikchi, H., Chuck, C., Zhang, A., and Niekum, S. A dual approach to imitation learning from observations with offline datasets. arXiv preprint 2xXiv:2406.088005 2024.
Sun, W., Gordon, G. Boots, B., and Bagnell, J. A. Dual policy iteration, 2019a.
Rajaraman, N., Han, Y., Yang, L., Liu, J., Jiao, J., and Ramchandran, K. On the value of interaction and function approximation in imitation learning. Advances in Neural Information Processing Systems, 34:1325-1 1336, 2021a.
Sun, W., Vemula, A., Boots, B., and Bagnell, D. Provably efficient imitation learning from observation alone. In International conference on machine learning, pp. 6036- 6045. PMLR, 2019b.
Rajaraman, N., Han, Y., Yang. L. F., Ramchandran, K., and Jiao, J. Provably breaking the quadratic error compound- ing barrier in imitation learning, optimally, 2021b. URL ht https:
Swamy, G., Choudhury, S., Bagnell, J. A., and Wu, S. Of moments and matching: A game-theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pp. 10022-10032. PMLR, 2021.
//tr///axivvvii org/ abs/ /2102222222 12948.
Reddy, S., Dragan, A. D., and Levine, S. SQIL: imitation learning via regularized behavioral cloning. 1rXiv:190..11000 2019a.
Swamy, G., Rajaraman, N., Peng, M., Choudhury, S., Bag- nell, J., Wu, S. Z., Jiao, J., and Ramchandran, K. Minimax optimal online imitation learning via replay estimation. Advances in Neural Information Processing Systems, 35:
Reddy, S., Dragan, A. D., and Levine, S. SQIL: imita- tion learning via regularized behavioral cloning. CoRR, abs/1905.1 1 108, 2019b. URL abs/1905.
7077-7088, 2022.
Ren, J., Swamy, G., Wu, Z. S., Bagnell, J. A., and Choud- hury, S. Hybrid inverse reinforcement learning. arXiv 2024.
Swamy, G., Wu, D., Choudhury, S., Bagnell, D., and Wu, S. Inverse reinforcement learning without reinforcement learning. In International Conference Machine Learn- ing, pp. -33399.3331..... PMLR, 2023.
preprint arXiv:2402.0888
Ross, S. and Bagnell, D. Efficient reductions for imita- tion learning. In International Conference on Artificial Intelligence and Statisties (AISTATS), 2010.
Syed, U. and Schapire, R. E. A game-theoretie approach apprenticeship learning. In Advances in Neural Informa- tion Processing Systems (NeurIPS), 2007.
Ross, S.. Gordon, G., and Bagnell, D. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
Tarbouriech, J., Lattimore, T., and O' O'Donoghue, B. Prob- abilistic inference in reinforcement learning done right. Advances in Neural Information Processing Systems, 36, 2024.
Samuelson, P. A. How deviant can you be? Journal of the American Statistical Association, 132224)155 1968.
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics en- gine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 5026-5033. 2012.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv: 1707.0677766377 2017.
Torabi, F., Warnell, G., and Stone, Behavioral cloning from observation. arXiv preprint 1805 2018a.
Sermanet, P.. Lynch, C., Chebotar, Y., Hsu, J., Jang, E.,
Schaal, S., Levine, S., and Brain, Time-contrastive
13

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Torabi, F., Warnell, G., and Stone, P. Generative ad- versarial imitation from observation. arXiv preprint 1007:0807.665555 2018b.
Conference on Neural Information Processing Systems, 2024.
Xu, T., Li, Z., Yu, Y., and Luo, Z.-Q. Provably efficient adversarial imitation learning with unknown transitions. arXiv preprint arXiv:2306. 06563, 2023.
Torabi, F., Warnell, G., and Stone, P. Recent advances in imitation learning from observation. arXiv preprint arXiv: 1905. 13566, 2019.
Xu, T., Zhang. Z., Chen, R., Sun, Y., and Yu, Y. Prov- ably and practically efficient adversarial imitation learn- ing with general function approximation. arXiv preprint arXiv:2411.00611 2024.
van Hasselt, H., Guez, A., and Silver, D. Deep rein- forcement learning with double q-learning, 2015. URL https: /apps://arxiie /ttp/v/rxvvvv.////e//eeeee.ae org/ abs / 15050..0 66461.
Viano, L., Huang, Y.-T., Kamalaruban, P., Weller, A., and Cevher, V. Robust inverse reinforcement learning un- der transition dynamics mismatch. Advances in Neural Information Processing Systems, 34:25917-25931 2021.
Yang, C., Ma, X., Huang, W., Sun, F., Liu, H., Huang, J., and Gan, C. Imitation learning from observations by minimizing inverse dynamies disagreement. Advances in neural information processing systems, 32, 2019.
Viano, L., Huang, Y.-T., Kamalaruban, P., Innes, C., Ra- mamoorthy, S., and Weller, A. Robust learning from observation with model misspecification, arXiv preprint arXiv:2202.06003, 2022a.
Yu, L., Yu, T., Song, J., Neiswanger, W., and Ermon, S. Of- fline imitation learning with suboptimal demonstrations via relaxed distribution matching. In Proceedings of the AAAI conference on artificial intelligence, volume 37. pp. 1 11016-11024, 2023.
Viano, L., Kamoutsi, A., Neu, G., Krawczuk, I., and Cevher, V. Proximal point imitation learning. Advances in Neu- ral Information Processing Systems, 35:24309-24326, 2022b.
Yu, X., Lyu, Y., and Tsang, I. Intrinsic reward driven imita- tion learning via generative model. In International con- ference on machine learning, pp. 10925 10935. PMLR, 2020.
Viano, L., Skoulakis, S., and Cevher, V. Imitation learning in discounted linear mdps without exploration assump- tions. In Forty-first International Conference on Machine Learning, 2024.
Zeng, S., Li, C., Garcia, A., and Hong, M. Maximum- likelihood inverse reinforcement learning with finite-time guarantees. Advances in Neural Information Processing Systems, 35:10122-1011 2022.
Vieillard, N., Pietquin, O., and Geist, M. Munchausen reinforcement learning. Advances in Neural Information Processing Systems, 33:4235 4246, 2020.
Zhang, H., Bai, F., Xiao, C., Gao, C., Xu, B., and Müller, M. 3-dqn: Improving deep q-learning by evolving the behavior. arXiv preprint arXiv:2501. 13, 2025.
Warmuth, M. K., Jagota, A. K., et al. Continuous and discrete-time nonlinear gradient descent: Relative loss bounds and convergence. In Electronic proceedings of the 5th International Symposium on Artificial Intelligence and Mathematics, volume 326. Citeseer, 1997.
Zhou, Q., Li, H., and Wang. J. Deep model-based reinforce- ment learning via estimated uncertainty and conservative policy optimization. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 34, pp. 6941 194116948, 2020.
Watson, J., Huang, S., and Heess, N. Coherent soft imitation learning. Advances in Neural Information Processing Systems, 36:14540-14583, 2023.
Zhu, Z., Lin, K., Dai, B., and Zhou, J. Off-policy imi- tation learning from observations. Advances in neural information processing systems, 33: 1240--121, 2020.
Wiklund, T. Another look at binomial and related distri- butions exceeding values close to their centre. arXiv preprint arXiv:2308.0544 05435, 2023.
Zinkevich, M. Online convex programming and generalized infinitesimal gradient ascent. In Fawcett, T. and Mishra, N. (eds.), Machine Learning, Proceedings of the Twenti- eth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pp. 928-939. AAAI Press, 2003.
Wu, R., Chen, Y., Swamy, G., Brantley, K., and Sun, W. Diffusing states and matching scores: A new framework for imitation learning. arXiv preprint arXiv:2410. 13855, 2024.
Wulfmeier, M., Bloesch, M., Vieillard, N., Ahuja, A., Born- schein, J., Huang, S., Sokolov, A., Barnes, M., Desjardins, G., Bewley, A., et al. Imitating language via scalable in- verse reinforcement learning. In The Tiiht Annual
14

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
A. Related Works
IL Theory The first theoretical guarantees obtained for the imitation learning problem dates back to the work of (Abbeel & Ng, 2004) and (Syed & Schapire, 2007) which notably used the idea of no-regret learning. However, their work requires either knowledge of the environment transitions or they require a suboptimal in the precision parameter E amount of expert trajectories to estimate those. Our theoretical guarantees are in the same setting of previous works like (Shani al., 2021) and (Xu et al., 2023) which do not require knowledge of the environment transitions priori but assumes online trajectory access to the environment. The main difference that their work focuses on the easier finite horizon setting. Additionally, their exploration techniques only applies to the tabular setting. Indeed, in the MuJoCo experiments in (Shani et al., 20,1), the authors do not attempt to implement the exploration mechanism required for their theoretical guarantees. In a similar way, also (Xu et al., 2023) can not be implemented beyond the tabular setting because it relies on reward free procedure requiring bonuses proportional to the number of visits to each state action pair. (Ren et al., 2024) suggest an algorithm which does not require exploration but it can not improve upon behavioural cloning in terms of expert trajectories. (Rajaraman et al., 2020; 2021b; Foster et al., 2024) analyze instead offline imitation learning (behavioural cloning) where no additional interaction with the environment is allowed. This setting is more general but it comes at the cost of additional assumptions such as policy realizability or worst depedence on the horizon on the required number of trajectories. (Foster et al., 2024) presents an analysis for general policy classes but they require a maximum likelihood oracle which can not be implemented exactly when using neural network function approximation.
There has been also a variety of studies tackling the problem of computationally efficient algorithm with linear function approximation such as (Kamoutsi et al., 2021; Viano et al., 2022b; 2024; Rajaraman et al., 2021a; Swamy et al., 2022). However, their proof techniques are strictly depending on the linearity of the dynamics therefore the experiments in continuous control tasks require changes in the algorithmic design. Albeit our guarantees are restricted to the tabular setting, the algorithm can be implemented with no modifications with neural networks.
Several works focus on the setting where expert queries are allowed at any state visited during the MDP interaction (Ross & Bagnell, 2010; Ross al., 201 I; Swamy et al., 2021) or that require a generative model for the algorithm updates (Swamy et al., 2022). Another recent work requires generative model to sample the initial state of the trajectory from the expert occupancy measure (Swamy et al., 2023). Our algorithm requires sampling only trajectories in the MDP therefore it does not leverage the aforementioned generative model assumption. In contrast, the setting of this work matches the most practical one adopted for example in (Ho et al., 2016; Ho & Ermon, 2016; Fu et al., 2018; Reddy et al., 2019a; Dadashi et al., 2021; Watson et al., 2023; Garg et al., 2021; Ni et al., 2021). In this case, the expert policy can not be queried and the learner access only a precollected dataset of expert demonstrations.
Theory for IL from States Only and with general function approximation their work does not use exploration mechanism. However their work requires an additional realizability assumption of the expert value function, it can only learn a difficult to store and deploy non stationary policy and provides suboptimal guarantees on TE in terms of the horizon dependence.
This setting has been firstly studied in (Sun et al., 2019b) in the finite horizon setting
The follow up from (Arora et al., 2020), still requires the realizability of the state value function which is not needed in our work. The work of Kidambi et al. (2021) uses the idea of exploration in state only finite horizon imitation learning. Their analysis for tabular MDP gives bound on K which has a worst horizon dependence and it requires the design of exploration bonuses tight to the structural properties of the MDP. Therefore, their NN experiments requires an empirical approximation of such bonuses while the SOAR framework applies naturally.
Wu et al. (2024) imposes expert score function realizability and that the expected state norm remains bounded during learning. The algorithm has provable guarantees but it requires expensive RL in the loop routine that we avoid in our work.
Exploration Techniques in Deep RL Ensemble of Q networks has also been used for training stabilization (Anschel et al., 2017). (Zhang et al., 2025) introduces exploration technique based on multiple actors. Ciosek et al. (2019) does not have theoretical guarantees but it uses the idea of constructing an optimistic critic using mean plus standard deviation but only to define an exploratory policy with which collecting data. Our approach instead maintains only one actor policy which is updated with the optimistic Q estimate. (Parker-Holder Holder et al., 2020; Lyu et al., 2022) exploration with ensemble of actors rather than critics. (Kurutach et al., 2018; Chua et al., 2018) uses an ensemble of networks trained to learn the transition model to improve the sample complexity in model based RL. (Henaff' et al., 2022) learns instead an inverse dynamics model
15

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
and via an encoder and decoder model and uses the features output by the encoder to compute elliptical potential bonuses which are standard in linear bandits AAbbasi-Yadkorf et al., 2011), 1). Moskovitz al. (2021) improved TD-3 (Fujimoto et al., 2018) using an ensemble of critics and bandit algorithm to find an aggregation rule balancing well the amount of optimism required by online exploration and pessimism required by off policy algorithms such as TD-3.
In addition, there are several deep RL work that takes a bayesian point of view to the problem, these algorithms often achieve remarkable performance but the algorithm implemented with deep networks requires usually adjustments creating a mismatch compared to the provable algorithms in the tabular case. Among those (Luis et al., 2023; Zhou et al., 2020; O' DDonoguue et al., 2018) use the Bellman equation for the state value function variance to train a network (dubbed U network) that models the uncertainty of the network predicting the Q values. They respectively prove that this trick improves the performances of SAC, PPO (Schulman et al., 2017) and DQN(Mnih et al., 2015). (Curi et al., 2020) uses the model uncertainty estimate in the update of the actor.
Moreover, building on the theoretical analysis of PSRL (Osband & Van Roy, 2014) and RLSVI (Osband et al., 2016b) that show sublinear bayesian regret bound. At any step, these algorithms sample from a posterior distribution either an MDP where to plan or a value function to follow greedly each step. Between one step and the other the posterior is updated given the new data. While the theorical analysis in the above works prescribe a randomization at the value function parameters level, in the deep RL version, dubbed Boostrapped DQN (Osband et al., 2016 the perturbation is performed implicitly maintaining set of Q networks and sampling uniformly at each round according to which network the agent chooses the greedy action. (Chen et al., 2017) improved upon Bootstrapped DQN using an aggregation rule. That is acting greedy with respect to the mean plus standard deviation of the q ensemble. Osband et al. (2018) further builds on this idea adding a differ prior to each network in the ensemble to increase diversity. Finally, Osband et al. (2023a) replaces the uniform sampling in (Osband et al., 2016a) with a learned distribution with an epistemic network (Osband et al., 2023b).
Furthermore, motivated by the bayesian regret bound proven in (O'Donoghue, 2021) in the tabular case and the one in (O' Donoghue, 2023), (Tarbouriech et al., 2024) proves a regret bound in the function approximation setting and showcased convincing performance in the Atari benchmark. Their algorithm requires to know the variance of the cost posterior distribution which is not available in the neural network experiments. Therefore, it is estimated using the standard deviation of an ensemble of cost network. In our work, we use an ensemble of Q networks and not cost networks.
Additionally, (Ishfaq et al., 2021) analyzed ensemble exploration techniques in the general function approximation setting. Their ensemble consists of different critics trained on the same state actions dataset but with rewards perturbed with a gaussian random vector. (Ishfaq et al., 2023; 2024) looked at efficient implementation of Thompson sampling in Deep RL and obtained convincing results in Atari and providing guarantees for linear MDPs and general function approximation respectively. Moreover, (Ishfaq et al., 2025) extended the above results for continuous action spaces. Unfortunately, these methods do not apply directly to imitation learning because they require a fixed reward function.
Exploration techniques in Deep IL As mentioned only few works investigated exploration techniques in Deep II., Apart from the previously mentioned works, (Yu et al., 2020) adopts a model based approach and used exploration bonuses based on prediction error of the next observed state (a.k.a. curiosity driven exploration (Pathak et al., 2017; Burda et al., 2018)). Finally we notice that ensembles have been used in IL theory IL also for goals different to exploration. In particular, (Swamy et al., 2022) partitioned the expert dataset in two subdataset and show that these technique allows for improved expert sample complexity bounds when the expert is deterministic.
State-only imitation learning Torabi et al. (2018b) tackled the problem of imitation learning from states only modifying the discriminator of GAIL (Ho & Ermon, 2016) to take as input state next state pairs instead of state action pairs. Further practical improvements have been proposed in (Zhu et al., 2020) that allows for the use of off-policy data. The works (Yang et al., 2019; Nair et al., 2017; Pathak et al., 2018; Radosavovic et al., 2021) use the idea of an inverse dynamic model while (Edwards et al., 2019; Ganai et al., 2023) develops a practical algorithm aiming at estimating the forward dynamic model. Furthermore, (Torabi et al., 2018a) introduces twist in behavioral cloning using inverse dynamic modelling to make applicable to state only expert datasets. A comprehensive literature review can be found in (Torabi et al., 2019). More recently, features/state only imitation learning has found application in non markovian decision making problems (Qin et al., 2024). Sikchi et al. (2022) introduce an algorithm that takes ad advantage of an offline ranker between trajectories to get strong empirical results in Lf0 setting. Another line of works (Gupta et al., 2017; Sermanet et al., 2018; Liu et al., 2019; Viano et al., 2021; 2022a; Gangwani & Peng, 2020; Cao & Sadigh, 2021; Gangwani et al., 2022) motivate imitation learning from observation alone arguing that the expert providing the demonstrations and the learner acts in slightly different environments.
16

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
In (Kim et al., 2022a; Sikchi al., 2024), the authors proposed convex programming based methods imitate an expert policy from expert state only demonstration and auxiliary arbitrary state action pairs. Several works (Ni et al., 2021; Kim et al., 2022b; Ma et al., 2022; Yu et al., 2023) introduce empirical methods to minimize an f-divergence between expert and learner state occupancy measure. Complementary, (Chang et al., 2023) minimizes the Wasserstein distance between expert and learner state occupancy measure. Their numerical results are convincing but no sample complexity bounds are provided. Convincing results have been obtained also in (Chang et al., 2024) that uses the idea boosting and (Wu et al., 2024) which uses a diffusion models inspired loss to update the cost.
B. Theoretical Analysis
B.1. Upper bounding the policy regret
Corollary B.1. Consider an MDP where max, supp(P(-ls,)) . =2 then for L 2 36 log with probability at least 1 5
[SSIAA
it holds that
8
min (*)* (s.)s A S.EESXA, ke [K].
[E|4]
Proof. Let us fix a state-action-next state triplet s, a, s', a batch index E [L] and an iteration index k [K]. Then, we consider the following stochastic estimator for the probability of transitioning s' from state taking action
(s,
ES..EER (8,,,8-5
(s No +2
E EsadER//a (88=s)
(s) +2
Notice that in above estimators the denominator corresponds to the number of visits of the pair s, a up to the time & E [K] within the batch & E [4]. i.e. (s, a). increased by 2 for technical reasons. At the numerator instead we have the sum of (s, a) indicators functions which equals one when the state following the state action pair 8, a equal to s'. Each of this indicator is random variable distributed according a Bernoulli random variable with mean P(s' [ss a). At this point, we use the technique introduced in (Cassel et al., 2024). In particular, will show that
2 Vs,a, E x A children(s), Ak E [K], 3A E [7.] P Ph (s'1s, a) s (s)) aD]
1
We distinguish 3 cases: N = 0, N/ a) = 1, (/ 2 2. If (/ = O, then, we have that
P/ 0 - P(s s,)).
If (s. 1, we distinguish two cases. P(slls, a) 2 3/4 then
1 Pt a) < < a))
Otherwise, if No = 1, and P(s'|s, a) S 1 then
3
1 (x=a) < P(s"|s, | a)) = = 0 = P(s'ss,
2 1
P
2
3
Finally, for N/ (s, a) > 2.we have that for ((s1(88)221
it holds that
(s,a))
2
1
(='(s, S (ss =1 (s. +2 <
(N(s,a) a) < P((s'ss,
where the last inequality holds for No 2 2. Otherwise, for P(s'|s, < Lemma 2) (adapted from (Wiklund, 2023, Corollary 1) to obtain
[) we can apply (Cassel et al., 2024,
TP (s'(s,)) < P(sll,,a) 2
Σ
1(== (N
2 .
E.
17

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
At this point notice that for any positive vector V [0, (1 4
81, it holds that
P (s' s, a)) vs' ccildren(s) s)
P (s' 2
As' children(s)
S
Pt ((((ss.))V(())
S E P(d)s,)))))))
L''SS
''ES
where the inequality holds because of the following implication between events
& P(d 1s, (8') as' E children(s)
Σ ()(d
E
P(dl)s,a\V(())
s' E'children(s)
8' d'Echildren(s)
Moreover, since the estimation at each state s' is independent we have that
lchildren(s)
P P5 & ((s'ss,a) Vs' E children(s))
II
P
2
E'children(ss)
Then, for any 8, E S x is concluded by the following chain of inequalities
min [
Σ
())
P((')s,a)V(o))
of ('chhidren(ss)
s' Echildron(s)
Σ
())
N
Σ P(5'(s,aV((5) ve 14
s' ('Cchidren(ss
E'chiddren(s)
II RE[A]
1 P
Σ
E
P(6(),.)VV()))
('Chhldrren(s)
(Echildren(s)
II RE[L]
11 P S (( Vs' children(s))
(childrenss)
[oog childrem(e))
II EE[L]
Therefore, choosing L 2
log(1/8)
we ensure that
log
[childrenn)] phildden (63)
P min Pl (v) 56.
[E(T]
For [childrenss) 2, we have that (log chibdremte))
))
Teantdron
< 36, therefore with union bound over the sets S, A and
K] we conclude that for L 36 log [8|AA|K) ) holds that
P
min
P (s, (s.aa A v.s.kESxAxx
K -
[E(T]]
B.2. Policy Regret Decomposition
Theorem 4.2. Policy Regret In a binarized MDP with |S] states and discount factor it holds that with probability 1 35, for any policy n², Regret, K, π²) ) is upper bounded by
+ V [V |A|
log |A| + πK
(1 7)
(1-7) 7)a
(1-7) 72
18

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
and for
losl (1-))(1->)) K
holds that using the update in (Min) or in (Mean-Std) it holds that Regret(K, x*) is upper
bounded by Ö
[IS|"|A|log(1) (og(1/6)
Proof. The theorem is proven with the following regret decomposition in virtue of Lemma 4.5 already presented the main text. Denoting 8t (s, a) & ct (s, + PPV a) g+++ (s, and g (s. a) & Qh+ (s, a) - Q* (s.a)
K
(1 Regret :Kineet=<KTTT) (K;T*)
= Σ k=1
Banat [(.
(s, (s) - x (s))]]
(BTRL)
d (8, - dt (s, 10 (s.a))
(Optimism)
k=
K
k
+
E tta
(s, a)]
Σ aaadi [o (s,
E
(Shift)
k=1
At this point, we bound each term with the following Lemmas, we obtain
V VK |K|SPP1AA |AI
log log/AA |A K
7)) + +
(1 71-
K K|8)² [A] 1og(1/6)
(1 11-7) 7)5
Bound on (BTRL) Then, we continue bounding the first term invoking the following lemma.
Lemma B.2. Let consider the sequence of policies { k=1 generated by Algorithm for all n > 0 then it holds that
(BTRL) < log|Ai
7
Proof.
K (())
[²{)]] = Σ E
x" =s))]
k=1
KEK
E Exndm
Σ
+1))
(2)
LEEK
Then, applying standard regret bound for Be the Regularized Leader (BTRL) holds that for all S
Σ (s) -x-----'0 > (s)) = log [A]
7
REK
Then, plugging into (2) we conclude that
(BTRL) S log [A]
"
Bound on (Shift) We follow the idea from (Moulin & Neu, 2023) controlling this term proving that two consecutive policies will have occupancy measures within a O11)) total variation distance.
K Lemma B.3. For the sequence of policies a } k= generated by Algorithm 2, for all n >> O, it holds that (Shift) S nk
(1-7)3.
19

---

IL-SOAR:
Imitation Learning with Soft Optimistic Actor cRitic
Proof. Let denote Qmax (1 -
K
d=t Q² - pltt)
)*)
(att 1 ((") + Σ
K-1
(0t, d=² ( - -ca aark .24)
77 (K 2)
11Q (K -1)
< Qmax
(1-
(1-9)
In the first inequality, we used Lemma C.4. Finally, noticing that
d" Qt QQtt1) = -000 ,0KK Q²)
-
-0t= $
Qmax
and that
K 0"
K
(Shift)
Σ
d" Q² - -Qu+1)
allows us to conclude the proof summing the two bounds.
Bound on (Optimism)
Lemma B.4. Let us consider an MDP where maXs aeSx, supp(P((|ss, a)) 2. For each E [K], if the Q in
Algorithm 2, are updated according to (Min) or (Mean-Std), the iterates produced by Algorithm 2 satisfy with probability 1 35 that
V K | log(1//)))
(Optimism) zō
Proof. For the optimism term we can observe that using the update of the Q values, we have that
(5, a) = 7(PV - -min P (s, (s)
Therefore, in virtue of Corollary 4.11 with probability
8 it holds that
(sssa) a) 2 0 A s,aeSxA
Therefore, with probability 1-6 8, we have that
K
K
d= (s - d (s.aa) (8, (ss.a))((0(ss
< Σ Σ d= (s.
Jomi 8,8
km
8,0
ppv((s -min sE(T)
min P/
At this point, using Lemma B.5 and a union bound we have that with probability 25, it holds that
K
d=* (s) - d=' d='(ss as] [s)
1= 5,8
K
[8]
[S] |S]
< 7
E saanddt
[V (x)
1)(1 (0011-77) 77 log
(1 716
K
E
(N' + 1)(1 (1) +4
Am
20

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
S log
[ + +1))
K
(N* [)sss)//+++ 1)(1 7)2
(1 (1-7)6
K
2 (N) 1)(1 (1)
E
+4
k=1
At this point, since L is geometrically distributed for every & [K] it holds that L Lmax with probability 1 1 6. Therefore, we can invoke Lemma C.2 to bound =k=1= E bound ensures that with probability 36, we have that
log(K/6)
for all k E [K]
(1-2)
s.andat
(N
2
Another union
(s, a) - d= (s.a)) [6,
(s, a)]
1=
|S] |A| LK2(K +1)
K |S| log
(2L [S] |A| log Lmmx) + 4log (2K/0))
(1 (1-7)6
(2L |S] |A| log mmxx) + 4log (2K/8))
7
V K 2 log(1111 (1/6)
=
where the Ô notation hides logarithmic factors in K, 7, 1,1 and [A].
Now, prove the part of the Theorem that considers the update for Q++1 given in (Mean-Stde Std). Under this update, we have that
(s,0)= =7 pP*().0) max L
V
1 L os) - -
Σ
(s .0
8=1
Then, applying Samuelson's inequality Lemma C.5, we have that
min P
(s,
1 L
Σ p
p&V+(s,
1 Σ
I PKV
1
RE[L]
t=1
moreover, it holds that
min PK 20
[E(2]
Therefore,
min (s) 2 max rE(T)
Σ (t
PRV - Σ
() 0
T.
t=1
which implies
(*ss,a) 27 (p) min V
2 0
EELL)
where the last inequality holds with probability - ö thanks to Corollary 4.11. Therefore, with probability 1 - 6, we have that
K
K
d= (s. d= d' (s,a) (8 a) <
Σ Σ
d (s)
Rm 8.0
k=
8.0
21

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
K
Σ p)y L Σ pAVm((sa))
(s.) - PP/V" +
L
where the last inequality holds removing the maximum. For the first term, we can use Lemma B.5 and continue as the previous proof show that with probability 1
K
I
V K KSS/² || log(110)
(v) (P 2ō
1 1-7
1=1
t=1
So, we are left with bounding the term
K
4
L
2
Ply
Σ
(s)
e=1
p=1
To this end, by Jensen's inequality we have that
K
2
PAyk(s,a)
1 Σ
pk (s;
T,
k=1
8=1
K
-PVV(s,.))
E sanndak
k=
P=1t=1
S
7
E
f=1t=1
5
E
2
k=
t=1
At this point, notice that invoking Lemma B.5, we have that for all é E [L] with probability 1 - 6
(p (s)))))))
2
[8)
[S] | TK +
2
(N L) 1)(1 772 log
+
(1 (1-7)8
K
(N) +1c//-+1)))))--))11 1)(1-7)
3/S|
(IS|IA) T +1)
12
12
(s) 1)(1 7)2 log
+
K2 +
(1 7)
(N* -))
3/S log (1)
(N) 11)(11111111 7)2
Therefore, plugging into the previous display we obtain
Σ
plV
1
Σ
P ())
8'1
22

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
3 log (1) (NV + 1)(1 - 7)2
E
k=1
K
L
3 3|5] log (1) (N a)/ + 1)(1 - 7)3
K
I
Σ
stand
h=1
K
3 L2 log (?)
K
E saanda
(N* + 1)(1 - 7)2
k=1
|S| log (1) 7)2
K
K
E s
(5 a) +1
Finally, bound E
standtt (s using Lemma C.2 under the event I. IImax
log(K/6) (1-3)
which holds with
probability 1 - 5. Thanks to an union bound we have that with probability 1 - 26,
K
2
Σ plV
L
|S| log (1) K
|S] |A| log (mmx + log (2K Lmax/
(1 7)2
K | |A (1g(1/))
Therefore, the proof is concluded also for the case of Q value being updated as in (Mean-Std).
Lemma B.5. With probability 1 - 6, it holds that for all s, a S x A.
|S| (V* - P/ S V (s, 1)(1-10) 7)2 log
|8|||] LK2 +
2 K
+
(1 (1-
ve,k x [K]
(N* a) + 1)(1-7)
In particular, the above statement implies that for all k E [K]
[S]
S LK +1)
2 K
(V* min [E(L)
p) < (N*(s, a)/L+ + 1)(1 (111-7) log
(1 7)6
2
(*, 1)(1 (1
Proof. Let us introduce the value class the possible value functions generated by Algorithm 2. i.e. { {EEIIS Il I - f(s) 2 0 Vs E $). Let us introduce Coov-covering set Ce.. (V) such that for any V E V
there exists V. Cec.r CV.. such that [ vll < Ecov. Therefore let us denote by 5 the element of C... (V) such that
yk FK S ccov. Then, let us consider a generie V E Ceno (V),
(v) (), K...ERR
Σ
((() = (K(8,a)
Σ 5
(V
23

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
1
Σ
V(5') (s,00,a))
()
Σ
(((8)))-((s1) ((()) n1 (c,==,a))
(x, a)
Then, notice that denoting (s, a) the state sample after a the n time the state action pair was visited we have that
(x,a))
Σ ([(ss.))v(ss)) (s.a=3,a) (1 E,
Σ (P((ssa)-V((((s..))) V(Sh(s))))
Applying directly the Azuma Hoeffding inequality is not possible because the number of elements in the sum, i.e. the number of visits N/ (s, a) not a random variable independent on the random variables (8, ) (s,a) (see (Lattimore
Szepesvári, 2020, Exercise 7.1) )
Therefore, we first apply the Azuma Hoeffding inequality for specific & and for a specific value of the visits N (8, a). That is, holds that with probability 8
(s, a) (16)
Σ -(()))))(()1) (0.==za)) S
2(1 7)
8,
Therefore via union bound for & [K] and (s, 10,1,..., K} have that with probability 1 5 it holds that for all k E [K]
No a) log( (K + 1)/8)
Σ
(PV(s,a)- -v(')) 1 (s.a=3,)) $
2(1 7)2
a ER
Therefore, we can conclude that with probability at least - 8 for all k [K]
log (K( (K + 1)/0) 2N (s, a) 7)
(v, a8 (s, &,
Σ
((6')115a==2)) S
Now, by another union bound over Cecon (V), [K], [Z] and S A and denoting
()) = (1 &
Σ
V(s')L
holds that
log (og(K(( + (8) |A| Cec. (V)) L/5)
P -7(s..))PP((ss.))
Vs, S KJ, VEV 21-8
(8) 7)2
Therefore, now let us consider the element v E Ceo.r (V) such that pk- pi
S Ccov. Then, we have that for all
i [4]
(V*(s, - ()) PV) a) P (P-PA)(V Vi)
= (V, PRi* (x,(() + 2ecov
log(K(K + 18 [A] Cen. (/)) 2ccov
No. (8, a) 7)
24

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
[
K(K + +1)
log
+ 2E
No (s 7)2
With Ccov = K-1 we get
S
S ||| LK2(K +1)
2 K
(s,a) - (s
(s log
+
7)6
Then, we can continue as follows
(), pAV*(s,a)
(s)*(s,a)= (s) +2
(s)
|S||A| +1)
(s) V No a)(1 - 7)2 log
(s, +2 +2
11-7)8
K
(s,
2
(N/ a) + +] 7-2 log
18]
||]||| LK2 +1)
2
(/ a) +2
(1-7)6
K
2
[8]
[S] |A] LK?KK +
2
(s, (N) a) + 2)(1 -7)
log
V (N)(s, a) + +) - 7
(1 (1-7)1 738
Finally, rearranging and using that N (8s a) (a.a)
(a.o) 1, obtain that with probability
6, holds
that for all E [L], [K], S SxA A
[S| (v* (s. V (N a)/L+ 1)(1 - 7)2
| LK2 +1)
2
log
+
(1-
K
(N* a) 1)(1 7)
B.3. Upper bound the regret of the reward player
Theorem 4.3, Cost Regret binarized MDP with (S states and discount factor ",, it holds that with probability 25,
- Regret, K; Ctrue) upper bounded by
S IA log (IS) [A]
4 K 1og(115)+ + VV
2 1D mg
Proof. We decompose the regret follows
,,ar -1
Ctrue ct 0,,, d
Ctrue
K
Ctrue ct d==t - -
K
Cirue ct d a
k=1
For the first term, we can invoke standard online gradient descent bound and get
K
2
Cerue
c,t - dm
-d
<
+ nk
d /2
17
25

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
2
+
nk
dm
||
/2
11
2
nK
< +
? 2
4 Therefore choosing 1/ V K. we get
K
Crrue - c² e,,,y d 22VK.
Then, we can handle the remaining two terms. In particular Σ ( (Ctrme - ch, d - is the sum of a martingale difference sequence. Therefore, applying the Azuma- Huum--Hoefdiing inequality it holds that with probability
K
Curue - ch, d= esy, S (og(/)))))
where we used that (crme ct,, dr o, )l < 2
for for all &s E [K]. Finally for the expert concentration term, we have that
Cirue c d dd
d < K
S
A
d
d
Then, any fixed state action pair 8, a with probability - -8/(1S) [Al) by Azuma-Hoefiding inequality holds that
log((S) |A|5-1)
dee (8) (s)
160=00- - (8) S
D
2 [
e'ED
Therefore, by union bound it holds that with probability 8,
[og(|S||A|| 5-1
d 11
S
2 2
Putting together, the bounds on the three terms allow conclude the proof.
C. Technical Lemmas
Lemma C.1. Consider the MDP M = (S,A,P,c,vor)) and two policies ,, : + DA Then consider for any Q E RISIIA and (s)) = (-1s), (+s)).,((ss))) and .v' be respectively the state-action and state value function of the equals
policy n in MDP M. Then, it holds that (1 (1 (vo: Da vxy)
( -C.0-c-pPVx) + Eandn ((1)) - x1(ss))
Proof.
(0' (
Then, using the property of occupancy measure have that (at (1-7) (vo, Vx of the policy n' in the MDP. Then, holds that
where Va is the value function
dr' Q)
(dr Tm Q - -pp)) + +11-)) (vo, + (p'))pPa)
26

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
dr Q - c (11-11 Vo. vo) + (7pTTor (m)) (0,, dd'', Q c - XPV) (1-7) 7) (Vo, v=)) + +(ETTT (1-7)Vo- Vx)
10 = (d' Q - c -Pv" + - 7) V - -v-) + (E
Rearranging and using the definition of aaaH yields the conclusion.
Lemma C.2. Let assume that L Lor all le E [K]. Then, it holds that with probability 1 -
K
E
saa
( +1 I 2L |S] |A| log (K Lmmxx) + 4 4log (2K Limax! (8)
Proof. Let us assume that 1 S Tmmax for all k E [K]. Invoking (?)Lemma (.4]cohen22019leamiing it holds that with probability 8
K
K L*
< 2
N*(si, ot)/L + 4 4log (2K Lmaax (
(
+1
1=
k= t=1
1 (na=staat
2 Σ
+ log (2K T.max/8 N* a)/s
8,0E8 x k=
K th
(s.mmshat)
S 2L Σ ΣΣ
4log (2K L2KLmax/O) Lmax/
(* +1
s,EEx Ak=
L
(s,a=ettat)
2L E ΣE
4log (2K Lmaxx /8)
8,0ESXA L El.
(1/==07 +1
K Σ log D
L
2L
+ 4 4lo (2K mmx
8,0ESxA
k= 1=
$ 2L [S] |A log (K (KLmax) + 4 (2KLmax/8)
where used Lemma C.3 for f(x) x
Lemma C.3. Let an 2 0 and oo) + [0, oo) be increasing function. then
T
T
ΣT
Oe
Σ
ouf faf +
> an)
f(x)dx
1=1
Proof. See (Orabona, 2023) Lemma 4.13.
Lemma C.4. The sequence of policies policy n. Then it holds that { k=1 generated by Algorithm and let df denote the occupancy measure for the -
11, < Qmma
Vk E [K]
(1 71
Proof. By Lemma in (Sun et al., 2019a) holds that
d=
k+1
<
E --9
1
Benght
[1)=+((-<0=- x *++ ++++(11=]11]]
27

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Then, notice that by 1-strong convexity of the KL divergence it holds that
2 souk
1
|| (-|z) - x x (-12)]1
- 2
1
BBnd
(+1x)
(sll )
E
sndtt Σ (a/x)
(1(u((z,)) - log
Σ
(
2
aEA
ned
1 Bedd log
2nd=t
(al)))
Σ
aEA
aEE
!1
k (a/x)(
" woddak
(alx))u(z,aa)
aEA
aEA
where the last inequality follows by Jensen's inequality and convexity - log. Hence, we continue the upper bound as follows
a ||17 x x++
++
7 E
Σ
Qu(=ra)) (** nh*1 (+(x))
max
(+|x) x (+1=))11
2
Which implies, by Jensen's 's inequality and diving both sides by E anddtk (1 (+|x()11)) that
E Bxnd+ [+|x) - ++ (-1x))) S Qmaxx.
L Lemma C.5. Samuelson's inequality Let us consider L scalars {Xi) and denote the sample mean as X
L Ek Xe and the empirical standard deviation as = / EX-((x x)2 then it holds that
8 L < X1 x S 16 A EE
Proof. Let us consider arbitrary vector R., Then, we have that ||o|loo I ||1|1g- At this point let us consider
== [X1 X, X,...,Xy Xz - X).. Moreover, let define as l the index such that |||| Xe- - X Then, we have that for all l E [4].
Xe x < X
x - -
Σ 8=1
(Xe x)= -L- 1o.
Therefore, rewriting the absolute value it holds that
X -L 18 S
X1
X -L-16 16
The next lemma says that the effective horizon in the original MDP and the binarized MDP is equal up to a log, ((S)) factor. Lemma C.6. It holds that
1
loga [S] + 2
<
1 //1/ (0///8/
1-7
Proof.
1
1
1
// logg |S|
1-7
7/1
log,
28

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
10
1
Tin
1
- 7
7bin
1og. +1
1
Thin
7
1=U
loga
|||
2
D. Implementation details
Environment: We use the Hopper-v5, Ant-v5, HalfChhetahh-5,, and Walker2d-v5 environments from OpenAI Gym.
The expert policy is trained using SAC. The training configuration uses 3000 epochs. The agent explores randomly for the first 10 episodes before starting policy learning. A replay buffer of I million experiences is used, with batch size of 100 and a learning rate of Ie-3. The temperature parameter (a) is set to 0.2. The policy updates occur every 50 steps, with I update per interval. After training 64 experts trajectories are collected to be used later for the agent training, The cumulative reward of the expert policies in the different environments are as given in the following table.
Expert Samples:
Table 1: Expert returns
Method
Ant-v5
Hopper-v5
Humanoid-v5
Walker2d-v5
Expert
4061.41
3500.87 1: 4.33
5237.48 1: 414.69
5580.39 :: 20.30
return
= 730.58
IL algorithms implementation: Our starting code base is taken from the repository of E-IRL? (Ni et al., 2021), and the implementation of the other algorithms are based this one. For more details about the implementation please refer repository. The most important hyperparameters reported in Table 2
ML-IRL, f-IRL and rkl: These algorithms were already implemented in the E-RL repository. The method leverages SAC as the underlying reinforcement learning algorithm and different type of objectives for the cost update. The multi-Q-nettork exploration bonus is implemented inside the SAC update, there we keep track of multiple Q-networks and use their mean and standard deviation to update the policy. The clipping is applied on the standard deviation which serves as the exploration bonus.
We started from the f-IRL implementation. maintaining the same hyperparameters for fair comparison. The key modification was removing reward model training from the RL loop, instead training it only once before entering the loop using behavioral cloning and L2 normalization, after which the reward model remained fixed throughout the training.
CSIL:
OPT-AIL (state-only and state-action): We started from the implementation of ML-IRL, and added the OPT-AIL. exploration bonus, incorporating optimism-regullriieed Bellman error minimization for Q-value functions as described in the original article (Xu et al., 2024).
The updated Q-loss can be formulated as:
Cq = (r (((1))((Q(ssd))) a T
log (a')s' (())) XE(Qo(s,
Where: - Qo is the current Q-network - Q3 is the target Q-network r is the learned reward > is the discount factor is the done flag a is the entropy coefficient is the optimism regularization parameter The expectation is taken across the data distribution sampled from the replay buffer, which includes stae-actiio-reward--ne state-done transitions and individual state-action pairs.
hhtpp::///t thub. www.w//w.www...mccm F/F.TRI/tree//aaa
29

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Table 2: Core Hyperparameters Across Environments
Parameter
Walker2d
Humanoid
Hopper
Ant
Number of Iterations
1.5
1M
M
1.2
Reward Network size
[64, 64]
[64. 64]
[64, 64]
[128, 128]
Policy Network size
[256, 256]
[256, 256]
[256, 256]
[256, 256]
Reward Learning Rate
le.4
le.4
le-4
le-4
SAC Learning Rate
le-3
1e-3
le-3
1e-3
6000
5000
Mean Return
3000
Networks=1
Networks=3, clip=5.0
Networks=4, c0ip=10.0
Networks=5, clip=5.0
1000
Networks=10, clpp=50.0
Expert
0.0
0.2
0.4
0.6 Episode
0.8
1.0
1.2
1e6
Mean return of ML-IRL Ant-v5 with different number of neural networks. The grid search for the clipping values was performed over the following values [0.1 0.5 1 10 50). Results are averaged over 3 seeds.
Figure 5:
SQIL (Reddy et al., 2019b): was implemented by initializing replay buffer with expert trajectories and assigning them reward of 1, while collecting additional on-policy trajectories from the agent's current policy with reward of O. During training, the SAC agent learns from both expert and agent-generated transitions, effectively learning to imitate expert behavior through the asymmetric reward structure. The agent updates policy by sampling from this mixed replay buffer, where the expert transitions provide high-reward signal guide the learning process.
GAILs: we used the implementation available from Basoble-Baslllee ines3 (Raffin et al., 2021). This is the only method not based on SAC.
D.1. Hyperparameters tuning
To determine how different numbers of neural networks changed the performance, we conducted an ablation study both the clipping value and the number of neural networks. We performed this analysis for the ML-IRL algorithm on the Ant-V5 environment. We chose this environment since previous experiments showed that its higher complexity led to higher variance the performance of different algorithms. It was necessary to perform grid search on the number of neural networks because we noticed that different numbers neural networks preferred different clipping values.
The results are reported in Figure As we observed, all cases, adding more neural networks leads to better performances. However, this improvement does not increase proportionally with the number of neural networks; in fact, the run with 10 neural networks is outperformed by the one with 4. This led us select 4 as the fixed value of neural networks, also justified by the much slower training time of the 10-network case.
For every environment and algorithm, we performed a grid search over different clipping values. The range clipping values varied across algorithms. Figure shows the different values used in the search and their impact on performance. The difference in performance across clipping values small simpler environments (e.g.. Hopper or Walker2d) while
30

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
becomes evident in more complex environments with larger state-action spaces. These plots also show the necessity of the clipping for the exploration bonus. In most environments and algorithms, when large clipping value is applied, it leads to performance degradation.
Empirically, the Q-network's 's standard deviation diverges due to unclipped Q-values. Without value clipping, high Q-values for specific state-action pairs increase the probability of being visited, causing more of these pairs to accumulate in the replay buffer across different rollouts and potentially amplifying the standard deviation across the q-network the next update. This justifies the necessity of clipping value on the exploration bonus.
31

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
1e3
Hopper
Walker2d
Hopper
1e3
Walker2d
Avarage Reeurn
3.0
Averogo Return
Averrag Return
2.0
N Return
1.5
2
2.5
0.0
0.4
0.8
0.0
1.2
0.0
0.4
0.0
1.2
Episode (x10%)
Episode (x10f)
Episode (x109)
Episode (x109)
1e3
Humenoid
1e3
1e3
Humanoid
163
Ant
5.0
Average Return
Average Return
Average Return
5.0
Average
2.5
2.5
0.0
0.0
0.4
0.5
0.0
Episode (x10%)
Episade (x10*)
Episode (x10f)
Episode (x10%)
clip-0.1
clip=1.0
cllp-10.0
100.0
clip=1000.0
dlp-0.1
clpp-1.0
dip- 10.0
clip-100.0
ciip-1000.0
0.55=0.5
clip=5.0
clip=50.0
clip=500.0
dlp=0055
dlip=5.0
clip= 50.0
clip=500.0
(a) CSIL Comparison
(b) ML-IRI. Comparison
1e3
Hopper
1e3
We ler2o
1c3
Hopper
1n3
Average Return
3.0
Aetroa Retum
Averoage Return
3.0
Average Return
1.5
1.5
0.0
0.4
0.8
0.0
0.6
1.2
Episode (x10%)
Episode (x10%)
Episode (x10%)
Humanoid
1e3
Ant
1.3
1e3
Averaag Return
Averoge Return
Average Return
Average Retum
22
2.5
0.0
0.4
0.8
0.0
0.5
1.0
0.0
0.4
0.8
0.0
0.3
1.0
Episode (x10%)
Episode (x (x109
Episode (x10%)
Episode (x10%)
clip=0.0
ciip=0.8
clip=3.0
dipanan
clip=30.0
dlpp-0.1
clip-1.0
clip-10.0
clip-100.0
clip=1000.0
clip-0.1
clip-1.0
cip-5.0
dip-110.0
clip-50.0
clip=0.5
clip=5.0
cl.g=50.0
clip=500.0
clig=0.5
clip=2.0
diip=20.0
clip-500.0
(c) ML-IRL-SA Comparison
(d) RKL Comparison
Figure 6: Comparison clipping values across different environments, showing the effect on average return for each environment.
32

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
E. Experiments with single expert trajectory
Here we report the we report the result the experiments using a single trajectory. Our findings indicate that the performance remained consistent regardless of the number of trajectories used and the performance are comparable to the ones with 16 trajectories. Notable differences in performance improvement were observed in Humanoid-v5 and ant state environments in the state only settings, where more pronounced gap evident.
Hoppervs
Wolkkerd--v5
1.0
0.8
0.4
0.2
0.50
0.25
0.50
0.75
1.00
1.25
0.0
0.2
0.4
0.6
0.8
1.0 1eb
0.0
0.5
1.0
1.5 leb
0.0
0.2
0.4
0.6
0.8
1.0
Episode
leb
Episode
Episode
Episode
166
ML-IBII
del
nel + SOWR (Ours)
....
Expert
MG-IR SONK (Ours)
Figure 7: Experiments from State-Only Expert Trajectories. expert trajectories, average over 3 seeds, L - 4 Clipping values o
ML-IRL: [Ant: 0.1, Hopper: 0.1, Walker2d: 50.0, Humanoid: 0.5], rkl: [Ant: 0.1, Hopper: 0.5, Walker2d: 1.0, Humanoid: 50.0]
Hopperyv
Wollier2div5
Humanoid
0.8
0.6
0.2
0.0
0.00
0.25
0.50
0.75
1.00
1.25
0.0
0.2
0.6
0.8
1.0
0.0
0.5
1.0
1.5
0.0
0.2
0.4
0.6
0.8
1.0
Episode
16h
Episode
Episode
Episode
MU-ILL (SA)
ML-RL (SA) SOAR (Ours)
Expert
cill
SOAR [Qurs]
Figure 8: Experiments from State-Action Expert Trajectories. expert trajectories, average over seeds, 4. Clipping values
CSIL: [Ant: 0.5, Hopper: 50.0, Walker2d: 0.1, Humanoid: 0.1], ML-IRL(SA): [Ant: 0.1, Hopper: 0.5, Walker2d: 0.1, Humanoid: 1.0]
F. Omitted Pseudocodes
This section introduces the omitted pseudocodes to clarify the implementation of the algorithms based SOAR. We first give pseudocode (see Algorithm 6) that mirrors Algorithm in the setting where deep neural network approximation is needed due to the continuous structure of the state-action space. The critic training is the same in the standard SAC (Haarnoja et al., 2018) but we report it in Algorithm 7 for safe completeness. Notice that we adopt the double critic training originally proposed (van Hasselt et al., 2015) to avoid an excessive underestimation the critics value
'Ntice that (van Hasselt et al., 2015) talks about excessive overestimation of the prediction target in the critic training rather than underestimation. This difference is due to the fact that their paper casts RL. as reward maximization while we adopt cost minimization perspective. For the reason take the maximum between the critics rather than the minimum done in (van Hasselt al., 2015).
33

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Algorithm 6 Base Method SOAR pseudocode
Require: Policy step size 77, step size a, expert dataset D Dre. discount factor y maximum standard deviation parameter
1: Initialize actor network Tyt randomly.
2: Initialize the cost network Cu. randomly.
3: Initialize the L critics {Qot..... :oo randomly.
4: Initialize the L target critics [Qqg.wa..... Qoromn randomly.
5: Initialize empty replay buffers {Dt) ) (One for each critic)
6: for k to K do
7: Ttè <- COLLECTRRAJECTORY// for each e [L].
8: Add to replay buffer D < <D, -
9: Let D = 1
10: Cwt + UPDATECOST(CCU DILY DK a) using the Base Method (such as CSIL, f-IRL or ML-IRL) ).
11: for = to do
Pot+, , Pottron UPDATECRITICS(D). Fppt 7177: Co+)
13:
end for
(Q,+ 9, OPTIMISTICQ-NNID {cot je
1 F-1 TD DR (-1) log uo (als) + Qut (s,
14:
.0)
(see Algorithm 5)
15: Define the loss C
16:
Update policy weights to wh*+1 using Adam (Kingma & Ba, 2015) on the loss C
17:
end for
18:
Return π
Algorithm 7 UPDATECRITICS
Require: D Tyy,, 0r77 7,Cm
1: Let B = {si, ai s4, d be a minibatch sampled from D
2: a + π(s() for all E [N].
3: Define Q Vor (si, ai) f- max (Qot.um.. (85s as), Los am (8s, ai)) for all 8s, as E
4: Backup; + (8i, az) + 7(1 - done;) (Qot (8s, as) + a log Twe T (a/18))
5: fay.00
1. EN (o0g.as (si, a;) Backup;
6: fgr.00
(oog.ess (Si,
Baackp;;
7: 00 ++100))
- (H.(11) 1Q V gp)
8: 8 +1122)
t- 0t. (2) nQ V -0)
9: 8t (1) + (1 Turg joh (or.
+ Tung (1.(1)
10: Ok+ 11,rg, + (1 Tharg (ar.targ.
+ Tame (2.(2)
11: 2gt (8, a) max ( Qofti.ces (8, Q Qottices (8, for all 8, E Dk:
12: Qgtti,om(8, = max (oot 11.ma (8, a), Quf 12 co (8, a)) for all s, E D
return Qoti1; 9o3 11.0x.
E.. Instantiating the cost update
We show after how the algorithmic template in Algorithm captures different imitation learning algorithms just changing the cost update. For example, f-RL with the reversed KL divergence (RKL) can be seen as Algorithm with the cost update described in Algorithm Moreover, our SOAR+RKL is obtained plugging in the cost update in Algorithm in Algorithm 6.
34

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
Algorithm 8 UPDATECOST for RKL (f-IRL, for reversed KL divergence) (Ni al., 2021)
Require: C, D.a pat, a, divergence generating function fx) (z) for the reversed KL divergence, prior distribution over trajectories p(r)-
1: Pw (T)(T) = p((T)e-ou(1 (r((r)
2: X +- arg w Da Dog Dx(s)] + BeNDA [log(1 Dx(s))
3: Estimate the density ratio:
4: (e(s) (+)
Pw(s) (+)
5: Compute the stochastic gradient
T
(sx) (E($t)
)
(-
Σ Vocu((r)) }
T
Br-pe
Σ
h
T
1=1
t=1
E pup
Σ h
(E((S)
Errpo
Σ
T
V Vocw (st)
T
po (s()))
1=1
1=1
6: 2 - vv
7: Return Cw
Next, we present the cost update for the algorithm MI.-IRL (Zeng et al., 2022). We present it for the state-action version. The state-only version is obtained simply omitting the action dependence everywhere.
Algorithm UPDATECOST for ML.-RRI. (State-Action version) (Zeng et al., 2022)
Require: cw, De.D D {0t at) rmm
Le Sample state-action trajectory TE { at 11=1 variable with parameter (1 - -1
from the expert dataset Dng where Le is geometric random
2: Compute the stochastic loss
LR
E
Σ 1=0
r Cu (s)
E a.))
-
1=0
3: W <- a Ln
4: Return Cw.
To conclude, we present the cost update for CSIL. Notice that since the cost used by CSIL does not leverage the information of the policy iteration & we can move the cost update before the main loop and keep a constant cost function fixed during the training of the policy. Notice that the CSIL. the reward is simply given by the log probabilities learned by the behavioural cloning policy. Therefore the reward parameters in CSIL coincides with the parameters of the behavioral cloning policy network. We point out that using reward of this form is similarly done in (Vieillard et al., 2020), We plan to explore further the connection between Munchausen RL and CSIL in future work.
Algorithm 10 UPDATECOST for CSIL (Watson et al., 2023)
Require: DRE.
1: Compute the behavioral cloning policy finding an approximate solution to the following problem.
arg
log (als)
e, mg
2: Return Cw. (8, a) log oo (a|s)
35

---

IL-SOAR: Imitation Learning with Soft Optimistic Actor cRitic
G. Details for the experiment on hard exploration task
The environment used for this experiment coincides with the construction used the lower bound for the number of environment interaction in (Moulin et al., 2025, Theorem 19). This is a simple two states MDP ( a low reward state and high reward state) with 20 actions per state. From the high reward state all actions identical. From the low reward state, all actions are identical but the one chosen by the deterministic expert which has just slightly higher probability to lead to the high reward state from the low reward state. Even observing the expert state occupancy measure perfectly, it is difficult for the learner to find out which is the action which the expert took. That is because all actions are almost identical but one.
36


Reviews:
[{"review_id": "FL2OGzhwGx", "reviewer": "Reviewer_PNta", "summary": "This paper introduces IL-SOAR, an imitation learning (IL) framework designed to bridge the gap between theoretical sample efficiency and practical scalability. The IL-SOAR framework alternates between updating a cost function (critic) and updating the policy (actor).\n\nA key contribution of IL-SOAR is the introduction of an optimistic critic, leveraging multiple critic networks to quantify uncertainty. Specifically, IL-SOAR constructs an optimistic critic that systematically underestimates the true expected cost, thus effectively promoting exploration.\n\nThe authors provide theoretical guarantees for the efficiency of this optimistic critic approach through regret analysis in a tabular setting, explicitly analyzing two aggregation methods for optimism: Min and Mean-Std.\n\nPractically, IL-SOAR extends this principle of optimism to continuous domains by applying the optimistic critic to pre-existing approaches, such as CISL, ML-IRL. Empirical results demonstrate superior performance relative to standard approaches, highlighting IL-SOAR as a theoretically grounded and practically effective online imitation learning method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks a lot for appreciating our work and your careful review!\n\nPlease find the answers to your questions/remarks in the following.\n\n***..experiments using explicitly exploration-critical tasks would strengthen the central claims of the paper.***\n\nIn order to assess the exploration capability of our algorithm we consider state only imitation learning in an \nhard tabular environment which is usually used to demonstrate lower bounds. \nWe refer to [1] section 4 for a detailed description of the environment. We make it slightly harder considering 20 actions (instead of 2 actions as in [1]). Among the actions available to the learner there is one action $a^\\star$ that has slightly higher probability to maintain the learner in the state visited more often by the expert.\nSince, we are in imitation learning from states only setting, the learner can not observe such action $a^\\star$ therefore it has to explore efficiently the action set to estimate $a^\\star$.\n\nWe use this environment for this experiment https://imgur.com/a/HI3LOeq. The results show that exploration is necessary in this environment. Indeed, it is evident that for $L=1$, that is when the algorithm does not have an  exploration mechanism, the learner fails at achieving expert performance while the learner succedes when multiple critics are added.  ( We will comment on the choice of $L$ later in our response)\n\nRegarding continuous control experiments, we are not really sure about which environment could be used to assess the exploration capability of an algorithm. Does the reviewer have some environments in mind ? \n\n[1] Osband & Van Roy 2016, On Lower Bounds for Regret in Reinforcement Learning\n\n***There appears to be a gap between theoretical expectations and practical observations...***\n\nIn figure 4 increasing the number of critics does not improve the performance further but this is not in contradiction with the theory.\nIndeed, notice that in Theorem 3.4 $K$ denotes the number of iterations while the number of critics is denoted by $L$. The performance of the algorithm is not expected to improve in a monotone way at the increase of $L$ because of the following tradeoff:\n\n1) $L$ should be large enough to ensure high enough probability of optimism according to Corollary 4.10 ( see also Lemma B.1)\n2) $L$ should be small enough to guarantee a tight bound on $\\delta^k$ according to Lemma 4.12 where the number of critics L appears at the numerator.\n\nTherefore, it is expected from the theoretical study that an excessively large value of $L$ leads to worst sample complexity guarantees. Ideally, $L$ should be set equal to the smallest value guaranteeing optimism with probability $1-\\delta$. This is the reasoning behind setting $L = 7/2\\log(2 SAK / \\delta)$ in our paper. \nIn order to verify the theory, we add an ablation on the number of $Q$ estimators ( going up to 100 as suggested) in the tabular setting which verifies that there is indeed a tradeoff in the choice of $L$. The image is visible via this anonymous link https://imgur.com/a/HI3LOeq\n\n***Question 1.*** \nFrom the theoretical analysis presented here the two aggregation rules are equivalent. In practice we think that mean+std offers more flexibility in tuning the clipping parameter. Therefore, while equivalent in the worst case the std based aggregation rule seems to achieve better instance dependent results. It would be interesting to confirm this observation theoretically in future work.\n\n***Question 2.***\n\nWe did not try to increase the number of critics above 10 in the continuous states and actions experiments with neural networks. We think this should make the algorithm’s computational cost excessively high compared to a single critic implementation. In the tabular environment considered here https://imgur.com/a/HI3LOeq we try a larger number of critics and see that there is a tradeoff in the choice of $L$ ( please see explanation above). \n\nThanks again for the time spent reviewing our paper, we remain available for further clarification if needed.\n\nBest,\nAuthors", "claims_and_evidence": "**Theoretical Claim:**\n\nIL-SOAR provides a theoretically efficient imitation learning framework.\n\n**Evidence:**\n\nThe authors present a detailed regret analysis for the tabular setting, considering both min and mean-std aggregations. They derive explicit theoretical guarantees, demonstrating that IL-SOAR achieves a regret bound matching established results in terms of $\\epsilon$, the value error between the imitator and the expert. This analysis rigorously supports the theoretical efficiency claim regarding both the number of critics and the number of samples.\n\n**Experimental Claim:**\n\nIL-SOAR demonstrates practical scalability and effectively enhances vanilla methods (which utilize a single critic) in continuous state-action imitation learning tasks under both state-only and state-action settings.\n\n**Evidence:**\n\nThe authors demonstrate empirical results through experiments conducted on several MuJoCo continuous control environments (Ant, Hopper, Walker2d, and Humanoid). They compare IL-SOAR methods (ML-IRL+SOAR, RKL+SOAR, CSIL+SOAR) against their vanilla methods (ML-IRL, RKL, CSIL) and strong baselines (GAIL, SQIL, OPT-AIL), showing consistent and significant improvements across different algorithms and tasks. These empirical results strongly support the practical effectiveness claim.", "methods_and_evaluation_criteria": "### Methods\n\nThe IL-SOAR method employs a simple structure commonly used in inverse reinforcement learning (IRL), alternating between updating the cost function (critic) and updating the policy (actor). A key methodological innovation is the optimistic critic, constructed using multiple critic networks to estimate uncertainty. This critic systematically underestimates the true expected cost, thereby encouraging effective exploration.\n\nThe authors evaluate two explicit methods for aggregating critic outputs to achieve optimism:\n\n1. Min aggregation (selecting the minimum critic estimate) (for a tabular case)\n2. Mean-Std aggregation (using the mean estimate minus the standard deviation across critics) (for both tabular and continuous cases)\n\n### Evaluation Criteria\n\nThe proposed method is empirically evaluated based on episode returns in standard continuous control benchmarks (MuJoCo environments such as Ant, Hopper, Walker2d, and Humanoid). These environments are widely recognized for assessing reinforcement and imitation learning methods, making them appropriate for evaluating practical performance.\n\nHowever, it is unclear whether these benchmarks genuinely require exploration to successfully recover the expert policy. To better highlight the importance of exploration, additional experiments using explicitly exploration-critical tasks would strengthen the central claims of the paper.", "theoretical_claims": "Although I have not verified every detail of the theoretical claims and proofs, they appear rigorous and well-supported, with comprehensive derivations provided in the appendix.", "experimental_designs_or_analyses": "Experimental design seems quite valid. About analyses on this, I remain some comments on a further section.", "relation_to_broader_scientific_literature": "The key contributions of this paper are summarized as follows:\n\n1.\tA theoretical analysis demonstrating that optimism in the face of uncertainty can lead to an efficient imitation learning algorithm.\n2.\tThe introduction of the IL-SOAR framework, which alternates between multiple critic (cost) updates and actor updates using an optimistic critic approach.\n3.\tEmpirical results showing that IL-SOAR improves imitation performance in continuous control domains.\n\nTo the best of my knowledge, these contributions are novel within the imitation learning community.", "essential_references_not_discussed": "I believe the manuscript includes all essential related references.", "other_strengths_and_weaknesses": "### Strengths\n\n- The paper effectively bridges theoretical principle with practical applicability, providing both solid theoretical guarantees and robust empirical results.\n- Experimental results demonstrates significant improvements over vanilla imitation learning approaches.\n- The method is generalizable and easily integrable into existing IL frameworks, providing broader applicability within the imitation learning community.\n\n### Weaknesses\n\n- Although the paper emphasizes exploration in IL, the evaluation benchmarks presented are not particularly exploration-critical tasks, potentially limiting insights into the effectiveness of the proposed exploration strategy in imitation learning. Including experimental results on explicitly exploration-critical benchmarks would significantly enhance the paper’s impact.\n- There appears to be a gap between theoretical expectations and practical observations regarding the relationship between performance and the number of critics. Specifically, Theorem 3.4 indicates that the upper bound of the average regret decreases as the number of critics (K) increases. However, Figure 4, assuming the number of neural networks corresponds to the number of Q-value functions, shows that performance improvement is not directly proportional to an increase in the number of Q-value functions. This finding is counterintuitive, as increased uncertainty estimation accuracy from more Q-value functions might be expected to lead to better performance. Conversely, in a pessimistic approach [1] within the offline RL context, performance consistently and significantly improves as the number of Q-value functions increases, highlighting a notable contrast to the current observation.\n\n[1] An et al., “Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble”, NeurIPS 2021.", "other_comments_or_suggestions": "Typos in the legends of Figures 3 and 7, as well as in Section D, should be corrected: “cisl” should be revised to “csil” (Coherent Soft Imitation Learning).", "questions_for_authors": "**Question 1.** This paper proposes two aggregation methods for Q-values in tabular settings: Min and Mean-Std. Under what conditions or scenarios would each aggregation method be advantageous compared to the other?\n\nAdditionally, why is the mean-std aggregation method particularly considered for OptimisticQ-NN in continuous state-action scenarios, rather than the min aggregation method?\n\n**Question 2.** Why is the performance improvement not directly proportional to the increase in the number of Q-value functions, as shown in Figure 4? Did the authors try the experiment with a more larger scale? (approximately 10~100 Q-value functions)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I4VGPEKrqU", "reviewer": "Reviewer_Kkc5", "summary": "The authors propose an optimistic $Q$-function update for SAC-based inverse RL algorithms. They prove sample complexity benefits in the tabular setting and show an approximation of their idealized algorithm can improve the performance of a suite of prior approaches with minimal increase in complexity.\n\n## Update After Rebuttal\nThe authors added in a discussion of some of the points of confusion I originally had to the paper and added in experiments I specifically requested. I already factored them doing so into my evaluation of the paper, and hence maintain my score", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nthanks a lot for appreciating our work \nand for all the suggestions ! Please find below the responses to your questions.\n\n***Have you thought ... overhead.***\n We agree with the reviewer! We think that interesting next steps are applying the exploration technique based on multiple critics in RL and RLHF. For this paper, we feel that it would be better to keep the message more coincise to imitation learning. We can definitely mention in the future directions this point.\n\n***Why are you adding 2 ...***\nA larger additional weight in the denominator is needed for technical reasons. \nIn particular, to invoke Cassel et al. (2024, Lemma 3) in the proof of Lemma B.1 which in turns applies Corollary 1 (b) in [1]. Intuitively, adding $2$ rather than $1$ in the denominator is needed to ensure that the estimators are more optimistically biased for a small number of visits.\n\n[1] Wiklund 2023,  Another look at binomial and related distributions exceeding values close to their centre\n\n***Given you're running ... interesting.***\n\nThanks for the suggestion. It would be nice to see if optimistic gradient descent + optimistic exploration comes with any benefit.\n\nWe speculate that proving last iterate convergence using optimistic updates for both reward and cost should be possible but it could deteriorate the sample complexity.\nAn alternative approach to get last iterate could be to strongly regularize the problem as in [2] but this comes at the cost of worst rates in $K$.\n\nIn the attempt we did to prove last iterate convergence with optimistic gradient descent ascent the problem is controlling the expected bonus under the last iterate distribution. Indeed, the technique we use to bound the average bonuses in Lemma C.2 does not seem helpful towards this goal.\n\n[2] Müller et al. Truly No-Regret Learning in Constrained MDPs\n\n***If it's ... SOAR has no memory overhead over standard SAC.***\n\nThanks for the suggestion ! We think that at the practical level this uncertainty quantification technique can be used effectively.\nWe are more hesitant in saying whether guarantees can be proven about this technique. We will definitely add a discussion about this technique.\n\n***In Remark 3.1, when you ... clarify this.***\n\nThe reviewer is right. We meant a single pair of critics in Remark 3.1 and we will clarify this in our revision. Thanks a lot for noticing this !\n\n***Response to the section \"Other comments and suggestions\"***\n\nThanks for pointing out the missing references, for suggesting how to improve the abstract and the overall writing (especially for Section 3.1 and for the presentation of the algorithm) ! We are happy to implement these changes and include a discussion about the missing references in our revision.\n\nFinally, thanks for suggesting AdRIL, HyPe and SMILING as possible algorithms to be boosted with SOAR. During the rebuttal time,\nwe run a SOAR boosted version of HyPe and we noticed that it improves over the base version of Hype in Ant, Humanoid and Hopper. The results are available here https://imgur.com/a/2Zse8xl\n\n\nThanks again for the time spent reviewing our paper, we remain available for further clarification if needed.\n\nBest,\nAuthors", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I read all the theorems and skimmed the proofs -- both looked like what I expected to see.", "experimental_designs_or_analyses": "Please include standard error bars in all the plots and specify the number of seeds used to compute the results in Figure 1.", "relation_to_broader_scientific_literature": "SOAR could realistically improve a wide set of prior SAC-based IRL methods, which I find quite interesting!", "essential_references_not_discussed": "I would make sure to cite https://papers.nips.cc/paper_files/paper/2007/hash/ca3ec598002d2e7662e2ef4bdd58278b-Abstract.html as the first paper to introduce the no-regret vs. no-regret flavor of inverse RL. It would also be good to add more of a discussion of https://arxiv.org/abs/2205.15397 as they also use several variants of deep ensembles in imitation learning -- some of the aggregation methods they explore might be interesting for you. Other SAC-based IRL algorithms that would be good to cite / potentially try out SOAR on include AdRIL (https://arxiv.org/abs/2103.03236), HyPE (https://arxiv.org/abs/2402.08848), and SMILING (https://arxiv.org/abs/2410.13855) -- the first and third have fairly unique ways of doing the cost function update, while the second has some interesting ideas re: policy updates. You should also cite https://www.sciencedirect.com/science/article/pii/S002200009791504X to be more transparent about your game-solving strategy for Eq. 1.", "other_strengths_and_weaknesses": "I think the greatest strength of this paper is the broadness of the technique proposed. I think the greatest weakness (which can easily be fixed) is the clarity with which it is presented.", "other_comments_or_suggestions": "There are several parts of this paper where I think the writing could be improved to make things more accessible:\n\n- Define $\\epsilon$ in the abstract or postpone its usage to later. Also, I think you meant to say \"tied\" not \"tight\" in the second paragraph.\n\n- From the way the paper is written, it is not at all clear that you could apply this technique to *all* SAC-based IRL methods. I'd suggest heavily emphasizing this in the abstract / intro to properly convey the general utility of the techniques you developed.\n\n- Treating the value function as a vector and the performance of a policy as an inner product (Line 80)  likely going to confuse a lot of people -- speaking from experience here.\n\n- I think the algorithm writeup is fairly sloppy and hard to follow if you don't already have a strong background in this sort of thing. I might suggest presenting the theory independently first before broadening out and discussing, e.g., the different ways of learning cost functions in practice.\n\n- Section 3.1 mostly reads as a statement of facts rather than a coherent narrative / clear proof sketch -- I strongly recommend rewriting this section to make it easier to follow if you aren't already an expert. I would also cut some of the alternatives (e.g. state-only, the mean - std. ideas) and postpone them to the appendix.\n\n- It would also be good to be more transparent about what you're doing in your description of the algorithms (e.g. I'd explicitly say you're doing NPG in Line 14 of Algorithm 2 and Zinkevich-style OGD in Line 8 of Algorithm 3). It then becomes much more clear that you're running two no-regret players against eachother to compute equilibria -- I would explicitly call this out, perhaps citing the list of \"primal\" algorithms in https://arxiv.org/abs/2103.03236 that follow a similar strategy.\n\n- In Line 250, you could perhaps cite https://dl.acm.org/doi/abs/10.1145/1390156.1390286 as an example of how if you knew the dynamics in closed form, you could collapse the trajectory-level mixture to a single policy.", "questions_for_authors": "1. Have you thought about whether your ideas could be useful for standard RL problems where we know the reward function? Nothing here seems tied to IRL per se in that portion of your analysis. It might be interesting to try out the optimistic critic on a variety of RL problems and potentially have another contribution with minimal theoretical overhead.\n\n2. Why are you adding 2 and not 1 to the denominator of the fraction in Line 11 of Algo 2?\n\n3. Given you're running two no-regret players against each other to solve the game, can you get last iterate convergence via optimism (in the other, game-theoretic sense :p)? If that works, it might be a nice result to add in. If not, I would still think explaining where the proof breaks to be interesting.\n\n4. If it's computationally feasible, could you try the Bayesian dropout version of an ensemble (https://proceedings.mlr.press/v48/gal16.pdf) and report the performance? This way, SOAR has no memory overhead over standard SAC.\n\n5. In Remark 3.1, when you say $L=1$ for most SAC implementations used for IRL, do you mean they have a single *pair* of critics? All SAC implementations I'm aware of don't have a single critic -- would be good to clarify this.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "58JON92Rud", "reviewer": "Reviewer_EDMt", "summary": "This paper presents SOAR: an algorithmic template that learns a policy from expert demonstrations with a primal dual style algorithm that alternates cost and policy updates. The method boosts consistently the performance of imitation learning algorithms based on Soft Actor Critic across various Mujoco tasks.\n\n## update after rebuttal\nI confirm my score. Authors addressed comments and added clarity and results to the original submission.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks a lot for appreciating our work !\n\nWe will make sure to capitalize the algorithm names in the figures.\n\nThanks again for the time spent reviewing our paper, we remain available for further clarification if needed.\n\nBest,\nAuthors", "claims_and_evidence": "Claims are presented clearly and supported by clear and convincing evidence including proofs and sketch proofs.", "methods_and_evaluation_criteria": "The proposed method and evaluation seem reasonable and informative for the application at hand. Comparisons across variants and different algorithms also support the claims made.", "theoretical_claims": "I reviewed the theoretical claims and equations, although I did not verify all mathematical derivations in detail. While it is possible that I may have overlooked some aspects, the theoretical claims appear to be correct to the best of my understanding.", "experimental_designs_or_analyses": "The experiments conducted replicate sets of experiments that are common and well known, as well as using well know algorithms for comparison. The design of the experiments and their analysis is valid and supports the claims.", "relation_to_broader_scientific_literature": "This work is relevant for the imitation learning community, and major parts of the literature therein are covered.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The paper is well presented and significant. Claims, theoretical contributions and experiments are presented in a clear way. It can have impact on imitation learning as well as applications such as in robotics, where reducing exploration time is a significant issue.", "other_comments_or_suggestions": "Please capitalize algorithms names in your figures.", "questions_for_authors": "Please refer to the other comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cOZ38tm7uG", "reviewer": "Reviewer_e44U", "summary": "The paper presents an imitation learning method for the tabular setting and establishes a probabilistic bound on its regret, considering both, regret due to the policy updates and the cost updates. The method is based on the principle of optimism in the face of uncertainty by learning a transition matrix the transition probabilities - thereby reducing the expected cost. Two different options, \"Min\" and \"Mean-Std\",  are presented for obtaining optimistic Q values based on this transition matrix. The algorithm returns a mixture policy, with regret below $\\epsilon$ with high probability, requiring $\\mathcal{O}(\\epsilon^{-2})$ iterations/environment interactions, and $\\mathcal{O}(\\epsilon^{-2})$ expert interactions.\n\nThe paper also proposes an extension for imitation learning methods based on SAC that uses ensembles to obtain optimistic Q-functions. The experiments show, that using these optimistic Q-functions can clearly improve the performance of the tested methods (Coherent Soft Imitation Learning, Maximum Likelihood imitation learning, RKL).\n\n## update after rebuttal\n\nBased on the rebuttal and the other reviews, I do not see the need for any updates.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer,\n\nThanks a lot for appreciating our work !\n\nPlease find our responses to your comments below.\n\n***The experimental design is sound, however, it uses too few seeds (5). It would also be useful to compare the performance for different data set sizes (the appendix does contain experiments with a single expert trajectory, though).***\n\nWe will increase the number of seeds. As can be appreciated in the appendix it seems that even with just 1 expert trajectory SOAR improves against the non exploratory counterparts.\n\n\n***Exploration strategies, also based on optimism, ... in sample efficiency.***\n\nWe definitely agree with the reviewer. Indeed, we hope that the main takeaway from our submission is that optimistic exploration in imitation learning is beneficial. This fact does not seem to be very well known in the current IL literature.\n\n***Very recently, Moulin, … manuscript.***\n\nThanks to the pointer to this reference we missed. We will add it in our revision \n\n***The main contribution … theme).***\n\nIn our opinion, the 2 contributions nicely complement each other. Indeed, our goal was to find a method achieving these two desiderata simultaneously:\n\n1) Guarantees in the tabular case.\n2) Convincing empirical performance in continuous control experiments with minimum algorithm modifications ( that is the way the critics are trained).\n\nThe ensemble based exploration technique implemented in SOAR is one possible way to achieve this goal.\nNotice that, the tabular guarantees per se could also have been achieved with classic analysis leveraging exploration bonuses but it is very unclear how bonus based algorithms can be implemented with neural networks. \n\n\n***In Algorithm 1, line 11, it seems like $k$\n should rather be  $K$***\n\nThanks for catching the typo ! We will fix it.\n\n***Line 372 refers to an \"attainable range\". How is this defined?***\n\nWe will clarify this. By attainable range we mean the interval in which the state value function can take values. Since for all $s \\in \\mathcal{S}$, we have that $V(s) \\in [0, (1-\\gamma)^{-1}]$ we consider as attainable range the interval $[0, (1-\\gamma)^{-1}]$.\n\nThanks again for the time spent reviewing our paper, we remain available for further clarification if needed.\n\nBest,\nAuthors", "claims_and_evidence": "The main claims, e.g. the regret bound (Theroem 3.4) are supported by the provided derivations.", "methods_and_evaluation_criteria": "The method of using optimism to improve exploration and thereby sample efficiency makes sense. The mujoco locomotion experiments are suitable as they are commonly used by related methods, allowing for a fair comparison.", "theoretical_claims": "I did not check the derivations in the appendix, but I could not find any issues with the theoretical claims in the main manuscript.", "experimental_designs_or_analyses": "The experimental design is sound, however, it uses too few seeds (5). It would also be useful to compare the performance for different data set sizes (the appendix does contain experiments with a single expert trajectory, though).", "relation_to_broader_scientific_literature": "Exploration strategies, also based on optimism, are very popular in reinforcement learning, but are much less common in imitation learning, presumably due to the fact that the demonstrations may guide the agent. However, the submission shows, that optimism in the face of uncertainty can result in significant improvements in sample efficiency.", "essential_references_not_discussed": "Very recently, Moulin, Neu and Viano have submitted the related work \"Optimistically Optimistic Exploration for Provably Efficient Infinite-Horizon Reinforcement and Imitation Learning\" to arxiv, which could be already discussed in the manuscript.", "other_strengths_and_weaknesses": "The main contribution seems to be the theoretical analysis for the tabular algorithm. The empirical results in the continuous setting, IL-SOAR, are of practical interest, but this algorithm is quite different from the tabular one. Hence, the different contributions are almost orthogonal (although optimistic Q-functions is, of course, a common theme).", "other_comments_or_suggestions": "In Algorithm 1, line 11, it seems like $K$ should rather be $k$.", "questions_for_authors": "Line 372 refers to an \"attainable range\". How is this defined?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a new algorithm to learn a policy from expert demonstrations by alternating between cost and policy updates. This method uses an optimistic -function update for SAC-based inverse RL algorithm. The sample efficiency of the proposed method is proven theoretically. The efficiency is also proven practically on MuJoCo environments, where this method is shown to outperform certain alternatives. 
The paper is well-written, and the combination of a strong theoretical analysis and a sound empirical evaluation makes this paper well-suited for ICML.
Decision: Accept (poster)