Demonstration-Regularized RL
Authors: Daniil Tiapkin
Denis Belomestny
Daniele Calandriello
Eric Moulines
Alexey Naumov
Pierre Perrault
Michal Valko
Pierre Menard
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
regularization in reinforcement leaning
learning with demonstrations
reinforcemenet learning with human feedback
Primary area: reinforcement learning
TL;DR: We showed a theoretically efficient way to inject expert demonstrations into RL agent and, moreover, into RLHF.

Abstract: Incorporating expert demonstrations has empirically helped to improve the sample efficiency of reinforcement learning (RL). This paper quantifies theoretically to what extent this extra information reduces RL's sample complexity. In particular, we study the demonstration-regularized reinforcement learning framework that leverages the expert demonstrations by $\mathrm{KL}$-regularization for a policy learned by behavior cloning. Our findings reveal that using $N^{\mathrm{E}}$ expert demonstrations enables the identification of an optimal policy at a sample complexity of order $\widetilde{\mathcal{O}}(\mathrm{Poly}(S,A,H)/(\varepsilon^2 N^{\mathrm{E}}))$ in finite and $\widetilde{\mathcal{O}}(\mathrm{Poly}(d,H)/(\varepsilon^2 N^{\mathrm{E}}))$ in linear Markov decision processes, where $\varepsilon$is the target precision, $H$ the horizon, $A$ the number of action, $S$ the number of states in the finite case and $d$ the dimension of the feature space in the linear case. As a by-product, we provide tight convergence guarantees for the behavior cloning procedure under general assumptions on the policy classes. Additionally, we establish that demonstration-regularized methods are provably efficient for reinforcement learning from human feedback (RLHF). In this respect, we provide theoretical evidence showing the benefits of KL-regularization for RLHF  in tabular and linear MDPs. 
Interestingly, we avoid pessimism injection by employing computationally feasible regularization to handle reward estimation uncertainty, thus setting our approach apart from the prior works.

Published as a conference paper at ICLR 2024
DEMONSTRATION-RRGGULARZZZ RL
Daniil Tiapkin' 1.2 Denis Belomestny Daniele Calandriello* Éric Moulines 1.5
Remi Munos Alexey Naumoy² Pierre Perrault Michal Valko*
Pierre Ménard
1CMAP, École Polytechnique HSSE University *Duisburg-Esee University
*Google DeepMind "Mohamed Bin Zayed University of AI, UAE tiapkin, eric. moul
IDEMIA
EENS Lyon
edu denis. belomestay@uni -ounii-due.cee de (dcalandriello, munos, valkom @gooole.. com anaumovOhse. ru pierre, perirrr.rrratttootoooo com pierre menard@ens-lyon
moulines ines /apolytechhiiieeeooee )
ABSTRACT
Incorporating expert demonstrations has empirically helped to improve the sam- ple efficiency of reinforcement learning (RL). This paper quantifies theoretically to what extent this extra information reduces RL's sample complexity. In par- ticular, study the wetponsstrrttooogggaaaaazaa reinforcement learning that lever- ages the expert demonstrations by Kl.reeuularization for policy learned by be- havior cloning. Our findings reveal that using NE expert demonstrations ables the identification of an optimal policy at sample complexity of order (Pooyy(S, A, H))(= in finite and (Pooyydd, H))(="N*)) in linear Markov decision processes, where E is the target precision, H the horizon, A the num- ber of action, S the number of states in the finite case and the dimension of the feature space in the linear case. As by-product, we provide tight conver- gence guarantees for the behavior cloning procedure under general assumptions on the policy classes. Additionally, we establish that demonstration-regulann methods are provably efficient for reinforcement learning from human feedback (RLHF). In this respect. provide thcoretical evidence showing the benefits of KL-regularization for RLHF in tabular and linear MDPs. Interestingly, we avoid pessimism injection by employing computationally feasible regularization to han- dle reward estimation uncertainty, thus setting our approach apart from the prior works.
INTRODUCTION
In reinforcement learning (RI., Sutton & Barto 1998), agents interact with environment max- imize the cumulative reward they collect. While RL has shown remarkable success in mastering complex games (Mnih al., 2013; Silver al., 2018; Berner et al., 2019), controlling physical sys- tems (Degrave et al., 2022), and enhancing computer science algorithms (Mankowitz et al., 2023), does face several challenges. In particular, RL algorithms suffer from large sample complexity, which is hindrance in scenarios where simulations are impractical and struggle in environments with sparse rewards (Goecks et al., 2020).
A remedy found to handle these limitations is incorporate information from pre-collected offline dataset in the learning process. Specifically, leveraging demonstrations from trajectrres without rewards has proven highly effective in redueing sample complexity, especially in fields like robotics (Zhu et al., 2018; Nair et al., 2020) and guiding exploration (Nair et al., 2018; Aytar al., 2018; Goecks et al., 2020).
However, from a theoretical perspective, little is known about the impact of this approach. Previous research has often focused on either offline RL (Rashidinejad al., 2021: Xie al., 2021; Yin al., 2021: Shi et al., 2022) or online RL (Jaksch et al., 2010; Azar et al., 2017; Fruit et al., 2018; Dann al.. 2017; Zanette & Brunskill, 2019b; Jin al., 2018). In this study, we aim to quantify how prior demonstrations from experts influence the sample complexity of various RI tasks, specifically two scenarios: best policy identification (BPI, Domingues al., 2021a; Al Marjani et al., 2021) and reinforcement learning from human feedback (RLHF), within the context of finite or linear Markov decision processes (Jin et al., 2020).
Imitation learning The case where the agent only observes expert demonstrations without further interaction with the environment corresponds to the well-known imitation learning problem. There

---

Published as a conference paper at ICLR 2024
two primary approaches in this setting: inverse reinforcement learning (Ng & Russell, 2000; Abbeel & Ng. 2004; Ho & Ermon, 2016) where the agent first infers a reward from demonstrations then finds optimal policy for this reward; and behavior cloning (Pomerleau, 1988; Ross & Bag- nell, 2010; Ross et al., 2011; Rajeswaran et al., 2018), a simpler method that employs supervised learning to imitate the expert. However collecting demonstration could be expansive and, further- more, imitation learning suffers from the compounding errors effect, where the agent can diverge from the expert's policy in unvisited states (Ross& & Bagnell, 2010; Rajaraman et al., 2020). Hence, imitation learning is often combined with an online learning phase where the agent directly interacts with the environment.
BPI with demonstrations In BPI with demonstrations, the agent observes expert demonstrations like in imitation learning but also has the opportunity to collect new trajectories, including reward information, by directly interacting with the environment. There are three main method categories' for BPI with demonstration: one employs an off-policy algorithm augmented with supervised learning loss and a replay buffer pre-filled the demonstrations (Hosu & Rebedea, 2016; Lakshmi- narayanan et al., 2016; Vecerík al., 2017: Hester et al., 2018), while second uses reinforcement learning with a modified reward supplemented by auxiliary rewards obtained by inverse reinforce- ment learning (Zhu al., 2018; Kang al., 2018). The third class, demmonstration-regullrize RL., which is the one we study in this paper, leverages behavior cloning to learn policy that imitates the expert and then applies reinforcement learning with regularization toward this behavior cloning policy (Rajeswaran et al., 2018; 8; Nair et al., 2018; Goecks et al., 2020; Pertsch et al., 2021).
RL We introduce particular demonstration--regllaiie that consists of several steps. We start by learning with maximum likelihood estimation from the demonstrations of behavior policy. This transfers the prior information from the demonstrations to a more practical representation: the behavior cloning policy. During the online phase, we aim to solve a trajectory Kullback-Leibhe divergence regularized MDP (Neu et al., 2017: Vieillard al., 2020; Tiapkin et al., 2023), penalizing the policy for deviating too far from the behavior cloning policy. We use the solution of this regularized MDP an estimate for the optimal policy in the unregularized MDP, effectively reducing BPI with demonstrations to regularized BPI.
mmmoostration-regularri
RL method
Consequently, we propose two new algorithms for BPI in regularized MDPs: The UCBVI-Ent+ algorithm, a variant of the UCBVI-Ent algorithm by Tiapkin et al. (2023) with improved sam- ple complexity, and the LSVI-UCB- Ent algorithm, its adaptation to the linear setting. When in- RL method, these algorithms yield sample com-
corporated into the demonstrationn-eeglare plexity rates for BPI with N demonstrations of order" Ö(Poly(S, A, ())((²N))) in finite and O(Poly(d, H))((2N*)) in linear MDPs, where e the target precision, H the horizon, A the number of action, S the number of states in the finite case and d the dimension of the feature space in the lin- ear case. Notably, these rates show that leveraging demonstrations can significantly improve upon the rates of BPI without demonstrations, which are of order O(Poly(S, A, H)/e²) in finite MDPs (Kaufmann et al., 2021; Ménard et al., 2021) and (Poly H)/=²) in linear MDPs (Taupin et al., 2023). This work, up to our knowledge, represents the first instance of sample complexity rates for BPI with demonstrations, establishing the provable efficiency of demoonstraioo--rggulaaie
RI.
Preference- BPI with demonstration In RL with demonstrations, the assumption typically entails the observation of rewards in the online learning phase. However, in reinforcement learning from human feedback, such that recommendation system (Chaves et al., 2022), roboties (Jain al., 2013; Christiano et al., 2017), clinical trials (Zhao et al., 2011) 1) or large language models fine-tuning (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang al., 2022), the reward is implicitly defined by human values. Our focus centers on preference-based RL (PHRI., Busa-Feked et al. 2014; Wirth al. 2017; Novoseller et al. 2020: Saha et al. 2023) where the observed preferences between two trajectories are essentially noisy reflections of the value of a link function evaluated at the difference between cumulative rewards for these trajectories.
Existing literature on PbRL. focuses either on the offline setting where the agent observes pre- collected dataset of trajectories and preferences (Zhu et al., 2023; Zhan et al., 2023a) or on the
'The boundary hetween the above families of methods is not strict, since for example, one can see the regularization in the third family a particular choice of auxiliary reward learned by inverse reinforeement learning that appears in the second class of methods. In the O-) notation we ignore terms poly-log in H,S, A, d, 1/5, 1/s and the notation Poly indicates polynomial dependencies.
2

---

Published as a conference paper at ICLR 2024
online setting where the agent sequentially samples pair of trajectories and observes the associated preference (Saha et al., 2023; Xu et al., 2020; Wang et al., 2023).
In this work, we explore hybrid setting that aligns more closely with what is done in practice (Ouyang et al., 2022). In this framework, which we call preference-based BPI with demonstration, the agent selects sampling policy based on expert-provided demonstrations used to generate tra- jectories and associated preferences. The offline collection of preference holds particular appeal in RLHF due to the substantial cost and latency associated with obtaining preference feedback. Fi- nally, in our setting. the agent engages with the environment by sequentially collecting reward-free trajectories and returns an estimate for the optimal policy.
coansttrattoo-rgulaaiieee RLHF To address this novel setting, we follow a similar approach that was used in RL with demonstrations. We employ the dataset of preferences sampled using the behavior cloning policy to estimate rewards. Then, we solve the MDP regularized towards the behavior cloning policy, equipped with the estimated reward. Using the same regularized BPI solvers, we establish a sample complexity for the demonstration-regguaan ((Poly(S, A, / N²) in finite MDPs and ((Poly / )) in linear MDPs. Intriguingly, these rates mirror those of RL with demonstrations, illustrating that RLHF with demonstrations does not pose a greater challenge than RL with demonstrations, Notably, these findings expand upon the similar observation made by Wang et al. (2023) in the absence of prior information.
RLHF method of order
We highlight our main contributions:
We establish that demonstration regularized RL is an efficient solution method for RL with N demonstrations, exhibiting a sample complexity of order ((PolysS, A, H)/(E" ))) in finite MDPs and ((Polydd, H))(="NB)) in linear MDPs.
We provide evidence that demonntraation-regullaiie methods can effectively address reinforce- ment learning from human feedback (RLHF) by collecting preferences offline and eliminating the necessity for pessimism (Zhan et al., 2023a). Interestingly, they achieve sample complexities similar to those in RL with demonstrations.
We prove performance guarantees for the behavior cloning procedure in terms of Kullback-Leiblee divergence from the expert policy. They are of order O(Poly(S, A, H)/NE) for finite MDPs and
(Poly(d, H) for linear MDPs.
We provide novel algorithms for regularized BPI in finite and linear MDPs with sample complexi- ties S A/())) and ( (Ass), correspondingly, where X is regularization parameter.
2 SETTING
MDPs We consider episodic MDP M 81, H, fpa (eEM:: frk heetr)) where S is the set of states with $1 the fixed initial state, A is the finite set of actions of size A, H is the number of steps in one episode, (s(s") [ss a) is the probability transition from state to state s' by performing action a in step h. And (s, a) E [0, 1] is the reward obtained by taking action a in state at step h.
We will consider two particular classes of MDPs.
Definition 1. (Finite MDP) An MDP M is finite if the state space S is finite with size denoted b. Definition 2. (Linear MDP) An MDP M (S, A1, H. (px the m. (rx (ra)ne is linear if the state space S is a measurable for a certain o-algebra Fs, and there exists known feature map v: S x - Rd, and unknown parameters Os E R2 and an unknown family of signed measure ph,i, h [H] E [d] with its vector form 14 Fs + Rd such that for all (h, s,a) E [H] x S x A and for any measurable set B E Fs, it holds (s(s, a) a)ssa)TTe or, and pa (B]s, a) Li 1h(s, (B) *(s, (B.).. Without loss of generality, we assume [|)(s I 1 for all (s, E S Sxxxx A and max(Ilun [[0,]la] S v for all h [H].
Policy & value functions A policy x is a collection of functions Te : S + Aa for all h E [M] where every To maps each state
probability over the action set. We denote by IT the set of policies. The value functions of policy x at step h and state is denoted by V,, and the optimal value functions, denoted by Vi,, are given by the Bellman respectively optimal Bellman equations (, a) = ra a) + pa. Vi+ (s)
(x) = max (s,
denotes the expectation operator with respect to the transition probabilities pa and Tag(s) & Ea (1w))(111b (-1)))99,,00 (0ss denotes the composition with the policy π at step h.
where by definition, V & 0. Furthermore, prf(s, a) & (s
Trajectory Kullback-L.eibler divergence We define the trajectory Kullback-Leibler divergence between policy æ and policy n' as the average of the Kullhack-Leeillrr divergence between policies
3

---

Published as a conference paper at ICLR 2024
at each step along a trajectory sampled with n,
(x = Ex Σ [LLmm(sN)lbt(s))
BEHAVIOR CLONING
In this section, we analyze the complexity of behavior cloning for imitation learning in finite and linear MDPs.
Imitation learning In imitation lcarning we are provided a dataset De { (si.ai...... alz), [] of _ independent reward-free trajectories sampled from a fixed unknown expert policy = The objective is to learn from these demonstrations a policy close to optimal. In order to get useful demonstrations we assume that the expert policy is close to optimal,
that is, (a1) vi= (s1) - ew for some small En > O.
Behavior cloning The simplest method for imitation learning is directly learn to replicate the expert policy in a supervised fashion. Precisely the behavior cloning policy x BC is obtained by minimizing the regative-lollikelhood over a class of policies F {- E II : Ta Fi.} with Fr
being a class of conditional distributions S + P(A) and where R, is some regularizer,
NR
BC
E arg min
log
[a[]],)))]) + RA(TT))
(1)
In order to provide convergence guarantees for behavior cloning, we make the following assump- tions. First, we assume some regularity conditions on the class of policies defined in terms of covering numbers of the class, see Appendix A for a definition.
Assumption 1. For all & [H]. there two positive constants Rz such that Yh [M] Vs e (0, 1) Wo, l1+)lo) S dy (R))))))).
Moreover, there is a constant y > such that for any E [H], nn E F₁ holds T. 2 7 for any (s, () E S x A.
The Assumption is a typical parametric assumption in density estimation, see e.g. Zhang (2002), with de being a covering dimension of the underlying parameter space. The part of the assumption on a minimal probability is needed to control KL-divergences (Zhang, 2006).
Next, we assume that smooth version of the expert policy belongs the class of hypotheses. Assumption 2. There is a constant к E (0, 1/2) such that a n-greedy version of the expert policy defined by n (als) (1 - (a + k/A belongs to the hypothesis class of policies: = F.
Note that a deterministic expert policy verifies Assumption 2 provided that is small enough and the policy class is rich enough. For x = 0, this assumption is never satisfied for any 7 > 0.
In the sequel, we provide examples of the policy class F and regularizers (Ra) for finite or linear MDPs such that the above assumptions are satisfied. We are now ready to state general performance guarantees for behavior cloning with KL regularization.
Theorem 1. Let Assumptions 1-2 be satisfied and let 0 < Ru(TT) S M for all h E [H] and any policy & Fi. Then with probability at least - 8, the behavior policy RC satisfies
(T (1"
6d H (log( Ae² /(yy A k)) log(2HN R₂ Rx((())
2HM NE
+ 1-K
18k
NE
This result shows that if the number of demonstrations EE is large enough and 7 = 1/N π A/N then the behavior cloning policy zBC converges to the expert policy x at fast rate of order O((d=H + A)/Nz) where we measure the "distance" between two policies by the trajectory Kullback-Lein Leibler divergence. The proof of this theorem is postponed to Appendix B and it heavily relies on verifying the so-called Bernstein condition (Bartlett & Mendelson, 2006).
3.1 FINITE MDPs
For finite MDPs, we chose logarithmic regularizer (TT(TT) E.,, log(1 / /T (als)) and the class of policies J (IT II: Fu (als) 2 1/(N² + A)). One check that Assumptions 1-2 hold and 0 < (TA) S SA log + A). We can apply Theorem to obtain the following bound for finite MDPs (see Appendix B.2 for additional details).
Corollary 1. For all N > for function class F and regularizer (Ra) (Ra)nem defined above, it holds
with probability at least 1 6,
KL ( (1 #C)
6SAH log(2e" N 1og(12H(N"))/0)
18AH
NE
NE
4

---

Published as a conference paper at ICLR 2024
Note that, imitation learning with a logarithmic regularizer is closely related to the statistical problem of conditional density estimation with Kullback-Leibler divergence loss, see for example Section 4.3 by van der Hoeven et al. (2023) and references therein. Additionally, we would like to emphasize that the presented upper bound is optimal up to poly-logarithmic terms, see Appendix B.5 for a corresponding lower bound.
Remark 1. In fact, the constraint added by the class of policies F is redundant with the effect of regularization and one directly optimize the whole set of policies in (I). It is then easy to BC obtain a closed formula for the behavior cloning policy *h (als) = (N (s. a) + 1)/(NN + A),
where we define the counts by (s) Ene (s, a) and (v, a) 1i 1 { a)) = as. Remark 2. Contrary to Ross & Bagnell (2010) and Rajaraman et al. (2020), our bound does not feature the optimality gap of the behavior policy but measures how close it is to the expert policy which is crucial to obtain the results of the next sections. Nevertheless, we can recover from our bound some of the results of the aforementioned references, see Appendix B.6 for details.
3.2 LINEAR MDPs
For the linear setting, need the expert policy to belong some well-behaved class of parametric policies. A first possibility would be to consider a greedy policy with respect to Q-value, linear in the feature space Th (s) E arg a (s)) ur, for some parameters uy, as it is done in the existing imitation learning literature (Rajaraman et al., 2021). However, under such a parametrization would be almost impossible learn an expert policy with high quality since small perturbation in the parameters w/, could lead to a completely different policy. We emphasize that if we assume that the expert policy is an optimal one, then Rajaraman et al. (2021) proposes a way to achieve a e-optimal policy but not how to reconstruct the expert policy itself. That is why we consider another natural parametrization where the log probability of the expert policy linear in the feature space. Assumption 3. For all h E [H], there exists an unknown parameter wh E ue with || ll2 S R for some known Rt 2 such that # (als) = exp(v'(s, a) "////(E
exp(U(s, (TTT)))).
For instance, this assumption is satisfied for optimal policy in entropy-regee reoooy-regularized linear MDPs, see Lemma in Appendix B.3. Under Assumption 3, suitable choice of policy class is given by F {- E II: TA E Fi) where
Fr [a(a]s)= K
-1- En'eA exp(t)(s, exp()"(s.a')Tnn) w. E ER,, ||wall2 lla S R
exp(v(s, www
(2)
and x = A/(NE + A). Furthermore, for the linear setting, we do not need regularization, that is, RA (m) 0. Equipped with this class of policies we can prove similar result as in the finite setting with the number of states replaced by the dimension d of the feature space.
Corollary 2. Under Assumption 3, function class F defined above and regularizer Rr. 0 for all
h E [H), it holds for all NE 2 A with probability at least 1 - 5,
8dH (log(2e) AN (log(48(N + log(H/8)))
18AH
(7
NE
N
Taking into account the fact that finite MDPs are specific within the broader category of linear MDPs, the lower bound presented in Appendix B.5 also shows the optimality of this result.
4
WWWNSSTRRATION-EGGLAAAAE
RL
In this section, we study reinforcement learning when demonstrations from an expert are also avail- able. First. we describe the regularized best policy identification framework that will be useful later.
Regularized best policy identification (BPI) Given some reference policy से and some regular- ization parameter A > 0, we consider the trajectory Kullback-Leibler divergence regularized value function V ($1) & V/2 (s1) - AKLuaJ(T, 7). In this value function, the policy æ is penalized for moving too far from the reference policy न. Interestingly, we can compute the value of policy x with the regularized Bellman equations, (Neu et al., 2017; Vieillard et al., 2020)
Q..AA = rh a) + Ph V.. (s..)
VT. n Q (s) XKL( (s))
where VT. + = O. We are interested in the best policy identification for this regularized value. Precisely, in regularized BPI, the agent interacts with MDP as follows: at the beginning of episode t, the agent picks up policy o' based only on the transitions collected up episode t - 1. Then new trajectory (with rewards) is sampled following the policy z' and is observed by the agent. At the end of each episode, the agent can decide to stop collecting new data, according to random stopping time (c = if the agent stops after the t-th episode), and output a policy # based on the observed transitions. An agent for regularized BPI is therefore made of triplet ((=') 1, R)
5

---

Published as a conference paper at ICLR 2024
Definition 3. (PAC algorithm for regularized BPI) An algorithm ((*))EEN is (e.8)-PAC for BPI regularized with policy गt and parameter with sample complexity C(E,A,
(s1) (+1) se i (s 2 21-8.
We can now describe the setting studied in this section.
BPI with demonstration We assume, as in Section 3, that first the agent observes N independent trajectories DE sampled from expert policy л². Then the setting the same as in BPL. Precisely, the agent interacts with the MDP as follows: at episode t, the agent selects a policy s' based on the collected transitions and the demonstrations. Then following the policy x² and observed by the agent. At the end of each episode, the agent stops according stopping rule (t t if the agent stops after the 1-th episode), and outputs policy .LRL
trajectory (with rewards) is sampled
Definition 4. (PAC algorithm for BPI with demonstration) An algorithm ((T ((T') is (e, PAC for BPI with demonstration with sample complexity ( if
P (v) - vi" (s1) S e, CS .6) 2
To tackle BPI with demonstration focus the following natural and simple approach.
annponstration-reguaaattt RL The main idea behind this method is to reduce BPI with demon- stration to regularized BPI. Indeed, in demoonttration-regllaiiee RL, the agent starts by learning through behavior cloning from the demonstration of a policy _C that imitates the expert policy, refer to Section 3 for details. Then the agent computes a policy R by performing regularized BPI with policy BC and some well-chosen parameter A. The policy R² is then returned as the guess for an optimal policy. The whole procedure is described in Algorithm 1. Intuitively the prior informa- tion contained in the demonstration compressed into handful representation namely the policy r BC Then this information is injected into the BPI procedure by encouraging the agent to output policy close to the behavior policy.
Algorithm Demonstratiinn-regulaaiia RL
1: Input: Precision parameter ERL, probability parameter ōRL, demonstrations Dr, regularization parameter A.
2: Compute behavior cloning policy BC = BehavLorCloning((P)))
3: Perform regularized BPI RL RegBPI( BC >, SRI., ŌRI.)
4: Output: policy , R!.
Using the previous results for regularized BPI, we next derive guarantees for demonstration- regularized RL. We start from a general black-box result that shows how the final policy error depends on the behavior cloning error, parameter X, and the quality of regularized BPI.
Theorem 2. Assume that there are an expert policy x such that (i((s)) - VI (s1) Vi"((s:) S eR and behavior cloning policy n satisfying (KLaay(="IxRR)) I EKL. Let x be ERL-optimal policy in X-regularized MDP with respect to π .hht is, Vi
BL V,B ERL. Then n fulfills
Vi'(s1) VT" (s1) S EE + ERL + AE
In particular, under the choice X" ERI./EkI. the policy RL is (2aRI. + en)-optimal in the original (non-regun regulaargari MDP.
Remark 3. We define an eтror in trajectory KL-divergence under the square root because the KL- divergence behaves quadratically in terms of the total variation distance by Pinsker's inequality. Remark 4 (BPI with prior policy). We would like to underline that we exploit all the prior informa- tion only through one fixed behavior cloning policy. However, as it is observable from the bounds of Theorem 2, our guarantees are not restricted to such type of policies and potentially could work with any prior policy close enough to a near-optimal one in trajectory Kullback-Leibhe divergence.
The proof of the theorem above is postponed to Appendix C. To apply this result and derive sample complexity for mnmooostration-reglliiiie BPI, we present the UCBVI-Ent* algorithm, a modifica- tion of the algorithm UCBVI-Ent proposed by Tiapkin et al. (2023), that achieves better rates for regularized BPI in the finite setting. In Appendix E, we also present the LSVI-UCB- Ent algorithm, direct adaptation of the UCBVI-Ent+ to the linear setting.

---

Published as a conference paper at ICLR 2024
Notably, the use of UCEVI-Ent by Tiapkin al. (2023) within the framework of demonstration- regularized methods, fails to yield acceleration through expert data incorporation due to its ((1/e²) sample complexity. In contrast, the enhanced variant, UCBVI-Ent*, exhibits more favorable com- plexity of (1/(Ex)) where == e/EKL, > & under the conditions stipulated in Theorem 2, for a skl sufficiently small. It is noteworthy that alternative approach employing the RL-Explore-Bi algorithm, introduced by Tiapkin et al. (2023), can also achieve such acceleration. However, AL-Explore- is associated with inferior rates terms of S and I and challenging to tend beyond finite settings.
The UCBVI-Ent+ algorithm works by sampling trajectories according an exploratory version of an optimistic solution for the regularized MDP which is characterized by the following rules.
UCBVI-Ent+ sampling rule To obtain the sampling rule at episode we first compute policy sf by optimistic planning in the regularized MDP,
(s.a) clip clip(r((s,a) + V (s, (s.
# (s) arg max (+) (+)) EEA
- =
with V V+1 = by convention, where से is a reference policy, 5' is an estimate of the transition probabilities, and be some bonus term taking into account estimation error for transition probabili- ties. Then we define family of policies that aim to explore actions for which Q-value is not well estimated at particular step. That is, for h' E [0, Jf]. the policy m=((k)) first follows the optimistic policy until step h where it selects an action leading to the largest width a confidence interval for the optimal Q-value,
(als) (als)=
if + M' i
[1 [1(a= arg (O,
(s,'')) ((,,)))))))
where Q' is lower bound on the optimal regularized Q-value function, see Appendix D.4. In particular, for h' 0 we have xf.00) #². The sampling rule is obtained by picking uniformly random one policy among the family x' 2²,,,, N [0, H] in each episode. Note that it is equivalent to sampling from a uniform mixture policy a mixx over all E [0, H), see Appendix D for more details. This algorithmic choice allows us exploit strong convexity of the KL-divergence and control the properties of a stopping rule, defined in Appendix D.4, that depends on the gap (Q', a) - O (s.a))²
The complete procedure is described in Algorithm 3 in Appendix D. We prove that for the well- calibrated bonus functions 62-7 and a stopping rule defined in Appendix D.4, the CCBVI-Enn++ algo- rithm (s, P for regularized BPI and provide a high-probabillty upper bound on its sample complexity. Additionally, a similar result holds for LSVI-UCB Ent algorithm. The next result is proved in Appendix D.5 and Appendix E.5.
Theorem 3. For all æ > 0, 8 E (0.1), 1), the UCBVI-Ent+ / LSVI-UCB UCBt algorithms defined in Appendix D.4 /Appendix E.4 are (EA for the regularized BPI with sample complexity
S2
C(E,6)= = HH' dd
C(E,8) 5
(finite
(linear).
dE
AE
Additionally, assume that the expert policy EE = =//2ooptimal and satisfies Assumption 3 in the linear case. Let n. BBC be the behavior cloning policy obtained using corresponding function sets described in Section 3. Then demoontrraioon-regalariz
RL based on UCBVI-Ent+ /ittettt//SSVV /LSVI-UCCB Ent with parameters ERLL e/4, dRL. 5/2 and X ((NEE/(SAHI) ( ((d))/(dH) is (e, (),5))PAC for BPI with demonstration in finite / linear MDPs and has sample complexity of order
C(E,N 5 g A
C(E,N = 5 H
(finite)
(linear).
NE
In the finite setting, UCBVI-Ent+ improves the previous fast-rate sample complexity result of order O(I 5* A*A/())) ) by Tiapkin et al. (2023). For the linear setting, we would like to acknowledge that LSVI-UCB-ENT is the first algorithm that achieves fast rates for exploration in regularized linear MDPs.
5 DEMONSTRATION-AEGULAATUTT
RLHF
In this section, we consider the problem of reinforcement learning with human feedback. We assume that the MDP is finite, i.e., [S] < +oo simplify the manipulations with the trajectory space.

---

Published as a conference paper at ICLR 2024
However, the state space could be arbitrarily large. In this setting, we do not observe the true reward function r" but have access to an oracle that provides a preference feedback between two trajectories. We assume that the preference is random variable with parameters that depend on the cumulative rewards of the trajectories as detailed in Assumption 4. Given a reward function r = {rr ) we define the reward of a trajectory (8 A) the sum of rewards collected over this trajectory r(r) En'n Th (s), os.).
Assumption (Preference-
model). Let To: Ti be two trajectories. The preference for Ti over trajeeed
is Bernoulli random variable with parameter g. (To, T)) oor" y" (To)), where or: R [o, 1] is a monotone increasing link function that satisfies inf o (x) = 1/C for Ç > 0.
The main example of the link function sigmoid function a(x) = 1/(1 + exp(-z)) that leads to the Bradley-Terry-Lue (BTL) model (Bradley & Terry. 1952) widely used in the literature (Wirth et al., 2017; Saha et al., 2023). We now introduce the learning framework.
Preference-basee BPI with demonstration We assume, as in Section 3, that the agent observes N independent trajectories DE sampled from an expert policy T. Then the learning divided in two phases:
1) Preference collection. Based on the observed expert trajectories DE. the agent selects a sampling policy R to generate data set of preferences DRM {(rd'. - ) consisting of pairs of tra- jectories and the sampled preferences. Specifically, both trajectories of the pair (rØ rf)) are sampled with the policy n and the associated preference o* is obtained according to the preference-bant model described in Assumption 4.
2) Reward-free interaction. Next, the agent interacts with the reward-free MDP as follows: at episode t, the agent selects a policy x' based on the collected transitions up to time t, demonstrations and preferences. Then a new trajectory (reward-free is sampled following the policy x' and is observed by the agent. At the end of each episode, the agent can decide to stop according to a stopping rule and outputs policy RRL...
Definition 5 (PAC algorithm for preference-basee BPI with demonstration). An algorithm ((T')LEN, R 4, n RL.HF) " is (e, 8PC for preference-based BPI with demonstrations and sample com- plexity C(e, 8) if (s:) vi" RLAF reward function r* is used in the value-function V*
(s:) S E, C(e, N(e,NV, 6)) 2 1 11-6d 6, where the unknown true
For the above setting, we provide a natural approach that combines with the maximum likelihood estimation of the reward given preferences dataset.
wwwpnsttaatioo-rguaaiiaae RL
cemoontraation-rgglaiizze
RLHF During the preference collection phase, the agent generates
a dataset comprising trajectories and observed preferences, denoted as DRM = {(Tō, a o²) )') executing the previously computed policy DCC Using this dataset, the agent can infer the reward via maximum likelihood estimation (MLE).
NMM by
Algorithm Demoonstration-reguulee
RLHF
1: Input: Precision parameter ERIMF, probability parameter òRIMP, demonstrations Dp. prefer- ences budget RM, regularization parameter X.
2: Compute behavior cloning policy _,CC = BehaviorCConingg(P.
3: Select sampling policy ~ BC and collect preference dataset DRM;
4: Compute reward estimate = RewardMLE(Gr, Dnm);
5: Perform regularized BPI using reward: ,RLHF RegBPI( X, ERLHP, ÖRLMP;
6: Output: policy x RLUF
The core idea behind this approach is to simplify the problem by transforming it into regularized BPI problem. The agent starts with behavior cloning applied to the expert dataset, resulting in the policy π BC During the preference collection phase, the agent generates a dataset comprising trajec- tories and observed preferences, denoted as DRM = {(TŌ o offoo))ooo))ff)) X"M by executing the previously computed policy π BC Using this dataset, the agent can infer the reward via MLE:
NRM
2 arg max YEG
Σ ot log (r((())) (((())))r(16)))) -(()))) (1 - o²) log
(1- (1-0(1(r)) -
(oc(((())r((r)))) r(1)))
where is function class for trajectory reward functions?. Finally, the agent computes x RR by performing regularized BPI with policy x BC a properly chosen regularization parameter À and the estimated reward f. The complete procedure outlined in Algorithm
For the theoretical guarantees on MLF estimate of rewards á refer to Appendix F.I.
8

---

Published as a conference paper at ICLR 2024
the behavior cloning policy NBCC for two purposes. First, it allows efficient offline collection of the preference dataset DRM, from which a high-quality estimate the reward can be derived. Second, regularization towards the behavior cloning policy л enables the injec- tion of information obtained from the demonstrations, while also avoiding the direct introduction of pessimism in the estimated reward in the previous works that handle offline datasets (Zhu al., 2023; Zhan et al., 2023a).
For this algorithm,
Remark 5. Zhan et al. (2023b) propose a similar two-stage setting of preference collection and reward-free interaction without prior demonstrations and propose an algorithm for this setup. How- ever, compared to their result, our pipeline adapted to any parametric function approximation of rewards and does not require solving any (non-convex) optimization problem during the preference collection phase.
Our approach to solve BPI with demonstration within the preference-based model framework draws inspiration from well-established methods for large language model RI. fine- tuning (Stiennon al., 2020; Ouyang et al., 2022: Lee et al., 2023). Specifically, our algorithm's policy learning phase is similar to solving an RL problem with policy-dependent rewards
Remark 6.
7 .RLFF (s. a) = f a) - x log T.LHF (als)/ (
This formulation, coupled with our prior stages of behavior cloning, akin supervised fine-tuning (SFT), and reward estimation through MLE based on trajectories generated by the SFT policy, mir- rors a simplified version of the three-phase RLHF pipeline.
The following sample complexity bounds for tabular and linear MDPs is simple corollary of The- orem and Theorem 3 and its proof is postponed to Appendix F.3.
RLHF). Let Assumption 4 hold. For 0 and E (0, 1), assume that an expert policy ev. e/8-optima and satisfies Assumption 3 in the linear case. Let The BC be the behavioral cloning policy obtained using function sets described in Section 3 and let the set G be defined in Lemma 19 for finite and in Lemma 20 for linear setting, respectively.
Corollary DDemonstration-regllaaa
If the following two conditions hold
NL) NRM 2 ( (2) 20(12//))) or NRM 2o(ceeDD)ee
for D = SA/d d in finite/ pmitt/llieaa linear MDPs and C, C, F BC) is single-policy concentrabillty coefficient defined in (20), Appendix F.3 (see also Zhan et al. 2023a), then demonstration-reglaaieee RLHF based on UCBVI-Ent-
LLSVI-UCB-EN with parameters
e/16, ōnL 5/3 and X =
X > ( (O(")/(dHH) is (E, for BPI with demonstration in finite linear
MDPs with sample complexity
C(E,N 80) 3 S" N
H N
(finite)
C(e, NE
(.N)E)))). 6)= Ō
(linear).
The conditions (1) and (2) control two different terms the reward estimation error presented in Theorem 8. Condition (1) shows that the small size of the expert dataset should be compensated by a larger dataset used for reward estimation and vice versa.
At the same time, condition (2) requires that at least one of these datasets is large enough to overcome the sub-exponential behavior of the error in the reward estimation problem. We remark that the second part of the condition (2) NRM C,/s² is unavoidable in the general case of offline learning even if the transitions are known due to a lower bound in Theorem 3 by Zhan et al. (2023a). However, as soon as the reward estimation error is small enough, we obtain the same sample complexity RL (see Section 4).
guarantees as in the demonstration-regullan
6 CONCLUSION
In this study, we introduced the BPI with demonstration framework and showed that demonstration- regularized RI, a widely employed technique, is not just practical but also theoretically efficient for this problem. Additionally, we proposed a novel preference-bar based BPI with demonstration ap- proach, where the agent gathers demonstrations offline. Notably, we proved that a demonstration- regularized RL method can also solve this problem efficiently without explicit pessimism injection. A compelling direction for future research could involve expanding the feedback mechanism in the preference-baee based setting, transitioning from pairwise comparison to preference ranking (Zhu et al., 2023). Additionally, it would be interesting to explore scenarios where the assumption a white- box preference-bacee based model, as proposed by Wang et al. (2023), is relaxed.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
D. Belomestny acknowledges the financial support from Deutsche Forschungsgemeinshaat (DFG), Grant Nr.497300407. The work of D. Belomestny and A. Naumov was supported by the grant for research centers in the field of Al provided by the Analytical Center for the Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D73032 (1000002)))) and the agreement with HSE University No. 70-2021-00139 The work of D. Tiapkin has been supported by the Paris flende-France Région in the framework of DIM A14IDE.
REFERENCES
Yasin Abbasi- Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances neural information processing systems, 24, 2011.
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In ICML '04; Proceedings of the !wenty-first international conference on Machine learning, pp. 1, New York, NY, USA, 2004. ACM. ISBN 1-58113-828-5.
Alekh Agarwal, Sham Kakade, Akshay Krishnamurthy, and Wen Sun. Flambe: Structural com- plexity and representation learning of low rank mdps. Advances neural information processing systems, 33:20095-20107, 2020.
Aymen Al Marjani, Aurélien Garivier, and Alexandre Proutiere. Navigating to the best policy in markov decision processes. Advances in Neural Information Processing Systems, 34:25852- 25864, 2021.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando de Freitas. Playing hard exploration games by watching youtube. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Gareett (eds.). Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018, URL https: httpe//ee/c.ccccccoee/eaaaaa
wwwpiiiinnnnnnnnnrneeeeeeeeeeeee neurips. cc/paper_ pdf.
Mohammad Gheshlaghi Azar, Ian Osband, and Rémi Munos. Minimax regret bounds for rein- forcement learning. In International Conference Machine Learning, 2017, URL https:
//arxiv.org wwg/pdff//t.ooooooo.oo..goo....p /1003. 05449.pd..
Peter L Bartlett and Shahar Mendelson. Empirical minimization. Probability theory and related
fields, 135(3))311-32)) -34,,333222 2006.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub W. Pachocki, Michael Petrov, Henrique Pond'c e de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor. Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. ArXiv, abs/1912.06663 2019.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-35,, 1952.
Róbert Busa-Fekete, Balázs Szörényi, Paul Weng, Weiwei Cheng, and Eyke Hüllermeier. Preference-basee reinforcement learning: Evolutionary direct policy search using a preference- based racing algorithm. Mach. Learn., 97(3):327-351 dec 2014. ISSN 0885-6125. doi:
10.11077/6109944444445533 URL https: /ttp::::////iooo///////// 10t.:/....1000/00000 100010000000994444--66666
Pedro Dalla Vecchia Chaves, Bruno L. Pereira, and Rodrygo L. T. Santos. Efficient online learning to rank for sequential music recommendation. In Proceedings of the ACM Web Conference 2022, WWW '22, pp. 1442-2450 New York, NY, USA, 2022. Association for Computing Machinery. ISBN 9781450390965.
doi: 10.11457348547.35122166 URL https: h/tp///o..oooo//////// org. /0g/10..0 1145/
3485447 0485447.3512116.
10

---

Published as a conference paper at ICLR 2024
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. I. Guyon, U. Von Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Gareett (eds.), Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL Utt.htttpss/pproceeennss neurips. pdf.
Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying PAC and regret: Uniform PAC bounds for episodic reinforcement learning. In Neural Information Processing Systems, 2017.
URL https: pettpa://arxcee wwt/ httpp////100.0771111111 0df10.pdf.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki. Diego de las Casas, Craig Don- ner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Mare Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva. Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897::4444199, February 2022. ISSN 1476-4687. doi: 10. 111008/41186-021-4455555 111586-021-04300
URL https: pettpa//doos http//ddooortt// org/oogg///// 10338/
Omar Darwiche Domingues, Pierre Ménard, Emilie Kaufmann, and Michal Valko. Episodic rein- forcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory, pp. 578-598. PMLR, 2021a.
Omar Darwiche Domingues, Pierre Menard, Matteo Pirotta, Emilie Kaufmann, and Michal Valko. Kernel-based reinforcement learning: A finite-time analysis. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2783-27922 PMLR, 18 18-24 Jul 2021b. URL mlr. prw.s/pee/seeee ht////13/////// /ww.//ommnaa.a....m/o html.
Richard M Dudley. Uniform central limit theorems, volume 142. Cambridge university press, 2014.
Ronan Fruit, Mattco Pirotta, Alessandro Lazaric, and Ronald Ortner. Efficient bias-span-constraimmg exploration-exxlootttion in reinforcement learning. In International Conference Machine Learning, pp. 1578-1586 PMLR, 2018.
Vinicius G. Goecks, Gregory M. Gremillion, Vernon J. Lawher, John Valasek, and Nicholas R. Waytowich. Integrating behavior cloning and reinforcement learning for improved performance in dense and sparse reward environments. In Proceedings of the 19th International Conference Autonomous Agents and MultiAgent Systems, AAMAS '20, pp. 465-473, Richland, SC, 2020. In- ternational Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450375 184.
Jean-Bastien Grill, Omar Darwiche Domingues, Pierre Menard, Remi Munos, and Michal Valko. Planning in entropy-regularized markov decision processes and games. In H. Wallach, H. Larochelle, A. Beygelzimer, F. ''AlcééBuuc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: httpppoeeenng.neeeeeee 40924fb222caa88633530000 Paaeeeeeeeeeaaaeeappa........
cc/ppaprr2219//11e
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Hor- gan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium Educational Advances in Artificial Intelligence, AAAI' 8.
18. AAAI Press, 2018, ISBN 978-1-57735-888
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Gamett (eds.), Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL p/p.oceeig///.oooeedini.....s f wtt.:///eerrotttooocooooraaaaaaa
wwworrpp.cc///eeeeeeeeeooooo cc/paper_ pdf.

---

Published as a conference paper at ICLR 2024
I.-A. Hosu and T. Rebedea. Playing atari games with deep reinforcement learning and human check- point replay. In ECA! Workshop on Evaluating General Purpose AI, 2016.
Ashesh Jain, Brian Wojeik, Thorsten Joachims, and Ashutosh Saxena. Learning trajectory pref- erences for manipulators via iterative improvement. In C.J. Burges, L. Bottou, M. Welling, Z.. Ghahramani, and K.Q. Weinherger (eds.), Advances in Neural Information Processing Sys- tems, volume 26. Curran Associates, Inc., 2013. URL https:: paper_
neurips.co pdf.
Thomas Jaksch, Ronald Ortner, and Peter Auer. Near-optimal regret bounds for reinforcement learning. Journal of Machine Learning Research, 99:1563-1600, 2010. URL http://www. jmlr. org/paperss httppppppplluuaaeeeppeae
jawsshhaaa pdf.
Chi Jin, Zeyuan Allen-Zhu, Sébastien Bubeck, and Michael I. Jordan. Is Q-learning provably ef- ficient? In Neural Information Processing Systems, 2018. URL https://arxiv.. org/ / pdf.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pp. 2137-2143. PMLR, 2020.
Anders Jonsson, Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Edouard Leurent, and Michal Valko. Planning in markov decision processes with gap-dependent sample complexity. Advances in Neural Information Processing Systems, 33:1253 -2222333,,,,, 2020.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning- In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2469-2478 2478. PMLR, 10-15 Jul 2018. URL https: httpss////ccceedeeepeeeee mlr. www.:mmww.............m....e Trresssv880/kang185 html..
Emilie Kaufmann, Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Edouard Leurent, and Michal Valko. Adaptive reward-free exploration. In Vitaly Feldman, Katrina Ligett, and Sivan Sabato (eds.), Proceedings of the 32nd International Conference on Algorithmic Learning Theory, volume 132 of Proceedings of Machine Learning Research, pp. 155-891. PMLR, 16-19 Mar 2021. URL https:://ppeet httpss//pproeeedingg mlr wwtp./pwwws..eeeeeeeeeeee https////kaaaaann1112 html.
A. S. Lakshminarayanan. S. Ozair, and Y. Bengio. Reinforcement learning with few expert demon- strations. In NIPS Workshop on Deep Learning for Action and Interaction, 2016.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint (rXXi:2309.0026: 2023.
Daniel J. Mankowitz, Andrea Michi, Anton Zhernov, Marco Gelmi, Marco Selvi, Cosmin Padu- raru, Edouard Leurent, Shariq Iqbal, Jean-Baptiste Lespiau, Alex Ahern, Thomas Köppe, Kevin Millikin, Stephen Gaffney, Sophie Elster, Jackson Broshear, Chris Gamble, Kieran Milan, Robert Tung, Minjae Hwang. Taylan Cemgil, Mohammadamin Barekatain, Yujia Li, Amol Mandhane, Thomas Hubert, Julian Schrittwieser, Demis Hassabis, Pushmeet Kohli, Martin Riedmiller, Oriol Vinyals, and David Silver. Faster sorting algorithms discovered using deep reinforcement learn- ing. Nature, 61 218(7964):257-266 263, Jun 2023. ISSN 1476-4687. doi: 10. 111088/4118866 0006--003300000000 URL httph:t///// //dos:://dooi. org/ 10. 1038/ 041186-08228 023-0600--9.
Pierre Ménard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 7599-7608. PMLR, 18 24 Jul 2021. URL http:: htttps:///pooceeinggs mlr pttppp///v//////rrr/2222 htmml.
12

---

Published as a conference paper at ICLR 2024
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In NIPS Deep Learn ing Workshop. 2013.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE Interna- tional Conference Robotics and Automation (ICRA), pp. 6292-6299. IEEE Press, 2018. doi: 10.1109/ICRA.0111 8463162. URL https://doi...ooo wttp://oo..ooooooo 1109/ICRA, 8463162.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. CoRR, abs/2006.09359, 2020. URL https::/aaxxit org/abs/ 2006. 09359.
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropyy-regularized markov decision processes. CoRR, abs/1 tesss/70557798, 2017.
07798.
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement reinforcementtlarrnnng learning. In Pat Langley (ed.), ICML, pp. 663-670. Morgan Kaufmann, 2000. ISBN 1-55860-707-2. URL http://bblp. uni moii-rrin tetrirrrrdee /e/ åb conf /aann//cmm////m hwtpm//mmmmmo....mmm htg.00
Ellen Novoseller, Yibing Wei, Yanan Sui, Yisong Yue, and Joel Burdick. Ducling posterior sam- pling for preference-based reinforcement learning. In Jonas Peters and David Sontag (eds.), Pro- ceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI), volume 124 of Proceedings of Machine Learning Research, pp. 1029-1038. PMLR, 03-06 Aug 2020. URL htttps://proceedingg mlr. aetess//v122/novoeleeee2222 html.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J. Lim. Demonstratioonguudded reinforcement learning with learned skills. 5th Conference Robot Learning, 2021.
Dean Pomerleau. ALVINN: autonomous land vehicle in neural network. In David S. Touretzky (ed.), Advances in Neural Information Processing Systems 1, JNIPS Conference, Denver, Col- orado, USA, 1988], pp. 305-313. Morgan Kaufmann, 1988. URL /ttt::/pppprss nips.cc/ cc/ paper/95- alvinn an autonomous land- fend-vehicleiir in- a -neurrk..ettrr..
Nived Rajaraman, Lin Yang. Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 2914-2924. Curran Asso- ciates, Inc., 2020. URL https: http///poeeeeeen..eeeeee wwwpeeeddiiiinnnnnnnneeeeeeeeeee wwwpurps.cccaaaaaeeeeee 0h10://2277000000009666600000080
Paper. .df.
Nived Rajaraman, Yanjun Han, Lin Yang, Jingbo Liu, Jiantao Jiao, and Kannan Ramchandran. On the value of interaction and function approximation in imitation learning. Advances in Neural Information Processing Systems, 34:1325-1331 1336, 2021.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations. In Proceedings of Robotics: Science and Systems (RSS), 2018.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging of- fline reinforcement learning and imitation learning: A tale of pessimism. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 11702-11716. Curran Associates, Inc., 2021. URL https://procceeding htwp::eeeeeiiinnneeiiiieeeeeeeee neurips...... www.rnee..ccpappppeeeee (tte6//2ccbbaac0442eeeeeooe
pdf.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 661-668, JMLR Workshop and Conference Proceedings, 2010.
13

---

Published as a conference paper at ICLR 2024
Stephane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and struc- tured prediction to no-regret online learning. In Geoffrey Gordon, David Dunson, and Miroslav Dudík (eds.), Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pp. 627-635, Fort Lauderdale, FI., USA, 11-13 Apr 2011. PMLR. URL htttps://proceeddnggs mlr press/ v15/ross11a 1a. html.
Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Ducling RL: reinforcement learning with tra- jectory preferences. In Francisco J. R. Ruiz, Jennifer G. Dy, and San-Willem van de Meent (eds.), International Conference Artificial Intelligence and Statistics, 25-27 April 2023, Palau de Congressos, Valencia, Spain, volume 206 of Proceedings of Machine Learning Research, pp. saha23a 6263-6289, PMLR, 2023. URL https: htRps:tt:://poooot httt:::://rroceednggg mlr. www.paeee0ee666- s0ha/aaa2///aaaaaaaaaaaaa html.
Igal Sason and Sergio Verdú. f -divergence inequalitiess IEEE Transactions on Information Theory, 62(11):5973-660) 6006, 2016. doi: 10.1 10.110997T22206226331555
Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 19967-20025. PMLR, 17-23 Jul 2022. URL https://ppt /proceeeiccce.c.s.. mlr. wwt.ppeessvvel..1 shw2cchhhccccc2c html.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy P. Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362, 2018
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates, Inc., 2020. URL attps://prooeeedngg neurips paf.r.ddf..
cc/paper www..pe.ccpppppee....eeeee
/file/
R. Sutton and A. Barto. Reinforcement Learning: an Introduction. MIT press, 1998.
Mohammad Sadegh Talebi and Odalric-Ambrym Maillard. Variance-aware regret bounds for undis-
counted reinforcement learning in mdps. In Algorithmic Learning Theory. pp. 770-805, 2018.
Jérôme Taupin, Yassir Jedra, and Alexandre Proutiere. Best policy identification in discounted linear MDPs. In Sixteenth European Workshop on Reinforcement Learning, 2023. URL https: //openreview nwt. htte//oorummnddS000ggaaae
Daniil Tiapkin, Denis Belomestny, Daniele Calandriello, Eric Moulines, Remi Munos, Alexey Nau- mov, Pierre Perrault, Yunhao Tang, Michal Valko, and Pierre Menard. Fast rates for maximum entropy exploration. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 34161-34221. PMLR, 23-29 Jul 2023. URL https::/ttooeeeeee http://pooceeee.eceeee///o/ mlr press/ w22///taakiiaaa
A.B. Tsybakov. Introduction Nonparametric Estimation. Springer Series in Statistics. Springer New York, 2008. ISBN 9780387790527. URL https: //doi wt..ooog/..././/.. 111700000107/1799.99
Sara A van de Geer. Empirical Processes in M-estimation, volume 6. Cambridge university press, 2000.
Dirk van der Hoeven, Nikita Zhivotovskiy, and Nicolò Cesa-Bianchi. High-probability risk bounds via sequential predictors, 2023.
Ramon van Handel. Probability in high dimensions. 2016. URL htttpp://webbmaaee wrtth.prinneto... APC5m00 pdf.
14

---

Published as a conference paper at ICLR 2024
Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin A. Riedmiller. Leveraging demon- strations for deep reinforeement learning robotics problems with sparse rewards. CoRR, abs/1707.08817, 2017. URL http://wwvvv..oaaaaaa/.// hatbs//177708811
Roman Vershynin. High- -igensiinl probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Rémi Munos, and Matthieu Geist. Leverage the average: An analysis of kl regularization in reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 978 77781713822 13829546.
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF more difficult than standard rI? CoRR, abs/2306. 14111, 2023, doi: 10.48550/arXiv.23065 14111. URL https://doi. http://////oi.oo11..4884885 arXiv. 2306 14111.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. J. Mach. Learn. Res., 18(1):4945-4990, jan 2017. ISSN 1532-4435.
Tengyang Xie, Nan Jiang, Huan Wang, Caiming Xiong, and Yu Bai. Policy finetun- ing: Bridging sample-efficient offline and online reinforcement learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27395-27407. Curran Associates, Inc., 2021. URL https: https://procceeding neurips newppps.ccpppperr files files/papeeseee /ttpe//pppppeee//ii/... 601eaa8aaab222.77666667661633883 Paper. pdf.
Yichong Xu, Ruosong Wang, Lin F. Yang, Aarti Singh, and Artur Dubrawski. Preference-based reinforcement learning with finite-time guarantees. In Proceedings of the 34th International Con- ference on Neural Information Processing Systems, 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 97819 9781713829546.
Yuhong Yang and A.R. Barron. An asymptotic property of model selection criteria. IEEE Transac- fions on Information Theory, 4411)95-1166, 1998. doi: 10.1 SWA.118..5993....
Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double vari- ance reduction. M. Ranzato, Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 7677-7688. Cur- ran Associates, Inc., 2021, URL https: / h/tp://ee/noccceeeeeeeeeeeeeee
files/lleppeee /
pdf.
Andrea Zanette and Emma Brunskill. Tighter wroblem-dependent regret bounds in reinforcement learning without domain knowledge using value function bounds. In Proceedings of the 36th International Conference Machine Learning, (ICML), 2019a.
Andrea Zanette and Emma Brunskill. Tighter
regret bounds in reinforcement learning without domain knowledge using value function bounds. In International Conference on Machine Learning, 2019b. URL https:///// / arxiiv. .arivvooree org.oogg//.. /9901.91..00 .02200.010...1 pdf.
Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lec, and Wen Sun. Provable offline reinforcement learning with human feedback. arXiv preprint arXiv:2305. 14816, 2023a.
Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D. Lee. How to query human feedback efficiently in rl? CoRR, abs/2305. 2023b. doi: 10.48550aaxxx.22355.8800 URL https: //doi . org/10. 48550/ arxiv 2305 18505.
Tong Zhang. Covering number bounds of certain regularized linear function classes. Journal of Machine Learning Research, 2(Mar):527-550, 2002.
Tong Zhang. From E-entropy to KL-entropy: Analysis of minimum information complex- ity density estimation. The Annals of Statistics, 34(5):2180 - 2210, 2006. doi: 10.1214/
009053606000000044,, URL htRps: :/doii org/10. 12100...1114410 1214/09003660000000
15

---

Published as a conference paper at ICLR 2024
Y. Zhao. D. Zeng. M. A. Socinski, and M. R. Kosorok. Reinforcement learning strategies for clinical trials in nonsmall cell lung cancer. Biometrics, 67(4):1422-1444 2011.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feed- back from pairwise k-wise comparisons. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learn- ing Research, pp. 43037-43067. PMLR, 23-29 Jul 2023. URL attpss:/procceddiin mlr. press/ 2202/2hu222 html.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei A. Rusu, Tom Erez, Serkan Cabi, Saran Tunyasu- vunakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and imitation learning for diverse visuomotor skills. CoRR, abs/1802.09564, 2018. URL http:/ http///axvvvvv./ www.:///gg//.../////// 09564.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. URL /tt////xii..or/a////9//////// 08593.
16

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
A Notation
18
B Behavior cloning
20
B.1 Proof for General setting
20
B.2 Proofs for Finite setting
.e..............................
21
Proofs for Linear setting
22
B.4 Concentration Results
23
B.5 Proof of Lower Bounds
27
B.6 Imitation Learning Guarantees
31
C. Proof for Demonstration-regalariee RL
35
Best Policy Identification in Regularized Finite MDPs Preliminaries..
36
D.2
Algorithm Description
36
D.3 Concentration Events
38
D.4 Confidence Intervals
39
D.5
Sample Complexity Bounds
40
E
Best Policy Identification in Regularized Linear MDPs
45
General Properties Linear MDPs
45
E.2
Algorithm Description
................................ ................................
45
E.3
Concentration Events
................................
47
E,4
Confidence Intervals
48
E.5
Sample Complexity Bounds
50
F
Demonstration-Reuune Regularized Preference-Based Learning
53
F1 Maximum Likelihood Estimation Reward Model
:e..............................
53
F2 Properties of Bracketing numbers
56
F3 Proof for Demoostration-regulaizz RLHF
57
G Deviation Inequalities
60
G.1 Deviation inequality for categorical distributions
60
G.2 Deviation inequality for sequence of Bernoulli random variables
60
G.3 Deviation incquality for bounded distributions
60
G.4 Deviation inequality for vector-valued nelf-normalized processes
60
G.5 Deviation incquality for sample covariance matrices
61
H Technical Lemmas
63
H.1 Counts to pseudo-counts.
63
H.2 Counts to pscudo-counts linear MDPs
63
H.3 On the Bernstein inequality
64
H.4 Change of policy..
64
17

---

Published as a conference paper at ICLR 2024
A
NOTATION
Table Table of notation use throughout the paper
Notation
Meaning
S
state space of size S action space of size A dimension linear MDP
A
H
length of one episode initial state stopping time
$1
T
trajectory space, T (S x A)
E
desired accuracy solving the problem desired upper bound failure probability probability transition reward function
8
(s
Th (8,8,))
VT VA
value of policy and optimal value
QN, O
Q-value of policy т and optimal Q-value regularized value of policy æ and optimal regularized value regularized Q-value of policy and optimal regularized Q-value space of all policies and space of policies on step h space of all policies and policies on step with minimal probability y
V
G.A. F.A.A
Q F.A.A. R.Abb
II,
II, II,
DE
expert dataset of size NŁ; DE - ( (s). aj. Sy an), i E [N expert policy
m
n E.x
k-greedy version of the expert policy
EB
sub-optimality gap of the expert policy: Vi*( - Ví"" (s1) S sE
BC
behavior cloning policy
Rr
regularizer for behavior cloning
F
class of policies for behavior cloning
dz
covering dimension of one-step policy class for behavior cloning
state that was visited at step during i episode
a
action that was picked h step during episode true reward function in preference-based model link function, see Assumption 4
<
linearity measure of link function, see Assumption 4 sampling policy for generation preference dataset preference dataset of size NRM, 'RRM - {(Tb) T ot)}
S
DRM
C, R BC)
hingle-polcyy concentrability coefficient, see (20)
G
class of trajectory rewards for reward modeling bracketing dimension of the induced preference models
dg
π ILL
policy for BPI with demonstration policy for preference based BPI with demonstration
π RRLHF
Cle, NE, 8)
sample complexity for BPI with demonstration
C(
sample complexity for regularized BPI non-negative real numbers positive natural numbers
R
N
[n]
set (1, (1,2....,,7) Euler's number
Aa
1-dimensional probability simplex: Aa & {o R E/ 2j =
Ax
set of distributions finite set X': Ax A clipping procedure clip(æ, m, M) max(min(s,, M),m)
clip(a, m, M)
Let (X, X) be measurable space and P(X) be the set of all probability measures this space. For p E P(X), we denote by E, the expectation w.r.t. p. For a random mapping & X + notation { p means Law(E) p. For any measures p.q q P(X), we denote their product measure by
18

---

Published as a conference paper at ICLR 2024
p 2 9. We also write Egp instead of Ep. For any p.g E P(X), the Kullback-Leibler divergence
between and is given by
KL q) = Epllog d2l dy
p < q: otherwise.
+00,
For any p E P(X) and f x + R, we denote = B I. In particular, for any E Aa and f (O...., d) + R. we use pf = f())p(E). Define (fr = (xr=p[(((8") (pf)². For any (s,a) S. transition kernel p(s,a) a) P(S) and f: S + RR define pf( Ep(s.a)f)) and (arplf((s.a) = Varp(=.a)]f]. For any $ E S. policy T(s) E P(S) and f: A R. set fs()) = Banx(t)]S(ss, a)] and (s))
p(f
(a) For a MDP M, policy conditional
and sequence of function (fr, h [H],, define E, E. f(sw, on expectation of D >m f(s,,ax) with respect to the sigma-algebra Fn = o [[(sw,an)) z h). where for any h E [H], we have on (55),, 8141 ph (sh,, an).
We define trajectory Kl--divegeence between two policies क ( [Th]iE(H] as follows
H Σ (SN,
KLeraj (T, T') ET
We write [(S, A, H) = Og((S A, (,,,,)) if there exist So. Ao; Ho, Eo, do and constant Cf such that for any S So, 2 Ao, H Z Ho, < En, < do, [S, A, H,T, S Cf.9 g(S, A, H,T. We write f(S, A, H, Ölg(S, &,,,,, if Ct.9 in the previous defini- tion is poly-logaritmmic in S, A, H. 1/6,
For any symmetric positive definite matrix A, we define the corresponding A-scalar product and A-norm as follows
(α.1)) = (x, Ay), Notice that ||A112 then | Vell=l12
||]|a== = (o,
Coverings, packings, and bracketings A pair (X' p) is called pseudometric space with metric p: X X' + 1 if p satisfies p(x, x) = 0 for all x X, p is symmetric, that is, Va, y E X p(ar, y) = p(y, ..), and p satisfies triangle inequality Va. v, p(x,,) + p(u, =) 2 p(x, z).
Definition 6 (s-covering and packing). Let (X' p) be (yseudoJmetriy space with a metric p: X x X R. The e-covering number ((, X p) is the size of the minimal s-cover of (X, p), that is,
min {|x] Vy X'En X plu, x) $e).
XCX
The e-packing number P(e, X,P) p) is the size of the maximal e-separated set of (X' p).
P(e, X, = max X|: Va + y X p(I,)) > e).
XCX
Definition 7 (s-bracketing). Let F: X + R a function class endowed with a norm ||- Given two functions l, u: X IR, bracket [e, u] is set of all functions f E F such that {(x:) f(x) s u(x) for all x E X. A e-bracktt a bracket [e, uj such that ||e- ul S e. The e-bracketing number (e, II-II) is the cardinality of the minimal set of e-brackets needed cover F,
(F,, II-I) min(l Af e J 3/6, u] N ((x) S f(x) S u(<))x X, ||e u/l S e).
19

---

Published as a conference paper at ICLR 2024
B BEHAVIOR CLONING
In this appendix, we gather the proofs of the results for behavior cloning presented in Section
B. PROOF FOR GENERAL SETTING
In this appendix, provide the proof of Theorem 1.
Theorem (Restatement of Theorem 1). Assume Assumptions 1-2 and that 0 S RA(AA) S M for all h E [H], for any policy π Fhh Let x æBC be solution to (1). Then with probability at least the behavior policy x BC satisfies
6d (log( /AAy A K)) log(2HN 1og(2HN*Br/(66) Rx/(16))
2HM
18k
KLuraj (Tueaa(T51IBB)
NE
NE
1 K
Proof. We commence by defining the one-step trajectory KL-divergence as follows:
KT 1hhh HC)
E
log
Th
(an (4h)
BC (anlsh))
Th
In particular, by the linearity expectation, the following holds
H
KLuraj (5t (|7
Σ
KLuaa (T (TT
=
h=1
Recall the definition of the x-greedy version of the expert policy
F. R (als) = (1 x) (als)
= (1 (1-k)
(alss) + (1 - -))
Next, we can decompose the one-step trajectory KL KL-divergence as follows
E. h BC (ap (h)) A
(an (a) ($x)
# (anlsn)) (8h)
KLtraj (TH BC) E
log
E,E log
E.r. h (ap (8h)
For the second term, have
x F
F(anlss)) (ap (sn)
] E log
Th (anlss
1
E log
- -log(1 - -R) S
E. Th
Th
(an (8)) + K/ (A(1 x))
1
where the last inequality follows from the fact that (1 x) log(1 x) > -r for any < 1, by convexity of the function æ + x log z. Next, we decompose the smoothed version the one-step trajectory KL to the of stochastic and empirical terms.
Th (anlsh)
]
N
Th E. (anls))
) log
Fh E. (at BC n
E,F
log
E
log
BC Th
(an (anlsa)
NE
(anlas))) 7
BC 1s
1=1
N
T
E.ss
(a
log
NE
BC (a (o(ls)))
To upper bound the first term we apply Lemma and obtain with probability at least
N
E, Ei (sa)
] log (an (sn)))))
E.N Th Th
(af.l)) (a s9))
E ExE
log
NE
BC Th
2 log(e e² /7) KL (T P1 PI[BBB)) d d(log E R2 R//7) + (og((1)))
NE
5(log(Ae) /ea/ A k)) d><llg(2N Rx/7) (1/6)
8x
3NE
20

---

Published as a conference paper at ICLR 2024
To control the second term, first notice that since JF has product structure, then by simple observation
H
TTh (
arg min
Σ (T(
va E [
Th = arg min Cn (T²)
TIE TH FFH
h=1
m.
for any functions (Ch M=1 the MLE estimation (1) implies
N
Th
BC
E
arg min RAEF
Σ
log
Th (1111))
1
+ Ra(Fn)
1=1
therefore the following holds
N
E.N
(a
(ss)
N
log
#
S M +
log
+ Rr (T BC)
RC (aj (s%)
Th (o.(aalls))
t=1
N
log
+ (T F.R)
$ M.
B. (ss))
Thus, we have
2 loo(e /7) KL (T ( dr (log(2N + (1/0))
KTuaaj ( ( BC
NE.
5log Ae /Ay A K)) ( (2. Rx/7) + log (1)0)))))
M
9K
+ 1 1-K
3NE
NE
This means that V KL ( || BC) satisfies quadratic inequality of the form < (a² + x²) / 2, we further have x² < a² 2b. As result
x² + b. Since
BC) 6 log(Ae3 / (Ay A x)) log(2N E R5 (17))
2M
18k 1-K
KLmmaj ( (TT))
NE
NE
To conclude the statement, we apply a union bound over h E [H] and sum over the final upper bound.
B.2 PROOFS FOR FINITE SETTING
We recall that for finite MDPs we chose a logarithmic regularizer Ra = E.o log(1 / (als)) and the policy class F { E II TT (als)
1/(NF + +)). One can check that Assumptions 1-2 holds for these choices and that 0 < Rr S SA log(A), Then we can apply Theorem to obtain the following bound for finite MDPs.
Corollary (Restatement of Corollary 1). For all NE 2 A, for function class F and regularizer
(Ra) defined above, with probability at least 1 1-6 5,
6SAH log(2c* BB) logl (2 (NE)/10)
18AH
KLuaaj (T
BC)
NE
NE
Proof. Let us start from simple observation that F, S A a subset of a unit ball norm. Therefore by a standard result in the covering numbers for finte-dimensional Banach spaces (see ie. Problem 5.5 by van Handel (2016))
log,W(s, Fr. (1-)) S log((3))
Thus, the parametric classes (Fi (F satisfies Assumption 1 with constants d, SA, R 7 E A). Next notice that for any expert policy, Assumption is satisfied with A/(NE + A) for this parametric family. Thus, we can apply Theorem and get
6SAH log((NE + A)e3) B + A))8)
KL (1 BC)
NE
2. (A)
18AH NE
NE
By upper bounding the first and the second terms under the assumption N > A we conclude the statement.
n
21

---

Published as a conference paper at ICLR 2024
B.3 PROOFS FOR LINEAR SETTING
We start from a natural example when Assumption is fulfilled and the sub-optimality error ég is small.
Lemma 1. Assume that the MDP M linear (see Definition 2) and consider the regularized MDP with uniform policy {(als) = L/nif [A] and with coefficient X (see Appendix E for more exposition), Then the optimal regularied policy z satisfies Assumption 3 with constant R = Hvd/X. Moreover, this policy is XH log(A)) -ppttiall.
Proof. At first, by Proposition 2, it holds that for an optimal policy - there are some weights w such that Q7, a) (vis,, a), w and, moreover, ||=]] H vd.
Then we notice that from the regularized Bellman equations it holds
T (als) = arg max π (. - A AKL(T] (T]][/(f((()))
QEAA (s) KL(T (nif(()))) }- A
exp((v((s,)) a), tw()) Z(s)
arg max
Therefore, Assumption 3 is satisfied with R = H HvdAA for x ग
To verify the suboptimality of this policy, we notice that न satisfies
arg max VT* (T
sell
therefore
(81) - A XKL (x [T}} & ti (81) - -KK (aaea)(T@z,. xl
Viiss) - 1/2 ViTia(ss) AH log(A).
Next, we provide the result for linear MDPs under Assumption 3, using the parametric assumption given in (2).
Corollary (Restatement of Corollary 2). Under Assumption 3, for all NE 2 A, for the function class F defined in (2) and regularizer Rr, = 0, for all E [H]. with probability at least 1 - 8,
6dH ( log (N)
18AH
KLtraj (T P [C)
NE
NE
Proof. We start by checking that Assumption holds. By construction of FA in (2), we have
inf inf E.i (sA)ESXA
Th Z NE + A
Next, we have to consider the covering dimension of the hypothesis set. First, we notice that for any two policies Th. ph Fr we have
(a(s))) 1h (a|s) = [(1 (als) (1 (o1-x))kk k) (1- (a]s) ph (a|s))
where Th. 1/ F, for Fy defined as follows
F Ta(als) =
EA' exp (s,, a')Ta)) [|wwlla s RR
exp(e((s, a''
Thus, is sufficient to compute the covering number for J/.- Let us define
( wp) = exp((((s, w),
Z(s, Σ
( w)).
aEA
22

---

Published as a conference paper at ICLR 2024
Then let wh, w/. be weight vectors that correspond to xy, and μl respectively. Then ((a wn)
D(als, w/h)
[aa(s)) - / (a|s)]
wn) (s, w/))
L[aas, wh)
2(aas, w/))
( wxn)
(ss, wh)
Z(s, w/h)
((s, wp)
((a s, wh)
((a w(/)
(aals, w/y))
2(s, 2/h
(ss, wh)
(a)s, wa)
Z(s, wh)
((s, wv)
Next, we analyze both terms separately. For the first term, have
( w(,)
exp( w/ waHH
(als, wh)
We notice that the absolute value of the expression under exponent is upper-bounded by |m2y w/ w/,1 ll. Let us assume that [|wn w/ l12 S then by the inequality |1 1- S 2| for any |ar] S 1, we have
1 (aass, w/))
< 2/ w/, 122-
(3)
(a wn.)
For the second term, we have by the definition of the normalization constant
7(s, w/)) 1
Ee (a'ls, wx)(1 (a''s, wx,)) / / (a'|s, wx)]
(ss, wh)
(ss, wh)
Eg D(a'ls wa) (a'(s Is, w/ / wh)l < 2 2|| w/ll2.
(ss, wh)
where in the end we applied (3). Finally, we have, for any policies TíK and n4 such that the corre- sponding weights wa and w², satisfies w w/ 2 1, that
(als) (a(ass) (a)ss (als)) = 4|| w/, 1l2-
(4)
Now construct e-net for (0, 1). Let Ne/a (W, 11-112) be e/4-net in the space of weights W = {w,, E Rdd ||w lla S R). It satisfies (see i.e. van Handel (2016))
log.N(e/4, W, I1-112)1 I dllo((22/ss.
Next, we show that policies with weights that correspond to a covering of size (Nee44 Wh, 11-122) forms e-net in Fi. Let Th E Fr be arbitrary policy with parameter w/. Let w/ be in the covering of size (e/44, W, I1-11)) be a parameter that satisfies [|w, w/ll2 S e/4 < 1. Let us fix 14 as a policy that corresponds w/, Since I|w w/ll2 S (4) is applicable. Thus
sup (als) pm (a]s)] 4 mm w/,12 S. || - -illoc (s.))SxA
Therefore, policies that correspond an E/4-net wa form an E-net in and we have an upper bound on the size of the s-net. As result, F, satisfies Assumption with a dimension d,- d. a scaling factor R2= = 12R and 7 = 1/(NE + A). Additionally, by construction of Fn. and Assumption 3, the last Assumption 2 holds with Theorem and obtain with probability at least 1
A/(NE + A). Therefore, can apply
6dH (log(e² (N + A)) (NE + A)R/O))
18AH NE
KLyraj (TT B BC)
NE
Using of A < R concludes the statement.
B.4 CONCENTRATION RESULTS
In this section, we state important results on the concentration of the stochastic error for the risk estimates.
Recall the definition of the x-greedy version of the expert policy as follows
"h (a)s)))) = (1 x) (als) + K = (1 - k) 7 (a]s) + (1 K)A
A
23

---

Published as a conference paper at ICLR 2024
Lemma 2. Let n. be fixed expert policy. Let (6 oi) be i..d. sequence of state-action pairs generated by following the policy NE at step h. For T E (0,1/A) let n policy such that for all (8,a) a) E S x A it holds (n(als) 2 7. Then for any ô E (0, and any K. < 1/2 with probability at least 8
Tho (aj Th (o) o%)
(an"(an(ss)
2 log /n) KLemg (
(T)) log(2/6)
log
F. log
N
Tu (a.((ass))
N
2 log(Ae* /(Ay Ax) log(2/6)
3N
1 1-
Proof. As first step, we can apply Bernstein inequality
"A (a
N
E.
E.n (an (anl(a [an) Thh (an (8h)
2Vrr R log
log(2/6)
log
EgE
log
2(1og v Vlog(A/x) log(2/6)
3N
Next, want to upper bound variance in terms of KLrraj [I square root variance terms of the second moment and Minkowski incquality
(TA.. We start from bound of
Th E.N (an sth)
Th E.r (anlan) Th Tn.(nnlsa (8n) ))
27
Var log Ta (an(sp))
Eg
log
R.n (a|s) 27
nh (a TTn (a(s)
E E
log
+ log
E (a1a
A (anlas)) Ta (usl sph)
27
log
(AA
log(1 - kn) m))
By
log
1 (1 ka
((B)
Term (A). The result below directly follows from Lemma 4 of Yang & Barron (1998). However, for completeness, we prove here.
First, we notice that
T T (h Th (a|s))
(A) Fag
Σ
TH (alsh) log?
LaEA
To analyze this term, define an f-divergence (Sason & Verdú, 2016) for function, f as follows
ah (als)) (a(as))
(T(T)()) = Σ
TA (als).
aEA
In particular, (s)) = DD for g(t) = t flog + (1 t) and, moreover for s(() = (1)
T (als) Th (a)s)
E (als)
log
aEAA
Then we notice that and are non-negative function and, moreover, its argument / takes values in (o, 1) 11,1 since for t = 1 both functions are zero. First, we analyze the ratio for f and g for any E (0, 1)
f(t) = blog + (1-t)
tlog2 (t)
r(t) g(t)
24

---

Published as a conference paper at ICLR 2024
To bound this function, let prove that is monotone for all t e (0. 1)
50
(t) = ltg(t) ((t + 1) log(t) + +2(1 1))
2 0.
(tlog t + (1 -t)]
Thus for any E (0, 1)
r(t) - lim tlog2(t)
t-1 logt + -t) = 2.
Next, we analyze the segment E (1, 1/7l.
flog²(t)
r(t)
t flogt t + (1 - S log(t) + log(1/7) + log(c>/7)
since
t (t) & (0 (((--t) =
Vt> > 1.
Therefore we have for any t (0, 1) 0 (1, 1/7| and as simple corollary
f(t) S log(o g(t) = (T) I log(c²/>) (I(TE)))]TA(S)))
Finally, we have
(A) log(e lo) (T (sw)) = log(e? /n) KT (m (
Term (B). We can rewrite this term follows using inequality (a + b)² 2aa 26²
B S 2
E
(alss)) log² [1+ (1- AT (an
+2 K
Next we analyze the function g(x) = x log (1 + 8/2)). Its derivative is equal to
g'(x) = log 1 + - log 1 +
E
2e
I t+
Since E > O. we can define x* as a root of equation g'(x) = for x maximum of gxx), thus for 0
0. Notice that will be
g(x) S g(x*) x² (log 1
2 ))
4
4ga
x²
S 4s.
(a+ + E)
x +
Therefore
8K
2
B
Final bound on variance
Combining these two bounds, we have
E.N (an (sa) Th
]
Var.e log
S (log(e8/7) KIeraa + / 8x/(11 - x)+ 2x2/(1 x)
na (an//11 (ss)
Using this bound on variance, can bound the main stochastic term
Var log
m1 "A (onlow) (aa
log(2/6)
2 loo(e? (2/( KL ( (T) 1og(2/8)
N
N
(4n/(1 - k) + R² (11 - k: log(2/6)
+ +2
N
Next, we use an inequality 2ab < a² b2 obtain
2// (Ak/(1 x)) x2/(1 - x)2) 1og(2/8)
< (4k/(1-k) + #*² x2/(11 k))2= 2og(2//)
N
N
Finally, since k/(1 x)) < conclude the statement.
25

---

Published as a conference paper at ICLR 2024
Next define ,, set of all one-step policies Tn (a|s) such that
inf Th (a 2
(a.0)EXAA
This set forms a metric space with a metric induced by foor
IT lloo
sup (s.a)xSxA
[TA T/. (a|s))
Lemma 3. Let F be sub-space of If with an induced metric, such that it satisfies for all (0,1)
log [e(,FF |1:11o))] S d dllo( dl/o(R/)).
for some positive constants R, d > O. Then with probability at least the following holds for all Th E FF simultaneously
1
Th (a
E.
E. "h (an (sh) Thh (an [sp]
log
E
log
N
Th (a
2 log(e? /v) KLrraj (T Th)
N
5 (log(AAe) / A A))) d(log((2 log(1/)))
8k
3N
1 1-
Proof. Let Ni be minimal E-net of F for that will be specified later. Combining Lemma 2 with union bound over N₁ we have for any nTy NÇ with probability at least 8
E.N (af.1sh) FA
E.s. Th (ap (ap, (8%)
log
E
log
N Aml
T (a
(ap (8))
log(e?
(on(22/7) (7) KT KTrraj (M (TT)) dlog(22/(eo))
(5)
N
2log Ae /(Ay A x)) dlog(2R/(s0))
5x
3N
1-
Next, we select an arbitrary policy Tn E F and let oh No be e-close policy to πy. Then
N
T F.
(a%,15%)
Th F.N: (an((a))n)) 99))
log
E
log
(a
(0%)
Th (un|s1
t=1
1
N
log
Th F.K
(at, (0%)))
EF
log
Th
F.N
(an (8h)
N
Tr. (a (11))
TT (ap (sh)
The (1)
Tí. (an, (sa))
Th. (an (sh)) 1){
log
E
log
Th (a (5)
We start from bounding the second term, which could be done follows
T (s n ( (s{)
T/ (aa [sa)
7 (a)s)
log
E,e log
2 max 8,a
log
N
TT sph)
TT (a)
Next, we use simple inequalities
T (als) = log 1
* (als) - (a|s)
(als) - n (a|s)]
s
log
TA (als)
(a)s))
and, in the opposite direction, we can use the same reasoning
T (als)
n& (al
> 8
log
log
(a|s)
T/. (a|s)
26

---

Published as a conference paper at ICLR 2024
Thus, applying (5) for the first term obtain
N
E.x A
(af lsh))
B.K (app 8hh)
Th (a(an 88)
log
E
log
(a (sf))
log(e? /7) KLuraj ( (TT,) (2og((2
N
2 2llog Ac (AAy A Ak) dlog(2R/(s6))
5x
1- + 22/77
3N
Next, we use similar inequality to obtain
log T % (an
(TLtra(TTTi)) ExE
(an) 9h)
n (an (9h)
Tyt (ax (sh). ]
KLuraj ( m() + E
log
S KTaraj ( + &
Finally, applying inequalities va bb+b= S va + Vb and 2ab - a²
E.x "h
(a (s,, (a a,,, (s%)
Th B.r (ap (sa)
log
E log
(ana sn)
2d 2d e2/ /7) KT KTaraj ( [ log (2)g(2/((0))
N
d looge"//1) /7) log(2R/(EE)) +
N
2(log (Ao3) (llg(Ace/(Ay/) ( dlog(2R/
5x
+ 22/7.
3N
1-
By rearranging the terms and taking = 7 x/(k= - k) we conclude the statement.
B.5 PROOF OF LOWER BOUNDS
B.5.1 GENERAL SETUP
In this section provide lower bound on estimation in KL-divergence using framework of Chapter 2 by Tsybakov (2008). Our goal is to obtain a lower bound on minimax risk that is defined follows
inf sup E,,.... [Ktraaj (+)),, WEF
where infimum is taken over all estimators that map the sampled trajectories (71...., TN) to a policy from the hypothesis class F 2 Fi x ...
Let consider specific type of MDPs where the transition kernel Pı (s, a) does not depend a state-action pair (s, a): V(s, a, h) E S x A x [H], VA E Fs (A]s, a) = (A) for fixed measures nh. In particular, for h 1 always have μlg. da, Dirae measure at initial state s
Then we define over the space of all policies the following specific distance defined through the Hellinger distance
Ph: (TT, m//)) / VEampan (T m)),
(T (VVAA((s)))
Σ aEA
2
27

---

Published as a conference paper at ICLR 2024
and we define the following distance for the space of full policies (the triangle inequality follows from Minkowski inequality)
p(
Pi (T)T)))) x/)-
(6)
Next we impose the following metric-specific assumption for our hypothesis classes Assumption 5. For all h E [M] a of the function class Fh with respect to the metric satisfies
Ve E (0, 1) log Ple, Fh Pn) 2 d, log( R//s)
for constants da 2 0 and R > 0. In particular, Lemma implies that
H
log P(e,
Σ dn log log (R/5). h=1
Theorem 4. Let Assumption holds and let us define D En=1 dh. Also we assume that for any F it holds (s, 2 R for y (0. 10)1 Let us assume D 5 and e² DDDRR. Then the following minimax lower bound holds
D
min max (IKLITa((T(T)) TEJ
16N log(e22 /7))
Proof. First, notice by the first part of Lemma 7
log(c? //)N
log(c? /T)N ( (,)
min max TEF
KLtraj (T]|
min max E * REF
where the expectation taken with respect sample TI. T....., TN. Next, we can follow the general reduction scheme, see Chapter 2.2 by Tsybakov (2008). By Markov inequality
min max E REF
I lolooee/// /7)
p #) 2 min max # REF
p( 2
D
4N (Nlog(e²/>))
Next, we use the reduction finite hypothesis class. Define < D/ (4loogee//)))) and take P is maximal (-separated set of size M + 11 = P(C,F, p) and enumerate all the policies in it as To. ...,...... TM-
Therefore we obtain
D 4N log(e2/n)
min max R =EF
TNME p(T, 2
V
D
min
max Jd
P TN
P(TT, +t) 2
4N log(e2/n)
Let us define v0 arg min MM..... M p(Tjj R). Then have that if a/ j. then
2p(T,, x) 2 p(TT, Æ) + p(T #) 2 TT-)
Since j * n², then by definition of Ç-separable set we have p( +) 2 C/2. As result
D
min max P PramNTNNT jE(O....,MM
p(TT, 7)
4N log(e2 /7)
min
max IP PT....TTTT [v=* # j].
je (E.
28

---

Published as a conference paper at ICLR 2024
Finally, taking infimum over all hypothesis tests obtain
D
min max TEJ
inf max Py[y + j] - Pe.M- V O M
AN 4N (og(e((22/))
To lower bound the right-hand side, apply Proposition 2.3 by Tsybakov (2008). Notice that the maximal e-packing is g-net (see Lemma 4.2.6 by Vershynin (2018)). Therefore, by Lemma 7 we
have
M
M
N
KL(P TMT,; |P P (n.o)
KT (Ti
M
M
1=1
<nlog(e? (olog(e?/>)
1
Σ
p² (T))
5D a ax
M
Thus by Proposition 2.3 by Tsybakov (2008)
TM
TM 1+ + vo.
Pe,M sup oct<l
log(T)
Next, select 7. in way such that
a4 + aa./2
1 2 = log(T.) = - (a. + Va./2).
log(T.)
Therefore we have
1 exp(log(M) - 1/2(a, + vx./2))
Pe.M 2
2 + exp((lgg(M) 1/2(o,, + v2,
Notice that the function f(x) exp(x)/(1 exp(x)) monotonically increasing, therefore it is enough to bound the expression under the exponent from below.
Let us assume that a. 2 = 5. Then we have
2 v2
log(M) - 1/2((a* + va./2) 2 log(M +1)
a.
log(2)
4
Alog(e2 /
D Dog
R
D
D
4log(ee (T)(e
log
-2)
D
To show that the expression above non-negative, is enough to guarantee
log(N) + log( /R2 /D) 2 = N e
Under this condition we have Pe.M 2 1/4 concluding the statement.
B.5.2 FINITE MDPs
For the case of finite MDPs, we additionally specialize the distributions f4, as uniform over S for all h > and 141 = Es.- Lemma 4. Let AA, E En 2, 1, } with 7 1/A. Then have for any a E 10
log P(e, AAm dx)) 2 (A 1) log((1 A7)/(2a)).
Proof. Let S = E Σラ o = 1, > 0}. Then there is mapping yo: [8.11-112) + (AA, dz) that defines an isometry between these two metrie spaces:
Va,y E S || vl12 dx(p(x), plx)) Va.
29

---

Published as a conference paper at ICLR 2024
where the square root is applied component wise.. Therefore, it is enough to estimate the packing number of the preimage of A A.y that is defined as follows (1) {xr E R Σ x 1, 32 2 v7} with the same Euclidean metric.
The next step to proceed with shift x + x+ VT that will be isometry between (1) and So - A7).
Next, lower bound the fg-distance the sphere by the Ez distance the first A coordinates and therefore it is enough to consider the packing number of S1 - A7) = ((0, 1) n
E Σ vi S }
Finally, we apply the volume argument. In particular, is enough to compute the volume of 5 Ar). To do it, notice that can represent the ball of radius Ary by 2 copy of 5% 186(1-A1) Ay)). Thus
(0 -A))) (1 - A7) 11-17)4-1
vol vol(BB
24-1
Finally we have by Proposition 4.2.12 by Vershynin (2018)
- 11 Aw
A-1
P(E,AA,,, dn)l
2z
Lemma' 5.
Let (x be metric space such that log iP (K, p)] 2 dloo((//)) d 2 1. and define the space X the following metric p(x. y) 1/4 Σ p( Then
log x p) 2 dK/2 2 log( (8E))
Proof. Consider the maximal e-separable set P of the space K. This set could be considered finite alphabet of size q 2 (R/e)á. Let us consider the set pk as the set words alphabet P of size q of length K with Hoeffding distance. Then notice that if there is two words (I, p that have Hoeffding distance at least aK for some constant & (0, 1). then
Σ Ellenvi 2 ae. plz, K
Therefore, if we consider aK separable set in pk in terms of Hoeffding distance, it will be automatically ae-separable in the original space X To find such use the Saberr-Varshamor bound from coding theory. As result
P(as, PK ,), N
1
(1- 1/9)" (1/9)*-5
The denominator could be interpreted as follows: Let XI,..., Xx be Ber(1/g) random variables.
[aK] Σ
(1/0)=/ -/00
nn-1/
Σ x, 2 a)K
To upper bound the last probability can apply the Chermoff-Hocfiie theorem
P
x (1- 1/9)
< <exp(- - kk(1k alll/g) K).
1=1
Take a = 1/2, then we have
kl(1/2||L/g) log
1
log
{)
9
log(2) log !
1 2
2
2
Thus, we have since d 2
P(E/2, ,)
(o/g/// exp(dK/2 log(R. (4:)))
By rescaling conclude the statement.
30

---

Published as a conference paper at ICLR 2024
Corollary 4. Assume that y S 1/(2A). Let us define Fr. A and F = Fu. Then Assumption 5 holds with constants d₁. (A - 2 for all h > 1, dj (A-1) 1) and R = 1/32.
As a result, as soon as H 2 2, 2, HSA 2 40 and n 2 512c HSA the following minimax lower bound holds
HSA 128NV log c /n)
min max E B ~ KT KT ( (T| REF
B.5.3 TECHIICALL LEMMAS
Lemma 6. Let {(Ki, pi)).= be collection of relaxed pseudometric spaces that satisfy
VE E (0, 1) log P(e, X,, P:) 2 d; (.log(()) R/e)
for some constants 5. S d2, Then the product space X X1 X ... XK with a pseudometric
p(( ,K),, (,1,...,, 9k)) = v V pl (I Vi) satisfies
K
Ve E (0, 1): log P(e, p) 2
Σ d, (R/E).
Proof. Let , Pk be maximal e-separated set in the corresponding spaces M Then we want to show that the set P1 ... Pk also e-separated set the product space.
Let f x be two point in P. Then
K
x(x,x'= Σ
(Z 2E
since x and are different at least one coordinate. As result, we have
log P(e, (Pe Pi) Σ log Ple, Mic
Σ d log R/e).
Lemma 7. Let n. n' II, Let p be an averaged Hellinger distance distance defined in (6). Then the following inequality holds
(T KT KTnraj( log(e? (og(
Proof. It enough to show that for two measures p, E An such that min; pi/g: 2 the following holds
(p. KL(pllg) log(e? (p.
The lower bound holds by Lemma 2.4 of Tsybakov (2008), and the upper bound holds by Lemma of Yang & Barron (1998).
B.6 IMITATION LEARNING GUARANTEES
In this appendix, we present guarantees that give behavior cloning procedure in the setting of imi- tation learning for finite MDPs and compare obtained results to (Ross Bagnell, 2010; Rajaraman al., 2020).
31

---

Published as a conference paper at ICLR 2024
General expert Using Pinsker inequality in the space of trajectories (see Lemma 9) and the fact the expert policy Eg-optimal we deduce the following bound on the optimality gap the behavior cloning policy with probability at least 1 -
SAH NE
(11 - V9T (81) S S
We remark that we obtain similar rate as BPI, see for example Ménard al. (2021), where instead of observing N demonstrations we collect the same number of trajectories (observing also the rewards) by interacting sequentially with the MDPs. This seems bit counter-intuitive since we expect to learn faster by directly observing the expert.
However, the deterministic expert, we get an improved rate using the following variance-aware Pinkser inequality.
Lemma 8. Let and n' be arbitrary policies. Then the following upper bound holds
H
(T - VI (81) <
4EA
Var Q7 (sw)
KLtmj(T|TTT) + 4H KI Krrraj (T/T))
L==1
Proof. Let start from Lemma 11 and Lemma 32.
H
(F((8) - VI (s1) = E.
Σ
m T/1 (sn)
2H
< Ex
Var 17. Q7 (sn) (L((T((s)((((((s))) KL (1
H
2H KLuaj (T(I''),
<
2Eg
Var. Q7 (sn)
(Trraa(T(lTT)) +
where in the last line we have applied Bruchy-Schwartz inequality. Next we apply Lemma 33 and obtain
H
E
Var
Q (sh)
< 2E,
Var Q (on)
+ 4/12 (T(raiTT((TT)
By inequality va SS + vb for any a, we have
H
(T1))) - (8) (81)
4E
Var Q7 (sn)
(Luaa(T||TT) + 4H KL ( (T)
Deterministic expert If we assume that the expert policy is deterministic, for example, a deter- ministic optimal policy, then we can improve the bound on the optimality gap since the variance term Lemma 8 zero,
SAH2 NE
(s1) Via (s1) SE Ō
Ross & Bagnell (2010) also consider behavior cloning with a deterministic expert and provide bound in terms of the classification-type emror of the behavior cloning policy to imitate the expert
H
BC (81) S Sæ + ex(1 where en BC) (s1) V/
1
E
BC
1as 13
H
We can easily recover our bound on the optimality gap of the behavior cloning policy from their bound by noting that Bg(= (T H, Lemma 10 for proof. In particular,
32

---

Published as a conference paper at ICLR 2024
also remark that the bound scales quadratically with the horizon H but also linearly with the number of actions and states SA.
By comparing this bound to the lower bound in Theorem 1.1 of Rajaraman et al. (2020) that it is optimal in its dependence on S, H and N. Additional dependence on a number of actions comes from the fact that our behavior cloning algorithm always outputs stochastic policy and obtains additional dependence on a number of actions.
We would like to underline that the bound in Lemma directly does not give any insights the performance of the algorithm in the case of non-deterninistic optimal expert, whereas Rajaraman et al. (2020) provides OSH2 /N guarantees using non-regularized behavior cloning algorithm. connected to the fact that for the optimal policy, it is enough to determine the subset of the support of T (s) to achieve the policy with the same value.
Finally, we would like to emphasize that our approach directly generalized to arbitrary parametric function approximation setting whereas the approach of Rajaraman et al. (2020) could be applied only in the setting of finite MDPs.
B.6.1 TECHNICAL. LEMMAS FOR IMITATION LEARNING
Lemma 9. Let M A, (mm) (rk) 81) be a finite MDP and let and x' be two any policies in M. Then
(+ Vi' (s1) S H (=
Proof. Let us define the trajectory distribution (T) for T (81,01..... 8H, ax). Then by the chain rule for KL-divergence have KL(g" (1"""""""" [==
Since Th E [0, 1]. we may apply variational formula for total variation distance and Pinkser's inequality
-V (V((s)) - VT (T (51)
S
Σ
Ex Irx (8h, E Brrlrasss an)
h=1
HT xh)) H KLina (T]] 1/22
Let us assume that a policy π is deterministic, then
ex (
1
E Σ
+a x($w]]
H
LREA
Notice that
Σ (al m(ah))
Σ
2
GEAA
acA
Therefore this quantity could be decomposed as follows
1 H
ex (
Σ
(TV()())
Lemma 10. Let be deterministic policy and n' be any policy. Then
ex (T') < KT (
H
33

---

Published as a conference paper at ICLR 2024
Proof. If the policy is deterministic, then
log
log
Th (an(sh)
T (TT (BN))1)))
log
EGEA (d(ss) & (s
--oo(1 (V(T)())) (f (s))))
By inequality log(1 x) I for any TV(T1(*)); (8n)).
have (L((((sW)I(((s)))
The following lemma is known as cerformance-diffeeeen lemma (see e.g. Kakade & Langford (2002) for a statement in the discounted setting). We provide proof for completeness. Lemma 11. Let π and n' be arbitrary policies. Then the following decomposition holds
(fr(s) (s:) = Ex,
Σ
T/Q2 (sn)
-h=1
Proof. Let proceed by backward induction over h E [H]. We want to show the following bound
(s) - V (s) En
Σ
Th
xf =s
For 11 both sides of the equation above are equal zero. Let us assume that the statement holds for any h > h. Then we have
(s) V (s) (8) N' Q' (5) π Q2 Q7 + - T'1Q='(3 (s) Ex V ( Vi (sh+1) + [ - ('
(sn)(s) s].
By the induction hypothesis and the tower property of mathematical expectation, conclude the statement.
L
34

---

Published as a conference paper at ICLR 2024
C PROOF FOR DEMONSTRATION-REGULAN
RL
Theorem (Restatement Theorem 2). Assume that there are an expert policy T such that (81)- Vi" (s1) S eæ and a behavior cloning policy n BC satisfying VKLuag(7-l|mBC) S EKI. Let π Rl. be eRL-Optinal policy x-regularized MDP with respect nBC, that is, Va (81) aBc S snL- Then Rl. fulfills
(=1 1/= V/""' (81) S SE + ERL + AE In particular, under the choice X ERL/EkLI the policy a BC is (2ERL. + ee)-optimal in the original (non-regularized) MDP.
Proof. We start from the following observation that comes from the assumption on expert policy and definition of regularized value
(s1 V/"" (s1) S SE + VT" (s1) - V(*""(ss) S EE Va A, (s1) + KLuaaj ( (TTBBBB (s1)
RL 7bc A.1 (s1) AKLuaj (T (17 EE + ERL + AE
where in the last inequality we apply assumptions on BC and RL
Here for completeness we present the proof of Thcorem
Theorem (Restatement of Theorem 3). For all E (0, the UCBVI-Ent /LSVI-UCB--ER
algorithms defined Appendix D.4 /Appendix E.4 are (e, 6PC C for the regularized BPI with sam-
ple complexity
S2 Aa
d A
((=,))=0r
(finite)
C(e, 5
(linear).
Additionally, assume that the expert policy is sE =/22opttimal and satisfies Assumption 3 in the linear case. Let n. BC be the behavior cloning policy obtained using corresponding function sets based on UCBVI- /LSVI-UCB- with parameters ERL e/4, 8τL 8/2 and O(Nee/SS (H)) (e(Nee/ddF) for BPI with demonstration in finite linear MDPs and has sample complexity of order
described in Section 3. Then demonstration--eeguaaaaa
(e, (e-A)
53 A
C .0) =
(finite)
CEE,N 8) 5
(linear).
NEg
NE
Proof. The first part of the statement is combination of Theorem and Theorem 6. The second part of the statement follows from an upper bound on E by behavior cloning (see Appendix B.2 and Appendix B.3) and Theorem 2.
35

---

Published as a conference paper at ICLR 2024
D BEST POLICY IDENTIFICATION IN REGULARIZED FINITE MDPS
In this appendix, we present and analyze the UCBVI-Ent+ algorithm for regularized BPI.
D.1 PRELIMINARIES
We first detail the general setting of KL-rregllarized MDPs.
Given reference policy से and some regularization parameter À 0, instead of looking the usual value function of a policy we consider the trajectory Kullback-Leibler divergence regularized value function V (+1) A VT (81) - AKT.tra(Pte (T, Ø). In this case, the value function the policy is penalized for moving too far from the reference policy म. Interestingly, we can compute the value of policy and the optimal value thanks to the regularized Bellman equations
(s, a) ra(s, a) Pa V A (s,
V ( AF (8) (T)))
A (8, a) rn(ss, a) Ph M.A. (s,a),
(5) = max A - (KL(TA REDA
A.A (8) arg max ( ( (KL((A())|
TEAd
where V regularized Bellman equations.
H- #
Note that for a the uniform policy we recover the entropy-
Next we define convex conjugate to A AKL (s)) as (s),,A) R +R
(a)()))A max {(T,, 2) AKL( TT (s)) Alog TEAA
Σ
(mllass expolss exp(xa//!
aaEA
and, with a sight abuse of notation extend the action this function to the Q-function as follows
(8) FF. (e).A.A (QEx
max { ( =
Thanks to the fact that the norm of gradients of (L ( (s)) tends infinity as ı tends border of simplex, we have an exact formula for the optimal policy by Fenchel-Legendre transform
A (8) = arg max { [ ( - AKT((T/(x()))
(
TEAA
Notice that have V ( E A since the gradient of + diverges the bound- ary of AA.
Finally, it is known that the smoothness property of (s,,G))A,A plays key role in reduced sample complexity for planning in regularized MDPs (Grill et al., 2019). For our general setting have that since (KT(-I/ax (s)) is X-strongly convex with respect to ||-||1, then ( is 1/A-strongly smooth with respect to the dual norm 11-11oo
(=) (z))) (V (F x')
21 || x 2/113..
We notice that KL-divergence is always non non-negative and, moreover, V (s) 2 0 for any s since the value of the reference policy गř non-negative.
D.2 ALGORITHM DESCRIPTION
In this appendix we present the UCBVI-EntE Ent+ algorithm, a modification of the algorithm UCBVI-Ent proposed by Tiapkin et al. (2023), that achieves better rates in the tabular setting. The UCBVI-EN Ent+ algorithm works by sampling trajectory according to an exploratory version of an optimistic solution of the regularized MDP and is characterized by the following rules.
36

---

Published as a conference paper at ICLR 2024
Sampling rule To obtain the sampling rule at episode t, we first compute policy it by optimistic planning in regularized MDP,
(s) clip (r((s,)) + V 1y (), O,
V
max REAA ( A (7K(7
(8)
+++
arg max (s) 1 ( TEDA
h
with V VH H+1 = O by convention, where p' is an estimate of the transition probabilities defined in Appendix D.3 and bed some bonus term, defined in (10), Appendix D.4, It takes into account an estimation error for transition probabilities. Then we define family of policies aimed explore actions for which Q-value is not well estimated at particular step. That is, for h' [0. H], the policy =<((h") first follows the optimistic policy æ' until step h where it selects action leading to the largest confidence interval for the optimal Q-value,
h 1.00 (als) (als)
if 5 N', if = N1
2.00 (als)
2(N' (a|s) = 1{a arg max, E -Q,
(a))) (als)
1b
where Q' is lower bound on the optimal regularized Q-value function, see Appendix D.4. In particular, for h 0 we have 7²-(0)  The sampling rule is obtained by picking up uniformly random one policy among the family x' x4((k)) h Unif(0, H]. Note that it is equivalent to sampling from uniform mixture policy π mix. over all h E [0, H].
Stopping rule and decision rule bound on the difference between the value of the optimal policy and the value of the current opti- mistic policy #..
To define the stopping rule, we first recursively build upper-
(s,
1
1h (+)))) + (s,
H
1 max a) (s, .0, 00.1)
(s) clip
*+ w((s) Th
+ 2X aeA
where bgup,e is bonus defined in (12), Appendix D.4, V is lower-bound on the optimal value by convention. Then the stopping time
function defined in Appendix D.4 and G in E N: (s1) S E} corresponds to the first episode when this upper-bound smaller than c. At this episode, we return the policy î #!.
The complete procedure described in Algorithm 3.
Algorithm 3 UCBVI-Ent+
1: Input: Target precision e, target probability bonus functions bt, DtKL
2: while true do
3: Compute Æ' by optimistic planning with (8).
4: Compute bound on the gap (i, a) with (9).
5: if G (x1) S then break
6: Sample h ~ Unif[[/] and set -" 7404)
7:
for h E [H] do
8: Play aj, ~ (s
9: Observe 1// ~ Pa (s%,
10: end for
Update transition estimates /t
12:
end while
13: Output policy = f
37

---

Published as a conference paper at ICLR 2024
D.3 CONCENTRATION EVENTS
We first define an estimate of the transition kernel. The number of times the state action-pair (s, was visited in step h in the first episodes are ni (s,a) - Ei= {( af = (s, a)}. Let n (s'1s, a) - Ei= (s), d s (s, a, (,) be the number of transitions from s to s at step
h. The empirical distribution defined as (s 8, a) ( (s,n a) if (s, a) 0 and ( a) _ 1/A for all E else.
Following the ideas of Ménard et al. (2021), we define the following concentration events. First, we define pseudo-counts as a sum of conditional expectations of the random variables that correspond to counts
(s. a) (s.a),
where nuix is mixture policy played on t-th step. In particular, we have
minit (s. H
oh 1x
(s, (s) a).
where Th (,(0(s,00) as) is greedy-modifiee policy 5' in stes:
,,(h')
Tha ) म (a|s)
if + h
1.0h' 2ich' TA
(als)
= 1.00') (als) 1 {fa a= arg max ((Ch((s) = (s,))))
T
if h
Let 3KL, ,gee, : (0, 1) N + RR, be some functions defined later on Lemma following favorable events
We define the
(KL (s, (s a)
(8) - Yt WH
(L)
(8) Yt Mt (G.d) n a) 2 1 (8))) (0)}
We also introduce an intersection of these events of interest, G(d) A EKL( n ( We prove that for the right choice of the functions 3KL, gent the above events hold with high probability. Lemma 12. For any 8 (0, 1) and for the following choices of functions B,
gKL (5. n) log(2SAH/)) + S log(e(1 +n)),
gent log(2SAH/6),
it holds that
2 1- 8/2, P[ (8)] 2 1 - 8/2,
1-8/2
In particular, P(G(5) 2 5-
Proof. Applying Theorem 9 and the union bound over h [H], () E S we get K[C (0)) 5/2.
By Theorem 10 and union bound, P[g (d) > 1 6/2.
The union bound over three prescribed
events concludes (6)] 2 1 - and P( 2 1 8.
Lemma 13. Assume conditions of Lemma 12. Then on event (KL for any J: S - [0, H) N, h E (H), (s, a) S
AH23KL(8, (,
Iph (x))(,,,,))
n/ (8,
38

---

Published as a conference paper at ICLR 2024
Proof. By Hölder and Pinsker inequalities
[p - [ a) S (1, a)) S (3,
Applying the definition of the event EKT. we conclude the statement.
Lemma 14. Assume conditions of Lemma 12. Then on event (K)( for any f: S 10, t t [M] (s, E A.
Ppn Phlf(s,a) H Phf(ss,)) +2
2H (6. (s,
A 11)
( a)
1 DA pn] < p)]s(s, + 2H
12 (ò, (5, 11)
(3,0a
Proof. Let us start from the first statement. We apply the second inequality of Lemma 32 and Lemma 33 obtain
[Pn - Ph]f(s,a)
& 2Var [f] a) KL(7))PA)
s 2 Varg [[((s))) (L( + 3H KL(PAIPN).
Since 0 S f(s) H we get
Vary [/] a) S ( & H (s a).
Finally, applying ab b, 2 O. we obtain the following inequality
1 H fl.(ss 4H² KL IPn).
p ph)f(s,a)
The definition of (8) implies the part of the statement. At the same time have trivial bound since f(s) E [0, H)
1
[pn -l,(s,)) < 211 S T (s) + 2H.
To prove the second statement, apply the first inequality of Lemma 32 and proceed similarly.
D.4 CONFIDENCE INTERVALS
Similar to Azar et al. (2017); Zanette Brunskill (2019a); Ménard et al. (2021), we define the upper confidence bound for the optimal regularized Q-function with Hoeffding bonuses.
Then we have the following sequences defined as follows
(s,a) clip as) + & V (s, as,
(s) max { (8) AKT
TEAA
Vi((s) = #1+1 (1())
(s)
V (s) 0,
and the lower confidence bound as follows
0% (s, a) clip(r (s, (s, -1
(s) max (TQ) (s) (s) TEAx
V +1 (s) 00
where we have two types of transition bonuses that will be specified before use. The Hoeffding bonuses are defined as follows
2123KL n a))l
10 (s,a) a)
(10)
n/ (8,
39

---

Published as a conference paper at ICLR 2024
Proposition 1. Let å E (0, 1). Assume Hoeffdiig bonuses (10). Then event ((6) for any t N, (h, 8,) H holds
(s. I Q (s.a) as)
(xn(s) S (s) (s).
Proof. Proceed by induction over h. For H + the statement is trivial. Now we assume that inequality holds for any h' > h for fixed h E [H]. Fix timestamp t E N and state- action pair (s, a) and assume that (s) H, i.e. no clipping occurs. Otherwise the inequality (s, & (2(s, a) is trivial. In particular, it implies nj (8, > 0.
In this case by Bellman equations (7) we have
14 As, a) - V (+1(s,0) - Ph A (s,a) + (s,
To show that the right-hand hand side is nen-negative. we start from the induction hypothesis
Th (nN p PaIV + (s,
The non nog-aegatyv of the expression above automatically holds from Lemma 13. To prove the second inequality on Q-value, we proceed exactly the same.
Finally, we have to show the inequality for V-values. To do it, we use the fact that V-value are computed by FF,(s)),, applied Q- Q-value
(s) (e)AA) (Q),,,, V.... (Q V (s)
(G).A) (Q))s)
Notice that (.) takes values in a probability simplex, thus, all partial derivatives of (2) are non-negative and therefore FF, is monotone in each coordinate. Thus, since
Q (s, & (s, S (), we have the same inequality Vi((s I V) S
(s(.).
D.5 SAMPLE COMPLEXITY BOUNDS
In this section, we provide guarantees for the regularization-awe gap that highly depends on the parameter X.
Let us recall the regularization-aware gap that is defined recursively, starting from G² A 0 and
a)) 1
p%. G ())))))
by
beap.((s. (s,
(11)
())() clip
++ wH(s) i
1 max (s, - 2 (s,a)) ,0.H) +
2X aed
where the additional bonus is defined as
(s. 4H23KL (5, (8,
(12)
and the corresponding stopping time for the algorithm
inf(l E N (m))
(13)
The next lemma justifies this choice of the stopping rule.
Lemma 15. Assume the choice of Hoeffding bonuses (10) and let the event G( defined in Lemma 12 holds. Then for any E M. S.h [H]
(5) T.A. S (s)
Proof. Let proceed by induction. For h H + 1 the statement is trivial. Assume that for any h' h the statement holds. Also assume that G (s) < H, otherwise the inequality on the policy emrr holds trivially. In particular, it holds that ni (8, a) > 0 for all a E A.
40

---

Published as a conference paper at ICLR 2024
We can start analysis from understanding the policy error by applying the smoothness of (s).A.A
(s) - (s) = A (Q7 - 1h ( - (5)
- (s),,) (V ( 1. (5, (s.-))
2x lie Q x. (s)
(5) Q ( (s)(_
Next we recall that
++1 (s) = V ( (())))
- -
Th
thus we have
( (Q)(s) (# (KL(T - = M/, +
-
and, by Bellman equations
V (s) (6) S =4+1 - (8)
10 11(()
2X
S # ph V= -V
#,4+1 (8) + 1 - ().).
2x
By induction hypothesis we have
(s) V (8) S t P
2x 110. - 2...)
1
Next, we apply Lemma 14
G, a) = p1 G G)M+1(8.)) + IPn - P
P G (s, + 4HL (s, & W)),,)),
s
[ H
n/ (3,
thus
A (8) VX (s) I WX + - 1 1
Q 1
2X
Finally, by the definition of || and Proposition 1
1
(s) A.h (s) S + (x(ss +
& (f).s).
max AEA
2X
Theorem 5. Let > 0, E (0, 1), S and X < H. Then UCBVI-Ent: algorithm with Hoeffding bonuses and a stopping rule & (13) is (s, P)C for the best policy identification in regularized MDPs.
Moreover, the stopping time i. is bounded as follows
1 SA (log:oogggSS//HA (SS//)))-SS--11 SL) SL)-L). -1)
where L O(log((SlH)))) log( (=))))
Proof. To show that UCBVI-Ent+ is (e, -PAC we notice that on event (()) for म = T² by Lemma 15
(s V < (s1) Se.
and the event ( holds with probability at least Next we show that the sample complexity is bounded by the quantity mentioned above.
41

---

Published as a conference paper at ICLR 2024
Step 1. Bound for (s1) First, start from bounding WH(s, a) and (s). By Lemma 14 can define the following upper bound for (s, a)
2 a)) (1 1+
ph G (s. 8H gKL (o, (6, (s,
n/ (s,
Therefore we obtain
(
2
8H2 (8, n (sh, ax))
(i(() S Eyt+1
[+
(/+1(5
H
ni (sh, an) 8h 1%
1
(2h,
max aEd
2X
By rolling this expression
18H2 n6,1%, (sh: an))
(81) & Ege11
1 1
H
12j, (sh, a%)
h=1
2
h 1
+ +
max aEA
(h(s a) 2% (sh,
H
2λ
Using the fact that (1 2/H2/H)* s c², have
gKL (5, (sh, an))
(11) s 2 H 1111
n/ (shh,
(A)
6²
E
Σ
max aEA
2 (sh, (s a))
2X
Lh=1
(B)
Term (A). The analysis of the term (A) follows Ménard et al. (2021): we switch counts to pseudo-
counts by Lemma 28 and obtain
H
" gkI (s,
(A) < 32e H
Σ
Σ
(s,
(s.a)ESxA
Term (B). For this term we analyze each summand over h separately. By Lemma 35
(2 a) 2 (sh, n) ) -E E,.11.00 (Qh an) 2% (8h, an)) 27 }
Egsti max
aeA
Next, we analyze the expression under the square. First, we have
(h(sn, an) 2¹ (sh, an) - 2 an) + P (itzl(sn,anh an).
By Lemma 13
Q an) Q (sh, an) S 46//'(s an) + Pa VV
h+1 1 az).
using V (5) 1i (s) < + [ 0 0h+ (s) and the definition of Hoeffding bonuses
(8)
(10), thus, rolling out this recursion
H Σ
2,3K (8,
(sh, an) 2 (sh, an) 4H
44HEE
nhe (Sw (sw)aw)
Sh
LM
By Lemma 28, Jensen inequality, and change of policy #++1 to ao (+1(()) by Lemma 35 we have
H 2B (SW, an:))
(s,, an) - Q2 (sh, an) 5 3/1322
E Ex(41,00
19h
(sw, aw) V
42

---

Published as a conference paper at ICLR 2024
By taking the square get
27 (s) - (sh, a)) )
E -
max DEA
H
2 (sw an ))
I 16H 1 (1.000
11100
The (sw,, an)) V
3h
The telescoping property of conditional expectation yields the final bound
H
22 (6. nh. (sw,, and
B max (sh, .EA
a)
a))
Σ
32H
)
h'=1
Finally, collecting bounds over all h E [H]
we have
16c H
H
(KI n (s,
B
(s,
(8, a) V
hm=1
The final bound for initial gap follows
H
3KL (s. (s. a)) (8, V1
(81) S 32e² 2
Σ
Σ
1 (3,
W=
(s,a)ESxA
16 H
H
H
BKL (5, (8,
di
())
a)) V1
hm 8,0
ht
Since S H, we have that H² S H3/2
Using a convention di. (s, a) = div (1.00) (s, we have
48c² H
H
H
3KL (s. a)) Tey a) V1
(5 <
dn
(s,
h=0
8,0
h'=1
By changing the summation order and noticing that
dr (s,a)
1
dh (1)
(s
H
h=D
for H +1 2H
2H we get
26c H
(s,0)
Gs)) 1
A
ΣΣ
h
mini
(s,
(s,
h=1
Step 2. Sum over t < i. Assume 1 > 0. In the case i = 0 the bound is trivially true. Notice that for any < t we have
G >
thus, summing upper bounds on (11) over all t < we have
96e 11
OKL (8, Th (s, a)) (s,
s(1-1) < Σ (()ss)
di
( V
(s.a,h)
Notice that (8,.) monotone and maximizes at t and d% mix. (s, n/ 7 Th (s, a). Thus, applying Lemma 29, have
384c² H SA gkl. (6, - log(().
E(1 ( <
Then by definition of 3KL
384c SAA
E(L ( 1)
(log(2SAH/)) + log((e)) log(t).
A
43

---

Published as a conference paper at ICLR 2024
Step 3. Solving the recurrence. Define A SAS and B log(2SAHH0) S. Our goal is to upper bound solutions to the following inequality
1 S A(Slog(() B) llg((t).
First, we obtain loose solution by using inequality log(t) S c /B that holds for any 2 1. Taking 3 = 1/3 we have
5 1 + 3.(35 - 1/3 + 2/3
Also we may assume that 2 2, thus S 1/2 and we achieve
2/3 6A(35 + B).
Solving this quadratic inequality in 1/33, we have
18A5 + (18A5) + 24AB
I (18AS + 244AB
2
Define L 3log
54AS + v18AB
Then we can easily upper bound the initial inequality as
follows
1 1 1 A(B SS)L.
44

---

Published as a conference paper at ICLR 2024
E BEST POLICY IDENTIFICATION IN REGULARIZED LINEAR MDPS
In this appendix, we first state some useful properties of regularized linear MDPs and describe the LSVI-UCB-ENT -nnt algorithm.
E.1 GENERAL. PROPERTIES OF LINEAR MDPS
Let us start with a description of how to generalize the techniques of regularized MDPs to the setup of linear function approximation (Jin al., 2020). Again, consider the case of KI--regularized MDPs with respect to a reference policy स. this setting the Q- and V-values could be defined through regularized Bellman equations
AH, a) (n a) + Pp PA (s,1
A (s) TT
XKL(TT
Moreover, for optimal Q- and V-functions we have
A a) (n(s a) + Ph 1 R.A max (TQ #EAr
AKL(T
Note that the value of policy could be arbitrarily negative, however, we know a priori that the optimal policy has non-negative value V V. E [0, HJ,, since the policy स itself has non-negative value.
In particular, under this assumption, we have the following simple proposition
Proposition 2. For linear MDP, for any policy such that V.,AA (s, a) 2 0 for any (s, A [H] there exists weights {w,, Jeeann such that for any (s, a, h) S A x [H] we have Q. (8, a) = (8 w.. Moreover, for any E [H] it holds [[ ll2 I 2H vd.
Proof. By Bellman equations
( a)) + Ph V...A ((8,))nn++ + Vo Σ (0sssa)
(w),,)), On +
Vi (nd(ss))
To show the second part, we use Definition 2. First, note that ||0 S vd and, the same time
So
V (i s Hvd.
since the value is bounded by H.
E.2 ALGORITHM DESCRIPTION
In this appendix, we describe the LSVI-UCB-Ent algorithm for regularized BPI in linear MDPs. LSVI-UCB Ent characterized by the following rules.
Sampling rule As for the UCBVI-Ent+ algorithm start with regularized optimistic planning under the linear function approximation
(s, (n, Th (s,
V1(() clip
max { T (s) AKL(T, (x))
(14)
FEAA
*t "h
(s) arg max (T) AKL(T, (s)))
REAA
45

---

Published as a conference paper at ICLR 2024
where b' is some bonus defined as follows
bj = B V..(ls "
(ss,a)) "
for > O bonus scaling factor, and the parameter w is obtained by least-square value iteration with Tikhonov regularization parameter (Jin et al., 2020),
Σ ra a + V (4+1(5)+2) s af) w + a
W
= arg mii
We notice that there is closed-form solution to this problem given by
1 [AA]
ph rh + V
(
where 4% (s), a and A Σラ.
" + al.
Then we also define family of exploratory policies by, for all h' E [H].
1 (a(()( ( = a1a E arg max m -=ll))aa((( -
(al())((al)) = # (als)
if 1/' if = 11 (15)
x² 7,(00) (a)ss
where Q' is some lower bound on the optimal regularized Q-value defined as follows
wh = [AR]]
Σ 2 I Vh+i (hh1
(16)
2 (s. 0) = (s,,s, T B (s) (ss a),
(s)) = clip
max
TQ - (II( 0,
TEAA
The sampling rule is then obtained by picking uniformly random policy among the exploratory policies, n' = *²*(h") for h Unif[H. Notice that it is equivalent using non-Markovian mix- ture policy x mix,t over all E [H). Additionally, it would be valuable to mention that computation of this policy could be done on-flight since we can compute Q and Q.
Stopping and decision rule In the linear setting
simple deterministic stopping rule = T for a fixed parameter T. In the finite setting, we were able to define an adaptive stopping rule by leveraging certain Bernstein-like inequality the gaps (see Lemma 14). However, it is not clear how to adapt such inequality to the linear setting.
As decision rule LSVI-UCB-EER returns the non-Markovian policy म, the uniform mixture over the optimistic policies (# heery The complete procedure is described in Algorithm 4.
Algorithm 4 WCV--UCB-Ent
1: Input: Number of episodes T, bonus function b', Tikhonov regularization parameter a.
2: for E (T] do
3: Compute #' by regularized optimistic planning with (14).
4: Sample h ~ Unnif (1, ..,., 11} and set n² ff(hh)
5: for h E [H] do
6: Play aj (s())
7: Observe s/4 ~ Pi (1)
8: end for
9: end for
Output the uniform mixture over { hemm
46

---

Published as a conference paper at ICLR 2024
E.3 CONCENTRATION EVENTS
In this section, the required concentration events for a proof of sample complexity for LSVI-UCB-ENE will be described. First, define several important objects. Let 1oद (1,,,, at) for any T E N E [H] and define
A ala + Σ MHOT Iº = ala + Σ Bymix.r an (s (s1).
where H' is uniform mixture policy of x= *(())) defined in (15) over all h' E (O..... H).
Let goone; (0, 1) x R R - R and gem; (0, x N R be some functions defined later in Lemma 16, We define the following favorable events for any fixed values of bonus scaling and Ridge coefficient a 2 that will be specified later.
goone (6, B)
vt E N, vh {H)
Σ 1h Vi (5h+1) - ph V A+1 (8% xx)
$ 2dH V geone f, B)
[A2]
Σ 1h (Vh+ (sh+1) pp V. (57x [5))
I 2dH v Boone (6, (6,1.. B)
[AA]]
Eaut (8) & vt MEN A A geat (6. t)Ta
We also introduce an intersection of these events of interest, ( B) - go B) n go We prove that for the right choice of the functions gooue, , the above events hold with high probability.
Lemma 16. Let B, a 2 1 be fixed. For any ō E (0, 1) and for the following choices of functions B
11 (11++))
(
B
geone (6,t A 2 2log
+ 5 + log 1 8d2
6
Hd
30m t) A 4 log (2t 1)/5) 4d log(t))) +3,
for any fixed a it holds that In particular, P(g( B)] 2
Pfgrone (5, 2 1-18/2 P (8) 2 1 55/2.
Proof. Let us fix h E [H) Then for all t E N by Lemma 17 we have || 2l S 2H / dt/a and by construction of A,, have Amin(A)) 2 Therefore, combination of Lemmas 25 and 26 for any fixed ε > O we have with probability at least - 6/H
Vi (h - pr V h+ (5h
d S 4
H(I+
82d
log
+ +dlog 1
2
ad
e/2
[AR] 1
8d112232
)
+ 8t2E2
+ dt +oloog log
1+
ag
Next we take Hd/t and obtain by using a and d >
1h { V/ 1 pn v Vininiii (5h 15t1 (7)
I 2/12d 2
H11+t%
+5
[A]]==
B 2
log 1 /222
Hd
47

---

Published as a conference paper at ICLR 2024
Taking the square root conclude the first half of the statement, the second half is exactly the same, since Lemma 25 gives a bound uniformly over all value functions.
By Theorem 27 and union bound over h [ Pl (6)] 2 6/2. The union bound over two prescribed events concludes (6, B3] 2 -
The proof of the following lemma remains exactly the same in Jin et al. (2020).
Lemma 17. [Lemma B.2 by Jin et al. (2020)) For any (t, h) E N x [H] the weights wh generated by LSVI-UCB-ENN satisfies
1 S 2H /t//
E.4 CONFIDENCE INTERVALS
In this section, we provide the confidence intervals on the optimal Q-function that is required for the proof of sample complexity of LSVI-UCB Ent.
We start from the specification of the required values of a and B.
24edHT
- 2(g (6,T) ( + 1),
B = 32dHH
log
(17)
8
where gemt is defined in Lemma 16. Proposition 3. Let and B satisfy (17). Then on the event G( defined in Lemma 16 we have
())s w.)) Qi a) - VV V (8 a) + T
(v(s, a), w/)) ( PM K+1 V + +
where ()) and A (s, a) satisfies
max |
< BV (A,
Proof. We provide the proof only for the first equation since proof of one statement completely reassembles the other. By Proposition 2 and Bellman equations we have
Q (). a) (1(s,, a), wF) (s, + Ph V (8,
therefore
//b 1h [AA]
Σ
WAr V
k+1 (sh+1)]
( ah), w/) awt
-oo[AA] w + [AN]] Σ
Vi
h+1 (8h-1) ph V h+1 (0% (7))
th
E1
82
[A
Σ
dhen V441 ViAnn (sh, a7))
Next, we analyze the last term Eg. By Definition 2 we have
Ph V T V (8h,ah)) = ( Wh: / [v h+ - V (osaill))))nnas)))))
thus
Es = |Ai Σ MH + +als aaIa-alaa [/ Phtt h+1 (6s)
- Vi+ti h+1 V Vexn4zl(s )n (s -a Ail / Vhun
htt V VeA (() (nd'')).
48

---

Published as a conference paper at ICLR 2024
As result, moving to Q-values directly have (b(s,a), - Q% (s, a) = (1b(5, a), w/ - =PA[V h+1 V V + (b(s,a),E + Ex)
(1,0,)
Next, we compute an upper bound for A (s. We start from the first term, where we apply Cauchy-Schwartz inequality and the second statement of Proposition 2
av) G.>l = 1<b(s, a), on 1a alli/s o)][aa))1 [
S 2// da (1s a)]
where we have used that || I 1/a. For the third term, we apply exactly the same con- struction and obtain the same upper bound. For the second term, also apply Cauchy-Schward inequality and Lemma 16
Σ dh (++1(1h+2) - pa pnV' 1-1
[A
S 2dH V goone f, (s, (8)Aa)))1)
Thus, we have
(s, [2HH / geone (ō, T) 4H /2d (,, (Ō, + / (
The only part is to show that for our particular choice of B it holds
2d 2dH1 / gonne (ò, I,, B) + 4/1 44 (em (8, 7) + + S B.
First, notice that
24cHT
4H 2dd + < 16Hd log
S B/2.
5
Thus, it is enough to show that
H 72)
B
2
B
2
geone (6, T) log
+ 5 + log
1 8/222
S
5
dH
16
dH
First, notice that since T 2 and (0, 1) then
H(1 + T2)
HT
2 2
+5 < 4
8
and also, using the inequality log(1 + x) S x for any x 2
log 1
1/
B dH
log
1 + 32
1
B 2)
+ log
32 8. 2.d1/222)
dH
+ 2 log(16 dT)).
32
dH
Thus, it is enough for B satisfy the following inequalities
log 24eHT
B
2T
B 2
B 8dH
4log
log(16d))
32dH
8
8dH
It is clear that the choice B defined in (17) satisfies all required inequalities.
Corollary (Confidence intervals validity). Let constant a and B defined in (17). Then on the event
G B) we have O (', a) > (s, 2 Q¹, (s, and V², (8) 2 Vi)a((s) 2 Vi((s for any I e [T], h E [H), (8, a) ES A.
49

---

Published as a conference paper at ICLR 2024
Proof. Let us prove using backward induction over h E [H]. For h H + 1 this statement is trivially true. Let us assume that the statement holds for any h' >> h. Thus by Proposition
a)) (8, = (s)
BB
+ Ph [ - Vitti (8)
Notice that (s, + ( Vi+1 2 0 for any s'. Thus, we have proven the required statement for Q-values. To show for V-values, we notice that if upper clipping in the definition V in (14) occurs, then the statement trivially holds. Otherwise, we have
2 0 and by induction hypothesis h+1
(+))) (o) (
( (o))
However, the function (s),A, is monotone since its gradients lies in probability simplex. Thus, (s) bound.
V
A.A.A (8) 2 0. Using exactly the same reasoning we may show the lower confidence
E.5 SAMPLE COMPLEXITY BOUNDS
In this section, we provide the sample complexity result of the LSVI-UCB-ERS algorithm. We start from general result that does not depend on the properties of linear MDPs, however, depends on the properties of the algorithms.
Lemma 18. Let constants a, B defined by (17). Then on event G B) defined in Lemma 16 for any t E N. S E [H]
V (s) (
Σ
Qn - (sh (em aEA
(
1111
max
21
Proof. Let us proceed by induction. For h H + 1 the statement is trivial. Assume that for any h' >> h the statement holds. Also assume that (s)(s) < H, otherwise the inequality on the policy error holds trivially. In particular, it holds that n/, (s, a) > 0 for all a E A.
We can start analysis from understanding the policy error by applying the smoothness of (a
V A. (s) F
F Q Qu. ( (s)
SPAACIAAM 1) + (VFF. (i (h((,
1h A (* ( - AKI (s) (+))
21
Next we recall that
A (+) = V =
#1 - (K
thus we have
((Dh)(s) ( 1 -KI (s) - + D. +
and, by Bellman equations
(s) (s) S #141 Q (s)
-
1
110 - 1
2X
I - ph VEAHI V
(s) + 2X
1 Q AA (Bc
Next, we start by changing the norm and using the properties of Q and Q (see Corollary 5)
th 12 (s) = mmx (), Qh.a((,a))
s max ( aed
(s,,)) 2
aEA
50

---

Published as a conference paper at ICLR 2024
Theorem 6. Let 0,8 (0. 1). Then LSVI-UCB-ENN Ent algorithm with choice of parameters described in (17) is PE. the best policy identification in regularized MDPs after
d
T o
iterates.
Proof. First, notice that the definition of the output policy म mixture policy over all ø allows us to define the following
T
V A. (51)
- V F,A,1 (81) = Σ
( - 1
T i=1
therefore it is enough to compute only the average regret of the presented procedure. In the sequel we assume the event (( B) for B defined in (17).
Step 1. Study of sub-optimality gap
Let us fix t E [T), then by Lemma 18 we have
H
@ - 2
V ( V/2" (81)
E
Eges1
max aaA
2X
Next, analyze each term separately, starting from the difference between Q-values inside. Let us fix [H], then by Proposition
(s. (s, = Q Q (.,nn((s,) - 1Q -(,,,
(18)
S Ph V Kitta (8, 4B1
Next, we have for any h' E [H] and any s' ES
V
V (8) S म +1 p 2ir 20))
therefore we can roll-out the equation (18) and obtain
(s, 2 (s,a) S 4BE,,
Σ
[Iv(s aw Maaas.11111
(sh, az) =(s,a)
W
Next, apply Lemma 35 for any fixed h [H]
Ege+s
max Q' 22i (sh, a)]s aEA
E Ege-i.c0
(s%) an)ls1
(sn
and for any W/ 2 h
Eg++1
[I alla) (sn. as) an) = (s. a) = B.1100 [ ansss,aw))lll an) (sn, an) (s.
Therefore, applying Jensen's inequality conditional measure and the tower property of condi- tional expectation
2 Σ ( and aw))l
max Q/ (Qh--0h (6,))) (0)(5)))))) a)lsi aEd
Ege+s
1 111000
|st
H
< 1613 H
Σ
E Eseti,0
[[rs(sw an alllss
M' h
"
16BB H
E Ege
|lvb an an)) >1ia,
M=1
51

---

Published as a conference paper at ICLR 2024
Summing over all h and recalling 7h as mixture policy of all mf+
O =21) ( as) I 16B2 H
E g
max REA
B 1.(0)
[[rb(sswaw)))) )n 11s1
h1=1 h=1
16B² H
E B
[| aw >
Step Summing sub-optimality gaps
Next, we sum all sub-optimality gaps and obtain
E (
Vo (1)) HH
( (811
Vi (81))
16B 1
T-1
[llb(sw an >lAgh |s|
$ H +
n m
A
M=1 mm
Next, apply the definition of event gent
A > 2 7 geat (8, Ta = [A x 2 2 (8, T) ]-
Notice that by the choice of a 2( ) 1) we have
A A 2geat (8, 2Ia Σ
E Emmix, [K(sh, (sh, an)]
Thus, may apply Lemma 31
Egets [W
$ 2
E c-
| an )n)l |ss
S 4 log det (x
To upper bound the determinant, we upper bound the operator norm using the triangle incquality
A 122 = 21a + Σ Ex (sb (1b )
2 (T 1),
therefore, combining with definition of B given in (17) we have
Σ (81) (s1) s H 64 322 HD
T
log(T +1)
24cdHT
log
X
5
yielding
H T
64 22 H5
log(T + +1)
24cdHT
(81 (
S
+
-log
XT
In particular, this implies that
T T= δ
A
is enough to obtain e-accurate policy.
52

---

Published as a conference paper at ICLR 2024
F DEMONSTRATIIO-REGGUAARRR PREFERENCE-BASER LEARNING
F.1 MAXIMUM LIKELIHOOD ESTIMATION FOR REWARD MODEL
In this section, discuss the maximum likelihood estimation problem for the reward estimation, following Zhan et al. (2023a). Let & be a function class of reward functions that satisfies the follow- ing assumption
Assumption 6. For a function class G we assume that the true reward belongs to it: r* E G.
Additionally, for the following function family Q = (ar (To. T) a((m)) -r(r)) : r E g)
equipped with an lo -norm has a finite bracketing dimension, that means there is d, > 0 and R such that
Ve E (0, log (o 1,11-l>>) S dg log log(Rg/s).
The bracketing numbers are commonly used in statistics for MLE, M-estimation, and, more gener- ally, in the empirical processes theory, see van de Geer (2000). Related to our setting, this assump- tion satisfied in the setting of tabular MDPs with a dimension dg = SAH and linear MDPs with dimensions dg dH, see Lemmas 19-20.
For each r E 9 and a pair of trajectories (To: TI) define the induced preference model as follows
g- (T T - (r((1)) 100)--((10 r(ro)). To measure the complexity of the reward class class Q = {gr: T x T [0, : E Fr), where T = (S x A) is space of all trajectories. See Definition 7 for the definition of bracketing numbers.
will use the bracketing numbers of the function
Given the dataset of preferences DRM estimate of the reward model as follows
((TO - ((To, ) we define the maximum likelihood
ARM
arg max rEg
Σ
o log (To 71) + log(1 ar (.(
(19)
The following result is a standard result on MLE estimation and, generally, M-estimation, see van de Geer (2000). The proof heavily uses PAC-Bayes techniques by Zhang (2006), see also Agarwal et al. (2020) for non-i.i.d. extension. We notice that a similar result could be extracted from Lemma by Zhan et al. (2023a), however, we did not find the proof of exactly this statement in their paper or references within.
Proposition Let Assumptions 4-6 hold. Let DRM be a preference dataset and assume that tra- jectories T5 and of were generated i.i.d. by following the policy т. Then for any 8 E 1) with probability least 1 1-8 8 the following bound for the MLE reward estimate given by solution to 19 holds
+ 2dg log (oggR//NRRR + log(1/6)
Erg- (T. mI) - ge (T T ))) )))
NRM
where 9t is a distribution over trajectories induced by policy
Proof. In the sequel, we drop the superscript from DRM and NRM to simplify the notation.
Let consider maximal set of e-brackets B of size We Q, l.lllc) and apply Lemma 2.1 by Zhang (2006) (or Lemma 21 by Agarwal et al. (2020)), where consider brackets [f, u] from B as parameters 6, prior π is a uniform over B, and the density ([e u]) is equal to a (properly weighted) Dirac measure
bracket [e u*1 that contains the MLE estimate (ties are resolved
arbitrary). implies that for any function L: B x D - holds
Ep ( L(D, [8*, u²1) - log ED [ Jee (e)))))]
(e, 2, [1 1
where D' an independent copy of the dataset D and the KL-divergence between a Dirac measure on an MLE bracket and the uniform distribution is computed exactly. Since we can control the exponential moment, by a simple Chernoff argument we have with probability at last - 5
log ED [e C(D' ,1" (r"aa)]]] < -L(D, 11", w(1l + log No 11-11l)))) + log(1/5).
53

---

Published as a conference paper at ICLR 2024
Next choose L(D, [e, u/) as log-likelihood ratio
N
of loga, (To, xt + (1 log(1 1 (TT of uog +t) + (1 - of) log(1 - L(TO) -f)
[(D, (e, ud) -
By the choice of a brackets [e, u] it holds (TO, T1) - (T 11) S u² (To: T1). Using the fact that F a solution to (19)
N
log + (1 0²) log( (1 g. (T +
- =
2 k=1
looog(TT +f) + (1 o²) log(1 f(rØ: +f)
-t log q, (To T + (1 0t) log(1 q. (T , +
5.
o log qi (T) 7 (1 0t) log(1 ( +f))
At the same time
- log Ep [elo' (0 [nD] = - log E exp
1 oloo 9- (To, 71) + (1 o) log(1 (1 (To 71)
2 olog wn (To, 71) + (1 - 0) log(1 - (To, 71)
where in the last expectation To. q", o on (To
By Fubini's theorem, have
log Ed (C(D' w[D] log En V 9 (T:
/1-14 11)) E(TU, T1))
Next, we study the expression under the square root. By the definition of the e-bracket we have e" (To.TT) S (=( mi) I u" (To, TI) and u" u"(To. T) - 8x (To, m) S e, therefore
(To TI ( T))= S v (To, TI (gr(To, m) + E) z v (To, m))g:(o, 71) + Vł.
and the similar bound for the second term. Applying incquality - log(x)) 2 - x we have
1 N log Ep (C(D",(1,"")) 2 1- Ero.T
v (... (1)
g: Ti)) (
+ /(1 (1- (1)
22E.
By the properties of the Hellinger distance du (see Section 2.4 and Lemma 2.3 by Tsybakov 2008) we have
1 1 -
V Val(oo: Ti (1):( v(1 2+(ro. (To, 11)
(TO,,
Br (Ber (err T1))] 2 E (a. T1) -10(to)))))(00)))))
Overall, we obtain
< 2 log.N((e, Q, 11.))0)) + log(1/0)
Ero. (To - (=((TT (To,
m))²
N
Taking E 1/N2 and applying the upper bound the bracketing number by Assumption 6 conclude the statement.
And also we have a simple corollary of this result that shows convergence of the reward models.
Theorem 7. Let Assumptions 4-6 hold. Let DRM be preference dataset and assume that trajec- tories Th and tf were generated i.i.d. by following the policy и. Then for any E (0,1) with probability at least - 8 the following bound holds
Erg.tenin [ (T))) -r(( ((m))) 2g log Rg/NRM) + S log(e?//)
NRM
where 5 1/(infe//11 H1 d'(x)) the non-linearity measure of the link function a defined in As-
sumption 4.
54

---

Published as a conference paper at ICLR 2024
Remark 7. We additionally notice that since two trajectories are i..d., we have
r-)(m)mm)))))))))))) B (()) (F())
f(T))))) 2V ((())))
In particular, it means that if the reward function is estimated up constant shift, then the MLE estimation eπrоr is zero.
Remark 8. In the setting of a sigmoid link function (x) = 1/(1 + exp(-x)) we have Ç (x){ yields the exponential dependence on the reward scaling. This could be avoided by using a more problem-dependent way to obtain a bound on rewards given bound on the induced preference model.
Proof. Follows from the application of Proposition 4 and the following application of mean-value theorem for any two fixed To. 71
[r (m)) rr(())) [f(m) f((o)) = o o (To r1) o o
(To (a ri)
(o (To mI) (T(T..T1)) A1]]
where & is a point between (T Tı) and (r(To, T1). The observation (o- = (()
(E))
yields the statement.
Next, we compute the required quantities dg for the case of finite and linear MDPs for a choice of 1/(1 + exp(-z) sigmoid function.
Lemma 19. Let reward function (rn(s,a) (N)( be an arbitrary function ra: S x A + [0. 1].
Let us define G {r(T) = Eh= (n(sh, an)). Then Assumption holds with constants dg = HSA and Rg 3H/2
Proof. Let us define a functional class of interest = (T, ((((n)((())))))))) r(rz)) Ire g} Since is a monotonically increasing function that satisfies o (x) S 1/4.. Thus, by combination of Lemma 21 and Lemma 22 we have
(e 2, (1-11=))) s (2e, 11-1oo))
Next we define function classes Fú. = {ru: S x A + [o, 1]} of one-step rewards. By Lemma 23 it holds
H
(2 S II h=1
H2,
Then we can associate a function space Fn with a parameters θ = [0, 1154 and by Lemma 24 and standard results in bounding of covering numbers of balls in normed spaces, van Handel (2016),
I N(s/2.10,, [1+ll=) s (3/
As result, we have
Ni, 11 HSA (3A
Lemma 20. Let reward function {ra(s, HeEH be parametrized as rh a) = as for vb S x A Rdd that satisfies [|v((s,)) a)lla & 1, and 0₂ E Or, for O₁ (0, Rd ||0x]| S Vd). Let us define G = (r(r) = E ra(sn, an)). Then Assumption holds with constants dg dH and Rg 3H v d/2.
Proof. Let define one-step rewards follows Fn (r,(s,a) = a)), On E Rd 110,1 s Vdj. Then following exactly the same reasoning in Lemma 19
H Wile, (H < Il Fg
h=1
55

---

Published as a conference paper at ICLR 2024
Next notice that Fr is Lipschtiz in fo-norm with respect to parameters 0, with constant 1:
(s, a) r/ (s, a)] ( (0h 0%)) 1 0%
Therefore, since b is ball of radius v d we obtain
Fh. 1
(3va/e)*.
As a result, we have
log. log.Vi(e,, 2.11loo)))))) < dH dH log(3// (d(=2))
F.2 PROPERTIES OF BRACKETING NUMBERS
In this section, we provide list elementary properties of bracketing numbers completeness. See Section 7 of Dudley (2014) for additional information.
Lemma 21. Let (G, (1-loo)) be normed space of functions
set X' that takes values in the interval I and let a: - [] be a monotonically increasing link function that satisfies supzer o' (x) C for C Then for F e > 0
{f a oog 9) it holds for any
F, S /
Proof. Let G be minimal set of e brackets that covers G. Then us take a set of brackets F = foool,o toelooo rou] [1, u] G}. The set of F consists of brackets since is monotone, and covers all the space F. Additionally, we have
lo o u(x) o ((x) = ((())(((()) {(x)) < Ce,
therefore the set F consists of Cs-brackets. By rescaling we conclude the statement.
Lemma 22. Let (G, (1.loo)) be normed space of functions over a set X and let us define set of functions over X X as (f(x,x') = g(x) g(x') g E g). Then holds
Nn 11+lo)) (,
Proof. Let G be minimal set of brackets that covers G. Let define the following set of brackets
F (1' ((x) u(xx), (2, u(x) {(x'){ [e, u] E G).
We may check that elements cover all the F. Let take f E and the corresponding g e G. Then let us take bracket [e, u] that covers g. In this case, we have
[e x') f(x) u(xx) S g(x) ,(x') S u(x) Elx') fa(a,
At the same time, we have
llfe fulloo 2 El1oo S 2E.
Thus, F is set of 2e-brackets that covers F.
Lemma 23. Let ( be sequence of spaces of functions over a set X equipped with a norm olo and let us define set F of functions over X as follows
K
F=
f( f(I
ge(ax)
g E &
A=1
Then the following bound holds
K NF < II N/(e/K,G.
k=1
56

---

Published as a conference paper at ICLR 2024
Proof. For any [K] let G₁ be minimal set of brackets that covers Gk. Then construct the set F as follows
F= = [e(x)) (u(()) (xx)). fu(x) -
Σ un (Ix) A4 E [K] [lx, ual E Gh gm }.
holds that |F| IT [G&l and also it clear that F consists of brackets and covers all the set F. Additionally, we notice that
I bu S Σ lle uxlloo Ke.
By rescaling we conclude the statement.
Lemma 24. Let ② be a set of parameters and let F = (fo(z) E Q be set of functions over X. Assume that So (x) is T--Lipschitz in with respect an arhitrary norm ||:]]
Va X (ff(z) fa)( L - 6'1l-
Then have
(e S S((E)
Proof. Let be a--ooerin of @ and let us define a set F = [le fo fo-eL, - X}. Let us show that F consists of brackets. Let fo E F, then there E X. By Lipchitzness
Va X [f((z) fx(2)) T.10 01 Le,
therefore bracket that corresponding e and indeed satisfy {(x) fz u(x) for any X. Also, we notice that |e - u||me 2gL, therefore by rescaling we conclude the statement.
F.3 PROOF FOR DEMONSTRATION-REGUUAR
RLHF
During this section we assume that the MDP is finite, i.e. |S| < +oo to simplify the manipulations with the trajectory space. However, the state space could be arbitrarily large.
In this section, we provide the proof for the demmonstraiion-rggulaizze RLHF pipeline defined in Algorithm 2. We start from the general oracle version of this inequality. For reward function let the value with respect to this value be defined as follows
H
V (s;r) En
Σ (sw, aNd 8h =
(E =
VV (s; X KKLraj (T|m).
A similar definition holds for Q-values. Additionally, let define the following coefficient defined by Zhan et al. (2023a) that additionally shows the closeness of T and BBC' in terms of the family of reward functions.
E Toa fr (T0) (*( (r(70) r(r)))
F, A max
0, sup rEg
(20)
E Poo
- - (r(70) -(())))
We notice that the denominator could be written
Byg (Tr - (T)) ( r(7))) = 2Vrr g no oo (r" - f].
Theorem 8. Let us assume that there is an underlying reward function r* such that
1. There is an expert policy F such that
(1155)))) - VI (81:7*) S ex;
57

---

Published as a conference paper at ICLR 2024
2. There is a behavior cloning policy x BC that satisfies
KT (T S EKL;
3. There is an estimate of reward function i that satisfies
VVar no [r* (T) - (Fr) S ERM:
Let RL be ERL Ept-ontimal policy in the X-regularized MDP for BBC and using é as rewards:
VBne 2, (81; Vince x.1 (81; S ENL.
RL
Then we have the following optimality guarantees x RL. in the unregularized MDP equipped with the true reward function
FRM 2sp. 2SRI. + 2λekI. 3ERMERL Vi(5))"") VI (81;T*)
4H KKL
A
Moreover, if we assume that Cr (9, BCB is finite, we have
(i((s)=")) VI (S1r*))
FRM + 25p + 2ERL + 2AEkL + (G, BBC). SRM.
Additionally, under the choice X positive solution to the equation 2x*2(20²2 22ERR. + E we have the following bound
(1(r") 1/T (si:") s 25g 4ERL + 6EKLERM min ( C. ERM)
where C, = Cr( (, is concentrability coefficient.
Proof.
We start from the following decomposition, using the assumption on the expert policy
(sa:"") V/r" (s1; VIT (s1:"") Vir (s1; + EE.
Next, we change the reward function as follows
VT (s1:7") - VT RL (s1:r") = V/"" (s1; V RL (51: + VT" (s1:7* P) VT (s1:-* f). (A)
(B)
We start from the analysis of the term (A). By properties of the behavior cloning policy and the BPI policy N RR
(A) = V ..1 (815)) + XKL ( 1| BC) VTBC ($15 - A KLrraj RL || BC)
S B, - Fe ($1; + Ae S ERL + AE
Next, we have to analyze the second term (B). We decompose it as follows
B ViT (s1:T" - f) - VIT
($1: - + ViT BC
(s1;T* f) VI (s1r"* f).
(C)
We start from the analysis of (D) because the analysis of term (C) depends on the concentrability assumption.
For the term finite
(D) we can apply the second part of Lemma 32 since the space of the trajectories is
D E Bo fr* (T) f(T)] - E Ey (T) f(T)]
< Var (r*(7) f(r)] KLuraj (LL (BC)) I v ERRs (Lr BCC)
58

---

Published as a conference paper at ICLR 2024
where applied an assumption the reward estimate. Next, we have to estimate the trajectory KL-divereence between Rf. and π BC.
Let us apply the e-optimality of the policy RRL with respect to reward f in the regularized MDP VT" (81; f) By rerranging the terms we have
(|= I Vi Vi (s1; + ERL - A KTrraj (1
KLuraj(n 11
(813f) V/" (sl;f) + +ERI) + +
EB + +ERL
+ d 11 (vr (s1; f -r") 1/"" (51; (s1;f--()) -r²)) + +ELL
VT (s1;r* -) ViTA (81; - f)) + ekkL. + EB + ERL
A
We notice that (B) A VT. (81; -)) V/ (815 - f), then we have the following recursion
ERM
B S (C)
((B) + (SE + ERL + A1
Let us denote 8² = (B) + ERL + dE / (C) +E ERL + A Then have the standard quadratic inequality in i, the maximal solution of which could be upper bounded as t² S a² + 26 which implies
B S
RM +EB + FRR + AE 2(C).
Next, we analyze the term (C).
Without concentrability assumption
By the first part of Lemma 32 we have
(C) E g [r (T) - f(T)] - E y nc
[r (T)
-[T]]]
2H
211
2Vrr BC KLtraj (T))
KL KLuraj ( 1TT)) s 622-EMEKT +
3 KK.
3
Overall, we have the final rates
2 FRM
4H 3 EKL
Vi VI (s1r²))
+ 22g + ZERL + 22 3ERMEKL +
X
With concentrability assumption we can upper bound (C) as follows
Now we assume that C, R, x BC) is finite. In this situation,
(C)- E Brongr Bi (To (m)) -
I =F, Var BB fr* - f] & V2 CG, R BC) ERM-
Overall, have the final rates
Vy VT RI. (81r*)) RRM + 22p + 2.RT. + 2λek + +30, ,, BC) SRM-
Under the choice of x To show the last part of the statement, we choose X* as a solution to the following quadratic equation 2 = 2XLERL + eiM- In particular, its positive solution is equal
x ERI + V V + 2E FRM
25
Applying firstly the quadratic formula and then the exact formula above we have
RM + 2ERL + 2x* E = 4A A 2ERL + 22 k. + 25 I 45RL + 3EKLERM-
A
Combining this bound with the bound performance we have
Vi2 (1(11r*) - VTT" (s1;**) < 25p 4SRL + GEKLERM + min (211E C, ERM)
where C, = (g, F. FF BC) is a concentrability coefficient.
59

---

Published as a conference paper at ICLR 2024
G DEVIATION INEQUALITIES
G.1 DEVIATION INEQUALITY FOR CATEGORICAL DISTRIBUTIONS
Next, we state the deviation inequality for categorical distributions by Jonsson al. (2020, Proposi- tion 1). Let (Xt)teN- be i.id. samples from a distribution supported on ( m. of probabilities given by p E 1m- where Am-1 the probability simplex of dimension m We denote by Pr. the empirical vector of probabilities, i.e., for all & {1.....ml m)
n Pn,k
-
Σ - 71
Note that an element E Am-1 can be seen as element of 1=-1 since Pm 1 - Exe Pk- This will be clear from the context.
Theorem 9. For all p E Am-1 and for all 8 E [0, 1]. P(En E N n kKL(p..)) > (1/)
0) + (m 1) log(e(1 + n/(m= 1))) 8.
G.2 DEVIATION INEQUALITY FOR SEQUENCE OF BERNOULLI RANDOM VARIABLES
Below, we state the deviation inequality for Bernoulli distributions by Dann et al. (2017, Lemma F.4). Let F, for t E N be filtration and (X.) be sequence of Bernoulli random variables with P(X, = 1[F--1 = Pt with P. being Ft-1 mee--ubuueaelle and X, being Fr F--masuurble.
Theorem 10. For all 0,
P
:::
X
Σ P1/2 log 8 8..
t=1
G.3 DEVIATION INEQUALITY FOR BOUNDED DISTRIBUTIONS
Below, we state the self-normalized Bernstein-type inequality by Domingues al. (2021b). Let (Yt)eN (We)seN be two sequences of random variables adapted to filtration (Fi)ten. We assume that the weights are in the unit interval wi E [0, 1] and predictable, i.e. F, measurable. We also assume that the random variables Y, are bounded [Yr] < b and centered E[ = 0. Consider the following quantities
S, a Σ W.Y.,
V₁ 2
Σ W E[Y
and W
Ws
and let h(x) A (x + 1) log(æ + 1) a be the Cramér transform of a Poisson distribution of param- eter Theorem 11 (Bersttin-tyyp concentration inequality). For all d > 0.
b.S.
2 1, (Vi/12 + 1)h
2 2log(1/4 + log(4e(21 < d.
V.+
The previous inequality can weakened obtain more explicit bound: if 2 with probability at least - for all 2 1,
| 2V% 1)/5) +3blog(4c(21 12llo((4e(2+11//00 1)/5)-
G.4 DEVIATION INEQUALITY FOR VECTOR- SELF-NORMALIZER PROCESSES
Next, we state Lemma D.4 by Jin et al. (2020). For any symmetric positive definite matrix A we define ||~||A = VIT Az.
Lemma 25 (Jin et al. (2020)), Let {s be a stochastic process on the state space S adapted to filtration 1o2 Let 2 be an RRd-valued stochastic process where v. is F F-predictable (X, is Fr-1 measurable) and |X||| < 1. Let A ala Σi= X,XI and let V be a family of
60

---

Published as a conference paper at ICLR 2024
function over the state-space S such that VV V. with probability at least 1
& H. Then for any (0,
VEENWWEV -B.
S 4H2
d
log
t
[
8t2
2
a
6
A-
where Ne is minimal 8-cover of with respect to the distance p(V, V' = sup (V(s) - (*))
Also, we state the result on the covering dimension of the class of bonus function, see Lemma D.6 by Jin et al. (2020) for similar result.
Lemma 26. Let V be a class of functions over S such that
max { x (),))s BV 0(s 1AA-)) (B TEAA
e :: V(s) clip
10, B), Amin (A) 2 a. Assume [s)(s S 1 for any (s,a) E x A. Then the covering number [Nel of the function space V with respect to the distance p(V, V' = sup.es (V(s) - V'(s) satisfies
is parameterized by tuple (w. A) such that ||w||| S B, E
4L #1 + d log (1(11 1+
8/ B2
log (s. p) & dlog 1
ac?
Proof. Following the approach of Jin et al. (2020), we reparametrize the following set by setting A = A2 and obtain
( 0.11.
V(s) clip
max TEAA
т n (bs,)) -)+ + ((,ss,)'AAsss
for |/0122 $ L, ||Al|1 - Baa Next, let V1, V/2 E V be two functions that corresponds to parame-
ters (wi, A1) and (w2, A2). Then, using non-expanding property of clip(-,0, 0, H) and max { we have
P(V,, V2) S sup max #ES A
(w] ( A )) (wiv(s,)) + Vo(s, A22
sup
[wi - w2] (s (8,8, a) Al0(s, V 123(())))))))) (
sxa
sup
[wi w2 +i
sup S
A2)
S || wella + VIAA - -
The rest of the proof follows Lemma D.6 by Jin et al. (2020) and uses the result on covering numbers of Euclidean balls in Rd.
G.5 DEVIATION INEQUALITY FOR SAMPLE COVARIANCE MATRICES
The following result generalizes Theorem 10 the case of linear MDPs and generalized counters.
Let [2 be sequence of random vectors of dimension d adapted to a filtration {F, such that ||Xell2 S 1 a.s.. Define a sequence of positive semi-definite matrices A, E[X,X] F(-1]. Notice that |A:1|2 Omax (At) S Also define A, Ala + Ej X,XJ and A, Ala +
Σj Ag.
Lemma 27. Let 8 E (0, Then the following event
geut' (6) = (vt 2 1: 3 A, ((
under the choice ((5 log( 1ogg(4 + 4d 4d +3 holds with probability least 1 6.
61

---

Published as a conference paper at ICLR 2024
Proof. Let fix vector from unit sphere ||0||2 1. Next, note that
Xj)
E.
(v, X3) 15)
AM;
Notice that AM, a martingale-differrnce sequence that satisfied
A.M S 2 and
E L Fy1 = Σ B|AM F- &
(u, [ S v'A.e.
==1
j=1
Therefore, applying self-normalized Bernstein inequality (Theorem 11) we have that with probabil- ity at least for 2
w A v A V Ae 1og((e(2t 1)/6) 3 3og + 1)/6).
Next, inequality 2ab S a² + b² implies that
1 In A. - v A.vl s 20r New + 4 + 1)/8).
Let us denote by Nr e-net over unit sphere of dimension d. By union bound over this net we have with probability at least 1 5
1 Ae + 4 (og((4e(2 1)/6)) + 1lg((WV))
ve E No 10 A, v A,o S
Let be an arbitrary vector on a unit sphere and let Ng be the closest vector to e in the e-net.
Then for any matrix A paxd we have
10 Av Av
6 Aol < |u" A(v 0) + ((v - 8) Ao| S 2/|1-1/2.
Next we notice that ||A, Arl12 - 2t and ||Aall S Then we have
Vu E Sa |o - r - 2 Agu + 4 (og(4e(21++ 1)/6) + 4 log([W + 31s.
Finally, have the upper bound on covering number [& N (3/e)² and, taking 1/t for all fixed 2 1 we have
E | - vT s 1 Ne + 4 log(4e(2 + 1)/6) + 4dlog(3)) + 3.
2
Thus, with probability at least - 8 we have for all t 1
1
T, 3(,, > A. A= 5 1(
2
where ((,,t) = log((4e 21 + 1)/5) + 4d log (31) +3
62

---

Published as a conference paper at ICLR 2024
H TECHNICAL LEMMAS
H. COUNTS TO PSEUDO-COUNTS
Here we state Lemma 8 and Lemma 9 by Ménard et al. (2021).
Lemma 28. On event g for any B(6, :) such that + 3(6, x)/x is non-increasing for x 1,
+ ((6, æ) is non-decreasing Yh [M]. (s. A.
WEN, 3(0,n) (3,
3(0, ni (8, V1
AIS
Lemma 29. For T E N and (u)) for sequence where us E 00, 1] and U2 - Ei= ue we ger
T
u++1
S 4log(UT++1 +
U.V
H.2 COUNTS TO PSEUDO-COUNTS IN LINEAR MDPs
Let [X be sequence of random vectors of dimension d adapted to a filtration {Fi such
that ||Xell2 S I a.s.. Define a sequence of positive semi-definite matrices A, x| Fr-
Notice that ||Ar||2 = T (As) S Also define A, Ala + Σj XJ and A, Ala +
Σj=1 Az-
Lemma 30. Let A 7 0 be a positive semi-definite matrix such that || S 1. Then log det(I + A) _ Tr(A) < 2 log det + A).
Proof. Follows from eigendecompositiinn for A and numeric inequality log(1 + +x) s x - 2 + r)) for all x E [o, 1].
The next result generalized Lemma D.2 by Jin et al. (2020); see also (Abbasi- Yadkori et al., 201 1). Also, it could be treated generalization of Lemma 29 for linear MDPs.
Lemma 31. Let || = for all 2 1. Then for any T
log (et(AA) (eo(
[ s log det(Ar) [X] A-
det(Ao)
Proof. First, we notice that Ar- F - measurable, thus
E - = E Ir [B=
-Tr(AA,-)] Tr
Notice that E, =
A, 1 is positive semi-definite matrix. Then by Lemma 30
log det + 2x) < E - < 2 log det(La E.).
At the same time, we have
(ee) det + A₁) = de det La + 11/2 (A,-11-11//
Thus by telescoping property
log (At(Ar) (Ao)
E[X s log det(Ar) det)
63

---

Published as a conference paper at ICLR 2024
H.3 ON THE BERNSTEIN INEQUALITY
We restate here a Bernstein-type inequality by Talebi & Maillard (2018).
Lemma 32 (Corollary 11 by Talebi Maillard, 2018). Let p. E As, where As denotes the probability simplex of dimension S. For all functions f: 8 [0, defined on S,
pf - qs - (V)r
b (T.0))))
qs pf S v 22ar (f) ( KL(,,
where use the expectation operator defined pf Es (s) and the variance operator defined as Var,(f) (f - (s)) p(f - pf)²
Lemma 33. Let q E As and function ʃ S [0, bl. then
Var, - 2arr, (f) + 4b2 KI(P. 2). Var, (f) < 2ar,, (f) + 4b2 KL(p, q)
Lemma 34. For p. E As, for 1.9: S + [o, b) two functions defined on S, we have that Vfr,(f) - 2Var, (g) + 2bpl - -gl and
Var,(f) S Varp(f) + 312 1p 9l12, where denote the absolute operator by [f((s) [f) all E S.
H.4 CHANGE OF POLICY
Let and n' be two Markovian policies and let 7 (hh) for h' E [H] be family of policies defined as follows
( (8) = (s)))
h + h'
(8)
h = h:.
Lemma 35. For any measurable function f: I. and any h S h it holds
[x[f(sn) = , (sn) sn]-
Moreover, for any measurable g: S - E and any h 2 h'
B Ig
ax) (s, a)] E9 -or) n [g(s (sw) (ww) (s,
Proof. At first, we recall that by definition of a kernel ps we have for any measurable f:
Eg [f on] = psf an), Ex [g(k, [s)] Tag(s)).
(21)
We show the first statement by induction over h < h'. For h = 1 the statement is trivial. Next, we assume that holds for hs and we have show for + 1 < W. By the tower property of conditional expectation and (21)
Ex [= =
an s1] = E=lp an s11] Ex E [pa (sh, an) |s [1 [TA [pr (sn) s1]-
We notice that since h < h' then nr n (b') Then can apply the induction hypothesis function Th (K') (p' ((sn)
E- [= = = [T fl.h) [M (p (sw))ss Ey [f()))))))))
(N) (1))
To show the second statement, use induction over all W. For h/' the statement is trivial. Next, we assume that it holds for h > h and we have to show it for h + 1. Again, using the tower property
Ey (g( (s41))sH anl E, EE [g(gs44 an ( sw, anl E,
an)
64

---

Published as a conference paper at ICLR 2024
(h') aht1 since 2 N. Thus, again applying the tower property, (21), and
We notice that Th+1 induction hypothesis
E [+(#++1, ax+1)) 1881, and
= Fn [T [ (1k) (* sn+ 18a, an and
= Er IPA (TT h+ 0 (8ps an)lsw, ow]
(k')
a [px (- 442 ( (sh,, as)lsw, and = Bging (0(s an+1)lsn awe
65


Reviews:
[{"review_id": "TbtN4zSMhO", "reviewer": "Reviewer_XPKe", "summary": "The authors propose KL-regularized online RL algorithms and provide an upper bound on the sample complexity of this algorithm in tabular MDP and linear MDP settings.", "questions": "I am not an expert in RL theory, I am a bit confused by the results presented in Corollary3 and Theorem6, where the bound presented in Corollary3 depends on $N^E$, whereas the bound in Theorem6 does not. Intuitively, I suppose the sample complexity would eventually depend on the size of the expert dataset. Can authors explain this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors provide a thorough analysis of the algorithms, to justify the efficiency of the RL algorithms with access to an expert dataset.", "weaknesses": "1. There is limited discussion on the relationship between the pure online learning version of LSVI-UCB and UCBVI+.\n2. The paper can be better organized, there are too many references pointing towards the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LOXTQUZkV0", "reviewer": "Reviewer_zApY", "summary": "The paper studied demonstration-regularized reinforcement learning (RL), where the learner first performs behavioral cloning on expert-generated demonstrations using maximum likelihood estimation. Then during the online interaction with the underlying environment, the learner penalizes deviation of the learned policy from the one learned in the behavioral cloning phase. The paper provided a theoretical analysis of these two phases. For behavioral cloning, the authors show that the KL divergence between the learned policy and the expert policy decreases linearly as the number of demonstrations grows. This holds for both tabular MDPs and linear MDPs under certain assumptions. Then based on this result, the authors further studied the regularized online learning scenario and the RLHF setting. In both cases, the authors are able to prove a fast convergence rate for the proposed algorithms.", "questions": "(1) Can you provide some empirical results to validate the theoretical findings of this paper?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) The paper performed a strong and solid theoretical study of behavioral cloning for both tabular and linear MDPs. The authors proved that the KL-divergence between the learner policy and the expert policy decreases linearly as the number of demonstrations grows. The authors also complemented the above positive results with a lower bound on the convergence rate. In terms of the dependency on the number of demonstrates, the upper bound and lower bound match. This is a nice and great result. Based on that, the authors further performed analysis on their proposed demonstration-regularized RL algorithms and the RLHF algorithms, and both achieved surprisingly fast convergence rates. Overall, the paper has made significant technical contributions.\n\n(2) The paper studied a very novel, interesting, yet challenging problem. The topic is of particular interest to the theoretical RL community, and I can forsee that the results of this paper significantly push the frontiers of RL theory and will drive more research along this line.", "weaknesses": "(1) It would be great to include some empirical studies, although this is purely a theory paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t81cK8U25y", "reviewer": "Reviewer_Muxw", "summary": "This paper studies demonstration-regularized RL where an agent is supposed to find a near optimal policy given an offline dataset that is collected from an expert policy. The paper theoretically shows that given $N^E$ expert samples, the sample complexity of finding a $\\epsilon$-optimal policy reduces by a factor of $1/N^E$ in both tabular and linear MDPs. Moreover, the paper extends the proposed method to RLHF and theoretically justify the efficiency of it.", "questions": "While the paper provides the lower bound for the imitation learning, is there any lower bound for the regularized RL?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides comprehensive theoretical results on various settings in demonstration-regularized RL.\n2. The results are nice and show a strong benefit using expert demonstrations.", "weaknesses": "1. I prefer that there is a separated \"related works\" section such that the presentation is clear.\n2. The contributions from the algorithm design part seem not significant. The algorithm is a combination of imitation learning and regularized RL.\n3. The results highly depends on the performance of the expert policy. However, in real life applications, obtaining expert demonstrations is usually expensive, and there might be far less offline demonstrations that that considered in this paper. Specifically, from Corollary 3, it seems that the benefit occurs when $N^E>H^3SA$, which is close to the typical sample complexity in standard RL, which might be too much in real applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HskecbvjVE", "reviewer": "Reviewer_UXmS", "summary": "This paper studies two new hybrid setting which are novel/uncommon in the literature: (i) demonstration regularized RL and (ii) Demonstration Regularized RLHF.\n\nIn (i) both expert demonstrations from an $\\epsilon$-optimal policy and online access to an MDP with reward function are possible. In (ii) the reward function is not available but it can be inferred thanks to a Preference Based Model introduced in Assumption 4.", "questions": "Q1) Is it possible to prove a bound which does not require the assumption that the expert is $\\epsilon$ optimal ?\n\nQ2) Why the regularization is needed in the tabular case but not in the linear one ? I am referring to Section 3.2\n\nQ3) How can UCBVI-Ent+ achieve $\\mathcal{O}(\\epsilon^{-1})$ sample complexity according to Theorem 5 while UCBVI-Ent achieves a worst sample complexity of $\\mathcal{O}(\\epsilon^{-2})$ ?\n\nQ4) What are the definitions of $\\pi^{t,(h)}$ and $\\tilde{\\pi}^t$ in Algorithm 3?\n\nQ5) In the setting of Demonstration Regularized RLHF is it necessary to have the coefficients defined in equation 3 in the bound ?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Both the newly introduced settings are interesting and matches practical situations.", "weaknesses": "There are several technical weaknesses in my opinion.\n\nThe main weakness is in my opinion that the setting seems \n\n1) The lower bound in Theorem 2 turns unfortunately vacuous in the limit of $\\gamma \\rightarrow 0$.\n\n2) It is unclear why the class of linear policies at the third line of Section 3.2 is considered to be not learnable. In fact, under this choice [1] proves in their Theorem 5 that it is possible to recover an $\\epsilon$-suboptimal policy compared to the expert with behavioural cloning.\n\n3) Corollary 3 requires the expert to be $\\mathcal{O}(\\epsilon)$ optimal but I would expect that, given the reward knowledge, it should be possible to prove a sample complexity bound without the assumption on the $\\mathcal{O}(\\epsilon)$ optimality of the expert.\nTo see this think to the case of any BPI algorithm which requires no expert at all to learn an $\\epsilon$-optimal policy.\n\n4) I think that Lemma 11 should be referred as the standard performance difference lemma.\n\n5) Just before Corollary 3, it is said  that \"UCBVI-Ent+ algorithm for regularized BPI. It is a modification of the algorithm UCBVI-Ent by\nTiapkin et al. (2023) with improvement sample complexity\". However, it is not explained which is the crucial difference between the two algorithms. In particular, also the settings are different because UCBVI-Ent+ uses reward information while UCBVI-Ent can be used only for maximum entropy exploration and not to solve Regularized MDPs but this difference is not explained in the main text.\n\n\n[1] ( Rajamaran et al., 2021 ) On the Value of Interaction and Function Approximation in Imitation Learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)