Model-Free Reinforcement Learning with the Decision-Estimation Coefficient
Authors: Dylan J Foster
Noah Golowich
Jian Qian
Alexander Rakhlin
Ayush Sekhari
Venue: NeurIPS 2023 poster (2023)
Keywords: Decision making
learning theory
bandits
reinforcement learning theory
online learning
decision-estimation coefficient
TL;DR: We combine the Decision-Estimation Coefficient (DEC) framework with a notion of "optimistic" estimation, allowing us to tackle model-free reinforcement learning in the DEC framework for the first time.

Abstract: We consider the problem of interactive decision making, encompassing structured bandits and reinforcement
learning with general function approximation. Recently, Foster et al. (2021) introduced the
Decision-Estimation Coefficient, a measure of statistical complexity that lower bounds the optimal regret for interactive decision
making, as well as a meta-algorithm, Estimation-to-Decisions, which achieves upper
bounds in terms of the same quantity. Estimation-to-Decisions is a reduction, which lifts
algorithms for (supervised) online estimation into algorithms for
decision making. In this paper, we show that by combining Estimation-to-Decisions with
a specialized form of "optimistic" estimation introduced by
Zhang (2022), it is possible to obtain guarantees
that improve upon those of Foster et al. (2021) by
accommodating more lenient notions of estimation error. We use this approach to derive regret bounds for
model-free reinforcement learning with value function approximation, and give structural results showing when it can and cannot help more generally.

Model-Free Reinforcement Learning with the Decision-Estimation Coefficient
Dylan Foster dyLanfosteremiccoooff...o
Noah Golowich nzgêmit.edu
Jian Qian jianqian@mi t. edu
Alexander Rakhlin makhlin@mit..m
Ayush Sekhari sekhari@mit t.
Abstract
We consider the problem of interactive decision making, encompassing structured bandits and reinforcement learning with general function approximation. Recently, Foster et al. [12] introduced the Decision-EEsimattio statistical complexity that lower bounds the optimal regret for interactive decision making, as well as a meta-algorithm, Estimatiio-to--eeciiinn,,, which achieves upper bounds in terms of the same quantity. Estimation-to Decisions a reduction. which lifts algorithms for (supervised) online estimation into algorithms for deci- sion making. In this paper, we show that by combining cetimation-to-Deissiie with specialized form of optimistic estimation introduced by Zhang [31], it is possible obtain guarantees that improve upon those of Foster al. [12] by accommodating more lenient notions of estimation error. We use this approach derive regret bounds for model-free reinforcement learning with value function approximation, and give structural results showing when it can and cannot help more generally.
Coefficient, a measure
tonnattiontt--Deciiioonn
1 Introduction
The theory of interactive decision making -ranging from bandits reinforcement learning with function approximation contains variety of sufficient conditions for sample-efficien learning, [25, 17, 26, 11, 19, 29, 18, 23, 3, 20, 10, 22, 9, 33], but necessary conditions have been comparatively unexplored. Recently, however, Foster et al. [12] introduced the (DEC), measure of statistical complexity which leads to upper and lower bounds on the optimal sample complexity for interactive decision making.
Coefficient
Regret bounds based on the Iatision-Estimatiinn Coefficient achieved by Estimation-to-Dectt Deciisions (E2D), a meta-algorithm which reduces the problem interactive decision making to supervised online estimation. While the Decision-Estimation Coefficient leads tight lower bounds regret for many problem settings, the upper bounds in Foster et al. [12] can be suboptimal in certain situations due to the need perform estimation with respect Hellinger distance, a stringent notion of estimation error. When specialized to reinforcement learning, the guarantees for the E2D meta- algorithm in Foster al. [12] are only tight for model-based settings (where function approximation is employed to model and estimate transition probabilities), and do not lead to meaningful guarantees for model-free settings with value function approximation. In this paper, we explore the prospect of developing tighter regret bounds suitable for model- -free settings.
We show that by combining Estimation-Be ciiimation-to-Diee tee--tt--Deeeee toDeeciions with optimistic online estima- tion, elegant technique recently introduced by Zhang [31], it is possible obtain regret bounds that improve upon Foster et al. [12] by accommodating weaker notions estimation error. Our main contributions are:
Contributions.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

We introduce new optimistic variant of the Docision-Estimation Coefficient, and show that variant of detimation-to-Deciiiin that incorporates optimistic estimation achieves regret bounds that scale with this quantity (Section 2). Using this approach, we derive the first regret bounds for Estimation-to- -oee--ss------sss applied to model-free reinforcement learning with bilinear classes [11] (Section 2.3).
We show that in general. whether or not optimistic estimation leads to improvement depends the divergence with respect to which estimation is performed: For symmetric divergences, optimistic estimation offers no improvement, but for asymmetric divergences, including those found in reinforcement learning, the improvement addition, we highlight settings in which combining optimistic estimation with Estimation-to- Decisions offers provable improvement over previous approaches that apply the technique with posterior sampling [31].
be drastic (Section 3). In
Perhaps the most important aspect of our work to elucidate the connection between the DEC framework and optimistic estimation, building foundations for further research into these techniques.
In what follows, we review the algorithm (Section 1.2), highlighting opportunities for improvement. In Section 2, we present our main results, including our application to model-free reinforcement learning. We close with discussion and structural results, highlighting situations in which optimistic estimation can and cannot help (Section 3).
Coefficient and Estimation-to-Deeee
meta-
1.1 Problem Setting
We adopt the Decision Making with Structured Observations (DMSO) framework of Foster et al. [12), which a general setting for interactive decision making that encompasses bandit problems (structured, contextual, and so forth) and reinforcement learning with function approximation.
The protocol consists of T rounds. For each round t I....,T::
1. The learner selects decision x' II, where IT is the decision space.
2. The learner receives a reward r' E R C R and observation o' E O sampled via (r",o') A(R O) is the underlying model.
M MM (I'), where M*
Above, R is the reward space and O is the observation space. The model (conditional distribution) M' represents the underlying environment, and unknown to the learner, but the learner assumed to have access to model class M C (II A(R O)) that is flexible enough to capture M
Assumption 1.1 (Realizability). The learner has access to a model class M containing the true model M
The model class M represents the learner's prior knowledge about the decision making problem, and allows one to appeal to estimation and function approximation. For structured bandit problems, models correspond to reward distributions, and M encodes structure in the reward landscape. For reinforcement learning problems, models correspond to Markov decision processes (MDPs), and M typically encodes structure in value functions or transition probabilities. We refer to Foster et al. [12] for further background.
For model M E M, [.) denotes the expectation under the process (r.o) M(),, (T) == EM [r] denotes the mean reward function, and Tм := arg max (*) de- notes the optimal decision. We measure performance in terms of regret, which is given by Regom Σ Exenge
far (m*)), where p' is the learner's randomization dis-
tribution for round l.
Additional notation. let (()) denote the set of all probability distributions over Z. For a model class M, co(M) denotes the convex hull. We write f = Ö(g) to denote that f O(g max(1, polylog(g)) }), and use S as shorthand for O(b).
For an integer n E N, we let [n]] denote the set {1....,
nn.. (nn...r For a set Z, we
1.2 Background: Cedimattionto-Deciitee and
Coefficient
To motivate our results, this section provides a primer on the cotnaationnt-Deeiiiion meta-algorithm and the Esstimatintiiniiinnnnno Coefficient. We refer to Foster et al. [12] for further background.

---

Algorithm Esstmattion-to-eecisiins (E2D) for General Divergences
parameters: Estimation oracle AlgEst Exp. parameter 7 > divergence ( (.ll1)).
2: for i 1,,,... , do
3: Compute estimate M Alges r'))),)),
4: p' <- arg me SuPMrM Expp/ | (Tx) (x) - D (M M) // //Eq. (2).
5: Sample decision X' ~ p' and update estimation oracle with (x',r',00
Online estimation. Estimation-to Diiimation-to-eecisoon (Algorithm 1) is reduction that lifts algorithms for on- line estimation into algorithms for decision making. An online estimation oracle, denoted by Algest an algorithm that, using knowledge of the class M, estimates the underlying model M from data in sequential fashion. At each round I, given the data H ( ,r', ","),"...,. (IT 1, 1,00
observed so far, the estimation oracle computes an estimate M Algés (1m",r','))) [ for the true model M
To measure the estimation oracle's performance, make use of a user-specified divergence- func- tion, which quantifies the discrepancy between models. Formally, we define divergence-liks like function (henceforth, "divergence") representing the discrepancy between the models M and M' at the decision и. Standard choices used in past work [12, 13, 15, 14, 27] include the squared error D (M, M³) (f (T) f (x))² for bandit problems, and squared Hellinger distance' (H M')
any function D II co(M) x co(M) + R+, with D (M M')
(T)) for RL.
where for distributions IP and Q, (I,, Q) == [(VPP- va 2 We then measure the estimation oracle's performance terms of cumulative estimation error with respect D, defined as
T
Est D
Σ
Exings [DD
M M.
(1)
where p' is the conditional distribution over n' given H²-1. We make the following assumption on the algorithm's 's performance.
Assumption 1.2. At each time E [T], the online estimation oracle Algest returns, given (IT), (o)),o'),.... ...., (m'-),," , with (r', M* and n' p', estimator M' II + A(R x O) such that Est Est. (T, 8), with probability at least where Est D (T, 5) is known upper bound.
For the squared error, one can obtain (T,5) Est (T) S Fog(FMM//) where FM = {f" M E M), and for Hellinger distance, it is possible to obtain Est H Bstt((T,6) = Est Pa (T, 6) S log(LM//)).
general version of the E2D meta-algorithm is displayed in Algorithm 1. At each timestep t, the algorithm queries estimation oracle to obtain an estimator M using the data (x² ,,,,, """,o"),... (T- -d o²=1) observed far. The algorithm then computes the decision distribution p' by solving a min-max optimization problem involving M and M (as well as the divergence D), and then samples the decision n' from this distribution.
Estimatioo-too-Deciiionss
The Decision-Estimation Coefficient. The min-max optimization problem in Algorithm 1 is derived from the Decision-Estimation Coefficient (DEC), a complexity measure whose value, for given scale parameter y > 0 and reference model M πI - A(R x O), given by
dec (M, = inf sup Excop ( - (M) - D7 (M M))
(2)
PEA(II) MEM
with dec, (M) sup d (M, M). Informally, the DEC measures the best tradeoff
between suboptimality (f (MT)) - fM (T)) and information gain (measured by D (M || M)) that can be achieved by decision distribution p in the face of a worst-case model M E M.
When (= (I-) is symmetric, write (T :) make this explicit.

---

The main result of Foster et al. [12] shows that the regret of E2D is controlled by the DEC and the estimation oracle's cumulative error Est Let M be any set for which M E M for all t almost surely. We have the following guarantee.
Theorem 1.1 (Foster al. [12]). Algorithu with exploration parameter y 0 guarantees that + ry Est almost surely.
RegoM I 7 dec., (M,
For the special case of Hellinger distance, standard algorithms (exponential weights) achieve Est. (T, x log(IM/6) with M = co(M), so that Theorem 1.1 gives (abbreviating dec,, = dec..)) DH):
Regom X dec", T + log(1
Opportunities for improvement. Foster et al. [12, 15] provide lower bounds on the regret for any decision making problem that have similar expression to (3), showing that the Decision-Estimation Coefficient with Hellinger distance (dect) (M)) plays a fundamental role in determining the statistical complexity of decision making. However, these lower bounds do contain the estimation term Est (T, 5) log( /0))) appearing in (3), and thus capture the price of moving from estimation to decision making (as characterized by the DEC), but not the complexity of estimation itself.
In general, the dependence on Est H| (T, 6) = log((M)/8) in the upper bound (3) can render the bound loose. In reinforcement learning, working with Hellinger distance necessitates modeling transition probabilities. While this leads to optimal results in some settings, in general the optimal rates for Hellinger estimation error can be prohibitively large, even in settings where model-free (or, value-based) methods, which directly model value functions, are known to succeed; this drawback is shared by all subsequent work based the DEC [13, 5, 15, 14, 27]. A natural solution is to replace Hellinger distance with a divergence (e.g.. based on Bellman error) tailored to value function approximation, but naive choices for D along these lines render dec.," (M, too large to give meaningful guarantees, and Foster et al. [12, 15] left this as an open problem.
2 eetimaationtt-Deciiiti with Optimistic Estimation
To derive improved regret bounds that address the shortcomings described in the prequel, we combine ciimattonnto-Decciionn with a specialized estimation approach introduced by Zhang [31] (see also Dann et al. [6], Agarwal and Zhang [1, 2]. Zhong et al. [32]), which refer to as optimistic estimation. We then use this approach to derive regret bounds for model-free reinforcement learning.
2.1 Optimistic Estimation
The idea of optimistic estimation is to augment the estimation objective with a bonus that biases the estimator toward models M .M for which the value fM1(T w. is large. We present general version of the technique.
Let divergence (= || estimation oracle A Algest is an algorithm which, at each step I, given the observations and rewards collected so far, computes an estimate for the underlying model. For technical reasons, it will be useful to consider randomized estimators (Foster et al. [12], Chen et al. [5]) that, at each round, produce a
be fixed. Following the development in Section 1.2, optimistic
distribution μe² E A(M) over models. Such estimators take the form μt" = Algést (((T), ,'',,,', "irr,,"',''))"" [=)=) where Je E A(M). For a parameter y > O. we define the optimistic estimation error as
T
OptEst = Σ
Extros E M D M
M-) +
(TTM)* - pit (TT.))
(4)
This quantity is similar to (1), but incorporates a bonus term 7 "(p"" (IT fa (TTZ))) which encourages the estimation algorithm to over-estimate the optimal value fM (Txr) ) for the underlying model M
Foor simpler model classes, it is possible improve upon (3) by moving from Hellinger distance lenient notions estimation error: bandit problems with Gaussian rewards, suffices consider DE (M, M) (T)( ((
(=))², which leads to upper bounds that scale with log]FMl < log [12].

---

Algorithm 2 Optimistic Estimation-to-Deciiiooi (E2D.Opt)
1: params: Estimation oracle AlgEs: Exp. param. 7 > divergence D with suff. stat. space V.
2: for == 1,,,... 7 do
3: Receive randomized estimator μ² AVV) Algest ( r')))
4: Compute
PEA(T) MEM p' arg min sup Enop Binny (TT)) (T) D
( (01lM) //Eq. (6).
5: Sample decision n' ~ ~'p' and update estimation oracle with (T',r",," 0²).
Assumption 2.1. At each time 1. E [T], the optimistic estimation oracle Algest returns, given (m), "rrr,"")...... ( 1, r, 1 o 1) with (r',,") (T') and x¹ p', randomized estimator u¹ E (MM) such that OptEst S OptEst, (T,8), with w.p. 1 6, where OptEst, (T,8) known upper bound.
For the case of contextual bandits, Zhang [31] proposes an augmented version of the ex- ponential weights algorithm which, for a learning rate parameter > 0. sets μ(M) x exp(-r(Z-((M) - - (TTx)) where T.(((a) is the squared prediction error for the rewards observed far. This method achieves E OptEst S log([FM)) + logl
and Zhang [31] combines this estimator with posterior sampling to achieve optimal contextual bandit regret. Agarwal and Zhang [1], Zhong et al. [32], Agarwal and Zhang [2] extend this development to rein- forcement learning, also using posterior sampling as the exploration mechanism." 3 In what follows, we combine optimistic estimation with Estimation-to-Deede Doceecis,, which provides a universal mechanism for exploration. Beyond giving guarantees which were previously out of reach for E2D (Section 2.3), this approach generalizes and subsumes posterior sampling, and can succeed in situations where posterior sampling fails (Section 3).
For the non-optimistic estimation error Est ? it is possible to obtain low error for well- behaved losses such as the square loss and Hellinger distance without the use of randomization by appealing to improper mixture estimators (e.g., Foster et al. [12[). We show in Section 3 that for such divergences, randomization does not lead to statistical improvements. For the optimistic estimation error (4), randomization essential due to the presence of the term - (f" " (Ta-) - f (T <
Remark 2.1.
Sufficient statistics. Before proceeding, note that many divergences of interest have the useful property that they depend on the estimated model M only through "sufficient statistic" for the model class under consideration. Formally, there exists a sufficient statistic space Vi and sufficient statistic Vb M + with the property that we can write (overloading notation)
(M II M') D" (*b() M'), (*) = ( and
Tм = *M)
for all models M, M' In this case, it suffices for the online estimation oracle to directly estimate the sufficient statistic by producing randomized estimator je E A(V). We measure performance via
T
OptEst, = En E Br Dnd (7' (e'I ( + + "f" ((""((TTT) 8dh (TT
(5)
1=1
Examples include bandit problems, where one may use squared estimation error D (...) and take (M( fM, and model-free reinforcement learning, where we show that by choosing the divergence D appropriately, one can use Q-value functions as a sufficient statistic. Note that we only focus on sufficient statistics for the first argument to D (- -), since this is the quantity we wish to estimate.
2.2 Algorithm and Main Result
We provide an optimistic variant of the E2D meta-algorithm (E2D.Opt) in Algorithm 2. At each timestep f, the algorithm calls the estimation oracle to obtain a randomized estimator μı' using the data
Daan et al. [6] also apply the optimistic estimation idea to model-free reinforcement learning, but do not provide online estimation guarantees.
5

---

(T',r', pr''.o).. o'......,TT............ (I'-),r""1 -p--1_0-1 collected so far. The algorithm then uses the estimator to compute a distribution p' E (III) and samples x¹ from this distribution, with the main change relative to Algorithm 1 being that the minimax problem in Algorithm 2 is derived from an "optimistic" variant Coefficient. For μ E A(M),
of the DEC, which we refer to as the Optimistic Decision-Estimen define
o-deo,
(M, O-deo,(M,,
inf sup Empp E M - ( DE MM
M)).
(6)
PEA(I) MEM
and o-dec. D (M) (0M)=supp (UM o-dec, (M. The Optimistic DEC has two difference from the original DEC. First, it is parameterized by a distribution j (M rather than reference model M 11 + AR x O), which reflects the use of randomized estimators; the value in (6) takes the expectation over reference model M drawn from this distribution (this modification also appears in the randomized DEC introduced in Foster et al. [12]. Second, and more critically, the optimal value 1 (T in (2) is replaced by the optimal value f (xx)) for the (randomized) reference model. This seemingly small change is the main advantage of incorporating optimistic estimation, and makes it possible to bound the Optimistic DEC for certain divergences D for which the value of the unmodified DEC would otherwise be unbounded (ef. Section 3).
Remark 2.2. When the divergence D admits sufficient statistic ab .M- - V for any distribution J E (MM,, if we define v E (V) via v(t) m({M E M: (MM = w²)). we have
inf sup E Exp Eap- [1( - (MT 7 D (wb M)]. o-dec (M, PEA(IT) MEM
In this case, by overloading notation slightly, SU)) o-dec (M,
we may write o-dec D (M)
Main result. Our main result shows that the regret of Optimistic Estimation-to- --e--isione is controlled by the Optimistic DEC and the optimistic estimation error for the oracle Alg
Theorem 2.1. For any ō > 0, Algorithm ensures that with probability at least 1 - 5,
RegoM I o-dec (M) T OptFsst (T,
This regret bound has the same structure as Theorem 1.1, with the DEC and estimation error replaced by their optimistic counterparts. In the remainder of the paper, we show that 1) by adopting asymmetric divergences specialized to reinforcement learning, this result leads to the first guarantees for model-fres RL with E2D, but 2) for symmetric divergences such as Hellinger distance, the result never improves upon Theorem 1.1.
Estimation with batching. For our application reinforcement learning, we generalize the results above to accomodate estimation algorithms that draw batches of multiple samples from each distribu- tion p'. Given a batch size n, we break the T rounds of the decision making protocol into K : T/n contiguous epochs (or, "iterations"). Within each epoch, the learner's 's distribution p is unchanged (we index by rather than i to reflect this), and we create a batch B* ( pra | by sampling nhd p² independently and observing (r*h1 otdt) M (mt for each I E [n]. We can then appeal to estimation algorithms of the form 1e* Algês ((B))==)). Regret bounds for variant of E2t.Opt with batching are given in Appendix B.
2.3 Application to Model-Free Reinforcement Learning
In this section, we use Optimistic Estimation-to-Deeiitte to provide sample-efficien guarantees for model-free reinforcement learning with bilinear classes [11], a general class of tractable reinforcement learning problems which encompasses many settings [17, 26, 7, 29, 18, 23, 3, 20, 10, 22, 9, 33).
Reinforcement learning preliminaries. To state our results, let us recall how reinforcement learning fits into the DMSO framework. We consider an episodic, finite-horizon reinforcement learning setting, With H denoting the horizon, each model M E .M specifies non-stationary Markov decision process
(S, A, {PA 1 Ja-1 { 1 d1 1, }. where S is the state space, A is the action space, PM S x A > (SS) is the probability transition distribution at step h, Rg :S x A - A(R) is the reward distribution, and dj E ((S)) is the initial state distribution. We allow the reward distribution and transition kernel to vary across models M, but assume that the initial state distribution is fixed.

---

For fixed MDP M E M,, each episode proceeds under the following protocol. At the beginning of the episode, the learner selects randomized, non-stationary policy = (Tj..... ................................ TT), where Th S + ((A); we let IRRS (for "randomized, non-stationary") denote the set of all such policies. The episode then evolves through the following process, beginning from 81 ~ d1: For h = 1,..., H: ap Tr (8k), Th ~ R (sh, an), and 8h+1 ~ PM 8h, aq). For notational convenience, we take 5H+1 to be a deterministic terminal state. We assume for simplicity that R C [0,1] (that is, Σ rh E [0, 1] almost surely). Within the DMSO framework, at each time t, the learning agent chooses σt' E IRRSS, then observes the cumulative reward r¹ - 2h=17 rh and trajectory o' = (si, ai, ri),..., (6 "n,, rH)) that results from executing n².
Value functions. The value for a policy x under M is given by (z) := EMM Eh=1 rn.. where [-] EM." the state-action value function and state value functions via 2N a) = EM," Em= TW 8h =
denotes expectation under the process above. For given model M and policy TT, we define
8, Oh a]. and Vi (s) EM," [EN= T'e 8% = s]. We define TM the optimal policy, which maximizes Q% 2, (s, for all states simultaneously. We abbreviate Qu: QM,TT.r
Value function approximation. To apply results to reinforcement learning, we take model-free (or, value function approximation) approach, and estimate value functions for the underlying MDP M*; this contrasts with model-based methods, such those considered in Foster et al. [12], which estimate transition probabilities for M* directly. We assume access to a class Q of value functions of the form Q = (Q1...., Q.)..
Assumption 2.2. The value function class Q has Q"", E Q. where M is the underlying model.
For Q = (Q....., QH) E Q. we define TQQ = (TQ..I...,,TT) TQ.H) via (s) arg m as) We define Ile {m Q E 2} as the induced policy class. While is not necessary for our results, we mention in passing that the class Q, under Assumption 2.2,
implicitly induces model class via Me == {M Q Qa,** Q).
Bilinear classes. The bilinear class framework [11] gives structural conditions for sample-efficient reinforcement learning that capture most known settings where tractable guarantees are possible. The following is an adaptation of the definition from Du et al. [11].*
Definition 2.1 (Bilinear class), An MDP M is said to be bilinear with dimension d relative to a class Q if:
(i) There exist functions Wa(:; Q + Rd, Xx(:;M) Q + Bd such that for all Q E Q and {H.
E M.T
Qn (sh, an) - max
Qn+1(sht1. ((XX.CQ:M)). (8) d'ed
(i) Let 24 == (Sh, Oh Th, 1h There exists a collection of estimation policies { I qee and a discrepancy function exat (-:-) Q x Z + IR such that for all Q E Q and h E [H].
(X,(Q, W. (Q'; M) E Mg oh m
Cost (Q'; [n)]]
If TQ TQ. we say that estimation is on-policy. We assume IEE (M (Q":+ zt)]] for all π.
We let duil Q M) denote the minimal dimension for which the bilinear property holds for M. For model class M, we define duil M) = sup M dui (Q; We let Lui (2; M) 2 1 denote any almost sure upper bound on c. (Q; zn)] under M.
'For the sake of simplicity, we adopt a less general definition than Du et al. [11]: We 1) assume that the "hypothesis class" is parameterized by the Q- -ufncttion class Q. and 2) limit to discrepancy functions that do not explicitly depend on the function Q indexing the factor (Q; M). The results here readily extend to the full definition.
For π and n'. n on n' denotes the policy that follows x for layers 1,..., h - 1 and follows T' for layers H.
7

---

2.4 Guarantees for Bilinear Classes
We now apply our main results to derive regret bounds for bilinear classes. We first provide optimistic estimation guarantees, then bound the Optimistic DEC, and conclude by applying Ept.
We take V/r Q as the sufficient statistic space, with (b(M) == QM:², and define (TQQ == Eeg-dd (s1. 1. oo (s))]. For the divergence D, we appeal to squared discrepancy, in the vein of
Jiang et al. [17], Du et al. [11]:
DE || M) ((BM)
B [
Co (Q:
(10)
We abbreviate OptEst OptEst, Pui and o-dec)/ (M, o-deo (M, p))
Estimation. To perform estimation, we approximate the average discrepancy in (14) from samples (drawing a batch of samples each step; Algorithm 3), then appeal to the exponential weights method for online learning, with a bonus to enforce optimism. See Algorithm 6 (deferred to Appendix D for space).
Proposition 2.1. For any batch size n (with K := T/n) and parameter 2 1, Algorithm 6, with an appropriate learning rate, ensures that with probability at least 1-6, 6,
OptEst < VKlog Q
K n1 )
1+
(11)
whenever M bilinear relative to and Assumption 2.2 is satisfied.
This result does not actually use the bilinear class structure, and gives bound on (14) for any choice of gest Similar to the optimistic estimation result given by Zhang [31] for contextual
bandits, this guarantee consists of "slow" term / [ resulting from optimism (which decays as y grows), and "fast" term. However, compared the contextual bandit setting, the fast term, + K). scales with the ratio which reflects sampling error in the estimated discrepancy, and necessitates large batch size. Previous algorithms for bilinear classes [17, 11] require large batch sizes for similar reasons (specifically, because the expectation in (14) appears inside the square, it is not possible to form unbiased estimate for Dar directly).
Bounding the DEC. A bound on the Optimistic DEC follows by adapting arguments in Foster et al. [12]: our result has slightly improved dependence on H compared the bounds in that work.
Proposition 2.2. Let M be any model class for which the bilinear class property holds relative to Q.
(I) In the on-policy case where Tt TQ. we have that for all y > 0, o-dech) S (Q.M(Q.M)
(2) In the general case (T * Tq). we have that for all 2 H²dmi MM, o-dec (M) S
Ha H2 (Q:M)
Combining Theorem B.1, Proposition 2, and Proposition 2.2, we obtain the following result.
Corollary 2.1 (Regret bound for bilinear classes). Let Q given. Assume that M* E M, where M is bilinear relative Q, and that Assumption 2.2 holds. Abbreviate d dsi( Q: and == Lbi ( M) == menMem Lbi( Q; M). For an appropriate choice of n and y, Algorithm 3, using the algorithm from Proposition 2.1 an oracle, enjoys the following guarantees with probability at least 1 1-6 8:
(1) In the on-policy case where eat =Q: RegOM S log (FFA/))oc((1TTH/))))))7
(2) In the general case where πQ
est + πQ: RegoM S
This is the first regret bound for model-free reinforcement learning with the Estimation-tt-Deeisse meta-algorithm. Importantly, the result scales only with the horizon, the dimension d, and the capacity log] Improving the dependence on 7 in Corollary 2.1 is an interesting question: currently, there are no algorithms for general bilinear classes that achieve VT regret without additional assumptions. Let us emphasize that regret bounds for bilinear classes can already be achieved by a number of existing methods [11, 19]. The contribution here to show that such guarantees can achieved through the general DEC framework, thereby placing this line of research stronger foundations.
8

---

Tighter guarantees under Bellman completeness. In Appendix B.2 (deferred for space), we adapt techniques from Agarwal and Zhang [2] to derive tighter estimation guarantees when Q satisfies a Bellman completeness property (e.g.. Zanette et al. [30], Jin et al. [19]), by appealing to refined algorithms tailored to squared Bellman error. We use this to derive tighter regret bounds for bilinear classes (T2/3 instead of 7334).
3 Understanding the Role of Optimistic Estimation
We close with discussion and interpretation of our results.
When does optimistic estimation help? Perhaps the most pressing question at this point is to understand when the regret bound for EpD.Opt (Theorem 2.1) improves upon the corresponding regret bound for vanilla E2D (Theorem 1.1). In what follows, we show that: (1) For any divergence D, the Optimistic DEC is equivalent to variant of the original DEC which incorporates randomized estimators [12], but with the arguments to the divergence flipped; (2) For divergences D that satisfy triangle inequality, this randomized DEC is equivalent to the original DEC itself. Together these results show that the improvement given by the Optimistic DEC is limited to asymmetric divergences such as the bilinear divergence in Section 2.3; for more traditional divergences such as Hellinger distance and squared error, the optimistic approach offers no improvement. Our results use the following regularity assumption, satisfied by all standard divergences.
Assumption 3.1. For all M, M E co(M). (fT f") - Lipp D (M M M) for a constant Lipp > O.
Given divergence D, we define the flipped divergence, which swaps the first and second arguments, by D= (M || M) == D (M || M). We define the Decision-Estimation Coefficient for randomized estimators [12, 5] as dec, (M, 14) = inf AMM SUPMEM [f (Tx) (*) ~ [D (M M]]] with dec? (M) == SuP dec,(M,, This definition is identical to (2), but allows M to be randomized.
Proposition 3.1. Whenever Assumption 3.1 holds, we have that for all y 0.
Lip o-deo, < /ec + 27
Lip
dec /
(12)
2
For settings in which there exists estimation oracle for which the flipped estimation error Est 5 = -E Eaast E M Da (M* I| M. is controlled, this result shows that to match the guarantee in Theorem 2.1, optimism is not required, and it suffices to run a variant of vanilla E2D that incorporates randomized estimators (cf. Foster et al. [12], Section 4.3).
We now turn the role of randomization. When D is convex in the first argument, we have dec, (M) I sup (c) dec, (M, M) = dec, (M), but it not immediately apparent whether the opposite direction of this inequality holds, and one might hope that working with the randomized DEC in Proposition 3.1 would lead to improvements over the nandranddd counterpart in Theorem 1.1. The next result shows that this not the case: Under mild assumptions on the divergence D, randomization offers no improvement.
Proposition 3.2. Let D be any bounded divergence such that for all M, M', M and क E II, (M M²) s C(D (M M) + D (M || M')). Then for all y > O, sup dec (M, M) S D
dec)/(2c) (M).
Implication for Hellinger distance. Squared Hellinger distance is symmetric, satisfies Assumption
3.1 with Tapp whenever R C [0, 1). and satisfies the condition in Proposition 3.2 with C = 2. Hence, by combining Proposition 3.1 with Proposition 3.2, we obtain the following corollary.
Corollary 3.1. If C [0, 1]. then o-dect, (M) - + S sup dec (M, M) S o-dect) / +
Vx > 0.
This shows that for Hellinger distance- at least from a statistical perspective-thed is no benefit to using the Optimistic DEC or randomized DEC compared to the original version. particular, this implics that regret bounds based on the randomized DEC with Hellinger distance (such as those found

---

in recent work of Chen et al. [5]) do not offer improvement over the guarantees for vanilla E2D in Foster et al. [12]. One caveat, though, that working with the Optimistic DEC, as well as randomized estimators, has potential to give computational improvement, as computing distribution E A(T)) that minimizes o-dec" (MM, 14) might be simpler than computing a corresponding distribution for dect; (M, .) with M co(M). We are not currently aware of any examples where such an improvement occurs, as even maintaining a distribution μ E ((M) requires intractably large memory for most classes of interest.
Implication for model-free RL. The bilinear divergence ( M) (B Σi=
1. (Q; (Q;zn)]) =1) that we adopt in Section 2.3 is asymmetric, as are closely re- lated divergences such squared Bellman error. Here, there are two reasons why optimistic estimation offers meaningful advantages.
(1) By Proposition 3.1 (Dai satisfies Assumption 3.1 with Lip = O(H)), natural alternative to optimistic estimation is estimate with respect to the flipped divergence Dos then appeal to Algorithm 3 of Foster et al. [12]. The issue with this approach is that minimizing the flipped estimation error, which takes the form
Est Du
Σ Ena E Exr D (Q )
Σ
E Bx
E Mr Sugel
E
E
is challenging in model-free settings; we are not aware of any algorithms that accomplish this.
(2) Alternatively, second choice is to perform estimation with respect to the un-flipped divergence Doi (which can be accomplished with Proposition 2.1 by taking - oc), and appeal to vanilla E2D (either Algorithm 1, or Algorithm 3 of Foster et al. [12] if one wishes to incorporate randomized estimators). However, the following result shows that unlike the Optimistic DEC, the original DEC with the divergence Das does not admit a favorable bound, even for tabular reinforcement learning.
Proposition 3.3. Let M be the class of all horizon-H H labular MDPs with |S| 2 and |A| 2. Consider the discrepancy function 1e5t(O (Q; 2n) = (Qn. an) - Th max Qh+1 (8h+1, d')) Then we have o-dect) (M) x 7. yet there exists M M for which dec (M, M)
H.
2 2H A
Insufficiency of posterior sampling. For contextual bandits, where o-dec (M) |A|
y [31], and
bilinear classes, where o-dec (M) x H-dy((M) (Proposition 2.2), a strategy that achieves the bound on the Optimistic DEC is posterior sampling (this is also the approach taken in Agarwal and Zhang [1, 2]. Zhong et al. [32]). That is, given distribution E ((M). choosing p(T)) = nt M{{M E
7
M TM })) in (6) certifies the desired bound on o-dec, (M, (u) for these examples. Optimistic Estimation-to-Dest Deeiiiissn subsumes and generalizes posterior sampling, but in light of the fact that this simple strategy succeeds for large classes of problems, it is reasonable to ask if there is a sense in which posterior sampling is universal, and whether can achieve the value of the Optimistic DEC for any model class. This would be desirable, since it would indicate that solving the minimax problem in Algorithm is not necessary. The following sample shows that this is not the case: there are model classes (specifically, MDPs with a constant number of actions) for which the regret of posterior sampling is exponentially large compared to the regret of Algorithm 2.
Proposition 3.4. Consider the divergence DH .). For any S SEP N and H 2 lSg, there exists a
class of horizon-H H MDPs M with |8| = and [AI = 3 that satisfies the following properties:
There exists an estimation oracle with OptEst" S log(S/)) w.p. least 8 for all 7 > 0.
Posterior sampling, which sets p' (T) = 1 ({M TM = π }},, has E. ReRegpM] 2 S ^ 28(H).
Algorithm 2 with divergence D DF(: has E[Regom] < Ö(VTlog(SS)
This shows that posterior sampling does not provide universal mechanism for exploration, and highlights the need for deliberate strategies such as E2D.
'While Foster et al. [12] do give regret bounds for model-free RL using this divergence, they only bound Bayesian regret, and do not provide an explicit algorithm for the frequentist setting.
10

---

Acknowledgements
NG is supported by a Fannie & John Hertz Foundation Fellowship and an NSF Graduate Fellowship. AR acknowledges support from the NSF through award DMS-2031883, from the ARO through award W911NF-21-1-0328, and from the DOE through award DE-SC0022199
References
[1] A. Agarwal and T. Zhang. Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. Advances in Neural Information Processing Systems, 35: 35284-35297, 2022.
[2] A. Agarwal and T. Zhang. Non-linear reinforcement learning in large action spaces: Structural conditions and samplc-effiiincyy of posterior sampling. Conference on Learning Theory, pages 2776-28 22682484422 14. PMLR, 2022.
[3] A. Ayoub, Z. Jia, C. Szepesvari, M. Wang. and L. Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning, pages 463 474. PMLR, 2020.
[4] N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games. Cambridge University Press, New York, NY, USA, 2006. ISBN 052 0921841089,
[5] F. Chen, S. Mei, and Y. Bai. Unified algorithms for RL with decision-estimation coefficients: No-regret, PAC, and reward-free learning. arXiv preprint (xXiv:2209.11455, 2022.
[6] C. Dann, M. Mohri, T. Zhang, and Zimmert. A provably efficient model-free posterior sam- pling method for episodic reinforcement learning. Advances in Neural Information P Processing Systems, 34:1 10100-20055 2021.
[7] S. Dean, H. Mania, Matni, B. Recht, and S. Tu. the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics, 20(4):633-67) 2020.
[8] O. D. Domingues, P. Ménard, E. Kaufmann, and M. Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory, pages 578-598. PMLR, 2021.
[9] S. Dong, B. Van Roy, and Z. Zhou. Provably efficient reinforcement learning with aggregated states. arXiv preprint /912.06666 2019.
[10] S. Du, A. Krishnamurthy, N. Jiang, A. Agarwal, M. Dudik, and J. Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning, pages 1665-1674. PMLR, 2019.
[11] S. S. Du, S. M. Kakade, J. D. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang. Bilinear classes: A structural framework for provable generalization RL. International Conference on Machine Learning, 2021.
[12] D. J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin. The statistical complexity of interactive 2021.
decision making. arXiv preprint arXiv:2
[13] D. J. Foster, A. Rakhlin, A. Sekhari, and K. Sridharan. On the complexity of adversarial decision making. Neural Information Processing Systems (NeurIPS), 2022.
[14] D. J. Foster, D. P. Foster, N. Golowich, and A. Rakhlin. On the complexity of multi-agent decision making: From learning in games to partial monitoring. Conference on Learning Theory (COLT), 2023.
[15] D. Foster, N. Golowich, and Y. Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. Conference on Learning Theory (COLT), 2023.
[16] D. A. Freedman. On tail probabilities for martingales. the Annals of Probability, pages 100-118, 1975.
[17] N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In International Conference on Machine Learning, pages 1704-1713, 2017.
[18) C. Jin, Z. Yang, Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference Learning Theory, pages 2137-2143, 2020.
11

---

[19] C. Jin, Q. Liu, and S. Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficiet algorithms. Neural Information Processing Systems, 2021.
[20] A. Krishnamurthy, A. Agarwal, and J. Langford. PAC reinforcement learning with rich observa- tions. In Advances in Neural Information Processing Systems, pages 1840-1848, 2016.
[21] T. Lattimore and C. Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
[22] L. Li. A unifying framework for computational reinforcement learning theory. Rutgers, The State University of New Jersey- -New Brunswick, 2009.
[23] A. Modi, N. Jiang, A. Tewari, and S. Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics, pages 2010-2020. PMLR, 2020.
[24] I. Osband and B. Van Roy. On lower bounds for regret in reinforcement learning. arXiv preprint 1608 02732, 2016.
[25] D. Russo and B. Van Roy. Eluder dimension and the sample complexity of optimistic exploration.
In Advances in Neural Information Processing Systems, pages 2256-2264, 2013.
[26] W. Sun, N. Jiang, A. Krishnamurthy, A. Agarwal, and J. Langford. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Conference on learning theory, pages 2898-2933. PMLR, 2019.
[27] A. Wagenmaker and D. J. Foster. Instance-opiimality in interactive decision making: Toward non-asymptotic theory. Conference on Learning Theory (COLT), 2023.
[28] Xie, J. Foster, Y. Bai, N. Jiang, and M. Kakade. The role of coverage online reinforcement learning. International Conference on Learning Representations (ICLR), 2023.
[29] L. Yang and M. Wang. Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning, pages 6995-7004. PMLR, 2019.
[30] A. Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning, pages 10978-10989. PMLR, 2020.
[31] T. Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science, 4(2):834-857, 2022.
[32] H. Zhong, W. Xiong, S. Zheng, L. Wang, Z. Wang. Z. Yang, and T. Zhang. A posterior sampling framework for interactive decision making, arXiv preprint arXiv:2211.0196 2022.
[33] D. Zhou, Q. Gu, and C. szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576 PMLR, 2021.
12

---

Contents of Appendix
A Additional Related Work
13
Additional Results
13
B.1
Optimistic citimaationto-Deciiiin with Batching
13
B.2
Model-Free RL: Tighter Guarantees under Bellman Completeness
14
B.3
Proofs from Appendix B.2
15
C Technical Tools
17
C.1 Preliminaries
17
C.2 Basic Online Learning Results
17
C.3 Online Learning with Completeness
18
Proofs from Section 2 and Section
30
D.I Proofs from Section 2
................................
D.2 Proofs from Section 3
A Additional Related Work
In this section we discuss additional related work not already covered.
Chen et al. [5] build on the DEC framework by giving regret bounds for a E2D that incorporates randomized estimators, but not optimism. For the case of finite classes, their guarantees scale as roughly
RegoM S dec, ddc,,((M T + log({M)/8).
where dec, PH (MM) is the DEC for randomized estimators defined in Section 3. As discussed in Section this regret bound cannot improve upon the guarantees for the original E2D method in Foster et al. [12] beyond constants, as we have sup dec (M, .) I dec,," In addition, since the algorithm does not incorporate optimism, cannot be directly applied to model-free reinforcement learning settings.
Foster et al. [15] give upper and lower bounds on optimal regret based variant of the DEC called the constrained Docision-Estimattin Coefficient. These results tighten the original regret bounds in Foster al. [12], but the upper bounds still scale with Est (T, 6) log(IM)/5), rendering them unsuitable for model-free RL. Nonetheless it would be interesting explore whether the techniques this work can be combined with optimistic estimation.
B Additional Results
B.1 Optimistic Estimation-to-Deriise with Batching
Algorithm 3 Optimistic Estimatiionto-Decisiioo (E2D.Opt) with Batching
parameters:
Online estimation oracle Algest with batch size
Exploration parameter 7
Divergence D(- || with sufficient statistic space W.
2: Let : T/n.
3: for k 1,2, K do
4: Receive randomized estimator 1²* E A(V) = Algést (B))
5: Get + arg min (EA() SuPMEM Encp E Gimutt -f f
-.. (5 (1 M) /1 Eq. (6).
6: Sample batch 8 = { pri o o where thi ~ ~pp and (rti ~ M (m and update estimation oracle with B*
For our application to reinforcement learning, it will be useful to generalize E2D. E2D.Opt to accomodate estimation algorithms that draw batches of multiple samples from each distribution p'. Given batch
13

---

size n, we break the T rounds of the decision making protocol into K := T/n contiguous epochs (or, "iterations"). Within each epoch, the learner's distribution p² unchanged (we index by rather than to reflect this), and we create a batch B* {(x* phi, )f by sampling n²d ~
independently and observing (piis ,0t11) ~ M* (TT*.1) for each E [n]. We consider estimation oracles
of the form μ² AlgÈst (IB*) ) and measure estimation error via
OptEst
Σ Baghupt R By Dr 1T(MM* M (+) + + (TT.) ( (m
(13)
We assume that the estimation oracle ensures that with probability at least 6, OptEst, OptEst, (K, 5), where OptEst, 2 (K, n. 6) is a known upper bound.
Algorithm 3 is variant of E2D. that incorporates batching. The algorithm updates the distribution p² in the same fashion as its non-batched counterpart, but does so only at the beginning of each epoch. The main guarantee for this method follows.
Theorem B.1. Let T E N be given, and let n be the batch size. For any 8 > O, Algorithm 3 ensures that with probability at least
RegoM S O-dec.. (M) + yn OptEst OptEst,((T/))) (T/n, n. n)
See Appendix D.1 for the proof. When working with divergences for which unbiased estimates are unavailable, this approach lead to stronger guarantees than Theorem 2.1. We refer to the proof of Proposition 2.1 for a concrete example.
B.2 Model-Free RL; Tighter Guarantees under Bellman Completeness
In this section, we show how to derive tighter estimation guarantees (and consequently tighter regret bounds) when Q satisfies a Bellman completeness assumption [30, 19]. For given MDP M, let [M](s a) := TM [ra + max.' [1hh+1, layer h.
8h. = 8, an = a) denote the Bellman operator for
Assumption B.1 (Completeness), We assume that Q Q1 for all h and Q++1 E @h+1, [Th" Qu+1] E 2h.
2H is a product class, and that
As before, we take v = Q, (MM := QMi*, and (TT)) = Exj-d, (N())) For the divergence D, we now appeal to squared Bellman error (e.g., [19, 28)):
H Σ M
Dibe (Q M)
[Qn an) - [Tr.
(14)
We abbreviate OptEst the = OptEst Pae and o-decthe(M. p)) = o-deo. Pu (M, pb).
Estimation. Algorithm 4 performs optimistic online estimation with squared Bellman error. The algorithm is an adaptation of a two-timescale exponential weights strategy originally introduced by Agarwal and Zhang [2] within optimistic posterior sampling algorithm referred to TS3. We show that this technique leads to a self-contained online estimation guarantee outside the context of the TS3 algorithm.
Proposition B.1 (Estimation for square Bellman error). Assume that Q satisfies completeness relative to M Moreover assume Σnn sup,,a rn(8, a) < 1 and supQ, h, Qx(8, a) < 1. Then for any > 1 and n < (log[]
+ +1) with batch size = H (K == T/m), X 1/8, (12 -1 and 8 > O, Algorithm 4 ensures that with probability least 1 5,
log| +
" 7l06(12)K/0KK +
K
OptEst S
(15)
72
whenever Q satisfies completeness relative M*?
Note that Proposition B.I does not make use of the bilinear class assumption, and only requires that Q satisfies completeness. As such, we expect that this result will find use more broadly.
Agarwal and Zhang [2] give tighter estimation error bound of roughly OptEst & ( + + but this result takes advantage of a Bellman rank assumption on the underlying MDP. The estimation error bound we state here does not require any structural assumptions on the MDP under consideration, but gives a worse rate.
14

---

Algorithm 4 Two-Timescale Exponential Weights for Bellman Complete Value Function Classes
1: Initialize S" 2.
2: for k I..... K do
3: For any Q, Q' E Q and h [H], define
QQ', Q) == Q'(hh"" "h aahi")) - r/ Q(+++11))
1 k-1
(²())) == 90 QQ'
S* 1) x exp
Σ
H
h=1
H
H A
2)2 Q)² ]
(2(Q)= == 1
Σ
(Q, + I log Eqr
exp
-
H
H
1=1
H
(Q) = p² (Q]5a-1) exp
-17 Σ
L'(Q)
-
1
Σ
max Q a)
H hm
4: Predict n*h
5:
for = 1, 1,.... H do
6:
Play ahd pd and obtain the trajectory of = (5f aa .r .r1"),.._...))))).. ...... (5 a rh'), where p¹ E ((1)) is decision distribution produced by any batched algorithm (e.g., Algorithm 3)
that selects decision adaptively based on μ².
Update +- St SUU af phd a ( JUU
7:
Regret bound for bilinear classes. To provide regret bounds, we assume that M satisfies the bilinear class property relative to Q as in Section 2.3. In addition to assuming that M is bilinear, make the following restrictions: (1) क
TQ. i.e. estimation is policy, (2) gsst (Q: =x)
Qu (Sh; a rh max.' Qh+1($4+1, a'), so that BM [ct." (Q; 2n)] is the average Bellman error for Q under M. 8 With this discrepancy function, Jensen's 's inequality implies that o-dec (MM) S
o-dec.. (M), so combining Theorem B.1, Proposition B.1, and Proposition 2.2, we obtain the following result.
Corollary B.1 (Regret bound under completeness). Let Q be given. Assume that M E M, where M is bilinear relative to Q, and that completeness holds. Moreover assume Σx= sups,a T' (s, a) S 1 and supq.h.s.a Qu(s, a) < 1. Abbreviate d = dai(Q: M). For appropriate choice of and % Algorithm 3, using Algorithm 4 (with appropriate parameter choice) as an oracle, enjoys the following guarantees with probability at least 1 ō:
RegDM S Hd /3 (log( (/3/33
(16)
This improves upon the 73/4-type rate in Corollary 2.1.
B.3 Proof's from Appendix B.2
Proposition B.1 is application of more general results given in Appendix C.3, which analyze generalization of Algorithm 4 for a more general online learning setting. To Proposition B.1, we simply apply these results to the reinforcement learning framework.
Proof of Proposition B.1. Let the batch size n H be fixed, and let K := T/n n be the number of epochs. Recall that for each step E [K]. the estimation oracle given a batch of examples B
takes the form oth M==1 where the p² and (r o (T"'.. Each observation (trajectory) k. k.l
((Tth) p.t. )
(st k. ",.""),..,, OH r.)l). We abbreviate Q* = -***
Estimation algorithm. For each step k. the randomized estimator μ²* selected as described in Algorithm 4. This algorithm is an instantiation of Algorithm 5 the general online learning setting described in Appendix C.3, with G = Q and for all h [H], XA = S x A, V/ = IR x S and
'These restrictions correspond to restricting attention to Q-type Bellman rank, special case of the bilinear class property [19, 12).
15

---

W = S" The unknown kernels are the transition distributions for the corresponding layers of the MDP M*, and the loss functions are
[n an), Q) = (s an);
(T,((T,8h++)) == Th + max Qn(s++1,a),
"
Eslfsihe == H max Q2
Finally, take 2 = (s a "), yi = (r% s and w = {s' heem It important to note that sh"" aah -h" ,8441 taken from different trajectories for h E [H] so yi. zj, independent from one other for h HH). Moreover, note that the distributions p" AIII) play the role of nature: the tuple (rf, yh w*) for h E [H] is determined by running a policy xt ~ p* in
the distribution the ground-truth MDP M With this configuration, observe that in the notation of Appendix C.3, have, for any Q
H1
Ext EQQ, Q, xi.n)²
E x (2 Q) - E(l,,2(uh +)
H
h=1
E m (sh [Th" Qn+i](sh, (sti)(s),an))
E h wut
h=1
E Braupt EMCA,
(Qu((h, - ( Qu+zl(sh
B Bumpt Dihe (Q M*).
and
H 1
Byg oo v' (Q)
Σ
A EN' kk max (i(s)1,a) max Qi(s" a)]
1=1
fM (Tar.) (xQ))
Estimation error bound. We take 12,3, so that Theorem C.1 implies that with probability at least 1 6,
Eq ( Eampse Dibe (Q M*) + aas" (TT)) - 1 (TTO)
S na lollog(((l (og()/)// + [og](//1 10//++0/ + a²K.
Then by taking a = - this further implies that with probability at least 1 - 5,
K
OptEst
Σ E Eange E D% (Q M*) + (p"" (( (m())))
S (no ( - log| 2 n + a²K)
H log Ω
n pog QK//8) K
n
Proof of Corollary B.1. We choose n = H and apply Algorithm 4 as the estimation ora- cle. We first consider the "trivial" parameter regime in which ( > 1/(226 (log( K/8) + 1)). Here, T S (/))) and thus
Regpm & Ha2 (log( Q 4/53)3
16

---

When the case above, does not hold, we proceed as in the theorem statement, choosing = Hd (log() ( - I 1/(216 (log() Combining Theorem B.1 and Propo-
sition B.1 then gives
RegDM X o-dec (M) . H log O
K
/
72
with probability at least 1 8. Next, using Proposition 2.2 bound o-dec,, (M) in the above display, it follows that
HdT
H² log Q
K
RegDM S
+1
+ y log( ( +
72"
We choose 7 = (log( ( 2 2/5/1//3 to obtain
RegDM S Hd (log() 2
with probability at least 1-
C
Technical Tools
C.1 Preliminaries
Lemma C.I. For all x E 10, we have
11 s 51-1/2 x/2, and ex s (e 1)x S 1+2x.
Lemma C.2. For all 2 -1/8, we have e 51-1++2 Proof of Lemma C.2. Let
= f(x)=
1+x 122 We have = e e 20 -1/8.
Thus, f' is monotonically decreasing on 2 -1/8, for x 1 f( 2 (0(
Hence, f()) is non-decreasing on x E [-1/8, 0J. Furthermore, -
+1 0
0. Thus fx) obtains maximum value æ and f(x) S f(0) =0
C.2 Basic Online Learning Results
In this section we state a technical lemma regarding the performance of the exponential weights algorithm for online learning. Let G be abstract of hypotheses. We consider the following online learning process.
For f = 1...., T:
Learner predicts (random) hypothesis g G.
Nature reveals E C (G - IR) and learner suffers loss E (g').
We define regret to the class G via
T Σ
T inf gEG
Regol
Bytnyt [f'))')
(g),
(17)
where je' E A(g) is the learner's randomization distribution for step t.
Lemma C.3. Consider the exponential weights update method with learning rate 1) 0, which sets
4 (() x exp
- -n (e(g))
ist
For any sequence of non-negative loss functions C "....,T,, this algorithm satisfies
" T 2
log|g
Regol
Eyg [ (g))))
(18)
1=1
17

---

In addition, for any sequence of loss functions E. ...... 1 S (2L) then
......"" E" with (0) E [-L, for all E G. if
T
logig S 4n
T E Eginn' [g))
log
Regol S 2n By [("")')) - Beg-na ( (1'))))
Σ
1=1
(19)
Proof of Lemma C.3.
A standard telescoping argument combined with the fact that Ei= (()) (e.g.. Assa-Bianchi and Lugosi [4])
-infges El=r (g) s
log (exp (Egeg -
gives that for any choice >> and any sequence of loss functions, exponential weights has
T
T log
log (20)
Regol
Σ
E g
1e (0)
+
Σ (() exp(-nfe Ex(g))ex((-p""(() gEG
7
t=
T
log/g
log
Σ (g) exp(-1((f())
Ep (C +
(21)
8E9
We first prove (18). Using that log(z) S for x 2 0 and exp( (x)) S 1 - x + have
2 for x 2 0, we
1 77 Bgr (e) + 7/² E E (())
log
Σ
(r(()) exp(eepppp)) (g)) exp(=n(())
2
DEE
so that
11 Regol
Σ
T
E Eguut [ (g)) log|g
2
To prove (19), we use that log(s) S x for 2 and exp(-x) S x 2x whenever |æ| S to get
log
Σ M exp(-n(e'(g) 9E9
Ey' (('')))) < Egnp' (e0()) - Eg [e (((())))
so that
log/91
Regol
I 2n
Σ 1=1
Bg [()(g)) Eg' ( +
"
C33 Online Learning with Completeness
In this section, give guarantees for online learning algorithm Algorithm 5, which generalizes the two-timescale exponential weights algorithm (Algorithm 4) of Agarwal and Zhang [2]. We describe and analyze the algorithm in general online learning framework, which abstracts away the core problem solved by Algorithm 4: value function estimation using a Bellman complete value function class.
Let be an abstract set of hypotheses. We consider and online learning process parameterized by positive integer H and E 1].
There are 2H outcome spaces, (Nu)netH) (Dhheet and W.
There are H unknown probability kernels {AC, X Phheee
There are 2H + 1 known loss functions (6A.1 Xr x g + [0, I]]eet (la,2 3h. x G [0, 1)]Meи. and 13 W x G + [0,1] 1].
Define 8
Ø. We consider the following process. For t 1,.., T:
18

---

Learner predicts (randomized) hypothesis f E G.
Nature reveals (#i, yi HE[H and w'. The outcomes (#j M can be chosen adaptively, but they are mutually independent given S 'The outcome w can be chosen adaptively based on 5 and (e]H) Each outcome v (x())) is drawn independently for each h [H].
The history updated via S t 5t 50 V Meem Ufw"}.
The learner suffers loss
E (ena ft) Bl0n,(2n 2t1)² + als(w', f*).
(22)
H
The learner's goal is to minimize form of regret for the cumulative loss given in (22). This loss function reflects two objective. The first objective involves the losses (e(m) with corresponding outcomes {ara hell). as well the H losses (f2,n) tied outcomes (yn heta which are generated stochastically based (rMM The primary objective is to minimize the primary error
H ΣΣ 1=1k=
(1.. - (len,2(yh. x
The secondary objective minimize Σ balw",$) and the ultimate goal is to minimize weighted sum of the two objectives.
This online learning setup, adapted from Agarwal and Zhang (2), generalizes the reinforcement learning setting in which Algorithm 4 operates. The adaptively chosen outcome zj. corresponds to the state-action pair the h-th step, (s). with the policy at time chosen in adaptive, potentially adversarial fashion. The conditionally stochastic outcome yf, corresponds to the reward and the next state (ri, +)), which is sampled independently from the MDP's reward distribution and transition distribution at step h, and is conditionally independent given (sj, aj,). The learner's objective the RI. framework is to predict a value function I = Q' that minimizes the squared Bellman error, realized by selecting the losses ( f') Qi (s), af,) and l2, (V). f*) r) max, 2h+1 (8) a). The secondary objective to predict the value function optimistically, with fg(w", f*) = max, Q(s;, a) and w' =
Algorithm 5 Two- Tioo-imescale Exponential Weights (adapted from Agarwal and Zhang (2])
1: Initialize Sn <- Ø.
2: for 1, 2, ...,, T do
3: For all f.g E G. define
Aj = (n.1(22.9)
(n,2(1h.
H
(( = (''0), x exp
-
#=1 Σ
A
H
h=1
1 H
Ai 5) ))
I'(f) H
Σ h=1
1) log Eyno((-11)
exp
1
Σ
H
h=1
1-1
x' p())==p([(( 1) x exp -
-1 (Alalw", ,f) + ff) 1)++)(0))
4: Sample and predict f p'
5: Observe {2), yi h wt. and update S, + S Uti
Ha Wi Uww".
To analyze Algorithm 5, we make a generalized realizability assumption and generalized complete- ness assumption; these assumptions abstract away the notions of realizability and completeness in RL.
19

---

Assumption C.1 (Realizability for online learning). There exists f G such that for all h E [H] and 2n Nn. we have
C1 (2 f²) (1n, f²) xnl.
Assumption C.2 (Completeness for online learning). For any G, there exists g such that for all h [H1] and 2h E Nh, we have
En,1 (a 9) = 1lln,2(Uhh 5) an)
For any f E G. we denote the corresponding E & satisfying this property by g = Tf. For functions g and and outcome Th. define
(g. f, I.) (xn. g) 1 f) mn);
this quantity generalizes the notion of Bellman enror for reinforcement learning. Recalling that (f.g) = ( (9 g). it follows immediately
() = ( - En2 f)) | xh1 = E,
In addition, let define
1 H
Elg. f. f.rnH) = Σ
(E
H
h=1
and
((f)== = := ls(w", ff).
The following result is the main theorem concerning the performance of Algorithm
Theorem C.1. Let X 1/8, n ((1"" 1)), and 0 < < 1. Under Assumption C.I and Assumption C.2, for any E (0, 1), with probability at least 1 1-6 8,
h
E. E [(0,, f) Ell,,(un. f) x11)² + 12 fi] zh
H1 (l., ft) -((e,,0n. +1,'² + 123lz( f*)
E-- [ jmps [8 fi ([)] 41.H]
< 2 (nB log(
C.3.1 Proof of Theorem C.1
For our analysis, it will be useful to consider the following offset version of the loss:
(g /) (g f)2 (BlC,,2(M,, f) ++/] (1,2(1h, ?
We 11.H shorthand for { Jnejen and further define
fg = и h=1
Σ fg.
8'()) == Ey (-15) 8' (1
z'()) == - log Eygg
f)),
Z:
log Es-p! exp (-n [Bi(f) 8' (f..f) -Z)(()))... -()))
17
20

---

Recall from Algorithm that
-0000=0(011.5- exp
- (i
1-1
)()= x exp
-1 Σ
(Bla(w,, [ +)+++(1)))
Thus, we can verify the following relationships:
exp
- Σ 89
d' 1-=
==1
1-1
Σ exp
-X
Σ 8 8''
M'Eg
11 Σ [B) (f.f) (F.) z
cxp
p(( =
(23)
Σ exp
-11 -"> (-(1)))+000))))33-
Meg
160 ( [1) = (g(g f) - 1000
1' (5) p' (f) - -1)0010)++))000 (111) (())))-)))))
In what follows, use E--11-] to abbreviate B[- 8"-"), where F == o(5
Proof of Theorem C.1. Under Assumption C.I. (2),, E[ln,2(ynn fr) x;,)² = 0. Thus, the first equality holds by definition
R BB (Cn,,(1hh:) ElG xh1)²²² + 123la(w', sil
T
H
{ =fl1)² + 123la(1', 150
H
hml
Σ E,- [ E-p
-12(((1) -
To prove the result, appeal to two technical lemmas, Lemma C.4 and Lemma C.5, stated below. Plugging the bound from Lemma C.5 into Lemma C.4, we have
T
Σ 1=1
E,
[ Smpt E(S, (,
æi.H + 63
Σ
E. [ (f) si.ul
t=l
1
64
Er B
wi.u] (48n)² 8² 8n8) log((S///00 + +S + 16 log|g]
128
+ log | + 1833T.
Rearranging, we obtain
Σ E. ( 12) sinn] 22 (10( log(()))) 3T).
1=1
We now state the technical lemmas. Lemma C.4 and Lemma C.5, used in the proof above
21

---

Lemma C.4. Under Assumption C.I, any 0 0 O n 1/24 and 0 < 8 < 1, we have
Bx-1 BByap E(f,f
[i:H|
+ 63
Ex - [R smpt (() æi.u
T
< 64
E. Sup (f) din + 18 + log G
Lemma C.5. Let X = 1/8 and rj 1/ 1/(216 (lo(((9))))))))) + 1)). Under Assumption C.2, for any
0 6 < L, with probability at least 1 6, we have
T
Σ [B (f) zi.s] S
E--1
E E(f, x1.H) )² x1.n]
128
t=1
+ (48r) 8n log( /97(69
+ +8 +66ogg1
The remainder of the proof is organized as follows.
Appendix C.3.2 presents basic technical lemmas.
Appendix C.3.3 presents the proof of Lemma C.4.
Appendix C.3.4 presents the proof of Lemma C.5.
Appendices C.3.6 and C.3.7 contain additional technical lemmas used the proof of Lemma C.5.
C.3.2 Basic properties
In this section, we present basic technical results that will be used within the proof of Lemma C.5 and Lemma C.4.
In this section, we will present some basic properties of (g, f), Elg, fi )2 and Z'(f) for any t E 9. f E G, wi E Unelan Xn. These properties are mainly due uub-Gaussian term in the definition of 8' (g. f). Recall
1 Σ
fg.
(l.1(1hh))) 11,2(1h f)) (E(en.(um. f) æh1 en. f))
"
Σ (1%h,
-Ele,. ff) 22)))²
H
1=1
+ ((,
f) 11/1 - ln (I ElCn,2(un, f) af)l)
H
Egg, fiiin) + 2(E(4A f) xh] - l,,2 En f. f,
1
Σ
H
h=1
Note that the quantity 2 f) 2 2,, f). conditioned on 2/. is a mean random
variable bounded by 1). Thus, we have
(g si.s] R[E(g, f |
(24)
Furthermore, since (vn, =x)l - 2 f) is sub-Gaussian with variance proxy 1/2 by Hoeffding's 's inequality, we have the following three lemmas.
Lemma C.6. For any S T. 9 E G and E R. have
Eleex(-o6) (9) x1.H] E[(ex(-c((1 -2) si.H]-
In addition, for any 0 < < 1, we have Elexp(-c6 (g. f)) zi.ul <
22

---

Proof of Lemma C.6. By the Elexp(-c8") (g. f)) zi.1l
of El f) x/1= (1,2(Vi f), we have
2c 2 (2h f) 21] (o |
=B/exp(-cE(g. f. = wi:n! E exp
s E exp((=o(11 20/ (1 si.1l
< E exxp(( (-xp(-c(111) J. ) 41.8]-
Lemma C.7. For any t s T. f and zis we have [[8 f)² ai.0l 5 5 ( 8 ri.h]
Proof of Lemma C.7. The result follows by writing
E 88 (g. f)2 si.
2
E(g, xim)) + Σ 2(B ah] (1.2
I 2
4
E(g, f. 21.H)
(Elen.2 2// ln,e(1h f)) f.
1
"
S f.
Σ
((B ((E(Cn.20n af f))x f(,h))
ai
H
h=1
S <5E(E fi <i.u].
where the first equality is by defintion, the second equality is by expanding the terms and notice that the cross terms have zero mean, the first inequality is obtained using the fact that |&, (g, f, (.4%) ] S and that the cross terms of the expansion of the second term have zero mean and the final inequality by 2(),,2(yn.))) [) xf (,.(1 f)] S
Lemma C.8. For all < T. as long as (-1.st-) [E(g, f, xi. ((
1/8, we have that for all f.
4 4E,- (z)) wi. 2 E Eg
) 2 0
almost surely. particular, E,, ((2 wi.H] 2
Proof of Lemma C.8. Recall (J( - log E (-1)) exp( 1:ppp-68' X5 (g. f)), thus have
- -X',-1Z'(f) +i.nl = E,- 1 Bg (-10) exp(-A6'(g, si.u|
< log E [g (1 (g, dí.nl
(Jensen)
< log E. FB / exp exp(-xpppppp (-1(1- 2A)E f. f.a.n))) xin)]
(Lemma C.6)
1 < log E, Eg 11 11 111 A(1 - 24)E(9, f. ..2)111))
æi.H
(Lemma C.1)
X < E. Eg S. minu = (11)) E(g, J, x
Recalling that Z'()) log E. Eg exp(-X5' X0' f)) and ('(f) Ey (. 8'(g f), obtain the following relationship via Jensen's 's inequality and second order expansion of the exponential function.
Lemma C.9. For any S T and realization of S' we have that as long as X 1/88 Z'() S ( and (f) < 9 Eg (g. for all g, f e G.
23

---

Proof of Lemma C.9. By Jensen's 's inequality,
1 () - log gg (-I)) exp( (g, f]) (- (1, 81)..
On the other hand, applying Lemma C.2, we have
()) = log l8 (-1)) exp( -xp f))
X2 (g. 1)
Thus
[ S (8 A (fn (g,
E (0
Another important fact we will use is that f satisfies , = 0. Furthermore, for any f E G, the pair T f, f always has 8' (T 0 for all f [T].
Lemma C.10. For any S T, under Assumption C.I we have d' d'( = 0. In addition, if Assumption C.2 is satisfied, then for any f. 8 (T(,, O.
Proof of Lemma C.10. By the definition f*
1
8' ft) = H
(lna(o (*) 2n.21hh 8²))² (E(e,,(yn. 1*) xij (n.2(th:-')) O.
h=1
Likewise, by the definition of f.
H 8(Tf.f) (1hh Tf) en.2(th. (B[C,,2(8n, f) 11al (n,2(1h. O.
1
Σ
C.3.3 Proof of Lemma C.4
In this section, we prove Lemma C.4, which gives a guarantee for the outer exponential weights update used within Algorithm 5.
Proof of Lemma C.4. The definition of the exponential weights update (in particular, (23)) implies that
17 -7-min min
Σ 3e(f)+ (8f.) +00)+
log |9|
log(E)~pt + Z((I)
T
log(E exp(-318(0()) + + Σ exp(-333epp((33pp (f, (f))))
+
log yupp" exp (3n23
t=1
where the final inequality holds due the fact that IX Z < E|3 for positive random variables X, Z (which in turn can be shown via a repeated application of Hölder's inequality). We further have
E,- (log(lE)
exp(-3110" (f. f))) vis0l
log( (T, [ ympt exp(- exp(-3318* (r.)) si.hl)
(Jensen)
log (E, Erp'' exp(-n(11 61) [ff, xin)))) aiml)
(Lemma C.6)
log [E,
E | (1 1711 - 6n) f, ff aim)))
(Lemma C.I)
E, Exp E(f. f( 2i.H]
2
24

---

Meanwhile, using Lemma C.2, we get
log(Br-1(Ey-pp ( æj.rl) S Bc-1 (Bspp (- + 917² (((())))))
æi.n]
Moreover, under Assumption C.1, the benchmark term is negative by
T -' - z2( + B((())
min Rx-1
T Σ
$ B
[t/l
Σ ==1
Ei- 1 (Z'()) 2i.u]]
(Lemma C.10 and the definition of L!
< 0.
(Lemma C.8)
Furthermore, the log-exponential term us constant- by its first order expansion:
exp(3nZ((f)) ai.ul)
T
log O exp(33) æi.s))
(Lemma C.9)
log( [E [ exp g (9. f)) xi.))
(=1
log [E,- E jmpt gg 5) exp(3no' (g. f)) æi.h))
(by Jensen)
(=1
Σ
f, log ( Ex,1 E jmpt Egngth exp(31(1 + f. f.r..))))))
(Lemma C.6)
E E [p E (1n 6n (n [i.]]]
(Lemma C.1)
T
<
32n
E Exmp' Z'(f) sin)]
(Lemma C.8)
By using the fact that [c(()) < 1 and combining the above displays, we obtain that
-log | <
E,-1
E jmp''
E(f, f, x1 æi=)]
T
32n
E,- E pmpp Z' (f) si.ul
[=l
3n 32
E,- [ uut 3n (f)]].
Rearranging yields the desired statement.
C.3.4 Proof of Lemma C.5
We state three technical lemmas, Lemmas C.11 to C.13, which are proven in subsequent subsections, then prove Lemma C.5 as a consequence.
25

---

Lemma C.11. Almost surely with respect to the draw of S", we have
B [)apt((18t-)) (f)
(
1 A
Ec-1 Epi
log
q f)
Term 1
E 1
Epp
[(e
1
log[9].
Term II
Lemma C.12. Under Assumption C.2, any 0 < < 1, with probability at least 5 over the draw of S we have for all S.
E, E Esaps (e
1) log 1
xi:H
f'(TfI.)
I 32n Ex- E (Z + &(f.J. fi..)...)))) Tog([G1T/6) si.h]
+ (24n) + 4n3) log( (5))
Lemma C.13. For any 1 S the following bound holds almost surely with respect to the draw of S::
B R Bynut
(e (0-1000000000000120000 (1400)000000)))
2 (1) | $ 17 1, B [15E xi.1)² + 280Z [ + 4n3².
Proof of Lemma C.5. By Lemma C.11. we have
T Σ Brupt Zf) xinn]= s x (Term - (Term II) + log/g/.
Thus under Assumption C.2, using the bound in Lemma C.12 for Term and using the bound in Lemma C.13 for Term II, we have with probability at least - 6,
T
E. suup' Z)()) | æi.el
T
1
32n E,- E /~pt (Z + E(f.f. log((//T8)) =
1=1
(2 4n log (
T
1 Σ
E, E [15]8(...+ +280Z æish]
4m
2
T
x log|g).
Reorganizing the terms, we have
(1- 32n
1-
log( (19 - 2807)
Σ
E,
E Soupl Z'(f) I [
1=1
[32
<
log log( (1917/8) + 15n)
Σ
B Egpg 8(S,f [1.H]
1=1
+ (24n² 3² + 4n3) log( ( + 2 T + log|g].
26

---

Using the assumed bound on and À gives
1
E. [10 gmpp Z'( fi.sl S
Bi- | [8(f fi [i
28
+ (24n² 3² + 4n log Log( 4n + log|g].
C.3.5 Proof of Lemma C.11 (Error Decomposition)
Proof of Lemma C.11. Recalling the relationships q" (0 f) = 96 f) e (60( and
(f) pf()) (1
the proof begins with the following manipulation:
1 (115++((155) log (TT(S r)
1 1/
log
g'( IT)
((TSI
E Egr+.
log
+ Epp++ -pt
log
pe
('TS(
q't2(T
(T
+ Exp.
log g (T
1 ( 1(000)+5161----00--- -1) log ((TST))
Egg
Ep. (( (0/3)/0)000000-22000000 -1) - (6TTT1)-)
A - (f))
-
Thus, taking expectation with respect to [.), dividing by X, rearranging, summing over on both sides, and taking advatnage of telescoping, we obtain
E
E (-1S--)) Zz( [i.H
Eyt [6-101000+40000-200--000--100111 -1) log q'(Tf ff] 1
1
R4
1=
ET-1 Epp (=0001100(0000000-00000011111) -1) (8((TI.)- 2'(0)]
wi.H
+
Σ E,- EE,, ( [T(. sf.ul
t=1
1 4T (TS (TS1
+ Ā
Eg Eypp' log " (TS
1ii ET IE s log
x T+1
Finally, we use that 8' (Tf. log
( 0, and log 1 log|g/.
C.3.6 Proof of Lemma C.12 (Bound on Term 1)
Toward proving Lemma C.12, we state and prove a series of technical lemmas, Lemmas C.14 to C.17.
Lemma C.14. Under Assumption C.2, for any 0 < 8 < 1, the event A defined below holds simultaneously for all t S T with probability least 5 over the draw of ST:
A'
sup log qr+ (TfIf) < 22log((9|T/0)
seg
27

---

-1 Σ ()(1)))
exp
Proof Lemma C.14. Recalling that (g
we have
exp
-A
)))
log
Est
sup
log
Est
sup
exp
- Σ
8 (g. 1)
(Lemma C.
f qi'n (
<
log
Egt
Σ
exp
-
8 (g.
f.
log
Σ
Es
exp
-
8' (g.
1.9
$ 2 log [|
(Lemma C.6)
where the final inequality uses 0 < À < 1/2 as well as the fact that for any fixed choice of f,9. conditioned æi. S the random variables 8 (g, (for s I independent. Then, by Markov's inequality and the union bound, we have the desired result.
Lemma C.15. For any T, & A 1/8, f. almost surely with respect to the draw of S and Fi. we have
(f = s 40 4:-40-Ep (00)--1ZZ'(f) zi.nl.
Proof of Lemma C.15. Using Lemma C.9, we get
B,-1
$ 2 E,
(Bn-atin)88001 f)l)² zi.n
< 2 E,-1
E Eyngre (8. [) witH
(by Jensen)
10 E, (11) E fin),)? [
(Lemma C.7)
40 E, (2) xi.u]-
(Lemma C.8)
Lemma C.16. For any t T, almost surely with respect the draw of S' and zi.H. we have f. 80l rinnl + 3,3.
E,
1
2i.1
< E, 1
Proof of Lemma C.16. Using Jensen's inequality, we have
z = log E, E (xpp Z(DD||
< Ex [| + (Z(
Thus by Lemma C.7 and Lemma C.15, have
E,- 1 wf.
< 3 E,-1 Expp
+12((0)
|
< E2 E Esap (1E(f (/)] | +38
Lemma C.17. For any 0 / 1/240, f. almost surely with respect the draw of St and xi
have
E,
< < + 44 (1 E("", CJ"" " xi.H) sin
+ 12n 2n.3
28

---

Proof of Lemma C.17. Jensen's inequality gives that
Z' < Pp (fr +
()) R (-11) 18 (g. []] (1)
We may then write
E
B
i Egpeps e (6 (f) + +e e 6n.' (f.f) + (() + (n)))t())) + 6ng' is'''" .5' 6n22
E,-1
- zi.1
6
where the second equality uses Jensen's inequality pull the y outside of the exponential and then the fact that a₁ ... dg S / Σ af for real numbers al...., ag. We control the six terms on the right band side separately follows:
Term (a): Using Lemma C.2 we get
(1)))(() wi.H S 1 + 6-6 + 36,22
Term (b): Using Lemma C.6 we get
(1p xi.n] S 1. e
Ex-1 E
Term (c):
Ex Lon_eenn (()'1()) (1)
2i.1
E,-1 (en By (g. rims]
< E, gg (.10) (xp (g. f)) si.]]
s E, gg (1 (6n(11 + E2 f. aaa))))
(Lemma C.6)
$ 16n Eg (-15) E(g, J, xi.H)² wish
(Lemma C.1)
< 64n E, (Z (f) sisul.
(Lemma C.8)
Term (d): Using Lemma C.2, have
E
E Eyaep Snpe"(()) win 1 + 6r/3 + 36.
Term (e):
E,-1 E Ep e (F')((')))) Di
< B Exp exp((x)))))) E2 (f f, ainu
(Lemma C.6)
+ 16n R, Ep E(8', x(. [ (Lemma C.1)
Term (f):
E.-
E ppp e =n')((")
xi:11
E, Bpinpt 1 - 36 (Z |ii
(Lemma C.2)
1 + (-6n + 1440m7)
En E Z'()) sisH
(Lemma C.15)
$ 1.
(Lemma C.8)
29

---

Proof of Lemma C.12. Combining the preceding lemmas, we have with probability at least 1 - over the draw of S", the event A' defined in Lemma C.14 holds for all v T simultaneously. and thus
E Bypps (6708(00+-(00--220)-111-1-- -1) log ('(
1
| |
1
s Eypst
(167/E-1(('(1))) By E(f', x )² di.H| + 12n2 2 2nB3 log q'(T)
16n By-
[ supp E x 1: )²) ((
log log((19 ri
+ 324m) + log (
where the first inequality is derived from Lemma C.17, and the second inequality follows from Lemma C.8 and Lemma C.14.
C.3.7 Proof of Lemma C.13 (Bound on Term IT)
Proof of Lemma C.13. We compute
(e (0-
E, Es-ps
| (wf00/00000000//2000000000000110 |
E, E
- 2n E, S~D' |0000001100000011220000000010110 mf.
< 1) [8 pp (F6,,)""++022((0))+1111)1111"" +332 |<i.m] +
5 1 1 [B [
where the second inequality applies Lemma C. (together with the fact that E 1/240))) the third inequality uses Young's 's inequality and the fact that | ir'()l S for f. and the final inequality applies Lemma C.15 (to bound 1 (f) Lemma 16 (to bound 2 12), and Lemma C.7 (to bound E. [6 #f.n)).
D Proofs from Section and Section
D.1 Proofs from Section 2
Proof of Theorem 2.1 and Theorem B.1. We prove Theorem B.1; Theorem 2.1 is the special case of this result which n 1. Observe that for Algorithm 3, we have
K
Regom n
Ext ( (TTr-) l (x*)].
We can rewrite this sum as
K
By k hp [frr" (Ix..) - far (TT*)]
By Bytnngh E mpr [n00 (mrppe) - (((( + E Branpe [(rm (MM))TT))) 5 (TTNO)
1=1
Expkopp Biite fri (IrTA) - frr (TT*) - y. D=* (5 MM
+ E Eerepe 7- DDDD' ( 110 + (f (mxxe)) - f:0t (RकEe)
[1O (TTEe)) - pm (Mt 7. D** (
(v (i+l (r + + OptEstt.
Ephapp E Biinnp
k=1
30

---

For each step k, by the choice of pt, we have
E Epp E Bii 1rrh (TTZE) fd (T²) - 7. D= ( || ]
< sup Enpr Biannue (1 (TTO)) ("*²) 7. D=* x5 [1
MEM
sup Bap B Binnet (MT)) - (*)() r D + M)
inf pEa(I) MEM
o-dec!, (M, S o-dec (M).
Finally, we use that probability least 1- 8, OptEst S OptEst, (K, (K,,,))
We next prove Proposition 2.1, which gives a bound on
K
H
OptEst
Σ
E tpt
E
Mr moch
Σ
MT
post (?:: + +y " (TTr)) -5
Algorithm 6 Optimistic Estimation for Bilinear Classes
1:
parameters:
Number of rounds T, Batch size
Learning rate 1) > O.
Discrepancy function ex0 =k: Exploration parameter > 0.
2: Let = T/n and 8 : Ø.
3: for k 1.2,.... K do
4: Form the randomized estimator jet via p (Q) x
(Q))), where
2
(Q)
h (Q; 2/
8y (TQ))
n2
5: Receive batch of samples B* { phi of , ) where ahi pr and (r*"),0tt))
(T
pp is the decision distribution produced by E2D.Opt (cf. Algorithm 3). F.1 "
6: Let 2/ (sh a rht 1 where recall ^ (s ai "...,,... (5
Proof of Proposition 2.1. Throughout this proof, we use the batched estimation notation from Appendix B.I. Let n be fixed, and let K : T/n be the number of epochs. Recall that for each step ho [K], the estimation oracle is given batch of examples B* = {(T fart",rt r )*)))) where tdd p² and (neid o²:1) (x**)) Each observation (trajectory) takes the form o (s" A.t bt rl)...... (s Al bi H H TH Throughout the proof, we use the notation
(Q) E ,** [ere (Q; =n)]
and abbreviate Q* A
Estimation algorithm. Define
n
())
Σ eest (Q: 2 n
where z/ : (5% o vhh 11+1). Let > 0 and > 0 be parameters whose values will be chosen the end of the proof. Defining
A.F kdd Alt
" (*(()) = Σ -
h=1
31

---

Algorithm 6 chooses
(²( x exp
-n -n> (())
ick
as the randomized estimator for epoch k.
Estimation error bound. Let abbreviate L Lui M*). Observe that for all k; (Q) S
:= = HL + a almost surely. Hence, Lemma C.3 implies that as long S 1/2R, we have
K
K
K
+ Q
B [et (Q))
(Q))
S 4n
Σ
E BQ (())
(25)
"
Am
1=1
A==1
For each k E [K], we have that for all Q e
2 Σ (((O))
H
"
(0)) 2
+ s 2H12 Σ (X((1)) + 2
h=1
As result, (25) implies that
K
(EX(O))
B
+
BQuua (( - (nqI)
(26)
K
()11)) ?
K
H
(EX101))
+ 8naa 2 + logl Ql
+ 8nHL?
E
Quapk
(27)
k=1h=1
k=1
h=1
Whenever 7 S
16/11 rearranging gives
K ΣΣ
EQ (E:(Q)) + ta Σ EQ [(FM)((T - (TT)) (
(28)
2
h=1
K H
E200))) + 8na2 K +
logl 2
(29)
1
A= A=1
We now appeal to the following lemma.
Lemma D.1. With probability at least - 6, it holds that for all l E [K], h E [H) and Q E Q
((E) (EX(1))² ecomc (n) {())()) + 2. (n),
(30)
where Econe (n) LV 21og( Q(KH/8)
Going forward, we condition the event in Lemma D.I. Applying the inequality (30) within (29), we have
K
E Equai (E)(2))) +
Σ
E Eq [fr (TT.) (Tq))
(31)
1=
logi Q
(E)(2))) + 8na2K +
+ K 2
k=
log Q
81ja² 2 K
+ +3H 2 cone (n),
(32)
where the last equality that (Q²)) 0 for all h [H], and [K]. which is consequence of Definition 2.1. Next, we recall that
Ean (EX()))
Σ Eqnp' DØ (Q Mr)
k=1k=1
32

---

A standard application of Freedman's 's inequality (c.f Lemma A.3 in Foster et al. [12]) implies that with probability at least 8.
K
K E Exppt B Dg (Q || Mr) LogH Ho)(H/6)).
E B
D (Q M*)
2
Putting everything together, we have that
E Npt Eq D (Q M²) + salf" (Tar""(TTT) (
S nnaKK K + 1og/ + HK (one(n)) (n) + HL² log(H/8).
Choosing 84
this gives
log/ Q
OptEst bi S na?K K +
H Ke K Ke (n) + HL2 log(H/6)
7
log| Ql
HL log([ (1+
S na 2 K +
n
We choose 7)
Ing O
1 which satisfies the constraints described carlier in the proof, and 1677
gives
K log Q
K
OptEst bi
$
RR.ooog + + (l 1+
?
K logl
(1 1 -
<
HL
HL
Proof of Lemma D.1. For any fixed [K] and Q E Q, Hoeffding's inequality implies that with
probability at least 1 - 8,
log(1/6)
εx(0) < LV
By a standard union bound follows that with probability at least 6, for all k E [K], h E [H]. and Q Q simultaneously.
2 2log(12)KH/5)) (n).
(Q) -()))-;;(0) L
n
Whenever this event occurs, the AM-GM inequality implics that (E)(2))² < 2(E) 2: 25 (n)
and likewise ( ((E) + (n).
Proof of Proposition 2.2. Let E A(Q) be fixed. Fix a E (0. 1). and let क be the randomized pol- icy that for each h h-independently plays &Q.A with probability 1 a/H and TQ with probability a/H. H. Let (TT) be the distribution induced by sampling Q E 11 and playing z Translated to our notation, Foster et al. [12] (cf. Proof Theorem 7.1, Eq. (152)) shows that for MDPs M, this strategy guarantees that for all 77 > 0.
Exe, Eq- (FQ fd (m) s a +
H M)
+
n
EQ.Q'~a ~ (Xn(Q; (a.
2n
h=1
dsi(2; M)
(BM.ROCNNT)
a +
+
Eq.
[eas 2n))
2m
2
h=1
33

---

In the on-policy case in which rst =Q
TQ. it suffices to set a = 0, which gives
BQ
(BM.TOPATi
(Q'
H
E (B [ext ; ]
h=1
Exsu B (BM. (Q': =
h=1
Ex Eq n D || M)],
where the last equality relabels Q' t Q. Setting = 27 yields the result.
In the general case where qQ # xQ. we have that for all Q' E Q
a E Trpp (B (1t (Q': =1)1) H (1- (1-a/H)# H-1 a/H) EQuu (BMITOUNTO"(C (Q:
We have - a(l-a/H)"11
1-1 2 2H
whenever a 1/2,, which gives
Bq ( - fM (+)) a (d .)
+ nH
E BQ ( || M)]
21
H² dail Q; M)
+
+7 Enpp E B ( M)],
27a
where the last line chooses 1) 7a/ H. To conclude, we set admissible whenever y 2 (Q;
which is
Proof Corollary 2.1. In both of the cases in the theorem statement (on-policy and off-policy). whenever 2 and n S VT (to ensure that K/n 2 1), combining Theorem B.1 and Proposition 2.1 gives
T + V nTllog RegoM S o-dech((M) T 7HL2 2 log( kH/))
n
We choose n = vT, which is admissible whenever T' is sufficiently large, and gives
RegNM E o-dect (M) T + HHL² log( (e((Q KH/0) + log Q T3/4
In the on-policy case, we have, from Proposition 2.2, that
Regom S HdT γ 10H Q 3//4
and in the off- off-policy case, have
RegoM S H²d H2d 1THHH1" /logl Q T//4
Choosing y balance yields the result.
Note that this result uses that the quantity MM ( =n)] only depends on the policy π through
34

---

D.2 Proofs from Section 3
Proof of Proposition 3.1. We begin with the upper bound. First, note that by Assumption 3.1 1 and the AM-GM inequality, we have
o-deo, (M) sup inf sup Eawp E M-n 1 (TT)) - (x) - D" (M ||
M)]]
MEA(M) PEATT) MEM
Lip
sup inf sup Exop E Mmp (TT) 2 17 [ |M + 27
WEA(M) PEA(T) MEM
2
(33)
Consider any fixed choice for μ A(M). By Sion's minimax theorem (see Foster et al. [12] for details), we have
inf sup Exu E Ei (~T) (T) ? D=T || M]
pEA(I) MEM
inf sup Eanp 18 Ma EMO PEA(IT) REA(M)
pd (TT)) - (T)(T) 2/4 1dtd (M M)] 9 (TT)) - (T)
sup inf Encp E B E EMWN
1 D (M M)). 2
REA(M) (EA(IT)
so that the main term in (33) is equal to
sup sup inf Exup Ex EMwr (T((TT)) - -5 (x) 7
D (M M]]
HEA(M) VEA(M) PEA(II)
2
$ sup inf sup Exuu ENT E EMwuy 8T (T^s) - (TT)
? D= (M 11 Mr
VEA(M) (EA(I)) μEA(M)
2
sup inf sup Expu EMS (5 (EA(M)PEA(I) NEM
ff (fx
7 D= (M D*M M)
Relabeling, this is equal
sup inf sup Exep Emay ( 2 D= D=(M || [M]
sup dec d 11). PED(M)
(EA(M) (TA( MEM
We now prove the lower bound. Using Assumption 3.1 and the AM-GM inequality once more, we have
O-dec, D (M)
sup inf sup ExE E M. (Txx) - { - 7. D" (M M)] MEA(M) PEA(IT) MEM
37 2
D= (M M)]
Lie
sup inf sup (EA(M) MEM
Exep Emap
- (s
27
sup sup inf Emp BI BMNu 8dh (TT)) - 5 2
37
D (M I|
M)
Lin
HEA(M) (E)()) PEA(I))
27
(34)
Using the minimax theorem in the same fashion as before, the main term in (34) is equal to
sup sup inf Enou ET EMaU - (T) (Tx)
3y
D. (M M)
vEA(M) (EA(M) PEA(U))
2
3y
Dt (M M)]
sup inf sup DEA(M) PEA(T)) PEA(M)
E Eaep E EMN EMW (1
2
3x
D7 (M M)
= sup inf sup Eapp EMEN 1 (
vEA(M) (TI(I) MEM
2
Relabeling, this is equal to
sup inf sup E Exp EE Mnp {d f (f 2
3y
D (M
M)
sup MEA(M)
dec,,/2(MM,H))
D
(EA(M) PEA(U)) MEM
35

---

Proof of Proposition 3.2. Let M he arbitrary. By Sion's minimax theorem (see Foster et al. [12] for
details), have dec,, M) sup inf Bx-p,M-ull [f (TTas) (TT - 7 D (M || M)]-
(EA(M) PET(IT)
By the assumed triangle inequality for D. we have that for all II,
EM (M M')] - C. Cm D (M || M)] + C. CM 1 M']
= -C' D M M)].
Applying this bound above, we have that dec (M, z
sup inf E Ea-p,Mau PEA(M) PEA(I)
- (T) -
/ (M'' || Mi]
20
sup sup inf EKOD,MMN (M (Txx) - -f (T) 20
E EM DD (M' Mn]]
vEA(M) (M)(M) PEA(TI)
sup inf sup BREN (p (TT( - -1 7
E EM DD (M' MD]
2C
rEALM) PEA(IT) MEM
sup dec (/22 VEA(M)
Proof of Proposition 3.3. The upper bound on o-dec)((M) follows from Proposition 2.2, so it remains to prove the lower bound on (Mc)
(M).
For a model M, define the Bellman operator Th" via
=
(1aa)
Th + max a(shtiaa) a'EA
(35)
Let
D (Q || M) Σ
MM [QQ((h) an) 1Th" 1h* an))²
h=1
and D (M MM D (Q M). By Jensen's inequality, it suffices lower bound
dec
Dan (M).
Let = (s. f) and A {a, b}. We consider a sub-family M' c C of deterministic combination lock MDPs parameterized by ă (ā. aH) E A", with M2 defined as follows.
The initial state is $1
5.
For each h = 1, 1...., H 1, if sn. = 5, then selecting aj = ãn transitions to $ = 5, and
selecting a # ã transitions 8h+1 = t. i self-looping terminal state: if s, = (, then
8h+1 regardless of the action taken.
If 8и = 5 and a = Tn. then ræ A > 0: all other state-action tuples have zero reward.
We choose M such that Q (s, a) = 0 for all (h, 8, a). We calculate that for all ă e A and all policies x IRRNS.
f - f(a(TT) = A Mp (a1.H # a).
D.Te (Q Maa = A pM (01.H d).
It follows that
dec Dane (M, 2 inf max Enepl A PM + α) 7 A Pp (aI. a)-
PEA(IT) dEA
> inf Ea-Uni((AM) Braple A pugg. (a1:1 + ā) 7 A (01: = a)].
TEA(I)
36

---

It is straightforward to see by induction that for all π E IRRSS, Un (P = =ā] -
2 2-# and Eg-Unit(A [PM (a1:1 # a.] 2 2
have
A
A
dec, Dao [,M M
2
2
2
The result follows by choosing A appropriately.
Proof of Proposition 3.4. Let H be given, and assume without loss of generality that S 2 + 2H-2 Let N == 2H-2 We construct a family of MDPs M = "h....,My! with deterministic rewards and transitions as follows.
{a. b).
The state space is S - TUE [s}. s is a deterministic initial state which appears only in layer 1. T represents depth H 2 binary tree (with the root at layer h 2), which states in total. E = (N + L..... 2N an auxiliary collection of self-looping terminal states, each of which can appears in layers h = ........ H. In total, we have [S] = 2 + 2
has N = 2 leaf states in layer H, labeled by (1...., N). and 2
The dynamics and rewards for MDP M, are as follows.
At layer h 1, $1 deterministically.
If a = a. we transition to $2 = N + E, and if a = b, we transition to the root state for the tree T. We receive reward.
For 2, states 8 E are self-looping and have no reward (i.e. 88+1 = 8h 8p. for h > 1).
For 2, states in T follow a standard deterministic binary tree structure (e.g., [24, 8)). Beginning from the root node at h 2, action transitions to the left successor, while action transitions the right successor, For M,, receive reward 1 for reaching the leaf node i E T at layer H, and receive zero reward for all other states. Note that the transition probabilities for this portion the MDP do not depend on i.
Online estimation. We first construct an online estimation algorithm for the class M. Recall that we adopt Hellinger distance, given by D [M, M) = D2 ((T)))((TT
We first note that since all M M have fd (TT the optimistic estimation error is equal the (non-optimistic) estimation error
Est
H
Σ 1=1
E
inpt E B (T
Observe that since all MDPs M MM have deterministic rewards and transitions, there exists function o (T) such that o ~ M(T) has o = 6² (нr) almost surely. It follows that
(M M (T)) 21(("" ( (m) # o" (T))
for all M,M' M' E M, so we have
Est. = -Σ Exting E Ep (M = Σ E Eg E Exinat 1100 (T') ( o (T7))]
For the estimation algorithm, we choose
# (M) o exp
IOO
(Tr)) # #
ist
Lemma C.3 implies that with probability 1, the sequence n: ,....,,"" satisfies
T
Σ E B (100(T²)) (x') (m') f
Σ Ho (T"
(TT
5 2
EE T Npp "T0'((') + 0'} log.M).
1=1
t=1
37

---

Since o² (5*) o', rearranging yields
E M rup [1o0 (x¹) + o') 2log|MM
From here, a standard application of Freedman's inequality [16) implies that with probability at least 1 6, E Pai Ey [T(o"* (m²) # o'}] x log(IM)/6) x log(S/5).
Lower bound for posterior sampling. Observe that for all M, E M, the unique optimal policy has TM, (s) = b at h = 1. Thus, the posterior sampling algorithm, which chooses y' p'(r)) y'( E MM] TM = x}), will never play a1 = a, and will never encounter states in E. As a result, the problem is equivalent (for this algorithm) to multi-armed bandit problem with N arms and noiseless binary rewards, which requires Regom! ž 2 the worst-case [21].
Upper bound for Algorithm 2. We first bound the Optimistic DEC for M. For any μ E A(M). we can write
o-dec" M)
inf sup En E EM [ (TT) (=) - D (M (T))
PEA(I)) MEM
inf sup Enep E Timp 1 (TT)) - f" (x) - 27 (1fo (T))
PEA(I)) MEM
inf sup Tm E Maup 1M (TTM) l (x) 2y T(o0 (T) + }
PEΔ(M) MEM
where the last equality uses that (TT)
for all M M. We choose p
(1
E) E M Txr + + ETa, where Ta is the policy that plays action deterministically.
Now, let M M be fixed. Since each model Mj transitions 82 N deterministically when a1 = a. we
ETEOy HO" + (0 (T) (H0 + (Ta)) 8 HIM + 4M}
and
E May Eamp (T) + o" (T)} 2 sm(M (M)).
Similarly, we have
(M)
= + (1 - s) (M)).
By choosing E = -- which is admissible whenever 1, we have
Enep T - (T) 27 1100((T) # 0r (T) S + H(M (M}) 2YEM((M) (M}) 1
This establishes that
o-dec (M)
(M)) v
for all 7 1. A regret bound of the form E E[RegoM] Theorem 2.1 with the estimation guarantee in the prequel and choosing appropriately.
now follows by invoking
38


Reviews:
[{"review_id": "RJ5lKRrLmw", "reviewer": "Reviewer_nAEr", "summary": "This paper adapted the DEC framework and E2D algorithm to incorporate model-free reinforcement learning. Specifically, this paper proposed the optimistic DEC and proved that it appears in the regret bound for the optimistic E2D algorithm with an optimistic estimation oracle.  The example of bilinear class is provided. The paper discussed when the optimistic E2D algorithm will have benefit than E2D in its original form. ", "questions": "What is the general picture of the lower bound side of optimistic DEC? Am I correct that oDEC is always smaller than DEC, and DEC serves as a lower bound, so it is not meaningful to discuss oDEC as a lower bound? ", "limitations": "na", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and provides concrete steps toward understanding the general framework of DEC. The original DEC framework is mostly model-based decision making, whereas the new framework can handle model-free RL. \n", "weaknesses": "This paper is largely built upon [13] and [32]. The additional results over [13] and [32] are a bit incremental. While using the DEC framework to provide a regret bound for bilinear class is nice, this does not new/improved regret bound for bilinear class. \n\nIn Appendix A, the authors discussed the relationship with constrained DEC [16], and mentioned that it would be interesting to explore whether optimistic estimation can be combined with the constrained DEC techniques. To make this paper stronger, one possible direction is to discuss the relationship with constrained DEC in details (either provide some results in this direction, or point out the technical difficulty to the combination). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive comments!\n\n> Am I correct that oDEC is always smaller than DEC, and DEC serves as a lower bound, so it is not meaningful to discuss oDEC as a lower bound?\n \nThe relation between the oDEC and the DEC depends on the choice of divergence and the model class. As shown by Corollary 3.1, for Hellinger divergence, the two are equivalent up to constants, which means that our results are always tighter than the guarantees in Foster et al. (2021). As shown by Proposition 3.3, there is a model class for which the oDEC with bilinear divergence is bounded, but the DEC with bilinear divergence can be exponential. In general though, the tightest relationship between oDEC and DEC that we are aware of is Proposition 3.1, which shows that the oDEC with divergence D is never larger than the DEC with the “flipped” version of D.\n\n> What is the general picture of the lower bound side of optimistic DEC?\n\nThe DEC defined with Hellinger divergence serves as a lower bound for expected regret. As mentioned before, when defined with respect to Hellinger divergence, the DEC and oDEC are equivalent. Hence, the oDEC with Hellinger divergence also serves as a lower bound on expected regret. Note that the regret upper bound we have obtained for bilinear classes is in terms of the oDEC with squared Bellman error, which in general is not equivalent to the (o)DEC with Hellinger divergence. Proving lower bounds based on variants of the DEC with alternative divergences D such as squared Bellman error is a fascinating question for future research, but is quite subtle, as the choice of D seems to be tied to the model class under consideration (i.e., depending on the choice of D, the oDEC may lead to tight results for some model classes but not others).\n\nLet us mention, however, that the goal of this work is not to obtain matching upper and lower bounds on regret in terms of a variant of the DEC, but rather to show that the estimation-to-decision framework for proving regret upper bounds extends to the model-free setting. The question of proving lower bounds is somewhat orthogonal to our focus, though we hope that our results can inspire future work along this direction.\n\n> This paper is largely built upon [13] and [32]. The additional results over [13] and [32] are a bit incremental. While using the DEC framework to provide a regret bound for bilinear class is nice, this does not new/improved regret bound for bilinear class.\n\nThe goal of this work is not to obtain a new algorithm for certain model classes, but rather to show for the first time that the DEC and estimation-to-decisions framework for proving regret upper bounds leads to meaningful guarantees in the model-free setting. Prior to our work, the DEC/estimation-to-decisions framework (Foster et al. 2021) offered the most general framework for RL with general function approximation. However, the upper bounds in Foster et al. (2021) are limited to model-based settings. On the other hand, complexity measures such as bilinear classes and bellman-eluder dimension apply to model-free settings, but are less general than the DEC. Our work gives a unifying perspective, and enjoys the generality of the DEC framework, yet also gives meaningful guarantees for model-free settings.\n\nA secondary advantage of the estimation-to-decisions framework is that it yields a modular algorithm, with a conceptual separation between the estimation and decision-making components. In contrast, the GOLF algorithm (for classes with bounded Bellman-Eluder dimension) is more ad-hoc in nature, employing global optimism to select a value function and its associated greedy policy, which is optimistic and consistent with all prior observations. The estimation and decision-making components of GOLF are thus tightly coupled. Our work shows that these components can be decoupled while still obtaining similar guarantees, which we believe constitutes a significant conceptual contribution.\n\nWe consider bilinear classes only as a stylized example to illustrate our techniques. Bilinear classes capture many known generalizations of tabular MDPs, including linear MDPs, linear mixture MDPs, MDPs with low Bellman rank, feature selection in low-rank MDPs, and many more. Note, however, that our techniques are not limited to this setting.\n\n"}, {"review_id": "ij8Md8pQII", "reviewer": "Reviewer_ooU7", "summary": "This paper contributes to a line of research on decision-estimation coefficients (and related algorithms). An optimistic variant of the original DEC is introduced and it is shown that this variant can lead to a new related meta-algorithm with optimism. This new structure and algorithm are shown to provide non-trivial regret bounds for model-free RL for bilinear classes (a known framework for sample efficient learning). It is also shown that the original DEC is insufficient to achieve the same. Finally it is shown that posterior sampling does not solve all the problems that E2D.Opt can solve.", "questions": "Does optimistic DEC subsume all of the original DEC results? Are there any downsides compared to the original E2D?\n\nWhat benefit does the batching offer if the regret bounds still scale as poly(K)? Why not just use one sample per batch?\n\nProp 3.4: shows that PS does not cover all the problems that E2D.Opt can solve. How about the other way?\n\nProp 3.4: What is the expectation over? Is this an average regret over MDPs in the class and if so, is posterior sampling misspecified? \n\nSuggestions:\n\n(14) does not seem to point to an equation in the main paper.\n\nLines 94-95: It may be more clear to say that “there exist algorithms/oracles” that do this.\n", "limitations": "No, but this work is theoretical. The limitations are either apparent from the assumptions or discussed adequately. A conclusion to summarize would be helpful though.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Overall this is a good paper that is presented clearly and makes an interesting contribution to a growing line of research. The application of E2D-style algorithms to model-free settings is important, as well as the ability to incorporate classes that were previously not handled.\n\nThe discussion of where advantages exist for optimistic DEC are thorough and helpful. The additional results also shed light on the problems that can be handled by both coefficients and other algorithms.", "weaknesses": "The new coefficient is used to reproduce an existing result. This leads to a natural new meta-algorithm and theorem, but since they are only shown in one specialized case, it’s unclear whether it’s useful beyond this one setting as a “meta-algorithm.” While it is apparent that one could swap out different divergences and oracles, it may be described better as a single algorithm, inspired by the original E2D.\n\nThe regret bound appears to have a slightly worse rate than what would be achievable with the original BiLin-UCB, which I believe has T^{2 / 3} if converted properly to regret.\n\nIt’s not clear how one would implement such an algorithm in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments and helpful suggestions!\n\n> Does optimistic DEC subsume all of the original DEC results? Are there any downsides compared to the original E2D?\n\nYes, the optimistic DEC—when equipped with Hellinger distance—recovers all of the main guarantees based on the DEC in Foster et al. (2021). By accommodating more general divergences, we generalize these results. The only possible downside we are currently aware of concerns computation: For some classes (e.g., linear models), there are computationally efficient algorithms for non-optimistic online estimation, but it is unclear whether *optimistic* online estimation can also be performed in a computationally efficient fashion.\n\n> What benefit does the batching offer if the regret bounds still scale as poly(K)? Why not just use one sample per batch?\n\nBatching is necessary for algorithms based on average bellman error (Prop 2.1 and Corollary 2.1) due to the following well-known technical issue regarding average bellman error: Because the quantity D_{bi} that we are interested in is the *square of an average* as opposed to an *average of a square*, we cannot take advantage of concentration across rounds/iterations, and multiple samples are needed to estimate this quantity well for each iteration (note that the guarantee in Proposition 2.1 is vacuous if there is n=1 sample per batch). This issue is precisely why well-known algorithms based on average bellman error such as OLIVE (Jiang et al. 2017) and BiLinUCB (Du et al. 2021) require multiple samples per iteration, and is one of the main reasons why these algorithms are analyzed in the PAC framework instead of regret.\n\n> Prop 3.4: shows that PS does not cover all the problems that E2D.Opt can solve. How about the other way?\n\nE2D.Opt subsumes all results we are aware of based on the frequentist posterior sampling framework used in Zhang (2021) and subsequent work (e.g., Agarwal and Zhang (2022), Zhong et al. (2022)). In addition, it can be shown that the DEC is bounded whenever the information ratio of Russo and Van Roy, which is commonly used to analyze posterior sampling and related algorithms, is bounded; see discussion in Foster et al (2021,2022). Understanding connections between the DEC and more general posterior sampling-like algorithms is an interesting question for future research.\n\n> Prop 3.4: What is the expectation over? Is this an average regret over MDPs in the class and if so, is posterior sampling misspecified?\n\nNote that Proposition 3.4 concerns a frequentist setting, and “Posterior Sampling” in this context refers to the frequentist posterior sampling approach based on optimistic estimation given by Zhang (2022): $\\mu^{t}$ is a randomized estimator (distribution over models) produced by an optimistic estimation algorithm, and the frequentist posterior sampling scheme samples a model from this distribution and plays the optimal decision for it. Since we are in a frequentist setting, the expectation only considers the randomness of the algorithm and the randomness of the samples drawn from the model itself.\n\nWe mention in passing that while the purpose of this example was to compare to the frequentist posterior sampling approach of Zhang et al. (2022), it is straightforward to show that our lower bound construction also applies to classical posterior sampling in the Bayesian framework with a well-specified prior. We are happy to include this result if it is of interest to the reviewer.\n\n> The new coefficient is used to reproduce an existing result. This leads to a natural new meta-algorithm and theorem, but since they are only shown in one specialized case, it’s unclear whether it’s useful beyond this one setting as a “meta-algorithm.” While it is apparent that one could swap out different divergences and oracles, it may be described better as a single algorithm, inspired by the original E2D.\n\nThe goal of this work is not to obtain a new algorithm for certain model classes, but rather to show for the first time that the DEC and estimation-to-decisions framework for proving regret upper bounds leads to meaningful guarantees in the model-free setting. Prior to our work, the DEC/estimation-to-decisions framework (Foster et al. 2021) offered the most general framework for RL with general function approximation, as well as the only complexity measure that leads to both upper and lower bounds on regret. However, the upper bounds in Foster et al. (2021) are limited to model-based settings. On the other hand, complexity measures such as bilinear classes and bellman-eluder dimension apply to model-free settings, but are less general than the DEC (and only give upper bounds, not lower bounds). Our work gives a unifying perspective, and enjoys the generality of the DEC framework, yet also gives meaningful guarantees for model-free settings.\n\n> The regret bound appears to have a slightly worse rate than what would be achievable with the original BiLin-UCB, which I believe has T^{2 / 3} if converted properly to regret.\n\nPlease refer to our response to Reviewer kpYc for detailed discussion around this issue.\n\nThanks again for the suggestions!"}, {"review_id": "imPGeNat3G", "reviewer": "Reviewer_kpYc", "summary": "This paper studies a variant of the recently proposed framework of Foster et al. for sequential decision making, based on a concept called \"decision estimation coefficient\" (DEC). The variant proposed here is based on enhancing the estimation step in the \"estimation-to-decisions\" (E2D) with an optimistic bias inspired by the recent work of Zhang. The authors show that this biased estimation scheme, coupled with an appropriately adjusted decision-making rule, can satisfy very similar regret guarantees for a wide range of sequential decision problems, and can provide improvements in certain settings. In particular, the authors show that their approach can handle a large class of tractable models for reinforcement learning called \"bilinear classes\".", "questions": "See above.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and the technical content is of excellent quality. The proposed approach is justified and explained well, and its analysis is also presented in an accessible way (at least on a high level). While the regret bounds for bilinear MDP classes is not novel in the sense that there exist other algorithms that achieve the same guarantees, I appreciate the conceptual contribution of showing that the DEC framework is also capable of tackling these (relatively) challenging problems.\n\nI appreciated the very careful comparison between all the relevant DEC variants in Section 3: the authors didn't just propose a technique and proved some bounds about it, but also explored a range of other opportunities and explained the differences between them in an accessible manner.", "weaknesses": "On the negative side, the rates that the authors derive are not particularly great: the scaling goes from $T^{2/3}$  in the setting with the most stringent assumptions all the way to $T^{5/6}$ as more and more assumptions are dropped. While the authors discuss this limitation quite openly, I would have appreciated some more discussion as to where this relatively poor scaling comes from. One contributing factor is certainly the use of batched estimation steps. My understanding is that some further looseness may come from the optimistic bonuses added to the estimation procedure, which makes the total estimation error grow polynomially with the number of updates (as opposed to logarithmically, which would allow getting sqrt{T} rates after putting everything together). I wonder though if this intuition is correct, and I would appreciate it if the authors could clarify what rate they would get if they could afford to set n=1 (without paying for it). Altogether, it would have been nice if the to compare the various notions of estimation error with the same care as what the DEC variants have received.\n\nOverall, I am leaning towards suggesting acceptance, but I would feel more strongly about my support if the authors were able to address my questions above in a satisfying way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive comments and helpful suggestions!\n\n> While the authors discuss this limitation quite openly, I would have appreciated some more discussion as to where this relatively poor scaling comes from. One contributing factor is certainly the use of batched estimation steps. My understanding is that some further looseness may come from the optimistic bonuses added to the estimation procedure, which makes the total estimation error grow polynomially with the number of updates (as opposed to logarithmically, which would allow getting sqrt{T} rates after putting everything together). I wonder though if this intuition is correct, and I would appreciate it if the authors could clarify what rate they would get if they could afford to set n=1 (without paying for it). \n\nThis is a great question. There is indeed room for further work along these lines. The reason for our suboptimal rates differs slightly in the two settings we consider: (1) Corollary 2.1, which makes no Bellman completeness assumption, and (2) Corollary B.1, which assumes Bellman completeness. We discuss both cases separately below.\n\n**Without the completeness assumption (Proposition 2.1 and Corollary 2.1):** For our results that do not make use of completeness, we use a divergence based on average bellman error. Batching is necessary for algorithms based on average bellman error (Prop 2.1 and Corollary 2.1) due to the following well-known technical issue regarding average bellman error: Because the quantity D_{bi} that we are interested in is the *square of an average* as opposed to an *average of a square*, we cannot take advantage of concentration across rounds/iterations, and multiple samples are needed to estimate this quantity well for each iteration (note that the guarantee in Proposition 2.1 is vacuous if there is n=1 sample per batch). This issue is precisely why well-known algorithms based on average bellman error such as OLIVE (Jiang et al. 2017) and BiLinUCB (Du et al. 2021) require multiple samples per iteration, and is one of the main reasons why these algorithms are analyzed in the PAC framework instead of regret.\n\nWithout bellman completeness, we obtain T^{3/4} regret in the on-policy case and T^{5/6} for the off-policy case. The reason that these rates are worse than \\sqrt{T} is due to the batching issue above (as well as the additional issue of going off policy in the latter case), and is not related to the use of optimistic estimation. If one is interested in PAC guarantees instead of regret, we expect that our analysis can be adapted to provide tighter guarantees.\n\n**With the completeness assumption:** Corollary B.1 gets a rate of T^{2/3}. The algorithm uses a small batch size of n=H, which causes no degradation in rates, and is only needed to ensure that certain conditional independence assumptions required by Theorem C.1 are satisfied; batching is required in Agarwal & Zhang (2022) for the same reason. The improvement in rate that we achieve in the bellman-complete setting (compared to Corollary 2.1) is due to the fact that the batch size does not grow with T, which is facilitated by the two-timescale exponential weights method from Agarwal and Zhang (2022), which takes advantage of the bellman-completeness assumption. In fact, as stated, their result their result would seem to a imply bound on the estimation error in Proposition B.1 that scales as $\\log|\\mathcal{Q}| + \\sqrt{\\log|\\mathcal{Q}|*T}/\\gamma$, which would lead to a \\sqrt{T} regret bound for decision making. Unfortunately, there is a gap in their proof (reference [1] in our paper) which is due to the optimistic bonuses added to the estimation procedure. Our self-contained proof fixes this gap, but this leads to a degradation in the rate. We suspect that there exists an estimation algorithm that can obtain the estimation bound claimed above, which would to \\sqrt{T} regret, but this likely requires new algorithmic ideas and is out of scope for the current paper.\n\nWe mention in passing that compared to other model-free algorithms based on completeness (e.g., GOLF), the reason for the technical difficulties around estimation above is that our results require *online* estimation guarantees rather than *offline* estimation guarantees. This is a more stringent requirement, but is necessary to take advantage of the DEC.\n\n\n> Altogether, it would have been nice if the to compare the various notions of estimation error with the same care as what the DEC variants have received.\n\nThank you for the useful suggestion! We will add discussion around this point in the final version. \n"}, {"review_id": "CHmfKhaExp", "reviewer": "Reviewer_DHV3", "summary": "The paper studies model-free RL algorithm using DEC as the complexity measure. ", "questions": "See above.", "limitations": "1. No discussion or solution on how to implement the algorithm.\n2. The analysis seems to build on two existing works: Zhang [32] and Foster et al. [13]. Not very exciting about the technique. \n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Extending previous model-based algorithm to model-free algorithm is a nice contribution. ", "weaknesses": "I believe this paper provides a nice contribution for the DEC family of papers. But I would like the authors to discuss more on the implication side. Why we care about extending model-free algorithms to bilinear classes? There are too many complexity measures for RL function approximation that are proposed recently so I am concerned about how useful those theories are for practice. \n\nThe algorithm for Bellman-Eluder dimension is model-free. Then can I ask why we need a new algorithm? \n\nI hope the authors could pay more attention to the computational side. So far, all the lines of DEC works are not computationally efficient and even lack any experiments. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review and useful suggestions!\n\n> I believe this paper provides a nice contribution for the DEC family of papers. But I would like the authors to discuss more on the implication side. Why we care about extending model-free algorithms to bilinear classes? There are too many complexity measures for RL function approximation that are proposed recently so I am concerned about how useful those theories are for practice. The algorithm for Bellman-Eluder dimension is model-free. Then can I ask why we need a new algorithm?\n\n\nThe goal of this work is not to obtain a new algorithm for certain model classes, but rather to show for the first time that the DEC and estimation-to-decisions framework for proving regret upper bounds leads to meaningful guarantees in the model-free setting. Prior to our work, the DEC/estimation-to-decisions framework (Foster et al. 2021) offered the most general framework for RL with general function approximation, as well as the only complexity measure that leads to both *upper* and *lower* bounds on regret. However, the upper bounds in Foster et al. (2021) are limited to model-based settings. On the other hand, complexity measures such as bilinear classes and bellman-eluder dimension apply to model-free settings, but are less general than the DEC (and only give upper bounds, not lower bounds). Our work gives a unifying perspective, and enjoys the generality of the DEC framework, yet also gives meaningful guarantees for model-free settings.\n\nA secondary advantage of the estimation-to-decisions framework is that it yields a modular algorithm, with a conceptual separation between the estimation and decision-making components. In contrast, the GOLF algorithm (for classes with bounded Bellman-Eluder dimension) is more ad-hoc in nature, employing global optimism to select a value function and its associated greedy policy, which is optimistic and consistent with all prior observations. The estimation and decision-making components of GOLF are thus tightly coupled. Our work shows that these components can be decoupled while still obtaining similar guarantees, which we believe constitutes a significant conceptual contribution.\n\nLet us mention in passing that we consider bilinear classes only as a stylized example to illustrate our techniques. Bilinear classes capture many known generalizations of tabular MDPs, including linear MDPs, linear mixture MDPs, MDPs with low Bellman rank, feature selection in low-rank MDPs, and many more. Note, however, that our techniques are not limited to this setting.\n\n> I hope the authors could pay more attention to the computational side. So far, all the lines of DEC works are not computationally efficient and even lack any experiments.\n\nWe emphasize that focusing on statistical complexity as opposed to computational efficiency is a common theme in the line of research on RL with general function approximation, and is not exclusive to the DEC (e.g., existing works on bellman rank, bilinear classes, bellman-eluder dimension, and so on are also inefficient). Understanding how to design computationally efficient algorithms for RL with general function approximation is a fascinating direction for future research, but the first step, which our paper and the papers above works towards, is to understand what is even possible statistically. Nonetheless, we hope that by placing RL theory on solid statistical foundations, our work can serve as a starting point for future work on efficient algorithms."}]

Author rebuttal:

Meta-review: All the reviewers are supportive of the substance of the paper's contributions. Most of the suggestions for improvement are centered around discussion and contextualization (for example, the possibility of combining optimism with constrained DEC; contextualization around optimality in the regret rates; and the primary benefit of the DEC framework being statistical rather than computational). I highly recommend that the authors include this discussion and contextualization in the camera-ready version.
Decision: Accept (poster)