Title: Model-Free Reinforcement Learning with the Decision-Estimation Coefficient
Venue: NeurIPS 2023
Accepted: yes
Keywords: Decision making

learning theory

bandits

reinforcement learning theory

online learning

decision-estimation coefficient
Abstract: We consider the problem of interactive decision making, encompassing structured bandits and reinforcement
learning with general function approximation. Recently, Foster et al. (2021) introduced the
Decision-Estimation Coefficient, a measure of statistical complexity that lower bounds the optimal regret for interactive decision
making, as well as a meta-algorithm, Estimation-to-Decisions, which achieves upper
bounds in terms of the same quantity. Estimation-to-Decisions is a reduction, which lifts
algorithms for (supervised) online estimation into algorithms for
decision making. In this paper, we show that by combining Estimation-to-Decisions with
a specialized form of "optimistic" estimation introduced by
Zhang (2022), it is possible to obtain guarantees
that improve upon those of Foster et al. (2021) by
accommodating more lenient notions of estimation error. We use this approach to derive regret bounds for
model-free reinforcement learning with value function approximation, and give structural results showing when it can and cannot help more generally.

Model-Free Reinforcement Learning with the Decision-Estimation Coefficient

Abstract

We consider the problem of interactive decision making, encompassing structured bandits and reinforcement learning with general function approximation. Recently, Foster et al. [12] introduced the Decision-Estimation Coefficient, a measure of statistical complexity that lower bounds the optimal regret for interactive decision making, as well as a meta-algorithm, Estimation-to-Decisions, which achieves upper bounds in terms of the same quantity. Estimation-to-Decisions is a reduction , which lifts algorithms for (supervised) online estimation into algorithms for decision making. In this paper, we show that by combining Estimation-to-Decisions with a specialized form of optimistic estimation introduced by Zhang [31], it is possible to obtain guarantees that improve upon those of Foster et al. [12] by accommodating more lenient notions of estimation error. We use this approach to derive regret bounds for model-free reinforcement learning with value function approximation, and give structural results showing when it can and cannot help more generally.

1 Introduction

The theory of interactive decision making-ranging from bandits to reinforcement learning with function approximation-contains a variety of sufficient conditions for sample-efficient learning, [25, 17, 26, 11, 19, 7, 29, 18, 23, 3, 20, 10, 22, 9, 33], but necessary conditions have been comparatively unexplored. Recently, however, Foster et al. [12] introduced the Decision-Estimation Coefficient (DEC), a measure of statistical complexity which leads to upper and lower bounds on the optimal sample complexity for interactive decision making.

Regret bounds based on the Decision-Estimation Coefficient are achieved by Estimation-to-Decisions ( E2D ), a meta-algorithm which reduces the problem of interactive decision making to supervised online estimation. While the Decision-Estimation Coefficient leads to tight lower bounds on regret for many problem settings, the upper bounds in Foster et al. [12] can be suboptimal in certain situations due to the need to perform estimation with respect to Hellinger distance , a stringent notion of estimation error. When specialized to reinforcement learning, the guarantees for the E2D metaalgorithm in Foster et al. [12] are only tight for model-based settings (where function approximation is employed to model and estimate transition probabilities), and do not lead to meaningful guarantees for model-free settings with value function approximation. In this paper, we explore the prospect of developing tighter regret bounds suitable for model-free settings.

Contributions. We show that by combining Estimation-to-Decisions with optimistic online estimation , an elegant technique recently introduced by Zhang [31], it is possible to obtain regret bounds that improve upon Foster et al. [12] by accommodating weaker notions of estimation error. Our main contributions are:

We introduce a new optimistic variant of the Decision-Estimation Coefficient, and show that a variant of Estimation-to-Decisions that incorporates optimistic estimation achieves regret bounds that scale with this quantity (Section 2). Using this approach, we derive the first regret bounds for Estimation-to-Decisions applied to model-free reinforcement learning with bilinear classes [11] (Section 2.3).

We show that in general, whether or not optimistic estimation leads to improvement depends on the divergence with respect to which estimation is performed: For symmetric divergences, optimistic estimation offers no improvement, but for asymmetric divergences, including those found in reinforcement learning, the improvement can be drastic (Section 3). In addition, we highlight settings in which combining optimistic estimation with Estimation-toDecisions offers provable improvement over previous approaches that apply the technique with posterior sampling [31].

Perhaps the most important aspect of our work is to elucidate the connection between the DEC framework and optimistic estimation, building foundations for further research into these techniques.

In what follows, we review the Decision-Estimation Coefficient and Estimation-to-Decisions metaalgorithm (Section 1.2), highlighting opportunities for improvement. In Section 2, we present our main results, including our application to model-free reinforcement learning. We close with discussion and structural results, highlighting situations in which optimistic estimation can and cannot help (Section 3).

1.1 Problem Setting

We adopt the Decision Making with Structured Observations (DMSO) framework of Foster et al. [12], which is a general setting for interactive decision making that encompasses bandit problems (structured, contextual, and so forth) and reinforcement learning with function approximation.

The protocol consists of T rounds. For each round t = 1 , . . . , T :

The learner selects a decision π t ∈ Π , where Π is the decision space .

The learner receives a reward r t ∈ R ⊆ R and observation o t ∈ O sampled via ( r t , o t ) ∼ M glyph[star] ( π t ) , where M glyph[star] : Π → ∆( R×O ) is the underlying model .

Above, R is the reward space and O is the observation space . The model (conditional distribution) M glyph[star] represents the underlying environment, and is unknown to the learner, but the learner is assumed to have access to a model class M⊂ (Π → ∆( R×O )) that is flexible enough to capture M glyph[star] .

Assumption 1.1 (Realizability) . The learner has access to a model class M containing the true model M glyph[star] .

The model class M represents the learner's prior knowledge about the decision making problem, and allows one to appeal to estimation and function approximation. For structured bandit problems, models correspond to reward distributions, and M encodes structure in the reward landscape. For reinforcement learning problems, models correspond to Markov decision processes (MDPs), and M typically encodes structure in value functions or transition probabilities. We refer to Foster et al. [12] for further background.

For a model M ∈ M , E M ,π [ · ] denotes the expectation under the process ( r, o ) ∼ M ( π ) , f M ( π ) := E M ,π [ r ] denotes the mean reward function, and π M := arg max π ∈ Π f M ( π ) denotes the optimal decision. We measure performance in terms of regret, which is given by Reg DM := ∑ T t =1 E π t ∼ p t [ f Mglyph[star] ( π Mglyph[star] ) -f Mglyph[star] ( π t )] , where p t is the learner's randomization distribution for round t .

Additional notation. For an integer n ∈ N , we let [ n ] denote the set { 1 , . . . , n } . For a set Z , we let ∆( Z ) denote the set of all probability distributions over Z . For a model class M , co( M ) denotes the convex hull. We write f = ˜ O ( g ) to denote that f = O ( g · max { 1 , polylog( g ) } ) , and use glyph[lessorsimilar] as shorthand for a = O ( b ) .

1.2 Background: Estimation-to-Decisions and Decision-Estimation Coefficient

To motivate our results, this section provides a primer on the Estimation-to-Decisions meta-algorithm and the Decision-Estimation Coefficient. We refer to Foster et al. [12] for further background.

Algorithm 1 Estimation-to-Decisions ( E2D ) for General Divergences

1: parameters : Estimation oracle Alg Est , Exp. parameter γ > 0 , divergence D π ( · ‖ · ) .

2: for t = 1 , 2 , · · · , T do

3: Compute estimate ̂ M t = Alg t Est ( { ( π i , r i , o i ) } t -1 i =1 ) .

4: p t ← arg min p ∈ ∆(Π) sup M ∈M E π ∼ p [ f M ( π M ) -f M ( π ) -γ · D π ( ̂ M t ‖ M )] . // Eq. (2) .

5: Sample decision π t ∼ p t and update estimation oracle with ( π t , r t , o t ) .

Online estimation. Estimation-to-Decisions (Algorithm 1) is a reduction that lifts algorithms for online estimation into algorithms for decision making. An online estimation oracle, denoted by Alg Est , is an algorithm that, using knowledge of the class M , estimates the underlying model M glyph[star] from data in a sequential fashion. At each round t , given the data H t -1 = ( π 1 , r 1 , o 1 ) , . . . , ( π t -1 , r t -1 , o t -1 ) observed so far, the estimation oracle computes an estimate ̂ M t = Alg t Est ( { ( π i , r i , o i ) } t -1 i =1 ) for the true model M glyph[star] .

To measure the estimation oracle's performance, we make use of a user-specified divergence-like function , which quantifies the discrepancy between models. Formally, we define a divergence-like function (henceforth, 'divergence') as any function D : Π × co( M ) × co( M ) → R + , with D π ( M ‖ M ′ ) representing the discrepancy between the models M and M ′ at the decision π . Standard choices used in past work [12, 13, 5, 15, 14, 27] include the squared error D π sq ( M,M ′ ) := ( f M ( π ) -f M ′ ( π )) 2 for bandit problems, and squared Hellinger distance 1 D π H ( M,M ′ ) := D 2 H ( M ( π ) , M ′ ( π )) for RL, where for distributions P and Q , D 2 H ( P , Q ) := ∫ ( √ d P - √ d Q ) 2 . We then measure the estimation oracle's performance in terms of cumulative estimation error with respect to D , defined as

where p t is the conditional distribution over π t given H t -1 . We make the following assumption on the algorithm's performance.

Assumption 1.2. At each time t ∈ [ T ] , the online estimation oracle Alg Est returns, given ( π 1 , r 1 , o 1 ) , . . . , ( π t -1 , r t -1 , o t -1 ) with ( r i , o i ) ∼ M glyph[star] ( π i ) and π i ∼ p i , an estimator ̂ M t : Π → ∆( R×O ) such that Est D ≤ Est D ( T, δ ) , with probability at least 1 -δ , where Est D ( T, δ ) is a known upper bound.

For the squared error, one can obtain Est sq ( T, δ ) := Est D sq ( T, δ ) glyph[lessorsimilar] log( |F M | /δ ) , where F M := { f M | M ∈ M} , and for Hellinger distance, it is possible to obtain Est H ( T, δ ) := Est D H ( T, δ ) glyph[lessorsimilar] log( |M| /δ ) .

Estimation-to-Decisions. Ageneral version of the E2D meta-algorithm is displayed in Algorithm 1. At each timestep t , the algorithm queries estimation oracle to obtain an estimator ̂ M t using the data ( π 1 , r 1 , o 1 ) , . . . , ( π t -1 , r t -1 , o t -1 ) observed so far. The algorithm then computes the decision distribution p t by solving a min-max optimization problem involving ̂ M t and M (as well as the divergence D ), and then samples the decision π t from this distribution.

The Decision-Estimation Coefficient. The min-max optimization problem in Algorithm 1 is derived from the Decision-Estimation Coefficient (DEC), a complexity measure whose value, for a given scale parameter γ > 0 and reference model M : Π → ∆( R×O ) , is given by

with dec D γ ( M ) := sup M ∈ co( M ) dec D γ ( M , M ) . Informally, the DEC measures the best tradeoff between suboptimality ( f M ( π M ) -f M ( π )) and information gain (measured by D π ( M ‖ M ) ) that can be achieved by a decision distribution p in the face of a worst-case model M ∈ M .

The main result of Foster et al. [12] shows that the regret of E2D is controlled by the DEC and the estimation oracle's cumulative error Est D . Let ̂ M be any set for which ̂ M t ∈ ̂ M for all t almost surely. We have the following guarantee.

Theorem 1.1 (Foster et al. [12]) . Algorithm 1 with exploration parameter γ > 0 guarantees that Reg DM ≤ sup M ∈ ̂ M dec D γ ( M , M ) · T + γ · Est D almost surely.

For the special case of Hellinger distance, standard algorithms (exponential weights) achieve Est H ( T, δ ) glyph[lessorsimilar] log( |M| /δ ) with ̂ M = co( M ) , so that Theorem 1.1 gives (abbreviating dec H γ ≡ dec D H γ ):

Opportunities for improvement. Foster et al. [12, 15] provide lower bounds on the regret for any decision making problem that have a similar expression to (3), showing that the Decision-Estimation Coefficient with Hellinger distance ( dec H γ ( M ) ) plays a fundamental role in determining the statistical complexity of decision making. However, these lower bounds do contain the estimation term Est H ( T, δ ) = log( |M| /δ ) ) appearing in (3), and thus capture the price of moving from estimation to decision making (as characterized by the DEC), but not the complexity of estimation itself.

In general, the dependence on Est H ( T, δ ) = log( |M| /δ ) in the upper bound (3) can render the bound loose. In reinforcement learning, working with Hellinger distance necessitates modeling transition probabilities. While this leads to optimal results in some settings, in general the optimal rates for Hellinger estimation error can be prohibitively large, even in settings where model-free (or, value-based ) methods, which directly model value functions, are known to succeed; this drawback is shared by all subsequent work based on the DEC [13, 5, 15, 14, 27]. A natural solution is to replace Hellinger distance with a divergence (e.g., based on Bellman error) tailored to value function approximation, but naive choices for D along these lines render dec D γ ( M , M ) too large to give meaningful guarantees, and Foster et al. [12, 15] left this as an open problem. 2

2 Estimation-to-Decisions with Optimistic Estimation

To derive improved regret bounds that address the shortcomings described in the prequel, we combine Estimation-to-Decisions with a specialized estimation approach introduced by Zhang [31] (see also Dann et al. [6], Agarwal and Zhang [1, 2], Zhong et al. [32]), which we refer to as optimistic estimation . We then use this approach to derive regret bounds for model-free reinforcement learning.

2.1 Optimistic Estimation

The idea of optimistic estimation is to augment the estimation objective with a bonus that biases the estimator toward models M ∈ M for which the value f M ( π M ) is large. We present a general version of the technique.

Let a divergence D π ( · ‖ · ) be fixed. Following the development in Section 1.2, an optimistic estimation oracle Alg Est is an algorithm which, at each step t , given the observations and rewards collected so far, computes an estimate for the underlying model. For technical reasons, it will be useful to consider randomized estimators (Foster et al. [12], Chen et al. [5]) that, at each round, produce a distribution µ t ∈ ∆( M ) over models. Such estimators take the form µ t = Alg t Est ( { ( π i , r i , o i ) } t -1 i =1 ) . where µ t ∈ ∆( M ) . For a parameter γ > 0 , we define the optimistic estimation error as

This quantity is similar to (1), but incorporates a bonus term γ -1 ( f Mglyph[star] ( π Mglyph[star] ) -f ̂ Mt ( π ̂ Mt )) , which encourages the estimation algorithm to over-estimate the optimal value f Mglyph[star] ( π Mglyph[star] ) for the underlying model M glyph[star] .

Algorithm 2 Optimistic Estimation-to-Decisions ( E2D.Opt )

1: params : Estimation oracle Alg Est , Exp. param. γ > 0 , divergence D with suff. stat. space Ψ .

2: for t = 1 , 2 , · · · , T do

3: Receive randomized estimator µ t ∈ ∆(Ψ) = Alg t Est ( { ( π i , r i , o i ) } t -1 i =1 ) .

4: Compute

5: Sample decision π t ∼ p t and update estimation oracle with ( π t , r t , o t ) .

Assumption 2.1. At each time t ∈ [ T ] , the optimistic estimation oracle Alg Est returns, given ( π 1 , r 1 , o 1 ) , . . . , ( π t -1 , r t -1 , o t -1 ) with ( r i , o i ) ∼ M glyph[star] ( π i ) and π i ∼ p i , a randomized estimator µ t ∈ ∆( M ) such that OptEst D γ ≤ OptEst D γ ( T, δ ) , with w.p. 1 -δ , where OptEst D γ ( T, δ ) is a known upper bound.

For the case of contextual bandits, Zhang [31] proposes an augmented version of the exponential weights algorithm which, for a learning rate parameter η > 0 , sets µ ( M ) ∝ exp ( -η ( L t ( f M ) -γ -1 f M ( π M ) )) , where L t ( f M ) is the squared prediction error for the rewards observed so far. This method achieves E [ OptEst sq γ ] glyph[lessorsimilar] log( |F M | ) + √ T log |F M | /γ , and Zhang [31] combines this estimator with posterior sampling to achieve optimal contextual bandit regret. Agarwal and Zhang [1], Zhong et al. [32], Agarwal and Zhang [2] extend this development to reinforcement learning, also using posterior sampling as the exploration mechanism. 3 In what follows, we combine optimistic estimation with Estimation-to-Decisions, which provides a universal mechanism for exploration. Beyond giving guarantees which were previously out of reach for E2D (Section 2.3), this approach generalizes and subsumes posterior sampling, and can succeed in situations where posterior sampling fails (Section 3).

Remark 2.1. For the non-optimistic estimation error Est D , it is possible to obtain low error for wellbehaved losses such as the square loss and Hellinger distance without the use of randomization by appealing to improper mixture estimators (e.g., Foster et al. [12]). We show in Section 3 that for such divergences, randomization does not lead to statistical improvements. For the optimistic estimation error (4), randomization is essential due to the presence of the term γ -1 ( f Mglyph[star] ( π Mglyph[star] ) -f ̂ Mt ( π ̂ Mt ) . glyph[triangleleft]

Sufficient statistics. Before proceeding, we note that many divergences of interest have the useful property that they depend on the estimated model ̂ M only through a 'sufficient statistic' for the model class under consideration. Formally, there exists a sufficient statistic space Ψ and sufficient statistic ψ : M→ Ψ with the property that we can write (overloading notation)

for all models M,M ′ . In this case, it suffices for the online estimation oracle to directly estimate the sufficient statistic by producing a randomized estimator µ t ∈ ∆(Ψ) . We measure performance via

Examples include bandit problems, where one may use squared estimation error D π sq ( · , · ) and take ψ ( M ) = f M , and model-free reinforcement learning, where we show that by choosing the divergence D appropriately, one can use Q-value functions as a sufficient statistic. Note that we only focus on sufficient statistics for the first argument to D π ( · ‖ · ) , since this is the quantity we wish to estimate.

2.2 Algorithm and Main Result

We provide an optimistic variant of the E2D meta-algorithm ( E2D.Opt ) in Algorithm 2. At each timestep t , the algorithm calls the estimation oracle to obtain a randomized estimator µ t using the data

( π 1 , r 1 , o 1 ) , . . . , ( π t -1 , r t -1 , o t -1 ) collected so far. The algorithm then uses the estimator to compute a distribution p t ∈ ∆(Π) and samples π t from this distribution, with the main change relative to Algorithm 1 being that the minimax problem in Algorithm 2 is derived from an 'optimistic' variant of the DEC, which we refer to as the Optimistic Decision-Estimation Coefficient . For µ ∈ ∆( M ) , define

and o-dec D γ ( M ) = sup µ ∈ ∆( M ) o-dec D γ ( M , µ ) . The Optimistic DEC has two difference from the original DEC. First, it is parameterized by a distribution µ ∈ ∆( M ) rather than a reference model M : Π → ∆( R×O ) , which reflects the use of randomized estimators; the value in (6) takes the expectation over a reference model M drawn from this distribution (this modification also appears in the randomized DEC introduced in Foster et al. [12]). Second, and more critically, the optimal value f M ( π M ) in (2) is replaced by the optimal value f M ( π M ) for the (randomized) reference model. This seemingly small change is the main advantage of incorporating optimistic estimation, and makes it possible to bound the Optimistic DEC for certain divergences D for which the value of the unmodified DEC would otherwise be unbounded (cf. Section 3).

Remark 2.2. When the divergence D admits a sufficient statistic ψ : M→ Ψ , for any distribution µ ∈ ∆( M ) , if we define ν ∈ ∆(Ψ) via ν ( ψ ) = µ ( { M ∈ M : ψ ( M ) = ψ } ) , we have

In this case, by overloading notation slightly, we may write o-dec D γ ( M ) = sup ν ∈ ∆(Ψ) o-dec D γ ( M , ν ) . glyph[triangleleft]

Main result. Our main result shows that the regret of Optimistic Estimation-to-Decisions is controlled by the Optimistic DEC and the optimistic estimation error for the oracle Alg Est .

Theorem 2.1. For any δ > 0 , Algorithm 2 ensures that with probability at least 1 -δ ,

This regret bound has the same structure as Theorem 1.1, with the DEC and estimation error replaced by their optimistic counterparts. In the remainder of the paper, we show that 1) by adopting asymmetric divergences specialized to reinforcement learning, this result leads to the first guarantees for model-free RL with E2D , but 2) for symmetric divergences such as Hellinger distance, the result never improves upon Theorem 1.1.

Estimation with batching. For our application to reinforcement learning, we generalize the results above to accomodate estimation algorithms that draw batches of multiple samples from each distribution p t . Given a batch size n , we break the T rounds of the decision making protocol into K := T/n contiguous epochs (or, 'iterations'). Within each epoch, the learner's distribution p k is unchanged (we index by k rather than t to reflect this), and we create a batch B k = { ( π k,l , r k,l , o k,l ) } n l =1 by sampling π k,l ∼ p k independently and observing ( r k,l , o k,l ) ∼ M glyph[star] ( π k,l ) for each l ∈ [ n ] . We can then appeal to estimation algorithms of the form µ k = Alg k Est ( { B k } k -1 i =1 ) . Regret bounds for a variant of E2D.Opt with batching are given in Appendix B.1.

2.3 Application to Model-Free Reinforcement Learning

In this section, we use Optimistic Estimation-to-Decisions to provide sample-efficient guarantees for model-free reinforcement learning with bilinear classes [11], a general class of tractable reinforcement learning problems which encompasses many settings [17, 26, 7, 29, 18, 23, 3, 20, 10, 22, 9, 33].

Reinforcement learning preliminaries. To state our results, let us recall how reinforcement learning fits into the DMSO framework. We consider an episodic, finite-horizon reinforcement learning setting. With H denoting the horizon, each model M ∈ M specifies a non-stationary Markov decision process M = { S , A , { P M h } H h =1 , { R M h } H h =1 , d 1 } , where S is the state space, A is the action space, P M h : S ×A → ∆( S ) is the probability transition distribution at step h , R M h : S ×A → ∆( R ) is the reward distribution, and d 1 ∈ ∆( S 1 ) is the initial state distribution. We allow the reward distribution and transition kernel to vary across models in M , but assume that the initial state distribution is fixed.

For a fixed MDP M ∈ M , each episode proceeds under the following protocol. At the beginning of the episode, the learner selects a randomized, non-stationary policy π = ( π 1 , . . . , π H ) , where π h : S → ∆( A ) ; we let Π RNS (for 'randomized, non-stationary') denote the set of all such policies. The episode then evolves through the following process, beginning from s 1 ∼ d 1 : For h = 1 , . . . , H : a h ∼ π h ( s h ) , r h ∼ R M h ( s h , a h ) , and s h +1 ∼ P M h ( · | s h , a h ) . For notational convenience, we take s H +1 to be a deterministic terminal state. We assume for simplicity that R ⊆ [0 , 1] (that is, ∑ H h =1 r h ∈ [0 , 1] almost surely). Within the DMSO framework, at each time t , the learning agent chooses π t ∈ Π RNS , then observes the cumulative reward r t = ∑ H h =1 r t h and trajectory o t := ( s t 1 , a t 1 , r t 1 ) , . . . , ( s t H , a t H , r t H ) that results from executing π t .

Value functions. The value for a policy π under M is given by f M ( π ) := E M ,π [∑ H h =1 r h ] , where E M ,π [ · ] denotes expectation under the process above. For a given model M and policy π , we define the state-action value function and state value functions via Q M ,π h ( s, a ) = E M ,π [∑ H h ′ = h r h ′ | s h = s, a h = a ] , and V M ,π h ( s ) = E M ,π [∑ H h ′ = h r h ′ | s h = s ] . We define π M as the optimal policy, which maximizes Q M ,π M h ( s, a ) for all states simultaneously. We abbreviate Q M ,glyph[star] ≡ Q M ,π M .

Value function approximation. To apply our results to reinforcement learning, we take a model-free (or, value function approximation) approach, and estimate value functions for the underlying MDP M glyph[star] ; this contrasts with model-based methods, such as those considered in Foster et al. [12], which estimate transition probabilities for M glyph[star] directly. We assume access to a class Q of value functions of the form Q = ( Q 1 , . . . , Q H ) .

Assumption 2.2. The value function class Q has Q Mglyph[star] ,glyph[star] ∈ Q , where M glyph[star] is the underlying model.

For Q = ( Q 1 , . . . , Q H ) ∈ Q , we define π Q = ( π Q, 1 , . . . , π Q,H ) via π Q,h ( s ) = arg max a ∈A Q h ( s, a ) . We define Π Q = { π Q | Q ∈ Q} as the induced policy class. While is not necessary for our results, we mention in passing that the class Q , under Assumption 2.2, implicitly induces a model class via M Q := { M | Q M ,glyph[star] ∈ Q} .

Bilinear classes. The bilinear class framework [11] gives structural conditions for sample-efficient reinforcement learning that capture most known settings where tractable guarantees are possible. The following is an adaptation of the definition from Du et al. [11]. 4

Definition 2.1 (Bilinear class) . An MDP M is said to be bilinear with dimension d relative to a class Q if:

(i) There exist functions W h ( · ; M ) : Q → R d , X h ( · ; M ) : Q → R d such that for all Q ∈ Q and h ∈ [ H ] ,

(ii) Let z h := ( s h , a h , r h , s h +1 ) . There exists a collection of estimation policies { π est Q } Q ∈Q and a discrepancy function glyph[lscript] est h ( · ; · ) : Q×Z → R such that for all Q,Q ′ ∈ Q and h ∈ [ H ] , 5

If π est Q = π Q , we say that estimation is on-policy . We assume | E M ,π [ glyph[lscript] est h ( Q M ,glyph[star] ; z h )] | = 0 for all π .

We let d bi ( Q ; M ) denote the minimal dimension d for which the bilinear property holds for M . For a model class M , we define d bi ( Q ; M ) = sup M ∈M d bi ( Q ; M ) . We let L bi ( Q ; M ) ≥ 1 denote any almost sure upper bound on | glyph[lscript] est h ( Q ; z h ) | under M .

2.4 Guarantees for Bilinear Classes

We now apply our main results to derive regret bounds for bilinear classes. We first provide optimistic estimation guarantees, then bound the Optimistic DEC, and conclude by applying E2D.Opt .

We take Ψ = Q as the sufficient statistic space, with ψ ( M ) := Q M ,glyph[star] , and define f Q ( π Q ) := E s 1 ∼ d 1 [ Q 1 ( s 1 , π Q ( s 1 ))] . For the divergence D , we appeal to squared discrepancy, in the vein of Jiang et al. [17], Du et al. [11]:

We abbreviate OptEst bi γ = OptEst D bi γ and o-dec bi γ ( M , µ ) = o-dec D bi γ ( M , µ ) .

Estimation. To perform estimation, we approximate the average discrepancy in (14) from samples (drawing a batch of n samples at each step; cf. Algorithm 3), then appeal to the exponential weights method for online learning, with a bonus to enforce optimism. See Algorithm 6 (deferred to Appendix D for space).

Proposition 2.1. For any batch size n (with K := T/n ) and parameter γ ≥ 1 , Algorithm 6, with an appropriate learning rate, ensures that with probability at least 1 -δ ,

whenever M glyph[star] is bilinear relative to Q and Assumption 2.2 is satisfied.

This result does not actually use the bilinear class structure, and gives a bound on (14) for any choice of glyph[lscript] est h . Similar to the optimistic estimation result given by Zhang [31] for contextual bandits, this guarantee consists of 'slow' term √ K log |Q| γ resulting from optimism (which decays as γ grows), and a 'fast' term. However, compared to the contextual bandit setting, the fast term, log( |Q| KH/δ ) ( 1 + K n ) , scales with the ratio K n , which reflects sampling error in the estimated discrepancy, and necessitates a large batch size. Previous algorithms for bilinear classes [17, 11] require large batch sizes for similar reasons (specifically, because the expectation in (14) appears inside the square, it is not possible to form an unbiased estimate for D bi directly).

Bounding the DEC. A bound on the Optimistic DEC follows by adapting arguments in Foster et al. [12]; our result has slightly improved dependence on H compared to the bounds in that work.

.

Proposition 2.2. Let M be any model class for which the bilinear class property holds relative to Q (1) In the on-policy case where π est Q = π Q , we have that for all γ > 0 , o-dec bi γ ( M ) glyph[lessorsimilar] H · d bi ( Q ; M ) γ . (2) In the general case ( π est Q = π Q ), we have that for all γ ≥ H 2 d bi ( Q ; M ) , o-dec bi γ ( M ) glyph[lessorsimilar] √ H 2 · d bi ( Q ; M ) γ .

glyph[negationslash]

Combining Theorem B.1, Proposition 2.1, and Proposition 2.2, we obtain the following result.

Corollary 2.1 (Regret bound for bilinear classes) . Let Q be given. Assume that M glyph[star] ∈ M , where M is bilinear relative to Q , and that Assumption 2.2 holds. Abbreviate d ≡ d bi ( Q ; M ) and L ≡ L bi ( Q ; M ) := sup M ∈M L bi ( Q ; M ) . For an appropriate choice of n and γ , Algorithm 3, using the algorithm from Proposition 2.1 as an oracle, enjoys the following guarantees with probability at least 1 -δ :

glyph[negationslash]

(1) In the on-policy case where π est Q = π Q : Reg DM glyph[lessorsimilar] ( H 2 dL 2 log( |Q| TH/δ )) 1 / 2 T 3 / 4 . (2) In the general case where π est Q = π Q : Reg DM glyph[lessorsimilar] ( H 6 d 2 L 4 log 3 ( |Q| TH/δ )) 1 / 6 T 5 / 6 .

This is the first regret bound for model-free reinforcement learning with the Estimation-to-Decisions meta-algorithm. Importantly, the result scales only with the horizon, the dimension d , and the capacity log |Q| . Improving the dependence on T in Corollary 2.1 is an interesting question: currently, there are no algorithms for general bilinear classes that achieve √ T regret without additional assumptions. Let us emphasize that regret bounds for bilinear classes can already be achieved by a number of existing methods [11, 19]. The contribution here is to show that such guarantees can be achieved through the general DEC framework, thereby placing this line of research on stronger foundations.

Tighter guarantees under Bellman completeness. In Appendix B.2 (deferred for space), we adapt techniques from Agarwal and Zhang [2] to derive tighter estimation guarantees when Q satisfies a Bellman completeness property (e.g., Zanette et al. [30], Jin et al. [19]), by appealing to refined algorithms tailored to squared Bellman error. We use this to derive tighter regret bounds for bilinear classes ( T 2 / 3 instead of T 3 / 4 ).

3 Understanding the Role of Optimistic Estimation

We close with discussion and interpretation of our results.

When does optimistic estimation help? Perhaps the most pressing question at this point is to understand when the regret bound for E2D.Opt (Theorem 2.1) improves upon the corresponding regret bound for vanilla E2D (Theorem 1.1). In what follows, we show that: (1) For any divergence D , the Optimistic DEC is equivalent to a variant of the original DEC which incorporates randomized estimators [12], but with the arguments to the divergence flipped ; (2) For divergences D that satisfy a triangle inequality, this randomized DEC is equivalent to the original DEC itself. Together these results show that the improvement given by the Optimistic DEC is limited to asymmetric divergences such as the bilinear divergence in Section 2.3; for more traditional divergences such as Hellinger distance and squared error, the optimistic approach offers no improvement. Our results use the following regularity assumption, satisfied by all standard divergences.

Given a divergence D , we define the flipped divergence , which swaps the first and second arguments, by q D π ( M ‖ M ) := D π ( M ‖ M ) . We define the Decision-Estimation Coefficient for randomized estimators [12, 5] as dec D γ ( M , µ ) = inf p ∈ ∆(Π) sup M ∈M E π ∼ p [ f M ( π M ) -f M ( π ) -γ · E M ∼ µ [ D π ( M ‖ M )]] , with dec D γ ( M ) := sup µ ∈ ∆( M ) dec D γ ( M , µ ) . This definition is identical to (2), but allows M to be randomized.

Proposition 3.1. Whenever Assumption 3.1 holds, we have that for all γ > 0 ,

For settings in which there exists an estimation oracle for which the flipped estimation error Est q D = ∑ T t =1 E π ∼ p t E ̂ M t ∼ µ t [ D π t ( M glyph[star] ‖ ̂ M t )] is controlled, this result shows that to match the guarantee in Theorem 2.1, optimism is not required, and it suffices to run a variant of vanilla E2D that incorporates randomized estimators (cf. Foster et al. [12], Section 4.3).

We now turn to the role of randomization. When D is convex in the first argument, we have dec D γ ( M ) ≤ sup M ∈ co( M ) dec D γ ( M , M ) = dec D γ ( M ) , but it is not immediately apparent whether the opposite direction of this inequality holds, and one might hope that working with the randomized DEC in Proposition 3.1 would lead to improvements over the non-randomized counterpart in Theorem 1.1. The next result shows that this is not the case: Under mild assumptions on the divergence D , randomization offers no improvement.

Proposition 3.2. Let D be any bounded divergence such that for all M,M ′ , M and π ∈ Π , D π ( M ‖ M ′ ) ≤ C ( D π ( M ‖ M ) + D π ( M ‖ M ′ )) . Then for all γ > 0 , sup M dec D γ ( M , M ) ≤ dec D γ/ (2 C ) ( M ) .

Implication for Hellinger distance. Squared Hellinger distance is symmetric, satisfies Assumption 3.1 with L lip = 1 whenever R ⊆ [0 , 1] , and satisfies the condition in Proposition 3.2 with C = 2 . Hence, by combining Proposition 3.1 with Proposition 3.2, we obtain the following corollary.

This shows that for Hellinger distance-at least from a statistical perspective-there is no benefit to using the Optimistic DEC or randomized DEC compared to the original version. In particular, this implies that regret bounds based on the randomized DEC with Hellinger distance (such as those found

in recent work of Chen et al. [5]) do not offer improvement over the guarantees for vanilla E2D in Foster et al. [12]. One caveat, though, is that working with the Optimistic DEC, as well as randomized estimators, has potential to give computational improvement, as computing a distribution p ∈ ∆(Π) that minimizes o-dec H γ ( M , µ ) might be simpler than computing a corresponding distribution for dec H γ ( M , M ) with M ∈ co( M ) . We are not currently aware of any examples where such an improvement occurs, as even maintaining a distribution µ ∈ ∆( M ) requires intractably large memory for most classes of interest.

Implication for model-free RL. The bilinear divergence D π bi ( Q ‖ M ) = ∑ H h =1 ( E M ,π [ glyph[lscript] est h ( Q ; z h )]) 2 that we adopt in Section 2.3 is asymmetric, as are closely related divergences such as squared Bellman error. Here, there are two reasons why optimistic estimation offers meaningful advantages.

(1) By Proposition 3.1 ( D bi satisfies Assumption 3.1 with L lip = O ( H ) ), a natural alternative to optimistic estimation is to estimate with respect to the flipped divergence q D bi , then appeal to Algorithm 3 of Foster et al. [12]. The issue with this approach is that minimizing the flipped estimation error, which takes the form

is challenging in model-free settings; we are not aware of any algorithms that accomplish this. 6

(2) Alternatively, a second choice is to perform estimation with respect to the un-flipped divergence D bi (which can be accomplished with Proposition 2.1 by taking γ →∞ ), and appeal to vanilla E2D (either Algorithm 1, or Algorithm 3 of Foster et al. [12] if one wishes to incorporate randomized estimators). However, the following result shows that unlike the Optimistic DEC, the original DEC with the divergence D bi does not admit a favorable bound, even for tabular reinforcement learning.

Proposition 3.3. Let M be the class of all horizonH tabular MDPs with |S| = 2 and |A| = 2 . Consider the discrepancy function glyph[lscript] est h ( Q ; z h ) = ( Q h ( s h , a h ) -r h -max a ′ ∈A Q h +1 ( s h +1 , a ′ )) . Then we have o-dec bi γ ( M ) glyph[lessorsimilar] H γ , yet there exists M ∈ M for which dec bi γ ( M , M ) glyph[greaterorsimilar] 2 H γ ∧ 1 .

Insufficiency of posterior sampling. For contextual bandits, where o-dec bi γ ( M ) glyph[lessorsimilar] |A| γ [31], and bilinear classes, where o-dec bi γ ( M ) glyph[lessorsimilar] H · d bi ( M ) γ (Proposition 2.2), a strategy that achieves the bound on the Optimistic DEC is posterior sampling (this is also the approach taken in Agarwal and Zhang [1, 2], Zhong et al. [32]). That is, given a distribution µ ∈ ∆( M ) , choosing p ( π ) = µ ( { M ∈ M| π M = π } ) in (6) certifies the desired bound on o-dec D γ ( M , µ ) for these examples. Optimistic Estimation-to-Decisions subsumes and generalizes posterior sampling, but in light of the fact that this simple strategy succeeds for large classes of problems, it is reasonable to ask if there is a sense in which posterior sampling is universal, and whether it can achieve the value of the Optimistic DEC for any model class. This would be desirable, since it would indicate that solving the minimax problem in Algorithm 2 is not necessary. The following sample shows that this is not the case: there are model classes (specifically, MDPs with a constant number of actions) for which the regret of posterior sampling is exponentially large compared to the regret of Algorithm 2.

Proposition 3.4. Consider the divergence D π H ( · , · ) . For any S ∈ N and H ≥ log 2 ( S ) , there exists a class of horizonH MDPs M with |S| = S and |A| = 3 that satisfies the following properties:

There exists an estimation oracle with OptEst H γ glyph[lessorsimilar] log( S/δ ) w.p. at least 1 -δ for all γ > 0 .

Posterior sampling, which sets p t ( π ) = µ t ( { M | π M = π } ) , has E [ Reg DM ] glyph[greaterorsimilar] S ∧ 2 Ω( H ) .

Algorithm 2 with divergence D = D π H ( · , · ) has E [ Reg DM ] ≤ ˜ O (√ T log( S ) ) .

This shows that posterior sampling does not provide a universal mechanism for exploration, and highlights the need for deliberate strategies such as E2D .

References

A. Agarwal and T. Zhang. Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. Advances in Neural Information Processing Systems , 35: 35284-35297, 2022.

A. Agarwal and T. Zhang. Non-linear reinforcement learning in large action spaces: Structural conditions and sample-efficiency of posterior sampling. In Conference on Learning Theory , pages 2776-2814. PMLR, 2022.

A. Ayoub, Z. Jia, C. Szepesvari, M. Wang, and L. Yang. Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning , pages 463-474. PMLR, 2020.

N. Cesa-Bianchi and G. Lugosi. Prediction, Learning, and Games . Cambridge University Press, New York, NY, USA, 2006. ISBN 0521841089.

F. Chen, S. Mei, and Y. Bai. Unified algorithms for RL with decision-estimation coefficients: No-regret, PAC, and reward-free learning. arXiv preprint arXiv:2209.11745 , 2022.

C. Dann, M. Mohri, T. Zhang, and J. Zimmert. A provably efficient model-free posterior sampling method for episodic reinforcement learning. Advances in Neural Information Processing Systems , 34:12040-12051, 2021.

S. Dean, H. Mania, N. Matni, B. Recht, and S. Tu. On the sample complexity of the linear quadratic regulator. Foundations of Computational Mathematics , 20(4):633-679, 2020.

O. D. Domingues, P. Ménard, E. Kaufmann, and M. Valko. Episodic reinforcement learning in finite mdps: Minimax lower bounds revisited. In Algorithmic Learning Theory , pages 578-598. PMLR, 2021.

S. Dong, B. Van Roy, and Z. Zhou. Provably efficient reinforcement learning with aggregated states. arXiv preprint arXiv:1912.06366 , 2019.

S. Du, A. Krishnamurthy, N. Jiang, A. Agarwal, M. Dudik, and J. Langford. Provably efficient RL with rich observations via latent state decoding. In International Conference on Machine Learning , pages 1665-1674. PMLR, 2019.

S. S. Du, S. M. Kakade, J. D. Lee, S. Lovett, G. Mahajan, W. Sun, and R. Wang. Bilinear classes: A structural framework for provable generalization in RL. International Conference on Machine Learning , 2021.

D. J. Foster, S. M. Kakade, J. Qian, and A. Rakhlin. The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 , 2021.

D. J. Foster, A. Rakhlin, A. Sekhari, and K. Sridharan. On the complexity of adversarial decision making. Neural Information Processing Systems (NeurIPS) , 2022.

D. J. Foster, D. P. Foster, N. Golowich, and A. Rakhlin. On the complexity of multi-agent decision making: From learning in games to partial monitoring. Conference on Learning Theory (COLT) , 2023.

D. J. Foster, N. Golowich, and Y. Han. Tight guarantees for interactive decision making with the decision-estimation coefficient. Conference on Learning Theory (COLT) , 2023.

D. A. Freedman. On tail probabilities for martingales. the Annals of Probability , pages 100-118, 1975.

N. Jiang, A. Krishnamurthy, A. Agarwal, J. Langford, and R. E. Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In International Conference on Machine Learning , pages 1704-1713, 2017.

C. Jin, Z. Yang, Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143, 2020.

C. Jin, Q. Liu, and S. Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. Neural Information Processing Systems , 2021.

A. Krishnamurthy, A. Agarwal, and J. Langford. PAC reinforcement learning with rich observations. In Advances in Neural Information Processing Systems , pages 1840-1848, 2016.

T. Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

L. Li. A unifying framework for computational reinforcement learning theory . Rutgers, The State University of New Jersey-New Brunswick, 2009.

A. Modi, N. Jiang, A. Tewari, and S. Singh. Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics , pages 2010-2020. PMLR, 2020.

I. Osband and B. Van Roy. On lower bounds for regret in reinforcement learning. arXiv preprint arXiv:1608.02732 , 2016.

D. Russo and B. Van Roy. Eluder dimension and the sample complexity of optimistic exploration. In Advances in Neural Information Processing Systems , pages 2256-2264, 2013.

W. Sun, N. Jiang, A. Krishnamurthy, A. Agarwal, and J. Langford. Model-based RL in contextual decision processes: PAC bounds and exponential improvements over model-free approaches. In Conference on learning theory , pages 2898-2933. PMLR, 2019.

A. Wagenmaker and D. J. Foster. Instance-optimality in interactive decision making: Toward a non-asymptotic theory. Conference on Learning Theory (COLT) , 2023.

T. Xie, D. J. Foster, Y. Bai, N. Jiang, and S. M. Kakade. The role of coverage in online reinforcement learning. International Conference on Learning Representations (ICLR) , 2023.

L. Yang and M. Wang. Sample-optimal parametric Q-learning using linearly additive features. In International Conference on Machine Learning , pages 6995-7004. PMLR, 2019.

A. Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

T. Zhang. Feel-good thompson sampling for contextual bandits and reinforcement learning. SIAM Journal on Mathematics of Data Science , 4(2):834-857, 2022.

H. Zhong, W. Xiong, S. Zheng, L. Wang, Z. Wang, Z. Yang, and T. Zhang. A posterior sampling framework for interactive decision making. arXiv preprint arXiv:2211.01962 , 2022.

D. Zhou, Q. Gu, and C. Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

Contents of Appendix

A Additional Related Work

In this section we discuss additional related work not already covered.

Chen et al. [5] build on the DEC framework by giving regret bounds for a E2D that incorporates randomized estimators, but not optimism. For the case of finite classes, their guarantees scale as roughly

where dec D H γ ( M ) is the DEC for randomized estimators defined in Section 3. As discussed in Section 3, this regret bound cannot improve upon the guarantees for the original E2D method in Foster et al. [12] beyond constants, as we have sup M dec H 4 γ ( M , M ) ≤ dec D H γ ( M ) . In addition, since the algorithm does not incorporate optimism, it cannot be directly applied to model-free reinforcement learning settings.

Foster et al. [15] give upper and lower bounds on optimal regret based on a variant of the DEC called the constrained Decision-Estimation Coefficient . These results tighten the original regret bounds in Foster et al. [12], but the upper bounds still scale with Est H ( T, δ ) = log( |M| /δ ) , rendering them unsuitable for model-free RL. Nonetheless it would be interesting to explore whether the techniques in this work can be combined with optimistic estimation.

B Additional Results

B.1 Optimistic Estimation-to-Decisions with Batching

Algorithm 3 Optimistic Estimation-to-Decisions ( E2D.Opt ) with Batching

1: parameters :

Online estimation oracle Alg Est with batch size n .

Exploration parameter γ > 0 .

Divergence D ( · ‖ · ) with sufficient statistic space Ψ .

2: Let K := T/n .

3: for k = 1 , 2 , · · · , K do

4: Receive randomized estimator µ k ∈ ∆(Ψ) = Alg t Est ( ( B i ) k -1 i =1 ) .

5: Get p k ← arg min p ∈ ∆(Π) sup M ∈M E π ∼ p E ̂ ψ ∼ µ k [ f ̂ ψ ( π ̂ ψ ) -f M ( π ) -γ · D π ( ̂ ψ ‖ M )] . // Eq. (6).

6: Sample batch B k = { ( π k,l , r k,l , o k,l ) } n l =1 where π k,l ∼ p k and ( r k,l , o k,l ) ∼ M glyph[star] ( π k.l ) , and update estimation oracle with B k .

For our application to reinforcement learning, it will be useful to generalize E2D.Opt to accomodate estimation algorithms that draw batches of multiple samples from each distribution p t . Given a batch

size n , we break the T rounds of the decision making protocol into K := T/n contiguous epochs (or, 'iterations'). Within each epoch, the learner's distribution p k is unchanged (we index by k rather than t to reflect this), and we create a batch B k = { ( π k,l , r k,l , o k,l ) } n l =1 by sampling π k,l ∼ p k independently and observing ( r k,l , o k,l ) ∼ M glyph[star] ( π k,l ) for each l ∈ [ n ] . We consider estimation oracles of the form µ k = Alg k Est ( { B k } k -1 i =1 ) , and measure estimation error via

We assume that the estimation oracle ensures that with probability at least 1 -δ , OptEst D γ ≤ OptEst D γ ( K,n,δ ) , where OptEst D γ ( K,n,δ ) is a known upper bound.

Algorithm 3 is a variant of E2D.Opt that incorporates batching. The algorithm updates the distribution p k in the same fashion as its non-batched counterpart, but does so only at the beginning of each epoch. The main guarantee for this method as follows.

Theorem B.1. Let T ∈ N be given, and let n be the batch size. For any δ > 0 , Algorithm 3 ensures that with probability at least 1 -δ , Reg DM ≤ o-dec D γ ( M ) · T + γn · OptEst D γ ( T/n,n,δ ) .

See Appendix D.1 for the proof. When working with divergences for which unbiased estimates are unavailable, this approach can lead to stronger guarantees than Theorem 2.1. We refer to the proof of Proposition 2.1 for a concrete example.

B.2 Model-Free RL: Tighter Guarantees under Bellman Completeness

In this section, we show how to derive tighter estimation guarantees (and consequently tighter regret bounds) when Q satisfies a Bellman completeness assumption [30, 19]. For a given MDP M , let T M h [ f ]( s, a ) := E M [ r h +max a ′ f ( s h +1 , a ′ ) | s h = s, a h = a ] denote the Bellman operator for layer h .

Assumption B.1 (Completeness) . We assume that Q = Q 1 ×··· × Q H is a product class, and that for all h and Q h +1 ∈ Q h +1 , [ T Mglyph[star] h Q h +1 ] ∈ Q h .

As before, we take Ψ = Q , ψ ( M ) := Q M ,glyph[star] , and f Q ( π Q ) := E s 1 ∼ d 1 [ Q 1 ( s 1 , π Q ( s 1 ))] . For the divergence D , we now appeal to squared Bellman error (e.g., [19, 28]):

We abbreviate OptEst sbe γ = OptEst D sbe γ and o-dec sbe γ ( M , µ ) = o-dec D sbe γ ( M , µ ) .

Estimation. Algorithm 4 performs optimistic online estimation with squared Bellman error. The algorithm is an adaptation of a two-timescale exponential weights strategy originally introduced by Agarwal and Zhang [2] within an optimistic posterior sampling algorithm referred to as TS3 . We show that this technique leads to a self-contained online estimation guarantee outside the context of the TS3 algorithm.

Proposition B.1 (Estimation for square Bellman error) . Assume that Q satisfies completeness relative to M glyph[star] . Moreover assume ∑ H h =1 sup s,a r h ( s, a ) ≤ 1 and sup Q,h,s,a Q h ( s, a ) ≤ 1 . Then for any γ ≥ 1 and η ≤ 1 / (2 16 (log( |Q| K/δ ) + 1)) , with batch size n = H ( K := T/n ), λ = 1 / 8 , β = (12 γH ) -1 and δ > 0 , Algorithm 4 ensures that with probability at least 1 -δ ,

whenever Q satisfies completeness relative to M glyph[star] . 7

Note that Proposition B.1 does not make use of the bilinear class assumption, and only requires that Q satisfies completeness. As such, we expect that this result will find use more broadly.

Algorithm 4 Two-Timescale Exponential Weights for Bellman Complete Value Function Classes

1: Initialize S 0 = ∅ .

2: for k = 1 , . . . , K do

3: For any Q,Q ′ ∈ Q and h ∈ [ H ] , define

4: Predict µ k .

5: for l = 1 , . . . , H do

6: Play π k,l ∼ p k and obtain the trajectory o k,l = ( s k,l 1 , a k,l 1 , r k,l 1 ) , . . . , ( s k,l H , a k,l H , r k,l H ) , where p k ∈ ∆(Π) is a decision distribution produced by any batched algorithm (e.g., Algorithm 3) that selects a decision adaptively based on µ k .

7: Update S k ← S k -1 ∪ ⋃ H l =1 { s k,l l , a k,l l , r k,l l , s k,l l +1 } ∪ ⋃ H l =1 { s k,l 1 } .

Regret bound for bilinear classes. To provide regret bounds, we assume that M glyph[star] satisfies the bilinear class property relative to Q as in Section 2.3. In addition to assuming that M glyph[star] is bilinear, we make the following restrictions: (1) π est Q = π Q , i.e. estimation is on policy, (2) glyph[lscript] est h ( Q ; z h ) = Q h ( s h , a h ) -r h -max a ′ Q h +1 ( s h +1 , a ′ ) , so that E M ,π [ glyph[lscript] est h ( Q ; z h )] is the average Bellman error for Q under M . 8 With this discrepancy function, Jensen's inequality implies that o-dec sbe γ ( M ) ≤ o-dec bi γ ( M ) , so combining Theorem B.1, Proposition B.1, and Proposition 2.2, we obtain the following result.

Corollary B.1 (Regret bound under completeness) . Let Q be given. Assume that M glyph[star] ∈ M , where M is bilinear relative to Q , and that completeness holds. Moreover assume ∑ H h =1 sup s,a r h ( s, a ) ≤ 1 and sup Q,h,s,a Q h ( s, a ) ≤ 1 . Abbreviate d ≡ d bi ( Q ; M ) . For an appropriate choice of n and γ , Algorithm 3, using Algorithm 4 (with appropriate parameter choice) as an oracle, enjoys the following guarantees with probability at least 1 -δ :

This improves upon the T 3 / 4 -type rate in Corollary 2.1.

B.3 Proofs from Appendix B.2

Proposition B.1 is an application of more general results given in Appendix C.3, which analyze a generalization of Algorithm 4 for a more general online learning setting. To Proposition B.1, we simply apply these results to the reinforcement learning framework.

Proof of Proposition B.1. Let the batch size n = H be fixed, and let K := T/n be the number of epochs. Recall that for each step k ∈ [ K ] , the estimation oracle is given a batch of examples B k = { ( π k,l , r k,l , o k,l ) } n l =1 where π k,l ∼ p k and ( r k,l , o k,l ) ∼ M glyph[star] ( π k,l ) . Each observation (trajectory) takes the form o k,l = ( s k,l 1 , a k,l 1 , r k,l 1 ) , . . . , ( s k,l H , a k,l H , r k,l H ) . We abbreviate Q glyph[star] = Q Mglyph[star] ,glyph[star] .

Estimation algorithm. For each step k , the randomized estimator µ k selected as described in Algorithm 4. This algorithm is an instantiation of Algorithm 5 in the general online learning setting described in Appendix C.3, with G = Q and for all h ∈ [ H ] , X h = S × A , Y h = R × S and

W = S H . The unknown kernels are the transition distributions for the corresponding layers of the MDP M glyph[star] , and the loss functions are

Finally, take x k h = ( s k,h h , a k,h h ) , y k h = ( r k,h h , s k,h h +1 ) and w k = { s k,h 1 } h ∈ [ H ] . It is important to note that s k,h h , a k,h h , r k,h h , s k,h h +1 are taken from different trajectories for h ∈ [ H ] , so y k h | x k h are independent from one other for h ∈ [ H ] . Moreover, note that the distributions p k ∈ ∆(Π) play the role of nature: the distribution of the tuple ( x k h , y k h , w k ) for h ∈ [ H ] is determined by running a policy π k ∼ p k in the ground-truth MDP M glyph[star] . With this configuration, observe that in the notation of Appendix C.3, we have, for any Q ,

and

Estimation error bound. We take α = 12 β , so that Theorem C.1 implies that with probability at least 1 -δ ,

Then by taking α = 1 γH , this further implies that with probability at least 1 -δ ,

Proof of Corollary B.1. We choose n = H and apply Algorithm 4 as the estimation oracle. We first consider the 'trivial' parameter regime in which Hd 1 / 3 (log( |Q| K/δ )) -1 / 5 T -1 / 3 ≥ 1 / (2 16 (log( |Q| K/δ ) + 1)) . Here, T glyph[lessorsimilar] Hd 1 / 3 (log( |Q| K/δ )) 4 / 5 T 2 / 3 , and thus

When the case above, does not hold, we proceed as in the theorem statement, choosing η = Hd 1 / 3 (log( |Q| K/δ )) -1 / 5 T -1 / 3 ≤ 1 / (2 16 (log( |Q| K/δ )+1)) . Combining Theorem B.1 and Proposition B.1 then gives

with probability at least 1 -δ . Next, using Proposition 2.2 to bound o-dec sbe γ ( M ) in the above display, it follows that

We choose γ = d 2 / 3 (log( |Q| K/δ )) -2 / 5 T 1 / 3 to obtain

with probability at least 1 -δ .

C Technical Tools

C.1 Preliminaries

Lemma C.1. For all x ∈ [0 , 1] , we have

Lemma C.2. For all x ≥ -1 / 8 , we have e -x ≤ 1 -x + x 2 .

Proof of Lemma C.2. Let f ( x ) = e -x -1+ x -x 2 . We have f ′′ ( x ) = e -x -2 < 0 for x ≥ -1 / 8 . Thus, f ′ is monotonically decreasing on x ≥ -1 / 8 , so for x ∈ [ -1 / 8 , 0] , f ′ ( x ) ≥ f ′ (0) = 0 . Hence, f ( x ) is non-decreasing on x ∈ [ -1 / 8 , 0] . Furthermore, f ′ ( x ) = -e -x +1 -2 x ≤ 0 for x ≥ 0 . Thus f ( x ) obtains maximum value at x = 0 , and f ( x ) ≤ f (0) = 0 .

C.2 Basic Online Learning Results

In this section we state a technical lemma regarding the performance of the exponential weights algorithm for online learning. Let G be an abstract set of hypotheses. We consider the following online learning process.

Learner predicts a (random) hypothesis g t ∈ G .

Nature reveals glyph[lscript] t ∈ L := ( G → R ) and learner suffers loss glyph[lscript] t ( g t ) .

We define regret to the class G via

where µ t ∈ ∆( G ) is the learner's randomization distribution for step t .

Lemma C.3. Consider the exponential weights update method with learning rate η > 0 , which sets

For any sequence of non-negative loss functions glyph[lscript] 1 , . . . , glyph[lscript] T , this algorithm satisfies

In addition, for any sequence of loss functions glyph[lscript] 1 , . . . , glyph[lscript] T with glyph[lscript] t ( g ) ∈ [ -L, L ] for all g ∈ G , if η ≤ (2 L ) -1 , then

Proof of Lemma C.3. A standard telescoping argument combined with the fact that -inf g ∈G ∑ T t =1 glyph[lscript] t ( g ) ≤ 1 η log ( exp ( ∑ g ∈G -η ∑ T t =1 glyph[lscript] t ( g ) )) (e.g., Cesa-Bianchi and Lugosi [4]) gives that for any choice η > 0 and any sequence of loss functions, exponential weights has

We first prove (18). Using that log( x ) ≤ x -1 for x ≥ 0 and exp( -x ) ≤ 1 -x + x 2 2 for x ≥ 0 , we have so that

To prove (19), we use that log( x ) ≤ x -1 for x ≥ 0 and exp( -x ) ≤ 1 -x +2 x 2 whenever | x | ≤ 1 to get

so that

C.3 Online Learning with Completeness

In this section, we give guarantees for an online learning algorithm Algorithm 5, which generalizes the two-timescale exponential weights algorithm (Algorithm 4) of Agarwal and Zhang [2]. We describe and analyze the algorithm in a general online learning framework, which abstracts away the core problem solved by Algorithm 4: value function estimation using a Bellman complete value function class.

Let G be an abstract set of hypotheses. We consider and online learning process parameterized by a positive integer H and α ∈ [0 , 1] .

There are 2 H +1 outcome spaces, {X h } h ∈ [ H ] , {Y h } h ∈ [ H ] , and W .

There are H unknown probability kernels {K h : X h →Y h } h ∈ [ H ] .

There are 2 H +1 known loss functions { glyph[lscript] h, 1 : X h ×G → [0 , 1] } h ∈ [ H ] , { glyph[lscript] h, 2 : Y h ×G → [0 , 1] } h ∈ H , and l 3 : W×G → [0 , 1] .

Define S 0 = ∅ . We consider the following process. For t = 1 , . . . , T :

Learner predicts a (randomized) hypothesis f t ∈ G .

Nature reveals { x t h , y t h } h ∈ [ H ] and w t . The outcomes { x t h } h ∈ [ H ] can be chosen adaptively, but they are mutually independent given S t -1 . The outcome w t can be chosen adaptively based on S t -1 and { x t h } h ∈ [ H ] . Each outcome y t h ∼ K h ( x t h ) is drawn independently for each h ∈ [ H ] .

The history is updated via S t ← S t -1 ⋃ { x t h , y t h } h ∈ [ H ] ⋃ { w t } .

The learner suffers loss

The learner's goal is to minimize a form of regret for the cumulative loss given in (22). This loss function reflects two objective . The first objective involves the H losses { glyph[lscript] 1 ,h } h ∈ [ H ] with corresponding outcomes { x h } h ∈ [ H ] , as well as the H losses { glyph[lscript] 2 ,h } h ∈ [ H ] tied to outcomes { y h } h ∈ [ H ] , which are generated stochastically based on { x h } h ∈ [ H ] . The primary objective is to minimize the primary error

The secondary objective is to minimize T ∑ t =1 l 3 ( w t , f t ) , and the ultimate goal is to minimize a weighted sum of the two objectives.

This online learning setup, adapted from Agarwal and Zhang [2], generalizes the reinforcement learning setting in which Algorithm 4 operates. The adaptively chosen outcome x t h corresponds to the state-action pair at the h -th step, ( s t h , a t h ) , with the policy at time t chosen in an adaptive, potentially adversarial fashion. The conditionally stochastic outcome y t h corresponds to the reward and the next state ( r t h , s t h +1 ) , which is sampled independently from the MDP's reward distribution and transition distribution at step h , and is conditionally independent given ( s t h , a t h ) . The learner's objective in the RL framework is to predict a value function f t = Q t that minimizes the squared Bellman error, realized by selecting the losses glyph[lscript] 1 ,h ( x t h , f t ) = Q t h ( s t h , a t h ) and glyph[lscript] 2 ,h ( y t h , f t ) = r t h + max a Q t h +1 ( s t h +1 , a ) . The secondary objective is to predict the value function optimistically, with glyph[lscript] 3 ( w t , f t ) = -max a Q ( s t 1 , a ) and w t = s t 1 .

Algorithm 5 Two-Timescale Exponential Weights (adapted from Agarwal and Zhang [2])

1: Initialize S 0 ← ∅ .

2: for t = 1 , 2 , . . . , T do

3: For all f, g ∈ G , define

4: Sample and predict f t ∼ p t .

5: Observe { x t h , y t h } h ∈ [ H ] , w t . and update S t ← S t -1 ⋃ { x t h , y t h } h ∈ [ H ] ⋃ { w t } .

To analyze Algorithm 5, we make a generalized realizability assumption and a generalized completeness assumption; these assumptions abstract away the notions of realizability and completeness in RL.

Assumption C.1 (Realizability for online learning) . There exists f ∗ ∈ G such that for all h ∈ [ H ] and x h ∈ X h , we have

Assumption C.2 (Completeness for online learning) . For any f ∈ G , there exists g ∈ G such that for all h ∈ [ H ] and x h ∈ X h , we have

For any f ∈ G , we denote the corresponding g ∈ G satisfying this property by g = T f .

For functions g and f and outcome x h , define

this quantity generalizes the notion of Bellman error for reinforcement learning. Recalling that ∆ t h ( f, g ) := glyph[lscript] h, 1 ( x t h , f ) -glyph[lscript] h, 2 ( y t h , g ) , it follows immediately

In addition, let us define and

The following result is the main theorem concerning the performance of Algorithm 5.

Theorem C.1. Let λ = 1 / 8 , η < 1 / (2 16 (log( |G| T/δ ) + 1)) , and 0 < β < 1 . Under Assumption C.1 and Assumption C.2, for any δ ∈ (0 , 1) , with probability at least 1 -δ ,

C.3.1 Proof of Theorem C.1

For our analysis, it will be useful to consider the following offset version of the loss:

We use x 1: H as a shorthand for { x h } h ∈ [ H ] and further define

Recall from Algorithm 5 that

Thus, we can verify the following relationships:

In what follows, we use E t -1 [ · ] to abbreviate E [ · | F t -1 ] , where F t -1 := σ ( S t -1 ) .

Proof of Theorem C.1. Under Assumption C.1, ( glyph[lscript] h, 1 ( x t h , f ∗ ) -E [ glyph[lscript] h, 2 ( y h , f ∗ ) | x t h ]) 2 = 0 . Thus, the first equality holds by definition as

To prove the result, we appeal to two technical lemmas, Lemma C.4 and Lemma C.5, stated below. Plugging the bound from Lemma C.5 into Lemma C.4, we have

Rearranging, we obtain

We now state the technical lemmas, Lemma C.4 and Lemma C.5, used in the proof above

Lemma C.4. Under Assumption C.1, for any 0 ≤ η ≤ 1 / 24 and 0 < β < 1 , we have

Lemma C.5. Let λ = 1 / 8 and η < 1 / (2 16 (log( |G| T/δ ) + 1)) . Under Assumption C.2, for any 0 < δ < 1 , with probability at least 1 -δ , we have

The remainder of the proof is organized as follows.

Appendix C.3.2 presents basic technical lemmas.

Appendix C.3.3 presents the proof of Lemma C.4.

Appendix C.3.4 presents the proof of Lemma C.5.

Appendices C.3.6 and C.3.7 contain additional technical lemmas used in the proof of Lemma C.5.

C.3.2 Basic properties

In this section, we present basic technical results that will be used within the proof of Lemma C.5 and Lemma C.4.

In this section, we will present some basic properties of δ t ( g, f ) , E ( g, f, x t 1: H ) 2 and Z t ( f ) for any t ∈ [ T ] , g, f ∈ G , x t 1: H ∈ ⋃ h ∈ [ H ] X h . These properties are mainly due to a sub-Gaussian term in the definition of δ t ( g, f ) . Recall

Note that the quantity E [ glyph[lscript] h, 2 ( y h , f ) | x t h ] -glyph[lscript] h, 2 ( y t h , f ) , conditioned on x t h , is a mean 0 random variable bounded by [ -1 , 1] . Thus, we have

Furthermore, since E [ glyph[lscript] h, 2 ( y h , f ) | x t h ] -glyph[lscript] h, 2 ( y t h , f ) is sub-Gaussian with variance proxy 1 / 2 by Hoeffding's inequality, we have the following three lemmas.

Lemma C.6. For any t ≤ T , g, f ∈ G and c ∈ R , we have

In addition, for any 0 < c < 1 2 , we have E [exp( -cδ t ( g, f )) | x t 1: H ] ≤ 1 .

Proof of Lemma C.6. By the 1 / 2 -sub-Gaussianity of E [ glyph[lscript] h, 2 ( y h , f ) | x t h ] -glyph[lscript] h, 2 ( y t h , f ) , we have E [exp( -cδ t ( g, f )) | x t ]

1: H

Lemma C.7. For any t ≤ T , g , f and x t 1: H , we have E [ δ t ( g, f ) 2 | x t 1: H ] ≤ 5 E [ E ( g, f, x t 1: H ) 2 | x t 1: H ] .

Proof of Lemma C.7. The result follows by writing

where the first equality is by defintion, the second equality is by expanding the terms and notice that the cross terms have zero mean, the first inequality is obtained using the fact that |E h ( g, f, x h ) | ≤ 1 and that the cross terms of the expansion of the second term have zero mean and the final inequality is by | E [ glyph[lscript] h, 2 ( y h , f ) | x t h ] -glyph[lscript] h, 2 ( y t h , f ) | ≤ 1 .

Lemma C.8. For all t ≤ T , as long as 0 ≤ λ ≤ 1 / 8 , we have that for all f almost surely. In particular, E t -1 [( Z t ( f ∗ )) | x t ] ≥ 0

Recalling that Z t ( f ) = -1 λ log E g ∼ q t ( ·| f ) exp( -λδ t ( g, f )) and δ t ( f ) = E g ∼ q t ( ·| f ) δ t ( g, f ) , we obtain the following relationship via Jensen's inequality and second order expansion of the exponential function.

LemmaC.9. For any t ≤ T and realization of S t -1 , we have that as long as λ < 1 / 8 , Z t ( f ) ≤ δ t ( f ) and | Z t ( f ) | ≤ 9 8 E g ∼ q t ( ·| f ) | δ t ( g, f ) | for all g, f ∈ G .

Proof of Lemma C.9. By Jensen's inequality,

On the other hand, applying Lemma C.2, we have

Thus

Another important fact we will use is that f ∗ satisfies δ t ( f ∗ , f ∗ ) = 0 . Furthermore, for any f ∈ G , the pair T f, f always has δ t ( T f, f ) = 0 for all t ∈ [ T ] .

Lemma C.10. For any t ≤ T , under Assumption C.1 we have δ t ( f ∗ , f ∗ ) = 0 . In addition, if Assumption C.2 is satisfied, then for any f , δ t ( T f, f ) = 0 .

Proof of Lemma C.10. By the definition of f ∗ ,

Likewise, by the definition of T f ,

C.3.3 Proof of Lemma C.4

In this section, we prove Lemma C.4, which gives a guarantee for the outer exponential weights update used within Algorithm 5.

Proof of Lemma C.4. The definition of the exponential weights update (in particular, (23)) implies that

where the final inequality holds due to the fact that E [ XYZ ] ≤ 3 √ E [ X 3 ] E [ Y 3 ] E [ Z 3 ] for positive random variables X,Y,Z (which in turn can be shown via a repeated application of Hölder's inequality). We further have

Meanwhile, using Lemma C.2, we get

Moreover, under Assumption C.1, the benchmark term is negative by

Furthermore, the log-exponential term is-up to a constant-bounded by its first order expansion:

By using the fact that | ι t ( f ) | ≤ 1 and combining the above displays, we obtain that

Rearranging yields the desired statement.

C.3.4 Proof of Lemma C.5

We state three technical lemmas, Lemmas C.11 to C.13, which are proven in subsequent subsections, then prove Lemma C.5 as a consequence.

Lemma C.11. Almost surely with respect to the draw of S T , we have

Lemma C.12. Under Assumption C.2, for any 0 < δ < 1 , with probability at least 1 -δ over the draw of S T , we have for all t ≤ T ,

Lemma C.13. For any t ≤ T , the following bound holds almost surely with respect to the draw of S t :

Proof of Lemma C.5. By Lemma C.11, we have

Thus under Assumption C.2, using the bound in Lemma C.12 for Term I and using the bound in Lemma C.13 for Term II, we have with probability at least 1 -δ ,

Reorganizing the terms, we have

Using the assumed bound on η and λ gives

C.3.5 Proof of Lemma C.11 (Error Decomposition)

Proof of Lemma C.11. Recalling the relationships q t +1 ( g | f ) = q t ( g | f ) · e -λ [ δ t ( g,f ) -Z t ( f )] and p t +1 ( f ) = p t ( f ) · e -η [ βι t ( f )+ δ t ( f,f ) -Z t ( f ) -Z t ] , the proof begins with the following manipulation:

Thus, taking expectation with respect to E t -1 [ · ] , dividing by λ , rearranging, summing over t on both sides, and taking advatnage of telescoping, we obtain

Finally, we use that δ t ( T f, f ) = 0 , log 1 q T +1 ( T f | f ) ≥ 0 , and log 1 q 1 ( T f | f ) = log |G| .

C.3.6 Proof of Lemma C.12 (Bound on Term I)

Toward proving Lemma C.12, we state and prove a series of technical lemmas, Lemmas C.14 to C.17.

Lemma C.14. Under Assumption C.2, for any 0 < δ < 1 , the event A t defined below holds simultaneously for all t ≤ T with probability at least 1 -δ over the draw of S T :

where the final inequality uses 0 < λ < 1 / 2 as well as the fact that for any fixed choice of f, g , conditioned on x s 1: H , s ≤ t , the random variables δ s ( g, f ) (for s ≤ t ) are independent. Then, by Markov's inequality and the union bound, we have the desired result.

Lemma C.15. For any t ≤ T , 0 ≤ λ ≤ 1 / 8 , f , almost surely with respect to the draw of S t and x t 1: H , we have

Proof of Lemma C.15. Using Lemma C.9, we get

Lemma C.16. For any t ≤ T , almost surely with respect to the draw of S t and x t 1: H , we have

Proof of Lemma C.16. Using Jensen's inequality, we have

Thus by Lemma C.7 and Lemma C.15, we have

Lemma C.17. For any 0 ≤ η ≤ 1 / 240 , f , almost surely with respect to the draw of S t and x t 1: H , we have

Proof of Lemma C.17. Jensen's inequality gives that

We may then write

where the second equality uses Jensen's inequality to pull the E f ′ ∼ p t [ · ] outside of the exponential and then the fact that a 1 · · · a 6 ≤ 1 6 ∑ 6 i =1 a 6 i for real numbers a 1 , . . . , a 6 . We control the six terms on the right hand side separately as follows:

Term (a): Using Lemma C.2 we get

Term (b): Using Lemma C.6 we get

Term (c):

Term (d): Using Lemma C.2, we have

Term (e):

Term (f):

Proof of Lemma C.12. Combining the preceding lemmas, we have with probability at least 1 -δ over the draw of S T , the event A t defined in Lemma C.14 holds for all t ≤ T simultaneously, and thus

where the first inequality is derived from Lemma C.17, and the second inequality follows from Lemma C.8 and Lemma C.14.

C.3.7 Proof of Lemma C.13 (Bound on Term II)

where the second inequality applies Lemma C.1 (together with the fact that η ∈ (0 , 1 / 240) ), the third inequality uses Young's inequality and the fact that | ι t ( f ) | ≤ 1 for all f , and the final inequality applies Lemma C.15 (to bound E t -1 [ | Z t ( f ) | 2 ] ), Lemma C.16 (to bound E t -1 [ | Z t | 2 ] ), and Lemma C.7 (to bound E t -1 [ δ t ( f, f ) | x t 1: H ] ) .

D Proofs from Section 2 and Section 3

D.1 Proofs from Section 2

Proof of Theorem 2.1 and Theorem B.1. We prove Theorem B.1; Theorem 2.1 is the special case of this result in which n = 1 . Observe that for Algorithm 3, we have

We can rewrite this sum as

For each step k , by the choice of p k , we have

Finally, we use that probability at least 1 -δ , OptEst D γ ≤ OptEst D γ ( K,n,δ ) .

We next prove Proposition 2.1, which gives a bound on

Algorithm 6 Optimistic Estimation for Bilinear Classes

1: parameters :

Number of rounds T , Batch size n

Learning rate η > 0 .

Discrepancy function glyph[lscript] est h ( Q ; z k,l h ) Exploration parameter γ > 0 .

2: Let K = T/n and B 0 := ∅ .

3: for k = 1 , 2 , · · · , K do

4: Form the randomized estimator µ k via µ k ( Q ) ∝ exp ( -η ∑ i<k glyph[lscript] i ( Q ) ) , where

5: Receive batch of samples B k = { ( π k,l , r k,l , o k,l ) } n l =1 where π k,l ∼ p k and ( r k,l , o k,l ) ∼ M glyph[star] ( π k,l ) .

// p k is the decision distribution produced by E2D.Opt (cf. Algorithm

6: Let z k,l h = ( s k,l h , a k,l h , r k,l h , s k,l h +1 ) , where we recall o k,l = ( s k,l 1 , a k, 1 1 , r k,l 1 ) , . . . , ( s k,l H , a k, 1 H , r k,l H )

3). .

Proof of Proposition 2.1. Throughout this proof, we use the batched estimation notation from Appendix B.1. Let n be fixed, and let K := T/n be the number of epochs. Recall that for each step k ∈ [ K ] , the estimation oracle is given a batch of examples B k = { ( π k,l , r k,l , o k,l ) } n l =1 where π k,l ∼ p k and ( r k,l , o k,l ) ∼ M glyph[star] ( π k,l ) . Each observation (trajectory) takes the form o k,l = ( s k,l 1 , a k,l 1 , r k,l 1 ) , . . . , ( s k,l H , a k,l H , r k,l H ) . Throughout the proof, we use the notation

where z k,l h := ( s k,l h , a k,l h , r k,l h , s k,l h +1 ) . Let η > 0 and α > 0 be parameters whose values will be chosen at the end of the proof. Defining

and abbreviate Q glyph[star] = Q Mglyph[star] ,glyph[star] .

Estimation algorithm. Define

Algorithm 6 chooses

as the randomized estimator for epoch k .

Estimation error bound. Let us abbreviate L = L bi ( Q ; M glyph[star] ) . Observe that for all k , | glyph[lscript] k ( Q ) | ≤ R := HL 2 + α almost surely. Hence, Lemma C.3 implies that as long as η ≤ 1 / 2 R , we have

For each k ∈ [ K ] , we have that for all Q ∈ Q ,

As a result, (25) implies that

Whenever η ≤ 1 16 HL 2 , rearranging gives

We now appeal to the following lemma.

Lemma D.1. With probability at least 1 -δ , it holds that for all k ∈ [ K ] , h ∈ [ H ] , and Q ∈ Q

,

where ε conc ( n ) := L √ 2 log( |Q| KH/δ ) n .

Going forward, we condition on the event in Lemma D.1. Applying the inequality (30) within (29), we have

where the last equality uses that E k h ( Q glyph[star] ) = 0 for all h ∈ [ H ] , and k ∈ [ K ] , which is a consequence of Definition 2.1. Next, we recall that

A standard application of Freedman's inequality (c.f Lemma A.3 in Foster et al. [12]) implies that with probability at least 1 -δ ,

Putting everything together, we have that

Choosing α = 1 8 γ , this gives

We choose η = √ log |Q| α 2 K ∧ 1 16 R , which satisfies the constraints described earlier in the proof, and gives

Proof of Lemma D.1. For any fixed k ∈ [ K ] and Q ∈ Q , Hoeffding's inequality implies that with probability at least 1 -δ ,

By a standard union bound it follows that with probability at least 1 -δ , for all k ∈ [ K ] , h ∈ [ H ] , and Q ∈ Q simultaneously,

Whenever this event occurs, the AM-GM inequality implies that

Proof of Proposition 2.2. Let µ ∈ ∆( Q ) be fixed. Fix α ∈ (0 , 1) , and let π α Q be the randomized policy that-for each h -independently plays π Q,h with probability 1 -α/H and π est Q,h with probability α/H . Let p ∈ ∆(Π) be the distribution induced by sampling Q ∈ µ and playing π α Q . Translated to our notation, Foster et al. [12] (cf. Proof of Theorem 7.1, Eq. (152)) shows that for all MDPs M , this strategy guarantees that for all η > 0 ,

In the on-policy case in which π est Q = π Q , it suffices to set α = 0 , which gives

where the last equality relabels Q ′ ← Q . Setting η = 2 γ yields the result.

glyph[negationslash]

In the general case where π est Q = π Q , we have that for all Q ′ ∈ Q , 9

We have α H (1 -α/H ) H -1 ≥ α 2 H whenever α ≤ 1 / 2 , which gives

where the last line chooses η = γα/H . To conclude, we set α = √ H 2 d bi ( Q ; M ) / 4 γ , which is admissible whenever γ ≥ H 2 d bi ( Q ; M ) .

Proof of Corollary 2.1. In both of the cases in the theorem statement (on-policy and off-policy), whenever γ ≥ 1 and n ≤ √ T (to ensure that K/n ≥ 1 ), combining Theorem B.1 and Proposition 2.1 gives

We choose n = √ T , which is admissible whenever T is sufficiently large, and gives

In the on-policy case, we have, from Proposition 2.2, that

and in the off-policy case, we have

Choosing γ to balance yields the result.

D.2 Proofs from Section 3

Proof of Proposition 3.1. We begin with the upper bound. First, note that by Assumption 3.1 and the AM-GM inequality, we have

Consider any fixed choice for µ ∈ ∆( M ) . By Sion's minimax theorem (see Foster et al. [12] for details), we have

so that the main term in (33) is equal to

Relabeling, this is equal to

We now prove the lower bound. Using Assumption 3.1 and the AM-GM inequality once more, we have

Using the minimax theorem in the same fashion as before, the main term in (34) is equal to

Relabeling, this is equal to

Proof of Proposition 3.2. Let M be arbitrary. By Sion's minimax theorem (see Foster et al. [12] for details), we have

By the assumed triangle inequality for D , we have that for all π ∈ Π ,

Applying this bound above, we have that

Proof of Proposition 3.3. The upper bound on o-dec bi γ ( M ) follows from Proposition 2.2, so it remains to prove the lower bound on dec bi γ ( M ) .

For a model M , define the Bellman operator T M h via

Let

and D π sbe ( M ‖ M ) := D π sbe ( Q M ,glyph[star] ‖ M ) . By Jensen's inequality, it suffices to lower bound dec D sbe γ ( M ) .

Let S = { s , t } and A = { a , b } . We consider a sub-family M ′ ⊂ M of deterministic combination lock MDPs parameterized by glyph[vector] a = ( glyph[vector] a 1 , . . . , glyph[vector] a H ) ∈ A H , with M glyph[vector] a defined as follows.

The initial state is s 1 = s .

glyph[negationslash]

For each h = 1 , . . . , H -1 , if s h = s , then selecting a h = glyph[vector] a h transitions to s h +1 = s , and selecting a h = glyph[vector] a h transitions to s h +1 = t . t is a self-looping terminal state: if s h = t , then s h +1 = t regardless of the action taken.

If s H = s and a H = glyph[vector] a H , then r H = ∆ > 0 ; all other state-action tuples have zero reward.

We choose M such that Q M ,glyph[star] h ( s, a ) = 0 for all ( h, s, a ) . We calculate that for all glyph[vector] a ∈ A H , and all policies π ∈ Π RNS , glyph[negationslash]

It follows that

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

It is straightforward to see by induction that for all π ∈ Π RNS , E glyph[vector] a ∼ Unif( A H ) [ P Mglyph[vector] a ,π ( a 1: H = glyph[vector] a )] ≤ 2 -H and E glyph[vector] a ∼ Unif( A H ) [ P Mglyph[vector] a ,π ( a 1: H = glyph[vector] a )] ≥ 1 2 , so we have

The result follows by choosing ∆ appropriately.

Proof of Proposition 3.4. Let H be given, and assume without loss of generality that S = 2 H -1 + 2 H -2 . Let N := 2 H -2 . We construct a family of MDPs M = { M 1 , . . . , M N } with deterministic rewards and transitions as follows.

A = { a , b } .

The state space is S = T ∪ E ∪ { s } . s is a deterministic initial state which appears only in layer 1 . T represents a depth H -2 binary tree (with the root at layer h = 2 ), which has N = 2 H -2 leaf states in layer H , labeled by { 1 , . . . , N } , and 2 H -1 -1 states in total. E = { N + 1 , . . . , 2 N } is an auxiliary collection of self-looping terminal states, each of which can appears in layers h = 2 , . . . , H . In total, we have |S| = 2 H -1 +2 H -2 .

The dynamics and rewards for MDP M i are as follows.

At layer h = 1 , s 1 = s deterministically.

If a 1 = a , we transition to s 2 = N + i ∈ E , and if a 1 = b , we transition to the root state for the tree T . We receive no reward.

For h ≥ 2 , all states s ∈ E are self-looping and have no reward (i.e. s h +1 = s h if s h ∈ E for h > 1) .

For h ≥ 2 , states in T follow a standard deterministic binary tree structure (e.g., [24, 8]). Beginning from the root node at h = 2 , action a transitions to the left successor, while action b transitions to the right successor. For M i , we receive reward 1 for reaching the leaf node i ∈ T at layer H , and receive zero reward for all other states. Note that the transition probabilities for this portion of the MDP do not depend on i .

Online estimation. We first construct an online estimation algorithm for the class M . Recall that we adopt Hellinger distance, given by D π H ( ̂ M,M ) = D 2 H ( ̂ M ( π ) , M ( π ) ) .

We first note that since all M ∈ M have f M ( π M ) = 1 , the optimistic estimation error is equal to the (non-optimistic) estimation error

Observe that since all MDPs M ∈ M have deterministic rewards and transitions, there exists a function o M ( π ) such that o ∼ M ( π ) has o = o M ( π ) almost surely. It follows that glyph[negationslash]

for all M,M ′ ∈ M , so we have

For the estimation algorithm, we choose

glyph[negationslash]

Lemma C.3 implies that with probability 1 , the sequence π 1 , . . . , π T satisfies glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

Since o Mglyph[star] ( π t ) = o t , rearranging yields

glyph[negationslash]

glyph[negationslash]

From here, a standard application of Freedman's inequality [16] implies that with probability at least 1 -δ , ∑ T t =1 E π t ∼ p t E ̂ M t ∼ µ t [ I { o ̂ Mt ( π t ) = o t } ] glyph[lessorsimilar] log( |M| /δ ) glyph[lessorsimilar] log( S/δ ) .

Lower bound for posterior sampling. Observe that for all M i ∈ M , the unique optimal policy has π Mi ( s ) = b at h = 1 . Thus, the posterior sampling algorithm, which chooses p t ( π ) = µ t ( { M ∈ M | π M = π } ) , will never play a 1 = a , and will never encounter states in E . As a result, the problem is equivalent (for this algorithm) to a multi-armed bandit problem with N arms and noiseless binary rewards, which requires E [ Reg DM ] glyph[greaterorsimilar] N glyph[greaterorsimilar] S in the worst-case [21].

Upper bound for Algorithm 2. We first bound the Optimistic DEC for M . For any µ ∈ ∆( M ) , we can write glyph[negationslash]

glyph[negationslash]

where the last equality uses that f M ( π M ) = 1 for all M ∈ M . We choose p = (1 -ε ) µ ( { M ∈ M | π M = ·} ) + επ a , where π a is the policy that plays action a deterministically.

Now, let M ∈ M be fixed. Since each model M i transitions to s 2 = N + i deterministically when a 1 = a , we glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

and

Similarly, we have

By choosing ε = γ -1 , which is admissible whenever γ ≥ 1 , we have glyph[negationslash]

This establishes that

for all γ ≥ 1 . A regret bound of the form E [ Reg DM ] ≤ ˜ O ( √ T log( S )) now follows by invoking Theorem 2.1 with the estimation guarantee in the prequel and choosing γ appropriately.

glyph[negationslash]

Peer reviews:
comments: SUMMARY:
The paper studies model-free RL algorithm using DEC as the complexity measure.

STRENGTHS:
Extending previous model-based algorithm to model-free algorithm is a nice contribution.

WEAKNESSES:
I believe this paper provides a nice contribution for the DEC family of papers. But I would like the authors to discuss more on the implication side. Why we care about extending model-free algorithms to bilinear classes? There are too many complexity measures for RL function approximation that are proposed recently so I am concerned about how useful those theories are for practice. 

The algorithm for Bellman-Eluder dimension is model-free. Then can I ask why we need a new algorithm? 

I hope the authors could pay more attention to the computational side. So far, all the lines of DEC works are not computationally efficient and even lack any experiments.

QUESTIONS:
See above.

LIMITATIONS:
1. No discussion or solution on how to implement the algorithm.
2. The analysis seems to build on two existing works: Zhang [32] and Foster et al. [13]. Not very exciting about the technique.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a variant of the recently proposed framework of Foster et al. for sequential decision making, based on a concept called \"decision estimation coefficient\" (DEC). The variant proposed here is based on enhancing the estimation step in the \"estimation-to-decisions\" (E2D) with an optimistic bias inspired by the recent work of Zhang. The authors show that this biased estimation scheme, coupled with an appropriately adjusted decision-making rule, can satisfy very similar regret guarantees for a wide range of sequential decision problems, and can provide improvements in certain settings. In particular, the authors show that their approach can handle a large class of tractable models for reinforcement learning called \"bilinear classes\".\n\nSTRENGTHS:\nThe paper is well-written and the technical content is of excellent quality. The proposed approach is justified and explained well, and its analysis is also presented in an accessible way (at least on a high level). While the regret bounds for bilinear MDP classes is not novel in the sense that there exist other algorithms that achieve the same guarantees, I appreciate the conceptual contribution of showing that the DEC framework is also capable of tackling these (relatively) challenging problems.\n\nI appreciated the very careful comparison between all the relevant DEC variants in Section 3: the authors didn't just propose a technique and proved some bounds about it, but also explored a range of other opportunities and explained the differences between them in an accessible manner.\n\nWEAKNESSES:\nOn the negative side, the rates that the authors derive are not particularly great: the scaling goes from $T^{2/3}$  in the setting with the most stringent assumptions all the way to $T^{5/6}$ as more and more assumptions are dropped. While the authors discuss this limitation quite openly, I would have appreciated some more discussion as to where this relatively poor scaling comes from. One contributing factor is certainly the use of batched estimation steps. My understanding is that some further looseness may come from the optimistic bonuses added to the estimation procedure, which makes the total estimation error grow polynomially with the number of updates (as opposed to logarithmically, which would allow getting sqrt{T} rates after putting everything together). I wonder though if this intuition is correct, and I would appreciate it if the authors could clarify what rate they would get if they could afford to set n=1 (without paying for it). Altogether, it would have been nice if the to compare the various notions of estimation error with the same care as what the DEC variants have received.\n\nOverall, I am leaning towards suggesting acceptance, but I would feel more strongly about my support if the authors were able to address my questions above in a satisfying way.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes to a line of research on decision-estimation coefficients (and related algorithms). An optimistic variant of the original DEC is introduced and it is shown that this variant can lead to a new related meta-algorithm with optimism. This new structure and algorithm are shown to provide non-trivial regret bounds for model-free RL for bilinear classes (a known framework for sample efficient learning). It is also shown that the original DEC is insufficient to achieve the same. Finally it is shown that posterior sampling does not solve all the problems that E2D.Opt can solve.\n\nSTRENGTHS:\nOverall this is a good paper that is presented clearly and makes an interesting contribution to a growing line of research. The application of E2D-style algorithms to model-free settings is important, as well as the ability to incorporate classes that were previously not handled.\n\nThe discussion of where advantages exist for optimistic DEC are thorough and helpful. The additional results also shed light on the problems that can be handled by both coefficients and other algorithms.\n\nWEAKNESSES:\nThe new coefficient is used to reproduce an existing result. This leads to a natural new meta-algorithm and theorem, but since they are only shown in one specialized case, it’s unclear whether it’s useful beyond this one setting as a “meta-algorithm.” While it is apparent that one could swap out different divergences and oracles, it may be described better as a single algorithm, inspired by the original E2D.\n\nThe regret bound appears to have a slightly worse rate than what would be achievable with the original BiLin-UCB, which I believe has T^{2 / 3} if converted properly to regret.\n\nIt’s not clear how one would implement such an algorithm in practice.\n\nQUESTIONS:\nDoes optimistic DEC subsume all of the original DEC results? Are there any downsides compared to the original E2D?\n\nWhat benefit does the batching offer if the regret bounds still scale as poly(K)? Why not just use one sample per batch?\n\nProp 3.4: shows that PS does not cover all the problems that E2D.Opt can solve. How about the other way?\n\nProp 3.4: What is the expectation over? Is this an average regret over MDPs in the class and if so, is posterior sampling misspecified? \n\nSuggestions:\n\n(14) does not seem to point to an equation in the main paper.\n\nLines 94-95: It may be more clear to say that “there exist algorithms/oracles” that do this.\n\nLIMITATIONS:\nNo, but this work is theoretical. The limitations are either apparent from the assumptions or discussed adequately. A conclusion to summarize would be helpful though.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-27"}

comments: SUMMARY:
This paper adapted the DEC framework and E2D algorithm to incorporate model-free reinforcement learning. Specifically, this paper proposed the optimistic DEC and proved that it appears in the regret bound for the optimistic E2D algorithm with an optimistic estimation oracle.  The example of bilinear class is provided. The paper discussed when the optimistic E2D algorithm will have benefit than E2D in its original form.

STRENGTHS:
This paper is well-written and provides concrete steps toward understanding the general framework of DEC. The original DEC framework is mostly model-based decision making, whereas the new framework can handle model-free RL.

WEAKNESSES:
This paper is largely built upon [13] and [32]. The additional results over [13] and [32] are a bit incremental. While using the DEC framework to provide a regret bound for bilinear class is nice, this does not new/improved regret bound for bilinear class. 

In Appendix A, the authors discussed the relationship with constrained DEC [16], and mentioned that it would be interesting to explore whether optimistic estimation can be combined with the constrained DEC techniques. To make this paper stronger, one possible direction is to discuss the relationship with constrained DEC in details (either provide some results in this direction, or point out the technical difficulty to the combination).

QUESTIONS:
What is the general picture of the lower bound side of optimistic DEC? Am I correct that oDEC is always smaller than DEC, and DEC serves as a lower bound, so it is not meaningful to discuss oDEC as a lower bound?

LIMITATIONS:
na
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-31