Title: Model-Free Active Exploration in Reinforcement Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: reinforcement learning; best policy identification; model free; exploration; sample complexity
Abstract: We study the problem of exploration in Reinforcement Learning and present a novel model-free solution. We adopt an information-theoretical viewpoint and start from the  instance-specific lower bound of the number of samples that have to be collected to identify a nearly-optimal policy. Deriving this lower bound along with the optimal exploration strategy entails solving an intricate optimization problem and requires a model of the system. In turn, most existing sample optimal exploration algorithms rely on estimating the model. We derive an approximation of the instance-specific lower bound that only involves quantities that can be inferred using model-free approaches. Leveraging this approximation, we devise an ensemble-based model-free exploration strategy  applicable to both tabular and continuous Markov decision processes. Numerical results demonstrate that our strategy is able to identify efficient policies faster than state-of-the-art exploration approaches.

Model-Free Active Exploration in Reinforcement Learning

Abstract

We study the problem of exploration in Reinforcement Learning and present a novel model-free solution. We adopt an information-theoretical viewpoint and start from the instance-specific lower bound of the number of samples that have to be collected to identify a nearly-optimal policy. Deriving this lower bound along with the optimal exploration strategy entails solving an intricate optimization problem and requires a model of the system. In turn, most existing sample optimal exploration algorithms rely on estimating the model. We derive an approximation of the instance-specific lower bound that only involves quantities that can be inferred using model-free approaches. Leveraging this approximation, we devise an ensemble-based model-free exploration strategy applicable to both tabular and continuous Markov decision processes. Numerical results demonstrate that our strategy is able to identify efficient policies faster than state-of-the-art exploration approaches.

1 Introduction

Efficient exploration remains a major challenge for reinforcement learning (RL) algorithms. Over the last two decades, several exploration strategies have been proposed in the literature, often designed with the aim of minimizing regret. These include model-based approaches such as Posterior Sampling for RL [36](PSRL) and Upper Confidence Bounds for RL [4, 25, 2](UCRL), along with model-free UCB-like methods [19, 56]. Regret minimization is a relevant objective when one cares about the rewards accumulated during the learning phase. Nevertheless, an often more important objective is to devise strategies that explore the environment so as to learn efficient policies using the fewest number of samples [16]. Such an objective, referred to as Best Policy Identification (BPI), has been investigated in simplistic Multi-Armed Bandit problems [16, 21] and more recently in tabular MDPs [28, 29]. For these problems, tight instance-specific sample complexity lower bounds are known, as well as model-based algorithms approaching these limits. However, model-based approaches may be computationally expensive or infeasible to obtain. In this paper, we investigate whether we can adapt the design of these algorithms so that they become model-free and hence more practical.

Inspired by [28, 29], we adopt an information-theoretical approach, and design our algorithms starting from an instance-specific lower bound on the sample complexity of learning a nearly-optimal policy in a Markov decision process (MDP). This lower bound is the value of an optimization problem, referred to as the lower bound problem, whose solution dictates the optimal exploration strategy in an environment. Algorithms designed on this instance-specific lower bound, rather than minimax bounds, result in truly adaptive methods, capable of tailoring their exploration strategy according to the specific MDP's learning difficulty. Our method estimates the solution to the lower bound problem and employs it as our exploration strategy. However, we face two major challenges: (1) the

Code repository: https://github.com/rssalessio/ModelFreeActiveExplorationRL

lower bound problem is non-convex and often intractable; (2) this lower bound problem depends on the initially unknown MDP. In [29], the authors propose MDP-NAS, a model-based algorithm that explores according to the estimated MDP. They convexify the lower bound problem and explore according to the solution of the resulting simplified problem. However, this latter problem still has a complicated dependency on the MDP. Moreover, extending MDP-NAS to large MDPs is challenging since it requires an estimate of the model, and the capability to perform policy iteration. Additionally, MDP-NAS employs a forced exploration technique to ensure that the parametric uncertainty (the uncertainty about the true underlying MDP) diminishes over time - a method, as we argue later, that we believe not to be efficient in handling this uncertainty.

We propose an alternative way to approximate the lower bound problem, so that its solution can be learnt via a model-free approach. This solution depends only on the Q -function and the variance of the value function. Both quantities can advantageously be inferred using classical stochastic approximation methods. To handle the parametric uncertainty, we propose an ensemble-based method using a bootstrapping technique. This technique is inspired by posterior sampling and allows us to quantify the uncertainty when estimating the Q -function and the variance of the value function.

Our contributions are as follows: (1) we shed light on the role of the instance-specific quantities needed to drive exploration in uncertain MDPs; (2) we derive an alternate upper bound of the lower bound problem that in turn can be approximated using quantities that can be learned in a model-free manner. We then evaluate the quality of this approximation on various environments: ( i ) a random MDP, ( ii ) the Riverswim environment [47], and ( iii ) the Forked Riverswim environment (a novel environment with high sample complexity); (3) based on this approximation, we present Model Free Best Policy Identification (MF-BPI), a model-free exploration algorithm for tabular and continuous MDPs. For the tabular MDPs, we test the performance of MF-BPI on the Riverswim and the Forked Riverswim environments, and compare it to that of Q-UCB [19, 56], PSRL[36], and MDP-NAS[29]. For continuous state-spaces, we compare our algorithm to IDS[33] and BSP [39] (Boostrapped DQN with randomized prior value functions) and assess their performance on hard-exploration problems from the DeepMind BSuite [41] (the DeepSea and the Cartpole swingup problems).

2 Related Work

The body of work related to exploration methods in RL problems is vast, and we mainly focus on online discounted MDPs (for the generative setting, refer to the analysis presented in [17, 28]). Exploration strategies in RL often draw inspiration from the approaches used in multi-armed bandit problems [26, 49], including ϵ -greedy exploration, Boltzmann exploration [57, 49, 26, 1], or more advanced procedures, such as Upper-Confidence Bounds (UCB) methods [2, 3, 26] or Bayesian procedures [52, 58, 14, 44]. We first discuss tabular MDPs, and then extend the discussion to the case of RL with function approximation.

Exploration in tabular MDPs. Numerous algorithms have been proposed with the aim of matching the PAC sample complexity minimax lower bound ˜ Ω ( | S || A | ε 2 (1 -γ ) 3 ) [25]. In the design of these algorithms, model-free approaches typically rely on a UCB-like exploration [2, 26], whereas modelbased methods leverage estimates of the MDP to drive the exploration. Some well-known model-free algorithms are MEDIAN-PAC [42], DELAYED Q-LEARNING [48] and Q-UCB [56, 19]. Some notable model-based algorithms include: DEL [34], an algorithm that achieves asymptotically optimal instance-dependent regret; UCRL [25], an algorithm that uses extended value-iteration to compute an optimistic MDP; PSRL [36], that uses posterior sampling to sample an MDP. Other algorithms include MBIE [47], E3 [22], R-MAX [9, 20], and MORMAX [50]. Most of existing algorithms are designed towards regret minimization. Recently, however, there has been a growing interest towards exploration strategies with minimal sample complexity, see e.g. [60, 28]. In [28, 29], the authors showed that computing an exploration strategy with minimal sample complexity requires to solve a non-convex problem. To overcome this challenge, they derived a tractable approximation of the lower bound problem, whose solution provides an efficient exploration policy under the generative model [28] and the forward model [29]. This policy necessitates an estimate of the model, and includes a forced exploration phase (an ϵ -soft policy to guarantee that all state-action pairs are visited infinitely often). In [51], the above procedure is extended to linear MDPs, but there again, computing an optimal exploration strategy remains challenging. On a side note, in [55], the authors provide an

alternative bound in the tabular case for episodic MDPs, and later extend it to linear MDPs [54]. The episodic setting is further explored in [53] for deterministic MDPs.

Exploration in Deep Reinforcement Learning (DRL). Exploration methods in DRL environments face several challenges, related to the fact that the state-action spaces are often continuous, and other issues related to training deep neural architectures [46]. The main issue in these large MDPs is that good exploration becomes extremely hard when either the reward is sparse/delayed or the observations contain distracting features [10, 59]. Numerous heuristics have been proposed to tackle these challenges, such as (1) adding an entropy term to the optimization problem to encourage the policy to be more randomized [31, 18] or (2) injecting noise in the observations/parameters [15, 43]. More generally, exploration techniques generally fall into two categories: uncertaintybased and intrinsic-motivation-based [59, 24]. Uncertainty-based methods decouple the uncertainty into parametric and aleatoric uncertainty. Parametric uncertainty [14, 32, 23, 59] quantifies the uncertainty in the parameters of the state-action value. This uncertainty vanishes as the agent explores and learns. The aleatoric uncertainty accounts for the inherent randomness of the environment and of the policy [32, 23, 59]. Various methods have been proposed to address the parametric uncertainty, including UCB-like mechanisms [11, 59], or TS-like (Thompson Sampling) techniques [38, 36, 5, 35, 37, 40]. However, computing a posterior of the Q -values is a difficult task. For instance, Bayesian DQN [5] extends Randomized Least-Squares Value Iteration (RLSVI) [38] by considering the features prior to the output layer of the deepQ network as a fixed feature vector, in order to recast the problem as a linear MDP. Non-parametric posterior sampling methods include Bootstrapped DQN (and Bootstrapped DQN with prior functions) [37, 39, 40], which maintains several independent Q -value functions and randomly samples one of them to explore the environment. Bootstrapped DQN was extended in various ways by integrating other techniques [6, 27]. For the sake of brevity, we refer the reader to the survey in [59] for an exhaustive list of algorithms. Most of these algorithms do not directly account for aleatoric uncertainty in the value function. This uncertainty is usually estimated using methods like Distributional RL [8, 13, 30]. Well-known exploration methods that account for both aleatoric and epistemic uncertainties include Double Uncertain Value Network (DUVN) [32] and Information Directed Sampling (IDS) [23, 33]. The former uses Bayesian dropout to measure the epistemic uncertainty, and the latter uses distributional RL [8] to estimate the variance of the returns. In addition, IDS uses bootstrapped DQN to estimate the parametric uncertainty in the form of a bound on the estimate of the suboptimality gaps. These uncertainties are then combined to compute an exploration strategy. Similarly, in [12], the authors propose UA-DQN, an approach that uses QR-DQN [13] to learn the parametric and aleatoric uncertainties from the quantile networks. Lastly, we refer the reader to [59, 45, 7] for the class of intrinsic-motivation-based methods.

3 Preliminaries

Markov Decision Process. We consider an infinite-horizon discounted Markov Decision Process (MDP), defined by the tuple ϕ = ( S, A, P, q, γ, p 0 ) . S is the state space, A is the action space, P : S × A → ∆( S ) is the distribution over the next state given a state-action pair ( s, a ) , q : S × A → ∆([0 , 1]) is the distribution of the collected reward (with support in [0 , 1] ), γ ∈ [0 , 1) is the discount factor and p 0 is the distribution over the initial state.

Let π : S → ∆( A ) be a stationary Markovian policy that maps a state to a distribution over actions, and denote by r ( s, a ) = E r ∼ q ( ·| s,a ) [ r ] the average reward collected when an action a is chosen in state s . We denote by V π ( s ) = E π ϕ [ ∑ t ≥ 0 γ t r ( s t , a t ) | s 0 = s ] the discounted value of policy π . We denote by π ⋆ an optimal stationary policy: for any s ∈ S , π ⋆ ( s ) ∈ arg max π V π ( s ) and define V ⋆ ( s ) = max π V π ( s ) . For the sake of simplicity, we assume that the MDP has a unique optimal policy (we extend our results to more general MDPs in the appendix). We further define Π ⋆ ε ( ϕ ) = { π : ∥ V π -V π ⋆ ∥ ∞ ≤ ε } , the set of ε -optimal policies in ϕ for ε ≥ 0 . Finally, to avoid technicalities, we assume (as in [29]) that the MDP ϕ is communicating (that is, for every pair of states ( s, s ′ ) , there exists a deterministic policy π such that state s ′ is accessible from state s using π ).

̸

We denote by Q π ( s, a ) := r ( s, a ) + γ E s ′ ∼ P ( ·| s,a ) [ V π ( s ′ )] the Q -function of π in state ( s, a ) . We also define the sub-optimality gap of action a in state s to be ∆( s, a ) := Q ⋆ ( s, π ⋆ ( s )) -Q ⋆ ( s, a ) , where Q ⋆ is the Q -function of π ⋆ , and let ∆ min := min s,a = π ⋆ ( s ) ∆( s, a ) be the minimum gap in ϕ . For some policy π , we define Var sa [ V π ] := Var s ′ ∼ P ( ·| s,a ) [ V π ( s ′ )] to be the variance of the value function V π in the next state after taking action a in state s . More generally, we define M k sa [ V π ] :=

-next state after taking action a in state s . We also let MD sa [ V π ] := ∥ V π -E s ′ ∼ P ( ·| s,a ) [ V π ] ∥ ∞ be the span of ϕ under π , i.e. , the maximum deviation from the mean of the next state value after taking action a in state s .

Best policy identification and sample complexity lower bounds. The MDP ϕ is initially unknown, and we are interested in the scenario where the agent interacts sequentially with ϕ . In each round t ∈ N , the agent selects an action a t and observes the next state and the reward ( s t +1 , r t ) : s t +1 ∼ P ( ·| s t , a t ) and r t ∼ q ( ·| s t , a t ) . The objective of the agent is to learn a policy in Π ⋆ ε ( ϕ ) (possibly π ⋆ ) as fast as possible. This objective is often formalized in a PAC framework where the learner has to stop interacting with the MDP when she can output an ε -optimal policy with probability at least 1 -δ . In this formalism, the learner strategy consists of (i) a sampling rule or exploration strategy; (ii) a stopping time τ ; (iii) an estimated optimal policy ˆ π . The strategy is called ( ε, δ ) -PAC if it stops almost surely, and P ϕ [ˆ π ∈ Π ⋆ ε ( ϕ )] ≥ 1 -δ . Interestingly, one may derive instance-specific lower bounds of the sample complexity E ϕ [ τ ] of any ( ε, δ ) -PAC algorithm [28, 29], which involves computing an optimal allocation vector ω opt ∈ ∆( S × A ) (where ∆( S × A ) is the set of distributions over S × A ) that specifies the proportion of times an agent needs to sample each pair ( s, a ) to confidently identify the optimal policy:

and ω opt = arg inf ω ∈ Ω( ϕ ) T ε ( ω ) -1 . Here, Alt ε ( ϕ ) is the set of confusing MDPs ψ such that the ε -optimal policies of ϕ are not ε -optimal in ψ , i.e., Alt ε ( ϕ ) := { ψ : ϕ ≪ ψ, Π ⋆ ε ( ϕ ) ∩ Π ⋆ ε ( ψ ) = ∅} . In this definition, if the next state and reward distributions under ψ are P ′ ( s, a ) and q ′ ( s, a ) , we write ϕ ≪ ψ if for all ( s, a ) the distributions of the next state and of the rewards satisfy P ( s, a ) ≪ P ′ ( s, a ) and q ( s, a ) ≪ q ′ ( s, a ) .We further let KL ϕ | ψ ( s, a ) := KL( P ( s, a ) , P ′ ( s, a )) + KL( q ( s, a ) , q ′ ( s, a )) . Ω( ϕ ) is the set of possible allocations; in the generative case it is ∆( S × A ) , while with navigation constraints we have Ω( ϕ ) := { ω ∈ ∆( S × A ) : ω ( s ) = ∑ s ′ ,a ′ P ( s | s ′ , a ′ ) ω ( s ′ , a ′ ) } , ∀ s ∈ S } , with ω ( s ) := ∑ a ω ( s, a ) . Finally, kl( a, b ) is the KL-divergence between two Bernoulli distributions of means a and b .

4 Towards Efficient Exploration Allocations

We aim to extend previous studies on best policy identification to online model-free exploration. In this section, we derive an approximation to the bound proposed in [28], involving quantities learnable via stochastic approximation, thereby enabling the use of model-free approaches.

The optimization problem (1) leading to instance-specific sample complexity lower bounds has an important interpretation [28, 29]. An allocation ω opt corresponds to an exploration strategy with minimal sample complexity. To devise an efficient exploration strategy, one could then think of estimating the MDP ϕ , and solving (1) for this estimated MDP to get an approximation of ω opt . There are two important challenges towards applying this approach:

(i) Estimating the model can be difficult, especially for MDPs with large state and action spaces, and arguably, a model-free method would be preferable.

(ii) The lower bound problem (1) is, in general, non-convex [28, 29].

A simple way to circumvent issue (ii) involves deriving an upper bound of the value of the sample complexity lower bound problem (1). Specifically, one may derive an upper bound U ( ω ) of T ε ( ω ) by convexifying the corresponding optimization problem. The exploration strategy can then be the ω ⋆ that achieves the infimum of U ( ω ) . This approach ensures that we identify an approximately optimal policy, at the cost of over-exploring at a rate corresponding to the gap U ( ω ⋆ ) -T ε ( ω opt ) . Note that using a lower bound of T ε ( ω ) would not guarantee the identification of an optimal policy, since we would explore "less" than required. The aforementioned approach was already used in [28] where the authors derive an explicit upper bound U 0 ( ω ) of T 0 ( ω ) . We also apply it, but derive an upper bound such that implementing the corresponding allocation ω ⋆ can be done in a model-free manner (hence solving the first issue (i)).

4.1 Upper bounds on T ε ( ω )

The next theorem presents the upper bound derived in [28].

Theorem 4.1 ([28]) . Consider a communicating MDP ϕ with a unique optimal policy π ⋆ . For all vectors ω ∈ ∆( S × A ) ,

̸

with

In the upper bound presented in this theorem, the following quantities characterize the hardness of learning the optimal policy: ∆( s, a ) represents the difficulty of learning that in state s action a is sub-optimal; the variance Var sa [ V ⋆ ] measures the aleatoric uncertainty in future state values; and the span MD sa [ V ⋆ ] of the optimal value function can be seen as another measure of aleatoric uncertainty, large whenever there is a significant variability in the value for the possible next states.

Estimating the span MD sa [ V ⋆ ] , in an online setting, is a challenging task for large MDPs. Our objective is to derive an alternative upper bound that, in turn, can be approximated using quantities that can be learned in a model-free manner. We observe that the variance of the value function, and more generally its moments M k sa [ V ⋆ ] 2 -k for k ≥ 1 (see Appendix C), are smaller than the span. By refining the proof techniques used in [28], we derive the following alternative upper bound.

Theorem 4.2. Let ε ≥ 0 and let k ( s, a ) := arg sup k ∈ N M k sa [ V ⋆ ] 2 -k (for brevity, we write k instead of k ( s, a ) ). Then, ∀ ω ∈ ∆( S × A ) , we have T ε ( ω ) ≤ U ( ω ) , with

̸

where C ( s ′ ) = max ( 4 , 16 γ 2 φ 2 M k s ′ ,π ⋆ ( s ′ ) [ V ⋆ ] 2 1 -k ) and φ is the golden ratio.

We can observe that in the worst case, the upper bound U ( ω ⋆ ) of the sample complexity lower bound, with ω ⋆ = arg inf ω U ( ω ) , scales as O ( | S || A | max s MD s,π ⋆ ( s ) [ V ⋆ ] 2 ∆ 2 min (1 -γ ) 2 ) . Since MD sa [ V ⋆ ] ≤ (1 -γ ) -1 , then U ( ω ⋆ ) scales at most as O ( | S || A | ∆ 2 min (1 -γ ) 4 ) . However, the following questions arise: (1) Can we select a single value of k that provides a good approximation across all states and actions? (2) How much does this bound improve on that of Theorem 4.1? As we illustrate in the example presented in the next subsection, we believe that actually selecting k = 1 for all states and actions leads to sufficiently good results. With this choice, we obtain the following approximation:

̸

where C ′ ( s ′ ) = max ( 4 , 16 γ 2 φ 2 Var s ′ ,π ⋆ ( s ′ ) [ V ⋆ ] ) . U 1 ( ω ) resembles the term in Theorem 4.1 (note that we do not know whether U 1 is a valid upper bound for T ε ). For the second question, our numerical experiments (presented below) suggest that U ( ω ) is a tighter upper bound than U 0 ( ω ) .

4.2 Example on Tabular MDPs

In Figure 1, we compare the characteristic time upper bounds obtained in the previous subsection. These upper bounds correspond to the allocations ω ⋆ , ω ⋆ 0 , and ω ⋆ 1 obtained by minimizing, over ∆( S × A ) 1 , U ( ω ) , U 0 ( ω ) , and U 1 ( ω ) , respectively. We evaluated these characteristic times on various MDPs: (1) a random MDP (see Sec. A in the appendix); (2) the RiverSwim environment

5

7

10

15

20

S

10

11

10

14

10

17

Characteristic Time

U

RandomMDP

5

7

10

15

20

S

10

8

10

9

10

10

10

11

5

7

11

15

21

S

10

11

10

17

10

23

10

29

U

(

ω glyph[star]

)

U

(

ω glyph[star]

0

)

U

(

ω glyph[star]

1

)

U

0

(

ω glyph[star]

0

)

|

|

|

|

|

|

[47]; (3) the Forked RiverSwim , a novel environment where the agent needs to constantly explore two different states to learn the optimal policy (compared to the RiverSwim environment, the sample complexity is higher; refer to Appendix A for a complete description).

We note that across all plots, the optimal allocation ω ⋆ 0 has a quite large characteristic time (black cross). Instead, the optimal allocation ω ⋆ (blue circle) computed using our new upper bound (3) achieves a lower characteristic time. When we evaluate ω ⋆ 0 on the new bound (3) (orange star), we observe similar characteristic times.

Finally, to verify that we can indeed choose k = 1 uniformly across states and actions, we evaluated the characteristic time ω ⋆ 1 computed using (4) (green triangle). Our results indicate that the performance is not different from those obtained with ω ⋆ , suggesting that the quantities of interest (gaps and variances) are enough to learn an efficient exploration allocation. We investigate the choice of k in more detail in Appendix A.

5 Model-Free Active Exploration Algorithms

In this section we present MF-BPI, a model-free exploration algorithm that leverages the optimal allocations obtained through the previously derived upper bound of the sample complexity lower bound. We first present an upper bound ˜ U ( ω ) of U ( ω ) , so that it is possible to derive a closed form solution of the optimal allocation (an idea previously proposed in [28]).

Proposition 5.1. Assume that ϕ has a unique optimal policy π ⋆ . For all ω ∈ ∆( S × A ) , we have:

̸

̸

with H ( s, a ) := 2+8 φ 2 M k sa [ V ⋆ ] 2 1 -k ∆( s,a ) 2 and H := max s ′ C ( s ′ )(1+ γ ) 2 ∆ 2 min (1 -γ ) 2 . The minimizer ˜ ω ⋆ := arg inf ω ˜ U ( ω ) satisfies ˜ ω ⋆ ( s, a ) ∝ H ( s, a ) for a = π ⋆ ( s ) and ˜ ω ⋆ ( s, π ⋆ ( s )) ∝ √ H ∑ s,a = π ⋆ ( s ) H ( s, a ) / | S | otherwise.

̸

In the MF-BPI algorithm, we estimate the gaps ∆( s, a ) and M k sa [ V ⋆ ] for a fixed small value of k (we later explain how to do this in a model-free manner.) and compute the corresponding allocation ˜ ω ⋆ . This allocation drives the exploration under MF-BPI. Using this design approach, we face two issues:

(1) Uniform k and regularization. It is impractical to estimate M k sa [ V ⋆ ] for multiple values of k . Instead, we fix a small value of k ( e.g. , k = 1 or k = 2 ) for all state-action pairs (refer to the previous section for a discussion on this choice). Then, to avoid excessively small values of the gaps in the denominator, we regularize the allocation ˜ ω ⋆ by replacing, in the expression of H ( s, a ) (resp. H min ), ∆( s, a ) (resp. ∆ min ) by (∆( s, a ) + λ ) (resp. (∆ min + λ ) ) for some λ > 0 .

(2) Handling parametric uncertainty via bootstrapping. The quantities ∆( s, a ) and M k sa [ V ⋆ ] required to compute ˜ ω ⋆ remain unknown during training, and we adopt the Certainty Equivalence principle, substituting the current estimates of these quantities to compute the exploration strategy.

Algorithm 1 Boostrapped MF-BPI (Boostrapped Model Free Best Policy Identification)

Require: Parameters ( λ, k, p ) ; ensemble size B ; learning rates { ( α t , β t ) } t .

2: for t = 0 , 1 , 2 , . . . , do

1: Initialize Q 1 ,b ( s, a ) ∼ U ([0 , 1 / (1 -γ )]) and M 1 ,b ( s, a ) ∼ U ([0 , 1 / (1 -γ ) 2 k ]) for all ( s, a ) ∈ S × A and b ∈ [ B ] .

3: Bootstrap a sample ( ˆ Q t , ˆ M t ) from the ensemble, and compute the allocation ω ( t ) using Proposition 5.1. Sample a t ∼ ω ( t ) ( s t , · ) ; observe ( r t , s t +1 ) ∼ q ( ·| s t , a t ) ⊗ P ( ·| s t , a t ) .

4: for b = 1 , . . . , B do

5: With probability p , using the experience ( s t , a t , r t , s t +1 ) , update Q t,b and M t,b using Equations (5) and (6).

6: end for

7: end for

By doing so, we are inherently introducing parametric uncertainty into these terms that is not taken into account by the allocation ˜ ω ⋆ . To deal with this uncertainty, the traditional method, as used e.g. in [28, 29]), involves using ϵ -soft exploration policies to guarantee that all state-action pairs are visited infinitely often. This ensures that the estimation errors vanish as time grows large. In practice, we find this type of forced exploration inefficient. In MF-BPI, we opt for a bootstrapping approach to manage parametric uncertainties, which can augment the traditional forced exploration step, leading to more principled exploration.

5.1 Exploration in tabular MDPs.

The pseudo-code of MF-BPI for tabular MDPs is presented in Algorithm 1. In round t , MFBPI explores the MDP using the allocation ω ( t ) estimating ˜ ω ⋆ . To compute this allocation, we use Proposition 5.1 and need (i) the sub-optimality gaps ∆( s, a ) , which can be easily derived from the Q -function; (ii) the 2 k -th moment M k sa [ V ⋆ ] , which can always be learnt by means of stochastic approximation. In fact, for any Markovian policy π and pair ( s, a ) we have M k sa [ V π ϕ ] =

1 γ 2 k E s ′ ∼ P ( ·| s,a ) [ δ π ( s, a, s ′ ) 2 k ] , where δ π ( s, a, s ′ ) = r ( s, a )+ γ E a ′ ∼ π ( ·| s ′ ) [ Q π ( s ′ , a ′ )] -Q π ( s, a ) is a variant of the TD-error. MF-BPI then uses an asynchronous two-timescale stochastic approximation algorithm to learn Q ⋆ and M k sa [ V ⋆ ] ,

where δ ′ t = r t + γ max a Q t +1 ( s t +1 , a ) -Q t +1 ( s t , a t ) , and { ( α t , β t ) } t ≥ 0 are learning rates satisfying ∑ t ≥ 0 α t ( s, a ) = ∑ t ≥ 0 β t ( s, a ) = ∞ , ∑ t ≥ 0 ( α t ( s, a ) 2 + β t ( s, a ) 2 ) ≤ ∞ , and α t ( s,a ) β t ( s,a ) → 0 .

̸

MF-BPI uses bootstrapping to handle parametric uncertainty. We maintain an ensemble of ( Q,M ) -values, with B members, from which we sample ( ˆ Q t , ˆ M t ) at time t . This sample is generated by sampling a uniform random variable ξ ∼ U ([0 , 1]) and, for each ( s, a ) set ˆ Q t ( s, a ) = Quantile ξ ( Q t, 1 ( s, a ) , . . . , Q t,B ( s, a )) (assuming a linear interpolation). This method is akin to sampling from the parametric uncertainty distribution (we perform the same operation also to compute ˆ M t ). This sample is used to compute the allocation ω ( t ) using Proposition 5.1 by setting ∆ t ( s, a ) = max a ′ ˆ Q t ( s, a ′ ) -ˆ Q t ( s, a ) , π ⋆ t ( s ) = arg max a ˆ Q t ( s, a ) and ∆ min ,t = min s,a = π ⋆ t ( s ) ∆ t ( s, a ) . Note that, the allocation ω ( t ) can be mixed with a uniform policy, to guarantee asymptotic convergence of the estimates. Upon observing an experience, with probability p , MF-BPI updates a member of the ensemble using this new experience. p tunes the rate at which the models are updated, similar to sampling with replacement, speeding up the learning process. Selecting a high value for p compromises the estimation of the parametric uncertainty, whereas choosing a low value may slow down the learning process.

Exploration without bootstrapping? To illustrate the need for our bootstrapping approach, we tried to use the allocation ω ( t ) mixed with a uniform allocation. In Figure 2, we show the results on Riverswim-like environments with 5 states. While forced exploration ensures infinite visits to all

0

10000

20000

30000

40000

50000

t

0

.

0

0

.

5

1

.

0

1

-

‖

V

/star

-

V

π /star

t

‖

∞

‖

V

/star

‖

∞

Optimal

0

10000

20000

30000

40000

50000

t

0

.

0

0

.

5

1

.

0

Optimal

ForkedRiverswim

Forced MF-BPI

state-action pairs, this guarantee only holds asymptotically. As a result, the allocation mainly focuses on the current MDP estimate, neglecting other plausible MDPs that could produce the same data. This makes the forced exploration approach too sluggish for effective convergence, suggesting its inadequacy for rapid policy learning. These results highlight the need to account for the uncertainty in Q,M when computing the allocation.

5.2 Extension to Deep Reinforcement Learning

To extend bootstrapped MF-BPI to continuous MDPs, we propose DBMF-BPI (see Algorithm 2, or Appendix B). DBMF-BPI uses the mechanism of prior networks from BSP [39](bootstrapping with additive prior) to account for uncertainty that does not originate from the observed data. As before, we keep an ensemble { Q θ 1 , . . . , Q θ B } of Q -values (with their target networks) and an ensemble { M τ 1 , . . . , M τ B } of M -values, as well as their prior networks. We use the same procedure as in the tabular case to compute ( ˆ Q t , ˆ M t ) at time t , except that we sample ξ ∼ U ([0 , 1]) every T s ∝ (1 -γ ) -1 training steps (or at the end of an episode) to make the training procedure more stable. The quantity ˆ Q t is used to compute π ⋆ t ( s t ) and ∆ t ( s t , a ) . We estimate ∆ min ,t via stochastic approximation, with the minimum gap from the last batch of transitions sampled from the replay buffer serving as a target. To derive the exploration strategy, we compute H t ( s t , a ) = 2+8 φ 2 ˆ M t ( s t ,a ) 2 1 -k (∆ t ( s t ,a )+ λ ) 2

̸

Finally, we obtain an ϵ t -soft exploration policy ω ( t ) ( s t , · ) by mixing ω ( t ) o ( s t , · ) / ∑ a ω ( t ) o ( s t , a ) with a uniform distribution (using an exploration parameter ϵ t ).

̸

Algorithm 2 DBMF-BPI (Deep Bootstrapped Model Free BPI)

Require: Parameters ( λ, k ) ; ensemble size B ; exploration rate { ϵ t } t ; estimate ∆ min , 0 ; mask probability p . 1: Initialize replay buffer D , networks Q θ b , M τ b and targets Q θ ′ b for all b ∈ [ B ] .

2: for t = 0 , 1 , 2 , . . . , do

3: Sampling step.

4: Compute allocation ω ( t ) ← ComputeAllocation ( s t , { Q θ b , M τ b } b ∈ [ B ] , ∆ min ,t , γ, λ, k, ϵ t ) .

6: Add transition z t = ( s t , a t , r t , s t +1 ) to the replay buffer D .

5: Sample a t ∼ ω ( t ) ( s t , · ) and observe ( r t , s t +1 ) ∼ q ( ·| s t , a t ) ⊗ P ( ·| s t , a t ) .

7: Training step.

8: Sample a batch B from D , and with probability p add the i th experience in B to a sub-batch B b , ∀ b ∈ [ B ] . Update the ( Q,M ) -values of the b th member in the ensemble using B b : { Q θ b , Q θ ′ b , M τ b } b ∈ [ B ] ← Training ( {B b , Q θ b , Q θ ′ b , M τ b } b ∈ [ B ] ) .

10: end for

9: Update estimate ∆ min ,t +1 ← EstimateMinimumGap (∆ min ,t , B , { Q θ b } b ∈ [ B ] ) .

6 Numerical Results

We evaluate the performance of MF-BPI on benchmark problems and compare it against state-of-theart methods (details can be found in Appendix A).

Tabular MDPs. In the tabular case, we compared various algorithms on the Riverswim and Forked Riverswim environments. We evaluate MF-BPI with (1) bootstrapping and with (2) the forced exploration step using an ϵ -soft exploration policy, MDP-NAS [29], PSRL [36] and Q-UCB [19, 56]. For MDP-NAS, the model of the MDP was initialized in an optimistic way (with additive smoothing).

In both environments, we varied the size of the state space. In Figure 3, we show 1 -∥ V ⋆ -V π ⋆ T ∥ ∞ ∥ V ⋆ ∥ ∞ , a performance measure for the estimated policy π ⋆ T after T = | S | × 10 4 steps with γ = 0 . 99 . Results (the higher the better) indicate that bootstrapped MF-BPI can compete with model-based and model-free algorithms on hard-exploration problems, without resorting to expensive model-based procedures. Details of the experiments, including the initialization of the algorithms, are provided in Appendix A.

5

10

20

30

50

S

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

1

-

‖

V

glyph[star]

-

V

π glyph[star]

T

‖∞

‖

V

glyph[star]

‖∞

5

9

19

29

49

S

ForkedRiverswim

Agent

Bootstrapped MF-BPI

Forced MF-BPI

Q-UCB

MDP-NaS

PSRL

|

|

|

|

Deep RL. In environments with continuous state space, we compared DBMF-BPI with BSP [40, 39] (Bootstrapped DQN with randomized priors) and IDS [33] (Information-Directed Sampling). We also evaluated DBMF-BPI against BSP2, a variant of BSP that uses the same masking mechanism as DBMF-BPI for updating the ensemble. These methods were tested on challenging exploration problems from the DeepMind behavior suite [41] with varying levels of difficulty: (1) a stochastic version of DeepSea and (2) the Cartpole swingup problem. The DeepSea problem includes a 5% probability of the agent slipping, i.e., that an incorrect action is executed, which increases the aleatoric variance.

The results for the Cartpole swingup problem are depicted in Figure 4 for various difficulty levels k (see also Appendix A.5 for more details), demonstrating the ability of DBMF-BPI to quickly learn an efficient policy. While BSP generally performs well, there is a notable difference in performance when compared to DBMF-BPI. For a fair comparison, we used the same network initialization across all methods, except for IDS. Untuned, IDS performed poorly; proper initialization improved its performance, but results remained unsatisfactory. In Figure 5, we present two exploration metrics

0

50

100

150

200

Episode

0

10000

20000

30000

Total upright time [steps]

Cartpole swingup - difficulty

k

= 10

DBMF-BPI

BSP

BSP2

IDS

1

3

5

10

Difficulty

k

0

20000

40000

60000

Total upright time [steps]

Cartpole swingup

-

episode 200

0

5000

10000

15000

20000

Timestep

t

0

.

4

0

.

5

0

.

6

0

.

7

Normalized entropy

Frequency of visit (entropy)

0

5000

10000

15000

20000

Timestep

t

Recent visit (entropy)

DBMF-BPI

BSP

BSP2

IDS

for difficulty k = 5 . The frequency of visits measures the uniformity and dispersion of visits across the state space, while the second metric evaluates the recency of visits to different regions, capturing how frequently the methods keep visiting previously visited states (a smaller value indicates that the agent tends to concentrate on a specific region of the state space). For detailed analysis, please refer to appendix A.

0

1000

2000

3000

Episode

0

200

400

Successful episodes

Slipping DeepSea - problem size 30

DBMF-BPI

BSP

BSP2

IDS

10

20

30

40

50

DeepSea problem size

0

.

0

0

.

1

0

.

2

0

.

3

0

.

4

std(

t

visit

)

Slipping DeepSea

-

Exploration

For the slipping DeepSea problem, results are depicted in Fig. 6 (see also Appendix A.4 for more details). Besides the number of successful episodes, we also display the standard deviation of ( t visit ) ij across all cells ( i, j ) , where ( t visit ) ij indicates the last timestep t that a cell ( i, j ) was visited (normalized by NT , the product of the grid size, and the number of episodes). The right plot shows std( t visit ) for different problem sizes, highlighting the good exploration properties of DBMF-BPI. Additional details and exploration metrics can be found in Appendix A.

7 Conclusions

In this work, we studied the problem of exploration in Reinforcement Learning and presented MFBPI, a model-free solution for both tabular and continuous state-space MDPs. To derive this method, we established a novel approximation of the instance-specific lower bound necessary for identifying nearly-optimal policies. Importantly, this approximation depends only on quantities learnable via stochastic approximation, paving the way towards model-free methods. Numerical results on hardexploration problems highlighted the effectiveness of our approach for learning efficient policies over state-of-the-art methods.

Acknowledgments

This research was supported by the Swedish Foundation for Strategic Research through the CLAS project (grant RIT17-0046) and partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The authors would also like to thank the anonymous reviewers for their valuable and insightful feedback. On a personal note, Alessio Russo wishes to personally thank Damianos Tranos, Yassir Jedra, Daniele Foffano, and Letizia Orsini for their invaluable assistance in reviewing the manuscript.

References

Peter Atkins, Peter William Atkins, and Julio de Paula. Atkins' physical chemistry . Oxford university press, 2014.

Peter Auer. Using confidence bounds for exploitation-exploration trade-offs. Journal of Machine Learning Research , 3(Nov):397-422, 2002.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning , 47:235-256, 2002.

Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 21, 2008.

Kamyar Azizzadenesheli, Emma Brunskill, and Animashree Anandkumar. Efficient exploration through bayesian deep q-networks. In 2018 Information Theory and Applications Workshop (ITA) , pages 1-9. IEEE, 2018.

Chenjia Bai, Lingxiao Wang, Lei Han, Jianye Hao, Animesh Garg, Peng Liu, and Zhaoran Wang. Principled exploration via optimistic bootstrapping and backward induction. In International Conference on Machine Learning , pages 577-587. PMLR, 2021.

Andrew G Barto. Intrinsic motivation and reinforcement learning. Intrinsically Motivated Learning in Natural and Artificial Systems, Springer , pages 17-47, 2013.

Marc G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning , pages 449-458. PMLR, 2017.

Ronen I Brafman and Moshe Tennenholtz. R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research , 3(Oct):213-231, 2002.

Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations , 2018.

Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q-ensembles. arXiv preprint arXiv:1706.01502 , 2017.

William R Clements, Bastien Van Delft, Benoît-Marie Robaglia, Reda Bahi Slaoui, and Sébastien Toth. Estimating risk and uncertainty in deep reinforcement learning. arXiv preprint arXiv:1905.09638 , 2019.

Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Richard Dearden, Nir Friedman, and Stuart Russell. Bayesian q-learning. In Proceedings of the 15th National Conference on Artificial Intelligence, American Association for Artificial Intelligence, July, 1998 , pages 761-768. AAAI Press, 1998.

Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295 , 2017.

Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Conference on Learning Theory , pages 998-1027. PMLR, 2016.

Mohammad Gheshlaghi Azar, Rémi Munos, and Hilbert J Kappen. Minimax pac bounds on the sample complexity of reinforcement learning with a generative model. Machine learning, Springer , 91:325-349, 2013.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning , pages 1861-1870. PMLR, 2018.

Chi Jin, Zeyuan Allen-Zhu, Sebastien Bubeck, and Michael I Jordan. Is q-learning provably efficient? Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Sham Machandranath Kakade. On the sample complexity of reinforcement learning . University of London, University College London (United Kingdom), 2003.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best arm identification in multi-armed bandit models. Journal of Machine Learning Research , 17:1-42, 2016.

Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, Springer , 49:209-232, 2002.

Johannes Kirschner and Andreas Krause. Information directed sampling and bandits with heteroscedastic noise. In Conference On Learning Theory , pages 358-384. PMLR, 2018.

Pawel Ladosz, Lilian Weng, Minwoo Kim, and Hyondong Oh. Exploration in deep reinforcement learning: A survey. Information Fusion , 2022.

Tor Lattimore and Marcus Hutter. Pac bounds for discounted mdps. In Algorithmic Learning Theory: 23rd International Conference, ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23 , pages 320-334. Springer, 2012.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In International Conference on Machine Learning , pages 6131-6141. PMLR, 2021.

Aymen Al Marjani and Alexandre Proutiere. Adaptive sampling for best policy identification in markov decision processes. In International Conference on Machine Learning , pages 74597468. PMLR, 2021.

Aymen Al Marjani, Aurélien Garivier, and Alexandre Proutiere. Navigating to the best policy in markov decision processes. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

Borislav Mavrin, Hengshuai Yao, Linglong Kong, Kaiwen Wu, and Yaoliang Yu. Distributional reinforcement learning for efficient exploration. In International Conference on Machine Learning , pages 4424-4434. PMLR, 2019.

Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning , pages 1928-1937. PMLR, 2016.

Thomas M Moerland, Joost Broekens, and Catholijn M Jonker. Efficient exploration with double uncertain value networks. In Deep Reinforcement Learning Symposium, NIPS 2017 , 2017.

Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Informationdirected exploration for deep reinforcement learning. In International Conference on Learning Representations , 2018.

Jungseul Ok, Alexandre Proutiere, and Damianos Tranos. Exploration in structured reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Ian Osband and Benjamin Van Roy. Bootstrapped thompson sampling and deep exploration. arXiv preprint arXiv:1507.00300 , 2015.

Ian Osband, Daniel Russo, and Benjamin Van Roy. (more) efficient reinforcement learning via posterior sampling. Advances in Neural Information Processing Systems (NeurIPS) , 26, 2013.

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems (NeurIPS) , 29, 2016.

Ian Osband, Benjamin Van Roy, and Zheng Wen. Generalization and exploration via randomized value functions. In International Conference on Machine Learning , pages 2377-2386. PMLR, 2016.

Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Ian Osband, Benjamin Van Roy, Daniel J Russo, and Zheng Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research , 20:1-62, 2019.

Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvári, Satinder Singh, Benjamin Van Roy, Richard Sutton, David Silver, and Hado van Hasselt. Behaviour suite for reinforcement learning. In International Conference on Learning Representations , 2020.

Jason Pazis, Ronald E Parr, and Jonathan P How. Improving pac exploration using the median of means. Advances in Neural Information Processing Systems (NeurIPS) , 29, 2016.

Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for exploration. In International Conference on Learning Representations , 2018.

Daniel J Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, et al. A tutorial on thompson sampling. Foundations and Trends® in Machine Learning , 11(1):1-96, 2018.

Richard M Ryan and Edward L Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology , 25(1):54-67, 2000.

Mohit Sewak. Deep reinforcement learning . Springer, 2019.

Alexander L Strehl and Michael L Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences , 74(8):1309-1331, 2008.

Alexander L Strehl, Lihong Li, Eric Wiewiora, John Langford, and Michael L Littman. Pac model-free reinforcement learning. In Proceedings of the 23rd International Conference on Machine learning , pages 881-888, 2006.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

István Szita and Csaba Szepesvári. Model-based reinforcement learning with nearly tight exploration complexity bounds. In Proceedings of the 27th International Conference on Machine Learning (ICML-10) , pages 1031-1038, 2010.

Jérôme Taupin, Yassir Jedra, and Alexandre Proutiere. Best policy identification in discounted linear mdps. In Sixteenth European Workshop on Reinforcement Learning , 2023.

William R Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika , 25(3-4):285-294, 1933.

Andrea Tirinzoni, Aymen Al Marjani, and Emilie Kaufmann. Near instance-optimal pac reinforcement learning for deterministic mdps. Advances in Neural Information Processing Systems (NeurIPS) , 35:8785-8798, 2022.

Andrew Wagenmaker and Kevin G Jamieson. Instance-dependent near-optimal policy identification in linear mdps via online experiment design. Advances in Neural Information Processing Systems (NeurIPS) , 35:5968-5981, 2022.

Andrew J Wagenmaker, Max Simchowitz, and Kevin Jamieson. Beyond no regret: Instancedependent pac reinforcement learning. In Conference on Learning Theory , pages 358-418. PMLR, 2022.

Yuanhao Wang, Kefan Dong, Xiaoyu Chen, and Liwei Wang. Q-learning with ucb exploration is sample efficient for infinite-horizon mdp. In International Conference on Learning Representations , 2019.

Christopher John Cornish Hellaby Watkins. Learning from delayed rewards. PhD thesis, Cambridge University, Cambridge, England , 1989.

Jeremy Wyatt. Exploration and inference in learning from reinforcement. PhD thesis, University of Edinburgh, Edinburgh, England , 1998.

Tianpei Yang, Hongyao Tang, Chenjia Bai, Jinyi Liu, Jianye Hao, Zhaopeng Meng, Peng Liu, and Zhen Wang. Exploration in deep reinforcement learning: a comprehensive survey. arXiv preprint arXiv:2109.06668 , 2021.

Andrea Zanette, Mykel J Kochenderfer, and Emma Brunskill. Almost horizon-free structureaware best policy identification with a generative model. In Advances in Neural Information Processing Systems (NeurIPS) , volume 32, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a way of obtaining tighter PAC bounds for model-free reinforcement learning. The new theoretical results allow the authors to propose new practical methods for exploration in both discrete and continuous state spaces. The proposed algorithms use ensembles of Q-values, and the results are very competitive when compared with model-based approaches.\n\nSTRENGTHS:\n* The new methods adapt to specific problems in an automated way.\n\n* The paper is well written, and the discussion is supported by relevant citations. The contributions are backed up by both theoretical arguments and empirical results on a range RL domains.\n\n* The contributions seem relevant and important. The results indicate that the proposed methods have merit.\n\n* The paper comes with a comprehensive and long appendix (which would be a pain to read for a busy reviewer), but the future readers will most certainly appreciate it. The appendix provides many details that contribute to the quality of the work.\n\nWEAKNESSES:\n* The authors should make it clear from the start that the new algorithms are ensembles. After reading a few pages of the paper, I was anticipating a clever algorithm that would be purely based on the proposed theory. It was a bit disappointing that ensembles were used at the end. I don't say that this is bad, well there is no free lunch, but it would be fair to expect that, when the new algorithms are mentioned for the first time, ensembles are mentioned too.\n\n* The paper argues that model-base approaches are expensive to run. The methods presented in this paper are ensembles. What is their runtime? Are that they really faster than model-based methods?\n\n* Since the new methods proposed in this paper are ensembles, I would expect some discussion on ensembles in RL. I would expect that there must exist ensemble based RL methods that could be competitive with the methods presented here. Literature review on ensembles in RL would be useful. A classic paper on this is: Wiering, M.A. and Van Hasselt, H., 2008. Ensemble algorithms in reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 38(4), pp.930-936.\n\nQUESTIONS:\n* How exactly is the alearotic uncertainty in the value function accounted for by the methods proposed in this paper?\n\n* There is probability p in Alg. 1. It would be useful if its role and rationale were explained.\n\n* The use of the variance of the value function is slick, and the authors are careful saying that this addresses the aleatoric uncertainty only, but it would be useful if the authors explained why epistemic uncertainty is not addressed.\n\nLIMITATIONS:\nI tried to discuss this in the weaknesses and questions sections above. Even though some of my comments may appear critical, I am very positive about this paper. This is solid work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a model-free exploration approach developed on an information theoretical basis. Firstly, the lower bound on the number of samples for a near-optimal policy is estimated, and based on this lower bound, the paper develops an exploration strategy for both tabular and deep RL approaches. The exploration strategy is further validated via experiments, where it is found to be superior to other competing approaches.\n\nSTRENGTHS:\nThe paper is generally well written and addresses an important problem. The developed exploration strategy having an information theoretic basis sets it apart from some of the previous approaches which have mainly been based on heuristics.\n\nWEAKNESSES:\nThe paper does not include experiments in more complex environments such as Montezuma’s revenge, where exploration is known to be a key challenge. A more detailed summary of the main intuitions and theoretical results from [1] could have made the paper more readable.\n\nQUESTIONS:\n1. Can more challenging environments possibly be added? If not, are there any fundamental limitations that make this infeasible?\n\n2. Wouldn’t the approximation errors in the deep RL version of the algorithm affect the suboptimality gap? Is this somehow accounted for?\n\n3. Apart from the current results, perhaps for some of the environments, the actual exploration trajectories could be traced/charted out to explicitly show how the exploration is modified. I noticed something along these lines included in the appendix, but it would be good to bring similar results forward into the main text. \n\n4. It would benefit readers to include more details about the main results in [1].\n\n5. Although it may be obvious, in the cartpole swingup environment, it would be good to include a brief paragraph regarding why/how increasing k makes the task more difficult.\n\n6. In line 297, what is N?\n\n7. I am not sure why Fig 3 shows the performance in Riverswim vs |S| and ForkedRiverswim vs N\n\n8. How is $\\Delta_{min}$ initialized in the Deep RL version?\n\nLIMITATIONS:\nCurrently, there is no explicit section outlining the limitations. I suggest the authors include a short paragraph about this covering at least the basic assumptions and requirements for the proposed to be applicable. Perhaps some comments regarding the scalability of the approach to more complex environments could also be added if that is indeed an issue.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a model free approach to exploration in RL, that is based around best policy identification. This technique, unlike prior work, uses stochastic approximation to learn a lower bound on the policy performance based on collected samples.\n\nSTRENGTHS:\n- The paper presents an model-free approach that's heavily guided by theory, and results show how this technique can be effective in improving exploration in terms of improved performance as well as reduced performance variance.\n\nWEAKNESSES:\n- The paper is a bit tough to get through; being so theoretically heavy, some intuitive high-level explanation of their implications would be really helpful for the reader to follow along the thought process of the authors.\n\n- It is unclear where the derivation of the many theorems is in the text. For example, looking for a derivation on how the authors reached at corollary 5.1, the closest reference I found in the appendix was section B.3, but that's just an explanation on how to use that corollary.\n\nQUESTIONS:\n- Could you please clarify where the proof and derivations of the different theorems are? For instance, for theorem 4.2 there is a nice explanation of the implications of the theorem, but it is not stated how that upper bound is derived. The same is true about corollary 5.1.\nPlease clarify this, as the evaluation of soundness of the paper heavily relies on being able to find how these theorems where proved.\n\n- Algorithm 1 only samples actions according to the allocation omega, does this mean that over time omega converges to pi^*?\n\n- Corollary 5.1, the denominator in  H_epsilon contains the term delta_min is defined in terms of the optimal policy for the MDP. How are you sampling according to corollary 5.1 (algorithm 1), when the optimal policy pi^* is needed to compute that value?\nThis is a bit unclear.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

comments: SUMMARY:
This paper introduces an approximation of a lower bound on the number of samples needed to identify a nearly optimal policy directly applicable to model free RL.
They further propose a model free exploration strategy that can be applied to the tabular and continuous MDPs.

STRENGTHS:
The paper is clear and well written. With experiments showing the proposed approach seems competitive with other exploration approaches.

WEAKNESSES:
The paper lacks a comparison of the current approach with more common exploration strategies in DeepRL such as 34, 37. 
The results remain very toy. It would be great to have an understanding of the kind of exploration the proposed approach performs well, with increasingly sparse rewards? on which circumstances is this preferable.

QUESTIONS:
Why have you used the 2k-th moments in the bound? 
Can the authors clarify which eq. Compute Allocation, training and estimate minimum gap refer to?
How is the mask probability being computed? are the results sensitive to this parameter?

LIMITATIONS:
The authors could provide a more thorough limitation description of the approach under which circumstances does the proposed approach fail, how critical and reasonable are the assumptions made, for instance the communicating MDPs. How does the approach perform in more challenging tasks such as Montezuma's revenge, and more structured exploration tasks. Under which types of exploration difficulty does the proposed approach excel or fail.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the exploration of reinforcement learning and introduces a novel model-free algorithm. The authors derive a new bound for the lower bound of the number of samples needed to identify a near-optimal policy. Based on that, they develop a model-free exploration strategy that is applicable to both tabular and continuous MDPs. Experimental results demonstrate that their strategy is competitive to existing approaches in efficiently identifying optimal policies.\n\nSTRENGTHS:\n- Theoretically, the new proposed bounds look novel and sound. They transform the original lower bound into a more manageable form, making it easier to handle and apply in practical scenarios.\n\n- Empirically, the experiments confirm the superiority of the proposed algorithm over existing methods, aligning with their claims.\n\nWEAKNESSES:\n- There seems to be a big gap between the theory and the algorithm. The theoretical results involve certain quantities (e.g., $\\Delta(s,a)$) that are unknown to the algorithm. The authors addressed this issue by approximating these quantities through Q-value (and M-value) learning, subsequently treating the approximated Q and M as ground truth for all computations. This introduces a substantial gap between the theory and the algorithm.\n\n- It would improve if the authors could provide an intuitive explanation of the terms in the complicated expressions presented in Theorems 4.1 and 4.2. Understanding and evaluating these results as a whole seems challenging without a better explanation.\n\n- Although I admit that the main contributions lie in the theoretical aspects, it would be better to conduct more complicated experiments.\n\nQUESTIONS:\n- Considering the gap between the theory and the algorithm mentioned above, is there any convergence guarantee for the algorithm?\n\n- When the authors select $\\bar k=1$ (line 211) and arrive at $\\bar{U}^1_{\\epsilon}$, does it remain an upper bound for $T_\\epsilon$?\n\n- I was a bit confused about the main idea. The authors first proposed an upper bound $\\tilde{U}$ for the well-established lower bound $T_0$, and then they proceeded to carry out all computations based on this new bound. Although I understand that it is their point to derive this approximation that makes the bound easier to handle, it remains unclear to me why this approach is rational. Typically, what we usually do is to derive a more manageable upper bound on another harder upper bound, or derive a more manageable lower bound on another harder lower bound. Then we just need to minimize the new bound, because in doing so, the original bound is indirectly minimized (or maximized). However, this paper proposed an upper bound for the lower bound, making the aforementioned relationship inapplicable. Therefore, I am curious what the underlying principle here is.\n\nLIMITATIONS:\nNo negative societal impact was identified.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-17"}