A Provably Efficient Sample Collection Strategy for Reinforcement Learning

( BD +D3/2S2A ) time steps to collect the B = ∑ s , a b ( s , a ) desired samples , in any unknown communicating MDP with S states , A actions and diameter D. Then we show how this general-purpose exploration algorithm can be paired with “ objective-specific ” strategies that prescribe the sample requirements to tackle a variety of settings — e.g. , model estimation , sparse reward discovery , goal-free cost-free exploration in communicating MDPs — for which we obtain improved or novel sample complexity guarantees . 1 Introduction . One of the challenges in online reinforcement learning ( RL ) is that the agent needs to trade off the exploration of the environment and the exploitation of the samples to optimize its behavior . Whenever the agent needs to gather information about a specific region of the Markov decision process ( MDP ) , it must plan for a policy to reach the desired states , despite not having exact knowledge of the environment dynamics . This makes solving the exploration-exploitation problem in RL highly non-trivial and it requires designing a specific strategy depending on the learning objective , such as PAC-MDP learning [ e.g. , 13 , 47 , 59 ] , regret minimization [ e.g. , 28 , 6 , 29 , 66 ] or pure exploration [ e.g. , 30 , 31 , 39 , 63 , 64 ] . A simpler scenario considered in the literature is to assume access to a generative model or sampling oracle ( SO ) [ 33 ] . Given any state-action pair ( s , a ) , the SO returns a next state s′ drawn from the transition probability p ( ·|s , a ) and a reward r ( s , a ) . In this case , it is possible to focus exclusively on where and how many samples to collect , while disregarding the problem of finding a suitable policy 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to obtain them . For instance , an SO can be used to obtain samples from the environment , which are combined with dynamic programming techniques to compute an ε-optimal policy . SO-based algorithms can be as simple as prescribing the same amount of samples from each state-action pair [ e.g . 35 , 33 , 5 , 16 , 46 , 1 , 38 ] or they may adaptively change the sample requirements on different stateaction pairs [ e.g . 15 , 58 , 62 ] . An SO is also used in Monte-Carlo planning [ 49 , 25 , 7 ] which focuses on computing the optimal action at the current state by optimizing over rollout trajectories sampled from the SO . Finally , in multi-armed bandit [ 37 ] , there are cases where each arm corresponds to a state ( or state-action ) , and “ pulling ” an arm translates into a call to an SO ( see e.g. , the pure exploration setting of [ 51 ] ) . Unfortunately , while an SO may be available in domains such as simulated robotics and computer games , this is not the case in the more general online RL setting . In this paper we tackle the exploration-exploitation problem in online RL by drawing inspiration from the SO assumption . Specifically , we define an approach that is decoupled in two parts : 1 ) an “ objective-specific ” algorithm that assumes access to an SO that ( adaptively ) prescribes the samples needed to achieve the learning objective of interest , and 2 ) an “ objective-agnostic ” algorithm that takes on the exploration challenge of collecting the samples requested by the SO-based algorithm as quickly as possible.1 Our main contributions can be summarized as follows : • We define the sample complexity of the objective-agnostic algorithm as the number of ( online ) steps needed to satisfy the prescribed sampling requirements . Leveraging recent techniques on exploration in the stochastic shortest path ( SSP ) problem [ 45 , 50 ] , we propose GOSPRL ( Goalbased Optimistic Sampling Procedure for RL ) , a conceptually simple and flexible exploration algorithm that learns how to “ generate ” the samples requested by any SO-based algorithm and we derive bounds on its sample complexity . • Leveraging the generality of our approach , we combine GOSPRL with problem-specific SO-based algorithms and readily obtain online RL algorithms in difficult exploration problems . While in general our decoupled approach may be suboptimal compared to exploration strategies designed to solve one specific problem , we obtain sample complexity guarantees that are on par or better than state-of-the-art algorithms in a range of problems . 1 ) GOSPRL solves the problem of sparse reward discovery in Õ ( D3/2S2A ) time steps , which improves the dependency on the diameter D w.r.t . a reward-free variant of UCRL2B [ 28 , 22 ] , as well as on S and A w.r.t . a MAXENT-type approach [ 26 , 17 ] . 2 ) GOSPRL improves over the method of [ 54 ] for model estimation , by removing their ergodicity assumption as well as achieving better sample complexity . 3 ) GOSPRL provably tackles the problem of goal-free cost-free exploration , for which no specific strategy is available . • We report numerical simulations supporting our theoretical findings and showing that pairing GOSPRL with SO-based algorithms outperforms both heuristic and theoretically grounded baselines in various problems . Related work . While to the best of our knowledge no other work directly addresses the problem of simulating an SO , a number of approaches are related to it . The problem solved by GOSPRL can be seen as a reward-free exploration problem , since it is not driven by any external reward but by the objective of covering the state space to quickly meet the sampling requirements . Standard exploration-exploitation algorithms , such as UCRL2 [ 28 ] in the undiscounted setting or RMAX [ 13 ] in the discounted one , implicitly encourage exploration to specific areas of the state-action space that are not estimated accurately enough . The objective of covering the state space is also studied in [ 26 , 17 ] with a Frank-Wolfe approach that optimizes a smooth aggregate function of the state visitations . Recent works on reward-free exploration ( RFE ) in the finite-horizon setting [ e.g. , 30 , 31 , 39 , 64 ] provide sufficient exploration so that an ε-optimal policy for any reward function can be computed . Our proposed solution shares high-level algorithmic principles with RFE approaches which incentivize the agent to visit insufficiently visited states via intrinsic reward . Nonetheless , our contribution significantly differs from existing RFE literature in two dimensions : 1 ) While we study the performance of GOSPRL in one goal-conditioned RFE problem ( Sect . 4.3 ) , our framework is much broader and it allows us to tackle a wider and diverse set of problems ( Sect . 4 and App . I ) ; 2 ) Our setting is horizon-agnostic and reset-free , which prevents from directly using any method or technical analysis in RFE designed for problems with an imposed planning horizon ( e.g. , finite-horizon or discounted ) . Finally , GOSPRL draws inspiration from the SSP formalism and solutions of [ 50 , 45 ] , but our approach critically differs from these works in three main ways : 1 ) we are interested in sample 1Alternatively , we can view it as a general approach to take any SO-based algorithm and convert it into an online RL algorithm . complexity guarantees rather than a regret analysis ; 2 ) we consider requirements ( i.e. , goals to sample ) that vary throughout the learning process , instead of an SSP problem with fixed goal state and cost function ; 3 ) we show how GOSPRL can serve as a sample collection component to tackle various learning problems other than regret minimization . 2 Problem Definition . We consider a finite and reset-free MDP [ 43 ] M : = 〈S , A , p , r , s0〉 , with S : = |S| states , A : = |A| actions and an arbitrary starting state s0 ∈ S. Calling an SO in any state-action pair ( s , a ) leads to two outcomes : a next state sampled from the transition probability distribution p ( ·|s , a ) ∈ ∆ ( S ) , and a reward r ( s , a ) ∈ R. A stationary deterministic policy is a mapping π : S → A from states to actions and we denote by ΠSD the set of all such policies . For any policy π and pair of states ( s , s′ ) , let τπ ( s→ s′ ) be the ( possibly infinite ) hitting time from s to s′ when executing π , i.e. , τπ ( s→ s′ ) : = inf { t ≥ 0 : st+1 = s′| s1 = s , π } , where st is the state visited at time step t. We introduce Dss′ : = min π∈ΠSD E [ τπ ( s→ s′ ) ] , Ds′ : = max s∈S\ { s′ } Dss′ , D : = max s′∈S Ds′ , where Dss′ is the shortest-path distance between s and s′ , Ds′ is the SSP-diameter of s′ [ 50 ] and D is the MDP diameter [ 28 ] . We now formalize the problem of simulating an SO ( i.e. , to generate the samples prescribed by an SO-based algorithm ) . At each time step t ≥ 1 the agent receives a function bt : S ×A → N , where bt ( s , a ) defines the total number of samples that need to be collected at ( s , a ) by time step t. We consider that ( bt ) t≥1 is an arbitrary sequence with each bt measurable w.r.t . the filtration up to time t ( i.e. , it may depend on the samples observed so far ) .2 We focus on the objective of designing an online algorithm that minimizes the time required to collect the prescribed samples . Since the environment is initially unknown , we need to trade off between exploring states and actions to improve estimates of the dynamics and exploiting current estimates to collect the required samples as quickly as possible . We formally define the performance metric as follows . Definition 1 . For any state-action pair , we denote by Nt ( s , a ) : = ∑t i=1 1 { ( si , ai ) = ( s , a ) } the number of visits to state s and action a up to ( and including ) time step t. Given a sampling requirement sequence b : = ( bt ) t≥1 with bt : S ×A → N and a confidence level δ ∈ ( 0 , 1 ) , we define the sample complexity of a learning algorithm A as C ( A , b , δ ) : = min { t > 0 : P ( ∀ ( s , a ) ∈ S ×A , Nt ( s , a ) ≥ bt ( s , a ) ) ≥ 1− δ } . With no additional condition , it is trivial to define problems such that C ( A , b , δ ) = +∞ for any algorithm . To avoid this case , we introduce the following assumptions . Assumption 1 . The MDP M is communicating with a finite and unknown diameter D < +∞ . Assumption 2 . There exist an unknown and bounded function b : S × A → N such that the sequence ( bt ) t≥1 verifies : ∀t ≥ 1 , ∀ ( s , a ) ∈ S ×A , bt ( s , a ) ≤ b ( s , a ) . Asm . 1 guarantees that whatever state needs to be sampled , there exists at least one policy that can reach it in finite time almost-surely ( notice that it is considerably weaker than the ergodicity assumption ( App . J ) often used in online RL , see e.g. , [ 60 , 40 , 24 ] ) . Asm . 2 ensures that the sequence of sampling requirements does not diverge and can thus be fulfilled in finite time . These assumptions guarantee that the problem in Def . 1 is well-posed and the sample complexity is bounded . A variety of problems can be cast under our decoupled approach , in the sense that they can be tackled by solving the problem of Def . 1 under a specific instantiation of the sampling requirement sequence ( bt ) t≥1 . For instance , consider the problem of covering the state-action space ( e.g. , to discover a hidden sparse reward ) , then the requirement is immediately defined as bt ( s , a ) = 1 . In Sect . 4 and App . I , we review problems where defining bt can be as simple as computing the sufficient number of samples needed to reach a certain level of accuracy in estimating a quantity of interest ( e.g. , model estimation ) or can be directly extracted from existing literature ( e.g. , ε-optimal policy learning ) . We now provide a simple worst-case lower bound on the sample complexity ( details in App . D ) . 2Allowing adaptive sampling requirements enables to pair GOSPRL with SO-based algorithms that adjust their requirements online as samples are being generated ( see e.g. , Sect . 4.2 ) . Algorithm 1 GOSPRL Algorithm Input : sampling requirement sequence ( bt ) t≥1 with bt : S ×A → N revealed at time t ( or anytime before ) . Initialize : Set G1 : = { s ∈ S : ∃a ∈ A , b1 ( s , a ) > 0 } , time step t : = 1 , counters N1 ( s , a ) : = 0 , attempt index k : = 1 and attempt counters U1 ( s , a ) : = 0 , ν1 ( s , a ) : = 0. while Gk is not empty do Define the SSP problem Mk with goal states Gk , and compute its optimistic shortest-path policy π̃k . Set flag = True and counter νk ( s , a ) : = 0. while flag do Execute action at : = π̃k ( st ) and observe next state st+1 ∼ p ( ·|st , at ) . Increment counters νk ( st , at ) and Nt ( st , at ) . if st+1 ∈ Gk or νk ( st , at ) > { Uk ( st , at ) ∨ 1 } then Set flag = False . end if Set t += 1. end while if st ∈ Gk then Execute an action a s.t.Nt ( st , a ) < bt ( st , a ) , observe next state st+1 ∼ p ( ·|st , a ) and set t += 1. end if Set Uk+1 ( s , a ) : = Uk ( s , a ) + νk ( s , a ) , k += 1 . Update the set of goal states Gk : = { s ∈ S : ∃a ∈ A , Nt−1 ( s , a ) < bt−1 ( s , a ) } . end while Lemma 1 . For any S ≥ 1 , there exists an MDP with S states satisfying Asm . 1 such that for any sampling requirement b : S → N satisfying Asm . 2 , min A C ( A , b , 12 ) = Ω ( ∑ s∈S Dsb ( s ) ) . Lem . 1 shows that the ( possibly non-stationary ) policy minimizing the time to collect all samples requires Ω ( ∑ sDsb ( s ) ) time steps in a worst-case MDP . We also notice that when the total sampling requirement B is concentrated on the state s for which Ds = D ( i.e. , b ( s′ ) = 0 , ∀s′ 6= s ) , the previous bound reduces to Ω ( BD ) .

Summary: This paper considers the problem of learning to sufficiently explore an MDP in order to effectively provide a generative model that can be used for downstream learning tasks by algorithms that require a generative model. An algorithm is given that leverages SSP results in order to force exploration of certain state-actions that may be required by the arbitrary down-stream algorithm. Sample complexity upper and lower bounds are proved to demonstrate the samples required to meet the budget prescribed by the downstream algorithm. Several example tasks that require non-trivial exploration are shown to make use of the algorithm.