A Provably Efficient Sample Collection Strategy for Reinforcement Learning

( BD +D3/2S2A ) time steps to collect the B = ∑ s , a b ( s , a ) desired samples , in any unknown communicating MDP with S states , A actions and diameter D. Then we show how this general-purpose exploration algorithm can be paired with “ objective-specific ” strategies that prescribe the sample requirements to tackle a variety of settings — e.g. , model estimation , sparse reward discovery , goal-free cost-free exploration in communicating MDPs — for which we obtain improved or novel sample complexity guarantees . 1 Introduction . One of the challenges in online reinforcement learning ( RL ) is that the agent needs to trade off the exploration of the environment and the exploitation of the samples to optimize its behavior . Whenever the agent needs to gather information about a specific region of the Markov decision process ( MDP ) , it must plan for a policy to reach the desired states , despite not having exact knowledge of the environment dynamics . This makes solving the exploration-exploitation problem in RL highly non-trivial and it requires designing a specific strategy depending on the learning objective , such as PAC-MDP learning [ e.g. , 13 , 47 , 59 ] , regret minimization [ e.g. , 28 , 6 , 29 , 66 ] or pure exploration [ e.g. , 30 , 31 , 39 , 63 , 64 ] . A simpler scenario considered in the literature is to assume access to a generative model or sampling oracle ( SO ) [ 33 ] . Given any state-action pair ( s , a ) , the SO returns a next state s′ drawn from the transition probability p ( ·|s , a ) and a reward r ( s , a ) . In this case , it is possible to focus exclusively on where and how many samples to collect , while disregarding the problem of finding a suitable policy 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . to obtain them . For instance , an SO can be used to obtain samples from the environment , which are combined with dynamic programming techniques to compute an ε-optimal policy . SO-based algorithms can be as simple as prescribing the same amount of samples from each state-action pair [ e.g . 35 , 33 , 5 , 16 , 46 , 1 , 38 ] or they may adaptively change the sample requirements on different stateaction pairs [ e.g . 15 , 58 , 62 ] . An SO is also used in Monte-Carlo planning [ 49 , 25 , 7 ] which focuses on computing the optimal action at the current state by optimizing over rollout trajectories sampled from the SO . Finally , in multi-armed bandit [ 37 ] , there are cases where each arm corresponds to a state ( or state-action ) , and “ pulling ” an arm translates into a call to an SO ( see e.g. , the pure exploration setting of [ 51 ] ) . Unfortunately , while an SO may be available in domains such as simulated robotics and computer games , this is not the case in the more general online RL setting . In this paper we tackle the exploration-exploitation problem in online RL by drawing inspiration from the SO assumption . Specifically , we define an approach that is decoupled in two parts : 1 ) an “ objective-specific ” algorithm that assumes access to an SO that ( adaptively ) prescribes the samples needed to achieve the learning objective of interest , and 2 ) an “ objective-agnostic ” algorithm that takes on the exploration challenge of collecting the samples requested by the SO-based algorithm as quickly as possible.1 Our main contributions can be summarized as follows : • We define the sample complexity of the objective-agnostic algorithm as the number of ( online ) steps needed to satisfy the prescribed sampling requirements . Leveraging recent techniques on exploration in the stochastic shortest path ( SSP ) problem [ 45 , 50 ] , we propose GOSPRL ( Goalbased Optimistic Sampling Procedure for RL ) , a conceptually simple and flexible exploration algorithm that learns how to “ generate ” the samples requested by any SO-based algorithm and we derive bounds on its sample complexity . • Leveraging the generality of our approach , we combine GOSPRL with problem-specific SO-based algorithms and readily obtain online RL algorithms in difficult exploration problems . While in general our decoupled approach may be suboptimal compared to exploration strategies designed to solve one specific problem , we obtain sample complexity guarantees that are on par or better than state-of-the-art algorithms in a range of problems . 1 ) GOSPRL solves the problem of sparse reward discovery in Õ ( D3/2S2A ) time steps , which improves the dependency on the diameter D w.r.t . a reward-free variant of UCRL2B [ 28 , 22 ] , as well as on S and A w.r.t . a MAXENT-type approach [ 26 , 17 ] . 2 ) GOSPRL improves over the method of [ 54 ] for model estimation , by removing their ergodicity assumption as well as achieving better sample complexity . 3 ) GOSPRL provably tackles the problem of goal-free cost-free exploration , for which no specific strategy is available . • We report numerical simulations supporting our theoretical findings and showing that pairing GOSPRL with SO-based algorithms outperforms both heuristic and theoretically grounded baselines in various problems . Related work . While to the best of our knowledge no other work directly addresses the problem of simulating an SO , a number of approaches are related to it . The problem solved by GOSPRL can be seen as a reward-free exploration problem , since it is not driven by any external reward but by the objective of covering the state space to quickly meet the sampling requirements . Standard exploration-exploitation algorithms , such as UCRL2 [ 28 ] in the undiscounted setting or RMAX [ 13 ] in the discounted one , implicitly encourage exploration to specific areas of the state-action space that are not estimated accurately enough . The objective of covering the state space is also studied in [ 26 , 17 ] with a Frank-Wolfe approach that optimizes a smooth aggregate function of the state visitations . Recent works on reward-free exploration ( RFE ) in the finite-horizon setting [ e.g. , 30 , 31 , 39 , 64 ] provide sufficient exploration so that an ε-optimal policy for any reward function can be computed . Our proposed solution shares high-level algorithmic principles with RFE approaches which incentivize the agent to visit insufficiently visited states via intrinsic reward . Nonetheless , our contribution significantly differs from existing RFE literature in two dimensions : 1 ) While we study the performance of GOSPRL in one goal-conditioned RFE problem ( Sect . 4.3 ) , our framework is much broader and it allows us to tackle a wider and diverse set of problems ( Sect . 4 and App . I ) ; 2 ) Our setting is horizon-agnostic and reset-free , which prevents from directly using any method or technical analysis in RFE designed for problems with an imposed planning horizon ( e.g. , finite-horizon or discounted ) . Finally , GOSPRL draws inspiration from the SSP formalism and solutions of [ 50 , 45 ] , but our approach critically differs from these works in three main ways : 1 ) we are interested in sample 1Alternatively , we can view it as a general approach to take any SO-based algorithm and convert it into an online RL algorithm . complexity guarantees rather than a regret analysis ; 2 ) we consider requirements ( i.e. , goals to sample ) that vary throughout the learning process , instead of an SSP problem with fixed goal state and cost function ; 3 ) we show how GOSPRL can serve as a sample collection component to tackle various learning problems other than regret minimization . 2 Problem Definition . We consider a finite and reset-free MDP [ 43 ] M : = 〈S , A , p , r , s0〉 , with S : = |S| states , A : = |A| actions and an arbitrary starting state s0 ∈ S. Calling an SO in any state-action pair ( s , a ) leads to two outcomes : a next state sampled from the transition probability distribution p ( ·|s , a ) ∈ ∆ ( S ) , and a reward r ( s , a ) ∈ R. A stationary deterministic policy is a mapping π : S → A from states to actions and we denote by ΠSD the set of all such policies . For any policy π and pair of states ( s , s′ ) , let τπ ( s→ s′ ) be the ( possibly infinite ) hitting time from s to s′ when executing π , i.e. , τπ ( s→ s′ ) : = inf { t ≥ 0 : st+1 = s′| s1 = s , π } , where st is the state visited at time step t. We introduce Dss′ : = min π∈ΠSD E [ τπ ( s→ s′ ) ] , Ds′ : = max s∈S\ { s′ } Dss′ , D : = max s′∈S Ds′ , where Dss′ is the shortest-path distance between s and s′ , Ds′ is the SSP-diameter of s′ [ 50 ] and D is the MDP diameter [ 28 ] . We now formalize the problem of simulating an SO ( i.e. , to generate the samples prescribed by an SO-based algorithm ) . At each time step t ≥ 1 the agent receives a function bt : S ×A → N , where bt ( s , a ) defines the total number of samples that need to be collected at ( s , a ) by time step t. We consider that ( bt ) t≥1 is an arbitrary sequence with each bt measurable w.r.t . the filtration up to time t ( i.e. , it may depend on the samples observed so far ) .2 We focus on the objective of designing an online algorithm that minimizes the time required to collect the prescribed samples . Since the environment is initially unknown , we need to trade off between exploring states and actions to improve estimates of the dynamics and exploiting current estimates to collect the required samples as quickly as possible . We formally define the performance metric as follows . Definition 1 . For any state-action pair , we denote by Nt ( s , a ) : = ∑t i=1 1 { ( si , ai ) = ( s , a ) } the number of visits to state s and action a up to ( and including ) time step t. Given a sampling requirement sequence b : = ( bt ) t≥1 with bt : S ×A → N and a confidence level δ ∈ ( 0 , 1 ) , we define the sample complexity of a learning algorithm A as C ( A , b , δ ) : = min { t > 0 : P ( ∀ ( s , a ) ∈ S ×A , Nt ( s , a ) ≥ bt ( s , a ) ) ≥ 1− δ } . With no additional condition , it is trivial to define problems such that C ( A , b , δ ) = +∞ for any algorithm . To avoid this case , we introduce the following assumptions . Assumption 1 . The MDP M is communicating with a finite and unknown diameter D < +∞ . Assumption 2 . There exist an unknown and bounded function b : S × A → N such that the sequence ( bt ) t≥1 verifies : ∀t ≥ 1 , ∀ ( s , a ) ∈ S ×A , bt ( s , a ) ≤ b ( s , a ) . Asm . 1 guarantees that whatever state needs to be sampled , there exists at least one policy that can reach it in finite time almost-surely ( notice that it is considerably weaker than the ergodicity assumption ( App . J ) often used in online RL , see e.g. , [ 60 , 40 , 24 ] ) . Asm . 2 ensures that the sequence of sampling requirements does not diverge and can thus be fulfilled in finite time . These assumptions guarantee that the problem in Def . 1 is well-posed and the sample complexity is bounded . A variety of problems can be cast under our decoupled approach , in the sense that they can be tackled by solving the problem of Def . 1 under a specific instantiation of the sampling requirement sequence ( bt ) t≥1 . For instance , consider the problem of covering the state-action space ( e.g. , to discover a hidden sparse reward ) , then the requirement is immediately defined as bt ( s , a ) = 1 . In Sect . 4 and App . I , we review problems where defining bt can be as simple as computing the sufficient number of samples needed to reach a certain level of accuracy in estimating a quantity of interest ( e.g. , model estimation ) or can be directly extracted from existing literature ( e.g. , ε-optimal policy learning ) . We now provide a simple worst-case lower bound on the sample complexity ( details in App . D ) . 2Allowing adaptive sampling requirements enables to pair GOSPRL with SO-based algorithms that adjust their requirements online as samples are being generated ( see e.g. , Sect . 4.2 ) . Algorithm 1 GOSPRL Algorithm Input : sampling requirement sequence ( bt ) t≥1 with bt : S ×A → N revealed at time t ( or anytime before ) . Initialize : Set G1 : = { s ∈ S : ∃a ∈ A , b1 ( s , a ) > 0 } , time step t : = 1 , counters N1 ( s , a ) : = 0 , attempt index k : = 1 and attempt counters U1 ( s , a ) : = 0 , ν1 ( s , a ) : = 0. while Gk is not empty do Define the SSP problem Mk with goal states Gk , and compute its optimistic shortest-path policy π̃k . Set flag = True and counter νk ( s , a ) : = 0. while flag do Execute action at : = π̃k ( st ) and observe next state st+1 ∼ p ( ·|st , at ) . Increment counters νk ( st , at ) and Nt ( st , at ) . if st+1 ∈ Gk or νk ( st , at ) > { Uk ( st , at ) ∨ 1 } then Set flag = False . end if Set t += 1. end while if st ∈ Gk then Execute an action a s.t.Nt ( st , a ) < bt ( st , a ) , observe next state st+1 ∼ p ( ·|st , a ) and set t += 1. end if Set Uk+1 ( s , a ) : = Uk ( s , a ) + νk ( s , a ) , k += 1 . Update the set of goal states Gk : = { s ∈ S : ∃a ∈ A , Nt−1 ( s , a ) < bt−1 ( s , a ) } . end while Lemma 1 . For any S ≥ 1 , there exists an MDP with S states satisfying Asm . 1 such that for any sampling requirement b : S → N satisfying Asm . 2 , min A C ( A , b , 12 ) = Ω ( ∑ s∈S Dsb ( s ) ) . Lem . 1 shows that the ( possibly non-stationary ) policy minimizing the time to collect all samples requires Ω ( ∑ sDsb ( s ) ) time steps in a worst-case MDP . We also notice that when the total sampling requirement B is concentrated on the state s for which Ds = D ( i.e. , b ( s′ ) = 0 , ∀s′ 6= s ) , the previous bound reduces to Ω ( BD ) .

Summary: The authors introduce a general technique for exploring the MDP in an SSP problem. Their approach extends the notion of reward-free exploration to  an "objective-agnostic" scenario, in which the goal is to generate a prescribed amount of samples from any state and action. The authors analyze the proposed algorithm and provide a sample complexity bound.  Then, the authors show this technique can be leveraged to solve specific scenarios which fit the communicating MDP setup. For these scenarios, the authors show the superiority of their approach with respect to previous works.