Near-Optimal Algorithms for Autonomous Exploration and Multi-Goal Stochastic Shortest Path

1 INTRODUCTION . Reinforcement learning ( RL ) with a known state space has been studied in a wide range of settings ( e.g. , Schmidhuber , 1991 ; Oudeyer et al. , 2007 ; Oudeyer and Kaplan , 2009 ; Baranes and Oudeyer , 2009 ) . When the state space is large , it is difficult for a learning agent to discover the whole environment . Instead , the agent can only explore a small portion of the environment . At a high level , we hope that the agent can discover states near the initial state , expand the range of known states by exploration , and learn near-optimal goal-conditioned policies for the known states . Because the agent discovers its known states of the environment incrementally , this learning problem was named Autonomous Exploration ( AX ) ( Lim and Auer , 2012 ; Tarbouriech et al. , 2020 ) . The autonomous exploration problem generalizes the Stochastic Shortest Path ( SSP ) problem ( Bertsekas et al. , 2000 ) where the agent aims to reach a predefined goal state while minimizing its total expected cost . However , in the autonomous exploration setting , the agent aims to discover a set of reachable states in a large environment and find the optimal policies to reach them . The autonomous exploration formulation is applicable to an increasing number of real-world RL problems , ranging from navigation in mazes ( Devo et al. , 2020 ) to game playing ( Mnih et al. , 2013 ) . For example , in the maze navigation problem , a robot aims to follow a predefined path in an unknown environment , and the robot has to discover and expand the size of regions known to itself autonomously without prior knowledge of the environment . This procedure also resembles some biological learning processes . See Lim and Auer ( 2012 ) for more discussions . Related Work . The setting of autonomous exploration was introduced by Lim and Auer ( 2012 ) , who gave the first algorithm , UcbExplore , with a sample complexity Õ ( L3S ( 1+ε ) LA/ε3 ) . Here L denotes the distance within which we hope the learning agent to discover , S ( 1+ε ) L denotes the total number of states within distance ( 1 + ε ) L from the starting state , A denotes the size of the action space , and ε denotes the error that we can tolerate . Recent work by Tarbouriech et al . ( 2020 ) designed the DisCo algorithm with a sample complexity bound Õ ( L3S2 ( 1+ε ) LA/ε 2 ) 1 , which improves the 1/ε 1We translate their absolute error ε to the relative error εL . Their bound has a refined form Õ ( L3S ( 1+ε ) LΓ ( 1+ε ) LA/ε 2 ) where Γ ( 1+ε ) L is the branching factor which in the worst case is S ( 1+ε ) L. dependency at the cost of a worse dependency on S ( 1+ε ) L. In this paper , we present new algorithms to further improve the sample complexity . 1.1 CONTRIBUTIONS . In this paper , we take important steps toward resolving the autonomous exploration problem . We compare our results with prior ones in Table 1.2 and we summarize our contributions below : 1 . We propose a new state discovery algorithm , Value-Optimistic Incremental State Discovery ( VOISD ) , which uses a value-optimistic method ( Neu and Pike-Burke , 2020 ) to estimate the expected cost of the optimal policy to reach these states . We prove this algorithm enjoys an Õ ( L3S ( 1+ε ) LA/ε 2 ) sample complexity , which improves prior results in Lim and Auer ( 2012 ) and Tarbouriech et al . ( 2020 ) . 2 . We connect the autonomous exploration problem to a new problem , multi-goal SSP and propose a new algorithm Re-MG-SSP that 1 ) satisfies a stronger criterion3 and 2 ) serves as a burn-in step for the next algorithm . 3 . We further propose Value-Aware Autonomous Exploration ( VALAE ) , which uses VOISD and Re-MG-SSP as initial steps and then uses the estimated value functions to guide our exploration . By doing so , for each state-action pair ( s , a ) , we derive an ( s , a ) -dependent sample complexity bound , which can exploit the variance information , and yield a sharper sample complexity bound than the bounds for VOISD and Re-MG-SSP . In particular , VALAE improves the dependency on L from cubic to linear . 4 . We give the first lower bound of the autonomous exploration problem . This lower bound shows VALAE is nearly minimax-optimal when the SL grows polynomially with respect to L . 1.2 MAIN DIFFICULTIES AND TECHNIQUE OVERVIEW . While our work borrows ideas from prior work on autonomous exploration ( Lim and Auer , 2012 ; Tarbouriech et al. , 2020 ) and recent advances in SSP ( Tarbouriech et al. , 2021 ) , we develop new techniques to overcome additional difficulties that are unique in autonomous exploration . Dependence between the Estimated Transition Matrix and Discovered States . Our algorithm incrementally adds new states to the set of discovered states K. To obtain a tight dependency on S ( 1+ε ) L , similar to the standard RL setting , one needs to use concentration on ( P̂s , a − Ps , a ) V ∗K , g instead of ‖P̂s , a − Ps , a‖1 ( used by Tarbouriech et al . ( 2020 ) ) where P̂s , a is the estimated transition , 2In ( Lim and Auer , 2012 ) and ( Tarbouriech et al. , 2020 ) , the cost is 1 uniformly for all state-action pairs . In this paper , we allow non-uniform costs . In Table , we consider uniform cost in Table 1 for fair comparisons . 3See Sect . 2 for different criteria . Ps , a is the true transition , and V ∗K , g is the value function of the optimal policy going to the state g restricted on the discovered states K. The main challenge is that the set of discovered states K is dependent on the samples collected , and thus V ∗K , g is dependent on P̂s , a . We can use the union bound on all the possible K , but the number of possible K is exponential in the number of states S. Our main technique is to construct a series of sets of states { s0 } = K0 ⊆ K1 ⊆ · · · ⊆ KZ = S→L , where Kz+1 is constructed by adding all the states that are reachable from s0 by some policy on Kz with expected cost no more than L. This series is only polynomially large so after applying the union bound we only have a logarithmic overhead . In order to use concentrations only on this sequence of sets , we also need to develop a modified definition of optimism . See Appendix B and Appendix C.2 for details . Connection between Autonomous Exploration and Multi-Goal SSP . In standard RL setting , it is known that in order to obtain a tight dependency on L , one needs to exploit the variance information in the value function ( Azar et al. , 2017 ) . However , in autonomous exploration , it is unclear how to exploit the variance information because even which state is in S→L is unknown . To this end , we first consider a simpler problem , multi-goal SSP , and extend the technique for singlegoal SSP ( Tarbouriech et al. , 2021 ) to this new problem ( cf . Alg . 3 ) . We also present a reduction from autonomous exploration to multi-goal SSP ( cf . Alg . 2 ) . These two techniques together yield the first tight dependency on L for autonomous exploration . 2 PRELIMINARIES . In this section , we introduce basic definitions and our problem setup . Notations . For any two vectorsX , Y ∈ RS , we write their inner product asXY : = ∑ s∈S X ( s ) Y ( s ) . We denote ‖X‖∞ : = maxs∈S |X ( S ) | , and if X is a probability distribution on S , we define V ( X , Y ) : = ∑ s∈S X ( s ) Y ( s ) 2 − ( ∑ s∈S X ( s ) Y ( s ) ) 2 . Markov Decision Process . We consider an MDP M : = 〈S , A , P , c , s0〉 , where S is the state space with size S , A is the action space with size A , and s0 ∈ S is the initial state . In state s , taking action a has a cost drawn i.i.d . from a distribution on [ cmin , 1 ] ( where cmin > 0 ) with expectation c ( s , a ) , and transits to the next state s′ with probability P ( s′|s , a ) . For convenience , we use Ps , a and Ps , a , s′ to denote P ( ·|s , a ) and P ( s′|s , a ) , respectively . A deterministic and stationary policy π : S → A is a mapping , and the agent following the policy π will take action π ( s ) at state s. For a fixed state g ∈ S we define the random variable tπg ( s ) as the number of steps it takes to reach state g starting from state swhen executing policy π , i.e . tπg ( s ) : = inf { t ≥ 0 : st+1 = g | s1 = s , π } . A policy π is a proper policy if for any state s ∈ S , tπg ( s ) < +∞ with probability 1 . Then we define the value function of a proper policy π with respect to the goal state g and its corresponding Q-function as follows : V πg ( s ) = E tπg ( s ) ∑ t=1 ct ( st , π ( st ) ) | s1 = s , Qπg ( s , a ) = E tπg ( s ) ∑ t=1 ct ( st , π ( st ) ) | s1 = s , π ( s1 ) = a , where ct ∈ [ cmin , 1 ] is the instantaneous cost at step t incurred by the state-action pair ( st , π ( st ) ) , and the expectation is taken over the random sequence of states generated by executing π starting from state s ∈ S. Here we have V πg ( g ) = 0 . We use πQ to denote the greedy policy over a vector Q ∈ RS×A , i.e . πQ ( s ) : = arg min a∈A Q ( s , a ) . For a fixed state g ∈ S , we denote V ∗g as the value function of the optimal policy with respect to goal state g , and here we list some important properties of V ∗g : there exists a stationary , deterministic and proper policy π∗ , such that its value function V ∗g : = V π∗ g and its corresponding Q-function Q∗g : = Q π∗ g satisfies the following Bellman optimality equations ( cf . Lem . 1 ) : Q∗g ( s , a ) = c ( s , a ) + Ps , aV ∗ g , V ∗ g ( s ) = min a∈A Q∗g ( s , a ) , ∀ ( s , a ) ∈ S ×A . Autonomous Exploration . Now we introduce the Autonomous Exploration problem . To formally discuss the setting , we need the following assumption on our MDP M . Assumption 1 . The action space contains a RESET action s.t . P ( s0|s , RESET ) = 1 for any s ∈ S . Moreover , taking RESET in any state s will incur a cost cRESET with probability 1 , where cRESET is a constant in [ cmin , 1 ] . Given any fixed length L ≥ 1 , the agent needs to learn the set of incrementally controllable states S→L . To introduce the concept of S→L , we first give the definition of policies restricted on a subset : Definition 1 ( Policy restricted on a subset ) . For any S ′ ⊆ S , a policy π is restricted on the set S ′ if π ( s ) = RESET for all s /∈ S ′ . Now we discuss the optimal policy restricted on a set of states K ⊆ S with respect to goal state g. We denote V ∗K , g ∈ RS as the value function of the optimal policy restricted on K with goal g , and Q∗K , g as the Q-function corresponding to V ∗ K , g . We consider the case that there exists at least one proper policy restricted on K with the goal state g. Then , V ∗K , g and Q∗K , g are finite , and they satisfy the following Bellman equations : Q∗K , g ( s , a ) = c ( s , a ) + Ps , aV ∗ K , g , ∀ ( s , a ) ∈ S ×A , V ∗K , g ( s ) = min a∈A Q∗K , g ( s , a ) , ∀s ∈ K , s 6= g , V ∗K , g ( s ) = Q ∗ K , g ( s , RESET ) = cRESET + V ∗ K , g ( s0 ) , ∀s /∈ K ∪ { g } , V ∗K , g ( g ) = 0 . We note that when K1 ⊆ K2 , for any g ∈ S , if V ∗K1 , g is finite , then V ∗ K2 , g is also finite , and we have V ∗K2 , g ≤ V ∗ K1 , g component-wise . And we note that for any s 6= g , we have mina∈AQ ∗ K , g ( s , a ) ≤ V ∗K , g ( s ) . Now we introduce the definition of incrementally controllable states S→L ( see Tarbouriech et al . ( 2020 ) for more intuitions on this definition . ) : Definition 2 ( Incrementally L-controllable states S→L ) . Let ≺ be any partial order on S . We denote S≺L as the set of states reachable from s0 with expected cost no more than L w.r.t . ≺ , which is defined as follows : • s0 ∈ S≺L , • if there is a policy π restricted on { s′ ∈ S≺L : s′ ≺ s } such that V πs ( s0 ) ≤ L , then s ∈ S ≺ L . The set of incrementally L-controllable states S→L is given by S→L = ⋃ ≺ S≺L . Learning Objective . In our settings , the learning agent knows the constants S , A and cmin , but the learning agent has no prior knowledge of the transition probability P or the cost function c ( · , · ) of the MDP M . We fix the length L ≥ 1 and an error parameter ε ∈ ( 0 , 1 ] . A learning algorithm of the autonomous exploration problem should output a set of states K ⊆ S that satisfies : • S→L ⊆ K , i.e. , the algorithm discovers all the states that we want to explore . The algorithm also outputs a set of policies { πs } s∈K that satisfy one of the following criteria : 1 . ( AXL on S→L ) ∀s ∈ S→L , V πss ( s0 ) ≤ ( 1 + ε ) L ; 2 . ( AX∗ on S→L ) ∀s ∈ S→L , V πss ( s0 ) ≤ V ∗S→L , s ( s0 ) + εL ; 3 . ( AXL on K ) ∀s ∈ K , V πss ( s0 ) ≤ ( 1 + ε ) L ; 4 . ( AX∗ on K ) ∀s ∈ K , V πss ( s0 ) ≤ V ∗K , s ( s0 ) + εL . We note that AX∗ on K is stronger than both AXL on S→L and AX ∗ on S→L , but it is not necessarily stronger than AXL on K , because we do not necessarily have V ∗K , s ( s0 ) ≤ L when s /∈ S→L . In the literature , AXL on S→L was studied in the original paper that proposed the autonomous exploration problem ( Lim and Auer , 2012 ) and AX∗ on K condition was studied in ( Tarbouriech et al. , 2020 ) . We denote T as the total number of steps the agent uses , and denote ( st , at ) as the state-action pair at the t-th step . We denote by ct ( st , at ) the instantaneous cost incurred at the t-th step . The performance of an algorithm is measured by the cumulative cost : CT : = T∑ t=1 ct ( st , at ) . Multi-goal SSP . We also study a new problem , multi-goal SSP , a natural generalization of the classical SSP problem . In multi-goal SSP , we consider an MDP and a fixed length L ≥ 1 . The MDP M satisfies Asmp . 1 , and all of its states are incrementally L-controllable , i.e . S→L = S. Algorithm 1 : Value-Optimistic Incremental State Discovery ( VOISD ) 1 Input : MDP M = 〈S , A , P , c , s0〉 , confidence δ ∈ ( 0 , 1 ) , error parameter ε ∈ ( 0 , 1 ] , and L ≥ 1 . 2 Initialize U ← { } , K ← { } , and snew ← s0 . Specify constants c1 = 6 , c2 = 72 , c3 = 2 √ 2 , c4 = 2 √ 2 . 3 Set ε← ε/3 , δ ← δ/2 , B ← 10L and πs0 ∈ Π ( { s0 } ) . 4 Set ψ ← 12000 ( L cminε ) 2 ln ( SA δ ) , and φ← 2dlog2 ψe . 5 For ( s , a , s′ ) ∈ S ×A× S , set N ( s , a ) ← 0 ; n ( s , a ) ← 0 ; N ( s , a , s′ ) ← 0 ; P̂s , a , s′ ← 0 ; θ ( s , a ) ← 0 ; ĉ ( s , a ) ← 0 . 6 for round r = 1 , 2 , · · · do 7 \\ ( a ) Discover Possible States in S→L 8 Add snew to K. Set s← snew . 9 for each a ∈ A do 10 while N ( s , a ) < φ do 11 Execute policy πs on MDP M until reaching state s. 12 Take action a , incur cost c and observe next state s′ ∼ P ( · | s , a ) . 13 Set N ( s , a ) ← N ( s , a ) + 1 , θ ( s , a ) ← θ ( s , a ) + c , N ( s , a , s′ ) ← N ( s , a , s′ ) + 1 . 14 If s′ /∈ K , add s′ to U . 15 Set ĉ ( s , a ) ← θ ( s , a ) N ( s , a ) and θ ( s , a ) ← 0 . 16 For all s′ ∈ S , set P̂s , a , s′ ← N ( s , a , s′ ) /N ( s , a ) , n ( s , a ) ← N ( s , a ) . 17 Stop the algorithm if U is empty . 18 \\ ( b ) Compute Optimistic Policy 19 For each g ∈ U , compute ( Qg , Vg ) : = VISGO ( S , A , K , s0 , g , cminε18 ) . 20 Choose a state s ∈ arg min g∈U Vg ( s0 ) . Stop the algorithm if Vs ( s0 ) > L. 21 Set the policy π̃ as the greedy policy over Qs . Remove s from U , set snew ← s and set πs ← π̃ . 22 Output : The discovered states K and their corresponding policies { πs } s∈K . In multi-goal SSP , a learning algorithm should output a set of policies { πs } s∈S , such that V πss ( s0 ) ≤ V ∗s ( s0 ) + εL for all s ∈ S. We observe that an algorithm that solves the autonomous exploration problem with AX∗ on S→L criterion can also solve the multi-goal SSP problem .

Summary: The authors consider the incremental autonomous exploration problem. That is, the agent faces an MDP and wants to learn near-optimal goal-conditioned policies to reach the states that are L-controllable, i.e. incrementally reachable from an initial state s_0 within L steps in expectation. The learning procedure is the following: the agent collects transition by interacting with the MDP, decides to stop, and outputs a set of states (that should be the L-controllable states) and a policy for each state ( that should be the associated near-optimal goal-conditioned policy). The authors propose the Value-Aware Autonomous Exploration (VALAE) algorithms. It works in 3 phases: in the first one a sub-algorithm Value-Optimistic Incremental State Discovery (VOISD) aims at identifying the set of  L-controllable states then sub-algorithm Reduce Autonomous Exploration to Multi-Goal SSP (Re-MG-SSP) as burn-in sample collector, finally a procedure close to the one of UCBExplore by Lime and Auer (2012). They prove a bound on the sample complexity ( the number of steps before the algorithm stops) of order O(L S_{2L} A / \epsilon^2 ) where S_L is the number of  L-controllable states, A the number of actions, \epsilon the error tolerated for the goal-conditioned policy. This bound improves prior results by Lim and Auer (2012) and Tarbouriech et al. (2020) with respect to the dependence on \epsilon or L. The authors also proved a lower bound of order  \Omega(L S_{L} A / \epsilon^2 )  implying that  VALAE is nearly minimax optimal if S_L grows polynomially with L.