Implicit Finite-Horizon Approximation and Efficient Optimal Algorithms for Stochastic Shortest Path

We introduce a generic template for developing regret minimization algorithms in the Stochastic Shortest Path ( SSP ) model , which achieves minimax optimal regret as long as certain properties are ensured . The key of our analysis is a new technique called implicit finite-horizon approximation , which approximates the SSP model by a finite-horizon counterpart only in the analysis without explicit implementation . Using this template , we develop two new algorithms : the first one is model-free ( the first in the literature to our knowledge ) and minimax optimal under strictly positive costs ; the second one is model-based and minimax optimal even with zero-cost state-action pairs , matching the best existing result from [ Tarbouriech et al. , 2021b ] . Importantly , both algorithms admit highly sparse updates , making them computationally more efficient than all existing algorithms . Moreover , both can be made completely parameter-free . 1 Introduction . We study the Stochastic Shortest Path ( SSP ) model , where an agent aims to reach a goal state with minimum cost in a stochastic environment . SSP is well-suited for modeling many real-world applications , such as robotic manipulation , car navigation , and others . Although it is widely studied empirically ( e.g. , [ Andrychowicz et al. , 2017 , Nasiriany et al. , 2019 ] ) and in optimal control theory ( e.g. , [ Bertsekas and Tsitsiklis , 1991 , Bertsekas and Yu , 2013 ] ) , it has received less attention under the regret minimization setting where a learner needs to learn the environment and improve her policy on-the-fly through repeated interaction . Specifically , the problem proceeds in K episodes . In each episode , the learner starts at a fixed initial state , sequentially takes action , suffers some cost , and transits to the next state , until reaching a predefined goal state . The performance of the learner is measured by her regret , which is the difference between her total costs and that of the best policy . Tarbouriech et al . [ 2020a ] develop the first regret minimization algorithm for SSP with a regret bound of Õ ( D3/2S √ AK/cmin ) , where D is the diameter , S is the number of states , A is the number of actions , and cmin is the minimum cost among all state-action pairs . Cohen et al . [ 2020 ] improve over their results and give a near optimal regret bound of Õ ( B ? S √ AK ) , where B ? ≤ D is the largest expected cost of the optimal policy starting from any state . Even more recently , Cohen et al . [ 2021 ] achieve minimax regret of Õ ( B ? √ SAK ) through a finite-horizon reduction technique , and concurrently Tarbouriech et al . [ 2021b ] also propose minimax optimal and parameter-free algorithms . Notably , all existing algorithms are model-based with space complexity Ω ( S2A ) . Moreover , they all 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . update the learner ’ s policy through full-planning ( a term taken from [ Efroni et al. , 2019 ] ) , incurring a relatively high time complexity . In this work , we further advance the state-of-the-art by proposing a generic template for regret minimization algorithms in SSP ( Algorithm 1 ) , which achieves minimax optimal regret as long as some properties are ensured . By instantiating our template differently , we make the following two key algorithmic contributions : • In Section 4 , we develop the first model-free SSP algorithm called LCB-ADVANTAGE-SSP ( Algorithm 2 ) . Similar to most model-free reinforcement learning algorithms , LCB-ADVANTAGESSP does not estimate the transition directly , enjoys a space complexity of Õ ( SA ) , and also takes only O ( 1 ) time to update certain statistics in each step , making it a highly efficient algorithm . It achieves a regret bound of Õ ( B ? √ SAK + B5 ? S 2A/c4min ) , which is minimax optimal when cmin > 0 . Moreover , it can be made parameter-free without worsening the regret bound . • In Section 5 , we develop another simple model-based algorithm called SVI-SSP ( Algorithm 3 ) , which achieves minimax regret Õ ( B ? √ SAK +B ? S 2A ) even when cmin = 0 , matching the best existing result by Tarbouriech et al . [ 2021b ] .1 Notably , compared to their algorithm ( as well as other model-based algorithms ) , SVI-SSP is computationally much more efficient since it updates each state-action pair only logarithmically many times , and each update only performs one-step planning ( again , a term taken from [ Efroni et al. , 2019 ] ) as opposed to full-planning ( such as value iteration or extended value iteration ) ; see more concrete time complexity comparisons in Section 5 . SVI-SSP can also be made parameter-free following the idea of [ Tarbouriech et al. , 2021b ] . We include a summary of regret bounds of all existing SSP algorithms as well as more complexity comparisons in Appendix A . Techniques Our main technical contribution is a new analysis framework called implicit finitehorizon approximation ( Section 3 ) , which is the key to analyze algorithms developed from our template . The high level idea is to approximate an SSP instance by a finite-horizon counterpart . However , the approximation only happens in the analysis , a key difference compared to [ Chen et al. , 2021 , Chen and Luo , 2021 , Cohen et al. , 2021 ] that explicitly implement such an approximation in their algorithms . As a result , our method not only avoids blowing up the space complexity by a factor of the horizon , but also allows one to derive a horizon-free regret bound ( more explanation to follow ) . In order to achieve the minimax optimal regret , our model-free algorithm LCB-ADVANTAGE-SSP uses a key variance reduction idea via a reference-advantage decomposition by [ Zhang et al. , 2020b ] . However , crucial distinctions exist . For example , we update the reference value function more frequently instead of only one time , which helps reduce the sample complexity and improve the lower-order term in the regret bound . We also maintain an empirical upper bound on the value function in a doubling manner , which is the key to eventually make the algorithm parameter-free . On the other hand , for our model-based algorithm SVI-SSP , we adopt a special Bernstein-style bonus term and bound the learner ’ s total variance via recursion , taking inspiration from [ Tarbouriech et al. , 2021b , Zhang et al. , 2020a ] . Empirical Evaluation We support our theoretical findings with experiments in Appendix H. Our model-free algorithm demonstrates a better convergence rate compared to vanilla Q learning with naive -greedy exploration . Our model-based algorithm has competitive performance compared to other model-based algorithms , while spending the least amount of time in updates . Related Work For a detailed comparison of existing results for the same problem , we refer the readers to [ Tarbouriech et al. , 2021b , Table 1 ] as well as our Table 1 . There are also several works [ Rosenberg and Mansour , 2020 , Chen et al. , 2021 , Chen and Luo , 2021 ] that consider the even more challenging SSP setting where the cost function is decided by an adversary and can change over time . Apart from regret minimization , Tarbouriech et al . [ 2021a ] study the sample complexity of SSP with a generative model ; Lim and Auer [ 2012 ] and Tarbouriech et al . [ 2020b ] investigate exploration problems involving multiple goal states ( multi-goal SSP ) . 1Depending on the available prior knowledge , the final bounds achieved by SVI-SSP are slightly different , but they all match that of EB-SSP . See [ Tarbouriech et al. , 2021b , Table 1 ] for more details . The special case of SSP with a fixed horizon has been studied extensively , for both stochastic costs ( e.g. , [ Azar et al. , 2017 , Jin et al. , 2018 , Efroni et al. , 2019 , Zanette and Brunskill , 2019 , Zhang et al. , 2020a ] ) and adversarial costs ( e.g. , [ Neu et al. , 2012 , Zimin and Neu , 2013 , Rosenberg and Mansour , 2019 , Jin et al. , 2020 ] ) . Importantly , recent works [ Wang et al. , 2020 , Zhang et al. , 2020a ] find that when the cost for each episode is at most a constant , it is in fact possible to obtain a regret bound with only logarithmic dependency on the horizon . Tarbouriech et al . [ 2021b ] generalize this concept to SSP and define horizon-free regret as a bound with only logarithmic dependence on the expected hitting time of the optimal policy starting from any state ( which is bounded by B ? /cmin ) . They also propose the first algorithm with horizon-free regret for SSP , which is important for arguing minimax optimality even when cmin = 0 . Notably , our model-based algorithm SVI-SSP also achieves horizon-free regret ( but the model-free one does not ) . 2 Preliminaries . An SSP instance is defined by a Markov Decision Process ( MDP ) M = ( S , A , sinit , g , c , P ) , where S is the state space , A is the action space , sinit ∈ S is the initial state , and g /∈ S is the goal state . When taking action a in state s , the learner suffers a cost drawn in an i.i.d manner from an unknown distribution with mean c ( s , a ) ∈ [ 0 , 1 ] and support [ cmin , 1 ] ( cmin ≥ 0 ) , and then transits to the next state s′ ∈ S+ = S ∪ { g } with probability Ps , a ( s′ ) . We assume that the transition P and the cost mean c are unknown to the learner , while all other parameters are known . The learning process goes as follows : the learner interacts with the environment for K episodes . In the k-th episode , the learner starts in initial state sinit , sequentially takes an action , suffers a cost , and transits to the next state until reaching the goal state g. More formally , at the i-th step of the k-th episode , the learner observes the current state ski ( with s k 1 = sinit ) , takes action a k i , suffers a cost c k i , and transits to the next state ski+1 ∼ Pski , aki . An episode ends when the current state is g , and we define the length of episode k as Ik , such that skIk+1 = g. Learning Objective At a high level , the learner ’ s goal is to reach the goal with a small total cost . To this end , we focus on proper policies — a ( stationary and deterministic ) policy π : S → A is a mapping that assigns an action π ( s ) to each state s ∈ S , and it is proper if the goal is reached with probability 1 when following π ( that is , taking action π ( s ) whenever in state s ) . Given a proper policy π , one can define the cost-to-go function V π : S → [ 0 , ∞ ) as V π ( s ) = E [ ∑I i=1 ci ∣∣∣P , π , s1 = s ] , where the expectation is with respect to the randomness of the cost ci incurred at state-action pair ( si , π ( si ) ) , next state si+1 ∼ Psi , π ( si ) , and the number of steps I before reaching g. The optimal proper policy π ? is then defined as a policy such that V π ? ( s ) = minπ∈Π V π ( s ) for all s ∈ S , where Π is the set of all proper policies assumed to be nonempty . The formal objective of the learner is then to minimize her regret against π ? , the difference between her total cost and that of the optimal proper policy , defined as RK = K∑ k=1 Ik∑ i=1 cki −K · V ? ( sinit ) , where we use V ? as a shorthand for V π ? . The minimax optimal regret is known to be Õ ( B ? √ SAK ) , where B ? = maxs∈S V ? ( s ) , and S = |S+| and A = |A| are the numbers of states ( including the goal state ) and actions respectively [ Cohen et al. , 2020 ] . Bellman Optimality Equation For a proper policy π , the corresponding action-value function Qπ : S × A → [ 0 , ∞ ) is defined as Qπ ( s , a ) = c ( s , a ) + Es′∼Ps , a [ V π ( s′ ) ] . Similarly , we use Q ? as a shorthand for Qπ ? . it is known that π ? satisfies the Bellman optimality equation : V ? ( s ) = mina∈AQ ? ( s , a ) for all s ∈ S [ Bertsekas and Tsitsiklis , 1991 ] . Assumption on cmin Similar to many previous works , our analysis requires cmin being known and strictly positive . When cmin is unknown or known to be 0 , a simple workaround is to solve a modified SSP instance with all observed costs clipped to if they are below some > 0 , so that cmin = > 0 . Then the regret in this modified SSP is similar to that in the original SSP up to an additive term of order O ( K ) [ Tarbouriech et al. , 2020a ] . Therefore , throughout the paper we assume that cmin is known and strictly positive unless explicitly stated otherwise . Algorithm 1 A General Algorithmic Template for SSP Initialize : t← 0 , s1 ← sinit , Q ( s , a ) ← 0 for all ( s , a ) ∈ S ×A . for k = 1 , . . . , K do repeat 1 Increment time step t +← 1 . 2 Take action at = argminaQ ( st , a ) , suffer cost ct , transit to and observe s ′ t. 3 Update Q ( so that it satisfies Property 1 and Property 2 ) . 4 if s′t 6= g then st+1 ← s′t ; else st+1 ← sinit , break . Record T ← t ( that is , the total number of steps ) . Other Notations For simplicity , we use CK = ∑K k=1 ∑Ik i=1 c k i in the analysis to denote the total costs suffered by the learner over K episodes . For a function X : S+ → R and a distribution P over S+ , denote by PX = ES∼P [ X ( S ) ] , PX2 = ES∼P [ X ( S ) 2 ] , and V ( P , X ) = VARS∼P [ X ( S ) ] the expectation , second moment , and variance of X ( S ) respectively where S is drawn from P . For a scalar x , define ( x ) + = max { x , 0 } , and denote by dxe2 = 2dlog2 xe and bxc2 = 2blog2 xc the closest power of two upper and lower bounding x respectively . For an integer m , [ m ] denotes the set { 1 , . . . , m } . In pseudocode , x +← y is a shorthand for the increment operation x← x+ y .

Summary: This paper presents an algorithmic template for the SSP model that unifies algorithms based on value optimism. Then, two instances of the template are presented and analyzed. The first instance is a model-free algorithm (the first for SSP) that achieves optimal regret but only when all costs are strictly positive. The second is a model-based algorithm that achieves optimal regret without any assumption on the cost. This algorithm enjoys improved computational complexity by performing 1-step planning instead of full-planning (also the first for SSP).