Implicit Finite-Horizon Approximation and Efficient Optimal Algorithms for Stochastic Shortest Path

We introduce a generic template for developing regret minimization algorithms in the Stochastic Shortest Path ( SSP ) model , which achieves minimax optimal regret as long as certain properties are ensured . The key of our analysis is a new technique called implicit finite-horizon approximation , which approximates the SSP model by a finite-horizon counterpart only in the analysis without explicit implementation . Using this template , we develop two new algorithms : the first one is model-free ( the first in the literature to our knowledge ) and minimax optimal under strictly positive costs ; the second one is model-based and minimax optimal even with zero-cost state-action pairs , matching the best existing result from [ Tarbouriech et al. , 2021b ] . Importantly , both algorithms admit highly sparse updates , making them computationally more efficient than all existing algorithms . Moreover , both can be made completely parameter-free . 1 Introduction . We study the Stochastic Shortest Path ( SSP ) model , where an agent aims to reach a goal state with minimum cost in a stochastic environment . SSP is well-suited for modeling many real-world applications , such as robotic manipulation , car navigation , and others . Although it is widely studied empirically ( e.g. , [ Andrychowicz et al. , 2017 , Nasiriany et al. , 2019 ] ) and in optimal control theory ( e.g. , [ Bertsekas and Tsitsiklis , 1991 , Bertsekas and Yu , 2013 ] ) , it has received less attention under the regret minimization setting where a learner needs to learn the environment and improve her policy on-the-fly through repeated interaction . Specifically , the problem proceeds in K episodes . In each episode , the learner starts at a fixed initial state , sequentially takes action , suffers some cost , and transits to the next state , until reaching a predefined goal state . The performance of the learner is measured by her regret , which is the difference between her total costs and that of the best policy . Tarbouriech et al . [ 2020a ] develop the first regret minimization algorithm for SSP with a regret bound of Õ ( D3/2S √ AK/cmin ) , where D is the diameter , S is the number of states , A is the number of actions , and cmin is the minimum cost among all state-action pairs . Cohen et al . [ 2020 ] improve over their results and give a near optimal regret bound of Õ ( B ? S √ AK ) , where B ? ≤ D is the largest expected cost of the optimal policy starting from any state . Even more recently , Cohen et al . [ 2021 ] achieve minimax regret of Õ ( B ? √ SAK ) through a finite-horizon reduction technique , and concurrently Tarbouriech et al . [ 2021b ] also propose minimax optimal and parameter-free algorithms . Notably , all existing algorithms are model-based with space complexity Ω ( S2A ) . Moreover , they all 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . update the learner ’ s policy through full-planning ( a term taken from [ Efroni et al. , 2019 ] ) , incurring a relatively high time complexity . In this work , we further advance the state-of-the-art by proposing a generic template for regret minimization algorithms in SSP ( Algorithm 1 ) , which achieves minimax optimal regret as long as some properties are ensured . By instantiating our template differently , we make the following two key algorithmic contributions : • In Section 4 , we develop the first model-free SSP algorithm called LCB-ADVANTAGE-SSP ( Algorithm 2 ) . Similar to most model-free reinforcement learning algorithms , LCB-ADVANTAGESSP does not estimate the transition directly , enjoys a space complexity of Õ ( SA ) , and also takes only O ( 1 ) time to update certain statistics in each step , making it a highly efficient algorithm . It achieves a regret bound of Õ ( B ? √ SAK + B5 ? S 2A/c4min ) , which is minimax optimal when cmin > 0 . Moreover , it can be made parameter-free without worsening the regret bound . • In Section 5 , we develop another simple model-based algorithm called SVI-SSP ( Algorithm 3 ) , which achieves minimax regret Õ ( B ? √ SAK +B ? S 2A ) even when cmin = 0 , matching the best existing result by Tarbouriech et al . [ 2021b ] .1 Notably , compared to their algorithm ( as well as other model-based algorithms ) , SVI-SSP is computationally much more efficient since it updates each state-action pair only logarithmically many times , and each update only performs one-step planning ( again , a term taken from [ Efroni et al. , 2019 ] ) as opposed to full-planning ( such as value iteration or extended value iteration ) ; see more concrete time complexity comparisons in Section 5 . SVI-SSP can also be made parameter-free following the idea of [ Tarbouriech et al. , 2021b ] . We include a summary of regret bounds of all existing SSP algorithms as well as more complexity comparisons in Appendix A . Techniques Our main technical contribution is a new analysis framework called implicit finitehorizon approximation ( Section 3 ) , which is the key to analyze algorithms developed from our template . The high level idea is to approximate an SSP instance by a finite-horizon counterpart . However , the approximation only happens in the analysis , a key difference compared to [ Chen et al. , 2021 , Chen and Luo , 2021 , Cohen et al. , 2021 ] that explicitly implement such an approximation in their algorithms . As a result , our method not only avoids blowing up the space complexity by a factor of the horizon , but also allows one to derive a horizon-free regret bound ( more explanation to follow ) . In order to achieve the minimax optimal regret , our model-free algorithm LCB-ADVANTAGE-SSP uses a key variance reduction idea via a reference-advantage decomposition by [ Zhang et al. , 2020b ] . However , crucial distinctions exist . For example , we update the reference value function more frequently instead of only one time , which helps reduce the sample complexity and improve the lower-order term in the regret bound . We also maintain an empirical upper bound on the value function in a doubling manner , which is the key to eventually make the algorithm parameter-free . On the other hand , for our model-based algorithm SVI-SSP , we adopt a special Bernstein-style bonus term and bound the learner ’ s total variance via recursion , taking inspiration from [ Tarbouriech et al. , 2021b , Zhang et al. , 2020a ] . Empirical Evaluation We support our theoretical findings with experiments in Appendix H. Our model-free algorithm demonstrates a better convergence rate compared to vanilla Q learning with naive -greedy exploration . Our model-based algorithm has competitive performance compared to other model-based algorithms , while spending the least amount of time in updates . Related Work For a detailed comparison of existing results for the same problem , we refer the readers to [ Tarbouriech et al. , 2021b , Table 1 ] as well as our Table 1 . There are also several works [ Rosenberg and Mansour , 2020 , Chen et al. , 2021 , Chen and Luo , 2021 ] that consider the even more challenging SSP setting where the cost function is decided by an adversary and can change over time . Apart from regret minimization , Tarbouriech et al . [ 2021a ] study the sample complexity of SSP with a generative model ; Lim and Auer [ 2012 ] and Tarbouriech et al . [ 2020b ] investigate exploration problems involving multiple goal states ( multi-goal SSP ) . 1Depending on the available prior knowledge , the final bounds achieved by SVI-SSP are slightly different , but they all match that of EB-SSP . See [ Tarbouriech et al. , 2021b , Table 1 ] for more details . The special case of SSP with a fixed horizon has been studied extensively , for both stochastic costs ( e.g. , [ Azar et al. , 2017 , Jin et al. , 2018 , Efroni et al. , 2019 , Zanette and Brunskill , 2019 , Zhang et al. , 2020a ] ) and adversarial costs ( e.g. , [ Neu et al. , 2012 , Zimin and Neu , 2013 , Rosenberg and Mansour , 2019 , Jin et al. , 2020 ] ) . Importantly , recent works [ Wang et al. , 2020 , Zhang et al. , 2020a ] find that when the cost for each episode is at most a constant , it is in fact possible to obtain a regret bound with only logarithmic dependency on the horizon . Tarbouriech et al . [ 2021b ] generalize this concept to SSP and define horizon-free regret as a bound with only logarithmic dependence on the expected hitting time of the optimal policy starting from any state ( which is bounded by B ? /cmin ) . They also propose the first algorithm with horizon-free regret for SSP , which is important for arguing minimax optimality even when cmin = 0 . Notably , our model-based algorithm SVI-SSP also achieves horizon-free regret ( but the model-free one does not ) . 2 Preliminaries . An SSP instance is defined by a Markov Decision Process ( MDP ) M = ( S , A , sinit , g , c , P ) , where S is the state space , A is the action space , sinit ∈ S is the initial state , and g /∈ S is the goal state . When taking action a in state s , the learner suffers a cost drawn in an i.i.d manner from an unknown distribution with mean c ( s , a ) ∈ [ 0 , 1 ] and support [ cmin , 1 ] ( cmin ≥ 0 ) , and then transits to the next state s′ ∈ S+ = S ∪ { g } with probability Ps , a ( s′ ) . We assume that the transition P and the cost mean c are unknown to the learner , while all other parameters are known . The learning process goes as follows : the learner interacts with the environment for K episodes . In the k-th episode , the learner starts in initial state sinit , sequentially takes an action , suffers a cost , and transits to the next state until reaching the goal state g. More formally , at the i-th step of the k-th episode , the learner observes the current state ski ( with s k 1 = sinit ) , takes action a k i , suffers a cost c k i , and transits to the next state ski+1 ∼ Pski , aki . An episode ends when the current state is g , and we define the length of episode k as Ik , such that skIk+1 = g. Learning Objective At a high level , the learner ’ s goal is to reach the goal with a small total cost . To this end , we focus on proper policies — a ( stationary and deterministic ) policy π : S → A is a mapping that assigns an action π ( s ) to each state s ∈ S , and it is proper if the goal is reached with probability 1 when following π ( that is , taking action π ( s ) whenever in state s ) . Given a proper policy π , one can define the cost-to-go function V π : S → [ 0 , ∞ ) as V π ( s ) = E [ ∑I i=1 ci ∣∣∣P , π , s1 = s ] , where the expectation is with respect to the randomness of the cost ci incurred at state-action pair ( si , π ( si ) ) , next state si+1 ∼ Psi , π ( si ) , and the number of steps I before reaching g. The optimal proper policy π ? is then defined as a policy such that V π ? ( s ) = minπ∈Π V π ( s ) for all s ∈ S , where Π is the set of all proper policies assumed to be nonempty . The formal objective of the learner is then to minimize her regret against π ? , the difference between her total cost and that of the optimal proper policy , defined as RK = K∑ k=1 Ik∑ i=1 cki −K · V ? ( sinit ) , where we use V ? as a shorthand for V π ? . The minimax optimal regret is known to be Õ ( B ? √ SAK ) , where B ? = maxs∈S V ? ( s ) , and S = |S+| and A = |A| are the numbers of states ( including the goal state ) and actions respectively [ Cohen et al. , 2020 ] . Bellman Optimality Equation For a proper policy π , the corresponding action-value function Qπ : S × A → [ 0 , ∞ ) is defined as Qπ ( s , a ) = c ( s , a ) + Es′∼Ps , a [ V π ( s′ ) ] . Similarly , we use Q ? as a shorthand for Qπ ? . it is known that π ? satisfies the Bellman optimality equation : V ? ( s ) = mina∈AQ ? ( s , a ) for all s ∈ S [ Bertsekas and Tsitsiklis , 1991 ] . Assumption on cmin Similar to many previous works , our analysis requires cmin being known and strictly positive . When cmin is unknown or known to be 0 , a simple workaround is to solve a modified SSP instance with all observed costs clipped to if they are below some > 0 , so that cmin = > 0 . Then the regret in this modified SSP is similar to that in the original SSP up to an additive term of order O ( K ) [ Tarbouriech et al. , 2020a ] . Therefore , throughout the paper we assume that cmin is known and strictly positive unless explicitly stated otherwise . Algorithm 1 A General Algorithmic Template for SSP Initialize : t← 0 , s1 ← sinit , Q ( s , a ) ← 0 for all ( s , a ) ∈ S ×A . for k = 1 , . . . , K do repeat 1 Increment time step t +← 1 . 2 Take action at = argminaQ ( st , a ) , suffer cost ct , transit to and observe s ′ t. 3 Update Q ( so that it satisfies Property 1 and Property 2 ) . 4 if s′t 6= g then st+1 ← s′t ; else st+1 ← sinit , break . Record T ← t ( that is , the total number of steps ) . Other Notations For simplicity , we use CK = ∑K k=1 ∑Ik i=1 c k i in the analysis to denote the total costs suffered by the learner over K episodes . For a function X : S+ → R and a distribution P over S+ , denote by PX = ES∼P [ X ( S ) ] , PX2 = ES∼P [ X ( S ) 2 ] , and V ( P , X ) = VARS∼P [ X ( S ) ] the expectation , second moment , and variance of X ( S ) respectively where S is drawn from P . For a scalar x , define ( x ) + = max { x , 0 } , and denote by dxe2 = 2dlog2 xe and bxc2 = 2blog2 xc the closest power of two upper and lower bounding x respectively . For an integer m , [ m ] denotes the set { 1 , . . . , m } . In pseudocode , x +← y is a shorthand for the increment operation x← x+ y .

Summary: The authors propose a generic template for regret minimization algorithms in SSP. They develop the first efficient model-free SSP algorithm. They introduced a technique called implicit finite-horizon approximation, which approximates the SSP model by a finite-horizon counterpart implicitly.