Minimax Regret for Stochastic Shortest Path

√ ( B2 ? + B ? ) |S||A|K ) where B ? is a bound on the expected cost of the optimal policy from any state , S is the state space , and A is the action space . This matches the Ω ( √ B2 ? |S||A|K ) lower bound of Rosenberg et al . [ 2020 ] for B ? ≥ 1 , and improves their regret bound by a factor of √ |S| . For B ? < 1 we prove a matching lower bound of Ω ( √ B ? |S||A|K ) . Our algorithm is based on a novel reduction from SSP to finite-horizon MDPs . To that end , we provide an algorithm for the finitehorizon setting whose leading term in the regret depends polynomially on the expected cost of the optimal policy and only logarithmically on the horizon . 1 Introduction . We study the stochastic shortest path ( SSP ) problem in which an agent aims to reach a predefined goal state while minimizing her total expected cost . This is one of the most basic models of reinforcement learning ( RL ) that includes both finite-horizon and discounted Markov Decision Processes ( MDPs ) as special cases . In addition , SSP captures a wide variety of realistic scenarios such as car navigation , game playing and drone flying . We study an online version of SSP in which both the immediate costs and transition distributions of the model are initially unknown to the agent . The agent interacts with the model for K episodes , in each of which she attempts to reach the goal state with minimal cumulative cost . A main challenge in the online model is found when instantaneous costs are small . For example , any learning algorithm that attempts to myopically minimize the accumulated costs might get caught in a cycle with zero cost and never reach the goal state . Nonetheless , even if the costs are not zero , only very small , the agent must be able to trade off the need to minimize costs with that of reaching the goal quickly . The online setting was originally suggested by Tarbouriech et al . [ 2020 ] who gave an algorithm with Õ ( K2/3 ) regret guarantee . In a follow-up work , Rosenberg et al . [ 2020 ] improved the previous bound to Õ ( B ? |S| √ |A|K ) , where S is the state space , A is the action space , and B ? is an upper bound on the total expected cost of the optimal policy when initialized at any state . Rosenberg et al . [ 2020 ] also provide a lower bound of Ω ( B ? √ |S||A|K ) – leaving a gap of √ |S| between the upper and lower bounds . In this work , unlike the previously mentioned works that assume the cost function is deterministic 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and known , we consider the case where the costs are i.i.d . and initially unknown . We prove upper and lower bounds for this case , proving that the optimal regret is of order Θ̃ ( √ ( B2 ? + B ? ) |S||A|K ) . The algorithms of both Tarbouriech et al . [ 2020 ] , Rosenberg et al . [ 2020 ] were based on a direct application of the “ Optimism in the Face of Uncertainty ” principle to the SSP model , following the ideas behind the UCRL2 algorithm [ Jaksch et al. , 2010 ] for average-reward MDPs . In this work we take a different approach . We propose a novel black-box reduction to finite-horizon MDPs , showing that the SSP problem is not harder than the finite-horizon setting assuming prior knowledge on the expected time it takes for the optimal policy to reach the goal state . While the reduction itself is simple , the analysis is highly nontrivial as one has to show that the goal state is indeed reached in every episode without incurring excessive costs in the process . The idea of reducing SSP to finite-horizon was previously used by Chen et al . [ 2020 ] , Chen and Luo [ 2021 ] for SSP with adversarially changing costs . However , they run one finite-horizon episode in every SSP episode and then simply try to reach the goal as fast as possible , while we restart a new finite-horizon episode every H steps . This modification is what enables us to obtain the optimal and improved dependence in the number of states . In addition , we provide a new algorithm for regret minimization in finite-horizon MDPs called ULCVI . We show that ( for large enough number of episodes ) its regret depends polynomially on the expected cost of the optimal policy B ? , and only logarithmically on the horizon length H. This implies that the correct measure for the regret is the expected cost of the optimal policy and not the length of the horizon . We note that regret with logarithmic dependence in the horizon H was also obtained by Zhang et al . [ 2020 ] , yet they make a much stronger assumption : that the cumulative cost of every trajectory is bounded by 1 . In contrast , we only assume that the expected cost of the optimal policy is bounded by some constant B ? , while other policies may suffer a cost of H. Our reduction , when combined with our finite-horizon algorithm ULCVI , guarantees SSP regret of Õ ( √ ( B2 ? + B ? ) |S||A|K ) . This matches the lower bound of Rosenberg et al . [ 2020 ] for B ? ≥ 1 up to logarithmic factors . However , their lower bound does not hold for B ? < 1 suggesting that this is not the correct rate in this case . Indeed , we prove a tighter lower bound of Ω ( √ B ? |S||A|K ) for B ? < 1 , showing that our regret guarantees are minimax optimal in all cases . As a final remark we note that , following our work , Tarbouriech et al . [ 2021 ] were able to obtain a comparable regret bound for SSP without prior knowledge of the optimal policy ’ s expected time to reach the goal state . 1.1 Additional related work . Planning for stochastic shortest path . Early work by Bertsekas and Tsitsiklis [ 1991 ] studied planning in SSPs , i.e. , computing the optimal strategy efficiently when parameters are known . Under certain assumptions , they established that the optimal strategy is a deterministic stationary policy and can be computed efficiently using standard planning algorithms , e.g. , Value Iteration and LP . Adversarial stochastic shortest path . Rosenberg and Mansour [ 2020 ] presented stochastic shortest path with adversarially changing costs . Their regret bounds were improved by Chen et al . [ 2020 ] , Chen and Luo [ 2021 ] using a reduction to online loop-free SSP ( see next paragraph ) . As mentioned before , our reduction is different and therefore able to remove the extra √ |S| factor in the regret . Regret minimization in MDPs . There is a vast literature on regret minimization in RL that mostly builds on the optimism principle . Most literature focuses on the tabular setting [ Jaksch et al. , 2010 , Azar et al. , 2017 , Jin et al. , 2018 , Fruit et al. , 2018 , Zanette and Brunskill , 2019 , Efroni et al. , 2019 , Simchowitz and Jamieson , 2019 ] , but recently it was extended to function approximation under various assumptions [ Yang and Wang , 2019 , Jin et al. , 2020b , Zanette et al. , 2020a , b ] . Online loop-free SSP . A different line of work considers finite-horizon MDPs with adversarially changing costs [ Neu et al. , 2010 , 2012 , Zimin and Neu , 2013 , Rosenberg and Mansour , 2019b , a , Jin et al. , 2020a , Cai et al. , 2020 , Shani et al. , 2020 , Lancewicki et al. , 2020 , Lee et al. , 2020 , Jin and Luo , 2020 ] . They refer to finite-horizon adversarial MDPs as online loop-free SSP . This is not to be confused with our setting in which the interaction between the agent and the environment ends only when ( and if ) the goal state is reached , and not after a fixed number of steps H. See Rosenberg and Mansour [ 2020 ] , Chen et al . [ 2020 ] for a discussion on the differences between the models . 2 Preliminaries and main results . An instance of the SSP problem is defined by an MDPM = ( S , A , P , c , sinit , g ) where S is a finite state space and A is a finite action space . The agent begins at an initial state sinit ∈ S , and ends her interaction withM by arriving at the goal state g ( where g 6∈ S ) . Whenever she plays action a in state s , she pays a cost C ∈ [ 0 , 1 ] drawn i.i.d . from a distribution with expectation c ( s , a ) ∈ [ 0 , 1 ] and the next state s′ ∈ S ∪ { g } is chosen with probability P ( s′ | s , a ) . Note that the transition function P satisfies ∑ s′∈S∪ { g } P ( s ′ | s , a ) = 1 for every ( s , a ) ∈ S× A . Proper policies . A stationary and deterministic policy π : S 7→ A is a mapping that selects action π ( s ) whenever the agent is at state s. A policy π is called proper if playing according to π ensures that the goal state is reached with probability 1 when starting from any state ( otherwise it is improper ) . In SSP , the agent has two goals : ( a ) reach the goal state ; ( b ) minimize the total expected cost . To facilitate the first goal , we make the basic assumption that there exists at least one proper policy . In particular , the goal state is reachable from every state , which is clearly a necessary assumption . Any policy π induces a cost-to-go function Jπ : S 7→ [ 0 , ∞ ] . The cost-to-go at state s is defined by Jπ ( s ) = limT→∞ Eπ [ ∑T t=1 c ( st , at ) | sinit = s ] , where the expectation is taken w.r.t the random sequence of states generated by playing according to π when the initial state is s. For a proper policy π , it follows that Jπ ( s ) is finite for all s ∈ S. However , note that Jπ ( s ) may be finite even if π is improper . We additionally denote by Tπ ( s ) the expected time it takes for π to reach g starting at state s ; in particular , if π is proper then Tπ ( s ) is finite for all s ∈ S , and if π is improper there must exist some state s such that Tπ ( s ) =∞ . Learning formulation . Here , the agent does not have any prior knowledge of the cost function c or transition function P. She interacts with the model in episodes : each episode starts at the fixed initial state sinit,1 and ends when the agent reaches the goal state g ( note that she might never reach the goal state ) . Success is measured by the agent ’ s regret over K such episodes , that is the difference between her total cost over the K episodes and the total expected cost of the optimal proper policy : RK = K∑ k=1 Ik∑ i=1 Cki – K · min π∈Πproper Jπ ( sinit ) , where Ik is the time it takes the agent to complete episode k ( which may be infinite ) , Cki is the cost suffered in the i-th step of episode k when the agent visited state-action pair ( ski , a k i ) , and Πproper is the set of all stationary , deterministic and proper policies ( that is not empty by assumption ) . In the case that Ik is infinite for some k , we define RK =∞ . We denote the optimal proper policy by π ? , Jπ ? ( sinit ) = arg minπ∈Πproper J π ( sinit ) . Moreover , let B ? > 0 be an upper bound on the values of Jπ ? and let T ? > 0 be an upper bound on the times Tπ ? , i.e. , B ? ≥ maxs∈S Jπ ? ( s ) and T ? ≥ maxs∈S Tπ ? ( s ) . Finally , let D = maxs∈S minπ∈Πproper T π ( s ) be the SSP-diameter , and note that B ? ≤ D ≤ T ? .

Summary: The paper proposed a novel reduction from stochastic shortest path (SSP) to finite-horizon Markov decision process (MDP). Based on this approach, the paper proposes a model-based algorithm, showing that the minimax regret is $\tilde{O}(\sqrt{(B_{\star}^2+B_{\star})|S||A|K})$, where where $B_{\star}$ is a bound on the expected cost of the optimal policy from any state, $S$ is the state space, and $A$ is the action space. This upper bound attains the lower bound $\Omega(\sqrt{B_{\star}^2|S||A|K})$ for $B_{\star} \ge 1$. The paper also prove a matching lower bound $\Omega(\sqrt{B_{\star}|S||A|K})$ for $B_{\star} < 1$. In addition, the paper provides a new algorithm for regret minimization in finite-horizon MDPs and the regret depends only logarithmically on the horizon length.