Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret

√ SAK ) , where K is the number of episodes , S is the number of states , A is the number of actions , and B ? bounds the expected cumulative cost of the optimal policy from any state , thus closing the gap with the lower bound . Interestingly , EB-SSP obtains this result while being parameter-free , i.e. , it does not require any prior knowledge of B ? , nor of T ? , which bounds the expected time-to-goal of the optimal policy from any state . Furthermore , we illustrate various cases ( e.g. , positive costs , or general costs when an order-accurate estimate of T ? is available ) where the regret only contains a logarithmic dependence on T ? , thus yielding the first ( nearly ) horizon-free regret bound beyond the finite-horizon MDP setting . 1 Introduction . Stochastic shortest path ( SSP ) is a goal-oriented reinforcement learning ( RL ) setting where the agent aims to reach a predefined goal state while minimizing its total expected cost [ Bertsekas , 1995 ] . In particular , the interaction between the agent and the environment ends only when ( and if ) the goal state is reached , so the length of an episode is not predetermined ( nor bounded ) and it is influenced by the agent ’ s behavior . SSP includes both finite-horizon and discounted Markov Decision Processes ( MDPs ) as special cases . Moreover , many common RL problems can be cast under the SSP formulation , such as game playing ( e.g. , Atari games ) or navigation ( e.g. , Mujoco mazes ) . We study the online learning problem in the SSP setting ( online SSP in short ) , where both the transition dynamics and the cost function are initially unknown and the agent interacts with the environment through multiple episodes . The learning objective is to achieve a performance as close ∗equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as possible to the optimal policy π ? , that is , the agent should achieve low regret ( i.e. , the cumulative difference between the total cost accumulated across episodes by the agent and by the optimal policy ) . We identify three desirable properties for a learning algorithm in online SSP . • Desired property 1 : Minimax . The information-theoretic lower bound on the regret is Ω ( B ? √ SAK ) [ Rosenberg et al. , 2020 ] , where K is the number of episodes , S is the number of states , A is the number of actions , and B ? bounds the total expected cost of the optimal policy starting from any state ( assuming for simplicity that B ? ≥ 1 ) . An algorithm for online SSP is ( nearly ) minimax optimal if its regret is bounded by Õ ( B ? √ SAK ) , up to logarithmic factors and lower-order terms . • Desired property 2 : Parameter-free . Another relevant dimension is the amount of prior knowledge required by the algorithm . While the knowledge of S , A , and the cost ( or reward ) range [ 0 , 1 ] is standard across regret-minimization settings ( e.g. , finite-horizon , discounted , average-reward ) , the complexity of learning in SSP problems may be linked to SSP-specific quantities such as B ? and T ? , which denotes the expected time-to-goal of the optimal policy from any state . An algorithm for online SSP is parameter-free if it relies neither on T ? nor B ? prior knowledge . • Desired property 3 : Horizon-free . A core challenge in SSP is to trade off between minimizing costs and quickly reaching the goal state . This is accentuated when the instantaneous costs are small , i.e. , when there is a mismatch between B ? and T ? . Indeed , while B ? ≤ T ? always holds since the cost range is [ 0 , 1 ] , the gap between the two may be arbitrarily large ( see e.g. , the simple example of App . A ) . The lower bound stipulates that the regret does depend on B ? , while the “ time horizon ” of the problem , i.e. , T ? should a priori not impact the regret , even as a lower-order term . An algorithm for online SSP is ( nearly ) horizon-free if its regret depends only logarithmically on T ? . Our definition extends the property of so-called horizon-free bounds recently uncovered in finitehorizon MDPs with total reward bounded by 1 [ Wang et al. , 2020a , Zhang et al. , 2021a , b ] . These bounds depend only logarithmically on the horizon H , which is the number of time steps by which any policy terminates . Such notion of horizon would clearly be too strong in the more general class of SSP , where some ( even most ) policies may never reach the goal , thus having unbounded time horizon . A more adequate notion of horizon in SSP is T ? , which bounds the expected time of the optimal policy to terminate the episode starting from any state . Finally , while the previous properties focus on the learning aspects of the algorithm , another important consideration is computational efficiency . It is desirable that a learning algorithm has run-time complexity at most polynomial in K , S , A , B ? , and T ? . All existing algorithms for online SSP , including the one proposed in this paper , meet such requirement . Related Work . Table 1 reviews the existing work on online learning in SSP . The setting was first studied by Tarbouriech et al . [ 2020a ] who gave a parameter-free algorithm with a Õ ( K3/2 ) regret guarantee . Rosenberg et al . [ 2020 ] then improved this result by deriving the first order-optimal algorithm with regret Õ ( B3/2 ? S √ AK ) in the parameter-free case and Õ ( B ? S √ AK ) if B ? is known ( to tune cost perturbation appropriately ) . Both approaches are model-optimistic,2 drawing inspiration from the ideas behind the UCRL2 algorithm [ Jaksch et al. , 2010 ] for average-reward MDPs . Concurrently to our work , Cohen et al . [ 2021 ] propose an algorithm for online SSP based on a blackbox reduction from SSP to finite-horizon MDPs . It successively tackles finite-horizon problems with horizon set to H = Ω ( T ? ) and costs augmented by a terminal cost set to cH ( s ) = Ω ( B ? I ( s 6= g ) ) , where g denotes the goal state . This finite-horizon construction guarantees that its optimal policy has a similar value function to the optimal policy in the original SSP instance up to a lower-order bias . Their algorithm comes with a regret bound of O ( B ? √ SAKL+ T 4 ? S 2AL5 ) , with L = log ( KT ? SAδ−1 ) ( with probability at least 1 − δ ) . It achieves a nearly minimax-optimal rate , however it relies on both T ? and B ? prior knowledge to tune the horizon and terminal cost in the reduction , respectively.3 2We refer the reader to Neu and Pike-Burke [ 2020 ] for details on the differences and interplay between model-optimistic and value-optimistic approaches . 3As mentioned by Cohen et al . [ 2021 , Remark 2 ] , in the case of positive costs lower bounded by cmin > 0 , their knowledge of T ? can be bypassed by replacing it with the upper bound T ? ≤ B ? /cmin . However , when generalizing from the cmin case to general costs with a perturbation argument , their regret guarantee worsens from Õ ( √ K + c−4min ) to Õ ( K 4/5 ) , because of the poor additive dependence on c−1min . Finally , all existing bounds contain lower-order dependencies either on T ? in the case of general costs , or on B ? /cmin in the case of positive costs lower bounded by cmin > 0 ( note that T ? ≤ B ? /cmin , which is one of the reasons why cmin can show up in existing bounds ) . As such , no existing analysis satisfies horizon-free properties for online SSP . Contributions . We summarize our main contributions as follows ( see also Table 1 ) : • We propose EB-SSP ( Exploration Bonus for SSP ) , a new algorithm for online SSP . It introduces a value-optimistic scheme to efficiently compute optimistic policies for SSP , by both perturbing the empirical costs with an exploration bonus and slightly biasing the empirical transitions towards reaching the goal from each state-action pair with positive probability . Under these biased transitions , all policies are in fact proper ( i.e. , they eventually reach the goal with probability 1 starting from any state ) . We decay the bias over time in a way that it only contributes to a lower-order regret term . See Sect . 3 for an overview of our algorithm and analysis . Note that EB-SSP is not based on a model-optimistic approach2 [ Tarbouriech et al. , 2020a , Rosenberg et al. , 2020 ] , and it does not rely on a reduction from SSP to finite-horizon [ Cohen et al. , 2021 ] ( i.e. , we operate at the level of the non-truncated SSP model ) ; • EB-SSP is the first algorithm to achieve the minimax regret rate of Õ ( B ? √ SAK ) while simulta- neously being parameter-free : it does not require to know nor estimate T ? , and it is able to bypass the knowledge of B ? at the cost of only logarithmic and lower-order contributions to the regret ; • EB-SSP is the first algorithm to achieve horizon-free regret for SSP in various cases : i ) positive costs , ii ) no almost-sure zero-cost cycles , and iii ) the general cost case when an order-accurate estimate of T ? is available ( i.e. , a value T ? such that T ? υ ≤ T ? ≤ λT ζ ? for some unknown constants υ , λ , ζ ≥ 1 is available ) . This property is especially relevant if T ? is much larger than B ? , which can occur in SSP models with very small instantaneous costs . Moreover , EB-SSP achieves its horizon-free guarantees while maintaining the minimax rate . For instance , under general costs when relying on T ? andB ? , its regret is Õ ( B ? √ SAK+B ? S 2A ) .4 To the best of our knowledge , EB-SSP yields the first set of ( nearly ) horizon-free bounds beyond the setting of finite-horizon MDPs . Additional Related Work . Planning in SSP : Early work by Bertsekas and Tsitsiklis [ 1991 ] , followed by [ e.g. , Bertsekas , 1995 , Bonet , 2007 , Kolobov et al. , 2011 , Bertsekas and Yu , 2013 , Guillot and Stauffer , 2020 ] , examine the planning problem in SSP , i.e. , how to compute an optimal policy when all parameters of the SSP model are known . Under mild assumptions , the optimal policy is deterministic and stationary and can be computed efficiently using standard planning techniques , e.g. , value iteration , policy iteration or linear programming . Regret minimization in MDPs : The exploration-exploitation dilemma in tabular MDPs has been extensively studied in finite-horizon [ e.g. , Azar et al. , 2017 , Jin et al. , 2018 , Zanette and Brunskill , 2019 , Efroni et al. , 2019 , Simchowitz and Jamieson , 2019 , Zhang et al. , 2020 , Neu and Pike-Burke , 2020 , Xu et al. , 2021 , Menard et al. , 2021 ] and infinite-horizon [ e.g. , Jaksch et al. , 2010 , Bartlett and Tewari , 2012 , Fruit et al. , 2018 , Wang et al. , 2020b , Qian et al. , 2019 , Wei et al. , 2020 ] . Other SSP-based settings : SSP with adversarial costs was investigated by Rosenberg and Mansour [ 2021 ] , Chen et al . [ 2021 ] , Chen and Luo [ 2021 ] .5 Tarbouriech et al . [ 2021 ] study the sample complexity of SSP with a generative model , as a standard regret-to-PAC conversion may not hold in SSP ( as opposed to finite-horizon ) . Exploration problems involving multiple goal states ( i.e. , multigoal SSP or goal-conditioned RL ) were analyzed by Lim and Auer [ 2012 ] , Tarbouriech et al . [ 2020b ] .

Summary: This paper studies regret minimization in stochastic shortest paths. It presents an algorithm with regret that is: minimax optimal, parameter-free and horizon-free (in various cases). Here, horizon-free refers to logarithmic dependence in the expected time of the optimal policy (that replaces the horizon in finite-horizon problems).