Stochastic Shortest Path: Minimax, Parameter-Free and Towards Horizon-Free Regret

√ SAK ) , where K is the number of episodes , S is the number of states , A is the number of actions , and B ? bounds the expected cumulative cost of the optimal policy from any state , thus closing the gap with the lower bound . Interestingly , EB-SSP obtains this result while being parameter-free , i.e. , it does not require any prior knowledge of B ? , nor of T ? , which bounds the expected time-to-goal of the optimal policy from any state . Furthermore , we illustrate various cases ( e.g. , positive costs , or general costs when an order-accurate estimate of T ? is available ) where the regret only contains a logarithmic dependence on T ? , thus yielding the first ( nearly ) horizon-free regret bound beyond the finite-horizon MDP setting . 1 Introduction . Stochastic shortest path ( SSP ) is a goal-oriented reinforcement learning ( RL ) setting where the agent aims to reach a predefined goal state while minimizing its total expected cost [ Bertsekas , 1995 ] . In particular , the interaction between the agent and the environment ends only when ( and if ) the goal state is reached , so the length of an episode is not predetermined ( nor bounded ) and it is influenced by the agent ’ s behavior . SSP includes both finite-horizon and discounted Markov Decision Processes ( MDPs ) as special cases . Moreover , many common RL problems can be cast under the SSP formulation , such as game playing ( e.g. , Atari games ) or navigation ( e.g. , Mujoco mazes ) . We study the online learning problem in the SSP setting ( online SSP in short ) , where both the transition dynamics and the cost function are initially unknown and the agent interacts with the environment through multiple episodes . The learning objective is to achieve a performance as close ∗equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . as possible to the optimal policy π ? , that is , the agent should achieve low regret ( i.e. , the cumulative difference between the total cost accumulated across episodes by the agent and by the optimal policy ) . We identify three desirable properties for a learning algorithm in online SSP . • Desired property 1 : Minimax . The information-theoretic lower bound on the regret is Ω ( B ? √ SAK ) [ Rosenberg et al. , 2020 ] , where K is the number of episodes , S is the number of states , A is the number of actions , and B ? bounds the total expected cost of the optimal policy starting from any state ( assuming for simplicity that B ? ≥ 1 ) . An algorithm for online SSP is ( nearly ) minimax optimal if its regret is bounded by Õ ( B ? √ SAK ) , up to logarithmic factors and lower-order terms . • Desired property 2 : Parameter-free . Another relevant dimension is the amount of prior knowledge required by the algorithm . While the knowledge of S , A , and the cost ( or reward ) range [ 0 , 1 ] is standard across regret-minimization settings ( e.g. , finite-horizon , discounted , average-reward ) , the complexity of learning in SSP problems may be linked to SSP-specific quantities such as B ? and T ? , which denotes the expected time-to-goal of the optimal policy from any state . An algorithm for online SSP is parameter-free if it relies neither on T ? nor B ? prior knowledge . • Desired property 3 : Horizon-free . A core challenge in SSP is to trade off between minimizing costs and quickly reaching the goal state . This is accentuated when the instantaneous costs are small , i.e. , when there is a mismatch between B ? and T ? . Indeed , while B ? ≤ T ? always holds since the cost range is [ 0 , 1 ] , the gap between the two may be arbitrarily large ( see e.g. , the simple example of App . A ) . The lower bound stipulates that the regret does depend on B ? , while the “ time horizon ” of the problem , i.e. , T ? should a priori not impact the regret , even as a lower-order term . An algorithm for online SSP is ( nearly ) horizon-free if its regret depends only logarithmically on T ? . Our definition extends the property of so-called horizon-free bounds recently uncovered in finitehorizon MDPs with total reward bounded by 1 [ Wang et al. , 2020a , Zhang et al. , 2021a , b ] . These bounds depend only logarithmically on the horizon H , which is the number of time steps by which any policy terminates . Such notion of horizon would clearly be too strong in the more general class of SSP , where some ( even most ) policies may never reach the goal , thus having unbounded time horizon . A more adequate notion of horizon in SSP is T ? , which bounds the expected time of the optimal policy to terminate the episode starting from any state . Finally , while the previous properties focus on the learning aspects of the algorithm , another important consideration is computational efficiency . It is desirable that a learning algorithm has run-time complexity at most polynomial in K , S , A , B ? , and T ? . All existing algorithms for online SSP , including the one proposed in this paper , meet such requirement . Related Work . Table 1 reviews the existing work on online learning in SSP . The setting was first studied by Tarbouriech et al . [ 2020a ] who gave a parameter-free algorithm with a Õ ( K3/2 ) regret guarantee . Rosenberg et al . [ 2020 ] then improved this result by deriving the first order-optimal algorithm with regret Õ ( B3/2 ? S √ AK ) in the parameter-free case and Õ ( B ? S √ AK ) if B ? is known ( to tune cost perturbation appropriately ) . Both approaches are model-optimistic,2 drawing inspiration from the ideas behind the UCRL2 algorithm [ Jaksch et al. , 2010 ] for average-reward MDPs . Concurrently to our work , Cohen et al . [ 2021 ] propose an algorithm for online SSP based on a blackbox reduction from SSP to finite-horizon MDPs . It successively tackles finite-horizon problems with horizon set to H = Ω ( T ? ) and costs augmented by a terminal cost set to cH ( s ) = Ω ( B ? I ( s 6= g ) ) , where g denotes the goal state . This finite-horizon construction guarantees that its optimal policy has a similar value function to the optimal policy in the original SSP instance up to a lower-order bias . Their algorithm comes with a regret bound of O ( B ? √ SAKL+ T 4 ? S 2AL5 ) , with L = log ( KT ? SAδ−1 ) ( with probability at least 1 − δ ) . It achieves a nearly minimax-optimal rate , however it relies on both T ? and B ? prior knowledge to tune the horizon and terminal cost in the reduction , respectively.3 2We refer the reader to Neu and Pike-Burke [ 2020 ] for details on the differences and interplay between model-optimistic and value-optimistic approaches . 3As mentioned by Cohen et al . [ 2021 , Remark 2 ] , in the case of positive costs lower bounded by cmin > 0 , their knowledge of T ? can be bypassed by replacing it with the upper bound T ? ≤ B ? /cmin . However , when generalizing from the cmin case to general costs with a perturbation argument , their regret guarantee worsens from Õ ( √ K + c−4min ) to Õ ( K 4/5 ) , because of the poor additive dependence on c−1min . Finally , all existing bounds contain lower-order dependencies either on T ? in the case of general costs , or on B ? /cmin in the case of positive costs lower bounded by cmin > 0 ( note that T ? ≤ B ? /cmin , which is one of the reasons why cmin can show up in existing bounds ) . As such , no existing analysis satisfies horizon-free properties for online SSP . Contributions . We summarize our main contributions as follows ( see also Table 1 ) : • We propose EB-SSP ( Exploration Bonus for SSP ) , a new algorithm for online SSP . It introduces a value-optimistic scheme to efficiently compute optimistic policies for SSP , by both perturbing the empirical costs with an exploration bonus and slightly biasing the empirical transitions towards reaching the goal from each state-action pair with positive probability . Under these biased transitions , all policies are in fact proper ( i.e. , they eventually reach the goal with probability 1 starting from any state ) . We decay the bias over time in a way that it only contributes to a lower-order regret term . See Sect . 3 for an overview of our algorithm and analysis . Note that EB-SSP is not based on a model-optimistic approach2 [ Tarbouriech et al. , 2020a , Rosenberg et al. , 2020 ] , and it does not rely on a reduction from SSP to finite-horizon [ Cohen et al. , 2021 ] ( i.e. , we operate at the level of the non-truncated SSP model ) ; • EB-SSP is the first algorithm to achieve the minimax regret rate of Õ ( B ? √ SAK ) while simulta- neously being parameter-free : it does not require to know nor estimate T ? , and it is able to bypass the knowledge of B ? at the cost of only logarithmic and lower-order contributions to the regret ; • EB-SSP is the first algorithm to achieve horizon-free regret for SSP in various cases : i ) positive costs , ii ) no almost-sure zero-cost cycles , and iii ) the general cost case when an order-accurate estimate of T ? is available ( i.e. , a value T ? such that T ? υ ≤ T ? ≤ λT ζ ? for some unknown constants υ , λ , ζ ≥ 1 is available ) . This property is especially relevant if T ? is much larger than B ? , which can occur in SSP models with very small instantaneous costs . Moreover , EB-SSP achieves its horizon-free guarantees while maintaining the minimax rate . For instance , under general costs when relying on T ? andB ? , its regret is Õ ( B ? √ SAK+B ? S 2A ) .4 To the best of our knowledge , EB-SSP yields the first set of ( nearly ) horizon-free bounds beyond the setting of finite-horizon MDPs . Additional Related Work . Planning in SSP : Early work by Bertsekas and Tsitsiklis [ 1991 ] , followed by [ e.g. , Bertsekas , 1995 , Bonet , 2007 , Kolobov et al. , 2011 , Bertsekas and Yu , 2013 , Guillot and Stauffer , 2020 ] , examine the planning problem in SSP , i.e. , how to compute an optimal policy when all parameters of the SSP model are known . Under mild assumptions , the optimal policy is deterministic and stationary and can be computed efficiently using standard planning techniques , e.g. , value iteration , policy iteration or linear programming . Regret minimization in MDPs : The exploration-exploitation dilemma in tabular MDPs has been extensively studied in finite-horizon [ e.g. , Azar et al. , 2017 , Jin et al. , 2018 , Zanette and Brunskill , 2019 , Efroni et al. , 2019 , Simchowitz and Jamieson , 2019 , Zhang et al. , 2020 , Neu and Pike-Burke , 2020 , Xu et al. , 2021 , Menard et al. , 2021 ] and infinite-horizon [ e.g. , Jaksch et al. , 2010 , Bartlett and Tewari , 2012 , Fruit et al. , 2018 , Wang et al. , 2020b , Qian et al. , 2019 , Wei et al. , 2020 ] . Other SSP-based settings : SSP with adversarial costs was investigated by Rosenberg and Mansour [ 2021 ] , Chen et al . [ 2021 ] , Chen and Luo [ 2021 ] .5 Tarbouriech et al . [ 2021 ] study the sample complexity of SSP with a generative model , as a standard regret-to-PAC conversion may not hold in SSP ( as opposed to finite-horizon ) . Exploration problems involving multiple goal states ( i.e. , multigoal SSP or goal-conditioned RL ) were analyzed by Lim and Auer [ 2012 ] , Tarbouriech et al . [ 2020b ] .

Summary: This paper investigates the problem of regret minimization of the stochastic shortest path, which is actively studied in the recent two years. The main contribution is a novel algorithm that enjoys nearly minimax optimal regret, matching the existing lower bound up to some log factors. Furthermore, the algorithm, as well as its variant, enjoy other desired properties, including parameter-free (does not require the knowledge of $B_\star$ or $T_\star$) and horizon-free (does not depend on $T_\star$)