Shortest-Path Constrained Reinforcement Learning for Sparse Reward Tasks

1 INTRODUCTION . Recently , deep reinforcement learning ( RL ) has achieved a large number of breakthroughs in many domains including video games ( Mnih et al. , 2015 ; Vinyals et al. , 2019 ) , and board games ( Silver et al. , 2017 ) . Nonetheless , a central challenge in reinforcement learning ( RL ) is the sample efficiency ( Kakade et al. , 2003 ) ; it has been shown that the RL algorithm requires a large number of samples for successful learning in MDP with large state and action space . Moreover , the success of RL algorithm heavily hinges on the quality of collected samples ; the RL algorithm tends to fail if the collected trajectory does not contain enough evaluative feedback ( e.g. , sparse or delayed reward ) . To circumvent this challenge , planning-based methods utilize the environment ’ s model to improve or create a policy instead of interacting with environment . Recently , combining the planning method with an efficient path search algorithm , such as Monte-Carlo tree search ( MCTS ) ( Norvig , 2002 ; Coulom , 2006 ) , has demonstrated successful results ( Guo et al. , 2016 ; Vodopivec et al. , 2017 ; Silver et al. , 2017 ) . However , such tree search methods would require an accurate model of MDP and the complexity of planning may grow intractably large for complex domain . Model-based RL methods attempt to learn a model instead of assuming that model is given , but learning an accurate model also requires a large number of samples , which is often even harder to achieve than solving the given task . Model-free RL methods can be learned solely from the environment reward , without the need of a ( learned ) model . However , both value-based and policy-based methods suffer from poor sample efficiency especially in sparse-reward tasks . To tackle sparse reward problems , researchers have proposed to learn an intrinsic bonus function that measures the novelty of the state that agent visits ( Schmidhuber , 1991 ; Oudeyer & Kaplan , 2009 ; Pathak et al. , 2017 ; Savinov et al. , 2018b ; Choi et al. , 2018 ; Burda et al. , 2018 ) . However , when such intrinsic bonus is added to the reward , it often requires a careful balancing between environment reward and bonus and scheduling of the bonus scale in order to guarantee the convergence to optimal solution . To tackle aforementioned challenge of sample efficiency in sparse reward tasks , we introduce a constrained-RL framework that improves the sample efficiency of any model-free RL algorithm in sparse-reward tasks , under the mild assumptions on MDP ( see Appendix G ) . Of note , though our framework will be formulated for policy-based methods , our final form of cost function ( Eq . ( 10 ) in Section 4 ) is applicable to both policy-based and value-based methods . We propose a novel k-shortest-path ( k-SP ) constraint ( Definition 7 ) that improves sample efficiency of policy learning ( See Figure 1 ) . The k-SP constraint is applied to a trajectory rolled out by a policy ; all of its sub-path of length k is required to be a shortest-path under the π-distance metric which we define in Section 3.1 . We prove that applying our constraint preserves the optimality for any MDP ( Theorem 3 ) , except the stochastic and multi-goal MDP which requires additional assumptions . We relax the hard constraint into a soft cost formulation ( Tessler et al. , 2019 ) , and use a reachability network ( Savinov et al. , 2018b ) ( RNet ) to efficiently learn the cost function in an off-policy manner . We summarize our contributions as the following : ( 1 ) We propose a novel constraint that can improve the sample efficiency of any model-free RL method in sparse reward tasks . ( 2 ) We present several theoretical results including the proof that our proposed constraint preserves the optimal policy of given MDP . ( 3 ) We present a numerical result in tabular RL setting to precisely evaluate the effectiveness of the proposed method . ( 4 ) We propose a practical way to implement our proposed constraint , and demonstrate that it provides a significant improvement on two complex deep RL domains . ( 5 ) We demonstrate that our method significantly improves the sample-efficiency of PPO , and outperforms existing novelty-seeking methods on two complex domains in sparse reward setting . 2 PRELIMINARIES . Markov Decision Process ( MDP ) . We model a task as an MDP tuple M = ( S , A , P , R , ρ , γ ) , where S is a state set , A is an action set , P is a transition probability , R is a reward function , ρ is an initial state distribution , and γ ∈ [ 0 , 1 ) is a discount factor . For each state s , the value of a policy π is denoted by V π ( s ) = Eπ [ ∑ t γ trt | s0 = s ] . Then , the goal is to find the optimal policy π∗ that maximizes the expected return : π∗ = arg max π Eπs∼ρ [ ∑ t γ trt | s0 = s ] = arg max π Es∼ρ [ V π ( s ) ] . ( 1 ) Constrained MDP . A constrained Markov Decision Process ( CMDP ) is an MDP with extra constraints that restrict the domain of allowed policies ( Altman , 1999 ) . Specifically , CMDP introduces a constraint functionC ( π ) that maps a policy to a scalar , and a thresholdα ∈ R. The objective of CMDP is to maximize the expected returnR ( τ ) = ∑ t γ trt of a trajectory τ = { s0 , a0 , r1 , s1 , a1 , r2 , s2 , . . . } subject to a constraint : π∗ = arg maxπ Eτ∼π [ R ( τ ) ] , s.t . C ( π ) ≤ α . A popular choice of constraint is based on the transition cost function ( Tessler et al. , 2019 ) c ( s , a , r , s′ ) ∈ R which assigns a scalar-valued cost to each transition . Then the constraint function for a policy π is defined as the discounted sum of the cost under the policy : C ( π ) = Eτ∼π [ ∑ t γ tc ( st , at , rt+1 , st+1 ) ] . In this work , we propose a shortest-path constraint , that provably preserves the optimal policy of the original unconstrained MDP , while reducing the trajectory space . We will use a cost function-based formulation to implement our constraint ( see Section 3 and 4 ) . 3 FORMULATION : k-SHORTEST PATH CONSTRAINT We define the k-shortest-path ( k-SP ) constraint to remove redundant transitions ( e.g. , unnecessarily going back and forth ) , leading to faster policy learning . We show two important properties of our constraint : ( 1 ) the optimal policy is preserved , and ( 2 ) the policy search space is reduced . In this work , we limit our focus to MDPs satisfying R ( s ) +γV ∗ ( s ) > 0 for all initial states s ∈ ρ and all rewarding states that optimal policy visits with non-zero probability s ∈ { s|r ( s ) 6= 0 , π∗ ( s ) > 0 } . We exploit this mild assumption to prove that our constraint preserves optimality . Intuitively , we exclude the case when the optimal strategy for the agent is at best choosing a “ lesser of evils ” ( i.e. , largest but negative value ) which often still means a failure . We note that this is often caused by unnatural reward function design ; in principle , we can avoid this by simply offsetting reward function by a constant −|mins∈ { s|π∗ ( s ) > 0 } V ∗ ( s ) | for every transition , assuming the policy is proper1 . Goalconditioned RL ( Nachum et al. , 2018 ) and most of the well-known domains such as Atari ( Bellemare et al. , 2013 ) , DeepMind Lab ( Beattie et al. , 2016 ) , MiniGrid ( Chevalier-Boisvert et al. , 2018 ) , etc. , satisfy this assumption . Also , for general settings with stochastic MDP and multi-goals , we require additional assumptions to prove the optimality guarantee ( See Appendix G for details ) . 3.1 SHORTEST-PATH POLICY AND SHORTEST-PATH CONSTRAINT . Let τ be a path defined by a sequence of states : τ = { s0 , . . . , s ` ( τ ) } , where ` ( τ ) is the length of a path τ ( i.e. , ` ( τ ) = |τ | − 1 ) . We denote the set of all paths from s to s′ by Ts , s′ . A path τ∗ from s to s′ is called a shortest path from s to s′ if ` ( τ ) is minimum , i.e. , ` ( τ∗ ) = minτ∈Ts , s′ ` ( τ ) . Now we will define similar concepts ( length , shortest path , etc . ) with respect to a policy . Intuitively , a policy that rolls out shortest paths ( up to some stochasticity ) to a goal state or between any state pairs should be a counterpart . We consider a set of all admissible paths from s to s′ under a policy π : Definition 1 ( Path set ) . T πs , s′ = { τ | s0 = s , s ` ( τ ) = s′ , pπ ( τ ) > 0 , { st } t < ` ( τ ) 6= s′ } . That is , T πs , s′ is a set of all paths that policy π may roll out from s and terminate once visiting s′ . If the MDP is a single-goal task , i.e. , there exists a unique ( rewarding ) goal state sg ∈ S such that sg is a terminal state , and R ( s ) > 0 if and only if s = sg , any shortest path from an initial state to the goal state is the optimal path with the highest return R ( τ ) , and a policy that rolls out a shortest path is therefore optimal ( see Lemma 4 ) .2 This is because all states except for sg are non-rewarding states , but in general MDPs this is not necessarily true . However , this motivates us to limit the domain of shortest path to among non-rewarding states . We define non-rewarding paths from s to s′ as follows : Definition 2 ( Non-rewarding path set ) . T πs , s′ , nr = { τ | τ ∈ T πs , s′ , { rt } t < ` ( τ ) = 0 } . In words , T πs , s′ , nr is a set of all non-rewarding paths from s to s′ rolled out by policy π ( i.e. , τ ∈ T πs , s′ ) without any associated reward except the last step ( i.e. , { rt } t < |τ | = 0 ) . Now we are ready to define a notion of length with respect to a policy and shortest path policy : Definition 3 ( π-distance from s to s′ ) . Dπnr ( s , s′ ) = logγ ( Eτ∼π : τ∈T π s , s′ , nr [ γ ` ( τ ) ] ) Definition 4 ( Shortest path distance from s to s′ ) . Dnr ( s , s′ ) = minπDπnr ( s , s′ ) . We define π-distance to be the log-mean-exponential of the length ` ( τ ) of non-rewarding paths τ ∈ T πs , s′ , nr . When there exists no admissible path from s to s′ under policy π , the path length is defined to be ∞ : Dπnr ( s , s′ ) = ∞ if T πs , s′ , nr = ∅ . We note that when both MDP and policy are deterministic , Dπ ( s , s′ ) recovers the natural definition of path length , Dπnr ( s , s ′ ) = ` ( τ ) . We call a policy a shortest-path policy from s to s′ if it roll outs a path with the smallest π-distance : Definition 5 ( Shortest path policy from s to s′ ) . π ∈ ΠSPs→s′ = { π ∈ Π | Dπnr ( s , s′ ) = Dnr ( s , s′ ) } . Finally , we will define the shortest-path ( SP ) constraint . Let SIR = { s | R ( s ) > 0 or ρ ( s ) > 0 } be the union of all initial and rewarding states , and Φπ = { ( s , s′ ) | s , s′ ∈ SIR , ρ ( s ) > 0 , T πs , s′ , nr 6= ∅ } be the subset of SIR such that agent may roll out . Then , the SP constraint is applied to the nonrewarding sub-paths between states in Φπ : T πΦ , nr = ⋃ ( s , s′ ) ∈Φπ T πs , s′ , nr . We note that these definitions are used in the proofs ( Appendix G ) . Now , we define the shortest-path constraint as follows : Definition 6 ( Shortest-path constraint ) . A policy π satisfies the shortest-path ( SP ) constraint if π ∈ ΠSP , where ΠSP = { π | For all s , s′ ∈ T πΦ , nr , it holds π ∈ ΠSPs→s′ } . 1It is an instance of potential-based reward shaping which has optimality guarantee ( Ng et al. , 1999 ) . 2We refer the readers to Appendix F for more detailed discussion and proofs for single-goal MDPs . Intuitively , the SP constraint forces a policy to transition between initial and rewarding states via shortest paths . The SP constraint would be particularly effective in sparse-reward settings , where the distance between rewarding states is large . Given these definitions , we can show that an optimal policy indeed satisfies the SP constraint in a general MDP setting . In other words , the shortest path constraint should not change optimality : Theorem 1 . For any MDP , an optimal policy π∗ satisfies the shortest-path constraint : π∗ ∈ ΠSP . Proof . See Appendix G for the proof . 3.2 RELAXATION : k-SHORTEST-PATH CONSTRAINT Implementing the shortest-path constraint is , however , intractable since it requires a distance predictor Dnr ( s , s ′ ) . Note that the distance predictor addresses the optimization problem , which might be as difficult as solving the given task . To circumvent this challenge , we consider its more tractable version , namely a k-shortest path constraint , which reduces the shortest-path problem Dnr ( s , s′ ) to a binary decision problem — is the state s′ reachable from s within k steps ? — also known as k-reachability ( Savinov et al. , 2018b ) . The k-shortest path constraint is defined as follows : Definition 7 ( k-shortest-path constraint ) . A policy π satisfies the k-shortest-path constraint if π ∈ ΠSPk , where ΠSPk = { π | For all s , s′ ∈ T πΦ , nr , Dπnr ( s , s′ ) ≤ k , it holds π ∈ ΠSPs→s′ } . ( 2 ) Note that the SP constraint ( Definition 6 ) is relaxed by adding a condition Dπnr ( s , s ′ ) ≤ k. In other words , the k-SP constraint is imposed only for s , s′-path whose length is not greater than k. From Eq . ( 2 ) , we can prove an important property and then Theorem 3 ( optimality ) : Lemma 2 . For an MDPM , ΠSPm ⊂ ΠSPk if k < m. Proof . It is true since { ( s , s′ ) | Dπnr ( s , s′ ) ≤ k } ⊂ { ( s , s′ ) | Dπnr ( s , s′ ) ≤ m } for k < m. Theorem 3 . For an MDPM and any k ∈ R , an optimal policy π∗ is a k-shortest-path policy . Proof . Theorem 1 tells π∗ ∈ ΠSP . Eq . ( 2 ) tells ΠSP = ΠSP∞ and Lemma 2 tells ΠSP∞ ⊂ ΠSPk . Collectively , we have π∗ ∈ ΠSP = ΠSP∞ ⊂ ΠSPk . In conclusion , Theorem 3 states that the k-SP constraint does not change the optimality of policy , and Lemma 2 states a larger k results in a larger reduction in policy search space . Thus , it motivates us to apply the k-SP constraint in policy search to more efficiently find an optimal policy . For the numerical experiment on measuring the reduction in the policy roll-outs space , please refer to Section 6.4 .

Summary: This paper proposes a new constraint for constrained MDP, based on k-shortest path, which helps improve sample efficiency for (model-free) RL algorithms in sparse-reward MDP, while theoretically proving that the constraint retains the same optimal policy in the original MDP. Intuitively, for sparse (positive) reward setting, the optimal policy should reach the positive reward states with the shortest path (as it has the lowest discounting). The relaxed form of the constraint considers the that the distance between two states is less than $k$, rather than being optimal length (which the optimal policy still also satisfies). The constraint is then converted into its Lagrangian form as a cost term to the reward (i.e. a type of reward shaping). Practically, this requires a k-reachability network (RNet), which is a binary distance discriminator judging whether the distance between two states are reachable within $k$ steps. This network is trained with contrastive loss, similar to prior work SPTM by Savinov et al., 2018. However, SPTM uses RNet for graph-based planning (i.e. the local distance between states), while this paper uses RNet as the cost/constraint on the policy objective function. Experiments were conducted in several maze navigation environments (2D grid world MiniGrid, to first person 3D maze environments in DeepMind Lab), showing promising results compared to several baselines which use intrinsic curiosity. Several ablations were performed on the hyperparameters ($k$, and tolerance $\delta t$ on the constraint), as well as some qualitative examples of the policies learned compared to novelty reward shaping.