C-Learning: Horizon-Aware Cumulative Accessibility Estimation

1 INTRODUCTION . Multi-goal reinforcement learning tackles the challenging problem of reaching multiple goals , and as a result , is an ideal framework for real-world agents that solve a diverse set of tasks . Despite progress in this field ( Kaelbling , 1993 ; Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Ghosh et al. , 2019 ) , current algorithms suffer from a set of limitations : an inability to find multiple paths to a goal , high sample complexity , and poor results in complex motion planning tasks . In this paper we propose C-learning , a method which addresses all of these shortcomings . Many multi-goal reinforcement learning algorithms are limited by learning only a single policy π ( a|s , g ) over actions a to reach goal g from state s. There is an unexplored trade-off between reaching the goal reliably and reaching it quickly . We illustrate this shortcoming in Figure 1a , which represents an environment where an agent must reach a goal on the opposite side of some predator . Shorter paths can reach the goal faster at the cost of a higher probability of being eaten . Existing algorithms do not allow a dynamic choice of whether to act safely or quickly at test time . The second limitation is sample complexity . Despite significant improvements ( Andrychowicz et al. , 2017 ; Ghosh et al. , 2019 ) , multi-goal reaching still requires a very large amount of environment interactions for effective learning . We argue that the optimal Q-function must be learned to high accuracy for the agent to achieve reasonable performance , and this leads to sample inefficiency . The same drawback of optimal Q-functions often causes agents to learn sub-optimal ways of reaching the intended goal . This issue is particularly true for motion planning tasks ( Qureshi et al. , 2020 ) , where current algorithms struggle . We propose to address these limitations by learning horizon-aware policies π ( a|s , g , h ) , which should be followed to reach goal g from state s in at most h steps . The introduction of a time horizon h naturally allows us to tune the speed/reliability trade-off , as an agent wishing to reach the goal faster should select a policy with a suitably small h value . To learn these policies , we introduce the optimal cumulative accessibility function C∗ ( s , a , g , h ) . This is a generalization of the state-action value function and corresponds to the probability of reaching goal g from state s after at most h steps if action a is taken , and the agent acts optimally thereafter . Intuitively it is similar to the optimal Q-function , but Q-functions rarely correspond to probabilities , whereas the C∗-function does so by construction . We derive Bellman backup update rules for C∗ , which allow it to be learned via minimization of unbiased estimates of the cross-entropy loss – this is in contrast to Q-learning , which optimizes biased estimates of the squared error . Policies π ( a|s , g , h ) can then be recovered from the C∗ function . We call our method cumulative accessibility estimation , or Clearning . Pong et al . ( 2018 ) proposed TDMs , a method involving horizon-aware policies . We point out that their method is roughly related to a non-cumulative version of ours with a different loss that does not enable the speed/reliability trade-off and is ill-suited for sparse rewards . We include a detailed discussion of TDMs in section 4 . One might expect that adding an extra dimension to the learning task , namely h , would increase the difficulty - as C∗ effectively contains the information of several optimal Q-functions for different discount factors . However , we argue that C∗ does not need to be learned to the same degree of accuracy as the optimal Q-function for the agent to solve the task . As a result , learning C∗ is more efficient , and converges in fewer environmental interactions . This property , combined with our proposed goal sampling technique and replay buffer used during training , provides empirical improvements over Q-function based methods . In addition to these advantages , learning C∗ is itself useful , containing information that the horizonaware policies do not . It estimates whether a goal g is reachable from the current state s within h steps . In contrast , π ( a|s , g , h ) simply returns some action , even for unreachable goals . We show that C∗ can be used to determine reachability with examples in a nonholonomic environment . Summary of contributions : ( i ) introducing C-functions and cumulative accessibility estimation for both discrete and continuous action spaces ; ( ii ) highlighting the importance of the speed vs reliability trade-off in finite horizon reinforcement learning ; ( iii ) introducing a novel replay buffer specially tailored for learning C∗ which builds on HER ( Andrychowicz et al. , 2017 ) ; and ( iv ) empirically showing the effectiveness of our method for goal-reaching as compared to existing alternatives , particularly in the context of complex motion planning tasks . 2 BACKGROUND AND RELATED WORK . Let us extend the Markov Decision Process ( MDP ) formalism ( Sutton et al. , 1998 ) for goal-reaching . We consider a set of actionsA , a state space S , and a goal set G. We assume access to a goal checking function G : S × G → { 0 , 1 } such that G ( s , g ) = 1 if and only if state s achieves goal g. For example , achieving the goal could mean exactly reaching a certain state , in which case G = S and a a1 a2 a3 s h s1 h− 1 s2 h− 2 s3 h− 3 g . . . . . . Figure 2 : Graphical model depicting trajectories from Pπ ( ·|· , g , h ) ( ·|s0 = s , a0 = a ) . Gray nodes denote fixed values , and white nodes stochastic ones . Nodes a , g and s are non-stochastic simply because they are conditioned on , not because they are always fixed within the environment . Note that the values of h decrease deterministically . Nodes corresponding to horizons could be separated from states , but are not for a more concise graph . G ( s , g ) = 1 ( s = g ) . For many continuous state-spaces , hitting a state exactly has zero probability . Here we can still take G = S , but letG ( s , g ) = 1 ( d ( s , g ) ≤ ) for some radius and metric d. More general choices are possible . For example , in the Dubins ’ Car environment which we describe in more detail later , the state consists of both the location and orientation of the car : S = R2 × S1 . We take G = R2 , and G ( s , g ) checks that the location of the car is within some small radius of g , ignoring the direction entirely . For a fixed g , G ( s , g ) can be thought of as a sparse reward function . In the goal-reaching setting , a policy π : S×G → P ( A ) , whereP ( A ) denotes the set of distributions over A , maps state-goal pairs to an action distribution . The environment dynamics are given by a starting distribution p ( s0 , g ) , usually taken as p ( s0 ) p ( g ) , and transition probabilities p ( st+1|st , at ) . States for which G ( s , g ) = 1 are considered terminal . Q-Learning : AQ-function ( Watkins & Dayan , 1992 ) for multi-goal reaching , Qπ : S×G×A → R , is defined by Qπ ( st , g , at ) = Eπ [ ∑∞ i=t γ i−tG ( st , g ) |st , at ] , where γ ∈ [ 0 , 1 ] is a discount factor and the expectation is with respect to state-action trajectories obtained by using π ( a|si , g ) . If π∗ is an optimal policy in the sense thatQπ ∗ ( s , g , a ) ≥ Qπ ( s , g , a ) for every π and ( s , g , a ) ∈ S×G×A , then Qπ ∗ matches the optimal Q-function , Q∗ , which obeys the Bellman equation : Q∗ ( s , g , a ) = Es′∼p ( ·|s , a ) [ G ( s , g ) + γmax a′∈A Q∗ ( s′ , g , a′ ) ] . ( 1 ) In deep Q-learning ( Mnih et al. , 2015 ) , Q∗ is parameterized with a neural network and learning is achieved by enforcing the relationship from equation 1 . This is done by minimizing∑ i L ( Q∗ ( si , gi , ai ) , yi ) , where yi corresponds to the expectation in equation 1 and is estimated using a replay buffer of stored tuples ( si , ai , gi , s′i ) . Note that s ′ i is the state the environment transitioned to after taking action ai from state si , and determines the value of yi . Typically L is chosen as a squared error loss , and the dependency of yi on Q∗ is ignored for backpropagation in order to stabilize training . Once Q∗ is learned , the optimal policy is recovered by π∗ ( a|s , g ) = 1 ( a = argmaxa′ Q∗ ( s , g , a′ ) ) . There is ample work extending and improving upon deep Q-learning ( Haarnoja et al. , 2018 ) . For example , Lillicrap et al . ( 2015 ) extend it to the continuous action space setting , and Fujimoto et al . ( 2018 ) further stabilize training . These improvements are fully compatible with goal-reaching ( Pong et al. , 2019 ; Bharadhwaj et al. , 2020a ; Ghosh et al. , 2019 ) . Andrychowicz et al . ( 2017 ) proposed Hindsight Experience Replay ( HER ) , which relabels past experience as achieved goals , and allows sample efficient learning from sparse rewards ( Nachum et al. , 2018 ) . 3 CUMULATIVE ACCESSIBILITY FUNCTIONS . We now consider horizon-aware policies π : S × G × N → P ( A ) , and define the cumulative accessibility function Cπ ( s , a , g , h ) , or C-function , as the probability of reaching goal g from state s in at most h steps by taking action a and following the policy π thereafter . By “ following the policy π thereafter ” we mean that after a , the next action a1 is sampled from π ( ·|s1 , g , h− 1 ) , a2 is sampled from π ( ·|s2 , g , h− 2 ) and so on . See Figure 2 for a graphical model depiction of how these trajectories are obtained . Importantly , an agent need not always act the same way at a particular state in order to reach a particular goal , thanks to horizon-awareness . We use Pπ ( ·|· , g , h ) ( ·|s0 = s , a0 = a ) to denote probabilities in which actions are drawn in this manner and transitions are drawn according to the environment p ( st+1|st , a ) . More formally , Cπ is given by : Cπ ( s , a , g , h ) = Pπ ( ·|· , g , h ) ( max t=0 , ... , h G ( st , g ) = 1 ∣∣∣∣s0 = s , a0 = a ) . ( 2 ) Proposition 1 : Cπ can be framed as a Q-function within the MDP formalism , and if π∗ is optimal in the sense that Cπ ∗ ( s , a , g , h ) ≥ Cπ ( s , a , g , h ) for every π and ( s , a , g , h ) ∈ S ×A×G ×N , then Cπ ∗ matches the optimal C-function , C∗ , which obeys the following equation : C∗ ( s , a , g , h ) = Es′∼p ( ·|s , a ) [ max a′∈A C∗ ( s′ , a′ , g , h− 1 ) ] if G ( s , g ) = 0 and h ≥ 1 , G ( s , g ) otherwise . ( 3 ) See appendix A for a detailed mathematical proof of this proposition . The proof proceeds by first deriving a recurrence relationship that holds for any Cπ . In an analogous manner to the Bellman equation in Q-learning , this recurrence involves an expectation over π ( ·|s′ , g , h − 1 ) , which , when replaced by a max returns the recursion for C∗ . Proposition 1 is relevant as it allows us to learn C∗ , enabling goal-reaching policies to be recovered : π∗ ( a|s , g , h ) = 1 ( a = argmax a′ C∗ ( s , a′ , g , h ) ) . ( 4 ) C∗ itself is useful for determining reachability . After maximizing over actions , it estimates whether a given goal is reachable from a state within some horizon . Comparing these probabilities for different horizons allows us to make a speed / reliability trade-off for reaching goals . We observe that an optimal C∗-function is non-decreasing in h , but this does not necessarily hold for non-optimal C-functions . For example , a horizon-aware policy could actively try to avoid the goal for high values of h , and the Cπ-function constructed from it would show lower probabilities of success for larger h. See appendix A for a concrete example of this counter-intuitive behavior . Proposition 2 : C∗ is non-decreasing in h. See appendix A for a detailed mathematical proof . Intuitively , the proof consists of showing that an optimal policy can not exhibit the pathology mentioned above . Given an optimal policy π∗ ( a|s , g , h ) for a fixed horizon h we construct a policy π̃ for h + 1 which always performs better , and lower bounds the performance of π∗ ( a|s , g , h+ 1 ) . In addition to being an elegant theoretical property , proposition 2 suggests that there is additional structure in a C∗ function which mitigates the added complexity from using horizon-aware policies . Indeed , in our preliminary experiments we used a non-cumulative version of C-functions ( see section 3.3 ) and obtained significantly improved performance upon changing to C-functions . Moreover , monotonicity in h could be encoded in the architecture of C∗ ( Sill , 1998 ; Wehenkel & Louppe , 2019 ) . However , we found that actively doing so hurt empirical performance ( appendix F ) . 3.1 SHORTCOMINGS OF Q-LEARNING Before describing our method for learningC∗ , we highlight a shortcoming ofQ-learning . Consider a 2D navigation environment where an agent can move deterministically in the cardinal directions , and fix s and g. For an optimal action a∗ , the optimalQ function will achieve some valueQ∗ ( s , g , a∗ ) ∈ [ 0 , 1 ] in the sparse reward setting . Taking a sub-optimal action a− initially results in the agent taking two extra steps to reach the intended goal , given that the agent acts optimally after the first action , so that Q∗ ( s , g , a− ) = γ2Q∗ ( s , g , a∗ ) . The value of γ is typically chosen close to 1 , for example 0.99 , to ensure that future rewards are not too heavily discounted . As a consequence γ2 ≈ 1 and thus the value of Q∗ at the optimal action is very close to its value at a sub-optimal action . We illustrate this issue in Figure 1b . In this scenario , recovering an optimal policy requires that the error between the learned Q-function and Q∗ should be at most ( 1−γ2 ) /2 ; this is reflected empirically by Q-learning having high sample complexity and learning sub-optimal paths . This shortcoming surfaces in any environment where taking a sub-optimal action results in a slightly longer path than an optimal one , as in e.g . motion planning tasks . The C∗ function does not have this shortcoming . Consider the same 2D navigation example , and let h∗ be the smallest horizon for which g can be reached from s. h∗ can be easily obtained from C∗ as the smallest h such that maxa C∗ ( s , a , g , h ) = 1 . Again , denoting a∗ as an optimal action and a− as a sub-optimal one , we have that C∗ ( s , a∗ , g , h∗ ) = 1 whereas C∗ ( s , a− , g , h∗ ) = 0 , which is illustrated in Figure 1b . Therefore , the threshold for error is much higher when learning the C∗ function . This property results in fewer interactions with the environment needed to learn C∗ and more efficient solutions .

Summary: The paper highlights a problem in existing goal-reaching RL agents, in that they do not explicitly allow for trading off speed (how fast you reach the goal) and reliability (how often you reach the goal). While this tradeoff is implicitly determined by the discount factor in training, the paper asserts that in practice the ability to more flexibly determine this during inference is more desirable. Given this shortcoming of existing work, the paper then proposes "C-Learning" which learns a policy conditioned on both a goal and a desired horizon (h) -- i.e., a time limit on the policy. The paper presents favorable results of C-Learning on a few simulated domains compared to existing goal-reaching RL agents.