C-Learning: Horizon-Aware Cumulative Accessibility Estimation

1 INTRODUCTION . Multi-goal reinforcement learning tackles the challenging problem of reaching multiple goals , and as a result , is an ideal framework for real-world agents that solve a diverse set of tasks . Despite progress in this field ( Kaelbling , 1993 ; Schaul et al. , 2015 ; Andrychowicz et al. , 2017 ; Ghosh et al. , 2019 ) , current algorithms suffer from a set of limitations : an inability to find multiple paths to a goal , high sample complexity , and poor results in complex motion planning tasks . In this paper we propose C-learning , a method which addresses all of these shortcomings . Many multi-goal reinforcement learning algorithms are limited by learning only a single policy π ( a|s , g ) over actions a to reach goal g from state s. There is an unexplored trade-off between reaching the goal reliably and reaching it quickly . We illustrate this shortcoming in Figure 1a , which represents an environment where an agent must reach a goal on the opposite side of some predator . Shorter paths can reach the goal faster at the cost of a higher probability of being eaten . Existing algorithms do not allow a dynamic choice of whether to act safely or quickly at test time . The second limitation is sample complexity . Despite significant improvements ( Andrychowicz et al. , 2017 ; Ghosh et al. , 2019 ) , multi-goal reaching still requires a very large amount of environment interactions for effective learning . We argue that the optimal Q-function must be learned to high accuracy for the agent to achieve reasonable performance , and this leads to sample inefficiency . The same drawback of optimal Q-functions often causes agents to learn sub-optimal ways of reaching the intended goal . This issue is particularly true for motion planning tasks ( Qureshi et al. , 2020 ) , where current algorithms struggle . We propose to address these limitations by learning horizon-aware policies π ( a|s , g , h ) , which should be followed to reach goal g from state s in at most h steps . The introduction of a time horizon h naturally allows us to tune the speed/reliability trade-off , as an agent wishing to reach the goal faster should select a policy with a suitably small h value . To learn these policies , we introduce the optimal cumulative accessibility function C∗ ( s , a , g , h ) . This is a generalization of the state-action value function and corresponds to the probability of reaching goal g from state s after at most h steps if action a is taken , and the agent acts optimally thereafter . Intuitively it is similar to the optimal Q-function , but Q-functions rarely correspond to probabilities , whereas the C∗-function does so by construction . We derive Bellman backup update rules for C∗ , which allow it to be learned via minimization of unbiased estimates of the cross-entropy loss – this is in contrast to Q-learning , which optimizes biased estimates of the squared error . Policies π ( a|s , g , h ) can then be recovered from the C∗ function . We call our method cumulative accessibility estimation , or Clearning . Pong et al . ( 2018 ) proposed TDMs , a method involving horizon-aware policies . We point out that their method is roughly related to a non-cumulative version of ours with a different loss that does not enable the speed/reliability trade-off and is ill-suited for sparse rewards . We include a detailed discussion of TDMs in section 4 . One might expect that adding an extra dimension to the learning task , namely h , would increase the difficulty - as C∗ effectively contains the information of several optimal Q-functions for different discount factors . However , we argue that C∗ does not need to be learned to the same degree of accuracy as the optimal Q-function for the agent to solve the task . As a result , learning C∗ is more efficient , and converges in fewer environmental interactions . This property , combined with our proposed goal sampling technique and replay buffer used during training , provides empirical improvements over Q-function based methods . In addition to these advantages , learning C∗ is itself useful , containing information that the horizonaware policies do not . It estimates whether a goal g is reachable from the current state s within h steps . In contrast , π ( a|s , g , h ) simply returns some action , even for unreachable goals . We show that C∗ can be used to determine reachability with examples in a nonholonomic environment . Summary of contributions : ( i ) introducing C-functions and cumulative accessibility estimation for both discrete and continuous action spaces ; ( ii ) highlighting the importance of the speed vs reliability trade-off in finite horizon reinforcement learning ; ( iii ) introducing a novel replay buffer specially tailored for learning C∗ which builds on HER ( Andrychowicz et al. , 2017 ) ; and ( iv ) empirically showing the effectiveness of our method for goal-reaching as compared to existing alternatives , particularly in the context of complex motion planning tasks . 2 BACKGROUND AND RELATED WORK . Let us extend the Markov Decision Process ( MDP ) formalism ( Sutton et al. , 1998 ) for goal-reaching . We consider a set of actionsA , a state space S , and a goal set G. We assume access to a goal checking function G : S × G → { 0 , 1 } such that G ( s , g ) = 1 if and only if state s achieves goal g. For example , achieving the goal could mean exactly reaching a certain state , in which case G = S and a a1 a2 a3 s h s1 h− 1 s2 h− 2 s3 h− 3 g . . . . . . Figure 2 : Graphical model depicting trajectories from Pπ ( ·|· , g , h ) ( ·|s0 = s , a0 = a ) . Gray nodes denote fixed values , and white nodes stochastic ones . Nodes a , g and s are non-stochastic simply because they are conditioned on , not because they are always fixed within the environment . Note that the values of h decrease deterministically . Nodes corresponding to horizons could be separated from states , but are not for a more concise graph . G ( s , g ) = 1 ( s = g ) . For many continuous state-spaces , hitting a state exactly has zero probability . Here we can still take G = S , but letG ( s , g ) = 1 ( d ( s , g ) ≤ ) for some radius and metric d. More general choices are possible . For example , in the Dubins ’ Car environment which we describe in more detail later , the state consists of both the location and orientation of the car : S = R2 × S1 . We take G = R2 , and G ( s , g ) checks that the location of the car is within some small radius of g , ignoring the direction entirely . For a fixed g , G ( s , g ) can be thought of as a sparse reward function . In the goal-reaching setting , a policy π : S×G → P ( A ) , whereP ( A ) denotes the set of distributions over A , maps state-goal pairs to an action distribution . The environment dynamics are given by a starting distribution p ( s0 , g ) , usually taken as p ( s0 ) p ( g ) , and transition probabilities p ( st+1|st , at ) . States for which G ( s , g ) = 1 are considered terminal . Q-Learning : AQ-function ( Watkins & Dayan , 1992 ) for multi-goal reaching , Qπ : S×G×A → R , is defined by Qπ ( st , g , at ) = Eπ [ ∑∞ i=t γ i−tG ( st , g ) |st , at ] , where γ ∈ [ 0 , 1 ] is a discount factor and the expectation is with respect to state-action trajectories obtained by using π ( a|si , g ) . If π∗ is an optimal policy in the sense thatQπ ∗ ( s , g , a ) ≥ Qπ ( s , g , a ) for every π and ( s , g , a ) ∈ S×G×A , then Qπ ∗ matches the optimal Q-function , Q∗ , which obeys the Bellman equation : Q∗ ( s , g , a ) = Es′∼p ( ·|s , a ) [ G ( s , g ) + γmax a′∈A Q∗ ( s′ , g , a′ ) ] . ( 1 ) In deep Q-learning ( Mnih et al. , 2015 ) , Q∗ is parameterized with a neural network and learning is achieved by enforcing the relationship from equation 1 . This is done by minimizing∑ i L ( Q∗ ( si , gi , ai ) , yi ) , where yi corresponds to the expectation in equation 1 and is estimated using a replay buffer of stored tuples ( si , ai , gi , s′i ) . Note that s ′ i is the state the environment transitioned to after taking action ai from state si , and determines the value of yi . Typically L is chosen as a squared error loss , and the dependency of yi on Q∗ is ignored for backpropagation in order to stabilize training . Once Q∗ is learned , the optimal policy is recovered by π∗ ( a|s , g ) = 1 ( a = argmaxa′ Q∗ ( s , g , a′ ) ) . There is ample work extending and improving upon deep Q-learning ( Haarnoja et al. , 2018 ) . For example , Lillicrap et al . ( 2015 ) extend it to the continuous action space setting , and Fujimoto et al . ( 2018 ) further stabilize training . These improvements are fully compatible with goal-reaching ( Pong et al. , 2019 ; Bharadhwaj et al. , 2020a ; Ghosh et al. , 2019 ) . Andrychowicz et al . ( 2017 ) proposed Hindsight Experience Replay ( HER ) , which relabels past experience as achieved goals , and allows sample efficient learning from sparse rewards ( Nachum et al. , 2018 ) . 3 CUMULATIVE ACCESSIBILITY FUNCTIONS . We now consider horizon-aware policies π : S × G × N → P ( A ) , and define the cumulative accessibility function Cπ ( s , a , g , h ) , or C-function , as the probability of reaching goal g from state s in at most h steps by taking action a and following the policy π thereafter . By “ following the policy π thereafter ” we mean that after a , the next action a1 is sampled from π ( ·|s1 , g , h− 1 ) , a2 is sampled from π ( ·|s2 , g , h− 2 ) and so on . See Figure 2 for a graphical model depiction of how these trajectories are obtained . Importantly , an agent need not always act the same way at a particular state in order to reach a particular goal , thanks to horizon-awareness . We use Pπ ( ·|· , g , h ) ( ·|s0 = s , a0 = a ) to denote probabilities in which actions are drawn in this manner and transitions are drawn according to the environment p ( st+1|st , a ) . More formally , Cπ is given by : Cπ ( s , a , g , h ) = Pπ ( ·|· , g , h ) ( max t=0 , ... , h G ( st , g ) = 1 ∣∣∣∣s0 = s , a0 = a ) . ( 2 ) Proposition 1 : Cπ can be framed as a Q-function within the MDP formalism , and if π∗ is optimal in the sense that Cπ ∗ ( s , a , g , h ) ≥ Cπ ( s , a , g , h ) for every π and ( s , a , g , h ) ∈ S ×A×G ×N , then Cπ ∗ matches the optimal C-function , C∗ , which obeys the following equation : C∗ ( s , a , g , h ) = Es′∼p ( ·|s , a ) [ max a′∈A C∗ ( s′ , a′ , g , h− 1 ) ] if G ( s , g ) = 0 and h ≥ 1 , G ( s , g ) otherwise . ( 3 ) See appendix A for a detailed mathematical proof of this proposition . The proof proceeds by first deriving a recurrence relationship that holds for any Cπ . In an analogous manner to the Bellman equation in Q-learning , this recurrence involves an expectation over π ( ·|s′ , g , h − 1 ) , which , when replaced by a max returns the recursion for C∗ . Proposition 1 is relevant as it allows us to learn C∗ , enabling goal-reaching policies to be recovered : π∗ ( a|s , g , h ) = 1 ( a = argmax a′ C∗ ( s , a′ , g , h ) ) . ( 4 ) C∗ itself is useful for determining reachability . After maximizing over actions , it estimates whether a given goal is reachable from a state within some horizon . Comparing these probabilities for different horizons allows us to make a speed / reliability trade-off for reaching goals . We observe that an optimal C∗-function is non-decreasing in h , but this does not necessarily hold for non-optimal C-functions . For example , a horizon-aware policy could actively try to avoid the goal for high values of h , and the Cπ-function constructed from it would show lower probabilities of success for larger h. See appendix A for a concrete example of this counter-intuitive behavior . Proposition 2 : C∗ is non-decreasing in h. See appendix A for a detailed mathematical proof . Intuitively , the proof consists of showing that an optimal policy can not exhibit the pathology mentioned above . Given an optimal policy π∗ ( a|s , g , h ) for a fixed horizon h we construct a policy π̃ for h + 1 which always performs better , and lower bounds the performance of π∗ ( a|s , g , h+ 1 ) . In addition to being an elegant theoretical property , proposition 2 suggests that there is additional structure in a C∗ function which mitigates the added complexity from using horizon-aware policies . Indeed , in our preliminary experiments we used a non-cumulative version of C-functions ( see section 3.3 ) and obtained significantly improved performance upon changing to C-functions . Moreover , monotonicity in h could be encoded in the architecture of C∗ ( Sill , 1998 ; Wehenkel & Louppe , 2019 ) . However , we found that actively doing so hurt empirical performance ( appendix F ) . 3.1 SHORTCOMINGS OF Q-LEARNING Before describing our method for learningC∗ , we highlight a shortcoming ofQ-learning . Consider a 2D navigation environment where an agent can move deterministically in the cardinal directions , and fix s and g. For an optimal action a∗ , the optimalQ function will achieve some valueQ∗ ( s , g , a∗ ) ∈ [ 0 , 1 ] in the sparse reward setting . Taking a sub-optimal action a− initially results in the agent taking two extra steps to reach the intended goal , given that the agent acts optimally after the first action , so that Q∗ ( s , g , a− ) = γ2Q∗ ( s , g , a∗ ) . The value of γ is typically chosen close to 1 , for example 0.99 , to ensure that future rewards are not too heavily discounted . As a consequence γ2 ≈ 1 and thus the value of Q∗ at the optimal action is very close to its value at a sub-optimal action . We illustrate this issue in Figure 1b . In this scenario , recovering an optimal policy requires that the error between the learned Q-function and Q∗ should be at most ( 1−γ2 ) /2 ; this is reflected empirically by Q-learning having high sample complexity and learning sub-optimal paths . This shortcoming surfaces in any environment where taking a sub-optimal action results in a slightly longer path than an optimal one , as in e.g . motion planning tasks . The C∗ function does not have this shortcoming . Consider the same 2D navigation example , and let h∗ be the smallest horizon for which g can be reached from s. h∗ can be easily obtained from C∗ as the smallest h such that maxa C∗ ( s , a , g , h ) = 1 . Again , denoting a∗ as an optimal action and a− as a sub-optimal one , we have that C∗ ( s , a∗ , g , h∗ ) = 1 whereas C∗ ( s , a− , g , h∗ ) = 0 , which is illustrated in Figure 1b . Therefore , the threshold for error is much higher when learning the C∗ function . This property results in fewer interactions with the environment needed to learn C∗ and more efficient solutions .

Summary: The paper proposes C-learning, which is an essentially a horizon aware Q-learning. In a nutshell, the authors proposes changing the Q function from Q(s, a) to C(s, a, h) where s is the state, a is the action and h is allowed time horizon i.e. the agent should get to the goal state using less than h states. Since C can be framed as a modified Q, the authors demonstrate that familiar Q-learning properties such as Bellman property and backprop can be used for C-learning.