Reachability Traces for Curriculum Design in Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) has been successfully used to train agents in domains such as robotics ( Zhu et al. , 2019 ) , Atari games ( Mnih et al. , 2015 ) and other complex games ( Silver et al. , 2018 ) . The universal principle underpinning these applications is the maximization of the long term expected rewards that the agent accumulates as it interacts with its environment . Following each interaction , the agent receives a numerical reward , which is ( generally ) directly or indirectly specified by a human user . The design of such reward functions is critical , as it fundamentally affects both the rate of learning ( Matignon et al. , 2006 ) , as well as the type of behaviors learned . Specifically , a sparse reward function makes learning challenging , as it deprives the agent of the necessary feedback required to improve its behavior . This has been a widely studied problem in RL , and although several potential solutions ( Ng et al. , 1999 ; Vecerik et al. , 2017 ; Narvekar et al. , 2020 ) have been proposed , it remains an active area of research . Apart from sparsity , the reward function could also suffer from improper specifications , such as inappropriately chosen reward values for certain states or actions , which could distract the agent from its intended task . The consequences of such mis-specifications have been recorded in works such as Burda et al . ( 2018 ) , where the RL agent , rewarded for curious behaviors , was shown to become distracted from its original task due to local sources of entropy ( A TV with randomly changing channels ) . Similar effects have also been observed in Clark & Amodei ( 2016 ) , where the agent , tasked with safely completing a boat race circuit , unexpectedly exploited the improperly designed reward function , and learned undesirable behaviors . In this work , we propose the idea of reachability traces , which is based solely on the reachability of the goal state , and is independent of other aspects of the reward function . Similar to the idea of eligibility traces ( Singh & Sutton , 1996 ) in classical RL , reachability traces model the temporal closeness ( i.e. , number of steps to the goal ) of states/state-action pairs leading to the goal by assigning diminishing reachability values ( traces ) to these states , looking backwards from the goal state . These reachability values are approximated through a reachability trace function , which is realized through a simple feedforward neural network , updated online during learning . We also show that alternatively , reachability traces could be learned using an MDP ( Markov Decision Process ) framework ( Puterman , 2014 ) , guaranteeing its convergence in tabular environments . Once learned , the reachability trace provides an indication of the temporal closeness to the goal state , which is used to autonomously determine a sequence of achievable subgoals , which are subsequently learned . The sequence of subgoals are chosen in increasing order of their reachability trace values , which ensures that subgoals appearing later in the sequence are temporally closer to the goal state , and thus have a higher chance of reaching the goal . The corresponding subpolicies , once learned , are used to provide the agent with action advice ( Fachantidis et al. , 2019 ) , thereby guiding it towards the goal region . The use of action advice ( accompanied by a non-zero probability of random exploration ) to guide our off-policy agent implies the preservation of its convergence properties in tabular environments . We demonstrate our approach in sparse ( discrete as well as continuous ) goal based RL tasks and compare its performance against several other baselines . We also discuss and empirically demonstrate other use cases of reachability traces , such as in environments with poorly designed reward functions . In summary , the main contributions of this work are : • The idea of reachability traces to model the temporal closeness to the goal . • A framework for automatically discovering and learning reachable subgoals . • An empirical comparison of our proposed approach to other baselines in a variety of reward sparse environmnents . 2 REACHABILITY TRACES . In goal based tasks , the aim of the agent is to learn a policy to reach a predetermined goal state . Depending on the reward function in question , the probability of reaching such a goal state may or may not be correlated with the ( action- ) value function of the agent . This is because the value function , which is solely designed to maximize the expected return , does not explicitly depend on goal state visits . For example , value function based learning would need to account for non-goal rewards , which ( if improperly specified ) could distract the agent from its intended task ( Burda et al. , 2018 ; Clark & Amodei , 2016 ) . Particularly in environments where the goal rewards are sparse , we posit that it is more beneficial to exploit the rare trajectories that lead to the goal state , by learning about their goal-reaching properties , and subsequently using this knowledge to guide the agent ’ s exploration . We posit that this goal-reaching property be characterized through reachability traces , which is solely based on the temporal distance to the goal state under the agent ’ s behavior policy . In classical RL , a concept that models the idea of temporal closeness is eligibility traces ( Singh & Sutton , 1996 ) . Although the primary motivation behind eligibility traces was to address the credit assignment problem , it was designed in a way that states/state-action pairs that were temporally more closely related to an event ( a state or action ) , were assigned higher trace values compared to those that were temporally further away . Here , we use a similar idea to build a reachability trace function using historically successful trajectories . Unlike eligibility traces , reachability traces are learned as a neural network and are concerned solely with reaching the goal state , and modeling the temporal distance to the goal , which in this work , is used to evaluate the suitability of potential subgoals . Assuming a behavior policy π , we consider a successful trajectory TS = { st ... st+k , ... st+N } which terminates at st+N ( goal state ) . For each st+i ∈ TS , we assign a reachability trace label eπ ( st+i ) as : eπ ( st+i ) = λ N−ie0 ( 1 ) where e0 ( set to 1 ) is the highest possible trace label and λ ( 0 < λ < 1 ) is the trace decay parameter . It is to be noted that traces are updated looking backwards from the terminal/goal state as per Equation 1 . Trace labels are assigned to ensure that states temporally closer to the goal state are associated with higher trace values , and those further away are associated with lower values . States in unsuccessful trajectories are assigned a trace label of 0 . These assigned trace labels are then be used to learn a reachability trace function φπ ( s ) : s → eπ ( s ) , which maps states ( or state-action pairs ) to their corresponding trace labels . Given enough successful trajectories , the learned reachability trace φ ( s ) would converge to the expected trace value : φ ( st ) = Eπ [ λNe0 ] ( 2 ) Our assumption is that prior to learning the trace function , our agent encounters at least one successful trajectory . We do not require that this trajectory be optimal ; only that it enounters the goal state G and terminates . As we obtain trace labels for multiple states upon visiting the goal , even a single successful trajectory could help build a rough estimate of the trace function . Our approach is also suitable to be used in the availability of ( suboptimal ) demonstrations , as described in Appendix D. Algorithm 1 Learning a reachability trace function 1 : Input : 2 : Trace decay parameter λ , stored trajectory length L , trace function φ ( s , θ ) , goal state G 3 : Initialize state s = s0 , successful trace buffer DφS and unsuccessful trace buffer DφU 4 : Output : Learned trace function φ 5 : while True do 6 : Interact with environment : take action a , observe r , s′ 7 : Store latest L states 8 : if Successful episode ( G is reached ) then 9 : for i from 1 to L do 10 : ei = λL−ie0 11 : end for 12 : Move the trace labels e and corresponding states into the buffer DφS 13 : else 14 : for i from 1 to L do 15 : ei = 0 16 : end for 17 : Move the trace labels e and corresponding states into the buffer DφU 18 : end if 19 : Sample set of state and trace labels ( ssamp , esamp ) , obtaining each sample from either DφS or DφU with equal probability 20 : Minimize ( esamp − φ ( ssamp , θ ) ) 2 with respect to trace network parameters θ 21 : s← s′ 22 : end while Since the trace function is indicative of closeness to the goal stateG , it is used as a basis for generating subsequent subgoals that are likely to be closer to G. In this work , the reachability trace is realized through a simple feedforward neural network , trained on data of states and corresponding trace labels generated from successful and unsuccessful trajectories , which are stored in separate buffers . The steps to be followed to learn a trace function have been described in Algorithm 1 . Here , successful and unsuccessful trajectories are sampled with equal probability , which causes the learned trace to be a biased estimate of that mentioned in Equation 2 . Despite this bias , we find that the learned reachability trace serves as an approximate measure of temporal proximity , and is still a useful basis for selecting subgoals . In addition , as our approach does not require generated suboals to lie on the optimal path , the use of this biased estimate is justified . Although reachability traces have been described only in terms of states ( for the sake of simplicity ) , similar to the case in eligibility traces , it can also be extended to state-action pairs ( φπ ( s , a ) ) . 3 SUBGOAL GENERATION AND CURRICULUM BUILDING . Once an appropriate trace function φ has been learned , we use it to identify and learn potential subgoals of increasing difficulty . The intuition is that when goal rewards are sparse , a high value of φ ( s ) or φ ( s , a ) indicates that the state is temporally close to the goal state , and thus may be less reachable ( from the starting state ) , while a low φ value may indicate that the state is further away from the goal state ( and thus temporally closer to the starting state ) , and thus may consitute an easier subgoal to be learned . In this work , we treat this temporal proximity as a proxy for subgoal difficulty , and use it as a basis for building a task curriculum ( Portelas et al. , 2020 ; Narvekar et al. , 2020 ) in sparse reward problems . The idea is to first learn simple subgoals ( low φ ) and progressively move to more difficult ones till the main task is solved . We consider a sparse-reward MDP ( main task ) M = { S , A , R , T , G , s0 } , with state space S , action space A , reward function R , and transition function T . s0 and G represent the start and terminal goal states respecitvely . Under the standard learning scenario , learning an optimal policy π∗ to solve this sparse reward MDP may become prohibitively sample inefficient . However , if we have learned a trace function φ , M can be simplified into a sequence of subtasksM = [ M0 ... Mi ... MN ] such that each Mi ∈ M is a subtask MDP Mi = { S , A , Ri , Ti , gi , si } sharing the same state-action space , with its own reward function Ri , start state si and terminal goal state gi . The starting state for the subsequent subtask is set as the goal state of the previous one ; that is , si = gi−1 . The transition function Ti is assumed to be identical to T , except that as per Ti , gi is terminal , whereas T assumes termination at G. The subtask sequence is such that for any two subtasks Mi , Mj , where j > i , φ ( gj ) > tφφ ( gi ) , tφ ( ≥ 1 ) is a subgoal trace scaling factor . Choosing large values of tφ would result in subgoals spaced temporally farther , while smaller values of tφ could be expected to return subgoals that are temporally close to one another . Guidelines for selecting tφ are described in Appendix B . The reward ri ( s ) corresponding to the reward function Ri for each subtask is binary , such that : ri ( s ) = { 1 s = gi 0 otherwise ( 3 ) In order to identify subgoals online , the agent is allowed to interact with the environment in an episodic manner , till it encounters a state smax corresponding to the highest φ value experienced during the episode , which also exceeds the trace value of the latest subgoal state gi . That is , if φ ( smax ) > tφφ ( gi ) , smax is considered to be the new subgoal ( gi+1 ) , and accordingly , subtask Mi+1 is generated with high rewards corresponding to state gi+1 , and 0 rewards for all other states . For each such subtask , a corresponding subpolicy is learned , following which the same process is repeated to generate the next subtask , and so on till the goal state G is reached . The fact that only previously visited states are chosen as subgoals ensures that the chosen subgoals are indeed reachable . At this point , the main taskM would be decomposed into several subtasks consisting of subgoals of progressively increasing difficulty ( Figure 6 , Appendix B ) , whose subpolicies are learned in sequence . Although these subpolicies are used to guide the agent towards the goal ( described in Section 4 ) , they can also potentially be reused ( Fernández & Veloso , 2006 ) to aid transfer learning ( Appendix E ) . The processes involved for subgoal generation and curriculum building are described in Algorithm 2 .

Summary: This paper proposed an automatic curriculum learning design method for goal-orientated Reinforcement Learning. The idea is to learn a reachability trace function to measure the temporal difference between a state and the final goal state. Then a curriculum is learnt based on the reachability trace function. The algorithm is evaluated on tabular and continuous robotic control environments.