Reachability Traces for Curriculum Design in Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) ( Sutton & Barto , 1998 ) has been successfully used to train agents in domains such as robotics ( Zhu et al. , 2019 ) , Atari games ( Mnih et al. , 2015 ) and other complex games ( Silver et al. , 2018 ) . The universal principle underpinning these applications is the maximization of the long term expected rewards that the agent accumulates as it interacts with its environment . Following each interaction , the agent receives a numerical reward , which is ( generally ) directly or indirectly specified by a human user . The design of such reward functions is critical , as it fundamentally affects both the rate of learning ( Matignon et al. , 2006 ) , as well as the type of behaviors learned . Specifically , a sparse reward function makes learning challenging , as it deprives the agent of the necessary feedback required to improve its behavior . This has been a widely studied problem in RL , and although several potential solutions ( Ng et al. , 1999 ; Vecerik et al. , 2017 ; Narvekar et al. , 2020 ) have been proposed , it remains an active area of research . Apart from sparsity , the reward function could also suffer from improper specifications , such as inappropriately chosen reward values for certain states or actions , which could distract the agent from its intended task . The consequences of such mis-specifications have been recorded in works such as Burda et al . ( 2018 ) , where the RL agent , rewarded for curious behaviors , was shown to become distracted from its original task due to local sources of entropy ( A TV with randomly changing channels ) . Similar effects have also been observed in Clark & Amodei ( 2016 ) , where the agent , tasked with safely completing a boat race circuit , unexpectedly exploited the improperly designed reward function , and learned undesirable behaviors . In this work , we propose the idea of reachability traces , which is based solely on the reachability of the goal state , and is independent of other aspects of the reward function . Similar to the idea of eligibility traces ( Singh & Sutton , 1996 ) in classical RL , reachability traces model the temporal closeness ( i.e. , number of steps to the goal ) of states/state-action pairs leading to the goal by assigning diminishing reachability values ( traces ) to these states , looking backwards from the goal state . These reachability values are approximated through a reachability trace function , which is realized through a simple feedforward neural network , updated online during learning . We also show that alternatively , reachability traces could be learned using an MDP ( Markov Decision Process ) framework ( Puterman , 2014 ) , guaranteeing its convergence in tabular environments . Once learned , the reachability trace provides an indication of the temporal closeness to the goal state , which is used to autonomously determine a sequence of achievable subgoals , which are subsequently learned . The sequence of subgoals are chosen in increasing order of their reachability trace values , which ensures that subgoals appearing later in the sequence are temporally closer to the goal state , and thus have a higher chance of reaching the goal . The corresponding subpolicies , once learned , are used to provide the agent with action advice ( Fachantidis et al. , 2019 ) , thereby guiding it towards the goal region . The use of action advice ( accompanied by a non-zero probability of random exploration ) to guide our off-policy agent implies the preservation of its convergence properties in tabular environments . We demonstrate our approach in sparse ( discrete as well as continuous ) goal based RL tasks and compare its performance against several other baselines . We also discuss and empirically demonstrate other use cases of reachability traces , such as in environments with poorly designed reward functions . In summary , the main contributions of this work are : • The idea of reachability traces to model the temporal closeness to the goal . • A framework for automatically discovering and learning reachable subgoals . • An empirical comparison of our proposed approach to other baselines in a variety of reward sparse environmnents . 2 REACHABILITY TRACES . In goal based tasks , the aim of the agent is to learn a policy to reach a predetermined goal state . Depending on the reward function in question , the probability of reaching such a goal state may or may not be correlated with the ( action- ) value function of the agent . This is because the value function , which is solely designed to maximize the expected return , does not explicitly depend on goal state visits . For example , value function based learning would need to account for non-goal rewards , which ( if improperly specified ) could distract the agent from its intended task ( Burda et al. , 2018 ; Clark & Amodei , 2016 ) . Particularly in environments where the goal rewards are sparse , we posit that it is more beneficial to exploit the rare trajectories that lead to the goal state , by learning about their goal-reaching properties , and subsequently using this knowledge to guide the agent ’ s exploration . We posit that this goal-reaching property be characterized through reachability traces , which is solely based on the temporal distance to the goal state under the agent ’ s behavior policy . In classical RL , a concept that models the idea of temporal closeness is eligibility traces ( Singh & Sutton , 1996 ) . Although the primary motivation behind eligibility traces was to address the credit assignment problem , it was designed in a way that states/state-action pairs that were temporally more closely related to an event ( a state or action ) , were assigned higher trace values compared to those that were temporally further away . Here , we use a similar idea to build a reachability trace function using historically successful trajectories . Unlike eligibility traces , reachability traces are learned as a neural network and are concerned solely with reaching the goal state , and modeling the temporal distance to the goal , which in this work , is used to evaluate the suitability of potential subgoals . Assuming a behavior policy π , we consider a successful trajectory TS = { st ... st+k , ... st+N } which terminates at st+N ( goal state ) . For each st+i ∈ TS , we assign a reachability trace label eπ ( st+i ) as : eπ ( st+i ) = λ N−ie0 ( 1 ) where e0 ( set to 1 ) is the highest possible trace label and λ ( 0 < λ < 1 ) is the trace decay parameter . It is to be noted that traces are updated looking backwards from the terminal/goal state as per Equation 1 . Trace labels are assigned to ensure that states temporally closer to the goal state are associated with higher trace values , and those further away are associated with lower values . States in unsuccessful trajectories are assigned a trace label of 0 . These assigned trace labels are then be used to learn a reachability trace function φπ ( s ) : s → eπ ( s ) , which maps states ( or state-action pairs ) to their corresponding trace labels . Given enough successful trajectories , the learned reachability trace φ ( s ) would converge to the expected trace value : φ ( st ) = Eπ [ λNe0 ] ( 2 ) Our assumption is that prior to learning the trace function , our agent encounters at least one successful trajectory . We do not require that this trajectory be optimal ; only that it enounters the goal state G and terminates . As we obtain trace labels for multiple states upon visiting the goal , even a single successful trajectory could help build a rough estimate of the trace function . Our approach is also suitable to be used in the availability of ( suboptimal ) demonstrations , as described in Appendix D. Algorithm 1 Learning a reachability trace function 1 : Input : 2 : Trace decay parameter λ , stored trajectory length L , trace function φ ( s , θ ) , goal state G 3 : Initialize state s = s0 , successful trace buffer DφS and unsuccessful trace buffer DφU 4 : Output : Learned trace function φ 5 : while True do 6 : Interact with environment : take action a , observe r , s′ 7 : Store latest L states 8 : if Successful episode ( G is reached ) then 9 : for i from 1 to L do 10 : ei = λL−ie0 11 : end for 12 : Move the trace labels e and corresponding states into the buffer DφS 13 : else 14 : for i from 1 to L do 15 : ei = 0 16 : end for 17 : Move the trace labels e and corresponding states into the buffer DφU 18 : end if 19 : Sample set of state and trace labels ( ssamp , esamp ) , obtaining each sample from either DφS or DφU with equal probability 20 : Minimize ( esamp − φ ( ssamp , θ ) ) 2 with respect to trace network parameters θ 21 : s← s′ 22 : end while Since the trace function is indicative of closeness to the goal stateG , it is used as a basis for generating subsequent subgoals that are likely to be closer to G. In this work , the reachability trace is realized through a simple feedforward neural network , trained on data of states and corresponding trace labels generated from successful and unsuccessful trajectories , which are stored in separate buffers . The steps to be followed to learn a trace function have been described in Algorithm 1 . Here , successful and unsuccessful trajectories are sampled with equal probability , which causes the learned trace to be a biased estimate of that mentioned in Equation 2 . Despite this bias , we find that the learned reachability trace serves as an approximate measure of temporal proximity , and is still a useful basis for selecting subgoals . In addition , as our approach does not require generated suboals to lie on the optimal path , the use of this biased estimate is justified . Although reachability traces have been described only in terms of states ( for the sake of simplicity ) , similar to the case in eligibility traces , it can also be extended to state-action pairs ( φπ ( s , a ) ) . 3 SUBGOAL GENERATION AND CURRICULUM BUILDING . Once an appropriate trace function φ has been learned , we use it to identify and learn potential subgoals of increasing difficulty . The intuition is that when goal rewards are sparse , a high value of φ ( s ) or φ ( s , a ) indicates that the state is temporally close to the goal state , and thus may be less reachable ( from the starting state ) , while a low φ value may indicate that the state is further away from the goal state ( and thus temporally closer to the starting state ) , and thus may consitute an easier subgoal to be learned . In this work , we treat this temporal proximity as a proxy for subgoal difficulty , and use it as a basis for building a task curriculum ( Portelas et al. , 2020 ; Narvekar et al. , 2020 ) in sparse reward problems . The idea is to first learn simple subgoals ( low φ ) and progressively move to more difficult ones till the main task is solved . We consider a sparse-reward MDP ( main task ) M = { S , A , R , T , G , s0 } , with state space S , action space A , reward function R , and transition function T . s0 and G represent the start and terminal goal states respecitvely . Under the standard learning scenario , learning an optimal policy π∗ to solve this sparse reward MDP may become prohibitively sample inefficient . However , if we have learned a trace function φ , M can be simplified into a sequence of subtasksM = [ M0 ... Mi ... MN ] such that each Mi ∈ M is a subtask MDP Mi = { S , A , Ri , Ti , gi , si } sharing the same state-action space , with its own reward function Ri , start state si and terminal goal state gi . The starting state for the subsequent subtask is set as the goal state of the previous one ; that is , si = gi−1 . The transition function Ti is assumed to be identical to T , except that as per Ti , gi is terminal , whereas T assumes termination at G. The subtask sequence is such that for any two subtasks Mi , Mj , where j > i , φ ( gj ) > tφφ ( gi ) , tφ ( ≥ 1 ) is a subgoal trace scaling factor . Choosing large values of tφ would result in subgoals spaced temporally farther , while smaller values of tφ could be expected to return subgoals that are temporally close to one another . Guidelines for selecting tφ are described in Appendix B . The reward ri ( s ) corresponding to the reward function Ri for each subtask is binary , such that : ri ( s ) = { 1 s = gi 0 otherwise ( 3 ) In order to identify subgoals online , the agent is allowed to interact with the environment in an episodic manner , till it encounters a state smax corresponding to the highest φ value experienced during the episode , which also exceeds the trace value of the latest subgoal state gi . That is , if φ ( smax ) > tφφ ( gi ) , smax is considered to be the new subgoal ( gi+1 ) , and accordingly , subtask Mi+1 is generated with high rewards corresponding to state gi+1 , and 0 rewards for all other states . For each such subtask , a corresponding subpolicy is learned , following which the same process is repeated to generate the next subtask , and so on till the goal state G is reached . The fact that only previously visited states are chosen as subgoals ensures that the chosen subgoals are indeed reachable . At this point , the main taskM would be decomposed into several subtasks consisting of subgoals of progressively increasing difficulty ( Figure 6 , Appendix B ) , whose subpolicies are learned in sequence . Although these subpolicies are used to guide the agent towards the goal ( described in Section 4 ) , they can also potentially be reused ( Fernández & Veloso , 2006 ) to aid transfer learning ( Appendix E ) . The processes involved for subgoal generation and curriculum building are described in Algorithm 2 .

Summary: This paper considers a sparse reward goal-based RL problem and introduces a notion of reachability trace for reinforcement learning, which intuitively approximates how reachable (or how far) the goal state is from the current state. The proposed idea is to divide the whole learning problem into three phases: 1) learning a reachability function w.r.t. the main goal state, 2) finding a sequence of subgoals that are gradually closer to the main goal, 3) learning a main policy while using the sequence subgoals (corresponding subpolicies) as an exploratory policy in epsilon-greedy action sampling. The empirical results on grid worlds and several maze environments show that the proposed method outperforms DDPG, RIS, PER, and EBU.