Explore with Dynamic Map: Graph Structured Reinforcement Learning

1 INTRODUCTION . How can humans learn to solve tasks with complex structure and delayed and sparse feedback ? Take a complicated navigation task as an example in which the goal is to go from a start state to an end state . A straightforward approach that one often takes is by decomposing this complicated task into a sequence of easier ones , identifying/forming some intermediate goals that help to get to the final goal , and finally choosing a route with the highest return among a poll of candidate routes that lead to the final goal . Despite recent success and progress in reinforcement learning ( RL ) approaches ( Mnih et al. , 2013 ; Fakoor et al. , 2020 ; Hansen et al. , 2018 ; Huang et al. , 2019 ; Silver et al. , 2016 ) , there is still a huge gap between how humans and RL agents learn . In many real-world applications , an RL agent only has access to sparse and delayed rewards , which by itself leads to two major challenges . ( C1 ) The first one is how can an agent effectively learn from sparse and delayed rewards ? One possible solution is to build a new reward function , known as an intrinsic reward , that helps expedite the learning process and ultimately solve a given task . Although this may seem like an appealing solution , it is often not obvious how to formulate an effective intrinsic reward . Recent works have introduced goal-oriented RL ( Schaul et al. , 2015 ; Chiang et al. , 2019 ) as a way of constructing intrinsic rewards . However , most goal-oriented RL algorithms require large amounts of reward shaping ( Chiang et al. , 2019 ) or human demonstrations ( Nair et al. , 2018 ) . ( C2 ) The second issue is how to retrieve relevant past experiences that help to learn faster and improve sample efficiency . Recent attention has focused much on episodic RL ( Blundell et al. , 2016 ) , which builds a non-parametric episodic memory to store past experiences , and thus can rapidly latch onto related ones through search with similarity . However , most episodic RL algorithms require additional memory space ( Pritzel et al. , 2017 ) . In this work , we propose Graph Structured Reinforcement Learning ( GSRL ) , which constructs a state-transition graph , leverages structural graph information , and presents solutions for the problems raised above . As shown in Figure 1 , when we encounter a complex task , we draw a map for planning with a long horizon ( Eysenbach et al. , 2019 ) and scan through the graph to seek related experiences within a short horizon ( Lee et al. , 2019 ) . Inspired by this , we propose a method to address the first problem ( C1 ) by learning to generate an appropriate goal ge based on map Ge0 at episode e and form the intrinsic rewards during goal generation . This goal generation can be optimized in hindsight . That is , we can update the goal generation model under the supervision generated by planning algorithms based on map Ge+10 at episode e + 1 . We can solve the second problem ( C2 ) by updating the value estimation of the current state with the trajectories that include neighborhood states . Note that GSRL draws a map to help agents plan an appropriate direction in the long horizon and learn an optimal strategy in the short horizon , which can be widely incorporated with various learning strategies such as DQN ( Mnih et al. , 2013 ) , DDPG ( Lillicrap et al. , 2015 ) , etc . However , arriving at a well-explored map in terms of state-transitions in the first place from scratch remains a challenging problem . In this regard , GSRL should be able to jointly optimize map construction , goal selection , and value estimation . As illustrated in Figure 2 , we design GSRL mainly following four steps : ( a ) At the beginning , an agent is located at the initial state and aims at the terminal state with an empty map ; ( b ) The agent then explores the environment with the current policy and records those explored states on the map ; ( c ) Considering the far distance to the terminal state and sparse reward during the exploration , we decompose the whole task into a sequence of easier tasks guided by goals ; ( d ) We leverage structured map information to select related trajectories to update the current policy ; The agent can repeat procedures ( b ) - ( d ) and finally reach the terminal state with a well-explored map . Our primary contribution is a novel algorithm that constructs a dynamic state-transition graph from scratch and leverages structured information to find an appropriate goal in goal selection with a long horizon and sample related trajectories in value estimation with a short horizon . This framework provides several benefits : ( 1 ) How to balance exploration and exploitation is a fundamental problem in any reinforcement learning ( Andrychowicz et al. , 2017 ; Nachum et al. , 2018 ) . GSRL learns an attention mechanism on the graph to decide whether to explore the states with high uncertainty or exploit the states with well estimated value . ( 2 ) Almost all the existing reinforcement learning approaches ( Paul et al. , 2019 ; Eysenbach et al. , 2019 ) utilize randomly sampled trajectories to update the current policy leading to inefficiency . GSRL selects highly related trajectories for updates through the local structure in the graph . ( 3 ) Unlike previous approaches ( Kipf et al. , 2020 ; Shang et al. , 2019 ) , GSRL constructs a dynamic graph through agent exploration without any additional object representation techniques and thus is not limited to several specific environments . Empirically , we find that our method constructs a useful map and learns an efficient policy for highly structured tasks . Comparisons with state-of-the-art RL methods show that GSRL is substantially more successful for boosting the performance of a graph-structured RL algorithm . 2 PRELIMINARIES . LetM = ( S , A , T , P , R ) be an Markov Decision Process ( MDP ) where S is the state space , A is the action space whose size is bounded by a constant , T ∈ Z+ is the episode length , P : S×A→ ∆ ( S ) is the transition function which takes a state-action pair and returns a distribution over states , and R : S × A→ ∆ ( R ) is the reward distribution . The rewards from the environment , called extrinsic , are usually sparse and , therefore , difficult to learn . To address this , goal-oriented RL introduces goal g ∈ G to build dense intrinsic rewards Rg : S × A × G → ∆ ( R ) . In order to make the learning procedure stable , the goal is limited to be updated episodically . A policy π : S × G → ∆ ( A ) prescribes a distribution over actions for each state and goal . At each timestep , the agent samples an action a ∼ π ( s , g ) and receives a corresponding reward rg ( s , a ) that indicates whether or not the agent has reached the goal . The episode terminates only after T timesteps even when the agent reaches the goal . The agent ’ s task is to maximize its cumulative discounted future reward . We use an off-policy algorithm to learn such a policy , as well as its associated goal-oriented Q-function and value function : Qπ ( s , a , g ) = E [ T−1∑ t=0 γt · rg ( at , st ) | st = s , at = a , π ] , V ∗ ( s , g ) = max a Q∗ ( s , a , g ) . ( 1 ) We use off-policy algorithms named DQN ( Mnih et al. , 2013 ) for discrete action space and DDPG ( Lillicrap et al. , 2015 ) for continuous action space to learn Q-functions and policies by utilizing off-policy data ( i.e. , data stored in the replay buffer ) . 3 GRAPH STRUCTURED REINFORCEMENT LEARNING . As illustrated in Figure 1 , GSRL constructs a dynamic graph on top of state transitions . However , learning and planning algorithms on the state transition graph are non-trivial . In this section , we provide a theoretical analysis to show that exploration without any constraint would lead to the explosion of the graph . In order to guide the exploration with directions , we develop a novel goaloriented RL framework which incorporates the structure information of the state-transition graph . Specifically , we first divide states into several groups according to their uncertainty and locations in the graph . We then adopt an attention mechanism to select an appropriate group and assign the state with the highest value in the graph as a goal to encourage further exploration . We also propose to update goal generation hindsightly and value estimation with related trajectories , to help RL agents learn efficiently . 3.1 EXPLORE WITH DYNAMIC GRAPH . We directly build a dynamic and directed graph using sample trajectories from the replay buffer . We denote the state-transition graph at timestep t in episode e by Get = < Vet , Eet > with relations Ret , the node space Vet represents state space SGet , edge space Eet represents action space AGet , and relations Ret represents transition PGet . We denote the graph ’ s boundary by ∂Get , which consists of observed but not fully explored states . An illustrated example of these notations above is available at Figure 7 in Appendix A . Let an agent start from an initial state s0 . Then Get grows from a single node ( i.e. , G00 = { s0 } ) and expands itself at each timestep , leading to the sequence { { G1t } T−1t=0 , { G2t } T−1t=0 , . . . , { GEt } T−1t=0 } where T denotes episode length and E is the number of episodes . Given that state-transition graph Get is always a connected directed graph , we describe the expansion behavior as consecutive expansion , which means no jumping across neighborhood allowed . We denote the graph increment at timestep t in episode e as ∆Get , and then we can ensure that Get ∪∆Get = Get+1 ⊆ Get ∪ ∂Get . Proposition 1 . ( adapted from ( Xu et al. , 2020 ) ) We assume that the probability of degree of an arbitrary state in the whole state-transition graph Gwhole being less than or equal to d is larger than p ( i.e. , P ( deg ( s ) ≤ d ) > p , ∀s ∈ SG ) . We then consider a sequence of consecutively expanding sub-graphs { { G1t } T−1t=0 , { G2t } T−1t=0 , . . . , { GEt } T−1t=0 } , starting with G01 = { s0 } , for all t ≥ 0 , e ≥ 1 . We can ensure that P ( |SGet | ≤ ) > p , where = d· ( d−1 ) T ·e+t−2 d−2 when d > 2 and = 1 , 3 when d = 1 , 2 respectively . Proof . The basic proof structure follows from ( Xu et al. , 2020 ) ; however , because our task and detailed setting are considerably different , several modifications are required as described in in Appendix D.1 . The proposition implies that even if the given assumption of the whole state-transition graph Gwhole has a small d and a large p ( i.e. , sparse graph ) , the guarantee of upper-bounding |SGet | becomes looser and weaker as t , e get larger . In other words , randomly appending Get through exploration would enhance computation complexity for learning and plan on the dynamic graph . In order to prevent Get from an explosion , we need to constrain ∆Get by guiding exploration with goals . In this paper , we propose to leverage the graph structure to assist agent exploration . Before the detailed introduction of exploration strategy , we firstly introduce the definition of certainty as Definition 1 . Given a state s , we define its certainty as to the number of candidate actions that have been already taken . In this paper , we build the state-transition graph G on the replay buffer and thus can adopt the out degree∗ of each state to approximately measure the certainty . In other words , we can approximate the certainty of state cert ( s ) ≈ deg ( s ) , where deg ( s ) denotes the degree of node s. Note that certainty of state in Definition 1 can be served as a local measurement to show the extent of exploration on some states , which is actually proportioned to the global measurement ( i.e. , the number of visited states ) in a deterministic environment ( See Appendix E.1 for details ) . In addition , we also can define uncertainty of state as the number of untaken candidate actions . Exploration Strategy . A simple but effective way is to do exploration based on goal-oriented RL , which decomposes the whole task into a sequential of goal-oriented tasks . In this case , newly explored states in each episode are guided by a specific goal . Therefore , we constrain the statetransition graph from the explosion and balance the exploration and exploitation through assigning an appropriate goal to the agent . In order to further discuss the definition of an appropriate goal , we here provide the definition of the optimal goal . As shown in Figure 4 , once we are accessible to the whole state-transition graph Gwhole , we can obtain the optimal solution by short path planning algorithm such as Dijkstra ’ s Algorithm . We define the optimal goal as the terminal state on the path . Definition 2 . For fully explored state-transition graph Gfull† , we define the optimal goal g∗full as the terminal state ( e.g. , s10 in Figure 4 ( a ) ) . For any not fully explored state-transition graph Ge0 , we define the optimal goal g∗e hindsightly , where we generate the optimal solution path Pe+1 based on the shortest path planning on the next episode graph Ge+10 and regard the reachable terminal state included both in Ge0 and Pe+1 as g∗e ( e.g. , s6 in Figure 4 ( c ) ) . One should be noted that the optimal goal is hindsight generated and keep approaching the terminal state during exploration . We further analyze and discuss the relations between goals in this paper and previous goal-oriented RL literature in Appendix E.2 . Proposition 2 . Assume that Q-value of each state is well estimated ( i.e. , Q = Q∗ ) , then optimal goal g∗e at the beginning timestep of any episode e is always included in the boundary of the statetransition graph Ge0 ( i.e. , ∂Ge0 ) . Proof . The proof of Proposition 2 can be found in Appendix D.2 . The proposition implies that the candidate set for each goal generation can be limited to the boundary . However , learning to generate an appropriate goal is non-trivial with respect to a dynamic and large-scale state-transition graph Get . One intuitive solution is to divide the whole candidate goal space ( i.e. , ∂Get ) into several candidate groups C1 , . . . , CN , where N is the number of groups . In order to both cover all the states and control exploration size in these groups , our group segment should follow the principle that ∪Nn=1Cn = ∂Get and Cm ∩ Cn = ∅ for any m 6= n ; m , n = 1 , 2 , . . . , N . Here , we set the last visited state slast in the previous episodes as C0 , since slast often is both close to the terminal state and has high uncertainty . As shown in Figure 3 , based on the last explored state , we build group Cn , n = 1 , . . . , N by extending scales in two different perspectives . We list two perspectives , as follows : • Extending from Neighbor Nodes . One perspective is to leverage local structure on graph , where Cn : = { s } , s ∈ Nn ( slast ) ∩∂Get for n = 1 , 2 , . . . , N , and CN : = ∂Get −∪N−1n=1 Cn . Nn ( slast ) means n-hop neighbors of slast . The intuitive motivation behind this is to keep the learning procedure stable by slowly adjusting the goal to explore the surrounding environment . ∗In this paper , we use ‘ degree ’ to represent ‘ out-degree ’ , unless otherwise stated . †We define the full explored graph Gfull as the graph covering the optimal solution , and the whole graph Gwhole as the graph covering all the state-transitions . Hence , we have that Gfull is a sub-graph of Gwhole ( i.e. , Gfull ⊆ Gwhole ) . • Extending from Uncertain Nodes . The other perspective is to utilize certainty information to guide goal generation , where Cn : = { s } , s ∈ Sd=|A|−n ∩ ∂Get for n = 1 , 2 , . . . , N , and CN : = ∂Get −∪Nn=1Cn . Sd=|A|−n denotes set of states whose degree equals |A|−n , and |A| is the size of action space . The intuitive motivation behind this is to eliminate uncertainty in the graph through exploration . We provide the complexity analysis for two perspectives as follows . Let d∂Get denote the maximum degree of states in ∂Get , and |S∂Get | denote the number of states in ∂Get . The complexity to construct C1 , . . . , CN by extending from neighbor nodes is O ( dN−1∂Get ) and by extending from uncertain nodes is O ( |S∂Get | ) . The detailed analysis and method are available in Appendix E.3 . Attention Strategy . One should be noted that extending from either neighbor or uncertain nodes can guarantee the number of groups C1 , . . . , CN on a dynamic graph Get is fixed . In order to keep all the groups not empty , we append slast to them at the beginning timestep of each episode . However , RL algorithms face an exploration-exploitation dilemma , which requires the agent to makes appropriate selection rather than random exploration over these groups . In order to the trade-off between exploration and exploitation , we apply an attention mechanism to select an appropriate group on the current situation . For each group , we learn an embedding vector to represent its feature ( i.e. , certainty for extending from uncertain nodes ) . In this paper , the attention module runs over N groups at each episode and select the appropriate one , which needs to take the features of other groups into consideration . Therefore , it ’ s natural to adopt self-attention mechanism ( Vaswani et al. , 2017 ) here . For simplicity , we structure an embedding vector for each group and denote all the features of groups as [ f1 , . . . , fN ] , and then define Q = K = V : = ( f1 , . . . , fN ) T ∈ RN×d . The self-attention can be defined as ATTφ ( C1 , . . . , CN ) = softmax ( QKT√ N ) V , ( 2 ) where ATTφ denotes self-attention function parameterized by φ . The output of the self-attention is then fed to a multi-layer perception ( MLP ) with ReLU activation function . The output of MLP is in the dimension RN×1 . And the selected group can be obtained according to CATT = arg max Cn σ ( MLP ( ATTφ ( C1 , . . . , CN ) ) ) , ( 3 ) where σ ( · ) denotes a sigmoid function , and CATT means the group selected according to the attention score . At the beginning timestep of any episode e , we have access to state-transition graph Ge0 and its boundary ∂Ge0 . We then divide ∂Ge0 into groups C1 , . . . , CN , and obtain CATT through the aforementioned attention mechanism . We select one state with the highest value in CATT as the goal , which can be formulated as ge = arg max s V ( s , ge−1 ) ‡ , ∀s ∈ CATT . ( 4 ) In this way , we can generate an appropriate goal ge to guide the agent exploration in the episode e .

Summary: This paper introduces Graph Structured Reinforcement Learning (GSRL) framework, able to balance exploration and exploitation in RL. Actually, GSRL builds a dynamic graph based on historical trajectories. Then in order to learn from sparse or delayed rewards and  be able to reach a distant goal, it decomposes the main task into a sequence of easier and shorter tasks. An attention strategy has also been proposed that is able to select an appropriate goal for each one of the easiest tasks. Experiments have been conducted on various robotics manipulation tasks showing that GSRL performs better compared to HER and MAP algorithms.