Explore with Dynamic Map: Graph Structured Reinforcement Learning

1 INTRODUCTION . How can humans learn to solve tasks with complex structure and delayed and sparse feedback ? Take a complicated navigation task as an example in which the goal is to go from a start state to an end state . A straightforward approach that one often takes is by decomposing this complicated task into a sequence of easier ones , identifying/forming some intermediate goals that help to get to the final goal , and finally choosing a route with the highest return among a poll of candidate routes that lead to the final goal . Despite recent success and progress in reinforcement learning ( RL ) approaches ( Mnih et al. , 2013 ; Fakoor et al. , 2020 ; Hansen et al. , 2018 ; Huang et al. , 2019 ; Silver et al. , 2016 ) , there is still a huge gap between how humans and RL agents learn . In many real-world applications , an RL agent only has access to sparse and delayed rewards , which by itself leads to two major challenges . ( C1 ) The first one is how can an agent effectively learn from sparse and delayed rewards ? One possible solution is to build a new reward function , known as an intrinsic reward , that helps expedite the learning process and ultimately solve a given task . Although this may seem like an appealing solution , it is often not obvious how to formulate an effective intrinsic reward . Recent works have introduced goal-oriented RL ( Schaul et al. , 2015 ; Chiang et al. , 2019 ) as a way of constructing intrinsic rewards . However , most goal-oriented RL algorithms require large amounts of reward shaping ( Chiang et al. , 2019 ) or human demonstrations ( Nair et al. , 2018 ) . ( C2 ) The second issue is how to retrieve relevant past experiences that help to learn faster and improve sample efficiency . Recent attention has focused much on episodic RL ( Blundell et al. , 2016 ) , which builds a non-parametric episodic memory to store past experiences , and thus can rapidly latch onto related ones through search with similarity . However , most episodic RL algorithms require additional memory space ( Pritzel et al. , 2017 ) . In this work , we propose Graph Structured Reinforcement Learning ( GSRL ) , which constructs a state-transition graph , leverages structural graph information , and presents solutions for the problems raised above . As shown in Figure 1 , when we encounter a complex task , we draw a map for planning with a long horizon ( Eysenbach et al. , 2019 ) and scan through the graph to seek related experiences within a short horizon ( Lee et al. , 2019 ) . Inspired by this , we propose a method to address the first problem ( C1 ) by learning to generate an appropriate goal ge based on map Ge0 at episode e and form the intrinsic rewards during goal generation . This goal generation can be optimized in hindsight . That is , we can update the goal generation model under the supervision generated by planning algorithms based on map Ge+10 at episode e + 1 . We can solve the second problem ( C2 ) by updating the value estimation of the current state with the trajectories that include neighborhood states . Note that GSRL draws a map to help agents plan an appropriate direction in the long horizon and learn an optimal strategy in the short horizon , which can be widely incorporated with various learning strategies such as DQN ( Mnih et al. , 2013 ) , DDPG ( Lillicrap et al. , 2015 ) , etc . However , arriving at a well-explored map in terms of state-transitions in the first place from scratch remains a challenging problem . In this regard , GSRL should be able to jointly optimize map construction , goal selection , and value estimation . As illustrated in Figure 2 , we design GSRL mainly following four steps : ( a ) At the beginning , an agent is located at the initial state and aims at the terminal state with an empty map ; ( b ) The agent then explores the environment with the current policy and records those explored states on the map ; ( c ) Considering the far distance to the terminal state and sparse reward during the exploration , we decompose the whole task into a sequence of easier tasks guided by goals ; ( d ) We leverage structured map information to select related trajectories to update the current policy ; The agent can repeat procedures ( b ) - ( d ) and finally reach the terminal state with a well-explored map . Our primary contribution is a novel algorithm that constructs a dynamic state-transition graph from scratch and leverages structured information to find an appropriate goal in goal selection with a long horizon and sample related trajectories in value estimation with a short horizon . This framework provides several benefits : ( 1 ) How to balance exploration and exploitation is a fundamental problem in any reinforcement learning ( Andrychowicz et al. , 2017 ; Nachum et al. , 2018 ) . GSRL learns an attention mechanism on the graph to decide whether to explore the states with high uncertainty or exploit the states with well estimated value . ( 2 ) Almost all the existing reinforcement learning approaches ( Paul et al. , 2019 ; Eysenbach et al. , 2019 ) utilize randomly sampled trajectories to update the current policy leading to inefficiency . GSRL selects highly related trajectories for updates through the local structure in the graph . ( 3 ) Unlike previous approaches ( Kipf et al. , 2020 ; Shang et al. , 2019 ) , GSRL constructs a dynamic graph through agent exploration without any additional object representation techniques and thus is not limited to several specific environments . Empirically , we find that our method constructs a useful map and learns an efficient policy for highly structured tasks . Comparisons with state-of-the-art RL methods show that GSRL is substantially more successful for boosting the performance of a graph-structured RL algorithm . 2 PRELIMINARIES . LetM = ( S , A , T , P , R ) be an Markov Decision Process ( MDP ) where S is the state space , A is the action space whose size is bounded by a constant , T ∈ Z+ is the episode length , P : S×A→ ∆ ( S ) is the transition function which takes a state-action pair and returns a distribution over states , and R : S × A→ ∆ ( R ) is the reward distribution . The rewards from the environment , called extrinsic , are usually sparse and , therefore , difficult to learn . To address this , goal-oriented RL introduces goal g ∈ G to build dense intrinsic rewards Rg : S × A × G → ∆ ( R ) . In order to make the learning procedure stable , the goal is limited to be updated episodically . A policy π : S × G → ∆ ( A ) prescribes a distribution over actions for each state and goal . At each timestep , the agent samples an action a ∼ π ( s , g ) and receives a corresponding reward rg ( s , a ) that indicates whether or not the agent has reached the goal . The episode terminates only after T timesteps even when the agent reaches the goal . The agent ’ s task is to maximize its cumulative discounted future reward . We use an off-policy algorithm to learn such a policy , as well as its associated goal-oriented Q-function and value function : Qπ ( s , a , g ) = E [ T−1∑ t=0 γt · rg ( at , st ) | st = s , at = a , π ] , V ∗ ( s , g ) = max a Q∗ ( s , a , g ) . ( 1 ) We use off-policy algorithms named DQN ( Mnih et al. , 2013 ) for discrete action space and DDPG ( Lillicrap et al. , 2015 ) for continuous action space to learn Q-functions and policies by utilizing off-policy data ( i.e. , data stored in the replay buffer ) . 3 GRAPH STRUCTURED REINFORCEMENT LEARNING . As illustrated in Figure 1 , GSRL constructs a dynamic graph on top of state transitions . However , learning and planning algorithms on the state transition graph are non-trivial . In this section , we provide a theoretical analysis to show that exploration without any constraint would lead to the explosion of the graph . In order to guide the exploration with directions , we develop a novel goaloriented RL framework which incorporates the structure information of the state-transition graph . Specifically , we first divide states into several groups according to their uncertainty and locations in the graph . We then adopt an attention mechanism to select an appropriate group and assign the state with the highest value in the graph as a goal to encourage further exploration . We also propose to update goal generation hindsightly and value estimation with related trajectories , to help RL agents learn efficiently . 3.1 EXPLORE WITH DYNAMIC GRAPH . We directly build a dynamic and directed graph using sample trajectories from the replay buffer . We denote the state-transition graph at timestep t in episode e by Get = < Vet , Eet > with relations Ret , the node space Vet represents state space SGet , edge space Eet represents action space AGet , and relations Ret represents transition PGet . We denote the graph ’ s boundary by ∂Get , which consists of observed but not fully explored states . An illustrated example of these notations above is available at Figure 7 in Appendix A . Let an agent start from an initial state s0 . Then Get grows from a single node ( i.e. , G00 = { s0 } ) and expands itself at each timestep , leading to the sequence { { G1t } T−1t=0 , { G2t } T−1t=0 , . . . , { GEt } T−1t=0 } where T denotes episode length and E is the number of episodes . Given that state-transition graph Get is always a connected directed graph , we describe the expansion behavior as consecutive expansion , which means no jumping across neighborhood allowed . We denote the graph increment at timestep t in episode e as ∆Get , and then we can ensure that Get ∪∆Get = Get+1 ⊆ Get ∪ ∂Get . Proposition 1 . ( adapted from ( Xu et al. , 2020 ) ) We assume that the probability of degree of an arbitrary state in the whole state-transition graph Gwhole being less than or equal to d is larger than p ( i.e. , P ( deg ( s ) ≤ d ) > p , ∀s ∈ SG ) . We then consider a sequence of consecutively expanding sub-graphs { { G1t } T−1t=0 , { G2t } T−1t=0 , . . . , { GEt } T−1t=0 } , starting with G01 = { s0 } , for all t ≥ 0 , e ≥ 1 . We can ensure that P ( |SGet | ≤ ) > p , where = d· ( d−1 ) T ·e+t−2 d−2 when d > 2 and = 1 , 3 when d = 1 , 2 respectively . Proof . The basic proof structure follows from ( Xu et al. , 2020 ) ; however , because our task and detailed setting are considerably different , several modifications are required as described in in Appendix D.1 . The proposition implies that even if the given assumption of the whole state-transition graph Gwhole has a small d and a large p ( i.e. , sparse graph ) , the guarantee of upper-bounding |SGet | becomes looser and weaker as t , e get larger . In other words , randomly appending Get through exploration would enhance computation complexity for learning and plan on the dynamic graph . In order to prevent Get from an explosion , we need to constrain ∆Get by guiding exploration with goals . In this paper , we propose to leverage the graph structure to assist agent exploration . Before the detailed introduction of exploration strategy , we firstly introduce the definition of certainty as Definition 1 . Given a state s , we define its certainty as to the number of candidate actions that have been already taken . In this paper , we build the state-transition graph G on the replay buffer and thus can adopt the out degree∗ of each state to approximately measure the certainty . In other words , we can approximate the certainty of state cert ( s ) ≈ deg ( s ) , where deg ( s ) denotes the degree of node s. Note that certainty of state in Definition 1 can be served as a local measurement to show the extent of exploration on some states , which is actually proportioned to the global measurement ( i.e. , the number of visited states ) in a deterministic environment ( See Appendix E.1 for details ) . In addition , we also can define uncertainty of state as the number of untaken candidate actions . Exploration Strategy . A simple but effective way is to do exploration based on goal-oriented RL , which decomposes the whole task into a sequential of goal-oriented tasks . In this case , newly explored states in each episode are guided by a specific goal . Therefore , we constrain the statetransition graph from the explosion and balance the exploration and exploitation through assigning an appropriate goal to the agent . In order to further discuss the definition of an appropriate goal , we here provide the definition of the optimal goal . As shown in Figure 4 , once we are accessible to the whole state-transition graph Gwhole , we can obtain the optimal solution by short path planning algorithm such as Dijkstra ’ s Algorithm . We define the optimal goal as the terminal state on the path . Definition 2 . For fully explored state-transition graph Gfull† , we define the optimal goal g∗full as the terminal state ( e.g. , s10 in Figure 4 ( a ) ) . For any not fully explored state-transition graph Ge0 , we define the optimal goal g∗e hindsightly , where we generate the optimal solution path Pe+1 based on the shortest path planning on the next episode graph Ge+10 and regard the reachable terminal state included both in Ge0 and Pe+1 as g∗e ( e.g. , s6 in Figure 4 ( c ) ) . One should be noted that the optimal goal is hindsight generated and keep approaching the terminal state during exploration . We further analyze and discuss the relations between goals in this paper and previous goal-oriented RL literature in Appendix E.2 . Proposition 2 . Assume that Q-value of each state is well estimated ( i.e. , Q = Q∗ ) , then optimal goal g∗e at the beginning timestep of any episode e is always included in the boundary of the statetransition graph Ge0 ( i.e. , ∂Ge0 ) . Proof . The proof of Proposition 2 can be found in Appendix D.2 . The proposition implies that the candidate set for each goal generation can be limited to the boundary . However , learning to generate an appropriate goal is non-trivial with respect to a dynamic and large-scale state-transition graph Get . One intuitive solution is to divide the whole candidate goal space ( i.e. , ∂Get ) into several candidate groups C1 , . . . , CN , where N is the number of groups . In order to both cover all the states and control exploration size in these groups , our group segment should follow the principle that ∪Nn=1Cn = ∂Get and Cm ∩ Cn = ∅ for any m 6= n ; m , n = 1 , 2 , . . . , N . Here , we set the last visited state slast in the previous episodes as C0 , since slast often is both close to the terminal state and has high uncertainty . As shown in Figure 3 , based on the last explored state , we build group Cn , n = 1 , . . . , N by extending scales in two different perspectives . We list two perspectives , as follows : • Extending from Neighbor Nodes . One perspective is to leverage local structure on graph , where Cn : = { s } , s ∈ Nn ( slast ) ∩∂Get for n = 1 , 2 , . . . , N , and CN : = ∂Get −∪N−1n=1 Cn . Nn ( slast ) means n-hop neighbors of slast . The intuitive motivation behind this is to keep the learning procedure stable by slowly adjusting the goal to explore the surrounding environment . ∗In this paper , we use ‘ degree ’ to represent ‘ out-degree ’ , unless otherwise stated . †We define the full explored graph Gfull as the graph covering the optimal solution , and the whole graph Gwhole as the graph covering all the state-transitions . Hence , we have that Gfull is a sub-graph of Gwhole ( i.e. , Gfull ⊆ Gwhole ) . • Extending from Uncertain Nodes . The other perspective is to utilize certainty information to guide goal generation , where Cn : = { s } , s ∈ Sd=|A|−n ∩ ∂Get for n = 1 , 2 , . . . , N , and CN : = ∂Get −∪Nn=1Cn . Sd=|A|−n denotes set of states whose degree equals |A|−n , and |A| is the size of action space . The intuitive motivation behind this is to eliminate uncertainty in the graph through exploration . We provide the complexity analysis for two perspectives as follows . Let d∂Get denote the maximum degree of states in ∂Get , and |S∂Get | denote the number of states in ∂Get . The complexity to construct C1 , . . . , CN by extending from neighbor nodes is O ( dN−1∂Get ) and by extending from uncertain nodes is O ( |S∂Get | ) . The detailed analysis and method are available in Appendix E.3 . Attention Strategy . One should be noted that extending from either neighbor or uncertain nodes can guarantee the number of groups C1 , . . . , CN on a dynamic graph Get is fixed . In order to keep all the groups not empty , we append slast to them at the beginning timestep of each episode . However , RL algorithms face an exploration-exploitation dilemma , which requires the agent to makes appropriate selection rather than random exploration over these groups . In order to the trade-off between exploration and exploitation , we apply an attention mechanism to select an appropriate group on the current situation . For each group , we learn an embedding vector to represent its feature ( i.e. , certainty for extending from uncertain nodes ) . In this paper , the attention module runs over N groups at each episode and select the appropriate one , which needs to take the features of other groups into consideration . Therefore , it ’ s natural to adopt self-attention mechanism ( Vaswani et al. , 2017 ) here . For simplicity , we structure an embedding vector for each group and denote all the features of groups as [ f1 , . . . , fN ] , and then define Q = K = V : = ( f1 , . . . , fN ) T ∈ RN×d . The self-attention can be defined as ATTφ ( C1 , . . . , CN ) = softmax ( QKT√ N ) V , ( 2 ) where ATTφ denotes self-attention function parameterized by φ . The output of the self-attention is then fed to a multi-layer perception ( MLP ) with ReLU activation function . The output of MLP is in the dimension RN×1 . And the selected group can be obtained according to CATT = arg max Cn σ ( MLP ( ATTφ ( C1 , . . . , CN ) ) ) , ( 3 ) where σ ( · ) denotes a sigmoid function , and CATT means the group selected according to the attention score . At the beginning timestep of any episode e , we have access to state-transition graph Ge0 and its boundary ∂Ge0 . We then divide ∂Ge0 into groups C1 , . . . , CN , and obtain CATT through the aforementioned attention mechanism . We select one state with the highest value in CATT as the goal , which can be formulated as ge = arg max s V ( s , ge−1 ) ‡ , ∀s ∈ CATT . ( 4 ) In this way , we can generate an appropriate goal ge to guide the agent exploration in the episode e .

Summary: This work presents a strategy for improving exploration and efficiency of RL by leveraging the graph structure of an episodic experience buffer. This strategy combines goal-oriented RL with structured exploration. The authors compare their proposed technique to two popular benchmarks for goal-reaching tasks. In addition, the authors provide some theoretical justification for their algorithmic choices.