Graph-Enhanced Exploration for Goal-oriented Reinforcement Learning

1 INTRODUCTION . Goal-oriented Reinforcement Learning ( GoRL ) ( Nasiriany et al. , 2019 ; Eysenbach et al. , 2019 ; Huang et al. , 2019 ; Levy et al. , 2017 ; Kulkarni et al. , 2016a ) allows RL methods to tackle complex tasks with long-term credit assignment and sparse reward . The basic idea of GoRL is to comprise a goal generator to generate intermediate goals ( also called subgoals in previous literature ( Zhang et al. , 2020 ; Paul et al. , 2019 ) ) which decomposes the original complicate task into a series of intermediate easier subtasks guided by goals , and a policy learner aiming to reach those generated goals . However , the performance of GoRL significantly relies on the generation of effective goals , which still remains a key challenge . One promising solution is to construct suitable abstraction graphs of the environment and generate goals by classical planning algorithms on graph . These abstraction graphs can either be built with the state-transitions based on the collected experience ( Eysenbach et al. , 2019 ) or the abstract structure of the environment based on the observation ( Shang et al. , 2019 ) . However , almost all existing approaches ( Huang et al. , 2019 ; Laskin et al. , 2020 ; Zhu et al. , 2019 ) focus on either graph construction based on the collected experience or agent exploitation over the graph by adopting classical planning techniques on graph ( e.g. , Dijkstra ’ s algorithm , A * search ) ( Laskin et al. , 2020 ; Eysenbach et al. , 2019 ; Huang et al. , 2019 ) . There has been limited literature pursuing more effective and informative exploration by strategical ( especially differentiable ) goal generation based on the abstraction graph , which is challenging as the graph built on either state-transitions or observations is inherently dynamic and sometimes even much large in scale . In summary , most existing methods still suffer from two aspects of difficulties . ( 1 ) The goal space is often large as the state space , leading to difficulty in effective exploration ( Zhang et al. , 2020 ) ( 2 ) The historical trajectories are collected by various goals , and random sampling would result in sample inefficiency for policy learning ( Fang et al. , 2019 ) , because of low reachability of the generated goals . In this paper , we propose Graph-enhanced GoRL ( G2RL ) , a novel GoRL framework that , based on a dynamically built state-transition graph , ( 1 ) develops a differentiable goal generator with hindsight supervision for effective environment explorations , and ( 2 ) designs a new trajectory sampling algorithm with goal proximity for efficient policy learning , as illustrated in Figure 1 ( a ) . Concretely , we construct the state-transition graph from the replay buffer , where the nodes are the states and edges are the transitions . We extract the graph directly from the experiences for discrete environment and applyK-bins discretization technique ( Kotsiantis & Kanellopoulos , 2006 ) for continuous environment . Notably , as shown in Figure 1 ( b ) , the graph in G2RL can be regarded as an auxiliary structural information for goal generation and trajectory sampling , and thus the discretization does not change a deterministic environment into a stochastic one . For effective exploration , we begin with revealing the connection between the out-degree of nodes and the status of exploration on the corresponding states . The intuition behind this is quite straightforward : if the out-degree of a node is small , it implies that its next states have not been much visited , namely this state itself is not well explored . In the light of it , we consider these not well explored nodes are valuable for exploration and define the set of these nodes as graph boundary . We then enforce the goal generated from the states in the boundary , as we theoretically show that the optimal goals are preserved within the graph boundary . For handling graph ’ s dynamics and large scale , we divide the graph boundary into several groups according to their out-degrees ( i.e. , the status of exploration ) . In such a way , our method is scalable , as it only needs to build a fix number of groups . We further develop an attention mechanism to select an appropriate group , in which the state with the highest value is assigned as intermediate goal . We introduce optimal goals as the supervision signals to learn our differentiable goal generator . These optimal goals are the most valuable and reachable states defined based on the planning algorithm on the graph in the next episode , and thus our goal generator is optimized in a hindsight manner . Furthermore , for efficient policy learning , almost all existing GoRL approaches ( Paul et al. , 2019 ; Eysenbach et al. , 2019 ) utilize randomly sampled trajectories to update the policy , leading to inefficiency as there are various subtasks guided by different goals . Instead , we design a novel relevance sampling technique , where the basic intuition is when updating the value of a state , the trajectories containing it and its neighboring states are likely to share similar goals . Hence , we call these trajectories relevant trajectories and give high priority to sampling these trajectories in policy learning . The contributions of this paper are as follows : • We propose a graph structure enhanced goal generator built upon attention mechanism , which achieves effective exploration with optimal goals and can be learned with hindsight supervisions . • We design a relevance sampling technique that leverages the state-transition graph information to select relevant experience sharing the similar directions ( i.e. , goals ) for efficient policy learning . • We theoretically show that optimal goals can be preserved in the boundary of the state-transition graph and empirically illustrate that our method is capable to capture these goals . We benchmark our method on various tasks , including discrete control and planning tasks on grid worlds and challenging continuous control tasks , which are widely used in GoRL literature ( Florensa et al. , 2018 ; Nachum et al. , 2018a ; b ; Savinov et al. , 2018 ) . Experimental results exhibit the superiority of G2RL on both asymptotic performance and sample efficiency compared with several state-of-the-art GoRL approaches , demonstrating the effectiveness of G2RL . 2 GRAPH CONSTRUCTION AND EXPLORATION OF GOAL-ORIENTED RL . 2.1 GOAL-ORIENTED REINFORCEMENT LEARNING . Let M = ( S , A , T , P , R ) be a Markov Decision Process ( MDP ) where S is the state space , A is the action space , T ∈ Z+ is the episode length , P : S × A → ∆ ( S ) is the transition function which takes a state-action pair and returns a distribution over states , and R : S ×A→ ∆ ( R ) is the reward distribution . The rewards from environments , called extrinsic , are usually sparse and thus , difficult to learn . To address this , following the basic idea of GoRL , we consider a framework comprising two main components : a goal generator aiming to generate goals g ∈ G at each episode ; and a policy learner aiming to maximize the intrinsic reward conditioned on generated goals , formulated as Rg : S × A × G → ∆ ( R ) , and performs a primary action at every timestep . Following prior methods ( Nachum et al. , 2018a ; Andrychowicz et al. , 2017 ; Zhang et al. , 2020 ) , we enforce goal space G as a subspace of S. A policy π : S ×G→ ∆ ( A ) prescribes a distribution over actions for each state and goal . We introduce π as a greedy deterministic policy . At each timestep , the agent samples an action a ∼ π ( s , g ) and receives a corresponding reward rg ( s , a ) that indicates whether or not the agent has reached the goal . The agent ’ s task is to maximize its cumulative discounted future reward . We use an off-policy algorithm to learn such a policy , as well as its associated goal-oriented Q-function and state value function : Qπ ( s , a , g ) = E [ T−1∑ t=0 γtrg ( at , st ) | st = s , at = a , π ] , V π ( s , g ) = ∑ a π ( a|s , g ) Qπ ( s , a , g ) . ( 1 ) 2.2 GRAPH CONSTRUCTION FROM REPLAY BUFFER . As Figure 1 ( b ) illustrates , we build a dynamic and directed graph on top of historical trajectories in the replay buffer at each timestep . We denote the state-transition graph at timestep t in episode e as Get = ( Vet , Eet ) with relations Ret , where the node space Vet represents state space SGet , edge space Eet represents transitions PGet , and relations Ret represents actions AGet . PGet determines the existence of transition from st to st+1 , and AGet shows the way to transit . For simplicity , we use Ge to denote Ge0 . See Appendix A1 for an illustrated example of notations . Note that it ’ s non-trivial to construct an abstraction graph with continuous environments . Existing approaches either organize the graph based on sampled trajectories ( Zhu et al. , 2019 ; Huang et al. , 2019 ) , or introduce a parameterized distance network ( Savinov et al. , 2018 ; Eysenbach et al. , 2019 ) to predict the distance of each pair of states and establish edges for those pairs whose distances are above the pre-defined threshold . In our implementation , we employ the K-bins discretization technique ( Kotsiantis & Kanellopoulos , 2006 ) to discretize the state and action spaces , as it has been shown that animals explore their environments with the help of grid cells activated on particular intervals ( O ’ Keefe & Dostrovsky , 1971 ; Banino et al. , 2018 ; Colombo et al. , 1998 ) .∗ Hence , in this case , each node of the graph represents a bin of states/actions with similar locations/directions ; and an edge exists when the number of transitions between pairs of states from two nodes reach a predefined threshold , and the corresponding relations show the actions of the transition . Formally , we can obtain discretized state and action spaces , denoted as Ŝ and Â . Notably , this discretization technique can consistently map a box to a fixed integer , and thus will not operate a deterministic MDP into a stochastic one . Without loss of generality , in the following parts , we mainly discuss in the context of discrete environments , as we have described how to extend it into continuous ones . Let an agent start from a start state sstart , aiming to a target state starget . Then , Get grows from a single node ( i.e. , G00 = { vstart } ) and expands itself at each timestep within each episode , leading to the sequence { { G1t } T−1t=0 , { G2t } T−1t=0 , . . . , { GEt } T−1t=0 } where T denotes episode length and E is the number of episodes . As the replay buffer always maintains the past experiences from the previous episodes , we have that Ge0 = Ge−1T−1 holds for ∀e = 1 , . . . , E. Given that Get is always a connected directed graph , we describe the expansion behavior as consecutive expansion , which naturally corresponds to exploration process of the agent of approaching the Gwhole , where we define Gwhole as the whole graph containing all possible transitions .

Summary: This paper improves exploration and learning efficiency in Goal-oriented Reinforcement Learning (GoRL). In particular, a new framework graph-enhanced GoRL (G2RL) is proposed by leveraging the state-transition graph. (1) G2RL produces under-explored goals (or goal states) by planning on the “one-episode ahead” graph to conduct directed and effective exploration. In the actual implementation, the goal is generated by a differentiable generator trained in a hindsight manner.  (2) In addition, G2RL proposes to use “relevant trajectories” based on the graph neighborhood information to improve the efficiency of policy learning. By combining (1) and (2), G2RL demonstrated strong performance on various environments. Theoretical analysis is provided on deterministic MDPs to support the design of the algorithm.