Graph-Enhanced Exploration for Goal-oriented Reinforcement Learning

1 INTRODUCTION . Goal-oriented Reinforcement Learning ( GoRL ) ( Nasiriany et al. , 2019 ; Eysenbach et al. , 2019 ; Huang et al. , 2019 ; Levy et al. , 2017 ; Kulkarni et al. , 2016a ) allows RL methods to tackle complex tasks with long-term credit assignment and sparse reward . The basic idea of GoRL is to comprise a goal generator to generate intermediate goals ( also called subgoals in previous literature ( Zhang et al. , 2020 ; Paul et al. , 2019 ) ) which decomposes the original complicate task into a series of intermediate easier subtasks guided by goals , and a policy learner aiming to reach those generated goals . However , the performance of GoRL significantly relies on the generation of effective goals , which still remains a key challenge . One promising solution is to construct suitable abstraction graphs of the environment and generate goals by classical planning algorithms on graph . These abstraction graphs can either be built with the state-transitions based on the collected experience ( Eysenbach et al. , 2019 ) or the abstract structure of the environment based on the observation ( Shang et al. , 2019 ) . However , almost all existing approaches ( Huang et al. , 2019 ; Laskin et al. , 2020 ; Zhu et al. , 2019 ) focus on either graph construction based on the collected experience or agent exploitation over the graph by adopting classical planning techniques on graph ( e.g. , Dijkstra ’ s algorithm , A * search ) ( Laskin et al. , 2020 ; Eysenbach et al. , 2019 ; Huang et al. , 2019 ) . There has been limited literature pursuing more effective and informative exploration by strategical ( especially differentiable ) goal generation based on the abstraction graph , which is challenging as the graph built on either state-transitions or observations is inherently dynamic and sometimes even much large in scale . In summary , most existing methods still suffer from two aspects of difficulties . ( 1 ) The goal space is often large as the state space , leading to difficulty in effective exploration ( Zhang et al. , 2020 ) ( 2 ) The historical trajectories are collected by various goals , and random sampling would result in sample inefficiency for policy learning ( Fang et al. , 2019 ) , because of low reachability of the generated goals . In this paper , we propose Graph-enhanced GoRL ( G2RL ) , a novel GoRL framework that , based on a dynamically built state-transition graph , ( 1 ) develops a differentiable goal generator with hindsight supervision for effective environment explorations , and ( 2 ) designs a new trajectory sampling algorithm with goal proximity for efficient policy learning , as illustrated in Figure 1 ( a ) . Concretely , we construct the state-transition graph from the replay buffer , where the nodes are the states and edges are the transitions . We extract the graph directly from the experiences for discrete environment and applyK-bins discretization technique ( Kotsiantis & Kanellopoulos , 2006 ) for continuous environment . Notably , as shown in Figure 1 ( b ) , the graph in G2RL can be regarded as an auxiliary structural information for goal generation and trajectory sampling , and thus the discretization does not change a deterministic environment into a stochastic one . For effective exploration , we begin with revealing the connection between the out-degree of nodes and the status of exploration on the corresponding states . The intuition behind this is quite straightforward : if the out-degree of a node is small , it implies that its next states have not been much visited , namely this state itself is not well explored . In the light of it , we consider these not well explored nodes are valuable for exploration and define the set of these nodes as graph boundary . We then enforce the goal generated from the states in the boundary , as we theoretically show that the optimal goals are preserved within the graph boundary . For handling graph ’ s dynamics and large scale , we divide the graph boundary into several groups according to their out-degrees ( i.e. , the status of exploration ) . In such a way , our method is scalable , as it only needs to build a fix number of groups . We further develop an attention mechanism to select an appropriate group , in which the state with the highest value is assigned as intermediate goal . We introduce optimal goals as the supervision signals to learn our differentiable goal generator . These optimal goals are the most valuable and reachable states defined based on the planning algorithm on the graph in the next episode , and thus our goal generator is optimized in a hindsight manner . Furthermore , for efficient policy learning , almost all existing GoRL approaches ( Paul et al. , 2019 ; Eysenbach et al. , 2019 ) utilize randomly sampled trajectories to update the policy , leading to inefficiency as there are various subtasks guided by different goals . Instead , we design a novel relevance sampling technique , where the basic intuition is when updating the value of a state , the trajectories containing it and its neighboring states are likely to share similar goals . Hence , we call these trajectories relevant trajectories and give high priority to sampling these trajectories in policy learning . The contributions of this paper are as follows : • We propose a graph structure enhanced goal generator built upon attention mechanism , which achieves effective exploration with optimal goals and can be learned with hindsight supervisions . • We design a relevance sampling technique that leverages the state-transition graph information to select relevant experience sharing the similar directions ( i.e. , goals ) for efficient policy learning . • We theoretically show that optimal goals can be preserved in the boundary of the state-transition graph and empirically illustrate that our method is capable to capture these goals . We benchmark our method on various tasks , including discrete control and planning tasks on grid worlds and challenging continuous control tasks , which are widely used in GoRL literature ( Florensa et al. , 2018 ; Nachum et al. , 2018a ; b ; Savinov et al. , 2018 ) . Experimental results exhibit the superiority of G2RL on both asymptotic performance and sample efficiency compared with several state-of-the-art GoRL approaches , demonstrating the effectiveness of G2RL . 2 GRAPH CONSTRUCTION AND EXPLORATION OF GOAL-ORIENTED RL . 2.1 GOAL-ORIENTED REINFORCEMENT LEARNING . Let M = ( S , A , T , P , R ) be a Markov Decision Process ( MDP ) where S is the state space , A is the action space , T ∈ Z+ is the episode length , P : S × A → ∆ ( S ) is the transition function which takes a state-action pair and returns a distribution over states , and R : S ×A→ ∆ ( R ) is the reward distribution . The rewards from environments , called extrinsic , are usually sparse and thus , difficult to learn . To address this , following the basic idea of GoRL , we consider a framework comprising two main components : a goal generator aiming to generate goals g ∈ G at each episode ; and a policy learner aiming to maximize the intrinsic reward conditioned on generated goals , formulated as Rg : S × A × G → ∆ ( R ) , and performs a primary action at every timestep . Following prior methods ( Nachum et al. , 2018a ; Andrychowicz et al. , 2017 ; Zhang et al. , 2020 ) , we enforce goal space G as a subspace of S. A policy π : S ×G→ ∆ ( A ) prescribes a distribution over actions for each state and goal . We introduce π as a greedy deterministic policy . At each timestep , the agent samples an action a ∼ π ( s , g ) and receives a corresponding reward rg ( s , a ) that indicates whether or not the agent has reached the goal . The agent ’ s task is to maximize its cumulative discounted future reward . We use an off-policy algorithm to learn such a policy , as well as its associated goal-oriented Q-function and state value function : Qπ ( s , a , g ) = E [ T−1∑ t=0 γtrg ( at , st ) | st = s , at = a , π ] , V π ( s , g ) = ∑ a π ( a|s , g ) Qπ ( s , a , g ) . ( 1 ) 2.2 GRAPH CONSTRUCTION FROM REPLAY BUFFER . As Figure 1 ( b ) illustrates , we build a dynamic and directed graph on top of historical trajectories in the replay buffer at each timestep . We denote the state-transition graph at timestep t in episode e as Get = ( Vet , Eet ) with relations Ret , where the node space Vet represents state space SGet , edge space Eet represents transitions PGet , and relations Ret represents actions AGet . PGet determines the existence of transition from st to st+1 , and AGet shows the way to transit . For simplicity , we use Ge to denote Ge0 . See Appendix A1 for an illustrated example of notations . Note that it ’ s non-trivial to construct an abstraction graph with continuous environments . Existing approaches either organize the graph based on sampled trajectories ( Zhu et al. , 2019 ; Huang et al. , 2019 ) , or introduce a parameterized distance network ( Savinov et al. , 2018 ; Eysenbach et al. , 2019 ) to predict the distance of each pair of states and establish edges for those pairs whose distances are above the pre-defined threshold . In our implementation , we employ the K-bins discretization technique ( Kotsiantis & Kanellopoulos , 2006 ) to discretize the state and action spaces , as it has been shown that animals explore their environments with the help of grid cells activated on particular intervals ( O ’ Keefe & Dostrovsky , 1971 ; Banino et al. , 2018 ; Colombo et al. , 1998 ) .∗ Hence , in this case , each node of the graph represents a bin of states/actions with similar locations/directions ; and an edge exists when the number of transitions between pairs of states from two nodes reach a predefined threshold , and the corresponding relations show the actions of the transition . Formally , we can obtain discretized state and action spaces , denoted as Ŝ and Â . Notably , this discretization technique can consistently map a box to a fixed integer , and thus will not operate a deterministic MDP into a stochastic one . Without loss of generality , in the following parts , we mainly discuss in the context of discrete environments , as we have described how to extend it into continuous ones . Let an agent start from a start state sstart , aiming to a target state starget . Then , Get grows from a single node ( i.e. , G00 = { vstart } ) and expands itself at each timestep within each episode , leading to the sequence { { G1t } T−1t=0 , { G2t } T−1t=0 , . . . , { GEt } T−1t=0 } where T denotes episode length and E is the number of episodes . As the replay buffer always maintains the past experiences from the previous episodes , we have that Ge0 = Ge−1T−1 holds for ∀e = 1 , . . . , E. Given that Get is always a connected directed graph , we describe the expansion behavior as consecutive expansion , which naturally corresponds to exploration process of the agent of approaching the Gwhole , where we define Gwhole as the whole graph containing all possible transitions .

Summary: In this work, the authors describe a novel graph-augmented RL method for goal conditioned tasks. The results shown outperform a number of baseline methods. The method is well motivated and theoreticaly grounded.