Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning

1 Introduction . Consider deploying a self-driving car to a new city . To be practical , the car should be able to explore the city such that it can learn to traverse from any starting location to any destination , since the destination may vary depending on the passenger . In the context of reinforcement learning ( RL ) , this problem is known as goal-conditioned RL ( GCRL ) [ 12 , 13 ] . Previous works [ 31 , 1 , 24 , 26 , 17 ] have tackled this problem by learning a goal-conditioned policy ( or value function ) applicable to any reward function or “ goal. ” However , the goal-conditioned policy often fails to scale to long-horizon goals [ 11 ] since the space of state-goal pairs grows intractably large over the horizon of the goal . To address this challenge , the agent needs to ( a ) explore the state-goal space such that it is proficient for any state-goal pair it might observe during test time and ( b ) reduce the effective goal horizon for the policy learning to be tractable . Recent work [ 25 , 11 ] has tackled long-horizon GCRL by leveraging model-based approaches to form plans consisting of lower temporal-resolution subgoals . The policy is then only required to operate for short horizons between these subgoals . One line of work learned a universal value function approximator ( UVFA ) [ 31 ] to make local policy decisions and to estimate distances used for building a landmark-based map , but assumed a low-dimensional state space where the proximity between the state and goal could be computed by the Euclidean distance [ 11 ] . Another line of research focused on visual navigation tasks conducted planning over 1The demo video and code can be found at https : //2016choang.github.io/sfl . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) graph representations of the environment [ 29 , 9 , 16 , 4 ] . However , these studies largely ignored the inherent exploration challenge present for large state spaces , and either assumed the availability of human demonstrations of exploring the state space [ 29 ] , the ability to spawn uniformly over the state space [ 9 , 16 ] , or the availability of ground-truth map information [ 4 ] . In this work , we aim to learn an agent that can tackle long-horizon GCRL tasks and address the associated challenges in exploration . Our key idea is to use successor features ( SF ) [ 15 , 2 ] — a representation that captures transition dynamics — to define a novel distance metric , Successor Feature Similarity ( SFS ) . First , we exploit the transfer ability of SF to formulate a goal-conditioned value function in terms of SFS between the current state and goal state . By just learning SF via self-supervised representation learning , we can directly obtain a goal-conditioned policy from SFS without any additional policy learning . Second , we leverage SFS to build a landmark-based graph representation of the environment ; the agent adds observed states as landmarks based on their SFSpredicted novelty and forms edges between landmarks by using SFS as a distance estimate . SF as an abstraction of transition dynamics is a natural solution for building this graph when we consider the MDP as a directed graph of states ( nodes ) and transitions ( edges ) following [ 11 ] . We use this graph to systematically explore the environment by planning paths towards landmarks at the “ frontier ” of the explored state space and executing each segment of these planned paths with the goal-conditioned policy . In evaluation , we similarly plan and execute paths towards ( long-horizon ) goals . We call this framework Successor Feature Landmarks ( SFL ) , illustrated in Figure 1 . Our contributions are as follows : ( i ) We use a single self-supervised learning component that captures dynamics information , SF , to build all the components of a graph-based planning framework , SFL . ( ii ) We claim that this construction enables knowledge sharing between each module of the framework and stabilizes the overall learning . ( iii ) We introduce the SFS metric , which serves as a distance estimate and enables the computation of a goal-conditioned Q-value function without further learning . We evaluate SFL against current graph-based methods in long-horizon goal-reaching RL and visual navigation on MiniGrid [ 6 ] , a 2D gridworld , and ViZDoom [ 37 ] , a visual 3D first-person view environment with large mazes . We observe that SFL outperforms state-of-the-art navigation baselines , most notably when goals are furthest away . In a setting where exploration is needed to collect training experience , SFL significantly outperforms the other methods which struggle to scale in ViZDoom ’ s high-dimensional state space . 2 Related Work . Goal-conditioned RL . Prior work has tackled GCRL by proposing variants of goal-conditioned value functions such as UVFAs which estimate cumulative reward for any given state-goal pair [ 23 , 34 , 31 , 26 ] . HER [ 1 ] improved the sample efficiency in training UVFAs by relabeling reached states as goals . Mapping State Space ( MSS ) [ 11 ] then extended UVFAs to long-horizon tasks by using a UVFA as both a goal-conditioned policy and a distance metric to build a graph for high-level planning . MSS also addressed exploration by selecting graph nodes to be at edge of the map ’ s explored region via farthest point sampling . However , this method was only evaluated in low-dimensional state spaces . LEAP [ 25 ] used goal-conditioned value functions to form and execute plans over latent subgoals , but largely ignored the exploration question . Conversely , other works [ 27 , 5 ] have worked on exploration for goal-reaching policies , but do not tackle the long-horizon case . ARC [ 10 ] proposed learning representations that measure state similarity according to the output of a maximum entropy goal-conditioned policy , which can be utilized towards exploration and long-horizon hierarchical RL . However , ARC assumes access to the goal-conditioned policy , which can be difficult to obtain in large-scale environments . Our method can achieve both efficient exploration and long-horizon goal-reaching in high-dimensional state spaces with a SF-based metric that acts as a goal-conditioned value function and distance estimate for graph-building . Graph-based planning . Recent approaches have tackled long-horizon tasks , often in the context of visual navigation , by conducting planning on high-level graph representations and deploying a low-level controller to locally move between nodes ; our framework also falls under this paradigm of graph-based planning . Works such as SPTM [ 29 ] and SoRB [ 9 ] used a deep network as a distance metric for finding shortest paths on the graph , but rely on human demonstrations or sampling from the replay buffer to populate graph nodes . SGM [ 16 ] introduced a two-way consistency check to promote sparsity in the graph , allowing these methods to scale to larger maps . However , these methods rely on assumptions about exploration , allowing the agent to spawn uniformly random across the state space during training . To address this , HTM [ 19 ] used a generative model to hallucinate samples for building the graph in a zero-shot manner , but was not evaluated in 3D visual environments . NTS [ 4 ] achieved exploration and long-horizon navigation as well as generalization to unseen maps by learning a geometric-based graph representation , but required access to a ground-truth map to train their supervised learning model . In contrast , our method achieves exploration by planning and executing paths towards landmarks near novel areas during training . SFL also does not require any ground-truth data ; it only needs to learn SF in a self-supervised manner . Successor features . Our work is inspired by recent efforts in developing successor features ( SF ) [ 15 , 2 ] . They have used SF to decompose the Q-value function into SF and the reward which enables efficient policy transfer [ 2 , 3 ] and to design transition-based intrinsic rewards [ 20 , 39 ] for efficient exploration . SF has also been used in the options framework [ 35 ] ; Eigenoptions [ 21 ] derived options from the eigendecomposition of SF , but did not yet apply them towards reward maximization . Successor Options [ 28 ] used SF to discover landmark states and design a latent reward for learning option policies , but was limited to low-dimensional state spaces . Our framework leverages a SF-based similarity metric to formulate a goal-conditioned policy , abstract the state space as a landmark graph for long-horizon planning and to model state-novelty for driving exploration . While the options policies proposed in these works have to be learned from a reward signal , we can obtain our goal-conditioned policy directly from the SF similarity metric without any policy learning . To our knowledge , this is first work that uses SF for graph-based planning and long-horizon GCRL tasks . 3 Preliminaries . 3.1 Goal-Conditioned RL . Goal-conditioned reinforcement learning ( GCRL ) tasks [ 12 ] are Markov Decision Processes ( MDP ) extended with a set of goals G and defined by a tuple ( S , A , G , RG , T , ) , where S is a state space , A an action set , RG : S ⇥A⇥ G ! R a goal-conditioned reward function , T the transition dynamics , and a discount factor . Following [ 11 , 36 ] , we focus on the setting where the goal space G is a subset of the state space S , and the agent can receive non-trivial rewards only when it can reach the goal ( i.e. , sparse-reward setting ) . We aim to find an optimal goal-conditioned policy ⇡ : S ⇥ G ! A to maximize the expected cumulative reward , V ⇡g ( s0 ) = E⇡ [ P t trt ] ; i.e. , goal-conditioned value function . We are especially interested in long-horizon tasks where goals are distant from the agent ’ s starting state , requiring the policy to operate over longer temporal sequences . Algorithm 1 Training 1 : Initialize : Graph G = ( L , E ) , parameter ✓ of SFS⇡̄✓ , replay buffer D , hyperparameter Texp , landmark transition count N l 2 : while env not done do 3 : lfront ⇠ Softmax ( 1Count ( L ) ) { Choose frontier landmark via count-based sampling } 4 : while lcurr 6= lfront do 5 : ltarget PLAN ( G , lcurr , lfront ) { Plan path to frontier landmark } 6 : ⌧traverse , lcurr Traverse ( ⇡l , ltarget ) { Traverse to ltarget with ⇡l ( Algorithm 3 in §H ) } 7 : G , N l Graph-Update ( G , ⌧traverse , SFS⇡̄✓ , N l ) { Update graph ( Algorithm 2 ) } 8 : end while 9 : ⌧rand = { st , at , rt } Texp t ⇠ ⇡̄ { Explore with random policy for Texp steps } 10 : G , N l Graph-Update ( G , ⌧rand , SFS⇡̄✓ , N l ) { Update graph ( Algorithm 2 ) } 11 : D D [ ⌧random 12 : Update ✓ from TD error with mini-batches sampled from D { Update SF parameters } 13 : end while 3.2 Successor Features . In the tabular setting , the successor representation ( SR ) [ 8 , 15 ] is defined as the expected discounted occupancy of futures state s0 starting in state s and action a and acting under a policy ⇡ : M⇡ ( s , a , s 0 ) = E⇡ hP1 t0=t t0 tI ( St0 = s0 ) St = s , At = a i ( 1 ) The SR M ( s , a ) is then a concatenation of M ( s , a , s0 ) , 8s 2 S. We may view SR as a representation of state similarity extended over the time dimension , as described in [ 21 ] . In addition , we note that the SR is solely determined by ⇡ and the transition dynamics of the environment p ( st+1|st , at ) . Successor features ( SF ) [ 2 , 15 ] extend SR [ 8 ] to high-dimensional , continuous state spaces in which function approximation is often used . SF ’ s formulation modifies the definition of SR by replacing enumeration over all states s0 with feature vector s0 . The SF ⇡ of a state-action pair ( s , a ) is then defined as : ⇡ ( s , a ) = E⇡ hP1 t0=t t0 t st0 St = s , At = a i ( 2 ) In addition , SF can be defined in terms of only the state , ⇡ ( s ) = Ea⇠⇡ ( s ) [ ⇡ ( s , a ) ] . SF allows decoupling the value function into the successor feature ( dynamics-relevant information ) with task ( reward function ) : if we assume that the one-step reward of transition ( s , a , s0 ) with feature ( s , a , s0 ) can be written as r ( s , a , s0 ) = ( s , a , s0 ) > w , where w are learnable weights to fit the reward function , we can write the Q-value function as follows [ 2 , 15 ] : Q⇡ ( s , a ) = E⇡ `` 1X t0=t t 0 tr ( St0 , At0 , St0+1 ) St = s , At = a # = E⇡ `` 1X t0=t t 0 t > t0w St = s , At = a # = ⇡ ( s , a ) > w ( 3 ) Consequently , the Q-value function separates into SF , which represents the policy-dependent transition dynamics , and the reward vector w for a particular task . Later in Section 4.3 , we will extend this formulation to the goal-conditioned setting and discuss our choices for w .

Summary: This paper presents Successor Feature Landmarks (SFL), a new graph-based planning framework based on successor features for long-horizon goal-conditioned RL (GCRL). The key idea is to use successor features to define a new metric, Successor Feature Similarity (SFS), to estimate the distance between two states or state-action pairs in high-dimensional environments, such as 3D visual environments. By leveraging SFS, a landmark-based graph is built to represent the explored state space. Both guided/directed exploration and random exploration are deployed to help the agent traverse between landmarks and explore unseen states of the environment respectively. Combining all the components, SFL is more efficient and scalable than previous graph-based methods for long-horizon GCRL.