Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning

1 Introduction . Consider deploying a self-driving car to a new city . To be practical , the car should be able to explore the city such that it can learn to traverse from any starting location to any destination , since the destination may vary depending on the passenger . In the context of reinforcement learning ( RL ) , this problem is known as goal-conditioned RL ( GCRL ) [ 12 , 13 ] . Previous works [ 31 , 1 , 24 , 26 , 17 ] have tackled this problem by learning a goal-conditioned policy ( or value function ) applicable to any reward function or “ goal. ” However , the goal-conditioned policy often fails to scale to long-horizon goals [ 11 ] since the space of state-goal pairs grows intractably large over the horizon of the goal . To address this challenge , the agent needs to ( a ) explore the state-goal space such that it is proficient for any state-goal pair it might observe during test time and ( b ) reduce the effective goal horizon for the policy learning to be tractable . Recent work [ 25 , 11 ] has tackled long-horizon GCRL by leveraging model-based approaches to form plans consisting of lower temporal-resolution subgoals . The policy is then only required to operate for short horizons between these subgoals . One line of work learned a universal value function approximator ( UVFA ) [ 31 ] to make local policy decisions and to estimate distances used for building a landmark-based map , but assumed a low-dimensional state space where the proximity between the state and goal could be computed by the Euclidean distance [ 11 ] . Another line of research focused on visual navigation tasks conducted planning over 1The demo video and code can be found at https : //2016choang.github.io/sfl . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) graph representations of the environment [ 29 , 9 , 16 , 4 ] . However , these studies largely ignored the inherent exploration challenge present for large state spaces , and either assumed the availability of human demonstrations of exploring the state space [ 29 ] , the ability to spawn uniformly over the state space [ 9 , 16 ] , or the availability of ground-truth map information [ 4 ] . In this work , we aim to learn an agent that can tackle long-horizon GCRL tasks and address the associated challenges in exploration . Our key idea is to use successor features ( SF ) [ 15 , 2 ] — a representation that captures transition dynamics — to define a novel distance metric , Successor Feature Similarity ( SFS ) . First , we exploit the transfer ability of SF to formulate a goal-conditioned value function in terms of SFS between the current state and goal state . By just learning SF via self-supervised representation learning , we can directly obtain a goal-conditioned policy from SFS without any additional policy learning . Second , we leverage SFS to build a landmark-based graph representation of the environment ; the agent adds observed states as landmarks based on their SFSpredicted novelty and forms edges between landmarks by using SFS as a distance estimate . SF as an abstraction of transition dynamics is a natural solution for building this graph when we consider the MDP as a directed graph of states ( nodes ) and transitions ( edges ) following [ 11 ] . We use this graph to systematically explore the environment by planning paths towards landmarks at the “ frontier ” of the explored state space and executing each segment of these planned paths with the goal-conditioned policy . In evaluation , we similarly plan and execute paths towards ( long-horizon ) goals . We call this framework Successor Feature Landmarks ( SFL ) , illustrated in Figure 1 . Our contributions are as follows : ( i ) We use a single self-supervised learning component that captures dynamics information , SF , to build all the components of a graph-based planning framework , SFL . ( ii ) We claim that this construction enables knowledge sharing between each module of the framework and stabilizes the overall learning . ( iii ) We introduce the SFS metric , which serves as a distance estimate and enables the computation of a goal-conditioned Q-value function without further learning . We evaluate SFL against current graph-based methods in long-horizon goal-reaching RL and visual navigation on MiniGrid [ 6 ] , a 2D gridworld , and ViZDoom [ 37 ] , a visual 3D first-person view environment with large mazes . We observe that SFL outperforms state-of-the-art navigation baselines , most notably when goals are furthest away . In a setting where exploration is needed to collect training experience , SFL significantly outperforms the other methods which struggle to scale in ViZDoom ’ s high-dimensional state space . 2 Related Work . Goal-conditioned RL . Prior work has tackled GCRL by proposing variants of goal-conditioned value functions such as UVFAs which estimate cumulative reward for any given state-goal pair [ 23 , 34 , 31 , 26 ] . HER [ 1 ] improved the sample efficiency in training UVFAs by relabeling reached states as goals . Mapping State Space ( MSS ) [ 11 ] then extended UVFAs to long-horizon tasks by using a UVFA as both a goal-conditioned policy and a distance metric to build a graph for high-level planning . MSS also addressed exploration by selecting graph nodes to be at edge of the map ’ s explored region via farthest point sampling . However , this method was only evaluated in low-dimensional state spaces . LEAP [ 25 ] used goal-conditioned value functions to form and execute plans over latent subgoals , but largely ignored the exploration question . Conversely , other works [ 27 , 5 ] have worked on exploration for goal-reaching policies , but do not tackle the long-horizon case . ARC [ 10 ] proposed learning representations that measure state similarity according to the output of a maximum entropy goal-conditioned policy , which can be utilized towards exploration and long-horizon hierarchical RL . However , ARC assumes access to the goal-conditioned policy , which can be difficult to obtain in large-scale environments . Our method can achieve both efficient exploration and long-horizon goal-reaching in high-dimensional state spaces with a SF-based metric that acts as a goal-conditioned value function and distance estimate for graph-building . Graph-based planning . Recent approaches have tackled long-horizon tasks , often in the context of visual navigation , by conducting planning on high-level graph representations and deploying a low-level controller to locally move between nodes ; our framework also falls under this paradigm of graph-based planning . Works such as SPTM [ 29 ] and SoRB [ 9 ] used a deep network as a distance metric for finding shortest paths on the graph , but rely on human demonstrations or sampling from the replay buffer to populate graph nodes . SGM [ 16 ] introduced a two-way consistency check to promote sparsity in the graph , allowing these methods to scale to larger maps . However , these methods rely on assumptions about exploration , allowing the agent to spawn uniformly random across the state space during training . To address this , HTM [ 19 ] used a generative model to hallucinate samples for building the graph in a zero-shot manner , but was not evaluated in 3D visual environments . NTS [ 4 ] achieved exploration and long-horizon navigation as well as generalization to unseen maps by learning a geometric-based graph representation , but required access to a ground-truth map to train their supervised learning model . In contrast , our method achieves exploration by planning and executing paths towards landmarks near novel areas during training . SFL also does not require any ground-truth data ; it only needs to learn SF in a self-supervised manner . Successor features . Our work is inspired by recent efforts in developing successor features ( SF ) [ 15 , 2 ] . They have used SF to decompose the Q-value function into SF and the reward which enables efficient policy transfer [ 2 , 3 ] and to design transition-based intrinsic rewards [ 20 , 39 ] for efficient exploration . SF has also been used in the options framework [ 35 ] ; Eigenoptions [ 21 ] derived options from the eigendecomposition of SF , but did not yet apply them towards reward maximization . Successor Options [ 28 ] used SF to discover landmark states and design a latent reward for learning option policies , but was limited to low-dimensional state spaces . Our framework leverages a SF-based similarity metric to formulate a goal-conditioned policy , abstract the state space as a landmark graph for long-horizon planning and to model state-novelty for driving exploration . While the options policies proposed in these works have to be learned from a reward signal , we can obtain our goal-conditioned policy directly from the SF similarity metric without any policy learning . To our knowledge , this is first work that uses SF for graph-based planning and long-horizon GCRL tasks . 3 Preliminaries . 3.1 Goal-Conditioned RL . Goal-conditioned reinforcement learning ( GCRL ) tasks [ 12 ] are Markov Decision Processes ( MDP ) extended with a set of goals G and defined by a tuple ( S , A , G , RG , T , ) , where S is a state space , A an action set , RG : S ⇥A⇥ G ! R a goal-conditioned reward function , T the transition dynamics , and a discount factor . Following [ 11 , 36 ] , we focus on the setting where the goal space G is a subset of the state space S , and the agent can receive non-trivial rewards only when it can reach the goal ( i.e. , sparse-reward setting ) . We aim to find an optimal goal-conditioned policy ⇡ : S ⇥ G ! A to maximize the expected cumulative reward , V ⇡g ( s0 ) = E⇡ [ P t trt ] ; i.e. , goal-conditioned value function . We are especially interested in long-horizon tasks where goals are distant from the agent ’ s starting state , requiring the policy to operate over longer temporal sequences . Algorithm 1 Training 1 : Initialize : Graph G = ( L , E ) , parameter ✓ of SFS⇡̄✓ , replay buffer D , hyperparameter Texp , landmark transition count N l 2 : while env not done do 3 : lfront ⇠ Softmax ( 1Count ( L ) ) { Choose frontier landmark via count-based sampling } 4 : while lcurr 6= lfront do 5 : ltarget PLAN ( G , lcurr , lfront ) { Plan path to frontier landmark } 6 : ⌧traverse , lcurr Traverse ( ⇡l , ltarget ) { Traverse to ltarget with ⇡l ( Algorithm 3 in §H ) } 7 : G , N l Graph-Update ( G , ⌧traverse , SFS⇡̄✓ , N l ) { Update graph ( Algorithm 2 ) } 8 : end while 9 : ⌧rand = { st , at , rt } Texp t ⇠ ⇡̄ { Explore with random policy for Texp steps } 10 : G , N l Graph-Update ( G , ⌧rand , SFS⇡̄✓ , N l ) { Update graph ( Algorithm 2 ) } 11 : D D [ ⌧random 12 : Update ✓ from TD error with mini-batches sampled from D { Update SF parameters } 13 : end while 3.2 Successor Features . In the tabular setting , the successor representation ( SR ) [ 8 , 15 ] is defined as the expected discounted occupancy of futures state s0 starting in state s and action a and acting under a policy ⇡ : M⇡ ( s , a , s 0 ) = E⇡ hP1 t0=t t0 tI ( St0 = s0 ) St = s , At = a i ( 1 ) The SR M ( s , a ) is then a concatenation of M ( s , a , s0 ) , 8s 2 S. We may view SR as a representation of state similarity extended over the time dimension , as described in [ 21 ] . In addition , we note that the SR is solely determined by ⇡ and the transition dynamics of the environment p ( st+1|st , at ) . Successor features ( SF ) [ 2 , 15 ] extend SR [ 8 ] to high-dimensional , continuous state spaces in which function approximation is often used . SF ’ s formulation modifies the definition of SR by replacing enumeration over all states s0 with feature vector s0 . The SF ⇡ of a state-action pair ( s , a ) is then defined as : ⇡ ( s , a ) = E⇡ hP1 t0=t t0 t st0 St = s , At = a i ( 2 ) In addition , SF can be defined in terms of only the state , ⇡ ( s ) = Ea⇠⇡ ( s ) [ ⇡ ( s , a ) ] . SF allows decoupling the value function into the successor feature ( dynamics-relevant information ) with task ( reward function ) : if we assume that the one-step reward of transition ( s , a , s0 ) with feature ( s , a , s0 ) can be written as r ( s , a , s0 ) = ( s , a , s0 ) > w , where w are learnable weights to fit the reward function , we can write the Q-value function as follows [ 2 , 15 ] : Q⇡ ( s , a ) = E⇡ `` 1X t0=t t 0 tr ( St0 , At0 , St0+1 ) St = s , At = a # = E⇡ `` 1X t0=t t 0 t > t0w St = s , At = a # = ⇡ ( s , a ) > w ( 3 ) Consequently , the Q-value function separates into SF , which represents the policy-dependent transition dynamics , and the reward vector w for a particular task . Later in Section 4.3 , we will extend this formulation to the goal-conditioned setting and discuss our choices for w .

Summary: The paper proposes a new metric, called the successor feature similarity, that measures the similarity between two states as the dot-product between the respective successor features. The authors then use this metric to guide goal-directed exploration when learning a goal-conditioned policy by choosing a goal with the lowest count. Novel goals are stored by keeping a buffer of visited goals, where new goals are only added if they are beyond a certain distance of any prior goal. Rewards are given by the SFS between a state and a goal (where a state-only SF is computed as the action-averaged SF). The authors compare this overall method to existing graph-based GCRL methods and find that the resulting method outperforms prior work, particularly in challenging first-person maze-navigation VizDoom tasks that require moving around multiple walls.