Possibility Before Utility: Learning And Using Hierarchical Affordances

1 INTRODUCTION . Reinforcement learning ( RL ) methods have recently achieved success in a variety of historically difficult domains ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Vinyals et al. , 2019 ) , but they continue to struggle on complex hierarchical tasks . Human-like intelligent agents are able to succeed in such tasks through an innate understanding of what their environment enables them to do . In other words , they do not waste time attempting the impossible . Gibson ( 1977 ) coins the term “ affordances ” to articulate the observation that humans and other animals largely interpret the world around them in terms of which behaviors the environment affords them . While some previous works apply the concept of affordances to the RL setting , none of these methods easily translate to environments with hierarchical tasks . In this work , we introduce Hierarchical Affordance Learning ( HAL ) , a method that addresses the challenges inherent to learning affordances over high-level subtasks , enabling more efficient learning in environments with complex subtask dependency structures . Many real-world environments have an underlying hierarchical dependency structure ( Fig . 1a ) , and successful completion of tasks in these environments requires understanding how to complete individual subtasks and knowing the relationships between them . Consider the task of preparing a simple pasta dish . Some sets of subtasks , like chopping vegetables or filling a pot with water , can be successfully performed in any order . However , there are many cases in which the dependencies between subtasks must be obeyed . For instance , it is inadvisable chop vegetables after having mixed them with the sauce , or to boil a pot of water before the pot is filled with water in the first place . Equipped with structural inductive biases that naturally allow for temporally extended reasoning over subtasks , hierarchical reinforcement learning ( HRL ) methods are well-suited for tasks with complex high-level dependencies . Existing HRL methods fall along a spectrum ranging from flexible approaches that discover useful subtasks automatically , to the structured approaches that provide some prior information about subtasks and their interdependencies . The former approaches ( e.g . Vezhnevets et al. , 2017 ; Eysenbach et al. , 2018 ) have seen limited success , as the automatic identification of hierarchical abstractions is an open problem in deep learning ( Hinton , 2021 ) . But approaches that endow the agent with more structure , to make complex tasks feasible , do so at the cost of rigid assumptions . Methods that use a form of finite automata ( Fig . 1b ) to express subtask dependencies ( e.g . Icarte et al. , 2020 ) require the set of symbols , or atomic propositions , provided to the agent to be complete , in that the history of symbols maps deterministically to the current context ( i.e . how much progress has been made ; which subtasks are available ) . Importantly , these methods and many others ( e.g . Andreas et al. , 2017 ; Sohn et al. , 2020 ) consider subtasks to be dependent merely on the completion of others . Unfortunately , these assumptions do not hold in the real world ( Fig . 1c ) . For instance , if one completes the subtask cook noodles , but they clumsily spill them all over the floor , are they now ready for the next subtask , mix noodles and sauce ? While the subtask cook noodles is somehow necessary for this further subtask , it is not sufficient to have completed it in the past . The only way for automata-based approaches to handle this complexity is to introduce a new symbol that indicates that the subtask has been undone . This is possible , but extraordinarily restrictive , since , unless the set of symbols is complete , none of the subtask completion information can be used to reliably learn and utilize subtask dependencies . Modeling probabilistic transitions allows the symbolic signal to be incomplete , but still requires a complete set of symbols , in addition to predefined contexts . In order to make use of incomplete symbolic information , our approach instead learns a representation of context grounded in the present state to determine which subtasks are possible ( Fig . 1d ) , rather than solely relying on symbols . The contributions of this paper are as follows . First we introduce milestones ( §4 ) , which serve the dual purpose of subgoals for training options ( Sutton et al. , 1999 ) and as high-level intents ( Kulkarni et al. , 2016 ) for training our affordance model . Milestones are a flexible alternative to atomic propositions used in automata-based approaches , and they are easier to specify due to less rigid assumptions . Unlike a dense reward function , the milestone signal does not need to be scaled or balanced carefully to account for competing extrinsic motives . Next , we introduce hierarchical affordances , which can be defined over any arbitrary set of milestones , and describe HAL ( §6 ) , a method which learns and utilizes a model of hierarchical affordances to prune impossible subtasks . Finally , we demonstrate HAL ’ s superior performance on two complex hierarchical tasks in terms of learning speed , robustness , generalizability , and ability to explore complex subtask hierarchies without extrinsic supervision , relative to baselines provided with the same information ( §7.3 ) . 2 RELATED WORK . Multi-task RL methods take advantage of shared task structure in order to generalize to new tasks from the same distribution ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ; Devin et al. , 2019 ; Sohn et al. , 2020 ; Lu et al. , 2021 ) . Sohn et al . ( 2020 ) learn subtask preconditions , but use symbol-based contexts and do not learn and use their model of preconditions concurrently . Instead they assume a naive policy can sufficiently reach all subtasks . Furthermore , they assume ground-truth affordances are provided at each step . Some works provide the agent with high-level task sketches ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ) describing the order in which subtasks must be completed . While these sketches are advertised as “ ungrounded ” Andreas et al . ( 2017 ) , they are in fact grounded by the inclusion of short sketches , which are the first to be introduced to the agent in a curriculum learning scheme ( Bengio et al. , 2009 ) . Our approach instead uses a direct signal , which alone need not determine task progress , and can learn without exposure to other tasks with shared structure . In using a set of discrete symbols to indicate subtask completion , our work is similar to the variety of approaches that apply temporal logic ( TL ) to the RL setting ( Yuan et al. , 2019 ; Hasanbeig et al. , 2018 ; 2020 ; Li et al. , 2017 ; 2018 ) . These works typically provide the agent with a TL formula , as well as assignments of atomic propositions at each time-step . Some works use reward shaping to encourage satisfaction of the formula ( Li et al. , 2017 ; 2018 ) , whereas others convert the TL formula to some finite state machine , which provides the agent with a structure that roughly expresses subtask dependencies ( Hasanbeig et al. , 2018 ; 2020 ; Yuan et al. , 2019 ) . Icarte et al . ( 2020 ) bypass this formula-to-automata conversion , and instead directly provide the automata to the agent in the form of a reward machine ( RM ) . While RMs are more expressive than LTL formulas , they are less flexible than HAL , which can deal with incomplete sets of symbols , as well as context stochasticity . Gibson ( 1977 ) introduces a theory of affordances , defined roughly as properties of the environment which must be measured relative to the agent . Heft ( 1989 ) and Chemero ( 2003 ) clarify affordances as relations between the agent and its environment . Khetarpal et al . ( 2020 ) formalize this relational definition of affordances in the context of RL , and model which low-level actions , given corresponding intents , are afforded in each state . In this work , milestones represent high-level intents corresponding to each subtask . They also demonstrate that modeling affordances speeds up and improves planning through the pruning of irrelevant actions , and allows for the learning of more accurate and generalizable partial world models . This approach does not directly translate to the hierarchical setting because subtasks , unlike actions , may fail for reasons other than affordances , meaning we do not have access to ground-truth affordance labels with which to train our model . Manoury et al . ( 2019 ) and Khazatsky et al . ( 2021 ) present approaches that can discover and use affordances to learn new skills , but their definition of affordances ( i.e . a behavior is either afforded or not , with no notion of preconditions ) does not translate to the hierarchical setting . 3 PRELIMINARIES . Our setting involves learning behavior policies in Markov Decision Processes ( MDP ) using RL . An MDP is defined by the state space S , action space A , reward function R : S × A → R , and state transition distribution P : S×A → 4 ( S ) . The objective is to learn a policy , π : S → 4 ( A ) , which selects actions that maximize expected future returns : G ( π ) = E [ ∑∞ t=0 γ trt | at ∼ π , st ∼ P ] , where γ is the discount factor . Sutton et al . ( 1999 ) introduce the options framework , which flexibly models hierarchical abstractions with minimal modification to the RL paradigm . Each option , o : = 〈Io , πo , βo〉 , is defined by an initiation set , Io ⊆ S , indicating where the option can be selected , the corresponding option policy , πo , and the termination condition , βo : S+ → [ 0 , 1 ] , indicating the probability of termination in each state . Options turn our typical MDP into a semi-Markov decision process ( SMDP ) since the state transition distribution is , in general , no longer dependent on the current state and action , but also the present option , which was decided in a previous time-step . The design of this framework allows options to be treated similarly to actions , except they may be executed across multiple time-steps , interrupted , composed , and learned as separate subpolicies . 4 MILESTONES AND HIERARCHICAL AFFORDANCES . We consider tasks that can be decomposed into subtasks , each represented by a milestone symbol , g ∈ G , where G is the set of symbols relevant to the task , and |G| = K. For each subtask , we introduce a separate option , 〈Ig , πg , βg〉 , and we call πg a subpolicy . At each time-step , in addition to the extrinsic reward signal provided by the environment to indicate success on the overall task , we have access to a milestone signal , which is a vector bt where each element btg ∈ { 0 , 1 } indicates whether g ∈ G was completed on time-step t. In our PASTA example , we might receive a milestone each time we cut a vegetable , make the sauce , cook the noodles , etc . Milestones serve two main purposes . Firstly , milestones function as option subgoals ( Sutton et al. , 1999 ) that are in this work used to train each subpolicy ( discussed in Section 5 ) . Secondly , each milestone represents the intent of its corresponding subtask—similar to the action intents introduced to learn action-level affordances in the work of Khetarpal et al . ( 2020 ) —which we use to learn hierarchical affordances ( discussed in Section 6 ) . In contrast to the standard options framework , primitive actions can only be executed as part of an option ’ s subpolicy in our method . Generally , policies trained solely over options have no guarantee of optimality ( Sutton et al. , 1999 ) , but we ensure the existence of an optimal solution by requiring gK ∈ G , where gK is the task ’ s final milestone ( indicating task success ) . When G = { gk } , our setting is standard , flat RL . Each additional milestone g′ added to G is useful as an intermediate signal so long as g′ corresponds to a unique behavior necessary for achieving gK . Hierarchical affordances are defined over G in the following way . The vector fs = f∗ ( s ) of size K represents which milestones are immediately achievable from the present state s , without requiring the collection of any intermediate milestones , where f is a hierarchical affordance classifier , and f∗ is the optimal one1 . Formally , fsg = 1 if at time t0 it is possible for future b T g = 1 without any b t j = 1 , j 6= g , where t0 < t < T . In PASTA , the milestone mix cooked noodles and sauce is not afforded at the beginning since cook noodles is required first . A successful policy trained within the vanilla options framework will eventually learn to execute options in contexts where they are most useful , regardless of each option ’ s predefined initiation set . However , hierarchical affordances give us a principled way to directly adjust this set : for subtask g , we can set Ig = { s | s ∈ S , fsg = 1 } . One can think of hierarchical affordances as using milestones to impose a state-grounded subtask dependency structure on top of the options framework , which we can use to prune impossible subtasks . If G = { g′ , gK } , an affordance-aware agent with access to optimal f∗ ( s ) will never initiate subtask gK from the beginning if g′ is a necessary intermediate behavior . Some logic-based RL approaches ( e.g . Yuan et al. , 2019 ; Icarte et al. , 2020 ) use atomic propositions as markers of subtask achievement to transition between contexts in a finite state machine . These approaches , and many other HRL works ( e.g . Andreas et al. , 2017 ; Sohn et al. , 2020 ) define subtask preconditions in terms of other subtasks . There are two forms of stochasticity that hierarchical affordances , by virtue of being grounded in the present state , can more naturally address than symbolically-defined dependencies . We can conceptualize potential agent trajectories as graphs where nodes represent the attainment of milestones , and edges are the segments between them . Node stochasticity is affordance-affecting randomness that occurs either when milestones are attained ( e.g . receiving varying quantity of an item ) , or at the beginning of the episode ( i.e . starting in different contexts ) . Edge stochasticity is when affordances change at any time within a segment . We treat edge stochasticity events as infrequent exceptions to the typical subtask dependency rules . For example , after cook noodles is complete , mix cooked noodles and sauce is afforded , even if the agent may eventually spill the noodles on the floor . By grounding these rules in the current state , an affordance-aware agent can detect and adapt to edge anomalies . In Section 6 , we describe in detail how hierarchical affordances are learned and used in stochastic environments where symbols alone would fail to reliably determine the current context .

Summary: This paper is in the realm of hierarchical reinforcement learning, and the aim is to employ a particular kind of affordance model: milestones of subtasks which can be achieved and a coupling between states and "affordable" subgoal completion (from that state). Basically, this makes it possible to limit action choices (or more general: bias choices) to actions/subpolicies for which it is known from experience that they are in reach from this state with this action/subpolicy. The paper introduces the HAL method, in which an indicator function for the milestones, a state generalization function (mapping similar affordance abilities to similar abstract states), and low level and hight level Q-functions are all learned in parallel. Using two domains, the method is evaluated and put against competetive baselines and it is shown that there are benefits in general, but also for specific aspects such as inexact milestone knowledge or stochasticity.