Possibility Before Utility: Learning And Using Hierarchical Affordances

1 INTRODUCTION . Reinforcement learning ( RL ) methods have recently achieved success in a variety of historically difficult domains ( Mnih et al. , 2015 ; Silver et al. , 2016 ; Vinyals et al. , 2019 ) , but they continue to struggle on complex hierarchical tasks . Human-like intelligent agents are able to succeed in such tasks through an innate understanding of what their environment enables them to do . In other words , they do not waste time attempting the impossible . Gibson ( 1977 ) coins the term “ affordances ” to articulate the observation that humans and other animals largely interpret the world around them in terms of which behaviors the environment affords them . While some previous works apply the concept of affordances to the RL setting , none of these methods easily translate to environments with hierarchical tasks . In this work , we introduce Hierarchical Affordance Learning ( HAL ) , a method that addresses the challenges inherent to learning affordances over high-level subtasks , enabling more efficient learning in environments with complex subtask dependency structures . Many real-world environments have an underlying hierarchical dependency structure ( Fig . 1a ) , and successful completion of tasks in these environments requires understanding how to complete individual subtasks and knowing the relationships between them . Consider the task of preparing a simple pasta dish . Some sets of subtasks , like chopping vegetables or filling a pot with water , can be successfully performed in any order . However , there are many cases in which the dependencies between subtasks must be obeyed . For instance , it is inadvisable chop vegetables after having mixed them with the sauce , or to boil a pot of water before the pot is filled with water in the first place . Equipped with structural inductive biases that naturally allow for temporally extended reasoning over subtasks , hierarchical reinforcement learning ( HRL ) methods are well-suited for tasks with complex high-level dependencies . Existing HRL methods fall along a spectrum ranging from flexible approaches that discover useful subtasks automatically , to the structured approaches that provide some prior information about subtasks and their interdependencies . The former approaches ( e.g . Vezhnevets et al. , 2017 ; Eysenbach et al. , 2018 ) have seen limited success , as the automatic identification of hierarchical abstractions is an open problem in deep learning ( Hinton , 2021 ) . But approaches that endow the agent with more structure , to make complex tasks feasible , do so at the cost of rigid assumptions . Methods that use a form of finite automata ( Fig . 1b ) to express subtask dependencies ( e.g . Icarte et al. , 2020 ) require the set of symbols , or atomic propositions , provided to the agent to be complete , in that the history of symbols maps deterministically to the current context ( i.e . how much progress has been made ; which subtasks are available ) . Importantly , these methods and many others ( e.g . Andreas et al. , 2017 ; Sohn et al. , 2020 ) consider subtasks to be dependent merely on the completion of others . Unfortunately , these assumptions do not hold in the real world ( Fig . 1c ) . For instance , if one completes the subtask cook noodles , but they clumsily spill them all over the floor , are they now ready for the next subtask , mix noodles and sauce ? While the subtask cook noodles is somehow necessary for this further subtask , it is not sufficient to have completed it in the past . The only way for automata-based approaches to handle this complexity is to introduce a new symbol that indicates that the subtask has been undone . This is possible , but extraordinarily restrictive , since , unless the set of symbols is complete , none of the subtask completion information can be used to reliably learn and utilize subtask dependencies . Modeling probabilistic transitions allows the symbolic signal to be incomplete , but still requires a complete set of symbols , in addition to predefined contexts . In order to make use of incomplete symbolic information , our approach instead learns a representation of context grounded in the present state to determine which subtasks are possible ( Fig . 1d ) , rather than solely relying on symbols . The contributions of this paper are as follows . First we introduce milestones ( §4 ) , which serve the dual purpose of subgoals for training options ( Sutton et al. , 1999 ) and as high-level intents ( Kulkarni et al. , 2016 ) for training our affordance model . Milestones are a flexible alternative to atomic propositions used in automata-based approaches , and they are easier to specify due to less rigid assumptions . Unlike a dense reward function , the milestone signal does not need to be scaled or balanced carefully to account for competing extrinsic motives . Next , we introduce hierarchical affordances , which can be defined over any arbitrary set of milestones , and describe HAL ( §6 ) , a method which learns and utilizes a model of hierarchical affordances to prune impossible subtasks . Finally , we demonstrate HAL ’ s superior performance on two complex hierarchical tasks in terms of learning speed , robustness , generalizability , and ability to explore complex subtask hierarchies without extrinsic supervision , relative to baselines provided with the same information ( §7.3 ) . 2 RELATED WORK . Multi-task RL methods take advantage of shared task structure in order to generalize to new tasks from the same distribution ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ; Devin et al. , 2019 ; Sohn et al. , 2020 ; Lu et al. , 2021 ) . Sohn et al . ( 2020 ) learn subtask preconditions , but use symbol-based contexts and do not learn and use their model of preconditions concurrently . Instead they assume a naive policy can sufficiently reach all subtasks . Furthermore , they assume ground-truth affordances are provided at each step . Some works provide the agent with high-level task sketches ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ) describing the order in which subtasks must be completed . While these sketches are advertised as “ ungrounded ” Andreas et al . ( 2017 ) , they are in fact grounded by the inclusion of short sketches , which are the first to be introduced to the agent in a curriculum learning scheme ( Bengio et al. , 2009 ) . Our approach instead uses a direct signal , which alone need not determine task progress , and can learn without exposure to other tasks with shared structure . In using a set of discrete symbols to indicate subtask completion , our work is similar to the variety of approaches that apply temporal logic ( TL ) to the RL setting ( Yuan et al. , 2019 ; Hasanbeig et al. , 2018 ; 2020 ; Li et al. , 2017 ; 2018 ) . These works typically provide the agent with a TL formula , as well as assignments of atomic propositions at each time-step . Some works use reward shaping to encourage satisfaction of the formula ( Li et al. , 2017 ; 2018 ) , whereas others convert the TL formula to some finite state machine , which provides the agent with a structure that roughly expresses subtask dependencies ( Hasanbeig et al. , 2018 ; 2020 ; Yuan et al. , 2019 ) . Icarte et al . ( 2020 ) bypass this formula-to-automata conversion , and instead directly provide the automata to the agent in the form of a reward machine ( RM ) . While RMs are more expressive than LTL formulas , they are less flexible than HAL , which can deal with incomplete sets of symbols , as well as context stochasticity . Gibson ( 1977 ) introduces a theory of affordances , defined roughly as properties of the environment which must be measured relative to the agent . Heft ( 1989 ) and Chemero ( 2003 ) clarify affordances as relations between the agent and its environment . Khetarpal et al . ( 2020 ) formalize this relational definition of affordances in the context of RL , and model which low-level actions , given corresponding intents , are afforded in each state . In this work , milestones represent high-level intents corresponding to each subtask . They also demonstrate that modeling affordances speeds up and improves planning through the pruning of irrelevant actions , and allows for the learning of more accurate and generalizable partial world models . This approach does not directly translate to the hierarchical setting because subtasks , unlike actions , may fail for reasons other than affordances , meaning we do not have access to ground-truth affordance labels with which to train our model . Manoury et al . ( 2019 ) and Khazatsky et al . ( 2021 ) present approaches that can discover and use affordances to learn new skills , but their definition of affordances ( i.e . a behavior is either afforded or not , with no notion of preconditions ) does not translate to the hierarchical setting . 3 PRELIMINARIES . Our setting involves learning behavior policies in Markov Decision Processes ( MDP ) using RL . An MDP is defined by the state space S , action space A , reward function R : S × A → R , and state transition distribution P : S×A → 4 ( S ) . The objective is to learn a policy , π : S → 4 ( A ) , which selects actions that maximize expected future returns : G ( π ) = E [ ∑∞ t=0 γ trt | at ∼ π , st ∼ P ] , where γ is the discount factor . Sutton et al . ( 1999 ) introduce the options framework , which flexibly models hierarchical abstractions with minimal modification to the RL paradigm . Each option , o : = 〈Io , πo , βo〉 , is defined by an initiation set , Io ⊆ S , indicating where the option can be selected , the corresponding option policy , πo , and the termination condition , βo : S+ → [ 0 , 1 ] , indicating the probability of termination in each state . Options turn our typical MDP into a semi-Markov decision process ( SMDP ) since the state transition distribution is , in general , no longer dependent on the current state and action , but also the present option , which was decided in a previous time-step . The design of this framework allows options to be treated similarly to actions , except they may be executed across multiple time-steps , interrupted , composed , and learned as separate subpolicies . 4 MILESTONES AND HIERARCHICAL AFFORDANCES . We consider tasks that can be decomposed into subtasks , each represented by a milestone symbol , g ∈ G , where G is the set of symbols relevant to the task , and |G| = K. For each subtask , we introduce a separate option , 〈Ig , πg , βg〉 , and we call πg a subpolicy . At each time-step , in addition to the extrinsic reward signal provided by the environment to indicate success on the overall task , we have access to a milestone signal , which is a vector bt where each element btg ∈ { 0 , 1 } indicates whether g ∈ G was completed on time-step t. In our PASTA example , we might receive a milestone each time we cut a vegetable , make the sauce , cook the noodles , etc . Milestones serve two main purposes . Firstly , milestones function as option subgoals ( Sutton et al. , 1999 ) that are in this work used to train each subpolicy ( discussed in Section 5 ) . Secondly , each milestone represents the intent of its corresponding subtask—similar to the action intents introduced to learn action-level affordances in the work of Khetarpal et al . ( 2020 ) —which we use to learn hierarchical affordances ( discussed in Section 6 ) . In contrast to the standard options framework , primitive actions can only be executed as part of an option ’ s subpolicy in our method . Generally , policies trained solely over options have no guarantee of optimality ( Sutton et al. , 1999 ) , but we ensure the existence of an optimal solution by requiring gK ∈ G , where gK is the task ’ s final milestone ( indicating task success ) . When G = { gk } , our setting is standard , flat RL . Each additional milestone g′ added to G is useful as an intermediate signal so long as g′ corresponds to a unique behavior necessary for achieving gK . Hierarchical affordances are defined over G in the following way . The vector fs = f∗ ( s ) of size K represents which milestones are immediately achievable from the present state s , without requiring the collection of any intermediate milestones , where f is a hierarchical affordance classifier , and f∗ is the optimal one1 . Formally , fsg = 1 if at time t0 it is possible for future b T g = 1 without any b t j = 1 , j 6= g , where t0 < t < T . In PASTA , the milestone mix cooked noodles and sauce is not afforded at the beginning since cook noodles is required first . A successful policy trained within the vanilla options framework will eventually learn to execute options in contexts where they are most useful , regardless of each option ’ s predefined initiation set . However , hierarchical affordances give us a principled way to directly adjust this set : for subtask g , we can set Ig = { s | s ∈ S , fsg = 1 } . One can think of hierarchical affordances as using milestones to impose a state-grounded subtask dependency structure on top of the options framework , which we can use to prune impossible subtasks . If G = { g′ , gK } , an affordance-aware agent with access to optimal f∗ ( s ) will never initiate subtask gK from the beginning if g′ is a necessary intermediate behavior . Some logic-based RL approaches ( e.g . Yuan et al. , 2019 ; Icarte et al. , 2020 ) use atomic propositions as markers of subtask achievement to transition between contexts in a finite state machine . These approaches , and many other HRL works ( e.g . Andreas et al. , 2017 ; Sohn et al. , 2020 ) define subtask preconditions in terms of other subtasks . There are two forms of stochasticity that hierarchical affordances , by virtue of being grounded in the present state , can more naturally address than symbolically-defined dependencies . We can conceptualize potential agent trajectories as graphs where nodes represent the attainment of milestones , and edges are the segments between them . Node stochasticity is affordance-affecting randomness that occurs either when milestones are attained ( e.g . receiving varying quantity of an item ) , or at the beginning of the episode ( i.e . starting in different contexts ) . Edge stochasticity is when affordances change at any time within a segment . We treat edge stochasticity events as infrequent exceptions to the typical subtask dependency rules . For example , after cook noodles is complete , mix cooked noodles and sauce is afforded , even if the agent may eventually spill the noodles on the floor . By grounding these rules in the current state , an affordance-aware agent can detect and adapt to edge anomalies . In Section 6 , we describe in detail how hierarchical affordances are learned and used in stochastic environments where symbols alone would fail to reliably determine the current context .

Summary: Given a task hierarchy specified as a dependency graph of milestones, this work employs a two-level hierarchical reinforcement-learning method similar to Kulkarni et al. 2016.  The lower level learns to achieve a given milestone in a classical fashion, while the upper level learns to choose milestones such that ultimately the agent will achieve the overall goal.  The core contribution is a method for pruning candidate milestones (upper-level actions) that are not currently achievable (afforded to the agent by the environment), using an affordance classifier trained on the fly.