Temporally Abstract Partial Models

1 Introduction . Intelligent agents flexibly reason about the applicability and effects of their actions over different time scales , which in turn allows them to consider different courses of action . Yet modeling the entire complexity of a realistic environment is quite difficult and requires a lot of data ( Kakade et al. , 2003 ) . Animals and people exhibit a powerful ability to control the modelling process by understanding which actions deserve any consideration at all in a situation . By anticipating only certain aspects of their effects over different time horizons may make models more predictable or easier to learn . In this paper we develop the theoretical underpinnings of how such an ability could be defined and studied in sequential decision making . We work in the context of model-based reinforcement learning ( MBRL ) ( Sutton and Barto , 2018 ) and temporal abstraction in the framework of options Sutton et al . ( 1999 ) . Theories of embodied cognition and perception suggest that humans are able to represent the world knowledge in the form of internal models across different time scales ( Pezzulo and Cisek , 2016 ) . Option models provide a framework for RL agents to exhibit the same capability . Options define a way of behaving , including a set of states in which an option can start , an internal policy that is used to make decisions while the option is executing , and a stochastic , state-dependent termination condition . Models of options predict the ( discounted ) reward that an option would receive over time and the ( discounted ) probability distribution over the states attained at termination ( Sutton et al. , 1999 ) . Consequently , option models enable the extension of dynamic programming and many other RL planning methods in order to achieve temporal abstraction , i.e . to be able to consider seamlessly different time scales of decision-making . Much of the work on learning and planning with options considers the case where they apply everywhere ( Bacon et al. , 2017 ; Harb et al. , 2017 ; Harutyunyan et al. , 2019b , a ) , with some notable recent exceptions which generalize the notion of initiation sets in the context of function approximation ( Khetarpal et al. , 2020b ) . Having options that are partially defined is very important in order to control the complexity of the planning and exploration process . However , the notion of partially defined option models , which make predictions only from a subset of states is the focus of our paper . ∗Correspondence to khimya.khetarpal @ mail.mcgill.ca 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In natural intelligence , the ability to make predictions across different scales is linked with the ability to understand the action possibilities ( i.e . affordances ) ( Gibson , 1977 ) which arise at the interface of an agent and an environment and are a key component of successful adaptive control ( Fikes et al. , 1972 ; Korf , 1983 ; Drescher , 1991 ; Cisek and Kalaska , 2010 ) . Recent work ( Khetarpal et al. , 2020a ) has described a way to implement affordances in RL agents , by formalizing a notion of intent over state space , and then defining an affordance as the set of state-action pairs that achieve that intent to a certain degree . One can then plan with partial , approximate models that map affordances to intents , incurring a quantifiable amount of error at the benefit of faster learning and deliberation . In this paper , we generalize the notion of intents and affordances to option models . As we will see in Sec . 3 , this is non-trivial and requires carefully inspecting the definition of option models . The resulting temporally abstract models are partial , in the sense that they apply only in certain states and options . Key Contributions . We present a framework defining temporally extended intents , affordances and abstract partial option models ( Sec . 3 ) . We derive theoretical results quantifying the loss incurred when using such models for planning , exposing trade-offs between single-step models and full option models ( Sec . 4 ) . Our theoretical guarantees provide insights and decouple the role of affordances from temporal abstraction . Empirically , we demonstrate end-to-end learning of affordances and partial option models , showcasing significant improvement in final performance and sample efficiency when used for planning in the Taxi domain ( Sec . 5 ) . 2 Background . In RL , a decision-making agent interacts with an environment through a sequence of actions , in order to learn a way of behaving ( aka policy ) that maximizes its value , i.e . long-term expected return ( Sutton and Barto , 2018 ) . This process is typically formalized as a Markov Decision Process ( MDP ) . A finite MDP is a tuple M = 〈S , A , r , P , γ〉 , where S is a finite set of states , A is a finite set of actions , r : S × A → [ 0 , Rmax ] is the reward function , P : S × A → Dist ( S ) is the transition dynamics , mapping state-action pairs to a distribution over next states , and γ ∈ [ 0 , 1 ) is the discount factor . At each time step t , the agent observes a state st ∈ S and takes an action at ∈ A drawn from its policy π : S → Dist ( A ) and , with probability P ( st+1|st , at ) , enters the next state st+1 ∈ S while receiving a numerical reward r ( st , at ) . The value function of policy π in state s is the expectation of the long-term return obtained by executing π from s , defined as : V π ( s ) = E [ ∑∞ t=0 γ tr ( St , At ) ∣∣S0 = s , At ∼ π ( ·|St ) , St+1 ∼ P ( ·|St , At ) ∀t ] . The goal of the agent is to find an optimal policy , π∗ = arg maxπ V π . If the model of the MDP , consisting of r and P , is given , the value iteration algorithm can be used to obtain the optimal value function , V ∗ , by computing the fixed-point of the Bellman equations ( Bellmann , 1957 ) : V ∗ ( s ) = maxa ( r ( s , a ) + γ ∑ s′ P ( s ′|s , a ) V ∗ ( s′ ) ) , ∀s . The optimal policy π∗ can be obtained by acting greedily with respect to V ∗ . Semi-Markov Decision Process ( SMDP ) . An SMDP ( Puterman , 1994 ) is a generalization of MDPs , in which the amount of time between two decision points is a random variable . The transition model of the environment is therefore a joint distribution over the next decision state and the time , conditioned on the current state and action . SMDPs obey Bellman equations similar to those for MDPs . Options . Options ( Sutton et al. , 1999 ) provide a framework for temporal abstraction which builds on SMDPs , but also leverages the fact that the agent acts in an underlying MDP . A Markovian option o is composed of an intra-option policy πo , a termination condition βo : S → Dist ( S ) , where βo ( s ) is the probability of terminating the option upon entering s , and an initiation set Io ⊆ S. Let Ω be the set of all options . In this document , we will use O ⊂ Ω to denote the set of options available to the agent and O ( s ) = { o|s ∈ Io } denote the set of options available at state s. In call-and-return option execution , when an agent is at a decision point , it examines its current state s , chooses o ∈ O ( s ) according to a policy over options πΩ ( s ) , then follows the internal policy πo , until the option terminates according to βo . Termination yields a new decision point , where this process is repeated . Option Models . The model of an option o predicts its reward and transition dynamics following a state s ∈ Io , as follows : r ( s , o ) . = E [ Rt+1 + γRt+2 + · · · + γk−1Rt+k|St = s , Ot = o ] , and p ( s′|s , o ) .= ∑∞ k=1 Pr ( Sk = s ′ , Tk = 1 , T0 < i < k = 0|S0 = s , A0 : k−1 ∼ πo , T0 : k−1 ∼ βo ) = ∑∞ k=1 γ kp ( s′ , k|s , o ) , where Ti is an indicator variable equal to 1 if the option terminates upon entering state i , and 0 otherwise . p ( s′ , k|s , o ) is the probability that option o terminates in s′ after exactly k time-steps , given that it started at s. Bellman optimality equations can then be expressed in terms of option models . The optimal state value function and state-option value function , V ∗Ω and Q∗Ω , are defined as follows : V ∗Ω ( s ) = max o∈O ( s ) Q∗Ω ( s , o ) and Q ∗ Ω ( s , o ) = r ( s , o ) + ∑ s′ p ( s′|s , o ) max o′∈O ( s′ ) Q∗Ω ( s ′ , o′ ) . Partial Models . MBRL methods build reward and transition models from data , which are then used to plan , e.g . by using the Bellman equations . However , learning an accurate model can be quite difficult , requiring a lot of data . Moreover , the model does not need to be accurate everywhere , as long as it is accurate in relevant places , and/or it provides useful information for identifying good actions . A useful approach is to build partial models ( Talvitie and Singh , 2009 ) , which only make predictions for specific parts of the observation-action space . Partial models come in two flavors : predicting only the outcome of a subset of state-action pairs , or making predictions only about certain parts of the observation space . Option models can be interpreted as partial models , of the first type , because they are defined only on states where the option applies . Affordances . Gibson ( 1977 ) coined the term “ affordances ” to describe the fact that certain states enable certain actions , in the context of embodied agents . For instance , a chair “ affords ” sitting for humans , water “ affords ” swimming for fish , etc . As a result , affordances are a function of the environment as well as the agent , and emerge out of their interaction . In the context of Object Oriented-MDPs ( Diuk et al. , 2008 ) , Abel et al . ( 2014 , 2015 ) define affordances as propositional functions on states , which assume the existence of objects and object class descriptions . We build on a more general notion of affordances in MDPs ( Khetarpal et al. , 2020a ) , defined as a relation between states and actions , where an action is affordable in a state if its desired outcome ( i.e . intent ) is likely to be achieved .

Summary: This paper describes options models as partial models, meaning that the options model predict only the outcome of a subset of state-action pairs (both the state transition and the reward). Options enable temporal abstraction by allowing an agent to select an option at a particular decision point and follow it until it satisfies the termination condition. The primary contribution of the paper is the development of theory which quantifies the loss incurred by using temporal abstraction (options) relative to single step decision making.