Temporally Abstract Partial Models
Authors: Khimya Khetarpal
Zafarali Ahmed
Gheorghe Comanici
Doina Precup
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
model-based reinforcement learning
temporal abstraction
options
partial models
affordances

Abstract: Humans and animals have the ability to reason and make predictions about different courses of action at many time scales. In reinforcement learning, option models (Sutton, Precup \& Singh, 1999; Precup, 2000) provide the framework for this kind of temporally abstract prediction and reasoning. Natural intelligent agents are also able to focus their attention on courses of action that are relevant or feasible in a given situation, sometimes termed affordable actions. In this paper, we define a notion of affordances for options, and develop temporally abstract partial option models, that take into account the fact that an option might be affordable only in certain situations. We analyze the trade-offs between estimation and approximation error in planning and learning when using such models, and identify some interesting special cases. Additionally, we empirically demonstrate the ability to learn both affordances and partial option models online resulting in improved sample efficiency and planning time in the Taxi domain.

Temporally Abstract Partial Models
Khimya Khetarpal +1,2, Zafarali Ahmed 3, Gheorghe Comanici 9, Doina Precup 12,33
McGill University, Milaa, DeepMind
Abstract
Humans and animals have the ability to reason and make predictions about different courses of action at many time scales. In reinforcement learning, option models (Sutton, Precup & Singh, 1999; Precup, 2000) provide the framework for this kind of temporally abstract prediction and reasoning. Natural intelligent agents are also able to focus their attention on courses of action that are relevant feasible in a given situation, sometimes termed affordable actions. In this paper, we define notion of affordances for options, and develop temporally abstract partial option models, that take into account the fact that an option might be affordable only certain situations. We analyze the trade-offs between estimation and approximation error in planning and learning when using such models, and identify some interesting special cases. Additionally, we empirically demonstrate the ability to learn both affordances and partial option models online resulting in improved sample efficiency and planning time in the Taxi domain.
Introduction
Intelligent agents flexibly reason about the applicability and effects of their actions over different time scales, which in turn allows them to consider different courses of action. Yet modeling the entire complexity of realistic environment is quite difficult and requires lot of data (Kakade et al., 2003). Animals and people exhibit a powerful ability to control the modelling process by understanding which actions deserve any consideration at all in situation. By anticipating only certain aspects of their effects over different time horizons may make models more predictable or easier to learn. In this paper develop the theoretical underpinnings of how such an ability could be defined and studied in sequential decision making. We work in the context of model-based reinforcement learning (MBRL) (Sutton and Barto, 2018) and temporal abstraction in the framework of options Sutton et al. (1999). Theories of embodied cognition and perception suggest that humans are able to represent the world knowledge in the form of internal models across different time scales (Pezzulo and Cisek, 2016). Option models provide a framework for RL agents to exhibit the same capability. Options define a way of behaving, including set of states in which an option can start, an internal policy that is used to make decisions while the option is executing, and stochastic, state-dependent termination condition. Models of options predict the (discounted) reward that an option would receive over time and the (discounted) probability distribution over the states attained at termination (Sutton et al., 1999). Consequently, option models enable the extension of dynamic programming and many other RL planning methods in order to achieve temporal abstraction, i.e. to be able to consider seamlessly different time scales of decision-making.
Much of the work on learning and planning with options considers the case where they apply ev- erywhere (Bacon et al., 2017; Harb et al., 2017; Harutyunyan et al., 2019h,a), with some notable recent exceptions which generalize the notion of initiation sets in the context of function approxima- tion (Khetarpal al., 2020b). Having options that are partially defined is very important in order to control the complexity of the planning and exploration process. However, the notion of partially defined option models, which make predictions only from subset of states is the focus of our paper.
"Corresoondnnce to chimya.khetarge @ @e@aill.mmm mail.mcgill.ca
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In natural intelligence, the ability to make predictions across different scales is linked with the ability to understand the action possibilities (i.e. affordances) (Gibson, 1977) which arise at the interface of an agent and an environment and are key component of successful adaptive control (Fikes et al., 1972; Korf, 1983; Drescher, 1991; Cisek and Kalaska, 2010). Recent work (Khetarpal et al., 2020a) has described way to implement affordances in RL agents, by formalizing notion of intent over state space, and then defining an affordance as the set of state-action pairs that achieve that intent to a certain degree. One can then plan with partial, approximate models that map affordances to intents, incurring quantifiable amount of error at the benefit of faster learning and deliberation. In this paper, generalize the notion of intents and affordances to option models. As we will see in Sec. 3, this is non-trivial and requires carefully inspecting the definition of option models. The resulting temporally abstract models are partial, in the sense that they apply only in certain states and options.
Key Contributions. We present framework defining temporally extended intents, affordances and abstract partial option models (Sec. 3). We derive theoretical results quantifying the loss incurred when using such models for planning, exposing trade-offs between single-step models and full option models (Sec. 4). Our theoretical guarantees provide insights and decouple the role of affordances from temporal abstraction. Empirically, we demonstrate end-to-end learning of affordances and partial option models, showcasing significant improvement in final performance and sample efficiency when used for planning in the Taxi domain (Sec. 5).
2 Background
In RL, a docision-making agent interacts with an environment through a sequence of actions. in order to Iearn way of behaving (aka policy) that maximizes its value. i.c. long-term expected return (Sutton and Barto, 2018). This process is typically formalized as a Markov Decision Process (MDP). A finite MDP is a tuple M = (S, r, P. 7), where S is finite set of states, A is a finite set of actions, r : S A + [0, R is the reward function, P- S x A - Dist(S) is the transition dynamics, mapping state-action pairs to a distribution over next states, and Y E [0, 1) is the discount factor. At each time step I, the agent observes a state 8; E S and takes an action a E A drawn from its policy π S - (ist(A) and, with probability P( 18t,, (8t,aa), az), enters the next state 8141 E S while receiving a numerical reward r(st, at). The value function of policy in state is the expectation of the long-term return obtained by executing from s, defined as: (s) = E IE 7'r((, At) S. = s, A, ~ (-))S),, Si+1 ~ P(- (S,. Ar) Vt].
The goal of the agent is to find an optimal policy, x' = arg max, V If the model of the MDP, consisting of and P, is given, the value iteration algorithm can be used to obtain the optimal value function, V*, by computing the fixed-point of the Bellman equations (Bellmann, 1957):
Ys. The optimal policy x²" can be obtained by V*(s)= = max, (f(s.a) + 7 E. P(s'|s,a)V* (s)) acting greedily with respect to V*.
Semi-Markov Decision Process (SMDP). An SMDP (Puterman, 1994) is generalization of MDPs, in which the amount of time between two decision points is random variable. The transition model of the environment is therefore a joint distribution over the next decision state and the time, conditioned the current state and action. SMDPs obey Bellman equations similar those for MDPs
Options. Options (Sutton et al., 1999) provide a framework for temporal abstraction which builds on SMDPs, but also leverages the fact that the agent acts an underlying MDP. A Markovian option is composed of an intra-option policy no. termination condition 3, S + Dist(S), where 3,(s) is the probability of terminating the option upon entering s, and an initiation set to C S. Let Ω be the set of all options. In this document, we will use O C l2 to denote the set of options available to the agent and Oss) = {ols E 1o} denote the set of options available at state s. In call-and-return option execution, when an agent is at decision point, it examines its current state s, chooses O E (s)) according to policy over options Tn((s) then follows the internal policy To until the option terminates according to Bo. Termination yields a new decision point, where this process is repeated.
Option Models. The model of an option predicts its reward and transition dynamics following a state 8 E To. as follows: r(s,0) E[R + 7 RRez + ... + - Re+x = 8, Or = o). and p(s |s, o) = Pr(s s' S.T = T,Tocick = O|So s, An To, Tock-1 ~ (b))
2

---

Passenger
Taxi
Destinations
Unaffordable Options
Affordable Optione
Figure 1: Illustration: Intents and affordances in simple navigation task. Intents include navigation to particular location to pick up or drop off a passenger. Affordances can indicate e.g. if passenger be dropped off (in the the passenger can succeed or fail (in the case when there is no passenger at the given location). Experiments in this domain are included in Sec.
where the passenger already in the taxi) or if option pickup
Ex=1 p''s',, Alss,), where T. is an indicator variable equal to the option terminates upon entering state i, and 0 otherwise. p(s', k|s, o) is the probability that option terminates in s' after exactly k time-steps, given that it started at s. Bellman optimality equations can then be expressed in terms of option models. The optimal state value function and state-option value function, Va and Qù are defined as follows:
(+)
max Qi(s, and oEO(s)
= r(8,0) + Ep(d'ss,)
max Ch(s', o'EO(s")
Partial Models. MBRL methods build reward and transition models from data, which are then used to plan, e.g. by using the Bellman equations. However, learning accurate model can be quite difficult, requiring a lot of data. Moreover, the model does not need to be accurate everywhere, as long it is accurate in relevant places, and/or it provides useful information for identifying good actions. A useful approach to build partial models (Talvitie and Singh, 2009), which only make predictions for specific parts of the observation-aee actionn space. Partial models come in two flavors: predicting only the outcome of a subset of state-action pairs, or making predictions only about certain parts of the observation space. Option models can be interpreted as partial models, of the first type, because they are defined only on states where the option applies.
Affordances. Gibson (1977) coined the term "affordances" to describe the fact that certain states enable certain actions, in the context of embodied agents. For instance, chair "affords" sitting for humans, water "affords" swimming for fish, etc. As result, affordances are function of the environment as well as the agent, and emerge out of their interaction. the context of Object Oriented-MDPs (Diuk et al., 2008), Abel et al. (2014, 2015) define affordances propositional functions on states, which assume the existence of objects and object class descriptions. We build on more general notion of affordances in MDPs (Khetarpal et al., 2020a), defined as a relation between states and actions, where an action is affordable state its desired outcome (i.c. intent) is likely to be achieved.
3 Affordances for Temporal Abstractions
We seek to reduce both the planning complexity when using option models, and the sample complexity of learning such models, by actively eliminating from consideration choices that are unlikely to improve the planning outcome. In particular, we build temporally abstract partial models informed by affordances. Previous work (Khetarpal et al., 2020a) has formalized affordances in RL by considering the desired outcome of primitive action, i.e. the intent associated with the action. We will now generalize this notion intents for options, which can be achieved over the duration of the option. To make this idea concrete, consider the example of taxicab, which needs to pick up passengers from given locations and drop them off desired destination. As discussed in Dietterich (2000), the use of abstraction, in both state space and time, can help solve this problem. In this context, an option could to navigate particular grid location and an intent would be to pick up a passenger, or to drop off the passenger currently in the car at the desired destination. Such intent limits the
3

---

space of possible options under consideration to those that have desired consequences. These intents capture long-term desired consequences of executing options.
Given the generalization of intents to temporal abstraction, the notion of affordance can still be defined similarly the primitive action case in Khetarppl et al. (2020a), by including state-option pairs which achieve the intent to a certain degree. Indeed, primitive affordances will be a special case of option affordances. Some examples of affordances for our illustration are depicted in Fig. An agent can then build partial models of only affordable options enabling it to not only "navigate in the affordance landscape" (Pezzulo and Cisek, 2016), but also better gauge action choices (Cisek and Kalaska, 2010).
3.1 Trajectory Based Option Models
In order to justify the upcoming definitions, we will start with a slight re-writing of the option models in terms of trajectories. A trajectory r(L) is random variable, denoting state-action sequence of length 1 > 1, T() (Sn, ..,...S.. S 1, trajectory of length for which So = 8. Further, let 78, f, 6') ) be a trajectory of length with So and S, =
,,
Overloading notation, let T(s; () denote
s' and r(s,s') s') a trajectory of any length & for which So and S, s'. The return is then 1-111 r( Ax). where |r| is the length of the G(T)) = k==
a deterministic function of a trajectory:
trajectory. The probability of observing a given trajectory (s, ag ..),, & Io. under option o is:
P(r=(90.00 (.))0)
a/ 1 -. (1-86()4+1)) (o
II A-
(x,)))
where the last fraction is there just to capture correctly termination at t. To simplify notation, denote this by Po(T(s, t)). We can define analogously the probability of a trajectory being generated by starting at state E to and ending at a given state s' after & steps by P. s'). The probability a trajectory of any length T(s, s') under o is then: PT(T(s, s')) E Pa(T(s, (rr(s,t,e s')) Let T(s, t, s') denote the set of all trajectories starting at ending s' and of length t and T(s, (
= U,T(s, t, s'). We can write the undiscounted transition model of an option o as:
P(5|ss,)) 0) = Er(o.)))T((.d)) (o(T(s,')) s'))
The discount on trajectory r will be denoted 7(T). If the discount factor is fixed per time step, this will simply be n|T all trajectories of the same length will have the same discount, which will allow us factor it out of products.
The reward model of an option is:
o(5,0,8')= E (n(o))))T Po(T(s, t, (T)ss
The expected discount for option o on a trajectory going from 8 to s is defined as:
To(s, s') = E Ero.LsTTTT))))) (o s')r²"
Note that when the action is primitive action, then 7o(8, 8') E. P( o)r We can re-write the optimal value function of an option as:
('(s,0)=
Σ Σ E'ES 1=1 T(s,tx*))
((((s,, max Q"
Note that the order of the two outer sums can be reversed. This form is equivalent to the one in Sutton al. (1999), but will be more useful for our results.
3.2 Option Affordances
We will now define an intent through a desired probability distribution in the space of all possible trajectories of an option. The goal will be obtain strict generalization of the results established in Khetarpal et al. (2020a) for primitive actions, in the case where each action is an option and 8(s) 1, Ys.
Definition 1 (Temporally Extended Intent ) A temporally extended intent of option E Q. 1 : - Dist(T) specifies for each state s, probability distribution over the space of trajectories

---

T. describing the intended result of executing in s. The associated intent model will be denoted by Pi(T]s, o) = [ T). A temporally extended intent I5¹ satisfied to a degree, Ss,a at state s S and option o E U2 if and only if:
di o), S Sa.0
(I)
where d a metric between probability distributions', and (ss, o) denotes the trajectory starting in state 8 and following the option o.
We note that primitive actions have "degenerate" trajectory, consisting of only the next state. Hence, the only reasonable choice there is to define intent based on the next-state distribution, as done in Khetarpal et al. (2020a). However, options have a whole trajectory, and defining intents on the trajectory distribution provides maximum flexibility. In practice, we expect that most useful intents would be defined in relation with the endpoint of the option, e.g. specifying an intended distribution over the state at the end of the option, or over the joint distribution of the state and duration. Further discussion of special cases is included in the Appendix. Based on this notion of temporally extended intents, affordances for options can be defined as follows:
Definition 2 (Option Affordances AFI-): Given a of options OCΩ and set of temporally extended intents ===VoooTT and 5 E E00,11 1]. we define the affor- dances AFz associated with I as relation AFz CSX O, such that o), IFI-,I,' is satisfied to at (s, o) to degree So.0 S55
Intuitively, we specify temporally extended intents such as "pick up passenger". "drop a passenger at destination", etc. such that the intent is satisfied to a certain degree. Affordances can then be defined as the subset of state-option pairs that can satisfy the intent to a that degree. Fig. 1 depicts a cartoon illustration of intents and corresponding option affordances in the classic Taxi environment.
4 Theoretical Analysis
We now analyze the value loss (Sec. 4) and planning loss (Sec. 4.2) induced by temporally extended intents I and corresponding temporally abstract affordances AF1-
Lemma 1. Given a finite set of option O C and set of temporally extended intents I
-ogolo'
that are satisfied to degrees Sa,or there exist constants (G7 -. ), such that:
max,,o,t,s EM(AANETTONN))
Po(T(s, t, 8')) -
87 (),
S SP
and
(2)
maxs,0 (r(, 0) EETP, [G(T]s o)]
S SR
where 7 == maxx,0 So.0r Th= == || and G(T) is the return on the trajectory T.
The proof is in the Appendix 1.1.1. We note that the error in the approximate probability distribution is bounded by the degree of intent satisfaction for each option i. (.... If intents are far from the true distribution P (i.e. much larger d in Def. 1) or misspecified, then the bounds above are predominantly governed by the approximation error induced due to the intent specification. Moreover, the approximate reward distribution also factor of the error in approximating praphiaabiippbbbbbliiiiii distribution.
4.1 Value Loss Bound
A set of temporally extended intents I define an intent-induced SMDP Mz-, in which the intents can be used to approximate the option transition and reward models. The lemma above establishes this approximation. which in turn allows us to compute the value loss incurred when planning in the intent-induced SMDP.
Theorem (Trajectory- Based Value-Loss Bound). Given a SMDP M corresponding to a finite set of options O and a set of temporally extended intents I = Uo defined on option trajectories (Def. 1), the value loss between the optimal policy for the original SMDP M and the optimal policy
In this work, use d to be the total variation.
5

---

Tz for the induced SMDP MI- is given by:
2 2R E 1/1
1 - -v- S
8 (1-x2-)
+
(1- ( ) 1-)
(4)
where O and C defined in Lemma 1, R max r(8, 0) is the maximum option reward,
y² y²== = Es (s, Exyg(s, s') and yo maxs,o
Es Yols, 8) are the maximum expected discount
factor for the intents and options respectively.
Proof is in Appendix A.2.1. Our result is strict generalization of the results established for primitive actions (Khetarpal et al., 2020a). Note that the value loss bound is better for temporally extended options than for primitives, due to the dependence on the maximum expected option discount (See Table 1). Note that in our bounds, R and Rmoz denote the maximum achievable reward for options and primitive actions respectively. Further interesting corollaries are included in the Appendix.
Value Loss Bound
Actions
Sub Sub-probability Intent 26 Rumes (1-7)-
Trajectory based Intent
Primitive
Temporally Extended
2 1 Y
2R
(1-
Table I: Value Loss Analysis. The maximum value loss incurred when considering intents shows that
while both primitive (I) and temporally extended intents (I")) predominantly depend on the intent approximation error Ç, temporally extended intents can result in gains contingent on the closeness of the intent model and maximum expected discounting of options and intents.
4.2 Planning Loss Bound
In this section, we analyze the effect of incorporating affordances and use temporally extended intents to build partial option models from data on the speed of planning. Similar results have previously been established to spell out the role of the planning horizon (Jiang et al., 2015) and to plan affordance- based partial models of primitive actions (Khetarpal et al., 2020a).
In practical scenarios, the agent may have limited information about the true model of the world. Moreover, might be infeasible and intractable to build a full model, especially real-life applica- tions. To address this, we consider the SMDP MMz induced by models associated with temporally extended intents and the associated affordances, and quantify the loss incurred when planning with this model.
Theorem 2 sinjectory-Based Planning-Loss Bound). Let I- be a set of temporally extended intents for a finite set of options O, and AFI the corresponding approximate SMDP over affordable state-option pairs AFI-. Then, the loss incurred when using M MA compute a policy & "MATI and then using this policy in the original MDP M (also known the certainty-equivalenet planning loss) can be bounded by:
5C (1 (1- (1-v² (1-y0)
2RO maz
- 2n
2 I- 11
5
2 i
S
+V
log
with probability at least 1- 8, where T and STe are defined in Lemma 1, R = max, r(s, o) is the maximum option reward, y² = maxa,o Es 7o(s, s') and y = maxx,o E. ro(8, 80) are the maximum expected discount factor for the intents and options respectively.
The proof is in Appendix A.3.1. The planning loss result generalizes the result for primitive actions provided Khetarpal et al. (2020a). We note a similar effect of incorporating affordances in partial models for temporally extended actions. The accuracy in approximation of the intent (via (CF S7h )) the size of affordable state-option pairs |AFz~ and the SMDP policy class I7 will induce trade-off between approximation of the intents and space of affordances. A key difference

---

Planning Loss Bound
Actions
Without Affordances
Affordance-aware
Primitive
2Bmes x (1-1)²
V 2n log
2Rmaz x (1-7)²
27 T +
2n log
21AFz [
TEA
2n (1-y)²
log
2|8||o lsxo
2n
2
2T log
21AJ
|
Table 2: On the role of affordances in actions and options. We decouple the role of the temporal extent of the options and the effects of incorporating affordances. Our analysis establishes improved guarantees for planning with option models. Further gains obtained when affordances rated, though at the cost of increased approximation error due to intents through Ç. We note that for simplicity, present the bounds obtained when intents are defined the distribution of an option's terminal state, a corollary of Theorem 2. The table highlights the trade-offs between estimation (via the model learning depending the data size n) and approximation (via the specification of intents).
incorpo-
in planning with the approximate partial option models MAFZ- is that the error can be controlled through the maximum expected discount factor for both intent and option models which in turn depends on the minimum expected duration of all affordable options.
Table 2 summarizes the effects of using temporally extended models and affordances. First, note that the planning with affordances introduces a trade-off between approximation and estimation in both primitive and temporally extended actions. Concretely, the approximation error is induced due to the specification of intents through (²,, whereas the estimation error is induced due to learning of the transition and has dependence the data size and the size of the policy class Ilz-..
5 Empirical Analysis
In this section, study the impact of using affordances to learn partial option models which are then used for planning, in order to corroborate the theoretical results established in Sec. 4. In Sec. 5.1, we hand designed set of affordances to show that it can improve training stability well sample efficiency when used to learn a single partial option model, conditioned on a state-option pair. Then, in Sec. 5.2 demonstrate the viability of learning the set of affordances at the same time the partial option model resulting in a set of affordances that were smaller than those that were hand designed. Environment. We consider the 5 5 Taxi domain (Diet- terich, 2000). The domain is a grid world with four des- ignated pickup/drop locations, marked R(ed), B(lue), G(reen), and Yelloow). See Fig. for illustration. The agent controls laxi and faces an episodic problem: the taxi starts in a randomly-chosen square and is given a goal location at which passenger must be dropped. The pas- senger is at one of the three other locations. To complete the task, the agent must drive the taxi to the passenger's location, pick them up. go to the destination, and drop the passenger there. The action space consists of six primitive actions: Up. Down, Left, Right, Pickup, and Drop. The agent gets a reward of -1 per step. +20 for successfully dropping the passenger the goal and - for dropping the passenger at the wrong location. There are a total of 25 (grid positions) x4 (goal destinations) x5 (passen- ger scenarios) = 500 states in this environment and the observation is one-hot vector.
Optinn Pokizies (896))
Intents
Acynchvonauu
(s. si
Learning
Ao)
M.
miols)
Planning
Model transtor Data trerader
Afordances Cytton Enveeremnnt
Krow Krowhegge
Figure 2: Experimental pipeline.
Option set O. We consider fixed set of taxi-centric options, defined as follows: Go to a grid position (25 options); Drop passenger at grid position (25 options); Pickup passenger from grid position (25 options). The options are pre-trained via value iteration and fixed for all our experiments. In total there are 75 x 500 37500 state-option pairs
7

---

Rate:
Succezz Rate
Soccess Rebe
0.5
23
Everything- Everything
Everytheng Everything
Prcup- Deop Everything
Everything Phocpp-
Dreppiieett--- Dventtirs (01i (ha Conerra
Dockg--rrppppooe Everything
Everyliing Pidcpp Drasgiaca
106
40M
staps
Environment steps
steps
(a) Data collection and model learn- ing with affordances.
(b) Planning with affordances.
(c) Data collection, model learning and planning with affordances.
The impact of affordance sets on success rate at different parts of the learning pipeline. (a) The use of affordances improves model learning even in the absence of any affor- dances during planning (blue dotted), (b) The use of affordances did not impact planning because the underlying quality of the model is the same. (c) When using affordances both during model learning and planning (blue dashed), the best performance is obtained. Curves are smoothed over 4 independent seeds using geplot's stat_smooth using a span of 0.1 and confidence interval of 95%.
Figure 3:
to collect transition data (st, o, T, S1TT, E.= rz) where option o was initiated at state s, and ended in state S1+7 after 7 steps, accumulating a reward of r. We execute options until termination or for Trax steps, whichever comes first. We learn linear models to predict the next state distribution Po. (8'18, o). option duration, L (s, o) and reward foz (s, o). where & denote parameter vectors. Affordances can be incorporated in model learning by selecting only affordable options during the data collection and mask the loss of unaffordable state-option transitions:
Experimental pipeline.
We use pre-trained options, o= (I, SS, (o(a)s). (Lo=5,,.(a(s)) (.(s)),
EiSTT MED A(s, o, s', , I- log Po. (s'lo, s) + (o, s) - T)² + (fog ]] (5)
where A(s, o, 8', 1) is if (s, o, s') is affordable according to the intent / and 0 otherwise. We use the learned models, M, in value iteration to obtain a policy over options (o)ss). Affordances can be incorporated into planning by only considering state-option pairs the affordance set (See Algorithm in the Appendix). We report the success rate, i.e., the proportion of episodes in which the agent successfully drops the passenger at the correct location. Data collection, learning, and evaluation happen asynchronously and simultancously (Fig 2) using the Launchpad framework (Yang al., 2021).
Intents and affordances are most useful in model learning when the affordance sets 5.1
more relevant.
In this section investigate the utility of using affordances on different aspects of the pipeline by considering a fixed set of affordances used either during model learning or planning. We first define three intent sets, I+, and their corresponding affordances:
Everything: All options are affordable at every state resulting in 37,500 state-option pairs in this affordance set.
2. Prckup+Drop: We build this set of affordances heuristically, by eliminating all options that simply go to grid position, resulting in 25, ,0000 state-option pairs.
3. Pickup+Drop @Gal: We create this affordance set of 4,000 state-option pairs that termi- nate at the four destination positions only.
When learning the partial model, using the most restrictive and relevant affordance set (PickupDDrop@Ca Goal) to collect data and mask the loss (*_, Everything) significantly improves the sample efficiency (Fig. 3(a)). The difference between Everything and Pickup+Drop was in- significant suggesting that the order of magnitude decrease in the number of statc-option pairs in the affordance set is important (See also Sec 5.2 for more analysis of the affordance set size). Additionally, using any affordance set enables the use of a higher learning rate for learning the model without divergence (Fig. B1). On the other hand, given the same option model, using affordance sets only during planning (Everything->> >*) does not create any improvement in the success rate (Fig. 3(b) or decrease the planning iterations (Fig. 4): the quality of the model dictates the success rate.
We will provide the source code for empirical analysis here.
8

---

Finally, using the most restrictive affordance set for both model learning and planning ToccuptDropGGool-PPee Sicku-ppccup Goal) can result further improvements in the sample effi- ciency (Fig. 3(c)) as well as accelerated planning time (Fig. 4)) demonstrating combined benefit of using affordances in more aspects of the pipeline.
5.2 Relevant affordances can be learned online and result in improved sample efficiency.
In this section, we demonstrate the ability to learn affordances at the same time as learning the partial option model. To do this, we train classifier, As(s, o, s', I) [0, 1] corresponding which predicts if a state-
Planning iterations
1000
to intent option pair is affordable. Pickup+Drop@Gan Goal is defined by 8 intents: four that are com-
everyt tillugt peardiiial
pestragge vonpytingg
300
pleted when the agent has passenger in the vehicle at the destinations; and four that are completed when the agent has dropped the passenger at the destinations. We convert As(s, o, s', I) into an indicator for Eq. 5, by ensuring that at least one of the intents in the intent set is affordable, A(s, o, 5
100
10M
20M
Environment steps
Figure 4: Improvements in planning iterations when using affordances. When using affordances during model learning and in both model learning and planning, we get sustained decrease in plan- ning iterations compared to not using them only using them during planning.
1 [(max)ez-((( (A(s, 0, I)) k] at some threshold value, k. When A: = 0, all state and options affordable. The affordance classifier learned at the same time as the option model, M, using the standard cross entropy objective:
-Eeez- c(s, o, s', I) log A(s, o, s', I) where c(s, o, s', I) is the intent completion function indicating if intent / was completed during the transition.
The threshold, k. controls the size of the affordance set (Fig. 5(a)) with larger k's resulting in smaller affordance sets. The learned affordance set for Pickup+Drop@@ Goal is 2,000 state-option pairs which smaller than what we heuristically defined (4,000 state-option pairs). Smaller affordance sets result in improved sample efficiency (Fig. 5(b)). We highlight that this not necessarily obvious since the learned affordance sets could remove potentially useful state-options pairs and k would be used to control how restrictive the sets are. These results show that affordances can be learned online for defined set of intents and result in good performance. In particular, there sample efficiency gains by using more restricted affordance sets.
Our results here demonstrate empirically that learning partial option model requires much fewer samples as opposed to learning a full model. We also corroborate this with theoretical guarantees sample and computational complexity of obtaining e-estimation of the optimal option value function, given only access to a generative model (See Appendix Sec. C).
Trreshold
Affordancess Set
Success note
0.30
30k
0.35
10N
23M
Environment steps
Environment steps
(a)
(b)
Figure 5: The impact of learning the affordance set for Pickup+Drop@ Goal on (a) size of the affordance set and (b) success in the downstream task. There is one-to-one correspondence between the threshold, k the affordance set size and the success rate on the taxi task. The learned affordance set for Pickup+Drop@ Goal is smaller than the heuristic used in Fig. 3(c).

---

Related Work
Affordances are viewed as the action opportunities (Gibson, 1977; Chemero, 2003), emerging out of the agent-environment interaction (Heft, 1989), and have been typically studied in AI as possibilities associated with an object (Slocum al., 2000; Fitzpatrick al., 2003; Lopes et al., 2007; Montesano et al., 2008; Cruz et al., 2016, 2018; Fulda et al., 2017; Song et al., 2015; Abel et al., 201 4). Affordances have also been formalized in RL without the assumption of objects (Khetarpal et al., 2020a). Our work presents the general case of temporal abstraction (Sutton et al., 1999).
The process model of behavior and cognition (Pezzulo and Cisek, 2016) in the space of affordances is expressed at multiple levels of abstraction. During interactive behavior, action representations at different levels of abstraction can indeed be mapped to findings about the way in which the human brain adaptively selects among predictions of outcomes at different time scales (Cisek and Kalaska, 2010; Pezzulo and Cisck, 2016).
In RL, the generalization of one-step action models to option models (Sutton et al., 1999) enables an agent to predict and reason at multiple time scales. Precup al. (1998) established dynamic programming results for option models which enjoy similar theoretical guarantees as primitive action models. Abel et al. (2019) proposed expected-length models of options. Our theoretical results can also be extended to expected-length option models.
Building agents that can represent and use predictive knowledge requires efficient solutions to cope with the combinatorial explosion of possibilities, especially in large environments. Partial models (Talvitie and Singh, 2009) provide an elegant solution to this problem, as they only model part of the observation. Existing methods focus on predictions for only some of the observations (Oh al., 2017; Amos et al., 2018; Guo et al., 2018; Gregor et al., 2019; Zhao et al., 2021), but they still model the effects of all actions and focus on single-step dynamics (Watters et al., 2019). Recent work by Xu et al. (2020) proposed deep RL approach to learn partial models with goals akin to intents, which is complementary to our work.
7 Conclusions and Limitations
We presented notions of intents and affordances that can be used together with options. They allow us to define lemporally abstract partial models, which extend option models to be conditioned on affordances. Our theoretical analysis suggests that modelling temporally extended dynamics for only relevant parts of the environment-agent interface provides two-fold benefits: 1) faster planning across different timescales (Sec. 4), and 2) improved sampled efficiency (Appendix Sec. C). However, these benefits can come at the cost of some increase in approximation bias, but this tradeoff can still be favourable. For example, in the low-data regime, intermediate-size affordances (much smaller than the entire state-option space) could really improve the speed of planning. Picking intents judiciously can also induce sample complexity gains, the approximation error due to the intent is manageable. Our empirical illustration shows that our approach can produce significant benefits.
I Limitations & Future Work. Our analysis assumes that the intents and options are fixed apriori. To learn intents, we envisage an iterative algorithm which alternates between learning intents and affordances, such that intents can be refined over time and the mis-specifications can also be self- corrected (Talvitie, 2017). Our analysis is complimentary to any method for providing or discovering intents. Another important future direction is build partial option models and leverage their predictions in large scale problems (Vinyals et al., 2019). Besides, it would be useful to relate our work to cognitive science models of intentional options, which can reason about the space of future affordances (Pezzulo and Cisek, 2016). Aligned with future affordances, promising research avenue study the emergence of new affordances the boundary of the agent-environment interaction in the presence of non-stationarity (Chandak et al., 2020).
Acknowledgments and Disclosure of Funding
The authors would like to thank Feryal Behbahani and Dave Abel for very detailed feedback, Martin Klissarov and Emmanuel Bengio for valuable comments on draft of this paper, and Joelle Pineau for feedback on ideas presented in this work. special thank you to Ahmed Touati for discussion and detailed notes (Azar et al., 2012) presented in RL theory reading group at Mila.
10

---

References
Abel, D., Barth-Maron, G., MacGlashan, J., and Tellex S. (2014). Toward affordance-aware planning. In First Workshop on Affordances: Affordances in Vision for Cognitive Robotics.
Abel, D., Hershkowitz, D. E.,, Barth-Maron, G., Brawner, S., O'Farrell, K., MacGlashan, J., and Tellex, S. (2015). Goal-based action priors. In Twenty-Fifth International Conference on Automated Planning and Scheduling.
Abel, D., Winder, J., desJardins, M., and Littman, M. (2019). The expected-length model of options. In International Joint Conference on Artificial Intelligence.
Amos, B., Dinh, L., Cabi, S., Rothörl, T., Colmenarejo, S. G., Muldal, A., Erez, T., Tassa, Y., de Freitas, N., and Denil, M. (2018). Learning awareness models. arXiv preprint arXiv: 1888.006318.
Azar, M. G., Munos, R., and Kappen, B. (2012). On the sample complexity reinforcement learning with generative model. arXiv preprint arXiv:/206.6444 6461.
Bacon, P.-L.,, Harb, J., and Precup, D). (2017). The option-critic architecture. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1726-1734.
Bellmann, R. (1957). Dynamic programming princeton university press. Princeton, NJ.
Chandak, Y., Theocharous, G., Nota, C., and Thomas, P. (2020). Lifelong learning with a changing action set, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3373-3380.
Chemero, A. (2003). An outline of a theory of affordances. Ecological psychology, 15(2):181-195
Cisek, P. and Kalaska, F. (2010). Neural mechanisms for interacting with world full of action choices. Annual review of neuroscience, 33:269-298.
Cruz, F., Magg, S., Weber, C., and Wermter, S. (2016). Training agents with interactive reinforcement learning and contextual affordances. IEEE Transactions on Cognitive and Developmental Systems, 8(4):271-284,
Cruz, F., Parisi, G. I., and Wermter, S. (2018). Multi-modal feedback for affordance-drivee interactive reinforcement learning. 2018 International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE
Dietterich, T. G. (2000). Hierarchical reinforcement learning with the maxq value function decompo- sition. Journal of Artificial Intelligence Research, 13:227-303.
Diuk, C., Cohen, A., and Littman, M. L. (2008). An object-oriented representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pages 240-247. ACM.
Drescher, G. L. (1991). Made-up Minds: A Constructivist Approach to Artificial Intelligence. MIT Press, Cambridge, MA, USA.
Fikes, R. E., Hart, P. E., and Nilsson, J. (1972). Learning and executing generalized robot plans. Artificial Intelligence.
Fitzpatrick, P., Metta, G., Natale, L., Rao, S., and Sandini, G. (2003). Learning about objects through action-initial steps towards artificial cognition. In 2003 IEEE International Conference on Robotics and Automation (Cat. No. 03CH37422), volume 3, pages 3140-3144 IEEE
Fulda, N., Ricks, D., Murdoch, B., and Wingate, D. (2017). What can you do with rock? affordance extraction via word embeddings. arXiv preprint arXiv: (003://77303422 03429.
Gibson, J. J. (1977). The theory of affordances. Hilldale, USA, 1(2).
Gregor, K., Rezende, D. J., Besse, F., Wu, Y., Merzic, H., and van den Oord, A. (2019). Shaping belief states with generative environment models for Advances in Neural Information Processing Systems, pages 13475-13487.

---

Guo, Z. D., Azar, M. G., Piot, B., Pires, B. A., and Munos, R. (2018). Neural predictive belief representations. arXiv preprint arXiv: 1e11.011177
Harb, J. Bacon, P.-L., Klissarov, M., and Precup. D. (2017). When waiting is not an option: Learning options with deliberation cost. arXiv preprint arXiv:/709.0457)
Harutyunyan, A., Dabney, W., Borsa, D., Heess, N., Munos, R., and Precup, D. (2019a). The termination critic. arXiv preprint arXiv:1902 09996.
Harutyunyan, A., Vrancx, P., Hamel, P., Nowé, A., and Precup, D. (2019b), Per-decision option discounting, In International Conference on Machine Learning, pages 2644-26522 PMLR.
Heft, H. (1989). Affordances and the body: An intentional analysis of gibson's ecological approach to visual perception. Journal for the theory of social behaviour, 19(1):1-30.
Jiang, N., Kulesza, A., Singh, S., and Lewis, R. (2015). The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1181-1189, International Foundation for Autonomous Agents and Multiagent Systems.
Kakade, S. M. al. (2003). On the sample complexity of reinforcement learning. PhD thesis, University of London London, England.
Kearns, M. J. and Singh, S. P. (1999). Finite-sample convergence rates for q-learning and indirect algorithms. In Advances in neural information processing systems, pages 996-1002.
Khetarpal, K., Ahmed, Z., Comanici, G., Abel, D., and Precup. D. (2020a). What can i do here? A theory of affordances in reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 5243-5253.
Khetarpal, K., Klissarov, M., Chevalier-Boisvert,, M., Bacon, P.-L., and Precup. D. (2020b). Options of interest: Temporal abstraction with interest functions. Proceedings of the AAAI Conference on Artificial Intelligence, pages 4444-4451.
Korf, R. E. (1983). Learning to Solve Problems by Searching for Macro-operators. PhD thesis, Pittsburgh, PA, USA.
Lawlor, J. (2020). jakelawlor/pnweolors:: pacific northwest inspired r color palette package.
Lopes, M., Melo, S., and Montesano, L. (2007). Affordance-basse imitation learning in robots. In 2007 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 1015-1021. IEEE.
Montesano. L., Lopes, M., Bernardino, A., and Santos- Santos-Victor, J. (2008). Learning object affordances: from sensory-motor coordination to imitation. IEEE Transactions on Robotics, 24(1):15-26.
Oh, J., Singh, S., and Lee, H. (2017). Value prediction network. In Advances in Neural Information Processing Systems, pages 61 18-6128.
Pezzulo, G. and Cisek, P. (2016). Navigating the affordance landscape: feedback control as process model of behavior and cognition. Trends in cognitive sciences, 424
Precup, D., Sutton, R. S., and Singh, S. (1998). Theoretical results on reinforcement learning with temporally abstract options. In European conference machine learning, pages 382-393. Springer.
Puterman, M. (1994). Markov decision processes. 1994. Jhon Wiley & Sons, New Jersey.
Slocum, A. C., Downey, C., and Beer, R. D. (2000). Further experiments in evolution of minimally cognitive behavior: From perceiving affordances to selective attention. In From animals to animats 6: Proceedings of the sixth international conference on simulation of adaptive behavior, pages 430-439.
12

---

Song. H. O., Fritz, M., Goehring, D., and Darrell, T. (2015). Learning to detect visual grasp affordance. IEEE Transactions on Automation Science and Engineering. 13(2):798 809.
Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction. MIT press.
Sutton, R. S., Precup, D., and Singh, S. (1999). Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial intelligence, 12((-2))111-211.
Talvitie, E. (2017). Self-correcting models for model-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence volume 31.
Talvitie, E. and Singh, S. P. (2009). Simple local models for complex dynamical systems. In Advances in Neural Information Processing Systems, pages 1617-1624.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., al. (2019). Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 375(7782):30-334.
Watters, N., Matthey, L.,, Bosnjak, M., Burgess, C. P., and Lerchner, A. (2019). Cobra: Data-efficient model-based rl through unsupervised object discovery and curiosity-drie drivenn exploration. arXiv preprint Xrv: ptt.///055......
Xu, D., Mandlekar, A., Martín-Martin, R., Zhu, Y., Savarese, S., and Fei-Fei, L. (2020). Deep affordance foresight: Planning through what can be done in the future.
Yang, F., Barth-Maron, G., Staúczyk, P., Hoffman, M., Liu, S., Kroiss, M., Pope, A., and Rrustemi,
A. (2021). Launchpad: A programming model for distributed machine learning research. arXiv preprint arXiv:2 10..045666..
Zhao, M., Liu, Z., Luan, S., Zhang, S., Precup, D., and Bengio, (2021). A consciossnes--nspiredd planning agent for model-based reinforcement learning. In Conference Neural Information Processing Systems. https:/arriv..rg/bbb//16....
13


Reviews:
[{"review_id": "wdp2-FOk9b", "reviewer": "Reviewer_VrS9", "summary": "This paper studies an option-based setting where each option is associated with an _intent_, a distribution over desired trajectories following the option execution. Intents induce _affordances_,  which are starting states that are likely to lead to the intents for each option. Intents and affordances induce a partial option model, where the usual transition model over options is replaced with the intents, and the model is partial over the affordances. The main theoretical questions concern the extent to which planning in this partial option model will lead to good performance in the original semi-MDP. Experiments are in a standard taxi domain with given options and intents.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive comments:\n* Very impressive engagement with previous literature throughout.\n* Given the previous work (Khetarpal et al., 2020a), this work is a well-motivated extension and a satisfying generalization of the previous results.\n* I like that the generalization considers intents over full trajectories, rather than just terminal states; the latter would have been fine too, but it is good to remain as general as possible.\n* The theoretical results are of some interest on their own, irrespective of applications.\n* I did not find any technical errors, though I have some confusions (see below).\n* The affordance learning results in Section 5.2 are clear and convincing.\n* The paper is very well written on the sentence level, with a few minor typos noted below.\n\n\nConcerns and criticisms:\n* This question would broadly apply to the previous work (Khetarpal et al., 2020a) just as much, but I am wondering if this line of work is meant to be “descriptive” or “prescriptive”. Is the idea that we want to analyze an intelligent agent that comes prepackaged with possibly suboptimal options and intents, which may be poorly matched with each other (descriptive)? Or is this work suggesting that we should design and learn an agent architecture involving options and intents (prescriptive)? From a prescriptive perspective, I would struggle to see the benefit of introducing intents and affordances into the standard option framework. A descriptive perspective could be interesting, but then I wonder if the proposed models would really be used to model natural phenomena.\n* The paper is difficult to read after the first two sections. This is partly because there is a lot of dense content, but there are also some opportunities to improve clarity without necessarily increasing word count. See also “confusions” below.\n* One thing that would improve clarity would be to have a dedicated discussion reviewing (Khetarpal et al., 2020a). I ended up reading this previous work, which provided a lot of useful context.\n* It would be useful to formally define the intent-induced MDP; right now it is just described informally in L184-185.\n* While space is limited, I would have liked to see more discussion of the implications of, and general intuition behind, the main theoretical results, especially how they are interestingly different from the results obtained in the previous work.\n* In the experiments, my understanding is that intents are always given, but the affordances are either given (section 5.1) or learned (section 5.2). However, the intents are only explicitly discussed in section 5.2. If my understanding is correct, it would be much more clear to define the intents along with the other environment details at the beginning of section 5.\n* My reading of L276-278 is that (model learning + planning) with affordances is claimed to be better than (model learning alone), but Figure 3(c) vs 3(a) does not support that claim (the results look almost identical).\n* The title could be much more specific.\n\n\nConfusions:\n* I am not sure what “sub-probability intent” means in Table 1\n* It would have been helpful to include code in the supplementary materials\n* In the experiments, it is not clear whether the state representation is tabular or something vector-based\n* L273 - 274 mentions option misspecification, I am not sure what that is referring to\n* It is sometimes not clear whether “affordance” or “affordable” is referring to both intents and affordances, or affordances alone\n* “Model learning alone” in section 5.1 is a bit of a confusing name, because I suspect that almost all of the limitations of this baseline are due to the affordances being not used during data collection (as opposed to masking the loss). Something like “data collection” may be more clear.\n\n\nMinor:\n* L46 typo: “a partial, approximate models”\n* L124 typo: “And intent”\n* L125 typo: “drop of the passenger”\n* L127 typo: “consequences of executing option.”\n* The placement of Table 1 is a bit distracting, it’s unclear whether we are supposed to read it immediately after the text above\n* In Equation 1, I don’t think that $\\tau(s)$ was previously defined (I only see $\\tau(t)$ and $\\tau(s, s’)$), but I can understand from context that it represents the trajectory distribution starting at $s$ and executing the option.\n* L199 typo: “corolaries” \n* Table 2 Rmax has typesetting issues in the top row\n* The notation for intention/affordance-induced MDP switches between including and excluding AF in the subscript\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See “concerns” above.\n", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "dA8OFceFDml", "reviewer": "Reviewer_4o6d", "summary": "This paper presents a formalism for semi-Markov decision processes that defines \"intention\" (desirable outcomes of behavior) and \"affordances\" (behaviors that achieve an intention). The main idea is that the agent may have many options, but may have a more limited set of intentions, and affordances can constrain the problem of selecting a good option. Theoretical results are also presented, bounding the sub-optimality caused by operating in intention/affordance/option space, rather than in primitive actions. Finally an illustrative empirical experiment gives a proof of concept for learning affordances, given fixed options and intentions.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "---Originality---\n\nThe paper acknowledges that, conceptually, this framework owes a great deal to recent work. The extension of that existing formalism, which dealt only with primitive actions, to the SMDP case is non-trivial and, as far as I know, novel.\n\n---Quality---\n\nI am not aware of technical flaws in the theoretical work and the experiments seem well-designed.\n\n---Clarity---\n\nI found the paper to be well-written and clearly presented. \n\n---Significance---\n\nI think the problem that this paper addresses has the potential to be very important. One of the things that has held options back from being a complete solution for temporal abstraction is that they are defined in terms of their mechanics (i.e. a policy etc.) and it is very hard to know what option policies the agent might want to have access to. The idea of instead working in terms of things the agent wants to cause in the world seems like it could relieve pressure on option selection -- it makes it okay to have lots of options, even if a lot of them are not very helpful. It seems like it might be easier to select useful intentions than to select useful options as well -- from experience in the world the agent might be able to identify circumstances that lead to reward and that it might want to recreate.\n\nIt also seems to me that this framework is much more important in the SMDP setting than in the MDP setting, where the space of possible behaviors and possible outcomes is already quite limited. \n\nAll that said, what is presented in the paper has practical limitations, especially that the options and intentions are presumed to be given. On its own, this paper is not likely to significantly impact practice in RL. However, it may be a base upon which follow-up work is built.\n\n---Summary of the Review---\n\nThe paper presents a sound and interesting extension to an existing theoretical framework. Though conceptually it builds on prior work, it grapples with important issues not previously addressed. I think this paper would be good to have in the literature.\n\n---After Discussion---\n\nThank you to the authors for the response. After reading the other reviews and discussing with the reviewers, I still feel confident that this paper makes a solid and novel contribution to an important problem. It does also have limitations, as discussed across all of the reviews. In balance, I am comfortable with my current rating.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not discuss negative societal impact, but I think this work is unlikely to have significant societal impact in the foreseeable future.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_2fsXhJCR3", "reviewer": "Reviewer_q4gZ", "summary": "This paper describes options models as partial models, meaning that the options model predict only the outcome of a subset of state-action pairs (both the state transition and the reward). Options enable temporal abstraction by allowing an agent to select an option at a particular decision point and follow it until it satisfies the termination condition. The primary contribution of the paper is the development of theory which quantifies the loss incurred by using temporal abstraction (options) relative to single step decision making. \n\n", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a clearly written paper which presents a variety of theoretical results. The majority of the work for the proofs is in the supplementary material, and the paper claims that it is somewhat distinct from Kheterpal 2020a (see line 195-196 and line 203-204). Irrespective of this point, the new bounds, to the best of my understanding, have not been demonstrated to be non-trivial. The motivation of the paper is that humans use abstraction to make better decisions, the bounds show that we can establish that plans with temporally abstract partial models are at least not unboundedly worse than standard plans. Experiments of sufficient complexity could help to motivate that empirically the methods presented here achieve the stated goal of better planning, but the 5x5 grid-worlds are just too simplistic to be convincing. \n\nComments:\n1. In figure 3 why does the success rate of the everything-everything option set get worse with more environment interactions. The claim is that this is due to option misspecification? What does that precisely mean. Which constants would option misspecification show up in in Table 2? \n\n2. In the experiments is everything-everything really the best baseline we can come up with? Shouldn't we compare to a non-options non-temporally-abstract method? And added bonus would be to compare the results both with unbounded computational budget and bounded. \n\n2. On line 332-333 the paper claims that theoretical analysis suggests that it is possible to achieve faster planning across different time scales. Where is the improvement in planning speed discussed or quantified? I see bounds on planning loss and value loss, but I don't see how either directly implies that planning speed is improved. I don't see any way to directly connect the degree of abstraction/planning speed improvement with degree of intent satisfaction. Moreover, the experiments themselves don't discuss how much time is saved in a planning cycle. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RDyIGbBNMzb", "reviewer": "Reviewer_R8s1", "summary": "This paper addresses the problem of learning and planning with options framework in RL. The authors extend an existing notations of affordness and intent in the context of options and perform a mathematical analysis of the bound and the value loss. The ideas are demonstrated on a taxi benchmark with predefined set of options to illustrates the benfit of the approach. \n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Most or even all of the ideas in the paper were already introduced and analyzed in previous: K. Khetarpal et. al. “What can I do here? A Theory of Affordness in Reinforcement Learning”, the main contribution here is the definition and extension of the framework to trajectories. The same bound and analysis was done in that light and appropriate bounds were derived. A missing piece in all this line of work of planning with value loss is the comparison to regret minimization in the context of options. At least something should be said about the differences between the two approaches. The paper in well structured and written clearly, and all of the bounds are aligned with previous results.\n\nMinor issues\n1. p.4 equation for probability under option o, there is a missing comma at the beginning <sa0,.. →< s,a0... \n\n2. p. 7 “Experimental pipeline”, T is not defined properly. Is it a constant for all option trajectories (hyper-parameter)? an RV?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an option model based on the notions of intent and affordances of options, and provides a theoretical analysis of the framework. While the empirical results are limited, it was felt that the conceptual and theoretical contributions of this paper are exciting, and could be a useful contribution to the field.
There were several points of confusion raised by the reviewers, and these points should be clarified in the final version.
Decision: Accept (Poster)