GrASP: Gradient-Based Affordance Selection for Planning
Authors: Vivek Veeriah
Zeyu Zheng
Richard Lewis
Satinder Singh
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
affordances

Abstract: Planning with a learned model is arguably a key component of intelligence. There are several challenges in realizing such a component in large-scale reinforcement learning (RL) problems. One such challenge is dealing effectively with continuous action spaces when using tree-search planning (e.g., it is not feasible to consider every action even at just the root node of the tree). In this paper we present a method for \emph{selecting} affordances useful for planning---for learning which small number of actions/options from a continuous space of actions/options to consider in the tree-expansion process during planning. We consider affordances that are goal-and-state-conditional mappings to actions/options as well as unconditional affordances that simply select actions/options available in all states. Our selection method is gradient based: we compute gradients through the planning procedure to update the parameters of the function that represents affordances. Our empirical work shows that it is feasible to learn to select both primitive-action and option  affordances, and that simultaneously learning to select affordances and planning with a learned value-equivalent model can outperform model-free RL. 

Under review as a conference paper at ICLR 2022
GRASP: GRADIENT-BASBR AFFORDANCE CE SELECTION FOR PLANNING
Anonymous authors Paper under double-blind review
ABSTRACT
Planning with learned model is arguably key component of intelligence. There are several challenges in realizing such a component in large-scale reinforcement learning (RL) problems. One such challenge is dealing effectively with continuous action spaces when using tree-search planning (e.g., it is not feasible to consider every action even at just the root node of the tree). In this paper we present a method for selecting affordances useful for planning for learning which small number of actions/options from a continuous space of actions/options to consider in the tree-expansion process during planning. We consider affordances that are mappings to actions/options as well as unconditional affordances that simply select actions/options available in all states. Our selection method is gradient based: we compute gradients through the planning procedure to update the parameters of the function that represents affordances. Our empirical work shows that it is feasible to learn to select both primitive- -aativn and option affordances, and that simultaneously learning select affordances and planning with a learned value-equivalent model can outperform model-free RL.
saaal-nddsstae-coddiiioo
The ability to plan using a model learned from experience is arguably a key component of intelligence. Repeated planning can help both in the better selection of the action execute in the current state as well as in providing better targets for updating policies and value functions. A landmark in the success of the use of planning model-based Deep RL (RL that uses neural-networfe as function approximators) was AlphaGo (Silver et al.] 2017) in which a simulator of the game of Go was used by the planner as model. More recently MuZero (Schrittwieser al.) 2020) has shown the capability of learning models from experience simultaneously with planning in solving an impressive number of large-scale RL problems (albeit on problems where a simulator is available and thus sample complexity is not big concern). Nevertheless, there remain interesting challenges in making planning with learned models become a routine part of Deep RL agents across the spectrum of RL domains. One such challenge, the subject of interest in this paper, is dealing effectively with continuous actions spaces where using tree-search based planning is not straightforward (e.g., just considering every action at the root node of the tree becomes infeasible). Accordingly, we consider RL domains with continuous action or option spaces (where options are temporally extended actions).
Our approach is select small number of affordances-to discover what actions/options are useful to consider in tree-based planning. The notion of affordances is a rich one in psychology that refers to many kinds of rgganism-enviromen from the idea of directly perceived high-level actions that environment states afford, such as object interactions (e.g., chairs afford sitting, cups afford grasping) or navigation actions (doorways afford passage), all restricted subsets of the vast space of possible actions the organism may perform. We implement affordances as parametric mappings from states (and goals in multi-takk settings) to actions/options. So selecting K affordances means learning K such mappings, and then there are at most K choices at each node the planning tree. What action/option the 2th affordance corresponds to depends the state corresponding to node. We also consider simply selecting K actions/options that are available in all states to evaluate how much conditioning on state matters.
relations (Gibsonn 1977). We take inspiration our work
Our main contributions are twofold: (1) the idea of selecting affordances to deal with continuous actions/options in planning, and (2) GrASP an algorithm for Gradient-bart based A Affordance Selection for Plamning. GrASP computes gradients through the planning procedure to update the parameters of the functions that represent affordances. We do not claim or show that GrASP is state the art for model-based RI. in continuous domains. Instead, we show empirically that despite the nonstationarity

---

Under review as a conference paper at ICLR 2022
of learning to select affordances whilst simultancously learning aalue-equivalent model that predicts their couseuunesss GrASP is fast enough at these two tasks for planning with the discovered affordances and their learned model to often be more sample efficient than model-free approaches in benchmark RI. control problems. A comparison with state of the art awaits further development of this new idea of using gradients to discover affordances for planning.
RELATED WORK
Other approaches to affordances in RL. Abel et al. (2014) defines affordances as a subset of actions available to an agent in state if the state meets certain preconditions defined through propositional functions. They introduce an approach for learning affordances in a scaffolded multi- task learning environment, where affordances learned from simple tasks with smaller state spaces are transferred onto difficult tasks with larger state spaces. With this approach, they demonstrate the utility of affordances to reduce the planning time in a learning agent. (Abel et al. (2015) extends this to the setting of goal-based priors: probability distributions over the optimality of each action for given state and goal pair. Both works are limited in their applicability as they rely on discovering affordances in Object Oriented-MDPS, which assumes objects, object classes, and their attributes as part of the state space (Diuk et al. 2008). is also limited in the sense that their affordance discovery approach heavily relies on carefully designed multi-task environment that provides curriculum to the learning agent. Khetarpal et al. (2020) defined affordances for MDPs and in simple discrete domains showed that hand-specified affordances allow for faster planning in an agent they reduce the number of actions available at a state and also allow for efficient learning of partial transition models of the environment.
Sherstov & Stone (2005) improved the time required to learn an optimal policy by transferring a reduced action set obtained from the policy of source task a new but similar target task. Rosman & Ramamoorthy
provide a method for learning Dirichlet action priors from a set of related tasks that were then shown to improve the planning speed. Other work (Even- al. 2006 Cruz
(2012) (2012)
et al. 2014 2016 2018 Zahavy et al., 2018) have also demonstrated the utility of action-pruning to allow an agent to scale up to environments with large action spaces.
In summary, none of the above approaches developed an approach to the discovery of affordances based on gradients through the planning procedure.
Gradient through trajectory in model to update policies. Another body of work, summarized below, can be seen as effectively using gradients through a trajectory generated from a learned model to update a policy. Our approach to learning affordances using gradients through a planning tree is different in at least two ways. First, we learn affordances that provide a set of actions as a function of state that agent should consider during planning. Second, we compute gradients through planning tree rather than a single trajectory. In our empirical work, we consider an ablation of GrASP in which learn single affordance mapping, in effect learning policy via gradients through trajectory as a means of comparing against an instance of this approach.
PIL.OO (Deseenoth & Rasmussen 2011) is approach that learned state transition models parametrized as Gaussian processes. The policy was derived from gradients obtained by differ- entiating through trajectory from the transition model and was shown to learn optimal policies in simple control tasks such as cartpole and mountaincar. Many approaches extended the idea from PILCO (Gal et al. 2016; Higuera et al. 2018 Chua al., 2018. Amos al., 2018) other kinds of models and domains. Dreamer, introduced by Hafner et (2020), used a recurrent architecture for learning latent transition models and directly learned the policy parameters from the world model by differentiating the unrolled model with the policy parameters. BBravan et al. (2020) introduced an approach that learns latent deterministic transition models and learned policy by backpropagating through transition model expansions, similar to that of Dreamer.
Continuous actions in RL. Stochastic Value Gradients (SVG) (Heess et al. 2015) allow learning of continuous control policies by obtaining value gradients of one-step model predictions and these value gradients have been shown to have reduced variance, thus producing successful learning algorithms such as Deterministic Policy Gradients (Silver al.) 2014), Deep Deterministic Policy Gradients (Casas) 2017) and Soft Actor-Critic (Haarnoja et al. (0188). These methods use gradients of one-step trajectories to update policies and thus quite different from GrASP.
2

---

Under review as a conference paper at ICLR 2022
Planning with continuous action spaces. Progressive widening (Couêtoux al. 2011 is early approach for planning in continuous- action domains when a simulator is available. It was recently extended to planning with value-equivalent models by Moerland et al. (0118) and Yang et al. (2020), but both of these have only been shown to work on domains with 1-D actions. MuZero (Schrittwieser al. 2020) is considered to be state-of-the-ar algorithm for planning with value-equivalent models in RL but was limited to discrete-action domains. More recently an extension of MuZero, called Sampled MuZero (Hubert et al.) 202T), was introduced for dealing with continuous actions. Sampled MuZero is different from GrASP in two important respects. Sampled MuZero uses the planning process to update a policy prior distribution over continuous action space and uses samples from this distribution to construct the planning tree, while GrASP uses gradients through the planning process to discover discrete set of affordance mappings which are used for building the tree. Like in MuZero, our GrASP agents learn value-equivalent models rather than observation prediction models. But unlike MuZero. our GrASP agents also learn option-models in addition to primitive- acctiioo models. (We could not directly compare our method against MuZero because code was released with the paper, and our attempts at a local implementation efficient enough to be usable did not succeed).
2 GRASP DETAILS
Key idea. Our RL agent is a planning agent that uses an affordance module that maps state representations to small set of K actions or options from continuous space for use in expanding look-ahead search tree. The affordance module is represented by a network with parameters @afford and so the planner is, in effect, parameterized by gafford The key idea underlying our method of discovering useful affordances for planning is that we can compute the gradient of performance loss with respect to gafford through the computations of the planner.
The planners we explore here use the affordance module and a learned value- equivalent model to expand a lookahead tree. In principle any tree-expansion procedure could be used including various MCTS algorithms. The key requirement is that performance-loss gradients w.r.to (afford can be computed through the planner's computations, then GrASP can discover affordances online. In our empirical work we will use two tree-expansion procedures: (I) shallow-depth complete trees with depth as a parameter. We use learned value functions at the leaf nodes to bootstrap value estimates at the root node and shallow depth can be sufficient, and the ability of GrASP to discover useful affordances means that we can have very few action choices at each state and thus afford to build complete lookahead trees. (2) UCT-based MCTS with number and depth of trajectories as parameters; here we choose affordances from the learned affordances based on an upper-confidence bonus.
High level overview. Figure pro- vides an overview of how the planning agent selects actions or options exe- cute in the environment using the plan- ning tree. The current observation from the environment x is encoded into an abstract state s which becomes the root node in the planning tree expanded us- ing fafford to generate actions/options at each abstract state node that is ex- panded, and using the value equivalent model to generate next abstract states and predictions of rewards and values. The reward and value predictions are backed-up to produce Q-values for each action or option, which are used con- struct a policy over the actions/options at the root node. An action/option is sampled from this policy and executed to produce a transition in the environ- ment. The next observation is encoded to new abstract-state, from which the agent repeats its planning procedure. There are thus five learned components: the novel affordance module fafford
Observation
penode
SO
Affordance-selection faford
pynamics
8
(o a,) ((s,))
5
S
S
Affordance-selection potord
a2
0x
plpamics
(S a,) (s,))
S
$2
Figure 1: GrASP using its model to perform lookahead planning with actions/options selected via the affordance module. The agent plans by either constructing K-ary depth- search tree or partial search tree through UCT.
3

---

Under review as a conference paper at ICLR 2022
and the four components of the value equivalent model, an observation encoder faneade, state-next state dynamics function flynamic, a reward predictor frew, and a value function approximator fV. In what follows specify the key aspects of the algorithm and how the module parameters are learned; full details are in the Appendix.
agent-environmen interaction. We assume discrete time setting in which agent interacts with a continuous-obsee observation environment using actions or options selected from a continuous space. For the most part we will treat options and actions if they are equivalent by overloading some terms such as reward and next observation. In the case of selecting actions, at each time step & the agent receives an observation X E R", selects an action a, E A = IR" and then receives reward r E R and new observation X(+1- In the case of selecting options, if option a E AA (note A is used to denote both the space of actions and options) is selected at time l, its policy is followed until termination and the agent receives reward (defined as the discounted sum of rewards from the start of the option until termination), and new observation Xt+n (if the option lasts for n time steps). The next option is then chosen at time t + n. The agent's s goal is the usual RL goal of maximizing discounted sum of rewards over an episode (all our tasks use discount factor of 0.99).
Affordance module. The agent encodes each observation x, into an abstract state S, using an encoding module describe below. The affordance module is a network with parameters @afford that maps abstract-states to K actions options a' ..... selected from A; i.c., fafford - {a² _a_,aa... ........ ax). In our experiments, we use a simple feedforward neural net architecture for the affordance module with K separate output heads, each head outputting different action/option.
Value-equivalent model representation. The agent plans with a value- equivalent model represented by four neural networks parameterized by 00 [Oencode, Odyyamicc (rew 0') J. These four modules are in common with the modules in MuZero's 's value equivalent model (Schrittwieser et al. 2020); the difference is that our agent does not have a policy-prediction module and that we allow for trriiitttansssions that are temporally extended while MuZero was restricted to iisionttaansiionss The functions computed by each module are defined as follows: The representation or encoding module fencode X - s. parameterized by gemoode, maps an observation x to an abstract state s E S Th. The abstract state representations are learned; they are not latent environment states or observation predictions. The value module fV + I parameterized by 0V estimates the value of an abstract state. The reward module frew : s, a - f E IR, parameterized by (rew, predicts the discounted sum of rewards until termination after executing For options, the agent learns to predict the expected duration of an option given that it is started in encoded state s. The dynamics module pdynamic 8, - s', parameterized by @eyynmii maps an abstract state s and action/option a to a predicted next abstract-state s' E S.
action or option given state
Value-equivalent model training. The parameters of the representation, dynamics and value networks are trained so that the resulting model estimates the value of the environment observation as accurately as possible (hence the term value-equivalent). The reward network is trained predict the rewards that the agent experiences from a given ibservation-aatioonoptioo pair.
Training operates by sampling a sequence of transitions from the replay buffer, described in pscudocode in Appendix, then using these transitions to update the neural networks of the learning agent. We briefly describe here the learning update of gM; the update of gafford is described below. Given a sequence of state-action/opiion-eewar transitions {x1, a1, X2, a2, F3, X3 xx,) ). ). the agent first constructs abstract-state Sı fencode (x1). then predicts subsequent abstract states pdynamic (s; 1). The model parameters 0 are then updated to minimize the loss:
n moddel Σ
fre ai)) + (0, (o.
where, 0; is the discounted sum of observed rewards bootstrapped by the predicted value function of the last abstract state in the sequence from target value function network (the value function network is copied into the target value function network periodically). Note that the model loss function not used to adapt the parameters of the affordance mappings. Model-learning and affordance-discovery are thus independent, though they do constrain each other indirectly because model-learning leads to learning of the abstract state representations which are imput to the affordance mappings and similarly the affordance mapping's 's choice of actions/options produce the trajectories for the replay buffer that used as data for model-learning.
4

---

Under review as a conference paper at ICLR 2022
Lookahead tree expansion using affordances and value-equivalent model. In the case of com- plete trees, the planning procedure expands a K-ary search tree to depth D, expanding each abstract state node with all K actions/options computed by the affordance module for each abstract state. In the case of UCT, the planner rolls out H trajectories to depth D) the trajectories then implicitly define partial tree. Each trajectory starts the root note. If there have not yet been selected at the root node, we choose one of those at random. Else, an expansion procedure selects greedily based on the sum of the current Q-values at the root node (see below for backup procedure) and UCT bonus that boosts the chances of selecting less frequently chosen affordances. At a non-root node whose abstract state has previously been encountered we follow the same expansion procedure (except there is no requirement to select all the affordances at least once). At a non-root node whose abstract state has not been encountered we pick an affordance at random.
more affordances that
Action selection via planning tree. Regardless of whether we have complete tree or partial tree produced by UCT the following recursive procedure is applied. The value of a leaf-node the value of the corresponding abstract state from the value network. The value of non-leaf node is a weighted sum of the Q-values for all the actions/options taken in the partial tree from that node. Suppose the abstract state of the non-leaf node under consideration is the actions/options in the tree are 01, 02, ... a. the Valuuee of (s, ) is the sum of the reward from the reward network for (s, ag) summed with the appropriately-diccountte The value of node 8 then is Σ. π (a(ss)((s, a;), where π is probability distribution over actions selected in the tree. For complete-tree based planning [{;]ss) Σ, exp(Q[s,a;)/7]
value of the successor node in the tree to (s, a;).
exp(Q(s.ar)/r) where the
Σ, n(s,a,)) where n(8, a;) the count of the number of times action a, is chosen in state s in the partial tree. Finally, the action executed at the current observation distribution defined by π at the root node.
temperature parameter, while for the UCT-based tree the weight ((al(s) = n(s.a,)
result of planning is selected by sampling from the
Gradient update for the affordance module. The agent updates the affordance module parame- ters by learning to maximize the value-estimate at the root node (so).. As described above, the value-estimate obtained via a value backup procedure through the tree, which yields the follow- ing expression: V(su) = Eb so.... .xx T (bb|su)
(S))ss) (So. b). where the summation is over the action/option-seleeteeo produced by the affordance module. Note that for both the complete tree and the partial tree produced by UCT, each of the affordances is selected at least once at the root. The recursive form of the value function leads to the following recursion to compute the needed gradient:
(V(ss)
Σ
* x(b/8))((s0. b)]
@gafford
0oafford
BE DEla),.0 ax}
@T((b/s)) agatford Q(50, b) + (b|ss)) fafford
aQ (Sg. b)
Σ
BE (A( .x)
Recall that ((as(s)) is a function gafford through the action/option selections a, that are outputs of the affordance module. Finally, aQ(sg.a))
digalood can be computed via recursive computation very similar
to the value backup procedure:
aQ a.) agatford
Drrew (so, a:) agatlord
@V(s) Doatford
where is the expected duration of option a,, This recursive gradient term can be efficiently computed with existing auto-differentiation packages (Griewank & Walther 2008) with minor additional computational complexity.
EXPERIMENTS WITH GRASP AGENTS
We report here experiments with GrASP agents on three hierarchical tasks requiring policies over a space of continuous options (note, we assume only the pretrained option policies are available, not the option-model and on nine domains from the DeepMind Control Suite (Tassa et al. 2018) that requiring policies over continuous primitive-actions, Our first aim is demonstrate that planning agent using GrASP is able to learn intuitively sensible affordances and to show that they discover good affordances quickly mnuuhh-whil simultaneously learning a value equivalent model to remain
5

---

Under review as a conference paper at ICLR 2022
competitive with strong model-free baseline. Furthermore that GrASP can do with both the simple complete-tred cemplete-trreeeeee planner and the more scalable UCT-based planner. We chose TD3 (Dankwa & Zheng. 2019) as our model-free off poolyyy baseline it produces stable learning and state-of-the-an art performance on many continuous control tasks using a non-distributed agent. Our second aim is to provide evidence that multiple (K>1) affordance mappings
useful for planning, in that the agent's 's comppinegd policy switches between the K affordance mappings, and yields better performance (asymptotically and in rate) than using any single individual mapping as a policy.
Neural Network Architecture and Hyperparameters. The TD3 and GrASP agents all used simple feedforward NN modules. We tuned the learning rate hyperparameters of TD3 and GrASP on the Point-Mass task from the DM Control Suite and the tuned hyperparameters were then used across all our experiments. The planning depth for K-ary search tree was tuned on Fish-Swim and depth of 2 was found to work well there. The hyperparameters related to UCT were set to be identical to the ones reported the MuZero algorithm (Schrittwieser et al. 2020). More details about the NN design and all the hyperparameters used can be found in the Appendix.
3.1 LEARNING OPTION- AFFORDANCES IN HIBRARCHICAL TASKS
Here we present results on 3 hierarchical control domains in which we provide the agents with pretrained navigation options from continuous space. In each of the 3 domains, the option space general but designed to be such that within the space there exist options that are object- or configuration-centio in ways that would recognize intuitively dances to use when planning. Furthermore, the space of multi-tasks for each domain is designed such that for GrASP to perform well across tasks it ought to find the intended affordances for planning. The main evaluation goal is to see if GrASP succeeds in doing so.
sensible environmental affor-
Tasks and option spaces:
Collect for Object-Centric Affordances We designed our first domain and tasks so it would be easy to visualize the affordances, and where to achieve high performance the affordances should be object-centric, i.e., options that are in some way oriented toward objects in the environment. The enviromment is a continuous 2D world where the start of each episode the agent and three different objects, denoted A, B, and C, are placed in random positions. A task, encoded in a goal vector given to the agent, involves collecting the three objects in a particular order. The agent receives a reward of 1 when it collects the objects in the correct sequence. The episode terminates without any reward when the agent picks an object out of order. There are five discrete primitive actions: up.down,left, right actions which move the agent in discrete steps of size e. and a collect action collects an object only if it is within e distance of the agent. We defined and pretrained a 2-D continuous space of navigate-aii aanate--nd-oolleet options that move the agent to within of z, y position and then execute the collect action. Note that the option space is not object-centric, but we expect GrASP to learn an affordance mapping that selects the 3 options corresponding to collecting the 3 objects.
Ant-Gather for Object-Cern Centric Affordances. The second domain and tasks involves navigation with an Ant agent. The agent starts random location with 3 apples and 5 bombs that are placed at fixed locations. The task is to navigate to and collect the apples while avoiding the bombs. The agent receives a reward of when it collects an apple and O when it collects bomb. We pretrained space of options to achieve any fcasible x, locations with u and velocities; so the space of options is not object-centric. Again, the goal is to see if GrASP can discover object-centric affordances this time with the additional aspect of avoiding certain objects.
Point-Mass for SrnfigurratinnCeetii Affordances. The third domain and tasks is adapted from the Point-Mass domain in DM Control Suite. The agent is a point-mass mass and must navigate and pass through three fixed locations in a specific order that is encoded in goal-vector given to the agent. The agent receives reward of at the episode termination if it crossed the three locations in the correct order, otherwise the reward is O. We pretrained a 4-D option space to achieve any feasible x-y location with any and velocity. In this case, the intended affordances are specific goals of achievement in configuration space (location and velocities).
Variants of the GrASP -based planning agents. We create distinct GrASP agents by varying type of affordance mapping. number of distinct affordance mapping heads, and planning algorithm for the complete-tre tree lookahead at depth D 2. We explored two kinds of affordance mappings: mappings conditioned on both abstract state and goal configuration, which we refer to as Goal-conditioned Affordances (abbreviated GA), and mappings that do not condition either states or goals, which

---

Under review as a conference paper at ICLR 2022
Herarcicca Pelm
Heracchcal Ant-Gitthr
Herardiical Collect Tack
Moan Epicodic Returm
4.3
Mean Epteadic
Epicodic
2 E
022
723 1101
Unvironment
Emvironment Step
Env invironment Step
Figure 2: Top: Learning performance of GrASP agents and the TD3 baseline on hierarchical tasks with pretrained continuous options. Bottom: Visualizations of object-centric options discovered by the GrASP SA-3 agent in the Collect task. Left panel shows the trajectories generated by the option chosen by each of 3 affordance heads, in different colors, from three different starting states for the agent. Right panel shows the same for a single start state but varying positions of objects.
we refer as Actions (abbreviated A), because they are similar conventional agent action sets that do not vary by state. For each kind of mapping we also explored varying the number of distinct mappings K, considering K 1, 2, 4, for Goal-conditioned Affordances (abbreviated GA- GA-2, GA-4, GA-8), and K 4, 8 for Actions (abbreviated A-4, A-8). We did not explore A-1 1 or A-2 because is not possible to solve these tasks with one or two fixed actions in the continuous space. Note that GA-1 has special status because is collapses the search tree to single trajectory and so implements in our framework the main idea in the algorithms described in the Related Work section Gradient through trajectory in a model update policies. In the Collect domain we also tested GrASP agent, SA-3, that has 3 affordance heads that condition on state but not on the goal vector.
Results. Figure 2 (top) shows the learning performance for the TD3 baseline, GA-1 (uses the same option-space as the GrASP agents), and the best performing conditioned Affordances and Action agents, GA-4 and A-8 (See Appendix for all learning curves). Each learning curve represents a mean over 5 random seeds and the shaded regions denote their standard errors. We note the following: (1) The GrASP agents (GA-4 and A-8) learn much faster and achieve much higher levels of performance than TD3 in all three tasks. In 500K steps the TD3 agent shows almost no learning; we continued training TD3 until 1M steps and plots its performance at that point as the dashed horizontal line. (2) In two out of three tasks, GA-4 outperforms A-8. (3) In all three tasks, GA-4 learns significantly faster that GA-I, demonstrating the benefits of tree-based planning with (bottom) visualizes the option affordances selected by the GrASP SA-3 agent in Collect; it is clear the agent has learned that the best options to plan with are those that go to objects. In effect has learned object-centric affordances in this environment. Similarly, the affordances learned in Ant-Gather object-centric in that they navigate to apples (while avoiding bombs) and the affordances learned in Point-Mass correspond to goals of achievement in configuration space.
Figure 2
3.2 LEARNING ACTION-AFFORDANEE IN DEEPMIND CONTROL SUITE
We evaluated GrASP on the following DM Control Suite tasks (Tassa al. 2018 Reacher-Easy, Cartpole-Swingur Point-Mass-Eaue Easy, Ball-In-Cup-CC Finger-Spin and Walker- Walker We selected these because standard model-free RL agent can learn a close-to-optime policy within 1M environment steps (adopting Srinivas et al. (2020)'s 's criterion). All agents observed the low-level state based features, not pixel observations. Agents also observed a target goal configuration which varied from episode to episode. In the appendix, we present results showing that GrASP is comparable to Dreamer (Hafner et al.) 2019) (whose code was available) in the experimental setting used by Dreamer in which the agent observes pixels. In addition to the GrASP -variants introduced in Section. 3.1 here we also consider versions using UCT for which we did a more limited exploration of the space of possible parameters: we fixed the number of affordance mappings to 4, used only goal-condition affordances, and varied number of trajectories to 20 and 50 (denoted UCT-20 and UCT-50 respectively).
Fish-Swim, Fish- Upright, Cheetah-Rur,
7

---

Under review as a conference paper at ICLR 2022
easy
point easy
ball cup:
1010
800
Epicadic
638
300
430
100
200
200
cartpole: swingup
fish: swim
fish: upright
500
000
Mean Epiende Retum
630-
$30
300
432-
400
230
200
300
230
230
cheetah:
walker: walk
330-
Mean Epicodi: Return
230-
230-
500
UCT: 50
235-
GA:
230
100
200
A:
GA:
TD3
30000 www.eo wooooee www.eeeeeeeeeeeee.. Tolcoo Erwiiron ment Step
www.oaapeeeeeeeeeec..... 50000/
Environment Step
Environment Step
Figure 3: Learning performance of four GrASP agents and the model-free baseline TD3 on nine DM Control Suite tasks. UCT-50 is the best performing UCT-based GrASP agent; GA-4 is the best performing of the unconditioneed-Actionss agents; GA-1 is of special interest because collapses the tree to a single trajectory; text. Learning curves for all agents are in the Appendix.
Affordance agents; A-8 is the best performing of the
Results. Figure shows the learning performance for the baseline TD3, GA-1, the best performing agent of the Good-ccoditioned Affordance and Action agents, GA-4 and A-8, the best performing UCT-based GrASP agent UCT-50. (We show this subset to make reading the graphs easier. See Appendix for all learning curves; most GrASP variants outperform TD3 in most domains), Each curve a mean over 5 random seeds; shaded regions are standard errors. Note the following: (1) All three of the good GrASP agents (GA-4, UCT-50, and A-8) learn faster achieve better asymptotic performance than the TD3 baseline in 8 out the tasks. Cartpole-Swinn where all configurations of GrASP fail improve the TD3 baseline agent. (2) In most tasks (8 out of 9) the GA-4 GrASP agent learning conditioned affordances performs better than the A-8 Finger-Spin the only task where GrASP
is the only domain
GrASP agent learning select unconditioned-action learning actions produces better performance than agents learning affordancces. (3) The GA-4 agent learning multiple (K 4) affordance mappings produces better asymptotic performance than GA-I, which is learning only one affordance mapping, in 4 of the 9 tasks. In 3 of the 9 tasks, the GA-1 agent is better, and in the other 2 tasks there is little difference. (4) The UCT-50 agent is not clear winner over the complete-tree based agents, doing slightly better in a couple of domains but slightly worse in the others; this may be because discovering affordances while bootstrapping with learned value functions allows for the simpler shallow-depll complete-tres planning compete well with the more sophisticated UCT-based planning.
Analysis of switching between affordance mappings. A single conditioned affordance mapping has the form a policy mapping states/goals to actions/options. The outputs of multiple affordance mappings interesting affordances to the extent that the agent's soaniing-computdd policy switches between the affordance mapping outputs, and the switching policy performance is better than using any single affordance mapping policy.
8

---

Under review as a conference paper at ICLR 2022
Swftchin Palryy Betum
Switching Palicy Return Affondance head Ruturn
Swiching Policy Affredance Heed 3 Return
Affordance leedd 4 &Retum
1981
3.44
0400
0.43
0.30
0.00
Epeasn Retues
Pp:0030 Returr
Figure 5: Top: Finger-Spin; Bottom: Walker-Walk. Distributions of differences in returns between the GA-4 agent planning with discovered affordances and agents following the policies of single affordance-mappinn heads.
We conducted two qualitative analyses of switching, In the first analysis we confirm that the agent policies do switch between affordance mappings by producing simple visualizations of the switching behavior of the trained policies across the domains. In all domains we observe rapid switching for all GA-k agents k 1. Figure shows an example in Fish-Swin of how GA-4 switches between actions computed by the 4 distinct affordance heads. In the 2nd analy- sis we compare the returns of the switching policy obtained by planning over the trained affor- dances to the returns of agents that use single one of the K affordance mappings as policy. For a 1000 distinct start-goal configurations we computed performance difference scores between the planning- with- affordances agents and the affordance affordance-mapping mapping-asppeeeeeeeoa -solicy agents. We summarized these differences with histograms for each affordance-mapping head. If these distributions are skewed to the positive side of zero, it implies that planning with discovered affordances obtains returns that are generally higher than following& single affordance-mant mappiing policy. Figure [ shows two interesting and representative cases (full results in the Appendix). Finger-Spin a domain where GA-4 showed substantial gains over TD-3 and GA-1, and the distributions of difference scores is positively skewed for all affordance heads. For Walker- Walk the asymptotic performance of GA-4 GA-1, TD-3 are comparable, and the distributions of differ- ences tightly clustered around zero. In general, across the domains and across the GA-k A >> agents we see positively skewed distributions when those agents outperform GA-1 and TD-3.
Ep Episode Slep
Figure 4: A visualization of the GA- 4 agent switching between actions output by the K = 4 distinct affor- dance heads. Partial episodes are from Fish-Swim.
CONCLUSION AND FUTURE WORK
The two main contributions of this work are (1) the idea of discovering affordances- or options that are considered at each imagined state in planning- address the challenge of planning with continuous actions/option; and (2) GrASP gradient-based method for discovering good action and option affordances by computing gradients through planning procedures update the parameters of functions that compute the affordances. We demonstrated the ability of GrASP to learn both primitive-action and option affordances quickly enough while simultaneously learning a value-equivalent model to yield performance on multiple continous control tasks that is competitive with often better than a strong model-free baseline. Analysis of the agent policies shows clear evidence that the learned affordance mappings are useful as affordances for planning and not merely as policies. We provided evidence for the generality of GrASP by combining it with two distinct planning procedures-2 a simple complete-tree lookahead planner and the more scalable UCT-and by showing its effectiveness in discovering both mnmitive-acttion and option affordances. In our view, the main limitation of the work presented here is that we have not yet integrated our affordance discovery into a sophisticated model-based RL algorithm like MuZero that employs a number of other ideas to scale to large domains. This is clear and exciting next step.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Abel, Gabriel Barth-Maron, James MacGlashan, and Stefanic Tellex. Toward affordance-aware planning. In First Workshop on Affordances: Affordances Vision for Cognitive Robotics, 2014.
David Abel, David Hershkowitz, Gabriel Barth-Maron, Stephen Brawner, Kevin O'Farrell, James MacGlashan, and Stefanie Tellex. Goal-based action priors. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 25, 2015.
Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gómez Colmenarejo, Alistair Muldal, Tom Erez, Yuval Tassa, Nando de Freitas, and Misha Denil. Learning awareness models. arXiv preprint arXiv:1804.06311 2018.
Arunkumar Byravan, Jost Tobias Springenberg. Abbas Abdolmaleki, Roland Hafner, Michael Neunert, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Imagined value gradients: Model-based policy optimization with tranferable latent dynamics models. In Conference on Robot Learning, pp. 566-589. PMLR, 2020.
Noe Casas. Deep deterministic policy gradient for urban traffic light control. arXiv preprint arXiv:1703.09033, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in a handful of trials using probabilistic dynamics models. arXiv preprint arXiv: 1805. 2018.
Adrien Couêtoux, Jean- Baaanttpttst Hoock, Nataliya Sokolovska, Olivier Teytaud, and Nicolas Bonnard. Continuous upper confidence trees. In International Conference on Learning and Intelligent Optimization, pp. 433-445. Springer, 2011.
Francisco Cruz, Sven Magg, Cornelius Weber, and Stefan Wermter. Improving reinforcement learning with interactive feedback and affordances. In 4th International Conference on Development and Learning and on Epigenetic Robotics, pp. 165-170. IEEE, 2014.
Francisco Cruz, Sven Magg, Cornelius Weber, and Stefan Wermter. Training agents with interac- tive reinforcement learning and contextual affordances. IEEE Transactions on Cognitive and Developmental Systems, 8(4):271-284) 2016.
Francisco Cruz, German Parisi, and Stefan Wermter. Multi-modal feedback for affordance-driven interactive reinforcement learning. In 2018 International Joint Conference Neural Networks (JJCN), pp. 1-8. IEEE, 2018.
Stephen Dankwa and Wenfeng Zheng. Twin-delayed ddpg: A deep reinforcement learning tech- nique to model a continuous movement of intelligent robot agent. Proceedings of the 3rd International Conference Vision, Image and Signal Processing, pp. 1-5, 2019.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (C),L-I1), pp. 465-472. Citeseer, 2011.
Carlos Diuk, Andre Cohen, and Michael L Littman. An object- representation for efficient reinforcement learning. In Proceedings of the 25th international conference on Machine learning, pp. 240-247, 2008
Eyal Even-Dar, Shie Mannor, Yishay Mansour, and Sridhar Mahadevan. Action elimination and stopping conditions for the multi-armes bandit and reinforcement learning problems. Journal of machine learning research, 7(6), 2006.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforee- ment learning. arXiv preprint
2017.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data- Exficient Machine Learning workshop, ICML, volume 4, pp. 25, 2016.
10

---

Under review as a conference paper at ICLR 2022
James J Gibson. The concept of afforddnccss. Perceiving, acting, and knowing, 1, 1977.
Andreas Griewank and Andrea Walther. Evaluating derivatives: principles and techniques of algorithmic differentiation, volume 105. Siam, 2008.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 18/2.
2018.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pp. 2555-2565, PMLR, 2019.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. International Conference on Learning Representations, 2020.
Nicolas Heess, Greg Wayne, David Silver, Timothy Lillicrap, Yuval Tassa, and Tom Erez. Learning continuous control policies by stochastic value gradients. arXiv preprint arXiv:1510.09142,, 2015.
Juan Camilo Gamboa Higuera, David Meger, and Gregory Dudek. Synthesizing neural network controllers with probabilistic model- based reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2538-2555 2544. IEEE, 2018.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and planning in complex action spaces. arXiv preprint arXiv:2/04.06300 2021.
Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, David Abel, and Doina Precup. What can do here? theory affordances reinforcement learning. International Conference on Machine Learning, pp. 5243-5245552 5253. PMLR, 2020.
Thomas M Moerland, Joost Brockens, Aske Plaat, and Catholijn M Jonker. A0c: Alpha zero in continuous action space. arXiv preprint arXiv: 180500961,, 2018.
Benjamin Rosman and Subramanian Ramamoorthy. What good actions? accelerating learning using learned action priors. 2012 IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDI.), pp. 1-6. IEEF, 2012.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, 588(7839):004-609, 2020.
Alexander A Sherstov and Peter Stone. Improving action selection in mdp's ' via knowledge transfer. In AAAI, volume 5, pp. 1024-1029, 2005.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pp. 387-395. PMLR, 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 55017676):55-3399 2017.
Aravind Srinivas, Michael I Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004..00 0410004404446..0...0000 2020.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint arXiv:1801.00699. 2018
Xuxi Yang, Werner Duvaud, and Peng Wei. Continuous control for searching and planning with learned model. arXiv preprint arXiv:2006.074 2020.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Leam what not to learn: Action climination with deep reinforcement learning. arXiv preprint 2018.
11

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 HARDWARE USED FOR OUR EXPERIMENTS
Each agent was trained single NVIDIA GeForce RTX 2080 Ti in all of our experiments.
A.2 EXPERIMENTS TO COMPARE WITH A MODEL-BASED RL APPROACH
In this section, we compare against Dreamer (Hafner et al. 2019) which is a model-basee approach that learns from pixel observations. Their experimental setup is different from those of ours considered in Section 3.2 in that they use pixel observations and they also use an action-repeat of 2. So we ran additional experiments with GrASP (GA=4) to match the experimental setting of that of Dreamer and compare their results here. We used Dreamer's 's open-sourced code for producing their learning curves. The results are averaged from random seeds.
Overall GrASP performs comparably with Dreamer matching or surpassing its performance in 3 out of tasks. GrASP in Finger-Spin reaches a higher asymptote and in Ball- In- Cup-Catch it seems to be learning relatively faster than Dreamer. The asymptotic performance is comparable in Ball- Sin--up--Core Cup-Catch and Palker-Walk, while Dreamer learns better in Cheetah-Run.
These results show that GrASP can handle tasks with pixel observation and can produce comparable performance with a state-of-the thrtt-rrrt model-based RL approach. We believe that this comparison along with results from Sections B.2 3.1 and the visualizations on hierarchical RL tasks clarifies our contribution, which is a method that identifies affordances that are useful for a planning agent (rather than improving ofte-offthhe-rt performance in specific continuous control tasks).
1s 2up: ratch
soiken welk
cheedat: Fu
Ripen opin
Scipa
Steza
Sapa
Figure 6: Learning performance of GIASP and Dreamer agents on four DM Control Suite tasks. We present GA=4 version as it was one of the best performing GrASP agents in Section. 3.2
A.3 SWITCHING BETWEEN AFFORDANCE MAPPINGS
Figure 7 shows the distribution of average episodic returns from 1000 episodes. Here we compare the returns of the switching policy obtained by planning over the trained affordances the returns of agents that use a single one of the K affordance mappings as a policy. We summarized these differences with histograms for each affordance- mapping head. If these distributions are skewed to the positive side of zero, it implies that planning with discovered affordances obtains returns that are generally higher than following a single affordance- sorodance-mapping policy. We conjecture that all the outputs of the affordance module collapsed to the same output, then this figure would be lot different: we would have seen the distributions to have concentrated around O. However, this is not the case in many of the domains: In Finger-Spin (shown in main text; Figure 5 top row), Fish-Swim, Cheetah-Run Fish- Aish-Uprihht and Reacher-Easy, -asy, the distributions are not concentrated at O and are spread over the range implying that these affordances indeed different from each other. In Walker Walk (shown in main text; Figure bottom row) and Ball-in-Cup -unCuCCCattch the distributions are concentrated at 0, which hints at the affordances collapsing similar outputs.
12

---

Under review as a conference paper at ICLR 2022
131
130
Figure 7: First Row: Fish-Swim; Second: Ball-in-Cuu -ill-in-Cup--ue Cup-Catch; Third: Cheetah Run; Fourth: Fish- Upright; Fifth: Reacher-Eass Easy. Each row corresponds to a DMControl Suite Task and summarizes the distributions of differences in returns between the GA-4 agent planning with discovered affordances and agents following the policies of single affordance-mapping heads.
A.4 ADDITIONAL VISUALIZATION FROM COLLECT DOMAIN
The visualizations in Figure 2 of the main text is from the SA-3 agent where the affordance heads condition on the state but not on the goal vector. For the GA-3 agent, the affordance heads condition on the state and goal vector, and so can essentially just learn affordances that directs the agent in picking up the next object that is required of the task. It has no incentive to learn to associate each affordance with collecting each of the objects at each state of the environment as has access the goal information. The SA-3 agent the contrary does not have the goal information and has to learn affordances that can allow it to solve all possible tasks defined in this environment.
The visualization of the affordances learned by the GA-3 agent is shown in Figure where the agent's start position is varied across the domain and trajectories from each of its affordances are shown. The task requires the agent to first collect the object and one of the affordances does associate with collecting that object. The remaining affordances however are leading the agent to different parts of the environment.
This visualization from GA-3 again shows that the affordances are learning something different from each other and are not collapsing to the same output.
A.5 COMPLETE LEARNING CURVES
We explored distinct GrASP agents by varying type of affordance mapping, number of distinct affordance mapping heads and planning algorithm for the DM Control tasks. We explored two kinds of affordance mappings: Goal- conl-coonii Affordances (GA) and mappings that do not condition on either states goals, which we refer to Actions (A), they similar conventional action
13

---

Under review as a conference paper at ICLR 2022
Figure 8: Visualizations of object-centric options discovered by GrASP GA-3 in the Collect task. The figure shows the trajectories generated by the option chosen by each of affordance heads, in different colors, from three different starting states for the agent. The task is fixed for each of the three starting states the agent.
space of an RL agent that available in all states within environment. For the GA mapping, we considered different number of affordance mapping heads K = 1,2, 4,8 8 (correspondingly abbreviated as GA-1, GA-2, GA-4 and GA-8), and K 4,8 for Actions (abbreviated as A-4 and A-8). For the experiments studying different number of affordance mapping heads, we fixed the planning procedure as a complete-tree lookahead of depth D 2. We also explored using GrASP agents with UCT as the planning procedure, instead of the complete-tree lookahead approach, for which we did a limited exploration of the space of possible parameters. We fixed the type of affordance mapping to cond-conni Affordances and the number of affordance mappings heads as 4, the number of simulation trajectories for UCT was varied to between 20 and 50.
In the main text, for the DM Control results, we presented learning curves from GA-4, A-8 and UCT-50 as they uniformly produced better learning performance when compared similar variants with different parameter settings. In Figure 9 we present the learning curves for different number of affordance mapping heads that we considered for Goal-conditioned Affordances and for Actions. We also present learning curves obtained from the GrASP agents using UCT in the same figure.
Identical experiments to the ones described for DM Control tasks were conducted for the Hierarchical Point-Mass and Ant-Gather tasks, and complete learning curves from those experiments presented in Figure I0 Since the UCT-based agents performed on similar levels to that of complete-tres lookahead agents on DM Control tasks, we did not explore using UCT on the hierarchical tasks.
A.6 IMPLEMENTATION DETAILS
Neural network architecture. We used the following NN architecture for the GrASP agent. The same architecture was used in both DM Control and Hicrarchical tasks. We used a three-layer MLP for the representation module. Each layer used 512 units. The reward, value and dynamics modules used a two-layer MLP, each with 512 units, to map the abstract state input into their respective outputs. The dynamics module maps from an abstract state and action/option selection to the next abstract state. The input to the dynamics module consists of concatenation of abstract state and action/option selection that is first embedded using two-layer MLP, again with 512 units each. The affordance module also used a two-layer MLP with 512 units each to map an abstract state to the multiple output heads, where each output head corresponds to an action/option selection made by the agent. The output of the affordance module was bounded by applying tanh activation. ELU activations were used throughout for all MLP layers.
The TD3 baseline agent consists of separate actor and critic modules, and as per its original presenta- tion (Heess al., 2015). no parameters were shared between those modules. The actor and critic used three-layer MIP each with 512 units which transformed their inputs to their respective outputs. ReLU activations were used for the MLP. The input to the actor is simply the observation received from the environment, while the input to the critic is the concatenation of the observation and the action selected by the agent at the same time step. The output of the actor is bounded between -1 and 1, which is achieved by applying tanh activation. This architecture is similar to the one used for the DDPG agent in Tassa et al. (2018).
Hy Hlyperaarameters used in experiments. We tuned the learning rates for the TD3 agent and the GrASP agent with conditoond Affordances with = 4 affordance mapping heads on the Point-Mass task from the DM Control Suite. We used this tuned learning rate for the rest of the
14

---

Under review as a conference paper at ICLR 2022
easy
point easy
cup:
1200
030
Maan Episadic Retum
600
320
200
150
cartpole: swingup
fish:
fish: upright
Episodic Retum
coo-
602
330
400
403
230
202
200
cheetah:
finger:
walker: walk
003
250
602
GA:2
Mean Epiconic
503
GA: 4
200
200
302
A:
UCT: 20
50-
102
SESEAEEEESAAREEEEEEEE Environment Step
D0 www.eeaeeeeeeeeeeaaaaaaa.. Environment Step
Environment
Figure 9: Learning performance of different variants of GrASP agent and the model-free baseline TD3 on nine DM Control Suite tasks. Recall that GA-1 is special interest because it collapses the tree single trajectory. We presented subset of the learning curves from this Figure as the main result in our main text.
Hierarchical Point Mass
risrarciacal Anc-aacher
Mean Episedic Return
3.0
2.0
GA.
Moan E
GA.
GA
GA/1
2.0
Environment Step
Environment Step
Figure 10: Learning performance of different variants of GrASP agents and the model-free TD3 baseline on 2 hierarchical tasks with pretrained continuous options. All the agents shown here learn select from the same pretrained space of option policies, thus making a fair comparison over the hierarchical agents. A subset of the learning curves from this Figure was presented in our main text.
DM Control Suite and for the Hierarchical tasks. We also used the same learning rate for different variants of the GrASP agent. The range of values for the initial learning rate hyperparameter was: (0.0001, 0.0003, 0.0005, 0.0007, 0.001, 0.003). For TD3, we tuned the learning rates of the actor and critic independently and found that actor learning rate of 0.0001 and critic learning rate of 0.001 produced optimal learning on Point-Mass task. We tuned the learning rate for the affordance module and found 0.001 to produce optimal learning the task. We did not tune the learning rate for the representation, reward, value and dynamics modules and set them arbitrarily to be 0.0001. We used an ADAM optimizer for learning the parameters of all the modules in TD3 and GrASP. The
15

---

Under review as a conference paper at ICLR 2022
hyperparameter for the optimizer was set its default value which is 1 10-8. The for the DM Control Suite were set to their default values and were not modified. The discount factor Y set to be 0.99. A replay buffer capable of storing 200k transitions was used for both GrASP and TD3 agents; the samples were drawn uniformly from this replay buffer. The TD3 baseline used soft updates to synchronize the target network with the online network. A smoothing factor of r 0.005 was used for these soft updates. The GrASP agents used hard updates to synchronize the target network parameters with the online network parameters and synchronized every T 1000 learning updates. We experimented using soft updates for GrASP agents but found that using hard updates produced better improvements than the version with soft updates.
Hierarchical tasks. In addition to the description provided in the main text, we provide more details about the hierarchical domains here.
Collect: This hierarchical task is described in the main text.
Point-Mass: The agent's start position randomly initialized from all possible states in its environ- ment. The task for the agent is to pass through the three objects at fixed locations in a specified order (defined through goal-vector) to receive a reward of 1. The episode successfully terminates when the agent receives reward of (i.e., successfully pass over the three objects in the required order). If the agent fails to complete the task, then the agent receives a reward of and the episode terminates when 500 steps reached. As there are three objects, there are 6 possible ways of ordering the three objects. Thus, there are 6 tasks defined using this environment.
Ant-Gather: This task is adapted from the Ant-Gather task introduced in Florensa et al. (2017). The task used in our experiments consists of three apples and five bombs located at fixed locations. The agent's s start position is randomly initialized from the possible states from this environment. The episode terminates when the agent either successfully navigates and collects all three apples from the environment when the number of episode steps reaches 500. The agent receives 1 as reward when it collects an apple and 0 when it collects a bomb.
B ADDITIONAL EXPERIMENTAN RESULTS
In this section, we present additional experiments that were designed for evaluating some of the design choices involving the GrASP agent. The design decisions made from these experiments were subsequently used for obtaining the results with the GrASP agent that were presented in the main
text.
Planning depths. Figure II Left shows the learning curve from three GrASP agents on Fish-Swim task from DM Control Suite, where all three agents use complete-tree lookahead search but plans with depths D 1, 2.3 respectively. The GrASP agent used Goal-conditioned A Affordances with K affordance mapping heads (i.e., GA-4). From this experiment, we observed a clear ordering of the agent's rate of learning w.r.to the planning depth: GrASP with D = 1 learned significantly faster than GrASP with D = 2.3. However, based on the final performance, GrASP with D marginally outperformed the other agents with D 1 and 3 planning depths. Thus, we selected to use D in all our main experiments.
GrASP agent that uses randomly initialized goal-conditioned affordances and actions. Fig- ure TT Right shows the learning performance of different agents on the Fish-Swim task. Specifically, the plot shows learning curves of GrASP agents GA-4 and A-4, and TD3 baseline. The plot also shows the performance of the GrASP agents that uses randomly initialized goal-conditioned affor- dances (RND GA-4) and actions (RND A-8). In other words, the RND GA-4 and RND A-8 agents do not learn the parameters of the affordance module. From this figure, we can clearly see that it is essential for the GrASP agent learn the parameters of its affordance module order to produce any learning on a given task. As these RND GA 4 and RND A-4 agents failed to produce any learning on the given task, we do not explore those agents on the remaining DM Control and Hierarchical tasks.
B. PSEUDOCODE FOR GRASP
Algorithm I presents the pseudocode for the GrASP agent that learns to select affordances in the form of actions or options.
16

---

Under review as a conference paper at ICLR 2022
Fish: swim
500
Mean Epioodic Return
602
AND GA:
402
RND A: 8
T03
Figure 11: Left: Learning performance of GrASP agents with different planning depths D = 1,2,3. All three GrASP agents use a complete-tree lookahead planning procedure and Goal-conditioned Affordances with K = 4 affordance mapping heads on Fish-Swim task. Right: Learning performance of GrASP agents GA-4 and A-8, model-free TD3 baseline, and GrASP agents with randomly initialized affordance modules RND GA-4 and RND A-8 on Fish-Swim task. The RND GA-4 and RND A-8 agents use identical architecture that of GA-4 and A-8, but do not learn the parameters of the affordance module.
Algorithm Learning Affordance Selections for Planning using GrASP
Initialization:
Initialize Agent's online parameters = {gencode gdynamic grew @V gafford) randomly Initialize Agent's target parameters {,...
Synchronize target parameters with the online parameters
Initialize Replay Buffer 1)
For each step:
# Agent-envvionmeet interaction loop Receive an observation Ie from environment
Encode the observation an abstract state 8, fenoode (æ:)
Sample an action ~ I (+)ss))
where π ( 8t)
result of complete-tree or UCT search
which uses rest of the agent modules: fdynamic, frew / fafford
Execute a, in the environment and receive next observation #141 and reward r++1 Append transition {xt, de, Pt-1: 2(+1} replay buffer D # Sample a minibatch, use it to obtain predictions and value targets from the Agent Sample sequence of transitions of length from replay buffer
fr,, a,, ri+1, Ti+1, ".. Fitn] ~
Encode observation v; an abstract state s; rencode (x:)
Obtain 8j fdynamie (8j-1, aj- 1). along with frew (1=-1, (8)),
for j i, i.....
Compute value target 0G. for = i.......
m:
i =r 71 + ... + ~ー max maxbe furtord paccode (#....)) fencodee b) b) where fencode, Q are obtained using the target parameters # Update Agent's parameters
Minimize [model -ijt frew- (s), + (6, - (s,))) (s)))
to update gencode gdynamic grew ,0V
Maximize Σ)= V(s,) to update gafford
Synchronize target parameters with the online parameters every T' learning updates
17


Reviews:
[{"review_id": "UJH91RO9uJl", "reviewer": "Reviewer_VDJ5", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is well structured and well written. \n- The proposed approach is articulated well.\n- The experimental section is good.\n- The authors have done a good job of explaining the challenges as the approach was developed. \n\n- While others in the multi-agent simulation community and the RL community have looked at incorporating affordance inspired concepts into agent models there are aspects of this work which is different and novel from other previous work. The consideration of how affordances will affect the generation of the actions branches in a planning tree is important. In my mind however, the fact the paper is considering the effect of goals on the nature of affordances (what the authors call Goal-conditioned affordances) is what makes this work different. \nThis is important because the literature on affordance theory from ecological psychology makes it clear that affordances are goal and intention dependent. In some previous works affordances are treated as static actions, whereas the concept is dynamic and agent dependent. \n\n- My criticisms of the paper are by no means major. I would have liked to have seen\n1. a more detailed motivating example as to why incorporating affordances in this way is important.\n2. the experimental domains selected could have been more sophisticated. I think the paper would have been better if there were less domains/games. Perhaps have a simple and complex domain. It is in the more complex domains (in my opinion) where it is worthwhile to incorporate a concept like an affordance.", "summary_of_the_paper": "This paper looks incorporating the concept of affordances from ecological psychology into the planning process. \nThe basic premise is that affordances represent the possible relevant actions available to an agent that are potentially moderated by their goals and the state of the world. This has the potential to reduce the size of planning search trees.\n\nIn this paper the authors claim two contributions (1) the incorporation of affordance consideration into the planning process and (2) an approach for discovering affordances which the authors have called GrASP as it is a gradient based approach. \n\nThe paper implements the approach and tests it experimentally on a range of domains from the DeepMind control suite.\n\n", "summary_of_the_review": "I think this a good paper. It builds on existing ideas on incorporating affordances but does it in a way that is novel.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nd4K1G4XxCn", "reviewer": "Reviewer_LdeK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n- As far as I am aware, this is a novel contribution and an interesting idea overall. Learning affordances seems to be closely related with learning options, which is a must for an agent to be able to learn how to perform complex tasks.\n- The paper is clearly written and is easy to follow. A reader with some moderate knowledge of RL should be able to re-implement the methods in this work.\n\nWeakness:\n- One thing is not quite clear to me:  what prevents the K heads from outputting similar actions? It could be possible that the initial set of affordances predicted by the model are quite similar, and it gets trapped in some local optimum that prevents the model from trying out other options. Does initialization play a big role?\nI feel like either I am missing something from the paper, or there is a little bit more context that's needed to understand when this method would work or fail.\nI would appreciate if the authors could clarify this.\n\n- The gradient update on affordance module. The summation over b is only considering K affordances, but if the total number of possible affordances is larger than K and you constraint to only K, you might never explore any of the one previously left out. How do you deal with this? Is there some type of exploration that's needed here?\n\n- Figure 4 is hard to interpret. I understand it is trying to show the selection across different affordances but maybe a line plot like this is not the right representation. Also, what am I supposed to get from this plot? Is switching often good? Is it bad? Some clarification around this plot would be useful.", "summary_of_the_paper": "The paper proposes dealing with tree search planning in continuous action spaces by learning affordances.\nThe authors propose an architecture composed of several modules, where they are able to plan ahead in a tree search manner by learning a module that expands K discrete possible affordance from each state node.\nThe results suggest that the proposed technique is able to deal with difficult tasks in which standard tree search methods would not be easy to apply.", "summary_of_the_review": "Overall an enjoyable read with interesting ideas. There might be some context or clarification missing, I am not convinced that the objective propose would explore available affordances/actions, and would not get stuck looking at a suboptimal subset.\nIf that is the case, it would be useful to know what are the requirements to prevent this from happening.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FCEjZlR6Vs7", "reviewer": "Reviewer_ohYy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n- Detailed ablation analysis of the proposed approach with shallow-depth complete trees and UCT-based MCTS.\n- Extended the Value equivalent model with affordance learning module\n- Qualitative analysis on the switches between affordance mapping outputs. Take-away: across the GA-k (k > 1) episode returns are positively skewed when those agents\noutperform GA-1 and TD-3.\n\n# Weakness\n\n- The claim that the paper introduced \"the idea of discovering affordances for planning in continuous domains\" seems inconsistent. The paper discusses Sampled MuZero in related work, that extends MuZero to continuous action space. In contrast, GrASP uses gradients through the planning process to discover a discrete set of affordance mappings which are used for building the tree. How is the discrete set of affordance decided from continuous action and state space? It seems like a design choice specific to each environment.\n- The paper does not compare the proposed approach with other SOTA model-based algorithms. While the authors mention \"GrASP algorithm is not the SOTA model-based RL in continuous domains\", it is still difficult to say how significant the contribution of the proposed approach is.\n    - GrASP is compared to another model-based algorithm Dreamer in the Appendix. This seems like a relevant comparison, however some observations like \"in *Ball-In-Cup-Catch* it seems to be learning relatively faster than Dreamer\" seem inconsistent with the plots shown. Dreamer performs better than GrASP on Walker:walk and cheetah:run. A discussion to analyze this relative performance between GrASP and Dreamer will be helpful to understand the merits of each of the models.\n    - The result section highlights which of the different variants of GrASP perform best in different scenarios. While this looks like a detailed ablation analysis, it does not provide the reviewer/reader with\n- Action and option affordance learning ablation is compared to a just model-free baseline (TD3). There seems no take-away message on what model works the best, and is task environment/hyperparameter dependent.\n- The claim \"*Cartpole-Swingup* is the only domain where all configurations of GrASP fail to improve over the TD3 baseline agent.\" seems incomplete. While at least one of the GrASP variants learn faster than TD3, the asymptotic performance for TD3 seems higher in reacher:easy and walker:walk.\n\n# Minor\n\n- Pg 5 \"Each trajectory starts at the root ~~note~~ node.\"\n- [Clarification] How to compare a discrete time \"step\" in TD3 vs GrASP?", "summary_of_the_paper": "The paper proposes a gradient-based affordance selection, i.e., action/option selection, in addition to the value equivalent modules for tree-expansion procedure(s) in planning.\nThe claim is that GrASP can learn primitive-action and option selection and plan in a continuous state and action space to outperform model-free RL.\nThe affordance module maps the state to K actions or options from continuous space. These affordances are later used for expanding the look-ahead search tree. The key idea is to compute the gradient of performance loss with respect to the affordance model's parameters through the planner. The method uses two expansion procedures: (1) shallow-depth complete trees, (2) UCT-based MCTS. The performance of TD3 is compared to different variants of the proposed algorithm on some tasks from the DM Control Suite.", "summary_of_the_review": "While I enjoyed the addition of affordance module in value equivalent network and found it interesting contribution to integrate learning and planning, the experiments do not analyze all the merits of the proposed approach as compared to prior work. It is unclear how design choices like the discrete set of actions/options is constructed from the continuous space.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "76m8u7dGlSC", "reviewer": "Reviewer_pYew", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper approaches the problem of continuous control from a discrete planning approach, and shows how the discrete set of actions/options can be updated with gradient descent. \n\n- The experiments highlight the advantage of GrASP when operating over a (separately learned) lower-dimensional option space. They also analyze the different aspects of the algorithm, including different number of heads, conditioning the affordance module on the goal, and the degree of switching between outputs.\n\nWeaknesses:\n- The proposed algorithm is only compared to TD3 in the main results. The paper acknowledges that it is not state of the art for model-based RL in continuous domains, but it would nonetheless be useful to include the comparisons to DREAMER in the main set of results, i.e., in Section 3.1 and Section 3.2. Other potential comparisons include [1] and [2].\n\n- Overall, it seems that GrASP has the greatest advantage when operating over option spaces. This could be because the action space is already narrowed down and the planner only needs to operate over a lower-dimensional action space, e.g., the tasks in Section 3.1 have 2-D or 4-D spaces. Or perhaps, it’s due to the fact that the effective planning horizon is shortened. In Section 3.1, I’m curious how GrASP performs on higher-dimensional option spaces and on tasks with varying horizon lengths.\n\nFinally, there are some details that are still unclear to me: \n- Does the TD3 baseline in Section 3.1 also operate over the pre-trained options? It’s not clear whether TD3 is performing poorly using primitive actions or with the pre-trained options.\n\n- In the implementation of the affordance module, is there some regularization to ensure the output actions/options do not collapse to the same output?\n\n- I’m also curious about the discovered actions/options. Do they generally concentrate around the optimal action(s) or are they diverse among themselves? I also assume that the visualizations in Figure 2 are from the A- agents, and that the GA- agents discover something different?\n\n[1] Chua et al. Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models. NeurIPS 2018.\n[2] Bharadhwaj et al. Model-Predictive Control via Cross-Entropy and Gradient-Based Optimization. L4DC 2020.", "summary_of_the_paper": "This paper proposes a gradient-based method for selecting affordances in planning. Specifically, it includes an affordance module that maps state representations to continuous actions/options. The planning procedure is aided by a learned model with an encoding, dynamics, reward, and value module. Most prior work uses the gradients through the trajectory under the learned model to update the policy/actions, whereas this work is about using them to update the collection of K potential actions to select from.", "summary_of_the_review": "This paper proposes a gradient-based approach for planning in continuous control problems. However, the experiments currently lack relevant comparisons to understand the significance of the proposed approach. I’m also curious how performance varies on tasks with different option spaces and different horizons.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a learning framework that allows for planning in continuous action spaces using tree search. Key to the approach is performing tree search over a discrete set of learned affordances that provide a compact abstraction of the action space that facilitates planning. The affordances are learned by passing gradients through a model-based planner that uses learned models of the dynamics, reward, and state-value functions. Experimental evaluations demonstrate the ability to perform tree search-based planning using the learned affordances in a variety of domains for which tree search would otherwise be difficult.

The paper is topical, both with regards to its consideration of affordances as temporal abstractions that facilitate planning as well as the broader notion of integrating planning and learning. Several reviewers agree that the means by which affordances are learned by passing gradients through the planner is both interesting and novel. The reviewers also emphasize that the paper is well written and easy to follow, and that the approach is reproducible as a result. The reviewers raised a few concerns with the initial submission, notably the need for experimental comparisons to other recent baselines, which are important to clarifying the significance of the contributions, and the susceptibility to collapse in the affordance distribution. The authors clarified some of these questions and proposed adding comparisons to other baselines (e.g., DREAMER, for which there is already a comparison in the appendix), however it is not clear whether the submission was updated accordingly. The authors are encouraged to take this feedback into account and to include a more thorough experimental evaluation in any future version of the paper.
Decision: Reject