Title: Multi-Step Generalized Policy Improvement by Leveraging Approximate Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: generalized policy improvement

successor features

transfer learning

model-based reinforcement learning
Abstract: We introduce a principled method for performing zero-shot transfer in reinforcement learning (RL) by exploiting approximate models of the environment. Zero-shot transfer in RL has been investigated by leveraging methods rooted in generalized policy improvement (GPI) and successor features (SFs). Although computationally efficient, these methods are model-free: they analyze a library of policies---each solving a particular task---and identify which action the agent should take. We investigate the more general setting where, in addition to a library of policies, the agent has access to an approximate environment model. Even though model-based RL algorithms can identify near-optimal policies, they are typically computationally intensive. We introduce $h$-GPI, a multi-step extension of GPI that interpolates between these extremes---standard model-free GPI and fully model-based planning---as a function of a parameter, $h$, regulating the amount of time the agent has to reason. We prove that $h$-GPI's performance lower bound is strictly better than GPI's, and show that $h$-GPI generally outperforms GPI as $h$ increases. Furthermore, we prove that as $h$ increases, $h$-GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by $h$-GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. These bounds strictly generalize those known in the literature. We evaluate $h$-GPI on challenging tabular and continuous-state problems under value function approximation and show that it consistently outperforms GPI and state-of-the-art competing methods under various levels of approximation errors.

Multi-Step Generalized Policy Improvement by Leveraging Approximate Models

Abstract

We introduce a principled method for performing zero-shot transfer in reinforcement learning (RL) by exploiting approximate models of the environment. Zeroshot transfer in RL has been investigated by leveraging methods rooted in generalized policy improvement (GPI) and successor features (SFs). Although computationally efficient, these methods are model-free: they analyze a library of policies-each solving a particular task-and identify which action the agent should take. We investigate the more general setting where, in addition to a library of policies, the agent has access to an approximate environment model. Even though model-based RL algorithms can identify near-optimal policies, they are typically computationally intensive. We introduce h -GPI, a multi-step extension of GPI that interpolates between these extremes-standard model-free GPI and fully model-based planning-as a function of a parameter, h , regulating the amount of time the agent has to reason. We prove that h -GPI's performance lower bound is strictly better than GPI's, and show that h -GPI generally outperforms GPI as h increases. Furthermore, we prove that as h increases, h -GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by h -GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. These bounds strictly generalize those known in the literature. We evaluate h -GPI on challenging tabular and continuous-state problems under value function approximation and show that it consistently outperforms GPI and state-of-the-art competing methods under various levels of approximation errors.

1 Introduction

Reinforcement learning (RL) (Sutton and Barto, 2018) algorithms have achieved remarkable performance in challenging tasks both in the model-free (Bellemare et al., 2020; Wurman et al., 2022) and the model-based (Bryant et al., 2022; Wu et al., 2022) settings. In these problems, agents are typically trained to optimize one particular reward function (a task ). However, designing agents that can adapt their decision-making policies to solve novel tasks in a zero-shot manner-i.e., without requiring any further learning-remains an important open problem.

Various principled and efficient policy transfer methods, which combine previously-acquired policies to solve new tasks in a zero-shot manner, have been proposed based on the combination of successor features (SFs) and generalized policy improvement (GPI) (Barreto et al., 2017; Borsa et al., 2019; Barreto et al., 2020; Kim et al., 2022). On the one hand, SFs allow agents to efficiently evaluate the performance of given policies in arbitrary sets of tasks. On the other hand, GPI extends the classic policy improvement procedure (Sutton and Barto, 2018) by analyzing a library of previously-learned

policies-each solving a particular task-and identifying a novel policy that simultaneously improves upon all policies in the agent's library.

We introduce h -GPI, a multi-step extension of GPI that interpolates between these extremes-standard model-free GPI and fully modelbased planning--as a function of a parameter, h , regulating the amount of time the agent has to reason. An h -GPI policy is computed by performing online planning for h steps, using an approximate model, and then employing GPI on all states reachable in h steps to identify (in a zero-shot manner) which actions the agent should take thereafter. See Figure 1 for an example.

⋮

GPI

bootstrapping

GPI

bootstrapping

GPI

bootstrapping

planning steps with

approximate model

h

zero-shot GPI in states

reachable in

steps

h

S

t

+

h

⋮

⋮

S

t

S

t

+

h

S

t

+

h

̂

Our work is partially motivated by the observation that state-of-the-art model-based RL algorithms (Hafner et al., 2019, 2021) often adapt poorly to local changes in the environment (Wan et al., 2022). Furthermore, environment models may have been learned and thus might be inaccurate. These observations underscore the risks involved in relying solely on approximate models for long-horizon planning scenarios, as model errors compound and may lead to catastrophic outcomes. Unlike techniques for dealing with approximate models, our method additionally exploits GPI's capabilities of efficiently performing zero-shot transfer in a model-free manner. Other approaches-similar in nature to h -GPI-make use of models and GPI-based techniques. Thakoor et al. (2022), for example, introduced GGPI, a method that learns state visitation models induced by particular policies to rapidly evaluate policies under a given, known reward function. The authors show that performing GPI over a particular type of non-stationary policy produces behaviors that outperform those in the agent's library policy. h -GPI, by contrast, learns a different type of model: an environment model, which is used to perform planning-i.e., action selection-rather than policy evaluation. Moreover, h -GPI exploits GPI to perform bootstrapping from all states reachable in h steps. In Section 5, we contrast our method with other related techniques, highlighting the unique aspects of our approach.

In this paper, we introduce the first principled method capable of exploiting approximate models and producing a multi-step extension of GPI with formal guarantees. We prove that h -GPI's performance lower bound is strictly better than GPI's, and show that h -GPI generally outperforms GPI as h increases. Furthermore, we prove that as h increases, h -GPI's performance becomes arbitrarily less susceptible to value function approximation errors in the agent's policy library. Finally, we introduce novel bounds characterizing the gains achievable by h -GPI as a function of approximation errors in both the agent's policy library and its (possibly learned) model. We empirically evaluate h -GPI in challenging tabular and continuous-state problems with value function approximation. Our findings show that it consistently outperforms both GPI and other state-of-the-art methods under various levels of approximation errors. These results, combined with our method's formal guarantees, indicate that h -GPI is an important first step towards bridging the gap between model-free GPI-based methods and fully model-based planning algorithms-while being robust to approximate models of the environment and value function estimation errors.

2 Background

Before introducing our contributions, we review key concepts and definitions related to model-free and model-based RL, SFs, and GPI.

2.1 Reinforcement learning

RL problems (Sutton and Barto, 2018) are typically modeled as Markov decision processes (MDPs). An MDP is a tuple M ≜ ( S , A , p, r, µ, γ ) , where S is a state space, A is an action space, p ( ·| s, a ) denotes the distribution over next states conditioned on a state and action, r : S × A × S ↦→ R is a reward function, µ is an initial state distribution, and γ ∈ [0 , 1) is a discount factor. Let S t , A t , and R t = r ( S t , A t , S t +1 ) be random variables corresponding to the state, action, and reward, respectively,

a

]

at time step t ≥ 0 . The goal of an RL agent is to learn a policy π : S ↦→ A that maximizes the expected discounted sum of rewards ( return ) ∑ ∞ t =0 γ t R t . The action-value function of a policy π is q π ( s, a ) ≜ E π [ ∑ ∞ i =0 γ i R t + i | S t = s, A t = a ] , where E π [ · ] denotes the expectation over trajectories induced by π and p . Given q π , a greedy policy can be defined as π ′ ( s ) ∈ arg max a q π ( s, a ) . It is guaranteed that q π ′ ( s, a ) ≥ q π ( s, a ) , for all ( s, a ) ∈ S × A . Computing q π and π ′ is done by processes known, respectively, as policy evaluation and policy improvement . Repeatedly alternating between policy evaluation and policy improvement steps is known to lead to an optimal policy, π ∗ ( s ) , which maximizes the expected return from all states s ∈ S (Puterman, 2014).

2.2 Successor features and GPI

The successor features (SFs) framework (Barreto et al., 2017) allows agents to efficiently evaluate the performance of a given policy when deployed under any linearly-representable reward functions, r w ( s, a, s ′ ) = ϕ ( s, a, s ′ ) · w , where ϕ ( s, a, s ′ ) ∈ R d are reward features and w ∈ R d are weights. Acontrolled Markov process ( S , A , p, µ, γ ) , i.e., an MDP without a reward function, when combined with reward features ϕ : S × A × S ↦→ R d , induces a family of MDPs:

Notice that the family of MDPs M ϕ represents all possible tasks (each defined by a reward function) that can be defined in the environment associated with the corresponding controlled Markov process. Given a policy π , its corresponding SFs, ψ π ( s, a ) ∈ R d , for a state-action pair ( s, a ) are defined as

where ϕ t ≜ ϕ ( S t , A t , S t +1 ) . Importantly, notice that given the SFs ψ π ( s, a ) of a policy π , it is possible to directly compute the action-value function q π w ( s, a ) of π , under any linearly-expressible reward functions, r w , as follows: 1

The equation described above-which uses SFs to evaluate a given policy under different reward functions-represents a process known as generalized policy evaluation (GPE) (Barreto et al., 2020). Notice that this process can be extended to the case where an agent has access to a set of previouslylearned policies, Π = { π i } n i =1 , and their corresponding SFs, Ψ = { ψ π i } n i =1 . Then, given any π i ∈ Π , GPE can be used to efficiently evaluate π i under any arbitrary reward function of interest, r w , via Equation (3): q π i w ( s, a ) = ψ π i ( s, a ) · w .

Generalized policy improvement. GPI generalizes the standard policy improvement process (discussed in Section 2.1). It assumes access to a set of policies (and corresponding action-value functions) and uses it to directly identify a novel policy optimizing a particular reward function, r w . Importantly, the novel policy is guaranteed to outperform all original policies the agent had access to.

Definition 1. (Barreto et al., 2020) Given a set of policies Π and a reward function r w , generalized policy improvement (GPI) is the process by which a policy, π ′ , is identified such that

Based on Equation (4) and the reward decomposition r w ( s, a, s ′ ) = ϕ ( s, a, s ′ ) · w , a generalized policy , π : S × W ↦→ A , can then be defined as follows:

Let q GPI w ( s, a ) be the action-value function of policy π GPI ( · ; w ) . The GPI theorem (Barreto et al., 2017) shows that π GPI ( · ; w ) satisfies Definition 1, and so Equation (5) can be used to identify a policy guaranteed to perform at least as well as any other policies π i ∈ Π , when tackling a new task, w . This theorem can be extended to cases where the agent only has access to an approximation of the action-value function, ˆ q π i , of a policy π i (Barreto et al., 2018).

2.3 Model-based RL

In model-free RL, agents do not have access to the environment state transition and reward functions, p and r , and must, instead, learn a policy solely based on samples obtained while interacting with the environment. Given an MDP, we denote m ≜ ( p, r ) as its model . In model-based RL, an agent is given (or has to learn) an approximate model, ˆ m ≈ m . Agents may exploit models in different ways; for example, (i) to perform background planning (i.e., generating simulated experiences to more rapidly learn a policy or value function, as in Dyna-style algorithms (Van Seijen and Sutton, 2013; Janner et al., 2019; Pan et al., 2019)); and (ii) to perform online planning (Barto et al., 1995; Chua et al., 2018; Efroni et al., 2020; Hansen et al., 2022).

In online planning methods, agents use a model to estimate the value of states by simulating the outcomes of a particular sequence of actions. This is also known as 'unrolling the model forwards', from the current state, for a given number of time steps. Typically, online planning procedures are performed from every state the agent encounters, as in model-predictive control (MPC) algorithms (Camacho and Alba, 2013; Chua et al., 2018). Importantly, recent work-such as that of Wan et al. (2022)-has shown that state-of-the-art model-based RL algorithms (Hafner et al., 2019, 2021) often fail catastrophically when deployed in settings where the environment (e.g., the MDP's reward function) may change. This observation underscores the risks involved in relying solely on approximate models for long-horizon planning scenarios, since it is well-known that model errors compound.

We address these limitations by introducing a zero-shot transfer technique based on online planning. Our technique interpolates between standard model-free GPI and fully model-based planning according to a parameter, h , regulating the amount of time the agent has to reason using an approximate model. Unlike previous methods that require re-learning or adapting models to tackle novel tasks, our technique exploits approximate models and SFs to solve any tasks (defined as linearly-expressible reward functions) without requiring any further learning . Importantly, it has strong theoretical guarantees regarding its performance lower bound under various types of approximation errors.

3 Multi-step generalized policy improvement

We now introduce our main contribution, h -GPI. This is a multi-step extension of GPI that combines online planning with approximate environment models to efficiently perform zero-shot transfer.

We start by defining the Bellman optimality operator T ∗ (and its multi-step variant) applied to actionvalue functions q ∈ Q , where Q is the space of action-value functions. To simplify notation, given a model m ≜ ( p, r ) , we denote E m [ · ] as the expectation operator with respect to S t +1 ∼ p ( . | S t , A t ) and R t ≜ r ( S t , A t , S t +1 ) . Similarly, we write r ( s, a ) ≜ E S t +1 ∼ p ( ·| s,a ) [ r ( s, a, S t +1 )] for brevity.

Definition 2. (Single- and multi-step Bellman optimality operators). Given a model m = ( p, r ) , the single-step Bellman optimality operator, T ∗ : Q ↦→ Q , is defined as:

The repeated application of T ∗ for h steps gives rise to the h -step Bellman operator, denoted ( T ∗ ) h q ( s, a ) ≜ T ∗ · · · T ∗ h -times q ( s, a ) . Efroni et al. (2018) showed that ( T ∗ ) h can be written as follows:

where µ i is any policy (in an arbitrary policy space) the agent could choose to deploy at time i .

It is well known that T ∗ is a contraction mapping and its fixed-point (for any initial q ∈ Q ) is lim h →∞ ( T ∗ ) h q ( s, a ) = q ∗ ( s, a ) (Puterman, 2014).

We now introduce our first contribution: h -step generalized policy improvement , or h -GPI:

Definition 3. Let Π = { π i } n i =1 be a set of policies and m = ( p, r ) be a model. Then, given a horizon h ≥ 0 , the h -GPI policy on state s is defined as

a

∈A

µ

1

,...,µ

h

-

1

E

m

[

h

-

1

∑

k

=0

γ

k

r

(

S

t

+

k

, µ

k

(

S

t

+

k

))

online planning

+

γ

h

a

′

∈A

π

∈

Π

q

π

(

S

t

+

h

, a

′

)

GPI

|

µ

0

(

S

t

) =

a

]

(10)

where µ i is any policy (in an arbitrary policy space) the agent could choose to deploy at time i .

Intuitively, h -GPI identifies a policy that returns the best possible action, a , by first planning with model m for h steps and then using GPI to estimate the future returns achievable from all states reachable in h steps. This is in contrast with standard GPI policies, which can only reason about the future returns achievable from the current state, S t , if following a given policy in Π . h -GPI, by contrast, uses a model to reason over the decisions made at states within h steps from s , as well as states, S t + h , reachable after h steps. This makes it possible for h -GPI to produce policies that exploit the model-free, zero-shot return estimates produced by GPI when evaluating states beyond a given horizon h ; in particular, states that would otherwise not be considered by the standard GPI procedure when determining return-maximizing actions. Notice, furthermore, that h -GPI by construction interpolates between model-free GPI and fully model-based planning:

h -GPI induces policies that interpolate between (i) standard model-free GPI (when h = 0 ) and (ii) fully model-based planning (when h →∞ ).

In practice, agents seldom have access to the true model, m , of an MDP, and instead rely on approximate models, ˆ m . The application of the Bellman operator under an approximate model, ˆ m , rather than the true MDP model, m , can be represented mathematically by replacing m with ˆ m in Equation (6). This gives rise to a model-dependent Bellman optimality operator, which we call T ∗ ˆ m . In what follows, we introduce Theorem 1, which extends the original GPI theorem (Barreto et al., 2017) and allows us to characterize the gains achievable by h -GPI as a function of approximation errors in both the agent's policy library ( Π ) and its (possibly learned) model ˆ m . Notice that Theorem 1 generalizes the original GPI theorem, which is recovered when h = 0 .

Theorem 1. Let Π = { π i } n i =1 be a set of policies, { ˆ q π i } n i =1 be approximations of their respective action-value functions, and ˆ m = (ˆ p, ˆ r ) be an approximate model such that, for all π i ∈ Π and all ( s, a ) ∈ S × A ,

Recall once again the definition of h -GPI (Definition 3):

Then,

where c ( ϵ r , ϵ p , h ) = 1 -γ h 1 -γ ( ϵ r + γϵ p v ∗ max ) , and v ∗ max ≜ | max s,a q ∗ ( s, a ) | .

Theorem 1 characterizes the performance lower bound of h -GPI as a function of the number of planning steps, h , and the approximation errors in the agent's model and action-value functions (i.e., errors ϵ , ϵ p , and ϵ r ).

Note that since ( T ∗ ) h max π ∈ Π q π ( s, a ) ≥ max π ∈ Π q π ( s, a ) , h -GPI's performance lower bound is strictly better than GPI's ( h = 0 ) assuming no model approximation errors ( ϵ r = ϵ p = 0 ). Furthermore, notice that as h →∞ , ( T ∗ ) h max π ∈ Π q π ( s, a ) converges to q ∗ and the approximation error term ( γ h ϵ ) in Equation (13) disappears. In other words, as h increases, h -GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. This implies that the planning horizon h trades off between two conflicting objectives. On the one hand, increasing the planning horizon h improves the performance of the h -GPI policy because the error term ( ϵ ) associated with value function approximations errors becomes irrelevant. On the other hand, longer horizons worsen h -GPI's performance lower bound since they increase its dependency on errors arising from approximate models. Intuitively, for small values of h , the agent relies more heavily on the assumption that its estimates { ˆ q π i } n i =1 are correct, given that these estimates are discounted by γ h . For large values of h , by contrast, the approximate model error term, c ( ϵ r , ϵ p , h ) , increases, and the agent relies more heavily on the assumption that its (learned) model is approximately correct.

Next, we introduce a novel upper-bound on the discrepancy between the value, q h -GPI w , of the h -GPI policy under a particular reward function, and the optimal action-value function, q ∗ w . Notice that, as in Theorem 1, we can recover the standard GPI bound when setting h = 0 . 2

Theorem 2. Let Π = { π ∗ i } n i =1 be a set of optimal policies with respect to reward weights { w i } n i =1 and w be arbitrary reward weights. Let ˆ m = (ˆ p, ˆ r w ) be an approximate model and { ˆ q π ∗ i w } n i =1 be approximations to the action-value functions of policies in Π , under the reward function r w , such that for all π i ∈ Π and all ( s, a ) ∈ S × A ,

We now extend the definition of h -GPI (Definition 3) to the case where this policy is defined under the assumption of an MPD with reward function r w :

Let ϕ max ≜ max s,a || ϕ ( s, a ) || . Then, it follows that

where c ( ϵ r , ϵ p , h ) = 1 -γ h 1 -γ ( ϵ r + γϵ p v ∗ max ) .

Theorem 2 precisely characterizes the optimality gap (i.e., the maximum difference) between the action-value function induced by h -GPI ( q h -GPI w ( s, a )) and the optimal action-value function ( q ∗ w ( s, a )) , as a function of (i) the reward weights { w i } n i =1 for which the policies in the agent's library, Π , are optimal; (ii) approximation errors in action-value functions { ˆ q π ∗ i w } n i =1 ; and (iii) approximation errors in the model ˆ m = (ˆ p, ˆ r w ) .

3.1 h -GPI under (learned) approximate models for zero-shot transfer

Recall that our goal is to perform zero-shot policy transfer to solve any tasks M ∈ M ϕ (Equation (1)). In order to employ h -GPI on any given reward function of the form r w ( S t , A t , S t +1 ) = ϕ t · w , we propose learning a model, ˆ m = (ˆ p, ˆ ϕ ) , that approximates both the state transition function, p , and reward features , ϕ . The key insight is that this model implicitly induces the space of all MDP models m = ( p, r w ) , based on (arbitrary) reward weights w ∈ R d . This allows employing RL model-based methods to multi-task settings without the need to re-train the model when solving new tasks defined by different reward functions. In Algorithm 1, we present a high-level description of how the h -GPI policy π h -GPI ( s, w ) can be computed given a model ˆ m = (ˆ p, ˆ ϕ ) , SFs { ˆ ψ π i } n i =1 , and horizon h ≥ 0 .

Tabular setting. In the tabular case, models can be learned via maximum likelihood estimates: ˆ m ( s ′ , ϕ | s, a ) ≈ N ( s,a, ϕ ,s ′ ) N ( s,a ) , where N ( s, a, ϕ , s ′ ) is the number of times the agent experienced the

4 ∈ a ∈A T ˆ m i w

transition ( s, a, ϕ , s ′ ) and N ( s, a ) is the number of times the agent selected action a in state s . To efficiently compute the action given by the policy π h -GPI ( s, w ) (line 3 of Algorithm 1), we extend the Forward-Backward Dynamic Programming (FB-DP) algorithm (Efroni et al., 2020) to the discountedSFs setting, in order to compute h -lookahead policies in real-time. The corresponding pseudocode can be found in Appendix B. Notably, given a state s , FB-DP time complexity is O ( N |A||S tot h | ) , where N is the maximal number of accessible states in one step (maximal 'nearest neighbors' from any state), and S tot h is total reachable states in h time steps from state s . Note that while we chose to extend FB-DP due to its efficiency and closed-form formulation, other planning techniques could have been used (e.g., Monte Carlo tree search (Tesauro and Galperin, 1996; Silver et al., 2017)).

Continuous-states setting. In the continuous-state setting, we extend the class of models composed of ensembles of probabilistic neural networks (Chua et al., 2018) to the SFs setting. These models are used in state-of-the-art single-task model-based RL algorithms (Janner et al., 2019; Yu et al., 2021). The learned model ˆ m φ ( s ′ , ϕ | s, a ) , parameterized by φ , is composed of an ensemble of n neural networks, { ˆ m φ i } n i =1 , each of which outputs the mean and diagonal covariance matrix of a multivariate Gaussian distribution: m φ i ( S t +1 , ϕ t | S t , A t ) = N ( µ φ i ( S t , A t ) , Σ φ i ( S t , A t )) . Each model in the ensemble is trained in parallel to minimize the following negative log-likelihood loss function, using different bootstraps of experiences in a buffer B = { ( S t , A t , ϕ t , S t +1 ) | t ≥ 0 } : L ( φ ) = E ( S t ,A t , ϕ t ,S t +1 ) ∼B [ -log m φ ( S t +1 , ϕ t | S t , A t )] . In practice, we use as B the buffer with experiences the agent collected while training the SFs for the training tasks. In order to compute the h -GPI policy in this setting, we approximate the expectation over next states by averaging over the predictions of the components of the ensemble (Chua et al., 2018).

We employ universal successor features approximators (USFAs) (Borsa et al., 2019) to learn SFs in the function approximation setting. Given sufficiently expressive USFAs, we can evaluate the value function of any policy π z (optimal for task z ) in any task w ∈ R d by simply computing ˆ q π z w ( s, a ) ≈ ˆ ψ ( s, a, z ) · w . Furthermore, when using a USFA to generalize to a new task w ′ , GPI (Equation 5) becomes: π GPI ( s ; w ) ∈ arg max a ∈A max w ∈M ˆ ψ ( s, a, w ) · w ′ , where M is typically a set of weight vectors used when training the USFA. Notably, we only require a single USFA ˆ ψ and a single model ˆ m to perform h -GPI given any M ∈ M ϕ , in contrast to other approaches that require learning a complex model for each policy in the library (Thakoor et al., 2022).

Notice that although we assume that during training the agent can observe and add to its buffer B the features ϕ t at each time step t , our method could also be applied to the setting where the features are learned beforehand in a pre-training phase (Hansen et al., 2020; Carvalho et al., 2023). Learning appropriate reward features ϕ is an important but orthogonal problem to the one tackled in this paper.

4 Experiments

We conduct tabular and deep RL experiments in three different domains to evaluate the effectiveness of h -GPI as a method for zero-shot policy transfer.

Environments. First, we consider the tabular FourRoom domain (Barreto et al., 2017). To make this domain more challenging, we made its transition function stochastic by including a 10% chance of the agent moving to a random direction after taking any action. In FourRoom, the reward features ϕ t ∈ R 3 correspond to binary vectors indicating whether the agent collected each of the three types of objects in the map. The second domain is Reacher (Alegre et al., 2022a), which consists of a robotic arm that can apply torque to each of its two joints. The features ϕ t ∈ R 4 are proportional

0

.

64

0

.

72

0

.

80

GPI

10-MPC

15-MPC

20-MPC

1-GPI

2-GPI

4-GPI

6-GPI

8-GPI

10-GPI

Median

0

.

64

0

.

72

0

.

80

IQM

0

.

56

0

.

64

0

.

72

0

.

80

Mean

Normalized Return

to the distance of the tip of the arm to four different targets. Finally, we extend the FetchPush domain (Plappert et al., 2018), which consists of a fetch robotic arm that must move a block to a target position on top of a table by pushing the block with its gripper. The reward features ϕ t ∈ R 4 are the negative distances of the block to each of the four target locations. The action space was discretized in the same way as usually done in Reacher. Importantly, the state space of this domain is high-dimensional ( S ⊂ R 19 ) and its dynamics are significantly more complex than Reacher's. A more detailed description of the domains can be found in Appendix B.

Baselines. We compare h -GPI with standard GPI (Barreto et al., 2017), which is equivalent to h -GPI when h = 0 . To showcase the importance of combining model-based planning with model-free action-value estimates, we include an SF-based MPC approach, which is equivalent to h -GPI but without GPI bootstrapping at time step h . That is, given a task w , it selects the first action from a sequence of actions that maximizes E ˆ m [ ∑ h -1 k =0 γ k ˆ ϕ t + k · w ] , and re-plans at every time step. In the function approximation case with USFAs, we also compare with Constrained GPI (CGPI) (Kim et al., 2022). CGPI is a state-of-the-art method that computes lower and upper bounds for the optimal value on new tasks and uses them to constrain value approximation errors when following the GPI policy.

0

1

2

3

4

5

6

7

8

9

10

h

0

.

55

0

.

60

0

.

65

0

.

70

0

.

75

0

.

80

0

.

85

0

.

90

Mean Normalized Return

h

-GPI

h

-GPI with

/epsilon1

∼ N

(0

,

0

.

05

2

)

h

-GPI with

/epsilon1

∼ N

(0

,

0

.

1

2

)

h

-GPI with

/epsilon1

∼ N

(0

,

0

.

2

2

)

SF-MPC (

h

= 10)

Based on the theoretical properties introduced in Section 3, we expect the following hypotheses to hold. H1: h -GPI performance better generalizes to unseen test tasks than standard GPI and, in general, improves as we increase h . H2: h -GPI is more robust to approximation errors in the learned SFs than GPI, and such errors have a decreasing impact as h increases. Additionally, we investigate whether H3: h -GPI can be scaled up to solve problems with high-dimensional state spaces. In each experiment, we report the median, interquartile mean (IQM), and mean normalized returns with their 95% stratified bootstrap confidence intervals (Agarwal et al., 2021), which are obtained by evaluating agents trained with 20 different random seeds on a set of test tasks. For fairness of comparison, we used the same agents for evaluating each method. We follow previous works (Borsa et al., 2019; Kim et al., 2022) and use as training tasks the weight vectors that form the standard basis of R d in all three domains. In FourRoom, we use 32 weights vectors equally spaced from the weight simplex { w | ∑ d i =1 w i = 1 , w i ≥ 0 } as test tasks. For Reacher and FetchPush, we follow Kim et al. (2022) and use weight vectors defined by {-1 , 1 } d as test tasks. Notice that this results in many tasks with negative weights, which are significantly different than the training tasks.

In Figure 2, we can observe that h -GPI is able to outperform GPI even with small planning horizons (see, e.g., 1 -GPI). As h increases, the performance on the test tasks consistently increases. This is in accordance with Theorem 1 and H1 . Notice that even with a small planning horizon of h = 2 , h -GPI is capable of matching the performance of SF-MPC even when SF-MPC is allowed to plan for ten times longer (i.e., h = 20 ).

Next, to investigate H2 , we evaluate h -GPI after artificially adding Gaussian noise with different standard deviations to the learned SFs of each of the agent's policies for every state and action. We can notice (in Figure 3) that standard GPI ( h = 0 ) catastrophically fails under all levels of

0

.

72

0

.

80

0

.

88

0

.

96

GPI-ST

CGPI

GPI

5-MPC

1-GPI

2-GPI

3-GPI

4-GPI

5-GPI

Median

0

.

80

0

.

88

0

.

96

IQM

0

.

72

0

.

80

0

.

88

Mean

Normalized Return

0

.

72

0

.

78

0

.

84

0

.

90

GPI-ST

CGPI

GPI

5-MPC

7-MPC

10-MPC

15-MPC

1-GPI

2-GPI

3-GPI

4-GPI

5-GPI

Median

0

.

80

0

.

84

0

.

88

IQM

0

.

72

0

.

78

0

.

84

0

.

90

Mean

Normalized Return

approximation errors. In all cases, h -GPI can closely approximate the same performance levels as those resulting from settings with no added noise. To do so, it suffices to use a sufficiently large h . In other words, h -GPI successfully and effectively identifies policies that are robust to noise due to its ability to combine model-free GPI generalization and planning under approximate/imperfect models for h steps. This is consistent with our theoretical results that show that value function approximation errors impact the performance's lower bound by a factor of γ h .

In Figures 4 and 5, we show the results in the Reacher and FetchPush domains, respectively. Given a test task w ′ , let GPI-ST be the policy that uses both the source/training (S) and target (T) tasks, M∪{ w ′ } , as input to the USFA when performing GPI. Similarly, we refer to GPI as the policy that performs GPI only over the training tasks M . The CGPI policy was unable to significantly improve performance over GPI-ST, which we hypothesize may be due to USFA errors in the training tasks, or due to overly small minimum rewards used in its upper-bound. Both are known limitations of CGPI (Kim et al., 2022). In both domains with function approximation using USFAs, h -GPI outperforms the competing methods, which supports H1 and H3 . In the FetchPush domain, in particular, SF-MPC is allowed to plan for up to h =15 steps; that is, three times longer than the maximum horizon we allow h -GPI to plan. Again, h -GPI consistently outperforms it (and other baselines) even when allowed to plan for significantly fewer steps. Interestingly, in the FetchPush domain, h -GPI's performance slightly decreases for intermediate values of h , which suggests that its policy is being affected by model approximation errors. Even so, the mean returns achieved, for all values of h are higher than the ones achieved by GPI. This demonstrates the robustness of h -GPI to approximate models of the environment and function approximation errors.

5 Discussion and related work

We now discuss the works most closely related to h -GPI. Further discussions can be found in Appendix C.

GPI and SFs. Previous works have extended GPI to safe RL (Gimelfarb et al., 2021; Feng et al., 2023), maximum-entropy RL (Hunt et al., 2019), unsupervised RL (Hansen et al., 2020), and hierarchical RL (e.g., via the options framework) (Barreto et al., 2019; Machado et al., 2023). Recently, Thakoor et al. (2022) introduced Geometric GPI (GGPI). GGPI uses geometric horizon models (GHMs) to learn the discounted future state-visitation distribution induced by particular policies to rapidly

evaluate policies under a given, known reward function. The authors show that performing GPI over a particular type of non-stationary policy produces behaviors that outperform those in the agent's library policy. h -GPI, by contrast, learns a different type of model: an environment model, which is used to perform planning-i.e., action selection-rather than policy evaluation. Additionally, GGPI requires learning separate GHMs for each policy in the library, whereas h -GPI can operate with a single model that predicts the next reward features, alongside a single USFA. Secondly, GGPI assumes that the reward function is known a priori , while we exploit SFs to generalize over all linear rewards given reward features. Bagot et al. (2023) introduced GPI-Tree Search (GPI-TS), which is closely related to h -GPI. GPI-TS uses GPI bootstrapping as backup value estimates at the leaf nodes of Monte Carlo tree search. However, GPI-TS does not employ SFs and was only designed to tackle single-task settings. Moreover, it assumes an oracle model of the environment. We, by contrast, exploit learned models to perform zero-shot transfer over multiple tasks and show how approximation errors in the model affect the performance of the h -GPI policy (see Theorems 1 and 2). Kim et al. (2022) introduced Constrained GPI (CGPI), which uses lower and upper bounds of the value of a new task to constrain the action-value estimates used when selecting the GPI policy's action. Although CGPI is able to deal with generalization errors in the values given by USFAs for target tasks, it is sensitive to errors in the value estimates for the training tasks. h -GPI, by contrast, can deal with approximation errors in the value functions of the source/base policies. Other works have studied methods for constructing a policy library for use with GPI (Zahavy et al., 2021; Nemecek and Parr, 2021; Alver and Precup, 2022; Alegre et al., 2022a) and for learning different SF-based representations (Lehnert and Littman, 2020; Touati and Ollivier, 2021). These methods solve important orthogonal problems and could potentially be combined with h -GPI.

Multi-step RL algorithms. Multi-step RL methods were extensively studied for policy evaluation, both in the model-free (Hessel et al., 2018; van Hasselt et al., 2018) and the model-based settings (Yao et al., 2009; Janner et al., 2019). Model value expansion algorithms (Feinberg et al., 2018; Buckman et al., 2018; Abbas et al., 2020) are a significant example of the latter. In this work, by contrast, we introduce a multi-step method for policy improvement in transfer learning settings. GX-Chen et al. (2022) introduced the η -return mixture, a new backup target for better credit assignment, which combines bootstrapping with standard value estimates and SFs, as a function of a parameter η . Efroni et al. (2018) studied multi-step greedy versions of the well-known dynamic programming (DP) policy iteration and value iteration algorithms (Bertsekas and Tsitsiklis, 1996), and Efroni et al. (2020) proposed a multi-step greedy real-time DP algorithm that replaces 1-step greedy policies with a h -step lookahead policy. Our work is also related to the techniques introduced by Sikchi et al. (2022) and Hansen et al. (2022), which combine planning and bootstrapping with learned value estimates. However, unlike h -GPI, these methods do not address the multi-policy and zero-shot transfer settings. We also note that there exists neuroscientific evidence that multi-step planning occurs in the brains of humans and other animals (Miller and Venditto, 2021). Finally, we believe that our work also shares similarities with the investigation performed by Tomov et al. (2021) on how model-free and model-based mechanisms are combined in human behavior.

6 Conclusions

We introduced h -GPI, a multi-step extension of GPI that interpolates between standard model-free GPI and fully model-based planning. Through novel theoretical and empirical results, we showed that h -GPI effectively exploits approximate models to solve novel tasks in a zero-shot manner. Notably, in our experiments, and consistent with the introduced theorems, we showed that h -GPI is less susceptible to value approximation errors and that it outperforms standard GPI and state-of-the-art competing baselines. Our findings hold even in high-dimensional problems where imperfect learned models are used. These results, combined with our method's strong formal guarantees, indicate that h -GPI is an important first step towards bridging the gap between model-free GPI-based methods and model-based planning algorithms-while also being robust to approximation errors.

Limitations and future work. The main limitation of h -GPI is that it can introduce computational overhead for large values of h . In future work, this can be tackled by designing heuristics for use when performing online planning, or via principled methods to dynamically select the best value of h . Another interesting direction is to integrate uncertainty estimation techniques (e.g., implicit value ensemble (Filos et al., 2022)) into the h -GPI policy to further reduce the impact of high model or value function approximation errors. Finally, regarding potential direct negative societal impacts of this work, we do not anticipate any.

Acknowledgments and disclosure of funding

This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001; CNPq (Grants 140500/2021-9, 304932/2021-3); FAPESP/MCTI/CGI (Grant 2020/05165-1); the Research Foundation Flanders (FWO) [G062819N]; the AI Research Program from the Flemish Government (Belgium); and the Francqui Foundation.

References

Abbas, Z., Sokota, S., Talvitie, E., and White, M. (2020). Selective dyna-style planning under limited model capacity. In III, H. D. and Singh, A., editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 1-10. PMLR. 5, C

Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G. (2021). Deep reinforcement learning at the edge of the statistical precipice. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems 34 (NeurIPS 2021) . 4

Alegre, L. N., Bazzan, A. L. C., and da Silva, B. C. (2022a). Optimistic linear support and successor features as a basis for optimal policy transfer. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 394-413. PMLR. 4, 5, B.2, B.3, C

Alegre, L. N., Felten, F., Talbi, E.-G., Danoy, G., Nowé, A., Bazzan, A. L. C., and da Silva, B. C. (2022b). MO-Gym: A library of multi-objective reinforcement learning environments. In Proceedings of the 34th Benelux Conference on Artificial Intelligence BNAIC/Benelearn 2022 . B.2, B.2

Alver, S. and Precup, D. (2022). Constructing a good behavior basis for transfer using generalized policy updates. In Proceedings of the Tenth International Conference on Learning Representations . 5, C

Bagot, L., D'eer, L., Latré, S., Schepper, T. D., and Mets, K. (2023). GPI-Tree Search: Algorithms for decisiontime planning with the general policy improvement theorem. In Proceedings of the Adaptive and Learning Agents Workshop (ALA 2023) . 5, C

Barreto, A., Borsa, D., Hou, S., Comanici, G., Aygün, E., Hamel, P., Toyama, D., Hunt, J., Mourad, S., Silver, D., and Precup, D. (2019). The option keyboard combining skills in reinforcement learning. In Proceedings of the 33rd International Conference on Neural Information Processing Systems , Red Hook, NY, USA. Curran Associates Inc. 5, C

Barreto, A., Borsa, D., Quan, J., Schaul, T., Silver, D., Hessel, M., Mankowitz, D., Zidek, A., and Munos, R. (2018). Transfer in deep reinforcement learning using successor features and generalised policy improvement. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 501-510, Stockholm, Sweden. 2.2

Barreto, A., Dabney, W., Munos, R., Hunt, J. J., Schaul, T., van Hasselt, H. P., and Silver, D. (2017). Successor features for transfer in reinforcement learning. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc. 1, 2.2, 2.2, 3, 4, A, A, A, B.2, B.2

Barreto, A., Hou, S., Borsa, D., Silver, D., and Precup, D. (2020). Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences , 117(48):30079-30087. 1, 2.2, 1

Barto, A. G., Bradtke, S. J., and Singh, S. P. (1995). Learning to act using real-time dynamic programming. Artificial Intelligence , 72(1):81-138. 2.3

Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. (2020). Autonomous navigation of stratospheric balloons using reinforcement learning. Nature , 588(7836):7782. 1

Bertsekas, D. and Tsitsiklis, J. (1996). Neuro-dynamic programming . Athena Scientific, Belmont, MA. 5, C

Borsa, D., Barreto, A., Quan, J., Mankowitz, D. J., Munos, R., Hasselt, H. V., Silver, D., and Schaul, T. (2019). Universal successor features approximators. In Proceedings of the 7th International Conference on Learning Representations (ICLR) . 1, 4, 4, B.3

Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., Paszke, A., VanderPlas, J., Wanderman-Milne, S., and Zhang, Q. (2018). JAX: composable transformations of Python+NumPy programs. B.3

Bryant, P., Pozzati, G., Zhu, W., Shenoy, A., Kundrotas, P., and Elofsson, A. (2022). Predicting the structure of large protein complexes using alphafold and monte carlo tree search. Nature Communications , 13(1):1-14. 1

Buckman, J., Hafner, D., Tucker, G., Brevdo, E., and Lee, H. (2018). Sample-efficient reinforcement learning with stochastic ensemble value expansion. In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., CesaBianchi, N., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc. 5, C

Camacho, E. and Alba, C. (2013). Model Predictive Control . Advanced Textbooks in Control and Signal Processing. Springer. 2.3

Carvalho, W. T., Filos, A., Lewis, R., Lee, H., and Singh, S. (2023). Composing task knowledge with modular successor feature approximators. In Proceedings of The Eleventh International Conference on Learning Representations . 4

Chua, K., Calandra, R., McAllister, R., and Levine, S. (2018). Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, pages 4759--4770, Red Hook, NY, USA. Curran Associates Inc. 2.3, 4, B.3

Dayan, P. (1993). Improving generalization for temporal difference learning: The successor representation. Neural Computation , 5(4):613-624. C

de Lazcano, R., Andreas, K., Tai, J. J., Lee, S. R., and Terry, J. (2023). Gymnasium robotics. B.2

Efroni, Y., Dalal, G., Scherrer, B., and Mannor, S. (2018). Beyond the one-step greedy approach in reinforcement learning. In Dy, J. G. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1386-1395. PMLR. 2, 5, A, C

Efroni, Y., Ghavamzadeh, M., and Mannor, S. (2020). Online planning with lookahead policies. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems 33 . 2.3, 4, 5, B.1, C

Feinberg, V., Wan, A., Stoica, I., Jordan, M. I., Gonzalez, J. E., and Levine, S. (2018). Model-based value estimation for efficient model-free reinforcement learning. CoRR , abs/1803.00101. 5, C

Feng, Z., Zhang, B., Bi, J., and Soh, H. (2023). Safety-constrained policy transfer with successor features. In 2023 IEEE International Conference on Robotics and Automation (ICRA) , pages 7219-7225. 5, C

Filos, A., Vértes, E., Marinho, Z., Farquhar, G., Borsa, D., Friesen, A., Behbahani, F., Schaul, T., Barreto, A., and Osindero, S. (2022). Model-value inconsistency as a signal for epistemic uncertainty. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 64746498. PMLR. 6, C

Fujimoto, S., Meger, D., and Precup, D. (2020). An equivalence between loss functions and non-uniform sampling in experience replay. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems , volume 33, pages 14219-14230. Curran Associates, Inc. B.3

Gimelfarb, M., Barreto, A., Sanner, S., and Lee, C.-G. (2021). Risk-aware transfer in reinforcement learning using successor features. In Proceedings of the 35th Annual Conference on Advances in Neural Information Processing Systems , Online. 5, B.2, B.2, C

GX-Chen, A., Chelu, V., Richards, B. A., and Pineau, J. (2022). A generalized bootstrap target for value-learning, efficiently combining value and feature predictions. In Proceedings of the Thirty-Sixth AAAI Conference on Artificial Intelligence , pages 6829-6837. AAAI Press. 5, C

Hafner, D., Lillicrap, T. P., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. (2019). Learning latent dynamics for planning from pixels. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2555-2565. PMLR. 1, 2.3

Hafner, D., Lillicrap, T. P., Norouzi, M., and Ba, J. (2021). Mastering atari with discrete world models. In Proceedings of The 9th International Conference on Learning Representations . OpenReview.net. 1, 2.3

Hansen, N. A., Su, H., and Wang, X. (2022). Temporal difference learning for model predictive control. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 8387-8406. PMLR. 2.3, 5, C

Hansen, S., Dabney, W., Barreto, A., Warde-Farley, D., de Wiele, T. V., and Mnih, V. (2020). Fast task inference with variational intrinsic successor features. In Proceedings of the 8th International Conference on Learning Representations . 4, 5, C

Hasselt, H. v., Guez, A., and Silver, D. (2016). Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence , AAAI'16, pages 2094--2100. AAAI Press. B.3

Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., and Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence , AAAI'18. AAAI Press. 5, C

Hunt, J., Barreto, A., Lillicrap, T., and Heess, N. (2019). Composing entropic policies using divergence correction. In Chaudhuri, K. and Salakhutdinov, R., editors, Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 2911-2920. PMLR. 5, C

Janner, M., Fu, J., Zhang, M., and Levine, S. (2019). When to trust your model: Model-based policy optimization. In Wallach, H., Larochelle, H., Beygelzimer, A., d' Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems (NIPS) 32 , pages 12519-12530. Curran Associates, Inc. 2.3, 4, 5, B.3, C

Kim, J., Park, S., and Kim, G. (2022). Constrained GPI for zero-shot transfer in reinforcement learning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems 35 . 1, 4, 4, 5, C

Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y., editors, Proceeding of the 3rd International Conference on Learning Representations (ICLR) , San Diego, CA. B.3

Lehnert, L. and Littman, M. L. (2020). Successor features combine elements of model-free and model-based reinforcement learning. Journal of Machine Learning Research , 21(196):1-53. 5, C

Machado, M. C., Barreto, A., and Precup, D. (2023). Temporal abstraction in reinforcement learning with the successor representation. Journal of Machine Learning Research (JMLR) , 24(80):1-69. 5, C

Miller, K. J. and Venditto, S. J. C. (2021). Multi-step planning in the brain. Current Opinion in Behavioral Sciences , 38:29-39. 5, C

Momennejad, I., Russek, E., Cheong, J. H., Botvinick, M., Daw, N., and Gershman, S. (2017). The successor representation in human reinforcement learning. Nature Human Behaviour , 1:680--692. C

Nemecek, M. and Parr, R. (2021). Policy caches with successor features. In Proceedings of the 38th International Conference on Machine Learning . 5, B.2, C

Pan, Y., Yao, H., Farahmand, A., and White, M. (2019). Hill climbing on value estimates for search-control in dyna. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 3209-3215, Macao, China. ijcai.org. 2.3

Plappert, M., Andrychowicz, M., Ray, A., McGrew, B., Baker, B., Powell, G., Schneider, J., Tobin, J., Chociej, M., Welinder, P., Kumar, V., and Zaremba, W. (2018). Multi-goal reinforcement learning: Challenging robotics environments and request for research. CoRR , abs/1802.09464. 4, B.2

Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons. 2.1, 3, A

Russek, E. M., Momennejad, I., Botvinick, M. M., Gershman, S. J., and Daw, N. D. (2017). Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLOS Computational Biology , 13(9):1-35. C

Schaul, T., Quan, J., Antonoglou, I., and Silver, D. (2016). Prioritized experience replay. In Proceedings of the 33rd International Conference on Learning Representations , Puerto Rico. B.3

Sikchi, H., Zhou, W., and Held, D. (2022). Learning off-policy with online planning. In Faust, A., Hsu, D., and Neumann, G., editors, Proceedings of the 5th Conference on Robot Learning , volume 164 of Proceedings of Machine Learning Research , pages 1622-1633. PMLR. 5, C

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D. (2017). Mastering the game of go without human knowledge. Nature , 550(7676):354-359. 4, B.1

Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction . The MIT Press, second edition. 1, 2.1

Tesauro, G. and Galperin, G. R. (1996). On-line policy improvement using monte-carlo search. In Mozer, M., Jordan, M. I., and Petsche, T., editors, Advances in Neural Information Processing Systems 9 , pages 1068-1074. MIT Press. 4, B.1

Thakoor, S., Rowland, M., Borsa, D., Dabney, W., Munos, R., and Barreto, A. (2022). Generalised policy improvement with geometric policy composition. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 21272-21307. PMLR. 1, 4, 5, C

Todorov, E., Erez, T., and Tassa, Y . (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033, Vilamoura, Portugal. B.2

Tomov, M., Schulz, E., and Gershman, S. (2021). Multi-task reinforcement learning in humans. Nature Human Behaviour , pages 1-10. 5, C

Touati, A. and Ollivier, Y. (2021). Learning one representation to optimize all rewards. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 13-23. Curran Associates, Inc. 5, C

van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. (2018). Deep reinforcement learning and the deadly triad. In Deep Reinforcement Learning Workshop, NeurIPS . 5, C

Van Seijen, H. and Sutton, R. S. (2013). Efficient planning in MDPs by small backups. In Proceedings of the 30th International Conference on International Conference on Machine Learning , ICML'13, pages III-361--III-369. JMLR.org. 2.3

Wan, Y., Rahimi-Kalahroudi, A., Rajendran, J., Momennejad, I., Chandar, S., and Van Seijen, H. H. (2022). Towards evaluating adaptivity of model-based reinforcement learning methods. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 22536-22561. PMLR. 1, 2.3

Wu, P., Escontrela, A., Hafner, D., Abbeel, P., and Goldberg, K. (2022). Daydreamer: World models for physical robot learning. In Liu, K., Kulic, D., and Ichnowski, J., editors, Proceedings of The 6th Conference on Robot Learning , volume 205 of Proceedings of Machine Learning Research , pages 2226-2240. PMLR. 1

Wurman, P. R., Barrett, S., Kawamoto, K., MacGlashan, J., Subramanian, K., Walsh, T. J., Capobianco, R., Devlic, A., Eckert, F., Fuchs, F., Gilpin, L., Khandelwal, P., Kompella, V., Lin, H., MacAlpine, P., Oller, D., Seno, T., Sherstan, C., Thomure, M. D., Aghabozorgi, H., Barrett, L., Douglas, R., Whitehead, D., Dürr, P., Stone, P., Spranger, M., and Kitano, H. (2022). Outracing champion gran turismo drivers with deep reinforcement learning. Nature , 602(7896):223-228. 1

Yao, H., Bhatnagar, S., Diao, D., Sutton, R. S., and Szepesvári, C. (2009). Multi-step dyna planning for policy evaluation and control. In Bengio, Y., Schuurmans, D., Lafferty, J., Williams, C., and Culotta, A., editors, Advances in Neural Information Processing Systems , volume 22. Curran Associates, Inc. 5, C

Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C. (2021). Combo: Conservative offline model-based policy optimization. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 28954-28967. Curran Associates, Inc. 4

Zahavy, T., Barreto, A., Mankowitz, D. J., Hou, S., O'Donoghue, B., Kemaev, I., and Singh, S. (2021). Discovering a set of policies for the worst case reward. In Proceedings of the 9th International Conference on Learning Representations . 5, C

A Proofs of theoretical results

We start by defining the Bellman optimality operator, T ∗ , and its multi-step variant, when applied to action-value functions q ∈ Q , where Q is the space of action-value functions. To simplify notation, given the (true) model, m ≜ ( p, r ) , of the underlying MDP, we denote E m [ · ] as the expectation operator with respect to S t +1 ∼ p ( . | S t , A t ) and R t ≜ r ( S t , A t , S t +1 ) . Similarly, we write r ( s, a ) ≜ E S t +1 ∼ p ( ·| s,a ) [ r ( s, a, S t +1 )] for brevity.

Definition (2) . (Single- and multi-step Bellman optimality operators). Given an MDP's model m = ( p, r ) , the single-step Bellman optimality operator, T ∗ : Q ↦→ Q , is defined as:

The repeated application of T ∗ for h steps gives rise to the h -step Bellman operator, denoted as ( T ∗ ) h q ( s, a ) ≜ T ∗ · · · T ∗ h -times q ( s, a ) . Efroni et al. (2018) showed that ( T ∗ ) h can be written as follows:

where µ i is any policy (in an arbitrary policy space) the agent could choose to deploy at time i . Definition 4. (Bellman evaluation operator). Given an MDP's model m = ( p, r ) and a policy π , the single-step Bellman evaluation operator, T π : Q ↦→ Q , is defined as:

It is well known that T ∗ and T π are contraction mappings and their fixed-point (for any initial q ∈ Q ) are lim h →∞ ( T ∗ ) h q ( s, a ) = q ∗ ( s, a ) and lim h →∞ ( T π ) h q ( s, a ) = q π ( s, a ) , respectively (Puterman, 2014).

Notice that in the definitions above, since m is the (true) model of a given MDP, we chose to omit m when writing the corresponding Bellman operators to make the notation more succinct. Formally, though, one could alternatively denote these operators as T ∗ m and T π m , respectively. In what follows, we always omit m whenever the corresponding operators are defined with respect to the true model of an MDP.

The Bellman optimality operator induced by an approximate model , ˆ m = (ˆ p, ˆ r ) , of the true underlying transition and reward functions, p and r , of a given MDP, is defined as:

Appendix

Now, recall our definition of h -step generalized policy improvement , or h -GPI:

Intuitively, an h -GPI policy identifies the best possible action, A t , when in state S t , by first planning with model m for h steps and then using GPI to estimate the future returns achievable from all states reachable in h steps. This is in contrast with standard GPI policies, which can only reason about the future returns achievable from the current state, S t , if following a given policy in Π . h -GPI, by contrast, uses a model to reason over the decisions made at states within h steps from S t , as well as states S t + h reachable after h steps. This makes it possible for h -GPI to produce policies that exploit the model-free, zero-shot return estimates produced by GPI when evaluating states beyond a given horizon h ; in particular, states that would otherwise not be considered by the standard GPI procedure when determining return-maximizing actions. Notice, furthermore, that h -GPI-by construction-interpolates between model-free GPI and fully model-based planning:

h -GPI induces policies that interpolate between (i) standard model-free GPI (when h = 0 ) and (ii) fully model-based planning (when h →∞ ).

Next, we introduce a few lemmas that will subsequently allow us to (i) precisely characterize the performance lower bound of an h -GPI policy; (ii) show that it is strictly better than GPI's performance lower bound; (iii) prove that as h increases, h -GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library.

The following lemma characterizes the loss introduced by applying the multi-step Bellman optimality operator, ( T ∗ ) h , to an approximate action-value function, ˆ q π ( s, a ) .

Lemma 1. Let Π = { π i } n i =1 be a set of policies, and { ˆ q π i } n i =1 be approximations to their respective action-value functions such that, for all π i ∈ Π and for all ( s, a ) ∈ S × A ,

Then,

Proof.

Above, (28) and (31) are due to the definition of ( T ∗ ) h introduced in Equation (20).

The next lemma characterizes the loss incurred by applying the Bellman optimality operator induced by an approximate model ˆ m , ( T ∗ ˆ m ) h , to an action-value function, q π , of a policy π .

Lemma 2. Let ˆ m = (ˆ p, ˆ r ) be an approximate model such that, for all ( s, a ) ∈ S × A ,

Let v ∗ max ≜ | max s,a q ∗ ( s, a ) | . Then, for any policy π and h ≥ 0 ,

where

Proof.

In the proof above, all terms shown in blue refer to approximate quantities-such as an approximate model of the environment or an approximate transition function. In the proof above, (35) follows from the recursive definition of ( T ∗ ) h showed in Equation (19). In (37), we add and subtract ∑ s ′ ˆ p ( s ′ | s, a ) max a ′ ∈A ( T ∗ ) h -1 q π ( s ′ , a ′ ) . (39) is obtained due to the property that ∑ s ′ ˆ p ( s ′ | s, a ) f ( s ′ ) ≤ max s ′ f ( s ′ ) , for any function f . Finally, (40) is due to ( T ∗ ) h -1 q π ( s ′ , a ′ ) ≤ v ∗ max .

We now show how to rewrite the inequality (40) recursively. Let ∆ h ( s, a ) ≜ | ( T ∗ ) h q π ( s, a ) -( T ∗ ˆ m ) h q π ( s, a ) | , for all ( s, a ) ∈ S × A . Then, by replacing the definition of ∆ h in (40), we obtain:

Because inequality (41) holds for any ( s, a ) , it also holds for the maximizer. Thus, we can omit ( s, a ) , and define ∆ h ≜ max s,a ∆ h ( s, a ) . The inequality (41) then becomes:

)

∣

∣

∣

∣

∣

Now we use the fact that, by definition, ∆ 0 ( s, a ) = | q π ( s, a ) -q π ( s, a ) | = 0 , to recursively expand inequality (42):

Finally, by using (44) and recalling the definition of ∆ h (i.e., ∆ h ≜ max s,a ∆ h ( s, a ) ), we can directly prove that

The result above bounds the difference between the action-value functions resulting from applying the Bellman optimality operator (defined with respect to the true model of the MDP) h times, and the action-value function resulting from applying the Bellman optimality operator (defined with respect to an approximate model, ˆ m ) h times. This allows us to precisely characterize-when repeatedly using the Bellman optimality operator over an initial action-value function-the maximum error incurred by using an approximate model of the environment.

The next two lemmas provide intermediate results that will be used in the proofs of Theorem 1 and Theorem 2.

Lemma 3. Let Π = { π i } n i =1 be a set of policies, { ˆ q π i } n i =1 be approximations to their respective action-value functions, and ˆ m = (ˆ p, ˆ r ) be an approximate model such that, for all π i ∈ Π and for all ( s, a ) ∈ S × A , the following holds:

Recall the definition of h -GPI (Definition 3):

Then, it follows that:

Proof.

Above, (49) is due to the definition of the Bellman evaluation operator (21), and (50) is due to the definition of the h -GPI policy (47). Finally, (51) is due to Lemma 1 and (52) follows from the use of Lemma 2.

Lemma 4. Let Π = { π i } n i =1 be a set of policies, { ˆ q π i } n i =1 be approximations to their respective action-value functions, and ˆ m = (ˆ p, ˆ r ) be an approximate model such that, for all π i ∈ Π , and for all ( s, a ) ∈ S × A :

Recall the definition of h -GPI (Definition 3):

Then,

Proof.

Above, (58) is by Lemma 3, (59) is due to the monotonicity of T ∗ (i.e. T ∗ q π ( s, a ) ≥ q π ( s, a ) ), and (60) is by Lemma 2.

We now use Lemmas 1, 2, and 4 to introduce Theorem 1, which extends the original GPI theorem (Barreto et al., 2017) and allows us to characterize the gains achievable by h -GPI as a function of approximation errors in both the agent's policy library, Π , and its (possibly learned) model, ˆ m . Notice that Theorem 1 generalizes the original GPI theorem, which is recovered when h = 0 .

Theorem (1) . Let Π = { π i } n i =1 be a set of policies, { ˆ q π i } n i =1 be approximations of their respective action-value functions, and ˆ m = (ˆ p, ˆ r ) be an approximate model such that (as previously-assumed in Lemmas 2 and 3), for all π i ∈ Π and all ( s, a ) ∈ S × A ,

Recall once again the definition of h -GPI (Definition 3):

Then,

where c ( ϵ r , ϵ p , h ) = 1 -γ h 1 -γ ( ϵ r + γϵ p v ∗ max ) .

Proof.

Above, (67) is due to the fixed-point of lim k →∞ ( T π h -GPI ) k q ( s, a ) being q h -GPI ( s, a ) for any q . (68) is due to the repeated application of Lemma 4. Finally, (70) is due to Lemma 1, and (71) is due to Lemma 2.

Theorem 1 characterizes the performance lower bound of h -GPI as a function of the number of planning steps, h , and the approximation errors in the agent's model and action-value functions (i.e., errors ϵ , ϵ p , and ϵ r ).

Note that since ( T ∗ ) h max π ∈ Π q π ( s, a ) ≥ max π ∈ Π q π ( s, a ) , h -GPI's performance lower bound is strictly better than GPI's ( h = 0 ) assuming no model approximation errors ( ϵ r = ϵ p = 0 ). Furthermore, notice that as h →∞ , ( T ∗ ) h max π ∈ Π q π ( s, a ) converges to q ∗ and the approximation error term ( γ h ϵ ) in Equation (13) disappears. In other words, as h increases, h -GPI's performance becomes arbitrarily less susceptible to sub-optimality in the agent's policy library. This implies that the planning horizon h trades off between two conflicting objectives. On the one hand, increasing the planning horizon h improves the performance of the h -GPI policy because the error term ( ϵ ) associated with value function approximations errors becomes irrelevant. On the other hand, longer horizons worsen h -GPI's performance lower bound since they increase its dependency on errors arising from approximate models. Intuitively, for small values of h , the agent relies more heavily on the assumption that its estimates { ˆ q π i } n i =1 are correct, given that these estimates are discounted by γ h . For large values of h , by contrast, the approximate model error term, c ( ϵ r , ϵ p , h ) , increases, and the agent relies more heavily on the assumption that its (learned) model is approximately correct.

Theorem (2) . Let Π = { π ∗ i } n i =1 be a set of optimal policies with respect to reward weights { w i } n i =1 and w be arbitrary reward weights. Let ˆ m = (ˆ p, ˆ r w ) be an approximate model and { ˆ q π ∗ i w } n i =1 be approximations to the action-value functions of policies in Π , under the reward function r w , such that for all π i ∈ Π and all ( s, a ) ∈ S × A ,

We now extend the definition of h -GPI (Definition 3) to the case where this policy is defined under the assumption of an MPD with reward function r w :

Let ϕ max ≜ max s,a || ϕ ( s, a ) || . Then, it follows that

where c ( ϵ r , ϵ p , h ) = 1 -γ h 1 -γ ( ϵ r + γϵ p v ∗ max ) .

Proof. The proof is an extension of Theorem 2 of Barreto et al. (2017) to the case where we use our Theorem 1 (i.e., we apply the bound introduced earlier to characterize the relation between the performance of h -GPI and the model and action-value function approximation errors), rather than using the GPI theorem, as defined by Barreto et al. (2017):

Above, (76) is due to Theorem 1 and (77) is due to the monotonicity of T ∗ . Finally, (78) is due to the application of Lemma 1 of Barreto et al. (2017), and (81) is due to the Cauchy-Schwarz's inequality.

Theorem 2, above, allows us to characterize h -GPI's performance (under approximation errors) with respect to the optimum action-value function, given an arbitrary reward function, r w :

Theorem 2 precisely characterizes the optimality gap (i.e., the maximum difference) between the action-value function induced by h -GPI ( q h -GPI w ( s, a )) and the optimal action-value function ( q ∗ w ( s, a )) , as a function of (i) the reward weights { w i } n i =1 for which the policies in the agent's library, Π , are optimal; (ii) approximation errors in action-value functions { ˆ q π ∗ i w } n i =1 ; and (iii) approximation errors in the model ˆ m = (ˆ p, ˆ r w ) .

B Experimental details

In this section, we provide a detailed description of the practical implementation of h -GPI, as well as details of the domains used in our experiments. The code necessary to reproduce our results is available in the Supplemental Material.

B.1 Computing h -GPI policies with SFs and Forward-Backward Dynamic Programming

In this section, we show how to extend the Forward-Backward Dynamic Programming (FB-DP) algorithm (Efroni et al., 2020) to the discounted-SFs setting, in order to compute h -GPI policies efficiently. The algorithm is described in Algorithm 3.

The algorithm works by first computing the sets of states achievable from the current state, s , after t = 0 , ..., h time steps (line 1 of Algorithm 3). The complete procedure is described in the ForwardPass algorithm (Algorithm 2). Next, the algorithm for computing h -GPI policies (Algorithm 3) assigns the value of states reachable in h steps using GPI bootstrapping with the SFs estimates (line 3). The values of states reachable in t = 1 , ..., h -1 time steps are then computed via dynamic programming (lines 4-5). Finally, the h -GPI action is identified in lines 6-8.

Regarding Algorithm 3, we first note that, given a state s , the algorithm's time complexity is O ( N |A||S tot h | ) , where N is the maximal number of accessible states in one step (i.e., maximal 'nearest neighbors' from any state), and |S tot h | is the total number of reachable states in h time steps from state s .

Note that while we chose to extend the original FB-DP algorithm to design a practical procedure for computing h -GPI policies, due to its efficiency and closed-form formulation, other planning techniques could have been used (e.g., Monte Carlo Tree Search (Tesauro and Galperin, 1996; Silver et al., 2017)).

Algorithm 2: Forward-Pass

B.2 Environments

Below, we describe the three domains used in our experiments-all of which are depicted in Figure 6.

G

S

L

FourRoom. The Four Room domain (Barreto et al., 2017; Gimelfarb et al., 2021) is a gridworld with dimensions of 13 × 13 , consisting of four rooms separated by walls. At each time step t , the agent occupies a cell and can move in one of four directions, denoted by the action set A = { up , down , left , right } . Differently from related works, we introduced a 10% chance of the agent moving to a random direction, rather than the direction given by the selected action. This makes the state transition function, p , stochastic. If the destination cell happens to be a wall, the agent remains in its current cell. The grid contains three different types of objects, as illustrated in the leftmost figure in Figure 6. There are four instances of each object type distributed throughout the grid. The state space is defined as the concatenation of the agent's current x-y coordinates and a set of binary variables indicating whether each object has been picked up or not: S = { 0 , 1 , ..., 12 } 2 ×{ 0 , 1 } 12 . The features ϕ ( s, a, s ′ ) ∈ { 0 , 1 } 3 are one-hot encoded vectors that indicate the type of object present in the current cell. If there are no objects in the current cell, the corresponding features are set to zero. The goal cell which the agent has to reach is located at the upper-right of the map. It contains one instance of each object, and entering this cell results in the termination of the episode. In this domain,

Algorithm 3: h -GPI with SFs and FB-DP

Input :

Model

ˆ

m

= (ˆ

p,

ˆ

ϕ

)

, SFs

{

ˆ

ψ

π

i

}

n

i

=1

,

planning horizon

h

0

, state

s

,

reward vector

w

.

1

{S

t

}

h

t

=0

ForwardPass( ˆ

m,h,s

)

2

for

s

∈ S

h

do

3

v

h

(

s

)

a

∈A

π

∈

Π

ˆ

ψ

π

(

s, a

)

·

w

4

for

t

=

h

-

1

, ...,

1

do

5

for

s

∈ S

t

do

6

v

t

(

s

)

a

∈A

[ ˆ

ϕ

(

s, a

)

·

w

+

γ

∑

s

′

ˆ

p

(

s

′

|

s, a

)

v

t

+1

(

s

)]

7

for

a

∈ A

do

8

(

T

∗

ˆ

m

)

h

π

∈

Π

ˆ

q

π

w

(

s, a

)

[ ˆ

ϕ

(

s, a

)

·

w

+

γ

∑

s

′

ˆ

p

(

s

′

|

s, a

)

v

1

(

s

)]

9

Return:

a

(

∗

ˆ

m

)

h

π

∈

Π

ˆ

q

π

w

(

s, a

)

T

a discount factor of γ = 0 . 95 was used. In our experiments, we used the implementation available on MO-Gymnasium (Alegre et al., 2022b).

Reacher. The Reacher domain is a classic domain in the SFs literature (Barreto et al., 2017; Gimelfarb et al., 2021; Nemecek and Parr, 2021; Alegre et al., 2022a). It consists of a two-joint robot arm that must reach different target locations with the tip of its arm (see the middle figure in Figure 6). The agent's state space S ⊂ R 6 consists of the sine and cosine of the angles of the central and elbow joints, as well as their angular velocities. The agent's initial state is the one shown in the middle figure in Figure 6. The action space, originally continuous, is discretized using 3 bins per dimension corresponding to maximum positive torque (+1), negative torque (-1), and zero torque for each actuator. This results in a total of 9 possible actions: A = {-1 , 0 , +1 } 2 . Each feature ϕ ( s, a, s ′ ) ∈ R 4 is defined as ϕ i ( s, a, s ′ ) = 1 -4∆( target i ) , i = 1 ... 4 , where ∆( target i ) is the Euclidean distance between the tip of the robot's arm and the i -th target's location. We used a discount factor of γ = 0 . 9 in this domain. We used the implementation of this domain available on MO-Gymnasium (Alegre et al., 2022b), which is based on the MuJoco robotics simulator (Todorov et al., 2012).

FetchPush. Finally, we extended the FetchPush domain (Plappert et al., 2018), which consists of a Fetch robotic arm that must move a block to a given target position on top of a table by pushing the block with its gripper (see the rightmost figure in Figure 6). Importantly, the state space of this domain, S ⊂ R 19 , is high-dimensional, and its dynamics are significantly more complex than that of Reacher's. In our experiments, we removed from the states all information related to the position of targets, as these are irrelevant since targets have fixed locations. We discretized the action space similarly to how this was done when defining the Reacher domain. The reward features ϕ ( s, a, s ′ ) ∈ R 4 correspond to the negative distances between the block and each of the four target locations on the table. Our implementation of this domain is an adaptation of the one available in Gymnasium-Robotics (de Lazcano et al., 2023).

B.3 Parameters

In the FourRoom experiments, the SFs of each policy were learned similarly as in Alegre et al. (2022a), using Q-learning and 5 Dyna updates per time step. Each policy was trained for 10 6 time steps using a learning rate of 0 . 1 and epsilon-greedy exploration with a probability of selecting a random action linearly decayed from 1 to 0 . 05 during half of the training period.

In the Reacher and FetchPush domains, we used multi-layer perception (MLP) neural networks to learn universal successor feature approximators (USFAs) (Borsa et al., 2019). The USFAs, ˆ ψ ( s, a, w ) , were modeled using MLPs with 4 layers of 256 neurons and ReLU non-linear activations. We used the Adam optimizer (Kingma and Ba, 2015) with a learning rate of 3 · 10 -4 , and mini-batches of size 256 . Recall that (as discussed in the main paper), we used vectors that form the standard basis of R d as training reward weights, M , where d is the dimension of the reward features vector ϕ . In our implementation, we also adopted popular DQN extensions to speed up and stabilize learning, such as prioritized experience replay (Schaul et al., 2016; Fujimoto et al., 2020) and Double Q-learning (Hasselt et al., 2016) with a target neural network updated after every 200 time steps. In both domains (Reacher and FetchPush), we trained the USFAs for a total of 2 · 10 5 time steps. In Reacher, the epsilon-greedy exploration rate was kept fixed at 0 . 05 , while in FetchPush, it was linearly decayed from 1 to 0 . 05 .

We used an ensemble of n =7 probabilistic neural networks in both the Reacher and FetchPush environments to learn the model m φ ( S t +1 , ϕ t | S t , A t ) . In the Reacher domain, each network was an MLP with 3 layers with 200 neurons each. In the FetchPush domain, we used an MLP with 4 layers with 400 neurons each. The probabilistic neural networks used to approximate the model were trained with early stopping based on a holdout validation subset with instances drawn from the experience buffer B -as commonly done when training such networks (Chua et al., 2018; Janner et al., 2019). In Figure 7, we depict the mean validation loss of the learned models both in the Reacher and FetchPush domains. As can be seen, the learned model rapidly becomes fairly accurate.

Compute. For the tabular experiments, we used an Intel i7-8700 CPU @ 3.20GHz computer with 32GB of RAM. For the experiments involving the function approximation setting, we used computers with NVIDIA A100-PCIE-40GB GPUs. For each random seed, the training period used for training

0

25000

50000

75000

100000

125000

150000

175000

200000

Training Time Step

0

.

0000

0

.

0025

0

.

0050

0

.

0075

0

.

0100

0

.

0125

0

.

0150

0

.

0175

0

.

0200

Mean Model Validation Loss

0

25000

50000

75000

100000

125000

150000

175000

200000

Training Time Step

0

.

0000

0

.

0025

0

.

0050

0

.

0075

0

.

0100

0

.

0125

0

.

0150

0

.

0175

0

.

0200

Mean Model Validation Loss

USFAs was of approximately 6.5 hours. We used the JAX library (Bradbury et al., 2018) for the implementation of neural networks.

C Related work

In this section, we discuss in more detail works that are related to h -GPI.

GPI and SFs. Previous works have extended GPI to safe RL (Gimelfarb et al., 2021; Feng et al., 2023), maximum-entropy RL (Hunt et al., 2019), unsupervised RL (Hansen et al., 2020), and hierarchical RL (e.g., via the options framework) (Barreto et al., 2019; Machado et al., 2023). Recently, Thakoor et al. (2022) introduced Geometric GPI (GGPI). GGPI uses geometric horizon models (GHMs) to learn the discounted future state-visitation distribution induced by particular policies to rapidly evaluate policies under a given, known reward function. The authors show that performing GPI over a particular type of non-stationary policy produces behaviors that outperform those in the agent's library policy. h -GPI, by contrast, learns a different type of model: an environment model, which is used to perform planning-i.e., action selection-rather than policy evaluation. Additionally, GGPI requires learning separate GHMs for each policy in the library, whereas h -GPI can operate with a single model that predicts the next reward features, alongside a single USFA. Secondly, GGPI assumes that the reward function is known a priori , while we exploit SFs to generalize over all linear rewards given reward features. Bagot et al. (2023) introduced GPI-Tree Search (GPI-TS), which is closely related to h -GPI. GPI-TS uses GPI bootstrapping as backup value estimates at the leaf nodes of a Monte Carlo tree search. However, GPI-TS does not employ SFs and was only employed in single-task settings. Moreover, it assumes an oracle model of the environment. We, by contrast, employ learned models to perform zero-shot transfer over multiple tasks, and we show how approximation errors in the model affect the performance of the h -GPI policy (see Theorems 1 and 2). Kim et al. (2022) introduced Constrained GPI (CGPI), which uses lower and upper bounds of the expected return of the optimal policy for a new task to constrain the action-values used when selecting the GPI policy's actions. Although CGPI is robust with respect to generalization errors in the values predicted by USFAs for new target tasks, it is sensitive to approximation errors in the values of the training tasks. h -GPI, by contrast, can inherently deal with approximation errors in the values of source/base policies. Other works have studied methods for constructing a policy library for use with GPI (Zahavy et al., 2021; Nemecek and Parr, 2021; Alver and Precup, 2022; Alegre et al., 2022a). These methods solve important orthogonal problems, and could potentially be combined with h -GPI.

Model-based RL and SFs. Russek et al. (2017) proposed a Dyna-style algorithm that performs background planning to update tabular successor representations (SRs) (Dayan, 1993). Momennejad et al. (2017) argued that this algorithm could better model human behavior on tasks with changing rewards when compared to pure model-free or model-based approaches. In this paper, by contrast, we employed online planning and leveraged GPI and SFs-both of which generalize SRs to continuousstate problems. Other works have studied methods for learning SF-based representations (Lehnert and Littman, 2020; Touati and Ollivier, 2021). Studying the behavior of h -GPI under different types of reward feature representations is an interesting direction for future research. We believe that our work

also shares similarities with the investigation performed by Tomov et al. (2021) on how model-free and model-based mechanisms are combined in human behavior.

Multi-step RL algorithms. Multi-step RL methods were extensively studied in the context of policy evaluation, both in the model-free (Hessel et al., 2018; van Hasselt et al., 2018) and the model-based settings (Yao et al., 2009; Janner et al., 2019). Model value expansion algorithms (Feinberg et al., 2018; Buckman et al., 2018; Abbas et al., 2020) are a representative example of the latter type of approach. In this paper, by contrast, we introduced a multi-step method for policy improvement, designed specifically to tackle transfer-learning settings. GX-Chen et al. (2022) introduced the η -return mixture, a new backup target for better credit assignment, which combines bootstrapping with standard value estimates and SFs, as a function of a parameter η . Efroni et al. (2018) studied multi-step greedy versions of the classic policy iteration and value iteration (Bertsekas and Tsitsiklis, 1996) dynamic programming (DP) algorithms. Efroni et al. (2020) proposed a multi-step greedy real-time DP algorithm that replaces 1-step greedy policies (used in policy improvement) with an h -step lookahead policy. The technique introduced in our paper is also related to the works of Sikchi et al. (2022) and Hansen et al. (2022), which combine planning and bootstrapping with a learned value estimate. However, unlike h -GPI, the abovementioned approaches did not address the multi-policy and zero-shot transfer settings. Another interesting use of multi-step prediction models is the work of Filos et al. (2022). In that work, the authors defined a measure of epistemic uncertainty based on the disagreement between the values resulting from applying the model-induced Bellman evaluation operator for different numbers of steps. Finally, we also emphasize to the interested reader that there exists neuroscientific evidence that multi-step planning occurs in the brains of humans and other animals (Miller and Venditto, 2021).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an to the Generalized Policy Improvement (GPI) framework, where the agent is able to leverage approximate models to derive a policy in a zero-shot fashion that adapts to unseen tasks related to others it has previously solved.\nThe paper derives an h-step planning horizon style action selection where the agent is able to trade-off long-term planning horizons for potentially compounding approximation errors in its update.\n\nThe authors provide theoretical results on the bound of improvement on their proposed method as it relates to model approximation error, and show that their approach is a consistent improvement over GPI in several standard RL benchmarks.\n\nSTRENGTHS:\n- The paper extends the GPI framework to leverage approximate models of the problem at hand. This provides a natural extension of the model-free approach of GPI, to a model based approach that allows an agent to take advantage of long-horizon planning when selecting actions.\n\n- The authors provide a theoretical bound on the improvement of their approach as it relates to the model approximation error.\n\n- The authors also presented a thorough experimental section with several standard benchmarks in the tabular and continuous state cases. These results show a consistent improvement over standard GPI.\n\nWEAKNESSES:\n- The main weakness I see is not of the method itself, but perhaps on how to apply it in practice. The theoretical results bounds the improvement of GPI based on the model approximation error, and in the limit one could set the h-step to 0 and recover GPI. \nHowever, since one of the main benefits is the potential of zero-shot adaptation to novel tasks, it is difficult to see how one would estimate the model error in practice when trying to solve a new task.\nHow would you know what h would be right when solving a new problem? According to Fig 3, the right h value seems to be quite important.\n\nQUESTIONS:\n- See weaknesses\n\n- Line 197-198, it states the \"increasing\" the planning horizon improves performance because the error term becomes irrelevant. Did you mean to say decreasing? Why would increasing horizon make error terms irrelevant? Wouldn't the error compound and therefore become more relevant to the performance?\n\n\nNitpick:\n\n- Notation in Eq. 7, and subsequent uses of max_{a' \\in A}(T).....max_{a \\in A} is not a function applied to the operator T itself, but rather to the result of the operator applied to q.\nThis should probably be max_{a' \\in A}( T*^h q(St+1, a;))....notice the parenthesis.\n\n\n- Eq 10, why the sudden change from pi to mu to refer to a policy? Is there any special meaning to it? If so, please clarify, if not, please pick a notation a be consistent with it.\n\n- line 116 reads a bit awkward, I think there are some typos there.\n- line 151 that's not the correct use of e.g.\n\nLIMITATIONS:\n- The authors that in practicality it would also be needed a way to assess the discrepancy between the model approximation and the true model to determine what the value of h should be.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a hybrid model-free and model-based policy improvement method for zero-shot policy generalization with successor features. The generalized value function is calculated using a generalized version of the multi-step Bellman operator, where a learned model is used in the rollout to predict future states and reward features.\n\nSTRENGTHS:\nThe idea to use multistep updates for generalized policy evaluation and improvement is novel to my knowledge. The paper is well written and relatively easy to understand. The authors carefully report detailed statistics for their experiments. I think that generally, the approach is a good idea and can be impactful if supported by the right set of experiments.\n\nWEAKNESSES:\nI am not sure about the meaning of the experimental results in section 4. As I understand it, the h-GPI approach amounts to planning ahead for h steps, and then obtaining the \"rest\" of the estimated future value for times after h using model-free successor features. If the learned model is relatively accurate, and the successor features are, compared to that, less accurate, then naturally the results improve as h increases and the decision-making relies more on the model over the successor features (as the authors also state after theorem 1). Likewise, it is unsurprising that simply ignoring the remaining value after h steps (as the SF-MPC baseline does iiuc, see footnote on page 8) performs worse than h-GPI. I think that a fair baseline would be SF-MPC for horizons much longer than h (the authors state themselves that SF-MPC underperforms since it \"requires long-term reasoning to collect items which are far from the agent\").\n\nAlternatively, one could make the argument that the \"cut-off\" done by h-GPI after h steps is a performance advantage over SF-MPC with much longer horizons, since rollouts can be shorter. However, I don't think that the authors are currently trying to make that argument; runtimes are barely mentioned (except for a brief discussion in the limitations section, but only in comparison to vanilla GPI).\n\nI would therefore suggest comparing to SF-MPC with much longer horizons, and if this should outperform h-GPI, maybe consider arguing from the perspective of run-time performance, supported by additional experiments on that.\n\nMinor issues: Should the reference in line 311 really be to figure 4, or should it be to figure 3? Also, it would probably help to quickly mention in the caption of figure 3 in what environment these experiments were run.\n\nQUESTIONS:\n- (See also above): What would happen if you increased the horizon for SF-MPC further?\n- How good is the dynamics model in the experiments for Figs 2,3, 4, and 5? Maybe it would be beneficial to report prediction errors (although not absolutely necessary if results for SF-MPC for long horizons are shown)\n- If the model quality is not perfect, is there a sweet-spot h at which the performance of h-GPI starts to decrease again with increasing h?\n\nLIMITATIONS:\nLimitations are discussed relatively briefly, it might be worth broadening this section a little.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper studies the combination of model-based planning/rl and model free generalized policy improvement and proposes h-GPI. The authors provide both theoretical and experimental results to support the method.

STRENGTHS:
1. This paper is clearly written and easy to follow. 
2. The motivation to study the intersection between model-based planning and GPI is also straightforward and clear.
3. The theoretical results are thorough, and are correct in my perspective.

WEAKNESSES:
1. For readers who are not familiar with GPI, Definition 1 is a bit confusing. What is $\pi'$ used for? In equation 5, $\pi'$ does not appear. Does that mean $\pi^{GPI}$ satisfies equation 4?
2. Can the authors explain how the policy set $\Pi$ is obtained?
3. Can the results generalize to the algorithm that replaces GPI to standard policy iteration/TD? It seems that h-GPI is a generalization of h-step model value expansion [1]. If so, I am interested in the comparison of h-GPI to MVE in both theory and experiments. Otherwise, the novelty compared to previous model-based value expansion works can be limited.
4. The current theoretical results only cover the bias of the proposed algorithm, which meets our expectation as model-free and model-based combinations require the trade-off of h. It would be more comprehensive if more analysis on the sample/computational complexity can be added.

[1] Vladimir Feinberg et al. Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

QUESTIONS:
See the weakness above.

LIMITATIONS:
See the weakness above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new hybrid solution of model-free and fully model-based reinforcement learning solutions for planning. It introduces a multi-step extension of generalized policy improvement by having certain planning steps to approximate model behavior, increasing the efficiency of the model. It is followed by zero-shot policy transfer.\n\nSTRENGTHS:\n1. Although the originality of the problem itself is not new, it does address the problem of working efficiency of current methods.\n2. The algorithm itself is very well-written, starting from basics of RL, MDPs and extended to the contribution itself.\n\nWEAKNESSES:\n1. The method itself lacks subjective comparison with other fully model-based or model-free solutions under same learning environment.\n2. Although the method addresses the problem of efficiency, it is not fully motivated and justified (why it should be useful and advancing from earlier model-free/model-based methods and why this is necessary?)\n3. The method is applied to two main setups - statistical one and deep learning based one. However, the setup looks a bit hand-crafted and lacks earlier reference. I do not think they comes from no-where - authors may address earlier works and add some general descriptions of those two settings, before getting into details.\n\nQUESTIONS:\n1. Why h-GPI should be useful and advancing from earlier model-free/model-based methods and why this is necessary?\n2. From high-level perspective, the proposed method looks like a fine-tuning of an approximators. So do you think this can be applied to \"conventional\" deep model-based planning methods? If so, do you think hGPI can still outperform?\n3. Do you think the method itself will be heavily rely on the generalization power of the universal SF feature approximator, rather than the model itself? It gives me this kind of instinct when reading the experimental sections.\n\nLIMITATIONS:\nThe authors listed some limitations of their methods at the very last section. Potential negative societal impact may not be applicable.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}