Successor Feature Representations
Authors: Chris Reinke
Xavier Alameda-Pineda
Venue: Accepted by TMLR ()

Abstract: Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in a new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application in high-dimensional tasks. But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to tasks where such a linear relationship exists. We propose a novel formulation of SR based on learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduce different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions, but also in the case of linearly decomposable reward functions.

Published in Transactions on Machine Learning Research (05/2023)
Successor Feature Representations
Chris Reinke
duris.veinke@inrii...
RobotLearn
INRIA Grenoble, LJK, UGA
Xavier Slameda-Pinede
zavier, www.aaalaamaaaaaa zwvier.daomedd--iieeddeei @innria
RobotLearn
INRIA Grenoble, LJK, UGA
Reviewed on OpenReview: Attps: // openreview. net/ met/teeruuueeee forum? id- IdFFfffrrDE!
Abstract
Transfer Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application high-dimensional tasks. But this came with the cost of having a lincar relationship between reward functions and successor features, limiting application to tasks where such linear relationship exists. We propose a novel formulation of SR based learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduce different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions, but also in the case of linearly decomposable reward functions.
1 Introduction
One of the goals of Artificial Intelligence (AI) is to design agents that have the same abilitics as humans, including to quickly adapt new enviromments and tasks. Reinforcement Learning (RL), the branch of AT that learns new behaviors based on the maximization of reward signals from the environment, has already successfully addressed many complex problems such playing computer games, chess, and even Go with superhuman performance (Mnih et al. 2015 Silver al. 2018 These impressive results are possible thanks to vast amount of interactions of the RT agents with their environment, ttakk during learning. Noneth such a strategy is unsuitable for settings where the amount of environment interactions is costly or restricted, for example, in robotics or when an agent has to adapt quickly to a new task or environment. Consider a caregiver robot in a hospital that has to learn a new task, such as new route to deliver meals. In such setting, the agent can not collect a vast amount of training samples but has to adapt quickly. Transfer learning aims provide mechanisms address this issue (Taylor & Stone, 2009 Lazaric 2012: Zhu et al., 2020). The rationale is to knowledge from previously encountered source tasks for new target task to improve the learning performance on this target task. The previous knowledge can help reduce the amount of interactions required to learn the new optimal behavior. For example, the caregiver robot could reuse knowledge about the layout of the hospital learned in previous source tasks (e.g. guiding a person) to learn to deliver meals.
The Successor Feature (SF) and General Policy Improvement (GPI) framework (Barreto et al. 2020 is a prominent transfer learning mechanism for tasks where only the reward function differs. SF is extension of

---

Published in Transactions on Machine Learning Research (05/2023)
Successor Representations (SR) Dayan 1993 SR represents policy (a learned behavior) by decoupling its dynamics from the expected rewards. The dynamics are described by the cumulative discounted probability of successor states for a start state i.e. a description of which future states the agent will visit if using the policy starting from s. This allows evaluating the policy by any reward function that defines rewards over visited states.
The SF framework extended SR by allowing high-dimensional tasks (state imputs) and by introducing the GPI procedure for the transfer of knowledge between tasks. Its basic premise is that rewards R(Ø) are defined based on low-dimensional feature vector ó E IT that describes the important features of a high- dimensional state. For our caregiver robot, this could be ID's of beds or rooms that it is visiting, in difference to its high-dimensional visual state input from camera. The rewards are then computed not based on its visual input but on the ID's of the beds or rooms it visits. The expected cumulative discounted successor features (1/ are learned for each policy that the robot learned in the past. It represents the dynamics in the feature space that the agent experiences for a policy. This corresponds to the rooms or bods the caregiver agent would visit if using the policy. This representation of feature dynamics is independent of the reward function. A behavior learned in previous task and described by this SF representation can be directly re-ovaluated for different reward function. In new task, i.c. for a new reward function, the GPI procedure re-evaluates the behaviors learned in previous tasks for it. It then selects at each state the behavior of a previous task if it improves the expected reward. This allows reusing behaviors learned in previous source tasks for a new target task. A similar transfer strategy can also be observed in the behavior of humans (Momennejad al. 2017 Momennejad]
2020
Tomov et al. 2021).
The SF&GPI framework Barreto et al. 2017:
2018) makes the assumption that rewards are linear position of the features & E I" via reward weight vector w; E R" that depends on the task i: o w; This assumption allows to effectively separate the feature dynamics of a behavior from the rewards and thus to re-evaluate previous behaviors given new reward function, ie. new weight vector wy. Nonetheless, this assumption also restricts the successful application of SF&GPI only to problems where such a lincar decomposition is possible or can be approximated..
We propose new formulation of the SR framework, called Successor Feature Representations (SFR), allow the usage of general reward functions over the feature space: r; R,))). SFR represents the cumulative discounted probability over the successor features, referred to as the g-function. We introduce SFRQ-learning (SFRQL) to learn the f-function. Our work is related to Janner et al. (2020 Touati & Ollivier (2021). and brings two important additional contributions. First, we provide mathematical proof of the convergence of SFRQL. Second, we demonstrate how SFRQL can be used for meta-RL, using the ε-function to re-evaluate behaviors learned previous tasks for new reward function Ry. Furthermore, SFRQL can also be used to transfer knowledge to new tasks using GPL.
The contribution of our paper is three-fold:
1. We introduce new RL algorithm, SFRQL, based on cumulative discounted probability of successor features.
2. We provide theoretical proofs of the convergence of SFRQL to the optimal policy and for guarantee of its transfer learning performance under the GPI procedure.
3. We experimentally compare SFRQL in tasks with linear and general reward functions, and for tasks with discrete and continuous features to standard Q-learning and the classical SF framework, demonstrating the interest and advantage of SFRQL.
2 Background
2.1
Reinforcement Learning
RL investigates algorithms solve multi-step decision problems, aiming to maximize the sum over future rewards (Sutton & Barto 2018h. RL problems are modeled as Markov Decision Processes (MDPs) which are defined tuple M (S, A, R,7), where S and A are the state and action set. An agent. transitions
2

---

Published in Transactions on Machine Learning Research (05/2023)
from state 8t to another state st+1 using action de time point collecting reward 7ei St "t" S1+1- This process is stochastic and the transition probability p(8++1 8t, as) describes which state 8t+1 is reached. The reward function R defines the scalar reward r R(s,, at St+1) E R for the transition. The goal in an MDP is to maximize the expected return G, = E [o 7 where R. = R(St, Ar. St+1). The discount factor y E [0. 1) weights collected rewards by discounting future rewards stronger. RL provides algorithms to learn a policy л : S A defining which action to take which state to maximise G.-
Value-based RL methods use the concept of value functions to learn the optimal policy. The stale-action value function, called Q-function, is defined the expected future return taking action as in St and then following policy w
(" Ex fre + 7r41 + - ..... Er (rx tQ A441)3
(1)
The Q-function can be recursively defined following the Bellman equation such that the current Q- value ("( ae) depends on the Q-value of the next state ("1(141, The optimal policy for an MDP can then be expressed based on the Q-function, by taking at every step the maximum action: it" (s) E arg m (s)
The optimal Q-function can be Icarned using temporal difference method such as Q-learning Watkins & Given a transition (Se, de, re, 81+1), the Q-value is updated according to:
Dayan 1992
Qx+1(8e, Qu(s,, az) + o& max Qu(S0+1;0++1) - Qu(s,, at)) act1
(2)
where ag (0, 1] is the learning rate al ileration k.
2.2 Successor Representations
Successor Representations (SR) were introduced as a generalization the value function Dayan 1993). SR represents the cumulative discounted probability successor states for a policy. Its state-action representa- tion [White [996) is defined as:
Do
M (St,
Cc ac,
Σ
p(S++ = (()s),at;T))
(3)
where p(S(+*= = s'|s,, (x; is the probability of being in state s' at time-point + l if the agent started in state 8t using action at and then following policy æ. Given a state-dependent reward function r, = R(s{), the Q-value, ie. the expected return, can be computed by:
(s) (s, S'ES
(4)
This allows to evaluate the expected return of learned policy for any given reward function. The SR framework has been restricted to low-dimensional and discrete state space because the representation of M (3] and the sum operator in the Q-value definition (4p are difficult to implement for high-dimensional and continuous states.
2.3 Transfer Learning with Successor Features
The Successor Feature (SF) framework (Gehring 2015 Barreto et al. 2017 2018 is an extension of SR handle high-dimensional, continuous state spaces and use the method for transfer learning. In the targeted transfer learning setting agents have to solve set of tasks (MDPs) M (M1 M2,..,, Mn). that differ only in their reward function. SF assumes that the reward function can be decomposed into linear combination of features & E &b C R and reward weight vector w, E R that is defined for a task M,:
(5 Ct, 81+1) p(9t, at, 81+1) Wi-
(5)
3

---

Published in Transactions on Machine Learning Research (05/2023)
We refer to such reward functions linear reward functions. Since the various tasks differ only in their reward functions, the features are the same for all tasks in M.
Given the decomposition above, it is also possible to rewrite the Q-function into an expected discounted sum over future features ( a) and the reward weight vector wy:
(s, = = Err (rx + + + Tr22 + +...) = fo: w. + 7 + 2
= E
w; = w (s, (w: w. oLtk
(6)
This decouples the dynamics of the policy in the feature space of the MDP from the expected rewards for such features. Thus, is now possible to evaluate the policy aH in a different task Mj using a simple multiplication of the weight vector w, with the ei-functio:: Q7 (s, = - (s. wy wy. Interestingly, the vb
function also follows the Bellman equation:
*º (s. a) = E [0 + yV ($1+1. ( az)
It can therefore be learned with conventional RL methods. Moreover, Lehnert & Littman 2019 showed the equivalence of SI-learnnng to Q-learning.
Being in new task M, the Generalized Policy Improvement (GPI) can be used select the action over all policies learned so far that behaves best:
T(s) E arg max max (s, = arg max max t (s,a) wy-
(8)
Barreto et al. 20118) proved that under the appropriate conditions for optimal policy approximation, the policy constructed in is close to the optimal one, and their difference is upper-bounder with:
-0' < 1 (Ir-rrile relloo + min ||r: rylloo +c).
2 (Ir-rilles
(9)
where - g max,,a (f) - g(s, The upper-bound between Q and Q has three components. First, the difference in reward (Ilr-rilla) between the current task M and the theoretically closest possible task Mj that can be linearly decomposed as in (5) Second, the difference (min, ||r: - -rylloo) between this theoretical task M, and the closest existing task M, A used by the GPT procedure (S) Third, an approximation error e. Very importantly, this result shows that the SF framework will only provide good approximation of the true Q-funttoon if the reward function can be represented using linear decomposition. If this is not the case, then the error in the approximation increases with the distance between the true reward function r and the best lincar approximation of it r; as stated by ||r-rillso For example, reward function R(Ø)= exp(-(Ø - )) using a Gaussian kernel to express a preferred feature around a could produce a large error.
In summary, SF&GPI extends the SR framework by introducing the concept of features over which the reward function is defined. This makes the transfer of knowledge between tasks with high-dimensional and continuous states possible. However, it introduces the limitation that reward functions linear combination of features and reward weight vector reducing their practical application for arbitrary reward functions.
3
Method: Successor Feature Representations
3.1 Definition and Foundations
The goal of this paper is to adapt the SR and SF frameworks to tasks with general reward functions R + + over state features ó d:
r(st, 0t, 81+1) R(g(st, at, 11+1)) ))) R(oe);
(10)

---

Published in Transactions on Machine Learning Research (05/2023)
where we define Φ, o( de, St+1). Under this assumption the Q-function can not be linearly decomposed into a part that describes feature dynamics and one that describes the rewards as in the SF framework ( To overcome this issue, we propose to use the future cumulative discounted probability of successor features, named f-function, which going to be the central mathematical object of the paper, as:
(s, a, 6)
Σ k=0
r * plor++ = Φ/s = s, 5, = =;;T)
(11)
where p(or+: = ōlst = 8,a, = a; R), or short p(o = @lst. ae; v), is the probability density function of the features at time + k following policy and conditioned to s and a being the state and action at time respectively. Note that ę depends only on the policy a but also on the state transition (constant through the paper). With the definition of the f-function, the Q-function rewrites:
2
2
S pl
Σ (=)
y* A 1 Σne 2h k=0
ac:
k==
J. R(Ø)
pllort 0/00, ar: T)d
R(O)E" (t,, d)le.
(12)
Both the proposed SFR and the SR frameworks exploit cumulative discounted sums of probability distri- butions. In the case of SR, the probability distributions are defined over the states (3) In contrast, the g-function of SFR defined over features (and not states). There exists also an inherent relationship be- tween SFR and SF. First, SF is a particular case of SFR when constrained linear reward functions, see Appendix A.4 Second, in the case of discrete features (Ø E N*), we can reformulate the SFR problem into linear reward function that is solved similarly to SF, see Appendix A.5 Therefore, SFR allows to consider these two cases within the same umbrella, and very importantly, to extend the philosophy of SF to continuous features and non-linear reward functions.
3.2 SFR Learning Algorithms
In order to learn the E-function, we introduce SFRQ-learning (SFRQL). Its update operator is an off- off-policy temporal difference update analogous to Q-learning and SFQL. Given a transition (8t, az S1+1, Qe) the SFRQL update operator is defined as:
(Sc, ae, 6) = (1 (1-ashl ax:) (sc, an + au (p(ór oisc, + (786
(13)
where 0++1 orggaax, S R( (81+1, o)ds
The following theorem provides convergence guarantees for SFRQL under the assumption that either p(ór = øist, (1;,,a;; π) is known, or an unbiased estimate can be constructed, and is one of the main results of this paper:
Theorem 1. (Convergence of SFRQL) For a sequence of state-action-feature {st, at St-1, 01 consider the SFRQL update given in (3). If the sequence of calle-action-eeabie triples visils each state, action infinitely often, and if the learning rate ag is an adapted sequence satisfying the Robbins-Moren conditions:
20
8
Σ
oz = 00,
a <<< x
(14)
then the sequence of function classes corresponding to the iterates converyes to the optimum, which corre- sponds to the optimal Q-function to which standard Q-learning updates would converge to:
[cn] + [E*] with Q'(s, (acaa)=/A (s, 0
(15)
The proof follows the same flow as for Q-learning and provided Appendix A.2 The main difference is that the convergence of SFRQL is proven up to additive function with zero reward, i.e. So R())s(s a, Φ)dØ
5

---

Published in Transactions on Machine Learning Research (05/2023)
0. More formally, the convergence is proven class of functions [g*], rather that to single function 5". This is not problematic, since additive functions with zero reward are meaningless for reinforcement learning.
As stated above, the previous result assumess that either p(é: 0/st, oz; TT is known, or an unbiased estimate can be constructed. We propose two different ways to approximate p(or = Φ st. ae; i)) from a given transition (st, (t; 8t+1, (e) so as to perform the É-update (13 The first instance a model-free version and detailed in the following section. A second instance uses a one-step SF model, called One-Step Model-based (MB)
SFRQL, which is further described in Appendix
MF SFRQL uses the same principle standard model-free temporal differ- ence learning methods. The update assumes for a given transition (st, St+1. @t) that the probability for the observed feature is p( = @x o²) = 1. Whereas for all other features (V E Φ o # Φt) the probability is p(ø' = &r (r( ae) = The resulting updates in case of discrete features are:
Model-free (MF) SFRQL:
VO Φ (st, ae Φ) - a0=aa{=(sa ac: Φ) + as [d=ó, + (1)1)0000)111)))))
0(+1 0)):
(16)
where 18= is the indicator function that feature & has been observed at time point t. Due to the stochastic update of the g-function and if the learning rate E (0, 1] discounts over time, the {-update will learn the true probability distribution p(ó = a.). Appendix D introduces the operator for contimuous features.
3.3 Meta SFRQ-Learning
After discussing SFRQL on a single task and showing its theoretical convergence, we can now investigate how it can be applied in transfer learning. Similar to the linear SF framework the g-function 5 allows to reevaluate a policy learned for task Mj in a new enviromment My:
{ (s,a),
(s,0))
(17)
This allows us to apply GPI in (8) for arbitrary reward functions in a similar manner to what was proposed for linear reward functions Barreto et al. (2018). We extend the GPI result to SFRQL as follows:
Thcorem 2. (Generalised policy improvement in SFRQL) Let M be the set of tasks, each one associated to a (possibly different) weighting function R2 E (D)). Let eoi be a representative of the optimal class of f-functions for task M,, i {1....,I, and let ę be approrimation to the optimal E-function, l|5T { [R., S,, Vi. Then, for another task M with weighting function R, the policy defined as:
(s) = arg max max / R(d)E" @b
(18)
satisfies:
2
I1 - 12 <
(min || R - Rd Rello(ole.o) + +c),
(19)
1 γ
where ||/l|g- = sups,a So If gl dob
The proof is provided in Appendix A.3
4 Experiments
We evaluated SFRQL in two environments The first has discrete features. It is a modified version of the object collection task by Barreto et al. (2117) having more complex features to allow the usage of general reward functions. See Appendix E.1 for experimental results in the original environment. The second environment, the racer environment, evaluates the agents in tasks with continuous features. We compared different versions of SFRQL, SFQL, and Q-learning which are abbreviated as SFR, SF, and Q respectively. The versions differentiate in if they are learning the feature and reward representations or these are given.
Source code at httpp://giitaas inria http://w...iit...eeeeeeema/e///a learning

---

Published in Transactions on Machine Learning Research (05/2023)
4.1 Discrete Features Object Collection Environment
The environment consist of 4 connected rooms (Fig. | a). The agent starts episode in position S and has to learn to reach goal position G. During an episode, the agent collects objects to gain further rewards. Each object has 2 properties: 1) color: orange or blue, and 2) form: box or triangle. The stale space is a high-dimensional vector E IR It encodes the agent's 's position by a 10 x 10 grid of two-dimensional Gaussian radial basis functions. Moreover, it includes a memory about which object has been already collected. Agents can move 4 dircctions. The features Φ E + {0, 1)3 are binary voctors. The first dimensions encode if an orange or blue object was picked up. The 2 following dimensions encode the form. The last dimension encodes if the agent reached goal G. For example, o' [1, 1,,0,,0,, 1,0,0, encodes that the agent picked up orange box.
Environment:
Tasks: Each agent learns sequentially 300 tasks which differ in their reward for collecting objects. We compared agents in settings: either in tasks with linear or general reward functions. For each linear task M, the rewards
o w, defined by linear combination of features and weight vector w, R The weights Wi,k for the first 4 dimensions define the rewards for collecting an object with a specific property. They are randomly sampled from a uniform distribution: Wi,k ~ U)-1,1). The final weight defines the reward for reaching the goal position which is wi,5 = 1 for all tasks. The general reward functions are sampled by assigning different reward to each possible combination of object properties 6j & using uniform sampling: R(())) u(-1,1), such that picking up an orange box might result in a reward of Ra = [1, 0)) 0.23, whereas blue box might result in R2(ó" 1,0,, 0)) 0.76
We compared SFRQL (SFR) Q-learning (Q) and classical Q-learning (SF) (Barreto el al. 2017). All agents use function approximation for their state-action functions (Q, 1,, or E). An independent mapping is used to map the values from the state for each of the 4 actions.
Agents:
We evaluated the agents under three conditions. First (Q, SF, SFR), the feature representation and the reward functions are given as defined in the environment and tasks. As the features are discrete, the
(a) Object Collection Environment
(b) Tasks with Linear Reward Functions
G
CSER-4
Average Return
600
200
100
160
180
200
220
240
260
280
300
S
Tasks Trained
Effect of Non-Linearity
(d) Tasks with General Reward Functions
Total Return Ratio
Q/ SFR
SF
SF-R
SF-1
SF-8
SFR
SER-R
CSER-4
Average Return
800
0.75
600
0.5
400
0.25
200
S210
1500
LA25
1.375
1,625
20
40
60
80
100
120
140
160
180
200
220
240
260
280
300
Mean Error of Linear Model
Tasks Trained
Figure 1: In the object collection environment (a), SFR reached the highest average reward per task for linear (b), and general reward functions (d). The average over 10 runs per algorithu and the standard error of the mean are depicted. (c) The difference between SFR and SF is stronger for general reward tasks that have strong non-linearities, i.c. where a linear reward model yields high error. SF can only reach less than 50% of SFR's performance in tasks with a mean lincar reward model error 1.625.

---

Published in Transactions on Machine Learning Research (05/2023)
f-function is approximated by an independent mapping for each action and possible feature & +. The Q-value Q(s, a) for the {-agents (Rq. 12 is computed by: (*,,a)) = E dee R( (8, a, (). In tasks with linear reward functions, the sampled reward weights w, were given to SF. For general reward functions, SF received an approximated weight vector w, based on a linear model that was trained before a task started on several uniformly sampled features and rewards.
In the second condition (SF-R, SFR-R), the feature representation is given, but the reward functions are approximated from observations during the tasks. In detail, the reward weights w, were approximated by linear model r øf w, for each task.
Under the third condition (SF-h, CSFR-h), also the feature representation ò E [0, 1Jah with / = 4 and h = 8 was approximated. The features were approximated with a linear model based on observed transitions by QL during the first 20 tasks. The procedure follows the multi-task learning protocol described by (Barreto et al. 2018) based on the work of (Caruanal 1997). As in this case the approximatod features are continuous, we introduce É-agent for continuous feat ures (CSFR) (Appendix D.2.4). CSFR discretizes each feature dimen- sion OR E [0, 1] in 11 bins with the bin centers: X {0.0, 0.1, .II,...,,,,,, 1.03. It learns for each dimension & and bin i the f-value EE (s, a, X,). Q-values (Eq. 12 are computed by (5, Σh= En Tx(X.) a, X;). CSFR-8 is omitted as it did not bring a significant advantage.
Each task was exccuted for 20, 000 steps, and the average performance over 10 runs per algorithm measured. We performed a grid-search over the parameters of each agent, reporting here the performance of the parameters with the highest total reward over all tasks.
Results: SFR outperformed SF and Q for tasks with linear and general reward functions (Fig. b, d). This was the case under the conditions where the features are given and the reward functions were either given (SF, SFR) or had to be Iearned (SF-R, SFR-R). When features are approximated, both agents (SF- 4, CSFR-4) had similar performance slightly above SFR-R. Approximating 8 features showed lower performance than features (SF-8, SF-4). We further studied the effect of the strength of the non-linearity in general reward functions on the performance of SFQL compared to SFRQL by evaluating them in tasks with different levels of non-linearity. We sampled general reward functions that resulted in different levels of the mean absolute model error if they are linearly approximated with min& (r()) - o w/ We trained SF and SFR in each of these conditions on 300 tasks and measured the ratio between the total return of SF to SFR (Fig. 1. c). The relative performance of SF compared to SFR (SF / SFR) decreases with the level of non-linearity. For reward functions that are nearly linear (mean error of 0.125). both have a similar performance. Whereas, for reward functions that are difficult to model with linear relation (mean crror of 1.6 1.625) SF reaches only less than 50% of the performance of SFR. This follows SF's theoretical limitation in showing the advantage of SFR over SF in non-lineat linear reward tasks.
4.2 Continuous Features Racer Environment
Environment and Tasks: We further evaluated the agents in a 2D environment with continuous features (Fig. [ a). The agent starts at a random position and drives around for 200 timesteps before the episode ends. Similar to a car, the agent has an orientation and momentum, so that it can only drive straight, or in a right or left curve. The agent reappears on the opposite side if exits one side. The distance to 3 markers are provided as features Φ E 3 Rewards depend on the distances =x=17&(0x) (o(()), where each component Tk has 1 or 2 preferred distances defined by Gaussian functions. For each of the 40 tasks, the number of Gaussians and their properties (1, o) are randomly sampled for each feature dimension. Fig. 2 (a) shows reward function with dark areas depicting higher rewards. The state space is high-dimensional vector E 2220 encoding the agent's position and orientation. The 2D position is encoded by 10 x 10 grid of two-dimenssonal Gaussian radial basis functions. Similarly, the orientation is also encoded using 20 Gaussian radial basis functions.
Σ
Agents: We used the CSFR agent as the task has continuous feature spaces. The reward weights w; for SF were approximated before each task based on randomly sampled feat ures and rewards. We further evaluated
8

---

Published in Transactions on Machine Learning Research (05/2023)
Racer Environment
(b) Tasks with General Reward Functions
SF
SE-R SF-3 - SF-6
CSFR CSTR-R
CSFR-3
80k
Average Retur
60k
40k
5
10
15
20
25
30
35
40
Tasks Trained
Figure 2: (a) Example of reward function for the racer environment based distances to its 3 markers. (b) SFRQL (SFR) reaches the highest average reward per task. SF yields performance even below Q as it is not able to model the reward function with its linear combination of weights and features. The average over 10 runs per agent and the standard error of the mean are depicted.
agents (SF-R, CSFR-R) that approximatod a linear reward model during task execution and agents (SF-h, CSFR-h) that approximated the features using data collected of the Q-learning agent from the first 10 tasks.
Results: CSFR reached the highest performance of all agents (Fig. 2 b) outperforming Q and SF. SF and all agents that approximated the reward functions (SF-R, CSFR-R) and features (SF-h, CSFR-h) reached only a performance below Q. They were not able to approximate sufficiently well the Q-function as their reward functions and features depend on a linear reward model which can not represent well the general reward functions in these tasks.
5 Discussion and Limitations
SFRQL compared to classical SFQL: SFRQL allows to disentangle the dynamics of policies in the feature space of a task from the associated reward, see (12). The experimental evaluation tasks with general reward functions (Fig. [ d, and Fig. 2 shows that SFRQL can therefore successfully apply GPI to transfer knowledge from learned tasks to ones. Given general reward function it can re-evaluate successfully learned policies. Instead, classical SFQL based lincar decomposition can not be directly applied given general reward function.
SFRQL also shows an increased performance over SF in environments with linear reward functions (Fig, a). This effect can not be attributed to differencee in their computation of a policy's expected return as both are correct (Appendix A.4). Additional experiments showed that SFRQL outperforms SF also single tasks without transfer. We hypothesis that SFRQL reduces the complexity of the problem for the function approximation compared to gi-function.
In principle, classical SFQL can also optimize general reward functions if features and reward weights are learned (SF-h). This is possible if the learned features describe the non-linear effects in the reward functions. Nonetheless, learning of features adds further challenges and shows reduced performance in our experiments.
Learning of Features:
The used feature approximation procodure for all SF-h and SFR-h agents by Barreto et al. 2017 learns leatures from observations sampled from initial tasks before the GPI procedure starts. Therefore, novel non-linearities potentially introduced later tasks are not well represent by the learned features. If instead features are learned alongside the GPI procedure, the problem on how coordinate both learning processes needs to be investigated. Importantly, gi-functions for older tasks would become unusable for the GPI procedure newer lask, because the feature representation changed between them.
Most significantly, our experimental results show that the performance of agents with learned features (SF-h) is strongly below the performance of SFRQL with given features (SFR and CSFR). Therefore, if features and reward functions are known then SFRQL outperforms SF, otherwise they are roughly equivalent. Using given features and reward functions natural for many applications as these are often known, for example in robotic tasks where they usually manually designed (Akalin Loutfi 2021).

---

Published in Transactions on Machine Learning Research (05/2023)
For tasks with continuous features (racer environment), SFRQL used suc- cessfully a discretization of each feature dimension, and learned the g-values independently for each di- mension. This strategy is viable for reward functions that are cumulative over the feature dimensions: r(0)= = Ex rx(o)) The Q-value can be computed by summing over the independent dimensions and the bins X: (=) = Ex EnEx (i(ar) (s, a, For more general reward functions, the space of all lea- ture combinations would need to be discrctized, which grows exponentially with each new dimension. As a solution the f-function could be directly defined over the continuous feature space, but this yields some problems. First, the computation of the expected return requires integral Q(s.a) S R(o) o) over features instead of a sum, which is a priori intractable. Second, the representation and training of the f-function, which would be defined over & continuum thus increasing the difficulty of approximating the function. Janner et. al. (2020) and Touuai & Ollivier 2222)) propose methods that might allow to represent a continuous E-function, but it is unclear if they converge and if they can be used for transfer learning.
Continuous Feature Spaces:
Computational Complexity: The improved performance of SFRQL and SFQL over Q-learning comes al the cost. of increased computational complexity. The GPI procedure (8) evaluates at each step the rh ""i-function over all previous experienced tasks in M. Hence, the computational complexity of both procedures increases linearly with each added task. A solution is to apply GPI only over subset of learned policies. Nonetheless, how to optimally select this subset is still an open question.
6 Related Work
Transfer methods in RL can be generally categorized according to the type of tasks between which transfer is possible and the type of transferred knowledge (Taylor & Stone 2009 Lazaric 2012: Zhu al. p020 : In the case of SF&GPI which SFRQL is part of, tasks only differ in their reward functions. The type of knowledge that is transferred are policies learned in source tasks which are re-evaluated in the target task and recombined using the GPT procedure. A natural use-case for SFRQL are continual problems
Transfer Learning:
[Khotarpal et al. 2020) where an agent has continually adapt to changing tasks, which are in our setting different reward functions.
Successor Representations & Features: SFRQL is a variant of SR. SR represent the cumulative prob- ability of future states (Dayan 1993: White 1996) whereas SFRQL represents the cumulative probability of low-dimensional future state features. SRs were restricted to low-dimensional state spaces using tabular representations (Lizotte et al.] 2008). Some recent extensions of SRs outside of transfer learning exists, for
example, as a basis for exploration (Machado et al. 2020 or active inference (Millidge & Buckley 2022).
SF extend SR to domains with Sigh-dimensional state spaces (first formulated by Gehring (0155) and then studied in depth in (Kulkarni et al. 2016 Barreto et al. 2017 2018 Zhang et al. 2017)), by predicting the future occurrence of low-dimensional features that are relevant to define the return. Several extensions to the SF framework have been proposed. One direction aims to learn appropriate features from data such as by optimally reconstructing rewards (Barreto et al., (017]), using the concept of mutual information (Hansen et al. 2019). or grouping of temporal similar states (Madjiheurem & Tonn 2019). Another direction is the generalization of the di-function over policies (Borsa et al. 2018) analogous to universal value function ap- proximation (Schaul et al. 2015). Similar approaches use successor maps (Madarasz) 2019 goal-conditioned policies (Ma et al. 2020). or succossor feature sets (Brantley et al., 2021]. Other dircctions include their application to POMDPs (Vertes & Sahani, 2019), combination with max-entropy principles (Vertes 2020 or hierarchical RT. (Barreto et al. (021). All these approaches build on the assumption of linear reward func- tions, whereas SFRQL allows the SF&GP1 framework to be used with general reward functions. Nonetheless, most of the extensions for linear SF can be combined with SFRQL.
SFR represents the dynamics of a policy in the feature space that is decoupled from the rewards allowing to roevaluate them under different reward functions. It shares therefore similar properties with model-based RL (Lehnert & Littman 2019). In general, model-based RL methods learn a one-step model of the environment dynamics p( 1t, az). Given a policy and an arbitrary reward function, rollouts can be performed using the learned model to evaluate the return. In practice, the rollouts have a high variance for long-term predictions rendering them ineffective. Recently, (Janner et al., 2020) proposed the m-moee framework that learns to represent ξ-values in continuous domains. Nonetheless, the application
Model-based RL:
10

---

Published in Transactions on Machine Learning Research (05/2023)
to transfer learning is not discussed and convergence is proven for SFRQL. This is the same case for the forward-backward MPD representation proposed in Touati & Ollivier (2021) ). (Tang et al. 2021) also proposes to decouple the dynamics in the state space from the rewards, but learn an internal representation of the rewards. This does not allow to reevaluate an policy to a new reward function without relearning the mapping-
7 Conclusion
The introduced SFR framework with its SFRQ-learning algorithm learns the expected cumulative discounted probability of successor features which disentangles the dynamics of a policy in the feature space of a task from the expected rewards. This allows SFRQL to reevaluate the expected return learned policies for general reward functions and to use for transfer learning utilizing GPI. We proved that SFRQL converges the optimal policy, and showed experimentally its improved performance over Q-learning and the classical SF framework for tasks with linear and general reward functions.
Broader Impact Statement
SFR represents general purpose RL algorithm for transfer scenarios without a specific potential for negative societal impact or et hical considerations.
Acknowledgments
This research is supported by H2020 SPRING project funded by the European Commission under the Horizon 2020 framework programme for Research and Innovation (H2020-ICTT-2019-2 GA #871245) well by the ML3RI project funded by French Research Agency under the Young Researchers programme #ANR-19-CE33-0000-00
References
Neziha Akalin and Amy Loutfi. Reinforcement learning approaches in social robotics. Sensors, 21(4):1292, 2021.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pp. 0555-005,, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowilz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501 510. PMLR, 2018.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 117(44):3077933000 2020.
André Barreto, Diana Borsa, Shaobo Hou, Ghcorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan Hunt, Shibl Mourad, David Silver, et. al. The option keyboard: Combining skills in reinforcement learning. arXie preprint arXiv:2106. 13100100..110,, 2021.
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features approximators. arXiv preprint 18122881.00722 2018.
Kianté Brantley, Soroush Mehri, and Geoffrey J Gordon. Successor feature sets: Generalizing successor representations across policies. arXiv preprint arXiw:2109.0226 2021.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-2 1997.
11

---

Published in Transactions on Machine Learning Research (05/2023)
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613 624, 1993.
Clement Gehring. Approximate linear successor representation. Reinforvement Learning Decision Making, 2015.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:1906.0500 2019.
Michael Janner, Igor Mordatch, and Sergey Levine. y-models: Generative temporal difference learning for infinite-horizon prediction. In NewrIPS, 2020.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Procup. Towards continual reinforcement Iearn- ing: A review and perspectives. arXio preprinl or 2rvi:22012. 13490, 2020.
Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606. 0626006,, 2016.
Alessandro Lazaric. Transfer reinforcement learning: framework and a survey. In Reinforcement Learn- ing, pp. 143-173 Springer, 2012.
Lucas Lehnert and Michael I. Littman. Successor features support model-based and model-free reinforcement learning. CoRR abs/1901.11499 2019.
Daniel Lizotte, Tao Wang, Michael Bowling, and Dale Schuurmans. Dual representations for dynamic programming. 2008.
Chen Ma, Dylan R Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. arXiv preprint or 20v:20001.0000 04025, 2020.
Marlos C Machado, Mare G Bellemare, and Michael Bowling. Count-based exploration with the successor representation ion. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 5125- 5133, 2020.
Tamas J Madarasz. Better transfer learning with inferred successor maps. arv preprint arXiv:1966.00 076663 2019.
Sephora Madjiheurem and Laura Toni. State2vec: Off-policy successor features approximators. preprint w:Xt::1910.10277 2019.
arXiv
Beren Millidge and Christopher L Buckley, Successor representation active inference. arXiv preprint arXiv:2207.098977 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Ida Momennejad. Learning structures: Predictive representations, replay, and generalization. Current Opinion in Behavioral Sciences, 32:155-166, 2020.
Ida Momennejad, Evan M Russek, Jin H Cheong, Matthew M Botvinick, Nathaniel Douglass Daw, and Samuel Gershman. The successor representation in human reinforcement learning. Nature Human Behavioor, 1(9):650 692, 2017.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference on machine learning, pp. 1312-1320, 2015.
David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforeement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419):11100 1144, 2018.
12

---

Published in Transactions on Machine Learning Research (05/2023)
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hongyao Tang, Jianye Hao, Guangyong Chen, Pengfei Chen, Chen Chen, Yaodong Yang, Luo Zhang, Wu- long Liu, and Zhaopeng Meng. Foresee then evaluate: Decomposing value estimation with latent future prediction. arXiv preprint ar (iix:2109.02220 2021.
Matthew Taylor and Peter Stone. Transfer learning for reinforcement learning domains: survey. Journal of Machine Learning Research, 10(7), 2009.
Momchil S Tomov, Eric Schulz, and Samuel J Gershman. Multi-task reinforcement learning in humans. Nature Human Behaviour, pp. 10, 2021.
Ahmed Touati and Yamn Ollivier. Learning one representation to optimize all rewards. arXie preprint arXiv:2103.079%, 2021.
John N Tsitsiklis. Asynchronous stochastic approximation and q-learning, Machine learning, 16(3):185 202, 1994.
Eszter Vertes. Probabilistic learning and computation in brains and machines. PhD thesis, UCL (University College London), 2020.
Eszter Vértes and Maneesh Sahani. A neurally plausible model learns successor representations in partially observable environments. Advances in Neural Information Processing Systems, 32, 2019.
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8(3-4):279 292, 1992.
Lisa M White. Temporal difference learning: eligibility traces and the successor representation for actions. Citeseer, 1996.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pp. 2371-2378. IEEE, 2017.
Zhuangdi Zhu, Kaixiang Lin, and Jiayu Zhou. Transfer learning deep reinforccment learning: A survey. arxiv preprint or Xic:20009. 07888, 2020.
13

---

Published in Transactions on Machine Learning Research (05/2023)
A Theoretical Results
This section provides proofs for the converge of SFRQL (Theorem D and the performance bound of the GPI procedure (Theorem 2) First it introduces some preliminaries relevant for Theorem before stating the proofs. The final two sections provide more background about the relation between classical SF and SFRQL.
A.1 Preliminaries
The following Propositions provide the background for the proof the convergence of SFRQL (Section A.2 Depending on the reward function R, there are several g-functions that correspond to the same Q function. Formally, this is equivalence relationship, and the quotient space has one-to-one correspondence with the Q-function space.
Proposition 1. (Equivalence between functions & and Q) Let R (') and E {5 8x
I s.l. E(s,a,)) ()))), a) E S x A, sups,a So (( a, o)de) oo and Q - I s.t. 1101 < xc). Let be defined as & ~n E2 e Ja R(o)Ci(s,a, (,, Φ) = fu R(o) Φ) V(s.a) E 8 x A. Then, is an equivalence relationship in E, and there is a bijective correspondence between the quotient space En and Q. The function class corresponding to & will be denoted by [E]
Proof. We will proof the statements sequentially.
~R is an equivalence relationship: To prove this we need to demonstrate that ~a symmetric, recip- rocal and transitive. The three are quite straightforward that ~aR defined with an equality.
Bijective correspondence: To prove the bijectivity, we will first prove that it is injective, then surjective. Regarding the injectivity: + + Qc + Qn, we prove it by contrapositive:
Q: =, = / (( a, d) Rlo)n(s,a. = [ [n].
(20)
In order to prove the surjectivity, we start from a function Q Q and select an arbitrary & E E, then the following function:
((s.a)
fo( Φ) H())(s a, o)d0 {()
(21)
satisfies that εq E and that Ja R(o)Eo(s, a, o)do = Q a), v(, S x A. We conclude that there is a bijective correspondence between the elements of Ex and of Q
Corollary 1. The bijection between Er and Q allows to induce a norm ||- || into ER from the supremum norm in Q, with which En is a Banach space (since Q Banach with || lbx)):
/ (1)) a.a
| = sup
= sup [Q(s, a)l = 1|1
(22)
5,8
Proof. The norm induced in the quotient space defined from the correspondence between ER and and is naturally defined as in the previous equation. The norm is well defined since it does not depend the class reprosentative. Therefore, all the metric propertics transferred, and En is immediately Banach with the norm ||
Similar to the Bellman equation for the Q-function, we can define a Bellman operator for the E-function, denoted by Te, as:
Te(E") as, = p(olst, + yEps (930a1)) (S(+1.11.00 0x+1,
(23)
with āef1 arggaaa, max. Jo (s+11, We can Tg to construct contractive operator:
14

---

Published in Transactions on Machine Learning Research (05/2023)
Proposition 2. (SFRQL has fixed point) The operator Tş is well-defined w.r.t. the equivalence ~, and therefore induces operalor Tn. defined over En- Tn is contractive w.r.t. . l|n. Since En is Banach, Th has unique fixed point and iterating TR starting anywhere converges to that point.
Proof. We prove the statements above one by one:
Let us first recall the definition of the operator T3 in removed the dependency for simplicity:
The operator Ta is well defined:
23 where we
To (e( 01, Φ) = p(olsc. an) + (p (5(8++1,0441,99
Let E2 E [E] two different representatives of class [g], we first observe that the optimal action does not depend on the representative:
arg max (( ø)dø arg max / (, afts a.+1 vs. (24)
E1 a++1
We now can write:
S. (())Te(E))(s),) at, - (5( ae; d))dó
= -
/ ( ((1 (1-1,, (2 (04111,,441,, 6) ds)+11dd
(25)
/ p((1-1187,00) / (141, E2 (d
because f1, fa E [c]. Therefore the operator TR((E)) (5) well defined in the quotient space, since the image of class does not depend on the function chosen to reprosent the class.
Contractive operator Thi The contractiveness of Ta can be proven directly:
/ R(Ø) (p(ols;, ar) + (e (((s41,
[T]((E)) - TR((T))l = sup
Se.ae
-p(olor, at) y Eee "" o)}) dø
/ E (E( Φ) n n(141,,7/41 Hd
=7 sup 8:.9
|sup
R(o)n(s41 a41, 2)de }
S7 sup 1l
((0)E(8++1, M)d0 - sup
act1
/ R(o)(((s)) - o))
<y sup
8111:9111
=> [n]ll
(26)
The contractiveness of TR can also be understood as being inherited from the standard Bellmann operator on Q. Indeed, given a function, one can easily see that applying the standard Bellman operator the Q function corresponding to & leads to the function corresponding to (n(()))
15

---

Published in Transactions on Machine Learning Research (05/2023)
Fixed point of Tx: To conclude the proof, use the fact that any contractive operator Banach space, in our case: Tn En + En has unique fixed point [5*]. and that for any starting point [So], the sequence [5n] = TR((S--)))) converges to [=*] w.r.t. to the corresponding norm ||[
In other words, successive applications of the operator Tn converge towards the class of optimal & functions [5*] or equivalently to an optimal & function defined up to an additive function k satisfying la k(s, a, R())d = 0, V(s,a) E x A (i.e. k Ker(s Rg)).
While these two results (Propositions) 1 and state the theorctical links to standard Q-lcarning formulations, the Tę operator defined in 23 is not usable in practice, because of the expectation.
A.2 Proof of Theorem
Propositions and 2 are useful to prove that the learning iterates converge in ER. Let us restate the dcfinition of the operator from 13
ST ( at, Φ) (st, au Φ) + as pp(:: øss; ae R) + 7ET (st11, @++1.0) Ex (st, Φ)
and the theoretical result:
Theorem 1. (Conwergence of SFRQL) For sequence of state-action-feature {st, at 8t-1, dili consider the SFRQL update given in [3]. If the sequence of dtote-action-featur triples visils each state, action infinitely often, and if the learning rate ag is an adapted sequence satisfying the Robbins-Mome conditions:
OA 00,
a x0
(27)
then the sequence of function classes corresponding the iterates converges to the optimum, which corre- sponds to the optimal Q-function to which slandard Q-learning updates would converge to:
[En] + [E*] with Q*(s,a) = / (s,
(28)
Proof. The proof re-uses the flow of the proof used for Q-lcarning operator above as:
Tsitsiklis 1994). Indeed, we rewrite the
(tt ar 0) +- Ex (Se, ar Q) + an (Te(EZ) ae, Φ) (sc, ae, 3) + E(S,, 0))
with & defined as:
(1) p(ói at; + 25 ($411,1+4,,1)) Φ) -E (p(0, 0180, de: m) + 75t ( ae b)}
Obviously satisfies [B{s} 0, which, together with the contractiveness of TR, is sufficient to demonstrate the convergence of the iterative procedure as done for Q-learning. In our case, the optimal function &" is defined up to an additive kernel function & Ker (5 - J. RE). The correspondence with the optimal Q learning function is a direct application of the corrospondence between the g- and Q-learning problems.
Generally speaking, the exact probability distribution of the features p(ø. dist, dei T) is not available. In our model-free algorithm, this distribution is replaced by the indicator function of the observed feature, whose expectation is equal to the exact distribution, and therefore the condition E[s}} 0 is satisfied. In our model-based algorithm proposed below, we need to be sure that the estimated distribution is not biased, so that the zero-mean condition is also satisfied.
A.3 Proof of Theorem 2
Let restate the result.
16

---

Published in Transactions on Machine Learning Research (05/2023)
Theorem 2. (Generalised policy improvement in SFRQL) Let M be the set of tasks, each associated to (possibly different) weighling function R4 E (Φ). Let ghi be a representative of the optimal class of E-functions for task Mi, i E {I,...,, and let E be an approzimation to the optimal E-function, l|5 t Ila, Ev, Then, for another task M with weighting function R, the policy defined as:
(()) = arg max max
(s,
(29)
J
satisfies:
2
1 - 1- v 1
(min || | Ra Rellp(ola.0) t
(30)
where [[f = s,a So 1f gl do.
Proof. The proof is stated two steps. First, we exploit the proof of Proposition and in particular (13) that states:
(Barreto et al, 2017
Q" - 1-7 sup |r(s, (.s,
Vi (I....,,I}
(31)
where Q" and Q* are the Q-functions associated to the optimal and i policies in the environment R. The conditions on the Q functions required the original proposition are satisfied because the f-functions satisfy them, and there an isometry between Q and & functions.
Because the above inequality is true for all training environments i, we can rewrite as:
2
Q1 - 10 S 1-y (minsupp((6.))
(32)
We now realise that, in the case of SFRQL, the reward functions rewrite as:
r(s.a) R())p(o)s;a)d) r)(s.a)= (
(33)
and therefore have:
(up/r(s,a) r((ssaa) = sup 8.0
(R( ((0))p(()ss
4 8.0
(34)
S sup s,8
R(Ø)) [c(o)) a)do
R
R Rillp(dlaa))
Similarly, due to the isometry between & and Q-learning, i.e. Proposition 2 we can write that:
||g E-11
||
[g*]]] 1|Q Q 10
2 min (r(8, r)) +e)
1-7
8,a
(35)
2
(min | Rellp(ols.a) + +
which proves the generalised policy improvement for SFRQL.
A.4 Relation between classical SFQL and SFRQL for linear reward functions
In the case of linear reward functions, i.c. where assumption (5) holds, possible show that SFRQL can be reduced to classical SF. Classical SF represents therefore specifie case of SFRQL. under this assumption.
17

---

Published in Transactions on Machine Learning Research (05/2023)
(Equality of classical SF and SFRQL for linear reward functions) Given the assumption that reward functions are linearily decomposable with
Theorem 3.
ri at S6+1) ((8t, d2, 86+1) Wi,
(36)
where @ E Rd are features for a transition and w, R" are the reward weight vector of task m, E M, then the classical SF and SFRQL fromework are equivalent.
Proof. We start with the definition the Q-value according to SFRQL. from 12
After replacing the reward
function RG with our linear assumption, the definition of the Q-function according to classical SF with the uiffnction can be recovered:
Q((s,a)
(" a, o (o)dó
S (" ar T
w
* p(o44 = -0180 8,0, =, T6 d$
(37)
pl =0882 8, a; の de
7 [0r+k]
k=
T
7 PLtk
w; = ass w,.
k=0
Please note, although both methods are equal in terms of their computed values, how these represented and learned differs between them. Thus, it is possible to see a performance difference of the methods in the experimental results where SFRQL outperforms SF in our environments.
A.5 Relation between classical SF and SFRQL for general reward functions
An inherent connection between classical SF and SFRQL exists for general reward functions that have either discrete feature space where the feature space is discretized. In these cases SFRQL can be viewed reformulation of the original feature space and general reward functions into a feature space with linear reward functions which are then solved with the classical SF mechanism. Nonctheless, in continuous fcature spaces, such a reformulation is not possible. SFRQL represents in this case a new algorit hin that is able lo converge to the optimal policy (Theorem in difference to the classical SF method. The following section analyzes cach of the three cases.
Discrete Feature Space Given an ordered set of discrete features á E (di...., Uj. pm + with m m= and general reward functions r=(o) R. We can reformulate this problem into linear decomposable problem using binary features tp {0, 13Th and reward weight vector w, R The feature vector &o represents binary pointer, where its j'th element (1,)) is 1 if the observed feature de is the j'th feature (Φ,) of the original feature set T:
Φ =
0
(38)
otherwise
The j'th element in the reward weight vector corresponds to the reward of the j'th feature in 4: wis r())). Given such reformulation, see that it is possible to reformulate the reward function using
18

---

Published in Transactions on Machine Learning Research (05/2023)
linear composition: (r) "o",, Based on the reformulation, can define the revised SF function:
()) a) = Rx Σ Deth 18t = =8, 8, a
(39)
If we concentrate on single element j of this vector function and replace the expectation by the probabilitics of the feature element being 0 1 at certain time point, then we recover SFRQL:
(8, a)5 = E
Pseth |8z = 8, =
k=0
En (1ij44 18t =
"( 0|s; s,a, a) 0 p(p s, a)
(40)
2 p 1/5t s,a, a;
plor+k = 0g|st = s. a) = a5))
k=0
In conclusion, this shows the underlying principle of how SFRQL solves problems with discrete feature spaces. It reformulates the feature space and the reward functions into a form that can then be solved via a lincar composition.
Discretized Feature Space Discretized continuous feature spaces as used by the CSFR agent can also be reformulated to a linear decomposable problem. The proof follows the same logic as the proof for discrete feature spaces. In this case binary index features for each bin of the discretized feature dimension of the CSFR can be constructed.
Continuous Feature Space In the case of continuous feature spaces 4 R" reconstruction of the feature space and the reward functions to construct problem with a lincar composition of rewards similar to the one given for discrete feature spaces not possible. In this case, it is not possible to define vector whos element point to each possible element in d as tb E R" is a uncountable set. Therefore, SFRQL can not recovered by reformulating the feature space and the reward functions. represents a class of algorithms that can not be roduced to classical SF. Nonctheless, the convergence proof of SFRQL also holds under this condition, whereas classical SF not solve such environments.
B One-Step SF Model-based (MB) SFRQL
we introduce a second SFRQL, procedure called One-step SF Model-based (MB) SFRQL that attempts reduce the variance of the update. To do so, MB SFRQL estimates the distribution over the successor features over time. Let p(d, øjst, arj #) denote the current estimate of the feature distribution. Given transition (St, (1, G1, St+1, Φi) the model is updated according to:
Besides the MF SFRQL update operator 16
E Φ: Po( a;; + Po(0lst. 04; T) (1 Pol st. a))
(41)
where 10=6, the indicator function that feature has been observed at time point and 3 [0, is the lcarning rate. After updating the model P.g it can be used for the É-update as defined in (3) Since the learned model Ps is independent from the reward function and from the policy, can be learned and used over all tasks.
19

---

Published in Transactions on Machine Learning Research (05/2023)
C
Experimental Details: Object Collection Environment
The object collection environment (Fig, a) was briefly introduced in Section 4.1 This section provides a formal description.
C.1 Environment
The enviromment is continuous two-dimensional area in which the agent moves. The position of the agent is point in the 2D space: (x, E [0, 1J2. The action space of the agent consists of four movement directions: A = {up, down, left, right}. Each action changes the position of the agent in a certain direction and is stochastic by adding Gaussian noise. For example, the action for going right updates the position according to Fz+1 = 24 + (N 0.05, = 0.005). Tf the new position ends in a wall (black areas in Fig. D a) that have a width of 0.04) or outside the environment, the agent is set back to its current position. Each environment has 12 objects. Each object has two properties with two possible values: color (orange, blue) and shape (box, triangle). If the agent reaches an object, it collects the object which then disappears. The objects occupy circular area with radius 0.04. At the beginning of episode the agent starts at location S with (zr, y)s (0.05, 0.05). An episode ends if the agent reaches the goal area G which is al position (z, y)G = (0.86, 0.86) and has a circular shape with radius 0.1. After an episode the agent is reset to the start position S and all collected objects reappear.
The state space of the agents consist of their position in the environment and the information about which objects they already collected during an episode. Following (Barreto et al.) 2017), the position is encoded using a radial basis function approach. This upscales the agent's (x, y) position to a high-dimensional vector Spos 1000 providing better signal for the function approximation of the different functions such as the to function.. The vector Spos is composed of the activation of two-dimensional Gaussian functions based on the agents position (z,y):
Spos exp (-
(ee-c,1)++ (u c3,2)
(42)
where Cj E 2K2 is the center of the jth Gaussian. The centers are laid out on a regular 10 x 10 grid over the area of the environment. The state also encodes the memory about the objects that the agent has already collected using binary vector mmem E {0, 1) The jth dimension encodes if the jth object has been taken (Smem. = 1) or not (Smem.j = 0). An additional constant term was added to the state to aid the function approxination. As a result, the state received by the agents is column vector with s s= ==sppss Hemm 1]" E IR
The features (fs ae, $142) E & C 10. in the environment describe the type of object that was collected by an agent during a slep or it reached the goal position. The first four feature dimensions encode binary the properties of a collected object and the last dimension if the goal area was reached. In total ( 6 possible features exists: 61 0, 0, O]T- standard observation, Φ [1, 1, collected orange bax, dy- [1,0,0, 0, 1, 1T collected an orange triangle, di [0, 1, 1, 0, T- collected blue box, Φ 1,0 T collected blue triangle, and ΦB = 0, 0, 1] reached the goal area.
Two types of tasks were evaluated in this environment that have either 1) linear or 2) general reward functions. 300 tasks, i.e reward functions, were sampled for each type. For linear tasks, the rewards r = o w, are defined by a linear combination of discrete features Φ E No and a weight vector w; E R The first four dimensions in w, define the reward that the agent receives for collecting objects having specific properties, e.g. being blue or being box. The weights for each of the four dimensions are randomly sampled from a uniform distribution: W ~ 1) for each task. The final weight defines the reward for reaching the goal state which is Ws = 1 for each task. For training agents in general reward tasks, general reward functions Rg for each task M, were sampled. These reward functions define for each of the four features (d2...., Φ5) that represent the collection of a specific object type an individual reward. Their rewards were sampled from uniform distribution: R, (0) (x(0)) and for reaching the goal area is (06) = 1 for all tasks. Reward functions of this form can not be linearly decomposed in features and weight vector.
(1-1, The reward for collecting no object is
20

---

Published in Transactions on Machine Learning Research (05/2023)
C.2 Algorithms
This section introduces the details of each evaluated algorithm. First the common elements are discussed before introducing their specific implementations.
All agents experience the tasks M E . of an environment soquentially. They are informed when a new lask starts. All algorithms receive the features o(s, a, 8') of the environment. For the action selection and [0. the agent performs a random
exploration, all agents use a e-greedy strategy. With probability action. Otherwise it selects the action that maximizes the expected return.
As the state space (s E R of the environments is high-dimensional and continuous, all agents use an approximation of their respective functions such as for the Q-function (Q(s.a) a) = Q(),a)) or the g-function (E(s, a, Φ) & E(8, a, o)). We describe the general function approximation procedure on the example of &- functions. not otherwise mentioned, all functions are approximated by a single linear mapping from the states to the function values. The parameters A E R.13x A|x|*) of the mapping have independent components of E 11s for each action a A and feature Φ: 0,
É(s, 0; = To a,
(43)
To learn ? we update the parameters g using stochastic gradient descent following the gradients Ve (05) of the loss basod on the SFRQL update [3
VHE o: fe((f)= = E (v(oe= = olst, as) + 75(1411, aet1, dz 05) Es ae &i 08))²}
(44)
with ēe+1 = arg max Σ ((+) a, oz;0),
where 0S = 05 but 0 is treated as a constant for the purpose of calculating the gradients Vo (O We used PyTorch for the computation of gradients and its stochastic gradient decent procedure (SGD) for updating the paramcters.
C.2.1 Learning of Features Representations
Besides the usage of the predefined environment features (Appendix C.1 the algorithms have been also evaluated with learned features on E 4n C RA. Learned features are represented by a linear mapping: on (st, S(41) s(c(st, $141) H. <(x) = 1+exp(-2) is an element-wise applied sigmoid function. The input is a concatenation of the current and next observation of the transition c(st,844) E 2g26 The learnable parameters H E 2226x define the linear mapping.
The learning procedure for H follows the multi-task framework described in [Barreto et al. 2017 based on CCrrunaa 1997).
learned to optimize the representation of the reward function r = Φ w; over several tasks. As the reward weights w, of a task M, are not known for learned features, they also need be learned per task. Both parameters (H, w) were learned on dataset. of transitions by the QL algorithm (Section C.2.2) during the initial 20 tasks of the object collection environment. Many of the transitions have reward of O which would produce a strong bias to learn features that always predict zero reward. Therefore, 75% of zero reward transitions have been filtered out. Based on the collected dataset the parameters H, and Wiel... were learned using a gradicnt decent optimization based on the following mcan squared error loss:
9141) (H, (.) = B(eLe { (5()())) H) w. ( -r:))
(45)
where Dj is the dataset of transitions from which the zero reward transitions were removed.
As each algorithm was evaluated for 10 runs (Section C.3 we also Iearned for cach run features from the dataset of transitions from the QL algorithm of the same run. The parameters H and Wiel, were optimized with Adam (learning rate of 0,003). They were optimized for 1, 000, 000 iterations. In each iteration, random batch of 128 transitions over all 20 tasks in the dataset was used. H and cach w, were initialized using a normal distribution with 0.05. Features da, with dimensions 4 and h were learned.
21

---

Published in Transactions on Machine Learning Research (05/2023)
C.2.2 QL
The Q-learning (QL) agent (Algorithm represents standard Q-learning (Watkins & Dayan 1992). The Q-function is approximated and updated using the following loss after each observed transition:
Lq(09)= =B (1( + max Q(1--1 0+11
((s),,;:89) als,, )
(46)
where 0 8 but 8 treated as a constant for the purpose of optimization, i. no gradients flow through it. Following Barreto et al. 2017) the parameters 89 are reinitialized for each task.
Algorithm 1: Q-lcarning (QL)
Input: exploration rate:
lcarning rate for the Q-function: a
for i ---1 to num tasks do initialize Q: 0 t small random initial values
new new _ew_cssodee-ee +- true for t t to num_steps do
if new episode then new bew_episode + false 8, f- initial state
With probability select random action de; hehrwii az arg max, ((5e, a) Take action and observe reward and next state 5e+1
if 8t41 is lerminal stale then
new new_episode true
Tt 0
else
L 7t 7
+- + 7e maxer: Q(5+11 0(+1)
Update 0 using SGD(a) with Lo (yy ((e, az)
$1 f- S111
C.2.3 SFQL
The classical successor feature algorithm (SF) is based on a linear decomposition of rewards in features and reward weights (Barrcto et al.) 2017) (Algorithm 2 If the agent lcarning the reward weights w, for task Mj then they are randomly initialized at the beginning of task. For the case of general reward functions and where the reward weights are given to the agents, the weights are learned to approximate a linear reward function before the task. See Section C.3 for description of the training procedure. After each transition the weights are updated by minimizing the error between the predicted rewards Φ(st, 8141) Twi w, and the observed reward ry:
Lw, (r(s,,,s') o(st, ar 8141) Tw:)) w:)
(47)
SF learns an approximated funcciioo for each task M,. The parameters of the o-function for the first task B are randomly initialized. For consccutive tasks, they initialized by copying them from the previous task (0 +- 0 The funncitioo of the current task M, updated after each observed transition with the loss based on
(0,) E ( (e, (:: 3s (st, ar (.,0))) 2
(48)
with 0et1 = arg max max
( a; 0.) W;
ke (1.2......
22

---

Published in Transactions on Machine Learning Research (05/2023)
where 0. = but 0." is treated constant for the purpose of optimization, i. gradients flow through it. Besides the current vi-function, SF also updates the #e-function which provided the GPT optimal action arg [1...( max; (s, 6) w,. The update uses the same loss as for the 48). but instead of using the GPI optimal action as next action, it usee the optimal action according to its own policy: a++1 arg max, 3he (s a) We
for the current transition:
update of the active -i-function
Algorithm 2: Classical SF Q-learning (SF) (Barreto al. 2017
Input:
exploration rate:
learning rate for diffnctions: a
learning rate for reward weights w: Ow
features & Ø
optional: reward weights for tasks: (W), W2.... W Wnam_tat tasks
for i 1 to num lasks do
if W; not provided then w; small random initial values
if i i then initialize eे: 0 + small random initial values else 0 + 0- new eeesodd + true for t 2- 1 to num mum_stops do if new episode then new new_episode +- false 8, < initial state
// GPI optimal policy With probability select random action otherwise + arg max, (e(s), a Take action and observe reward rz and next state 81+1 Update W; using SGD(aw) with Lw = (r: - Φ(3,, au st+1) w.)²
c c arg ma.., max, 4'x ()) a) Tw;
if St+1 a terminal state then new episode + true Th 0
else
L 7h- 7-7
// GPI optimal next action for task i
ā,ه1 + aarg max, (1 (x((0+1; TwT w;
y y 1-s(s, az, s141) (5441, ac-1) Update 0 using SGD(a) with Lo = (y (3(.
if c then
ā++1 + -rag max, (c(4441,a))) Tw
optimal next action for task
+ --1 01, S2+1) + (841)
Update t using SGD(a) with Lup = (y (e((),)))))
8t f 8141
C.2.4 SFRQL
The SFRQL agents (Algorithms 3 4 allow to reevaluate policies in tasks with general reward functions. If the reward function is not given, an approximation R, of the reward function for each task M, learned. The parameters for the approximation are randomly initialized the beginning each task. After each observed transition the approximation is updated according to the following loss:
Cx(OR) E (r(se, ar, S1+1) - (o( ae s1+1); 0))))))))))
(49)
In the case of tasks with linear reward functions the reward approximation becomes R. (o(s,, aes st41); R
o(st, ae St41) 8R. Thus with 8R w, we recover the same procedure as for SF (47). For non-linear, general reward functions represented R with neural network. The input of the network is the feature
23

---

Published in Transactions on Machine Learning Research (05/2023)
((5t, de, s6+1). The network has one hidden layer with 10 neurons having ReLu activations. The output is linear mapping to the scalar reward E IR.
All SFRQL agents learn an approximation of the E-ffunction for each task M,. Analogous to SF, the parameters of the f-function for the first task of are randomly initialized. For consecutive tasks, they are initialized by copying them from the previous task (6f + 6)_.). The fi-function of the current task M; is updated after each observed Transition with the loss given in 444p. The SFRQL agents differ in their setting for p(o, = dlst, az) in the updates which is described the upcoming sections. Besides the current E--function, the SFRQL agents also update the fe-function which provided the GPI optimal action for the current transition: c arg max.k maxa. Eeco (s ae, () (d)). The update uses the same loss for the update of the active fi-function (44). but instoad of using the GPI optimal action as next action, it uses the optimal action according to its own policy: 0c+1 = max. Eeeo Eo(#r+1, a, Rc()) Please note, when computing Q-values (EGED Ea(s, 6) (o)) the g-values are clipped 0 as they can not be negative: Eu(s, a, Φ) ma (x,,0, a,
The model-free SFRQL agent (Algorithm 3 uses stochastic update for the Ç- functions. Given a transition, we set p(ór = 11 a.) = for the observed feature Φ = ((8 Ct 8141) and p(ó: = 0|st, ax) = 0 for all other features Φ + (5 Gt. St41).
MF SFRQL (SFR):
MB SFRQL (MB SFR): The one-step SF mosel-based SFRQL agent (Algorithm 4 uses an approxi- mated model p to predict p(d; Φbst, a:) to reduce the variance of the g-function update. The model is by linear mapping for each action. It uses a softmax activation to produce a valid distribution over c:
exp(s T 0%.0)
ā(s, 0;00)= Evees exp(s AP
(50)
a.
where 8 E R13 As P is valid for each task in M, its weights Hr are only randomly initialized at the beginning of the first task. For each observed transition, the model updated using the following loss:
ve E x Ep(0P) = =E [ = @lst, or) - p(s,, ae o: o;)))²
(51)
where we set p(d; = oise, at) for the observed feature & = o(3,, a,, sc+1) and p(ø: = o/st, a)) = for all other features 6 + a(st, ae, 81+1).
Continuous SFRQL (CSFR): Under the condition of approximating feature representation (CSFR-h). the feature space becomes continuous. In this case are using discretized version of the SFRQL algorihm that is explained in detail under Appendix D.2.4. The agent uses for the object collection enviromment linear mapping to approximate the v-values given the observation s.
C.3 Experimental Procedure
All agents were evaluated in both task types (linear or general reward function) on 300 tasks. The agents experienced the tasks sequentially, each for 20.000 steps. The agents had knowledge when a task change happened. Each agent was evaluated for 10 repetitions to measure their average performance. Each repetition used a different random seed that impacted the following elements: a) the sampling of the tasks, b) the random initialization of function approximator parameters, c) the stochastic behavior of the environments when (aking steps, and d) the e-greedy action selection of the agents. The tasks, i.e. the reward functions, were different between the repetitions of particular agent, but identical to the same repetition of different agent. Thus, all algorithms were evaluated over the same tasks.
The SF agents (SF, SFRQL) were evaluated under two conditions regarding their reward model of a (ask. First, the reward weights or the reward function given to them. Second, that they have to learn the reward weights or the reward function online during the training (SF-R, SF-h, SFR-R, CSFR-h). As the SF does not support general reward functions, it not possible provide the SF agent with the reward function in the first condition. As solution, before the agent was trained new task M,, linear model of the
24

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 3: Model-free SFRQL (SFR)
Input:
exploration rate:
learning rate for g-functions:
learning rate for reward models R: aR
features & Ø optional: reward functions for lasks: {Rg, Re..... Rnam tasks
for i --- 1 to num tasks do
if Rd not provided then initialize RG: 0 + small random initial values if = then initialize Es: of < small random initial values else 0f + 05_1 new episode + true for t to num steps do if new episode then new new_episode +- false 8, initial state
arg (1.2.... max. Eg Es(st, a,,,,,,,, a, (o)
// GPI optimal policy
With probability e select random action ae, otherwise as + arg max, E. Ec(st, (,)
Take action a and observe reward r; and next state 8141
if R₂ not provided then L Update f.t using SGD(aR) with CR (re ac, 81+1))²
if st+1 is terminal state then
new episode +- true
%t 0
else L 7t 7-
// GPI optimal next action for task i
ā.t1 + arg max, argke(1,2.....) Eg Eu(s141, a, (o)
foreach @ E do if à = as,, 8141) then ys + 1 + 7ei 2.+1, else yo 71E((9441,, 0c+1, Φ)
Update of using SGD(a) with Le = Ey( - Esss x)
if c then
// optimal next action for task
ā++1 < arg mag, Eg a,,411,,,0) foreach & E + do if ó = Ó(s1, ar 9141) then yo + 1 + 7e
else Yo ( 0e,1,
Update o5 using SGD(a) with LE = Ey Ec(st, 2)
81
f- 8:41
25

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 4: One Step SF-Model SFRQL (MB SFR)
Input:
exploration rate: learning rate for f-functions:
learning rate for reward models R: aR
learning rate for the one-step SF model p: 3
features & or 6
optional: reward functions for tasks: { Ro..... Rnmm_tasks tasks
initialize jr: 0º + small random initial values for i 1-1 to num num_tasks do
if R; not provided then initialize RG: 0 < small random initial values if = then initialize E. of small random initial values else 0f 65 new episode true for t t to num steps do if new_epiee episode then new episodee +- false 8, initial state
C C arg [1. max, Eg (s a, (6)
// GPI optimal policy
With probability e select random action de, otherwise a < arg max, E. Eo( a, Φ) (d) Take action a and observe reward rz and next state 8141
if R₁ not provided then L Update Of using SGD(aR) with La (re Rt 8t+1)) foreach & do
L if b= 6($;, ar, s1+1) then Vs + else yo +
Update 6r using SGD(3) with Lp = E (ys - p( az, 6))² if 8t+1 is lerminal state then new new_episode +- true
%/- 0
else L 7 4-
// GPI optimal next action for task i
ā++1 +- arg max, argke(1.2....)) Eg a, ())(
forcach Φ E & do
L Uo - ae, Φ) + 7181(8411,44111 Update of using SGD(a) with LE Eylus - Es(,,, at, 2))2
if c then
// optimal next action for task
ātt1 + arg max, E. 0) do L Uo p(st, ar Φ) ( Ø))
foreach
Update Q using SGD(a) with LG Ey(yo Ec(ss, 2))²
81
8141
26

---

Published in Transactions on Machine Learning Research (05/2023)
reward (Ø) 6 ib was fitted. The initial approximation i; was randomly initialized and then fitted for 10.000 iterations using gradient descent procedure based on the absolute mean error (Ll norm):
Ar; = 17 |
Σ
R((0) - ΦTw,, Ws,
(52)
DE
with a learning rate of v 1.0 that yielded the best results tested over several learning rates.
The hyperparameters of the algorithms were to the same values as in (Barreto et al. 2017). A grid search over the learning rates of all algorithms was performed. Each learning rale was evaluated for three different settings which are listed in Table I If algorithms had several learning rates, then all possible combinations were evaluated. This resulted in different mumber of evaluations per algorithm and condition: Q- 3, SF- 3, SF-R- 9, SI-4 - 9, ST-8 9, SFR- 3, SFR-R - 9, CSFR-3 - 9, MB SFR- 9, MB SFR-R - 27. In total, 100 parameter combinations were evaluated. The reported performances in the figures are for the parameter combination that resulted in the highest cumulative total reward averaged over all 10 repetitions in the respective environment. Please note, the learning rates a and aw are set half of the rales defined in (Barreto al.) (017)). This is necessary due to the differences in calculating the loss and the gradients in the current paper. We use mean squared error loss formulations, whereas (Barreto et al, 2017) uses absolute error losses. The probability for random actions of the c-Groody action selection was set to = 0.15 and the discount rate to y 0.95. The initial weights @ for the function approximators were randomly sampled from a standard distribution with Oiit ~ N(u== 0, 0.01).
Hyperparameters:
Table 1: Evaluated Learning Rates in the Object Collection Environment
Parameter Description
Values
Learning rate of the Q, vi and g-function Learning rate of the reward weights the reward model Learning rate of the One-Step SF Model
{0.0025, 0.005, 0.025] {0.025, 0.05, 0.075) (0.2, 0.4, 0.6)
Ows 0R B
Computational Resources and Performance: Experiments were conducted on a cluster with a variety of node types (Xeon SKL Gold 6130 with 2.10GHz, Xeon SKL Gold 5218 with 2.30GHz, Xeon SKI. Gold 6126 with 2.60GHz, Xeon SKL Gold 6244 with 3.60GHl,, each with 192 GB Ram, GPU). The time for evaluating one repetition of a certain parameter combination over the 300 tasks depended on the algorithm and the task type. Linear reward function tasks: Q s 1h, SF s 4h, SF-R s 4h, SF-4 x 4h, SF-8 x 4h, SFR Ro 15h, SFR-R x 42h, CSFR-4 z 14h, MB SFR x 16h, and MB SFR-R v 43h. General reward function tasks: Q 1h, SF 5h, SF-R 4h, SF-4 xs 4h, SF-8 Rs 5h, SFR R 14h, SFR-R Rs 68h, CSFR-4 13h, MB SFR 18h, and MB SFR-R 67h. Please note, the reported times do not represent well the computational complexity of the algorithms, as the algorithms were not optimized for speed, and some use different software packages (numpy pytorch) for their individual computations.
C.4 Effect of Increasing Non-linearity in General Reward Task
We further studied the effect of general reward functions on the performance of classical SF compared to SFRQL (Fig. 1 c). We evaluated the agents in tasks with different levels of difficulty in relation how well their reward functions can be approximated by linear model. Seven difficulty levels have been evaluated. For each level, the agents were trained sequentially on 300 tasks as for the experiments with general reward functions. The reward functions for each level were sampled with the following procedure. Several general reward functions were randomly sampled as previously described. For each reward function, a lincar model of a reward weight vector w was filled using the same gradient descent procedure as in Eq. 52 The final average absolute model error after 10.000 iterations was measured. Each of the seven difficulty levels defines a range of model errors its tasks have with the following increasing ranges: {[0.0, 0.25), (0.25, 0.5)...., [1.5, each difficulty level, 300 reward functions were selected that yield linear model are the respective range of the level.
1.75)). For
27

---

Published in Transactions on Machine Learning Research (05/2023)
Q, SF, and SFR were each trained on 300 tasks, i.c. reward functions, on each difficulty level. As hyper- parameters were the best performing parameters from the previous general reward task experiments used. We measured the ratio between the total return over 300 tasks of Q-learning and MF SFRQL (Q/SFR), and SF and MF SFRQL (SF/SFR) Fig. (c) shows the results, using as x-axis the mcan average absolute model error defined by the bracket of each difficully level. The results show that the relative performance of SF compared to SFR reduces with higher non-linearity of the reward functions. For reward functions that are nearly linear (mean error of 0.125), both have a similar performance. Whereas, for reward functions that are difficult to model with linear relation (mean error of 1.625) SF reaches only less than 50% of the performance of MF SFRQT..
D Experimental Details: Racer Environment
This section extends the brief introduction to the racer environment (Fig. 2 a) given in Section 4.2
D.1 Environment
The environment is a continuous two-dimensional area in which the agent drives similar to a car. The position of the agent is a point in the 2D space: p (z, y) E [0, 1J2. Moreover, the agent has an orientation which it faces: 8 [-T, r1]. The action space of the agent consists of three movement directions: A (right, straight, left }. Each action changes the position of the agent depending on its current position and orientation. The action straight changes the agent's position by 0.075 towards its orientation 0. The action right changes the orientation of the agent to 0 + f and its position 0.06 towards this new direction, whereas left the direction to 1 changes. The environment is stochastic by adding Gaussian noise with a = 0.005 to the final position x, y, and orientation 0. Tf the agent drives outside the area (a, y) E [0, 1J2, then it reappcars on the other opposite side. The environment resembles therefore torus (or donut). As consequence, distances d(pa,P2) Py) are also measure in this space, so that the positions pz (0.1, 0.5) and Py (0.9, 0.5) have not distance of 0.8 but d(pz, Py) 0.2. The environment has 3 markers at the positions m1 = (0.25, 0.75), my = (0.75, 0.25), and m3 (0.75, 0.6). The fcatures mcasure the distance of the agent to each marker: & E R$ with dx = m. Each feature dimensions is normalized to be Φ E [0, 1]. At the beginning of an episode the agent is randomly placed and oriented in environment. An episode ends after 200 time steps.
The state space of the agents is similarly constructed as for the object collection environment. The agent's position is encoded with 10 10 radial basis functions Spos E R1100 defined in 42 In difference, that the distances are measure according to the torus shape. A similar radial basis function approach is also used to encode the orientation Sori E 20 of the agent using 20 equally distributed ganssian centers in [-T, x] and a = .T. Please note, π and - are also connected in this space, i.e. d(TT, -T) = 0. The combination of the position and oricntation of the agent is the final state: = [sposs TTil" E 120
The reward functions define preferred positions in the environment based on the features, i.c. the distance of the agent to the markers. A preference function 94 exists for each distance. The functions are composed of maximization over Gaussian components that evaluate the agents distance:
R(6) = Σ (m() with Ti == max exp (- (Øx 14,)
3
}
(53)
Jj
Reward functions are randomly generated by sampling the number of Gaussian components m to be 1 or 2. The properties of each component are sampled according to 2y ~ ((0.0,007), and oj 21(0.011,0000 Fig. 2 (a) illustrates one such randomly sampled reward function where dark areas represent locations with high rewards.
D.2 Algorithms
We evaluated Q, SF, SF-R, SF-h (sce Appendix C.2 for their full description) and CSFR, CSFR-R, and CSFR- h in the racer environment. In difference to their implementation for the object collection environment, they used a different neural network architecture to approximate their respective value functions.
28

---

Published in Transactions on Machine Learning Research (05/2023)
D.2.1 Learning of Feature Representations
Similar to the object collection environment, learned features were also evaluated for the racer environment with the procedure described in Section C.2.1 In difference, the dataset D' consisted of transitions from the Q agent from the initial 10 tasks of the racer environment. Moreover, zero reward transitions were not filtered as the rewards are more dense.
D.2.2 QL
Q-learning uses a fully connected leedforward network with bias and a ReLU activation for hidden layers. It has 2 hidden layers with 20 neurons each.
D.2.3 SFQL
SFQL uses a feodforward network with bias and ReL.U activation for hidden layers. has for each of the three feature dimensions separate fully connected subnetwork. Each subnetwork has 2 hidden layers with 20 neurons each.
D.2.4 Continuous Model-free SFRQL
The racer environment has continuous leatures e E R$. Therefore, the MF SFRQL procedure (Alg. 3 can not be directly applied as it is designed for discrete feature spaces. We introduce here MF SFRQL procedure for continuous feature spaces (CSFR). It is feature dimension independent, and discretized version of SFRQL. As the reward functions as:
53 are sum over the individual feature dimensions, the Q-value can be computed
(s, / R(o))"
ELe, rx((x) (s, ox))
(54)
where 4y is the feature space for each feature dimension which is t = [0, 1] in the racer environment. & is a E-function for the feature dimension &. (54) shows that the ç-function can be independently represented over each individual feature dimension ds, instead of over the full features Φ. This reduces the complexity of the approximation. Please note, when computing Q-values g-values are clipped at as they can not be negative: Ée(s, a, () max(0, (s, Ø))
Moreover, we introduce discretization of the f-function that discretizes the space of each feature dimension k in U = =11 bins with the centers:
maz min
Jiiin (
+ jAQ :0 =<jjK]] with AΦe
(55)
where AΦk is the distance between the centers, and amnin = 0.0 is the lowest center, and ΦF = 1.0 the largest center. Given this discretization and the decomposition of the Q-function according to 54 the Q-values can be computed by:
(T) = - R(x)5"(s, & XEX,
(56)
Alg. lists the complete CSFR procedure with the update steps for the g-functions. Similar to the SF agent, CSFR uses a feedforward network with bias and ReLU activation for hidden layers. It has for each of the three feature dimensions separate fully connected subnetwork. Each subnetwork has 2 hidden layers with 20 neurons each. The discretized outputs per feature dimension share the last hidden layer per subnetwork.
The fuunncion is updated according to the following procedure. Instead of providing a discrete learning signal to the model, we encode the observed feature using continuous activation functions around each bin center. Given the j'th bin center of dimension k, Fa.. its value is encoded to be 1.0 the feature value of this dimension aligns with the center (øx- 2kz). Otherwise, the encoding for the bin decreases lincarily based the distance between the bin center and the value (J dixl) and reaches 0 if the value is equal to
29

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 5: Mrdel-free SFRQL for Continuous Features (CSFR)
Input:
exploration rate: learning rate for f-functions:
learning rate for reward models R: oR
features & or & R
components of reward functions for Lasks: (R) = (rt, (r.,,r ra), Ra,.... Roum_taskss
discretization parameters: X, Ad
for i <- to num tasks do
if i then L VkE(I....)] initialize E): 0f +- small random values else
L VkE(I,...m)) 8 of new new_epii _ewieieddee__t true for t + 1 to mum steps do if new_episode then new episode false 8, + initial state
11
c arg m (1.2....) max, Ex=1 Exex. (s a, (2)
// GPI policy
With probability e select a random action de, hewwiie at f arg max, Ek=1 Ezex, (s,, a, (x)
&
Take action dr and observe reward rz and next state 8141 if St+1 is a terminal state then new episode +- true
7t 0
else
1. 7 4-
// GPI optimal next action for task
H ā.t1 +- aag max, ar8)E(1......)) Ek=1 ΣxEx, Sp as,
Φ, + o(s,, ar St+1) for ++11 1 to n do
foreach x E Xx do
yk..r max (, |=- (x- 7e (141,111,
Update of using SGD(a) with Le = En ExEx, (Uk,or - (8t, x)) if e then
// optimal next action for task
āzf1 < arg max, Σi= Erex, (E(,,, a, x))((())
for k + 1 to n do foreach x E Xx do
Y0,5 f- max 0, (x- + 2e (
Update of using SGD(a) with Le = =Eh ExEx, (Nex - (st, 2:)2
St + 8141
30

---

Published in Transactions on Machine Learning Research (05/2023)
neighboring bin center, i.. has distance 2 Adk. We represent this encoding for each feature dimension k by us E (0, 1) with:
(1.23) Voej<u uk.j = max (0, (1- [R.j - 0x.))
(57)
AΦR
To learn & we update the parameters H using stochastic gradient descent following the gradients V Cg( of the loss based on the SFRQL update 13
Vo E 4: (e((f)= E
Wu + 1 - Ex( ar: 05))
k=1
(58)
with O arg max Σ a4)
ZEXx
where = 3 is the number of feature dimensions and & is the vector of the U discretized g-values for dimension k.
D.3
Experimental Procedure
All agents were evaluated on 40 tasks. The agents experienced the tasks soquentially, each for 1000 episodes (200, 000 steps per task). The agents had knowledge when a task change happened. Each agent was evaluated for 10 repetitions to measure their average performance. Fach repetition used a different random seed that impacted the following elements: a) the sampling of the tasks, b) the random initialization of function approximator parameters, c) the stochastic behavior of the environments when taking steps, and d) the e-greedy action selection of the agents. The tasks, i.e. the reward functions, were different between the repetitions of particular agent, but identical to the same repetition of different agent. Thus, all algorithrs were evaluated over the same tasks.
SF was evaluated under two conditions. First, by learning reward weights w; with the iterative gradient decent mothod in (5)). The weights were trained for 10, 000 iterations with an learning rate 1.0. At cach iteration, 50 random points in the (ask were sampled and their features and rewards are used for the training step. Second, by learning the reward weights online during the training (SF-R).
Hyperparameters A grid search over the learning rates of all algorithms was performed. Each learning rate was evaluated for three different settings which are listed in Table 2 If algorithms had several learning rates, then all possible combinations were evaluated. This resulted in a different number of evaluations per algorithm and condition: Q- 4, SF- 4, SF-R - 12, SF-3 - 12, SF-6 12, CSFR- 4, CSFR-R 12, CSFR-3 12. In total, parameter combinat ions were evaluated. The reported performances in the figures are for the parameter combination that resulted in the highest cumulative total reward averaged over all 10 repetitions in the respective environment. The probability for random actions of the e-Greedy action selection set to 0.15 and the discount rate to 7 0.9. The initial weights and biases 0 for the function approxinators were initialized according to an uniform distribution with 0 ~ U( Vk. v Vk), where k features
Table 2: Evaluated Learning Rates in the Racer Environment
Parameter Description
Values
Learning rate of the Q v,, and g-function (0.0025, 0.005, 0.025, 0.53 Learning rate of the reward weights
O'w
{0.025, 0.05, 0.075)
Experiments were conducted on the same cluster as for the object collection environment experiments. The time for evaluating one repetition of a certain parameter combination over the 40 tasks depended on the algorithm: Q x 9h, SF s 73h, SF-R ss 70h, SF-3 20h SF-6 x 19h, CSFR z 88h, CSFR-R 80h, and CSFR-3 75h. Please note, the reported times do not represent well the computational complexity of the algorithms, as the algorithms were not optimized for speed, and some different. software packages (numpy pytorch) for their individual computations.
Computational Resources and Performance:
31

---

Published in Transactions on Machine Learning Research (05/2023)
E
Additional Experimental Results
This section reports additional results and experiments:
1. Evaluation of the agents in the object collection task by Barreto et al. (0017)
Report the total return and the statistical significance of differences between agents for all exper- iments
E.1 Object Collection Task by Barreto et al. (2017)
We additionally evaluated all agents in the object collection task by Barreto et al. 2017
The environment differs to the modified object collec- tion task (Appendix [ only in terms of the objects and features. The environment has 3 object types: orange, blue, and pink (Fig. 3 The feature encode if the agent has collected one of these object types or if it reached the goal arca. The first three dimensions of the features ((se, de; 8t+1) & C {0, 1]a encode which object lype collected. The last dimension encodes if the goal area was reached. In total |$| = possible features exists: Φ1 [0,0, 0, 0] - standard observation, Φ2 [1 1,0,0.0 0, 0 collected orange object, Ø3 [0, 0, 0T collected blue object, Φ4 = [0, 0, 0, collected a pink object, and db = [0, 0, 1| reached the goal area. Agents were also evaluated with learned features that have either a dimension of h 4 or = 8. The features were learned according to the procedure described in Appendix C.2.1
Environment:
G
S
Figure 3: Object collection
vironment from (Barreto et al., 2017) with 3 object types: or- ange, blue, pink.
The rewards r = o w; are defined by a lincar combination of discrete features Φ E N and weight. vector E R2. The first three dimensions in w define the reward that the agent receives for collecting object types. The final weight defines the reward for reaching the goal state which is W4 = 1 for each task. All agents were trained in on 300 randomly generated linear reward functions with the same experimental procedure as described in Appendix. C For each task the reward weights for the 3 objects are randomly sampled from a uniform distribution: W ~ ((-1,1).
of the
Results: The results (Fig. 4 follow closely the results from the modified object collection task (Fig. b. and 5 a). MF SFRQL (SFR) reaches the highest performance outperforming SF in terms of learning speed and asymptotic performance. is followed by MB SFR and SF which show no statistical significant difference between each other in their final performance. Nonetheless, MB has higher learning speed during the initial 40 tasks. The results for the agents that learn the reward weights online (SF-R, SFR-R, and MB SFR-R) follow the same trend with SFR-R outperforming SF-R slightly. Nonetheless, the E-agents have a much stronger learning speed during the initial 70 tasks compared to SF-R, due to the errors in the approximation the weight vectors, especially at the beginning of new task. The agent with approximated features (SF-4, SFR-h) have a similar performance that is below the performance of all other SF agents. All SF agents can clearly outperform standard Q-learning.
Please note, the SF agents with with approximated features show slighlty better performance in the experimental results by (Barreto et al. 2017) than in our experiments. (Barreto et al. 2017) does not provide the hyperparameters for the approximation procedure of the features. Therefore, difference between their and our hyperparameters could explain the different results.
E.2
Total Return in Transfer Learning Experiments and Statistical Significant Differences
Fig. shows for each of the transfer learning experiments in the object collection and the racer environment the total return that each agent accumulated over all tasks. Each dot besides the boxplot shows the total
32

---

Published in Transactions on Machine Learning Research (05/2023)
(a) Environment by Barreto et al 2017 with Linear Reward Functions
SE-R
SF-4
S1-8
SIR
SER-R
CSFR-4
MB SFR
MB SFR-R
Average Return
400
200
10
20
30
40
50
60
80
90
100
110
120
130
140
150
160
170
180
190
200
210
220
230
240
250
260
270
280
290
300
Tasks Trained
Total Return and Statistical Significance Tests
Q
SF
SF-R
SF-4
SF-8
SFR
SFR-R
CSFR
MB SER
MB SFR-R
250k
Return
200k
150k
100k
SI
p-value
Q
SF-R
SF
0.001
< 0,001
SP-R
0000
SF-4
00000
0.000
1000
100
0.002
SF-8
0.001
00000
< 0.001
< 0,000
SFR
0,000
SFT
S000
0.001
0.000
CSFR-4
< 0.001
0.002
0.001
MB SFR
0.121
0.001
0.001
MB SFR-R
0.076
0.001
00001
SF-4
0.001
<
0.001
SF-8
< 0.001
0.001
0.473
0.001
0.001
SFR
0.001
0.001
0.001
0.001
SFR-R
0.001
0.001
0.623
CSFR-4
0.001
0.001
MB SFR
0.001
Figure 4: MF SFRQL (SFR) outperforms SF in the object collection environment by Barreto et al. (011)
both in terms of asymptotic performance and learning speed. (a) The average over 10 runs the average reward per task per algorithm and the standard error of the mean are depicted. tasks in each evaluated condition. The table shows the p-values of pairwise Mann-Wiin WhintWy U tests between the agents.
(b) Total return over the 300
return for each of the 10 repetitions. The box ranges from the upper to the lower quartile. The whiskers represent the upper and lower fence. The mean and standard deviation are indicated by the dashed line and the median by the solid line. The tables in Fig. report the p-value of pairwise Mann Whitney test. A significant different total return can be expected if p 0.05.
For the object collection environment (Fig b), SFRQL (SFR) outperforms SF in both conditions, in tasks with linear and general reward functions. However, the effect is stronger in tasks with general reward functions where SF has more problems to correctly approximate the reward function with its linear approach. For the conditions, where the agents learn a reward model online (SF-R, SFR-R, MB SFR-R) and where the features are approximated (SF-h, CSFR-h) the difference between the algorithms in the general reward case is as strong due the effect of their poor approximated reward models.
In the racer environment (Fig 5 c) SF has poor performance below standard Q-learning it not appropriately approximate the reward functions with a lincar model. In difference CSFR outperforms Q. Also in this environment for the conditions, where the agents learn a reward model online (SF-R, CSFR-R) and where the features are approximated (SF-h, CSFR-h) the difference between the algorithms is not as strong due the effect of their poor approximated reward models.
33

---

Published in Transactions on Machine Learning Research (05/2023)
(a) Object Collection Enviromment with Linear Reward Functions
CSFR-4
200k
Total Return
150k
100k
p- vavuuue
SF
SP-R
SF-4
SFR
1007
CSFR-4
MB SER
MB SPR-R
SF
0.001
< 1000
0,001
2
0.001
0.000
> 0.001
0.001
0.001
0.000
< 0.001
< 0.001
0000
0.001
1000
0000
0,000
0.001
0.000
0.001
0.001
0.001
SF-4
0.970
0.001
0.241
SF-8
< 0.001
00000
0.000
0.001
0.001
SFR
0.001
0.186
0.001
SFR-R
0.011
0.001
0.054
CSFR-1
MB SFR
0.001
(b) Object Collection Environment with General Reward Functions
SF-4
SFR-R
CSFR-4
MB
MB SFR-R
250k
Total Return
200k
150k
100k
50k
p-value
SF-R
SF-8
SFR-R
CSFR-4
SFR-R
Q
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
0.001
< 0.001
2 0.001
< 0.001
< 0.001
0.001
< 0.001
< 0.001
SF-R
SF-4
0.104
1000
0.001
0.186
0.623
< 0.001
0.017
SF-8
0.001
< 0.001
0.001
< 0.001
< 0.001
SFR
0.001
0.000
< 0.001
0,001
SFR-R
0.121
< 0.001
0.241
CSFR-4
0.001
0.007
MB SFR
< 0.001
c Racer Environment with General Reward Functions
SF-R
SF-3
SF-6
CSFR
CSER-R
CSFR-3
3M
Total Return
2M
1.5M
p-value
SE-R
SF-3
3F-6
CSFR
CSFR-R
CSFR-3
0.001
0.001
0000
0.001
0,000
1,000
1000
0.001
0.001
0.001
SF
0.021
1000
1000
0.001
0.014
0.089
SF-R
0.001
0.623
0.001
SF-3
0.031
0.001
0.001
0.001
SF-6
CSPR
0.001
0.001
CSFR-R
0.001
Figure 5: Total return over all tasks in each evaluated condition. The tables show the p-values of pairwise Bonn-WWitney U tests between the agents. See the text for more information.
34


Reviews:
[{"review_id": "TA6DifoYfZ", "reviewer": "Reviewer_6PrE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and follows a solid structure. The experiments seem rigorous and the observations are fairly well supported by intuitive reasoning. \n\nI do not think the paper adds enough value to garner interest for a TMLR audience. As I see it, the only difference from the usual SF formulation is not using the reward being linear-in-$\\phi$ assumption. Having described this variation, the paper does not do anymore to solve for the issues that arise with this new variation, i.e. 1) Features are not learnt but assumed given, at least for cases when SFR performs better than SF, and 2) For the continuous case, features are discretized so as to handle the summation over the SF and the R.  \nOverall, I don't think the paper offers much useful insight for the community. It more or less has the same limitations as the prior SF work, i.e. features are assumed given and the continuous case is not handled well beyond simple discretization.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a vairation on the Successor Features formulation where the rewards are not assumed to be linearly representable using the features $\\phi$. They derive an analogus TD-based update for such a version and then prove that Q-learning converges when assuming the $\\psi$ analog is approximated well. The authors finally test their algorithm, which they call SFRQL, on two fairly simple domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Since my main qualm is with the motivation of the project, I unfortunately do not have any detailed suggestions for securing my recommendation for acceptance. I believe that would require the authors showing results/comments showcasing where one would gain using SFRs over SFs (besides the linear reward vs general reward case). I also think assuming access to given features is a harsh contraint to have. If the authors can convince the reader that there is an actual case for their exact setting to be useful, I would be open to discussing further.", "broader_impact_concerns": "Not Applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uBce3MO9OB", "reviewer": "Reviewer_p3ha", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The algorithm is very effective compared to the baseline.\n2. The author provides the code, and the code is more extensively documented.\n\n\nWeakness:\n1.  In the abstract, \"But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to such tasks.\" can you explain in more detail why having this linear relationship would result in limiting its application to such tasks?\n2. In the summary of the author's contribution, the authors say that a new algorithm is proposed and provides proof of convergence and better experimental results. I note that the algorithm combines Successor Representations (SR) and Successor Features (SF). The authors should have elaborated more on the shortcomings of the original algorithm and the limitations of the currently available algorithms that this algorithm can address. I noticed that the introduction was partly written, but it was more of a declarative expression of existing work and lacked sufficient discussion.\n3. The experimental scenario is too simple, the authors only did experiments on two environments, and I think the conclusions drawn from too simple experiments are insufficient. As far as I know, 'Successor Features for Transfer in Reinforcement Learning' contains other experimental settings, as well as some experimental scenarios for subsequent work, etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The author designed a new algorithm SFRQL, based on a cumulative discounted probability of successor features and provided theoretical proof of the convergence of SFRQL to the optimal policy and for a guarantee of its transfer learning performance under the GPI procedure.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The motivation of this article is not clear enough and I expect the authors to elaborate more and be able to enrich the comparison of experiments.", "broader_impact_concerns": "All of my concerns have been stated.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JtwbD18iF7", "reviewer": "Reviewer_pd9Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and the results, as far as I could tell, are correct. The SFR feature is somewhat derivative in that it builds on SRs and SFs quite heavily (even for the theoretical results), but as far as I can tell it is novel.\n\nSome of the mathematical notation needs to be tidied up (see comments below), but overall I didn't find any major issues.\n\nIn summary, I do think this is an interesting and valuable contribution to the community, so am supportive of it being accepted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Successor Feature Representations (SFRs), which are a combination of ideas from Successor Representations (SRs) and Successor Features (SFs), which can be used in problems where reward functions change between tasks, but dynamics remain the same. It extends the SF framework by relaxing the assumption of linearity of the rewards with respect to the features, and permitting general reward functions. Using this new formulation, the authors introduce the SFRQL algorithm, prove its convergence to the optimal policy, and evaluate it empirically.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Suggested changes / additions\n\n1. In the experiments, would it make sense to do policy evaluation, with different reward funcitons, using the learned SFRs? This would give a sense of their expressivity.\n1. In the proof of Theorem 1 you say \"Indeed, we rewrite the operator above as...\". Please spell out this rewrite, as it's not immediately clear they're equal.\n1. In the proof of theorem 1, please clarify where \"up to an additive kernel funciton $\\kappa$\" came from, as it's not clear.\n\n# Questions / suggestions\n\n1. In equation 9 in the $\\min_j$ term, does $j$ range over the set of available tasks?\n1. In equation 9, $i$ is not qualified in the RHS\n1. In equation 10, shouldn't the $s_{t+1}$ term in $\\phi$ be features, rather than states?\n1. In the LHS of equation 11, is it still $\\phi\\equiv\\phi(s_t, a_t, s_{t+1})$ like you just defined above it?\n1. Same question for the transition denoted just above equation 13.\n1. Are the right plots in Figure 1 evaluated on the training tasks, or on separate tasks?\n1. In the proof of theorem 1 where you rewrite the operator, shouldn't the $\\epsilon$ term also depend on $\\pi$?\n\n# Minor recommendations\n\n1. In the second paragraph of page 2: \"This assumption allows **us** to effectively ...\"\n1. Top of page 3: \"... defining which action to take in **each** state to maximise...\"\n1. In the end of the first paragraph of section 6: \"where an agent has **to** continually adapt...\"\n", "broader_impact_concerns": "None that I noticed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces Successor Feature Representations (SFRs), which can be used in problems where reward functions change between tasks, but dynamics remain the same. It extends the SF framework by permitting general reward functions and introduces the SFRQL algorithm, which is proven to converge to the optimal policy.
All the reviewers agree that the paper is well-written and clearly organized, and the empirical results show significant improvements over the considered baselines.
The main concerns are related to the incremental nature of the contribution and the simplicity of the experimental scenarios.
Even if I partly share these concerns, I think the paper deserves publication.
I ask the authors to update the paper according to what they have written in the rebuttals and to carefully consider reviewers' suggestions for improving the paper.
Decision: Accept (poster)