Xi-learning: Successor Feature Transfer Learning for General Reward Functions
Authors: Chris Reinke
Xavier Alameda-Pineda
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
transfer learning
meta learning
successor features

Abstract: Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor features (SF) are a prominent transfer mechanism in domains where the reward function changes between tasks. They reevaluate the expected return of previously learned policies in a new target task and to transfer their knowledge. A limiting factor of the SF framework is its assumption that rewards linearly decompose into successor features and a reward weight vector. We propose a novel SF mechanism, $\xi$-learning, based on learning the cumulative discounted probability of successor features. Crucially, $\xi$-learning allows to reevaluate the expected return of policies for general reward functions. We introduce two $\xi$-learning variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on $\xi$-learning with function approximation demonstrate the prominent advantage of $\xi$-learning over available mechanisms not only for general reward functions, but also in the case of linearly decomposable reward functions.

Under review as a conference paper at ICLR 2022
SUCCESSOR FEATURE TRANSFER LEARNING FOR GENERAL REWARD FUNCTIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor features (SF) are a prominent transfer mechanism in domains where the reward function changes between tasks. They reevaluate the expected return of previously learned policies in a new target task and to transfer their knowledge. A limiting factor of the SF framework its assumption that rewards linearly decompose into successor features and reward weight vector. We propose novel SF mechanism, &- learning, based on learning the cumulative discounted probability of successor features. Crucially, Ç-learning allows to reevaluate the expected return of policies for general reward functions. We introduce two E-learning variations, prove its convergence, and provide a guarantee its transfer performance. Experimental evaluations based on g-learning with function approximation demonstrate the prominent advantage of Ç-learning over available mechanisms not only for general reward functions, but also in the case of linearly decomposable reward functions. |
INTRODUCTION
Reinforcement Learning (RL) successfully addressed many complex problems such as playing computer games, chess, and even Go with superhuman performance (Mnih al. 2015 Silver et al. 2018). These impressive results are possible thanks to a vast amount of interactions the RL agent with its environment/aakk Such strategy is unsuitable in settings where the agent has to perform and learn at the same time. Consider, for example, a care giver robot in a hospital that has to learn a new task, such new route to deliver meals. In such setting, the agent can not collect vast amount of training samples but has to adapt quickly instead. Transfer learning aims to provide mechanisms quickly adapt agents in such settings (Taylor and Stone 2009 Lazarie 2012 Zhu al.. 2020). The rationale is to use knowledge from previously encountered source tasks for a new target task improve the learning performance on the target task. The previous knowledge can help reducing the amount of interactions required to learn the new optimal behavior. For example, the care giver robot could reuse knowledge about the layout of the hospital it learned in previous source tasks (e.g. guiding a person) to learn to deliver meals.
The Successor Feature (SF) and General Policy Improvement (GPI) framework (Barreto al. 2020) is prominent transfer learning mechanism for tasks where only the reward function differs. Its basic premise is that the rewards which the RL agent tries to maximize are defined based on low- dimensional feature descriptor @ E " For our care-giver robot this could be ID's of beds or rooms that it is visiting, in difference to its high-dimensional visual state input from a camera. The rewards then computed not based on its visual input but on the ID's 's of the beds or rooms that it visits. The expected cumulative discounted successor features (1)) are learned for each behavior that the robot learned in the past. It represents the dynamics in the feature space that the agent experiences for behavior. This corresponds to the rooms or beds the care-giver agent would visit if using the behavior. This representation of feature dynamics is independent from the reward function. A behavior learned in previous task and described by this SF representation can be directly re-evaluated for a different reward function. In new task, i.c. for new reward function, the GPI procedure re-evaluates the behaviors learned in previous tasks for it. It then selects at each state the behavior of a previous task it improves the expected reward. This allows to reuse behaviors learned in previous source tasks
Source code at wtt.ttss://// /ttttps://tiyyrr// com 3xuzxff3

---

Under review as a conference paper at ICLR 2022
for a new target task. A similar transfer strategy can also be observed in the behavior of humans (Momennejad et al. 2017 Momennejad 2020) Tomov et al. (021).
The classical SF&GPI framework (Barreto et al.
2017 2018) makes the assumption that rewards are a linear composition of the features & E I" via a reward weight vector w; E I" that depends on the task i: r; = o w,, This assumption allows effectively separate the feature dynamics of behavior from the rewards and thus to re-evaluate previous behaviors given a new reward function, i.e. new weight vector w.. Nonetheless, this assumption also restricts successful application of &PI only to problems where such linear decomposition is possible. We investigate the application of the SF&GPI framework to general reward functions: r; = (Ø) over the feature space. We propose to learn the cumulative discounted probability over the successor features, named & f-function, and refer to the proposed framework as f-learning. Our work related to Janner et al. (2020); (Touati and Ollivier (2021), and brings two important additional contributions. First, we provide mathematical proof of the convergence of Ç-learning- Second, we demonstrate how g-learning can be used for meta-RL, using the g-function to re-evaluate behaviors learned in previous tasks for new reward function Ry. Furthermore, E-learning can also be used to transfer knowledge to new tasks using GPI.
The contribution of our paper is three-fold:
We introduce a new RL algorithm, ç-learning based on a cumulative discounted probability of successor features, and two variants of its update operator.
We provide theoretical proofs of the convergence of g-learning to the optimal policy and for a guarantee of its transfer learning performance under the GPI procedure.
We experimentally compare g-learning in tasks with linear and general reward functions, and for tasks with discrete and continuous features to standard Q-learning and the classical SF framework, demonstrating the interest and advantage of g-learning.
BACKGROUND
2.1 REINFORCEMENT LEARNING
RL investigates algorithms to solve multi-step decision problems, aiming to maximize the sum over future rewards (Sutton and Barto 2018). RL problems are modeled as Markov Decision Processes (MDPs) which are defined as a tuple M = (S,A, R, 7), where S and A are the state and action set. An agent transitions from a state 8, to another state 81+1 using action a time point I collecting a reward rr: St "e'"t, 81+1- This process is stochastic and the transition probability p( ((st+1 az) describes which state St+1 is reached. The reward function defines the scalar reward = R(ss.. dg. $141) IR for the transition. The goal in an MDP is to maximize the expected return G. = E |Σ 7 Re++]. where R, = R(S,, A,, St+1). The discount factor y E [0, 1) weights collected rewards by discounting future rewards stronger. RL provides algorithms to learn a policy π S - A defining which action to take in which state to maximise G.-
Vasue- RI. methods the concept of value functions to learn the optimal policy. The state- action value function, called Q-function, is defined as the expected future return taking action ae in st and then following policy æ:
(5() = Ex fre + 7T+t1 + + 7 + +...) E, {
(1) The Q-function can be recursively defined following the Bellman equation such that the current Q-value (?T((s,, az) depends on the maximum Q-value of the next state (s++11 a2+1). The optimal policy for an MDP can then be expressed based on the Q-function, by taking every step the maximum action: x² -s) argmax, (s.ss) a).
+ max Q ( a
The optimal Q-function can be learned using a temporal difference method such as Q-learning (Watkins and Dayan 1992). Given a transition (8t, az, Tt, 1), the Q-value is updated according to:
a.) = Qu(se, at) + au r: + max Qu Qu(56+1, 4441) De(st,at) as+1
(2)
where on (0, 1] is the learning rate at iteration k.
2

---

Under review as a conference paper at ICLR 2022
2.2 TRANSFER LEARNING AND THE SF&GPI FRAMEWORK
We are interested in the transfer learning setting where the agent has to solve a set of tasks M {M,, .,,..., Mm), J. that in our case differ only in their reward function. The Successor Feature (SF) framework provides principled way to perform transfer learning (Barreto et al. 2017: 2018).
SF assumes that the reward function can be decomposed into a linear combination of features ④ E 4 c. R" and a reward weight vector W; E R" that is defined for a task M,:
(1 Ce. St+1) c(st. at, St+1) W.
(3)
We refer to such reward functions as linear reward functions. Since the various tasks differ only in their reward functions, the features are the same for all tasks in M.
Given the decomposition above, it is also possible to rewrite the Q-function into an expected discounted sum over future features (s, a) and the reward weight vector W.
Q = E Err + + + r + +...) E w. + I wi + 22 .... E 1 w, = (s. a)
w. Tw:
(4)
This decouples the dynamics of the policy ny in the feature space of the MDP from the expected rewards for such features. Thus, is now possible to evaluate the policy n; in a different task M, using a simple multiplication of the weight vector w with the v-function: Q7 (s, a) s: (s, a) w Interestingly, the vp function also follows the Bellman equation:
2² (s, a) = E { + r (s1+1, ग ((s
(5)
and can therefore be learned with conventional RL methods. Moreover, (Lehnert and Littman showed the equivalence of SF-learning to Q-learning
2019
Being in a new task M, the Generalized Policy Improvement (GPI) can be used select the action over all policies learned so far that behaves best:
(()) argmax max Q7,' (s, argmax max (s,
w, Tw,,
(Barreto et al. 2018) proved that under the appropriate conditions for optimal policy approximates, the policy constructed in (6) close to the optimal one, and their difference is upper-bounded:
(Ir-ralloo + min ||r - ryllo + +c)
||Q - 17
where Ilf 1 00 max,,a (f a) gs, For an arbitrary reward function the result can be interpreted in the following manner. Given the arbitrary task M, we identify the theoretically closest possible linear reward task M, with r;. For this theoretically closest task, we search the linear task M, in our set of task M (from which we also construct the GPI optimal policy (6)) which is closest to it. The upper bound between Q" and Q is then defined by 1) the difference between task M and the theoretically closest possible linear task M;: || rilloci and by 2) the difference between theoretical task M, and the closest task My: min, ||r, rylloo If our new task M is also linear then r rú and the first term in would vanish.
Very importantly, this result shows that the SF framework will only provide a good approximation of the true Q-funttion if the reward function in task can be represented using linear decomposition. If this is not the case then the error in the approximation increases with the distance between the true reward function and the best linear approximation of it r, stated by ||r rilloc-
METHOD: E É-LEARNING
3.1 DEFINITION AND FOUNDATIONS OF É-LEARNING
The goal of this paper is to investigate the application of SF&GPI to tasks with general reward functions R: T + I. over state features & E 4:
r(s,, 04 8141) R(0(8t, ae, 81+1)) R(d.);
(8)
3

---

Under review as a conference paper at ICLR 2022
where define Φ Ō(st, at, (t, S1+1). Under this assumption the Q-function not be linearly decomposed into a part that describes feature dynamics and one that describes the rewards as in the linear SF framework (3). To overcome this issue, propose to define the expected cumulative dis- counted prohability of successor features or g-function, which going to be the central mathematical object of the paper, as:
(" a, Φ)
Σ
* p(oc+* = -ost = 8.00 = a: aT:T)
k=
where p(or+ = =|s; = s, ar = a), , or in short p(cr+* Øsst, oe; rT,, is the probability density function of the features time t + k. following policy and conditioned to and a being the state and action at time respectively. Note that 5" depends not only on the policy т but also on the state transition (constant through the paper). With the definition of the ξ-function, the Q-function rewrites (this is compatible with SFQL in the linear reward case, see Appendix A.6):
aii L. p(oc+ a;:))
Q" (st, az)
1 (l( [R(or=))}
Σ
Σn **
k=0
k=0
(10)
-f R())
/ R(o)E= az 0)d0.
Σ * p(or++ = a13 a5;
km0
Depending on the reward function R, there are several g-functions that correspond to the same Q function. Formally, this is equivalence relationship, and the quotient space has one-to-one correspondence with the Q-function space.
Proposition 1. (Equivalence between functions & and Q) Let Q = {Q : x A R s.t. ||Q| < )o). Let ~ be defined as & ~ &2 + So REI = So RE2. Then, ~ is an equivalence relationship, and there is bijective correspondence between the quotient space E. and Q Corollary 1.
The bijection between E., and allows induce a norm || 11~ into E.. from the supremum norm in Q, with which E. is Banach space (since Q is Banach with || 11
| = sup / R(O)E(s, o)do sup Q(s, a)l blo s.a
sup
(11)
8,0
Similar to the Bellman equation for the Q-function, we can define a Bellman operator for the f-function, denoted by Tg. as:
Tg(5") = p(or = Ølst, at) + yEp (a (s141, 0)11,0))
(12)
As in the of the Q-function, Proposition 2. (E-learning has a fixed point) The operator Tg is well-defined w.r.f. the equivalence ru and therefore induces operator T- defined over E.. T~ is contractive w.r.t. Il Since E Banach, Tw, has a unique fixed point and iterating T- starting anywhere converges to that point.
use Ts construct contractive operator:
In other words, successive applications of the operator T~ converge towards the class of optimal functions [**] or equivalently to an optimal & function defined up to an additive function satisfying Jp k(s,a, a, Φ) R( 0, ((s a) S A (i.e. h E Ker(g - RE)).
While these two results state (see Appendix A for the proofs) the theoretical links to standard Q-learning formulations, the Te operator defined in (12) is not usable in practice, because of the expectation. In the next section, we define the optimisation iterate, prove its convergence, and provide two variants to perform the & updates.
3.2 Ş-LEARNING ALGORITHMS
In order to learn the Ç-function, we introduce the Ç-learning update operator, which an off- policy temporal difference method analogous to Q-learning. Given a transition (st, at S141, Φi) the E-learning update operator is defined as:
& (8,, az, ④) +- E (st, de, Φ) + an (p(0: = Φ15,, uz) + 78 (81+1, - (8r, 0r, [)], (13)
where āe+1- = argmax, So (1441, 1,4,,, a, d)
The following of the main results of the manuscript, stating the convergence of E-learning:
4

---

Under review as a conference paper at ICLR 2022
Theorem 1. (Convergence of E-learning) For sequence of state-action (st, at, S1+1, 6t tizo consider the E-learning update given in (13). If the sequence of state-action cere-accion-fffnaee ffature triples visits each state, action infinitely often, and if the learning rate a is adapted sequence satisfying the Robbins-Mon Monro conditions:
02
(14)
k=1
then the sequence of function classes corresponding to the iterates converges to the optimum, which corresponds to the optimal Q-function to which standard Q-learning updates would converge to:
with Q"(s.a)= / (s,
[cn] + 15T)
(15)
The proof is provided in Appendix A and follows the same flow for Q-learning
The previous theorem provides convergence guarantees under the assumption that either p(ór = ølst, (e: T)) is known, an unbiased estimate can be constructed. We propose two different ways to approximate p(ot- = 6|8t, de; (e)7)) from a given transition (8t, Oz, 1 (x) so as to perform the É-update (3) The first instance is model-free version and detailed in the following section. A second instance uses one-step SF model, called One-Step Model-bass based (MB) E-learming, which is further described in Sec. B
Model-free (MF) f-Learning: MF & g-learning uses the same principle as standard model-free temporal difference learning methods. The update assumes for given transition (8t, de; 11+1, , dt) that the probability for the observed feature is p(o = de|st, az) = 1. Whereas for all other features (Vo' E 4, ø' * Φi) the probability is p(ø' = se ar) = O, see Appendix D for continuous features. The resulting updates are:
Φ= D:: (T(s,, ar, ②) or: {"(st, (6, t (1- a) ar, 0') + a 0++1,6').
(I- ai の) + a (1+ 78 ($141, o))
(16)
Due to the stochastic update of the É-function and if the learning rate (0, 1] discounts over time, the f-update will learn the true probability of p(ø = Ød|st, 4t). A potential problem with the MF procedure is that might induce a high variance when the true feature probabilities are not binary.
3.3 META L-LEARNING
After discussing Ç-learning single task and showing its theoretical convergence, we can now investigate how can be applied in transfer learning. Similar to the linear SF framework the f-function allows to reevaluate policy learned for task M,, { in new environment My:
(s. = -J R,(()) (s, Φ)
(17)
This allows us to apply GPI in 66) for arbitrary reward functions in a similar manner to what was proposed for linear reward functions in (Barreto al. 2018). We extend the GPI result to the Ç-learning framework as follows:
Theorem 2. (Generalised policy improvement in g-learning) Let M be the set of tasks, each one associated (possibly different) weighting function R; L' L Let ş"î be representative of the optimal class of f-fucctonns for task M,, i [I,...,I}. 1}. optimal E-function, I|g [" I S.Vi, Vi. Then, for another task M with weighting function R, the policy defined as:
and let ETh be an approximation the
x(s) arg max max
R(o)E** o)do,
(18)
satisfies:
2
11 -5 S I γ
(min || R - Rd Rellp(ole.)) + +s),
(19)
where [[fllg -uullg So If gl dø.
The proof is provided in Appendix
5

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
We evaluated Ç-learning in two environments. The first has discrete features. It is modified version of the object collection task by Barreto al. (2017) having more complex features to allow general reward functions. See Appendix I. for experimental results in the original environment. The second environment, the racer environment, evaluates the agents tasks with continuous features.
4.1 DISCRETE FEATURES OBJECT COLLECTION ENVIRONMENT
Environment:
The environment consist of 4 rooms (Fig. I a). The agent starts episode in position S and has to learn to reach the goal position G. During an episode, the agent can collect objects to gain further rewards. Each object has 2 properties: 1) color: orange blue, and 2) form: box or triangle. The state space a high-dimensional vector s E R 12 encodes the agent's position using 10 10 grid of (wo-dimensional Gaussian radial basis functions. Moreover, includes a memory about which object has been already collected. Agents can move in directions. The features + (0, 1)5
binary vectors. The first dimensions encode if orange blue object was picked up. The 2 following dimensions encode the form. The last dimension encodes the agent reached goal G. For example, Ø [1, 0, 1, 0, 0] encodes that the agent picked up an orange box.
Fach agent learns sequentially 300 tasks which differ in their reward for collecting objects. We compared agents in two settings: either in tasks with linear or general reward functions. For each linear task M,, the rewards vector w; RR. The weights wi, for the first 4 dimensions define the rewards for collecting object with specific property. They are randomly sampled from a uniform distribution: Wi,k ~ U4(-1, The final weight defines the reward for reaching the goal position which is wi,5 1 for each task. The general reward functions are sampled by assigning different reward to each possible combination of object properties dy E & using uniform sampling: (2 U(-1, 1), such that picking up an orange box might result in a reward of (4 [1, 0, 1, 0) 0.23.
Tasks:
Φ w; are defined by a linear combination of features and a weight
(a) Collection Environment
(b) Tasks with Linear Reward Functions
SEQL
MB
Average Return
600
400
200
100
260
Tasks Trained
(c) Effect of Non- Non-Linearity
(d) Tasks with General Reward Functions
QL Xi
SEQL/XX Xi
QT.
SPQ
MP xi
MB Xi
Total Return Ratio
Average Return
800
600
0.5
400
0.125
0.375
0.025
0.875
11225
1.375
G25
20
40
60
80
100
120
140
160
180
200
220
240
260
280
300
Mean Error Linear Model
Tasks Trained
Figure In the (a) object collection environment, E-learning reached the highest average reward task for (b) linear, and (d) general reward functions. The average over 10 runs per algorithm and the standard error of the mean are depicted. (c) The performance difference between g-learning and SFQL is stronger for general reward tasks that have high non-linear,ty, i.e. where linear reward model yields a high error. SFQL can only reach less than 50% of MF É-learning's performance in tasks with mean linear reward model error of 1.625.

---

Under review as a conference paper at ICLR 2022
Agents: We compared f-learning Q-learning (QL), and classical SF Q-learning (SFQL) (Barreto et al. 2017). All agents use function approximation for their state-action functions (Q, v. or {- function). An independent linear mapping is used map the values from the state for each of the actions. As the features are discrete, the {-function and mgg-o are approximated by an independent mapping for each action and possible feature 4. The Q-value Q a) for the Ç-agents (Eq. I0) is computed by: Q (s, a) = Eoeo R( a, Φ). The reward functions of each task are given to the E-agents. For SFQL, the sampled reward weights w, were given in tasks with linear reward functions. For general reward functions, linear model & w. approximating the rewards was learned for each task and its weights W, given to SFQL (see Appendix C.3 for details), Each tasks was executed for 20, 000 steps, and the average performance over 10 runs per algorithm was measured. We performed a grid-search over the parameters of each agent, reporting here the performance of the parameters with the highest total reward over all tasks.
Results: E-learning outperformed SFQL and QL for tasks with linear and general reward functions [Fg..[- b; d). MF showed slight advantage over MB ç-learning in both settings. We further studied the effect non-linearity of general reward functions on the performance of classical SF compared to ç-learning by evaluating them in tasks with different levels of non- won-linearity We sampled general reward functions that resulted in different levels of mean absolute model error they are linearly approximated with min (r()) & w|. We trained SFQL and MF g-learning in each of these conditions on 300 tasks and measured the ratio between the total return of SFQL and MF & (Fig. []. The relative performance of SFQL compared to MF reduces with higher non- linearin of the reward functions. For reward functions that are nearly linear (mean error of 0.125), both have a similar performance. Whereas, for reward functions that difficult model with linear relation (mean error of 1.625) SFQL reaches only less than 50% of the performance of g-learning. This follows SFQL's theoretical limitation in (7) and shows the advantage of & learning over SFQL in non-linear reward tasks.
4.2 CONTINUOUS FEATURES RACER ENVIRONMENT
Environment and Tasks: We further evaluated the agents in an environment with continuous features (Fig. 2 a). The agent is randomly placed in the environment and has drive around for 200 timesteps before the episode ends. Similar to car, the agent has an orientation and momentum, so that it can only drive straight, or in a right or left curve. The agent reappears on the opposite side it exits one side. The distance to 3 markers are provided as features Φ E R33 Rewards depend on the distances = = T where each component тı has 1 or preferred distances defined by Gaussian functions. For each of the 37 tasks, the number of Gaussians and their properties (/s, a) are randomly sampled for each feature dimension. Fig. (a) shows reward function with dark areas depicting higher rewards. The agent has to learn to drive around in such way as to maximize its trajectory over positions with high rewards. The state space is high-dimensional vector E R120 encoding the agent's 's position and orientation. As before, the 2D position is encoded using 10 x 10 grid of wio-dimensional Gaussian radial basis functions. Similarly, the orientation also encoded using 20 Gaussian radial basis functions.
(a) Racer Environment
(b) Tasks with General Reward Functions
SFQU
CMF Xi
80k
Average Retur
40k
5
10
15
25
30
35
Tasks Trained
(a) Example of a reward function for the racer environment based on distances to its markers. (b) E-learning reaches the highest average reward per task. SFQL yields performance below QL as it not able to model the reward function with its linear combination of weights and features. The average over 10 runs per agent and the standard error of the mean depicted.
Figure 2:
7

---

Under review as a conference paper at ICLR 2022
Agents: We introduce MF Ş-agent for continuous features (CMF E) (Appendix D.2.3 D.2.3
CMF & dis-
cretizes each feature dimension dy E [0. 1] in 11 bins with the bin centers: X = 0.0. 0.1 1.0).
It learns for each dimension io and bin i the f-value EI (s, a, X²). Q-values (Eq. I0 computed by: Q* (s. a) = Σ Σ rs (x (s, a, X,). SFQL learns & for the continuous, non-discretized feature space. It received an approximated weight vector w; that was trained before the task started on several uniformly sampled features and rewards.
Results: {-learning reached the highest performance of all agents (Fig. 2 b) outperforming QL and SFQL. SFQL reaches only a low performance below QL, because it is not able to sufficiently well approximate the general reward functions with its linear reward model. This shows the advantage of Ç-learning over SFQL in environments with general reward functions.
DISCUSSION
Performance of ç-learning compared classical SF&GPI: Ç-learning allows to disentangle the dynamics of policies in the feature space of a task from the associated reward, see (TO). The experimental evaluation in tasks with general reward functions (Fig. dd, and Fig. 2 shows that ç-learning can therefore successfully apply GPI to transfer knowledge from learned tasks to new ones. Given a general reward function it can re-evaluate successfully learned policies for knowledge transfer. Instead, classical SFQL based on a linear decomposition (3) can not be directly applied given a general reward function. In this case a linear approximation has to be learned which shows inferior performance to E-learning that directly uses the true reward function.
ç-learning also shows an increased performance over SFQL in environments with linear reward functions (Fig. D a). This effect not be attributed to differences in their computation of a policy's expected return as both are correct (Appendix A.6 A possible explanation that E-learning reduces the complexity for the function approximation of the f-function compared to the vi-function in SFQL.
Continuous Feature Spaces: For tasks with continuous features (racer environment), E-learning used successfully a discretization of each feature dimension, and learned the ξ-values independently for each dimension. This strategy viable for reward functions that are cumulative over the feature dimensions: r(Ø)= = Σx Fx ØR. The Q-value can be computed by summing over the independent dimensions and the bins X: (" a) = E. Erex (2( (s, a. a. ). For more general reward func- tions, the space of all feature combinations would need to be discretized, which grows exponentially with each new dimension. As solution the g-function could be directly defined over the continuous feature space, but this yields some problems. First, the computation of the expected return requires an integral Q(s, a) = Sace R()){(s, a, Φ) over features instead of a sum, which is a priori intractable. Second, the representation and training of the ξ-function, which would be defined over a continuum thus increasing the difficulty of approximating the function. Jaanerr al. (2020) and Touati and Ollivier (2021) propose methods that might allow to represent continuous E-function, but it is unclear if they converge and if they can be used for transfer learning.
Learning of Features: In principle, classical SF&GPI can also optimize general reward functions if features and reward weights are learned. This is possible if the leamed features describe the non-linear effects in the reward functions. Nonetheless, learning of features adds further challenges and shows to reduce performance. Barreto et al. (2017) learns features from observations sampled from several tasks before the SF&GPI starts. Therefore, novel non-linearities potentially introduced at later tasks are not well represent by the learned features. If instead features are learned alongside the SF&GPI procedure, the problem on how to coordinate both learning processes needs to be investigated. Importantly, gi-functions for older tasks would become unusable for the GPI procedure on newer task, because the feature representation changed between them.
Moreover, replication (Sec. E.I of the object collection task from Barreto et al. (217)) shows the performance of learned features is below the performance of given features. MF Xi reaches a final average reward per task of 850 with given features and reward functions. The best performance of SFQL with learned features only reaches a final performance of 575 (Fig. 2 in (Barreto et al. (017)).
In summary, if features and reward functions are known then É-learning outperforms SFQL. And using given features and reward functions is natural for many applications as these are often known, for example in robotic tasks where they are usually manually designed (Akalin and Loutfi 202I).
8

---

Under review as a conference paper at ICLR 2022
Computational Complexity: The improved performance of SFQL and Ç-learning over QL in the transfer learning setting comes at the cost of an increased computational complexity. The GPI procedure (6) of both approaches requires to evaluate at each step the uf*i-function or fuinttnttoon over all previous experienced tasks in M. As a consequence, the computational complexity increases linearly with each new environment that is added. A solution is to apply GPI only over subset of learned policies, Nonetheless, an open question is still how to optimally select this subset.
6 RELATED WORK
Transfer Learning: Transfer methods in RL can be generally categorized according to the type of tasks between which transfer is possible and the type of transferred knowledge [Taylor and Stone 2009 Lazarie 2012, /Zuu et al. 2020). In the case of SF&GPI which E-learning is part of, tasks only differ in their reward functions. The type of knowledge that is transferred are policies learned in source tasks which are re-evaluated in the target task and recombined using the GPI procedure. A natural use-case for Ç-learning are continual problems (Khetarpal et al. 2020) where an agent has continually adapt to changing tasks, which are in our setting different reward functions.
Successor Features: SF are based on the concept of successor representations (Dayan 1993
Momennejad] 2020). Successor representations predict the future occurrence of all states for policy in the same manner as SF for features. Their application is restricted to cow-dimensional state spaces using tabular representations. SF extended them to domains with high-dimensional state spaces (Kulkarni et al. 2016 /Zhnn et al., 2017 Barreto et al. 2017 2018), by predicting the future occurrence of fow-dimensional features that are relevant to define the return. Several extensions to the SF framework have been proposed. One direction aims to learn appropriate features from data such as by optimally reconstruct rewards (Barreto et al. 2017), using the concept of mutual information (Hansen et al., 2019), the grouping of temporal similar states (Madjiheurem and Toni 2019). Another direction is the generalization of the ui-function over policies (Borsa et al. 2018) analogous universal value function approximation (Schaul et al.) 2015). Similar approaches
use successor maps (Madarasz) 2019), goal-conditioned policies (Ma et al., 2020), or successor
feature sets (Brantley al., (021). Other directions include their application to POMDPs (Vértes
and Sahani 2019). combination with max-entrops principles (Vertes) 2020), or hierarchical RL (Barreto et al. [022 In difference to E-learning all these approaches build on the assumption of linear reward functions, whereas &- g-learning allows the SF&GPI framework to be used with general reward functions. Nonetheless, most of the extensions for linear SF can be combined with g-learning.
Model-based RL: SF represent the dynamics of policy in the feature space that is decoupled from the rewards allowing to recvaluate them under different reward functions. It shares therefore similar properties with model-based RL (Lehnert and Lituman| 2019). In general, model-based RL methods learn a one-step model of the environment dynamics p(8 St, az). Given policy and an arbitrary reward function, rollouts can be performed using the learned model to evaluate the return. In practice, the rollouts have high variance for long-term predictions rendering them ineffective. Recently, (Janner et al., 2020) proposed the m-oodel framework that learns to represent Ç-values in continuous domains. Nonetheless, the application convergence is proven as for E-learning. This is the same case for the forward -ackdddr MPD representation proposed in Touati and Ollivier (202T). (Tang et al. 2021) also proposes to decouple the dynamics in the state space from the rewards, but learn an internal representation of the rewards. This does not allow to reevaluate policy to new reward function without relearning the mapping.
transfer learning is not discussed and no
CONCLUSION
The introduced ç-learning framework learns the expected cumulative discounted probability of successor features which disentangles the dynamics of a policy in the feature space of a task from the expected rewards. This allows g-learning to reevaluate the expected return of learned policies for general reward functions and to use it for transfer learning utilizing GPI. We proved that E-learning converges to the optimal policy, and showed experimentally its improved performance over Q-lcarning and the classical SF framework for tasks with linear and general reward functions.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
E-learning and its associated optimization algorithms represent general RL procedures similar to Q-learning, Their potential negative societal impact depends on their application domains which range over all possible societal areas in similar manner as for other general RL procedures.
Beyond the topic of the paper, we did our best cite the relevant literature and to fairly compare with previous ideas, concepts and methods, To that aim, all agents are trained and evaluated within the same software environment, and under the very same experimental settings.
REPRODUCIBILITY STATEMENT
In order to ensure high changes of reproducibility we provided lots of details of the method and experiments associated to the paper. In particular, we have provided the proofs for all mathematical results announced in the main paper (see Appendix A). These constitute the theoretical foundation of the proposed Ç f-learning methodology. Secondly, have provided all experimental details (methods, and environments) required for reproducing our experiments, namely: appendix C for the object collection and D for the racer environment respectively. In addition, provide additional results in appendix to completely illustrate the interest of the proposed method. Finally, we provided an anonymous link to the source code, so that reviewers can run if necessary.
REFERENCES
N. Akalin and A. Loutfi. Reinforcement learning approaches in social robotics. Sensors, 21(4): 2021.
A. Barreto, W. Dabney, R. Munos, J. Hunt, T. Schaul, H. P. van Hasselt, and D. Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pages 4055-4065, 2017.
A. Barreto, D. Borsa, J. Quan, T. Schaul, D. Silver, M. Hessel, D. Mankowitz, A. Zidek, and R. Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement, In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
A. Barreto, S. Hou, D. Borsa, D. Silver, and D. Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 11 17748):30079-3008 2020.
A. Barreto, D. Borsa, S. Hou, G. Comanici, E. Aygün, P. Hamel, D. Toyama, J. Hunt, S. Mourad, D. Silver, et al. The option keyboard: Combining skills in reinforcement learning. arXiv preprint arXiv:2106. 13105, 2021.
D. Borsa, A. Barreto, J. Quan, D. Mankowitz, R. Munos, H. van Hasselt, D. Silver, and T. Schaul. Universal successor features approximators. urXiv preprint arXiv: 1812.0766, 2018
K. Brantley, S. Mehri, and G. J. Gordon. Successor feature sets: Generalizing successor representa- tions across policies. arXiv preprint arXiv:2103.026551 2021.
Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613-624, 1993.
S. Hansen, W. Dabney, A. Barreto, T. Van de Wiele, D. Warde-Farley, and V. Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint arXiv:/906.0500 2019.
M. Janner, I. Mordatch, and S. Levine. m-models: Generative temporal difference learning for infinite-horizon prediction. In NeurlPS, 2020.
K. Khetarpal, M. Riemer, I. Rish, and D. Precup. Towards continual reinforcement learning: A review and perspectives. arXiv preprint arXiv:2012. 13490, 2020.
T. D. Kulkarni, A. Saeedi, S. Gautam, and S. J. Gershman. Deep successor reinforcement learning. arXiv preprint arXiv: 16Xiv:1606.0200 1606.02366666.... 2016.
10

---

Under review as a conference paper at ICLR 2022
A. Lazaric. Transfer in reinforcement learning: a framework and survey. In Reinforcement Learning. pages 143-173. Springer, 2012.
L. Lehnert and M. L. Littman. Successor features support model-based and model-free reinforcement learning. CoRR abs/1901. 1437, 2019.
C. Ma, D. Ashley, J. Wen, and Y. Bengio. Universal successor features for transfer reinforcement learning, arXiv preprint arXiv:2001.0402. 2020.
T. J. Madarasz. Better transfer learning with inferred successor maps. arXiv: 07663, 2019.
arXiv preprint
Madjiheurem and L. Toni. State2vec: Off-policy successor features approximators. arXiv preprint arXiv: 10277, 2019.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-555 533, 2015.
I. Momennejad. Learning structures: Predictive representations, replay, and generalization. Current Opinion in Behavioral Sciences, 32:155-166, 2020.
I. Momennejad, E. M. Russek, J. H. Cheong, M. M. Botvinick, N. D. Daw, and S. J. Gershman. The successor representation human reinforcement learning. Nature Human Behaviour, 1(9): 680-692 2017.
T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320, 2015.
D. Silver, T. Hubert, Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, Ku- maran, T. Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140-11444 2018.
R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
H. Tang, J. Hao, G. Chen, P. Chen, C. Chen, Y. Yang, L. Zhang, W. Liu, and Z. Meng. Fore- see then evaluate: Decomposing value estimation with latent future prediction. arXiv preprint arXiv:2103.02225 2021.
M. E. Taylor and P. Stone. Transfer learning for reinforcement learning domains: A survey. Journal of Machine Learning Research, 10(7), 2009.
M. S. Tomov, E. Schulz, and Gershman. Multi-task reinforcement learning humans. Nature Human Behaviour, pages 1-10, 2021.
A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. arXiv preprint arXiv:2103.07945, 2021.
N. Tsitsiklis. Asynchronous stochastic approximation and q-learning. Machine learning, 16(3): 185 185-202, 1994.
E. Vertes. Probabilistic learning and compulation in brains and machines. PhD thesis, UCL (University College London), 2020.
E. Vértes and M. Sahani. A neurally plausible model learns successor representations in partially observable environments, arXiv preprint arXiv: 09480, 2019.
C. J. Watkins and P. Dayan. Q-learning. Machine learning, 8(3-4):279-28) 292, 1992.
Zhang, J. Springenberg. Boedecker, and W. Burgard. Deep reinforcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 237 2371-2378. IEEE, 2017.
Z. Zhu, K. Lin, and Zhou. Transfer learning in deep reinforcement learning: A survey. arXiv preprint arXiv:2009.0788 2020.
11

---

Under review as a conference paper at ICLR 2022
A THEORETICAL PROOFS
A.1 PROOF OF PROPOSITION
Let us start by recalling the original statement the main paper.
Proposition (Equivalence between functions & and Q) Let Q = {Q :Q: A s.l. |||| < oo). Let be defined & ~ < Jo RG = So R$2- Then, is equivalence relationship, and there is bijective correspondence between the quotient space E_ and Q
Proof. We will proof the statements sequentially.
is an equivalence relationship: To prove this need to demonstrate that is symmetric, reciprocal and transitive. The three are quite straightforward since: ç ~ &, VC. & ~ n e ~ E, VE, and 7) = E~v
Bijective correspondence: To prove the bijectivity, we will first prove that is injective, then surjective. Regarding the injectivity: [<] + [n] = Qe 7 Qn we prove it by contrapositive:
- - / R(O)E(s
J. (() + [ [n]-
(20)
In order to prove the surjectivity, we start from function Q E Q and select arbitrary E E, then the following function:
a(s,a) So R(o)(s,, a, o)
EQ 0,
E(s,a, Φ)
(21)
satisfies that tq E and that I R(o)Eq Φ)dø a(s V(s, a) S x A. We conclude that there bijective correspondence between the elements E. and of
A.2 PROOF OF COROLLARY
Let us recall the result:
Corollary 1. The bijection between E. and Q allows to induce norm || ||~ into E. from the supremum norm 2, with which E., Banach space (since Banach with lloo):
| -sup 1/ R())(ss, sup Q(s, ||Q
(22)
Proof. The norm induced the quotient space defined from the correspondence between E- and Q and is naturally defined as in the previous equation. The norm is well defined since it does not depend on the class representative. Therefore, all the metric properties are transferred, and E., is immediately Banach with the norm || ||..
A.3 PROOF OF PROPOSITION 2
Let's restate the result:
Proposition (E-learning has a fixed point) The operator Tg well-defined w.r.t. the equivalence ~, and therefore induces an operator Tw. defined over E... T. is contractive w.r.t. -|- Since E. Banach, Tw. has a unique fixed point and iterating T-. starting anywhere converges to that point.
Proof. We prove the statements above one by one:
12

---

Under review as a conference paper at ICLR 2022
The operator T~ is well defined: Let us first recall the definition of the operator Tş in 12 where we removed the dependency on æ for simplicity:
Tg(E)= p(d. = =))), + (eiii)+-Bpepeeeeeee Pppessaetsleee (5($++1, 6))
Let Eτ, &2 E [g) two different representatives of class [s], we can write:
- - o))d
S 4 R(d) , p(s. - E2(s', oo))d'''
(23)
/ pls p(6',a((s,)) / ((O)(i((Ed d'), - E2(s, d', d
because E, f2 E [S]. Therefore the operator T-((E) (e) is well defined in the quotient space, since the image of class does not depend on the function chosen to represent the class.
Contractive operator T~:: The contractiveness of T- can be proven directly:
/ R(() (p(ols, a) + Tppce a's. tE(s', d),
|T~ (Ie) - (())ll sup 8.a
-p(ols, a) EEpss (n(s o))) dø
/
=7 sup
- nyss, a', Φ)
(24)
6,0
{1/ R(O)(E(s) d', nis', a', 2))do
=y sup sup Ep(s' v..)
s.a
/
S sysup sup
R(o)(E(s',d' - a d)
8',a'
=y [[<] [n]l
The contractiveness of Th can also be understood being inherited from the standard Bellmann operator on Q. Indeed, given & function, one can easily see that applying the standard Bellman operator the function corresponding & leads the function corresponding ([]))
Fixed point T..: To conclude the proof, use the fact that any contractive operator Banach space, in our case: T~ : E., + E. has a unique fixed point [g*], and that for any starting point [C)].
the sequence [c..] T. ((E converges to [[*] w.r.t. the corresponding norm ||[E]]
A.4 PROOF THBOREM
These two propositions will be useful to prove that the & learning iterates converge in E... Let us restate the definition of the operator from (3)):
(st, ae, Φ) E (St, ae Φ) + as (p(:: ōlse, ae; л) YET (s1+1, a.+1.)) Φ) Ex (st, as 6)
and the theoretical result:
Theorem 1. (Convergence of -learning) For sequence of state-action (st, at Se+1, de Loe consider the E-learning update given (13). If the sequence of state-action-feature triples visits each state, action infinitely often, and if the learning rate a is adapted sequence satisfying the Robbins- Monro conditions:
Og DC,
o < 00
(25)
13

---

Under review as a conference paper at ICLR 2022
then the sequence of function classes corresponding to the iterates converges to the optinum, which corresponds to the optimal Q-function to which standard Q-learning updates would converge to:
[cn] + [5"]
with (s(s,a)= R(O)E" (ss,,,27 2).d
(26)
Proof. The proof re-uses the flow of the proof used for Q-learning Tsitsiklis 1994.. Indeed, we rewrite the operator above as:
( at, o) + EX (st. au Φ) as [Tg(5E) az の) Ex (st, az, Φ) E(s,, an 0.
with defined as:
E(St, at, u) p(o;= = 0/ a4; T) + 7ER (8141, C++1, ())
-F. {p(1, 01 o4; + 75t (8++1, 0t-1,
Obviously satisfies E(c) O. which, together with the contractiveness of Tw,, is sufficient to demonstrate the convergence of the iterative procedure as done for Q-learning. In our case, the optimal function e is defined up an additive kernel function k E Ker. The correspondence with the optimal Q learning function is a direct application of the correspondence between the Ç- and Q-learning problems.
A.5 PROOF OF THEBRREM 2
Let us restate the result.
Theorem 2. (Generalised policy improvement in Ş-learning) Let M be the set of tasks, each one associated (possibly different) weighting function R₃ L (T). Let gnt be representative of the optimal class of & E-functions for task M,, i E {1...., I}. and let { be an approximation the optimal E-function, I 1l I E, Then, for another task M with weighting function R, the
policy defined as:
(s) arg max max R(O)E (s, d)do
(27)
satisfies:
I1 - 1r 5 1
(min R - Rellp(ole.o) +,),
(28)
where [If = sups,a So If gl do.
Proof. The proof is stated in two steps. First, we exploit the proof of Proposition of (Barreto et al. 2017), and in particular (13) that states:
||Q - Q² - 1-7
2
+E))
+e).
Vi (I,
(29)
where Q' and Q" are the Q-functions associated to the optimal and π policies in the environment R. The conditions on the Q functions required in the original proposition are satisfied because the f-functions satisfy them, and there is an isometry between Q and functions.
Because the above inequality is true for all training environments i, we can rewrite as:
II" 1= S - mmnnsup (r(, a) rs(ss (s) +e)) We now realise that, in the case of g-learning, the reward functions rewrite as: r((s,a)= = / R
2
(30)
r(s, a) / A(o)p(ols, a)dó
(31)
and therefore have:
sup (r(s, - r((s,a)) ssup
(R(o) (,(o))p(dlss
S.a
8.
4
(32)
S sup
[R(d) - R2(,) p()s. a)de
8.a
R
R Rillxobba
14

---

Under review as a conference paper at ICLR 2022
Similarly, due the isometry between and Q-learning, i.e. Proposition
write that:
l|5- - ||
||<-]
[6"]]|~ IQ - -Q"
2
<
minnspp (r(88 a) a)) +
7 8,a
(33)
1 (min || R Rello(ole.)) + +
which proves the generalised policy improvement for g-learning.
A.6 RELATION BETWEEN CLASSICAL SF AND Ç- S-LEANNING FOR LINEAR REWARD FUNCTIONS
In the case of linear reward functions, i.c. where assumption (3) holds, is possible to show that {- learning can be reduced to classical SF. Classical SF represents therefore a specific case of g-learning under this assumption.
Theorem 3. (Equality of classical SF and g-learning for linear reward functions) Given the assump- tion that reward functions are linearily decomposable with
Te(St, Gr. St+1) = 6(st. Qt St+1) Wi
where & E RR are features for a transition and w, E IRn are the reward weight vector of task my M, then the classical SF and Ş-learning framework are equivalent.
Proof. We start with the definition of the Q-value according to g-learning from 10) After replacing the reward function R; with our linear assumption, the definition of the Q-function according to classical SF with the & f-function can be recovered:
[
Q((s,a)=
5dd (" 42, R))
4
("), at, (1, Φ)Φ w,d
w
k p((0++ =018: = 8,a a; T) dø
k=0
pl0r+k = 6/st s,a, 0; T do
7 E {0l+k]
w
k=0
8
* Weth
( W:.
W;
Please note, although both methods equal in terms of their computed values, how these represented and learned differs between them. Thus, it is possible to see performance difference of the methods in the experimental results where Ç-learning outperforms SFQL in our environments.
B ONE-STEP SF MODEL-BASED (MB) Ç-LEARNING
Besides the MF E-learning update operator (16). we introduce second É-learning procedure called One-step SF Model-based (MB) E-Learning that attempts to reduce the variance of the update. To do MB g-learning estimates the distribution over the successor features over time. Let p(o, = Sist aei x) denote the current estimate the feature distribution. Given a transition (st, Ct S141, (i)) the model is updated according to:
di (0 St, a)))
Po(0) St, (;)) (1 poolollsss,ss Polo' S1, a3
o + à.: Po(0' TT)TT) +
15

---

Under review as a conference paper at ICLR 2022
where 3 E [0. 1] is the learning rate. After updating the model Ps. it can be used for the É-update as defined in (13). Since the learned model Pe is independent from the reward function and from the policy, it can be learned and used over all tasks.
C EXPERIMENTAL DETAILS: OBJECT COLLECTION ENVIRONMENT
The object collection environment (Fig. T provides a formal description.
a) was briefly introduced in Section 4.1
This section
C.I ENVIRONMENT
The environment is continuous two-dimensional area in which the agent moves. The position of the agent is a point in the 2D space: (2r, E [0, 1).. The action space of the agent consists of four movement directions: .A {up. down, left, right3. Each action changes the position of the agent in a certain direction and is stochastic by adding a Gaussian noise. For example. the action for going right updates the position according to It+1 xt Ni2 = 0.05, a 0.005). If the new position ends in a wall (black areas in Fig. -- a) that have a width of 0.04) or outside the environment, the agent is set back to its current position. Each environment has 12 objects. Each object has two properties with two possible values: color (orange, blue) and shape (box, triangle). If the agent reaches an object, it collects the object which then disappears. The objects occupy a circular area with radius 0.04. At the beginning of an episode the agent starts at location S with (æ, y)s (0.05, 0.05). An episode ends if the agent reaches the goal area G which is at position (I, y)g = (0.86, 0.86) and has a circular shape with radius 0.1. After episode the agent is reset to the start position S and all collected objects reappear.
The state space of the agents consist of their position in the environment and the information about which objects they already collected during an episode. Following (Barreto et al. 2017). the position encoded using radial basis function approach. This upscales the agent's (I, position
high-dimensional vector Spos E IR 100 providing better signal for the function approximation of the different functions such as the v or f-function. The vector Spos is composed of the activation of two-dimensional Gaussian functions based on the agents position (z, v):
Spos = exp (- (xx (x-C,,)²² + (y -,,2)²
(34)
where çj E R22 is the center of the 1"h Gaussian. The centers are laid out on a regular 10 x 10 grid over the area of the environment. The state also encodes the memory about the objects that the agent has already collected using a binary vector Smem E {0, 1 The jth dimension encodes if the jth object has been taken (5mem.J = 1) or not (5mem.j = 0). An additional constant term was added to the state to aid the function approximation, As a result, the state received by the agents is column vector with [s sTem= E 113
The features o a,, $t+1) E {0, 11.5 in the environment describe the type of object that was collected by an agent during a step or if it reached the goal position. The first four feature dimensions encode binary the properties of collected object and the last dimension if the goal area was reached. In total
|+| = 6 possible features exists: Φ1 0,,0,0 0, 0J standard observation, Φ2 [1, 0, 1, 1, 0 collected an orange box, 63 = [1, 0, 0, 1, O] collected an orange triangle, Φ4 00, 1,
collected blue box, Φ3 = 00, 1, 0, 1, collected blue triangle, and Φb = [0, 0, 0, O, 1] reached the goal area.
Two types of tasks were evaluated in this environment that have either 1) linear or 2) general reward functions. 300 tasks, i.c. reward functions, were sampled for each type. For linear tasks, the rewards r = o W; are defined by linear combination of discrete features & E No and weight vector w; E R The first four dimensions in w; define the reward that the agent receives for collecting objects having specific properties, e.g. being blue or being a box. The weights for each of the four dimensions are randomly sampled from uniform distribution: WkE[1,2,3.4 task. The final weight defines the reward for reaching the goal state which is W3 = for each task. For training agents in general reward tasks, general reward functions R2 for each task M, were sampled. These reward functions define for each of the four features (Φ2..... Ø5) that represent the collection of a specific object type an individual reward. Their rewards were sampled from uniform
(1,-1, for each
16

---

Under review as a conference paper at ICLR 2022
distribution: (e. 1-),11) The reward for collecting object is Rx(01) 0 and for reaching the goal area is R3 (0bs = 1 for all tasks. Reward functions of this form can not be linearly decomposed features and a weight vector.
C.2 ALGORITHM
This section introduces the details of each evaluated algorithm. First the common elements discussed before introducing their specific implementations.
All agents experience the tasks M M of an environment sequentially. They informed when a new task starts. All algorithms receive the features o(s, a, s') of the environment. For the action selection and exploration, all agents use e-greedy strategy. With probability e E [0, 1] the agent performs random action. Otherwise it selects the action that maximizes the expected return.
As the state space (s IR 13) of the environments an approximation of their respective functions such as for the Q-function (Q(s, a) Q(s, a)) or the E-function (É(s. a, σ) = {(s, a, Φ)). We deseribe the general function approximation procedure on the example of f-functions. If not otherwise mentioned, all functions are approximated by a single linear mapping from the states to the function values. The parameters 0 E |113/|A| of the mapping have independent components of IR113 for each action E A and feature e E Φ:
nigh-dimensional and continuous, all agents use
É(s, oz os, 05) = T a.
(35)
To learn & we update the parameters 0 using stochastic gradient descent following the gradients Vos (f5) of the loss based on the ç-learning update 13)
VΦ E : Cg((5) = E (b(le = olst, a.) + 7E($141, a+11, o: 06) E( ae, Φ0 (2:5)) }
(36)
with āe+l argmax Σ R(0)((0-11,,000
where Ø = 6 but @ is treated constant for the purpose of calculating the gradients V Lg (BS). We used PyTorch for the computation of gradients and its stochastic gradient decent procedure (SGD) for updating the parameters.
C.2.1 QL
The Q-learning (QL.) agent (Algorithm I represents standard Q-learning (Watkins and Dayan 1992). The Q-function is approximated and updated using the following loss after each observed transition:
{ (1ee,, (e, s(41) ) + max a E
( -E (
:0:1:0001))00)00 Q(5), ac; (::
(37)
where 00 = 8 but 0 is treated as constant for the purpose of optimization, i.c no gradients flow through Following (Barreto et al. 2017) the parameters 09 are reinitialized for each new task.
C.2.2 SFQL
The classical successor feature algorithm (SFQL) based on linear decomposition of rewards in features and reward weights (Barreto et al. 2017) (Algorithm 2). If the agent is learning the reward weights W, for a task M, then they are randomly initialized at the beginning of a task. For the case of general reward functions and where the reward weights given the agents, the weights are learned to approximate linear reward function before the task. See Section C.3 for a description of the training procedure. After each transition the weights updated by minimizing the error between the predicted rewards Φ(8t, dt, St+1) w, and the observed reward rt:
Lw, (w.) = E (r(s,a,8)) - O(s,, Or 8141) Tw.) (w.)²)
(38)
PoyTorch v1.4: http:tt///yy/occ..ooor
17

---

Under review as a conference paper at ICLR 2022
Algorithm 1: Q-learning (QL)
Input: exploration rate:
learning rate for the Q-function:
for i +- 1 to num Jasks do
initialize Q: 02 < small random initial values new _episdde f- true for i 4- 1 to num_steps do
new see__ee then new new_episode + false
9t mitiil state
With probability select random action ag, otherwise ae argmax, ((s(: a)
Take action ae and observe reward re and next state $141 if 8++1 is terminal state then
new hew_episode-t true
Y 0
else
+
- Ti + 7/ maxaee Q(s+11, ax+1)
Update 0 using SGD(a) with Le (y) (00 a:))
8t t $6+1
SFQI. learns an approximated vi-ffunctonn for each task M₁. The parameters of the vi-function for the first task 0 are randomly initialized. For consecutive tasks, they are initialized by copying them from the previous task (0) + 1 The - of the current task M, is updated after each observed transition with the loss based on 55:
(0,)) = F. (0(60 oe 86-1) + 770 C.t1; 0,7) -----ssee 22 ar 0.:) 2
(39)
with ae+1 = argmax
max KE (1....)....
(5 0 Wi
where 0,"h 0, but 03¹ treated constant for the purpose of optimization, i. gradients flow through it. Besides the current fu-fuccton,, SFQL also updates the vi-function which provided the GPI optimal action for the current transition:
argmax (1.2....)) max; (s, b) w. The
update uses the same loss as for the update of the active fu-function (39 but instead of using the GPI optimal action as next action, uses the optimal action according to own policy: äx+1 argmax, (o a) We
C.2.3 Č-LEARNING
The E-learning agents (Algorithms allow reevaluate policies in tasks with general reward functions. If the reward function not given, approximation A. of the reward function for each task M; is learned. The parameters for the approximation are randomly initialized at the heginning of each task. After each observed transition the approximation is updated according the following loss:
CR(OR) =E (r(se: de, se+1) R2 (o(s,, a, s(+1):
(40)
In the case of tasks with linear reward functions the reward approximation becomes Ri(o(se, at, st+1): 0R) Φ(8t, az, 8t+1) Thus with 0 = w; we recover the same proce- dure for SFQL (38). For non-linear, general reward functions we represented ft with neural network. The input of the network is the feature o(s,, at $t+1). The network has one hidden layer with 10 neurons having ReLu activations. The output is linear mapping to the scalar reward e R.
All E-learning agents learn approximation of the functioo for each task Mj- Analogous to SFQL,, the parameters of the Ê-function for the first task of are randomly initialized. For con-
18

---

Under review as a conference paper at ICLR 2022
Algorithm 2: Classical SF Q-learning (SFQL) (Barreto et al, 2017)
Input:
exploration rate:
learning rate for yi-functions:
learning rate for reward weights w: Ow
features Ø
optional: reward weights for tasks: {W1s W2,.... W Woom_taske tasks
for i 4-- to num num_tasks do
if w; not provided then f- small random initial values
if' then initialize wh: 0%¹ +- small random initial values else 0 0 new new_episode true for t + to num_steps do if new sew__epsodee then new new_episode false initial state
2rgmaAXkg11,2...... max, thx (5t, T
// GPI optimal policy
With probability e select random action ae, otherwise at + argmax, a) w; Take action a, and observe reward and next state 81+1 Update W; using SGD(aw) with Lw = (Te - 0(st, des 8141) w.)² Tw,)² if 8 is terminal state then new _ew_eeisode_ee true
0
else
I. Yt +-1 7
// GPI opt optimal next action for task i
0z-1 + argmax, argkE(L.2....) (x(
y y o(si, ae: 8t+1) + ā(t1)
Update 0 using SGD(a) with C. = (y (s))))))))
if of & then
dr+1 t- argmax, (a(++11, a) We // optimal next action for task
+ Ø(si, at, st+1) + (1-1,.04
Update 0 using SGD(a) with by = (y (8c)
8: f Sc+1
secutive tasks, they are initialized by copying them from the previous task (0 + ff The E--function of the current task M, is updated after each observed transition with the loss given in (46). The ç-learning agents differ in their setting for p(0r ø182; in the updates which is described in the upcoming sections. Besides the current É--function, the Ç-learning agents also update the f.-function which provided the GPI optimal action for the current transition: (1.2..) maxa; Eeco (st, 4t, Φ) (o). The update uses the same loss as for the update of the active fi-function (46), but instead of using the GPI optimal action as next action, uses the optimal action according to its own policy: āt+1 = max, Eoco Ec a, Φ) Re(Ø).
stochastic update for the & functions. Given a transition, we set p(Ø; = Φ ft. az) = for the observed feature Ø = 6(8,, 0z 8t+1) and p( = s at) = 0 for all other features + o(s, at 8t+1).
MF E-learning: The model-free E-learning agent (Algorithm
MB Ellearning: The one-step SF model- & llarning agent (Algorithm 4 uses an approximated model ğ to predict p(ó, ølst, ax) reduce the variance of the f-function update. The model is by
19

---

Under review as a conference paper at ICLR 2022
linear mapping for each action. It uses softmax activation to produce valid distribution over +:
exp(s T O ( p(s, o; 0%) Eereo exp(s 0
(41)
where 0% E ERIS As is valid for each task in M, its weights 0r are only randomly initialized at the beginning of the first task. For each observed transition, the model is updated using the following loss:
V : : cp( = (v(á, ølst, at) - p(st, at, o: 0²)))))}
(42)
where we set p(Ø. ols;, at) = 1 for the observed feature & = a ae, 91+1) and p(øe = 618t; 0e) = for all other features @ * ((8t, 01, 81+1).
C.3 EXPERIMENTAL PROCEDURE
All agents were evaluated in both task types (linear or general reward function) on 300 tasks. The agents experienced the tasks sequentially, each for 20.000 steps. The agents had knowledge when a task change happened. Each agent was evaluated for 10 repetitions to measure their average performance. Each repetition used a different random seed that impacted the following elements: a) the sampling of the tasks, b) the random initialization of function approximator parameters, c) the stochastic behavior of the environments when taking steps, and d) the e-greedy action selection of the agents. The tasks, i.c. the reward functions, were different between the repetitions of a particular agent, but identical to the same repetition of different agent. Thus, all algorithms were evaluated over the same tasks.
The SF agents (SFQL, (-learning) were evaluated under two conditions. First, that they have to learn the reward weights or the reward function online during the training (indicated by (O) in figures). Second, the reward weights the reward function is given to them. As the SFQL does not support general reward functions, it is not possible to provide the SFQL agent with the reward function in the second condition. As solution, before the agent was trained of the reward R; (Ø) o² úi was fitted. The initial approximation tiy was randomly initialized and then fitted for 10.000 iterations using gradient descent procedure based on the absolute mean error (L1 norm):
new task Mi linear model
1
Aw;
17
(())
TT wi
(43)
with learning rate of 1.0 that yielded the best results tested over several learning rates.
Hyperparameters: The hyperparameters of the algorithms were set to the same values as in (Barreto et al. 2017). A grid search over the learning rates of all algorithms was performed. Each learning rate was evaluated for three different settings which are listed in Table [ If algorithms had several learning rates, then all possible combinations were evaluated. This resulted
different number of evaluations per algorithm and condition: QL- 3, SFQL (O) 9, MF Xi (0) 9, MB Xi (O) 27, SFQL 3, MF Xi 3, MB Xi In total, 63 parameter combinations were evaluated. The reported performances in the figures are for the parameter combination that resulted in the highest cumulative total reward averaged over all 10 repetitions the learning rates and Ow are set to half of the rates defined in (Barreto et al.) 2017). This is necessary due to the differences in calculating the loss and the gradients in the current paper. We use mean squared error loss formulations, whereas (Barreto et al. 2017) uses absolute error losses. The probability for random actions of the c-Gredyy action selection was set to = 0.15 and the discount rate to y 0.95. The initial weights 0 for the function approximators were randomly sampled from standard distribution with Omn ~ N(p | = 0, a = 0.01).
the respective environment. Please note,
Computational Resources: Experiments were conducted on a cluster with a variety of node types (Xeon SKL Gold 61330 with 10GHz, Xeon SKL Gold 5218 with 2.30GHz, Xeon SKL Gold 6126 with 2.60GHz, Xeon SKL Gold 6244 with 3. 3.60GHz, each with 192 GB Ram, no GPU). The time for evaluating one repetition of certain parameter combination over the 300 tasks depended on the algorithm and the task type. Linear reward function tasks: QL ss 1h, SFQL (O) as 4h, MF & (O) 42h, MB & (O) R 43h, SFQL 4h, MF & x 15h, and MB & R 16h. General reward function
20

---

Under review as a conference paper at ICLR 2022
Table 1: Evaluated Learning Rates in the Object Collection Environment
Parameter
Description
Values
Learning rate of the Q, th, and f-function
{0.0025, 0.005, 0.025) {0.025, 0.05, 0.075) {0.2, 0.4, 0.6)
Oww O.R
Learning rate of the reward weights or the reward model Learning rate of the One-Step SF Model
tasks: QL ~ 1h. SFQL (0) v 4h, MF (0) ~ 68h, MB (O) = 67h, SFQL = 5h, MF & R2 14h, and MB & = 18h. Please note, the reported times do represent well the computational complexity of the algorithms, as the algorithms were not optimized for speed, and some use different software packages (numpy or pytorch) for their individual computations.
C.4 EFFECT OF INCREASING NON-LINEARITY IN GENERAL REWARD TASK
We further studied the effect of general reward functions the performance of classical SF compared to g-learning (Fig. T c). We evaluated the agents in tasks with different levels of difficulty in relation how well their reward functions can be approximated by linear model. Seven difficulty levels have been evaluated. For each level, the agents were trained sequentially on 300 tasks as for the experiments with general reward functions. The reward functions for each level were sampled with the following procedure. Several general reward functions were randomly sampled as previously described. For each reward function, linear model of a reward weight vector was fitted using the same gradient descent procedure as in Eq. 43 The final average absolute model error after 10.000 iterations was measured. Each of the seven difficulty levels defines a range of model errors its tasks have with the following increasing ranges: {[0.0, 0.25), [0.25, 0.5]..... [1.5, 1.75] For each difficulty level, 300 reward functions were selected that yield a linear model are in the respective range of the level.
Q-Learnig, SFQL, and MF Xi-learning were each trained on 300 tasks, i.c. reward functions, on each difficulty level. As hyperparameters were the best performing parameters from the previous general reward task experiments used. We measured the ratio between the total return over 300 tasks of QL and MF Xi-learning (QL/MF Xi), and SFQL and MF Xi-learning (SFQL/MF Xi). Fig. [ с shows the results, using as x-axis the mean average absolute model error defined by the bracket of each difficulty level. The results show that the relative performance of SFQL compared MF Xi reduces with higher non-linearity of the reward functions. For reward functions that are nearly linear (mean error 0. 0.125), both have a similar performance. Whereas, for reward functions that are difficult to model with a linear relation (mean error of 1.625) SFQL reaches only less than 50% of the performance of MF Xi-learning.
21

---

Under review as a conference paper at ICLR 2022
Algorithm 3: Model-frre E-learning (MF {)
Input: exploration rate:
learning rate for f-functions:
learning rate for reward models R: aR
features Ø
optional: reward functions for tasks: {R R2..... Rowm_ton tasks
for i +- to num_lasks do
if R2 not provided then initialize RG: 0 + small random initial values if then initialize E.: of +- small random initial values else of + 05 new _ew_episode true for t + to num__tttee steesssessett do if new ree__ee then new new_episode false < initial state
é - arggnaxe max, Eg (e( a, Φ) (0() // GPI optimal policy
With probability e select a random action a, otherwise ag < argmax, Eg Eo(sE, a, Φ) (d)
Take action ae and observe reward re and next state 81+1
R2 not provided then
Update 8 using SGD(a with Ca = (r+ a1,
if 1 is terminal state then
new new_episoue hew_eppiodeeee true
Y
else
L +-
// GPI optimal next action for task i Q--1 & argmax, argke(1,2....)) E. a,
foreach @ E Φ do
if o = o(st, S141) then ba + 1 + 1 0(41)) else yo 1
Update of using SGD(a) with Le = Es El(o,, ae o))²
if c. o then
// optimal next action for task
āo+1 + argmax, Es foreach o E Φ do if o p(st, a1, then yo + + 0u 01+1, else yo + 1664 ae+1, ②)
Update 05 using SGD(a) with Lę = Eg( Ec((s, ar, 0)
81+1
22

---

Under review as a conference paper at ICLR 2022
Algorithm 4: One Step SF-Model & E-learning (MB g)
Input:
exploration rate:
learning rate for E-functions: a learning rate for reward models R: OR
learning rate for the one-step SF model p: B
features Ø
optional: reward functions for tasks: {Ri, R2,... Raum tusks)
initialize p: 8² small random initial values
for i + to nwm_tasks do
RG not provided then initialize R4 0 + small random initial values if then initialize E of small random initial values else of 0 new_episee new_episode &- true for t < 1 to num steps do if new _ee__eesodee then new new_episode false 8; - imitial state
c argmax;e(1.2...)) max, Eg (e a, (o))) // GPI optimal policy
With probability e select random action ae, otherwise
ag 6 argmax, Eg Ee(s,, a, Φ) (d)
Take action d4 and observe reward ry and next state 81+1
if R2 not provided then
Update 8tt using SGD(aR) with Ln (re (o, at, 8141))²
foreach Φ Φ do
if e(st, at, 8c+1) then yo f- else Us +
Update 8 using SGD(B) with Cp 2. p(s), x))²
s+41 is terminal state then
new _ew_eepisdeeeao + true
7 0
else
L 7tt +
// GPI optimal next action for task
ūz-1 + argmax, argke(1,.....) Eg a)
foreach + do
No p(s ar, o) T a++1, 0)
Update of using SGD(a) with L5 = [v( Es( ar x))2
if c # then
// optimal next action task
de+1 +- argmax, E. (, R((())
foreach & E do
Uo + p(st,, ar, Φ) + ax+1,
Update 05 using SGD(o) with Ls Es(yo Ed(s,, at, o))²
8:
Se+1
23

---

Under review as a conference paper at ICLR 2022
D EXPERIMENTAL DETAILS: RACER ENVIRONMENT
This section extends the brief introduction to the racer environment (Fig. 2 a) given in Section 4.2
D.1 ENVIRONMENT
The environment is continuous two-dimensional area in which the agent drives similar car. The position of the agent is point in the 2D space:
(I, y) E [0, 1]. Moreover, the agent has an orientation which it faces: / E [-T, T1]. The action space of the agent consists of three movement directions: A = (right, straight, left }. Each action changes the position of the agent depending on its current position and orientation. The action straight changes the agent's 's position by 0.075 towards its orientation 0. The action right changes the orientation of the agent to 0 + द and its position 0.06 towards this new direction, whereas left the direction to 0 - IT changes. The environment is stochastic by adding Gaussian noise with 0 = 0.005 to the final position I, Y. and orientation A. If the agent drives outside the area (x, y) E [0, 1] then it reappears on the other opposite side. The environment resembles therefore torus (or donut). As a consequence, distances d(pa. Py) are also measure in this space, that the positions pa (0.1, 0.5) and Py (0.9, 0.5) have a distance of 0.8 but d(pa.Pr) = 0.2. The environment has 3 markers at the positions m = (0.25, 0.75), m2 = (0.75, 0.25), and my (0.75, 0.6). The features measure the distance of the agent to each marker: Φ E R2 with OR d(p, m²). Each feature dimensions is normalized to be de E [0, 1). At the beginning an episode the agent is randomly placed and oriented environment. An episode ends after 200 time steps.
The state space of the agents similarly constructed as for the object collection environment. The agent's 's position is encoded with 10 x 10 radial basis functions spos E R 100 as defined in 34 In
difference, that the distances are measure according to the torus shape. similar radial basis function approach is also used to encode the orientation See E R220 of the agent using 20 equally distributed gaussian centers in [- т] and o = ST. Please note, π and - are also connected in this space, ie. d( - O. The combination of the position and orientation of the agent is the final state:
8 = [ pos: % ori
E TR. 120
The reward functions define preferred positions in the environment based on the features, i.e. the distance of the agent to the markers. A preference function r' exists for each distance. The functions composed of maximization over m Gaussian components that evaluate the agents distance:
1 R(Ø)= = Σ rx(x)) with rı max exp (- [-(o--1) 14))
3
(44)
jj
1m
Reward functions are randomly generated by sampling the number of Gaussian components m to be 1 or 2. The properties of each component are sampled according to μí ~ 2(000, 0.7), and oj 2(0.001, 0.01). Fig. _ a illustrates one such randomly sampled reward function where dark areas represent locations with high rewards.
D.2 ALGORIHTMS
We evaluated Q-learning, SFQL (O), SFQL, and MF ç-learning (see Section C for their full description) in the racer environment. In difference to their implementation for the object collection environment, they used a different neural network architecture to approximate their respective value functions.
D.2.1 QL
Q-learning uses a fully connected feedforward network with bias and a ReLU activation for hidden layers. It has hidden layers with 20 neurons each.
D.2.2 SFQI.
Q-learning uses feedforward network with bias and a ReLU activation for hidden layers. It has for each the three feature dimensions separate fully connected subnetwork. Each subnetwork has hidden layers with 20 neurons each.
24

---

Under review as a conference paper at ICLR 2022
D.2.3 CONTINUOUS MODEL-FREE Ū-LEARNING
The racer environment has continuous features ò R Therefore, the MF Ç-learning procedure (Alg. 3 can not be directly applied as it is designed for discrete feature spaces. We introduce here MF learmingg procedure for continuous feature spaces (CMF g-learning). is feature dimension independent, and discretized version of Ç-learning. As the reward functions 44 are sum over the individual feature dimensions, the Q-value can be computed as:
/ R(O)5" (s, a, () = -E k
(*( =
rh (((()) (s. )db
(45)
where Dy is the feature space for each feature dimension which is 4b [0. 1] in the racer environment. & a f-function for the feature dimension k. (45) shows that the E-function can be independently represented over each individual feature dimension Φs, instead of over the full features Φ. This reduces the complexity of the approximation.
Moreover, we introduce discretization of the ξ-function that discretizes the space of each feature dimension U = 11 bins with the centers:
O
mar
min
X = { o + jAde 0 < j jKU with AΦL.
where Aøv is the distance between the centers, and ø 0.0 is the lowest center, and 0% 1.0 the largest center. Given this discretization and the decomposition of the Q-function according to (45). the Q-values can be computed by:
(s Σ E (s,)""ss,a,
FEXI
Alg. 5 lists the complete CMF f-learming procedure with the update steps for the Ê-functions. Similar to the SFQI. agent, the CMF Xi uses a feedforward network with bias and ReLU activation for hidden layers. It has for each of the three feature dimensions separate fully connected subnetwork. Each subnetwork has hidden layers with 20 neurons each. The discretized outputs per feature dimension share the last hidden layer per subnetwork.
The É-function is updated according to the following procedure. Instead of providing discrete learning signal to the model, encode the observed feature using continuous activation functions around each bin center. Given the j 'tt bin center of dimension k, Ik.j. its value is encoded be 1.0 if the feature value of this dimension aligns with the center (Ø), Ik.j). Otherwise, the encoding for the bin decreases linearily based on the distance between the bin center and the value ([ark.j - Φx and reaches 0 if the value is equal to neighboring bin center, i.e. has distance 2 AΦg. We represent this encoding for each feature dimension & by uy E (0, 1) with:
VLE Vocieu Uks = max (, (1 [a +b
To learn & update the parameters 8 using stochastic gradient descent following the gradients aas (85) of the loss based on the g-learning update 13
3
Uk + (8 aet1; 05) - (5e, ar; 05) 2
VΦ E +: (e(09) = E
(46)
with ā,t1 argmax ΣΣ 00
A ZEN,
where n 3 is the number of feature dimensions and Ea the vector the discretized f-values for dimension k.
D.3 EXPERIMENTAL PROCEDURE
All agents were evaluated on 37 tasks. The agents experienced the tasks sequentially, each for 1000 episodes (200, 000 steps per task). The agents had knowledge when task change happened.
25

---

Under review as a conference paper at ICLR 2022
Each agent was evaluated for 10 repetitions to measure their average performance. Each repetition used a different random seed that impacted the following elements: a) the sampling of the tasks, b) the random initialization of function approximator parameters, c) the stochastic behavior of the environments when taking steps, and d) the e-greedy action selection of the agents. The tasks, i.e. different between the repetitions of particular agent, but identical the
the reward functions, same repetition of a different agent. Thus, all algorithms were evaluated over the same tasks.
SFQL was evaluated under two conditions. First, by learning the reward weights online during the training (indicated by (O) in figures). Second, the reward weights were trained with the iterative gradient decent method (4)) The weights were trained for 10, 000 iterations with an learning rate of 1.0. At each iteration, 50 random points in the task were sampled and their features and rewards are used for the training step.
Hyperparameters A grid search over the learning rates of all algorithms was performed. Each learning rate was evaluated for three different settings which are listed in Table 2 If algorithms had several learning rates, then all possible combinations were evaluated. This resulted in different number of evaluations per algorithm and condition: QL- SFQL (0)- 12, SFQL - 4, CMF Xi In total, 24 parameter combinations the parameter combination that resulted in the highest cumulative total reward averaged over all 10 repetitions in the respective environment. The probability for random actions of the e-Greedy action selection was set to é = 0.15 and the discount rate to 0.9. The initial weights and biases for the function approximators were initialized according to an uniform distribution with 8 ~ 21-lu((kk k, vk), where k features
evaluated. The reported performances in the figures are for
Table 2: Evaluated Learning Rates in the Racer Environment
Parameter
Description
Values
Learning rate of the Q vh. and f-function Learning rate of the reward weights
(0.0025. 0.005, 0.025, 0.5) {0.025, 0.05, 0.075)
Ow
Computational Resources: Experiments were conducted the same cluster as for the object collection environment experiments. The time for evaluating one repetition of a certain parameter combination over the 37 tasks depended the algorithm: QL = 9h, SFQL (O) 70h, SFQL se 73h, and CMF & = 88h. Please note, the reported times do not represent well the computational complexity of the algorithms, as the algorithms were not optimized for speed, and some use different software packages (numpy or pytorch) for their individual computations.
26

---

Under review as a conference paper at ICLR 2022
Algorithm 5: Model-free Ç g-learning for Continuous Features (CMF)) {)
Input:
exploration rate:
learning rate for ξ-functions: & learning rate for reward models R: aR features á E R
components of reward functions for tasks: (R1 (ri, r2 22, rh), R2:.... Raum_taske tasks
discretization parameters: X, Aø
for i +- 1 to num tasks do
if 1 then
Vee(I.....) initialize Eg: 0f f- small random values
else
(KE(I.... G- t of
new new_ep _epeisdddde--- t- true for t t to num num_steps do if new new_episode then new new_episode false
8; - initial state
c argmax max, Er=1 Exex, a,)s,,
GPI policy
With probability c select random action at, otherwise
argmax, Ex= ΣxEx, Cx a, ())((()) Take action ae and observe reward rt and next state $141 if st+1 is terminal state then new new_ jow_episode- true
0
else
L Ytt +-7
// GPI optimal next action for task i
Ūz-1 + argmax, (f Ei=1 Exex,
Φ: < o(1,, ae, $1+1) for l to do foreach x E X do
Uk.z +- max 0,
|m-de,e] A0
+ 7651
Update of using SGD(a) with Le -tt Exex, (yk.z (t), if then
ae
// optimal next action for task
Øc+1 argmax, E ExEx, ())())))))(())) for l: to do
foreach x E Xx do
Uk.z +- max 0,
(2- + 0+41, Ad
Update 05 using SGD(a) with Cę Ek= EvEx, (8k. Eg(s ae, 2))
8:
Se+1
27

---

Under review as a conference paper at ICLR 2022
E ADDITIONAL EXPERIMENTAL RESULTS
This section reports additional results and experiments:
Report of the total return and the statistical significance differences between agents for all experiments
2. Evaluation of the agents in the original object collection task by Barreto et al. (2017)
E. OBJECT COLLECTION TASK BY BARRETO ET AL. (2017)
We additionally evaluated all agents in the original object collection task by Barreto et al. 2017
The environment differs to the modified object collection task (Section. [ only in terms of the objects and features. The environment has 3 object types: orange, blue, and pink (Fig. 3 The feature encode if the agent has collected one of these object types or if it reached the goal area. The first three dimensions the features ((5t, dt, St+1) 104 encode which object type is collected. The last dimension encodes the goal area was reached.
Environment:
In total | possible features exists: Φ1 [0, O, O, 0] standard observation, Φ [1,0, 0, 0, collected an orange object, Φ3
[0, 1, 0J collected blue object, de [0, 0, 1, 0J T collected pink object, and Φ5 [0, O, 0, 1] reached the goal area.
Figure 3:
Object collec-
tion environment from (Bar- reto et al. (017)) with 3 object types: orange, blue, pink.
The rewards
Φ w, defined by linear combination of discrete features o E N and a weight vector w E I The first three dimensions w define the reward that the agent receives for collecting one of the object types. The final weight defines the reward for reaching the goal state which is W, for each task. All agents were trained in on 300 randomly generated linear reward functions with the same experimental procedure as described in Section. C For each task the reward weights for the 3 objects are randomly sampled from a uniform distribution: WxE(1223) ~ (((111)
The results (Fig. 4 follow closely the results from the modified object collection task [rig---a b, and [5] a). MF reaches the highest performance outperforming SFQL in terms of learning speed and asymptotic performance. It is followed by MB and SFQL which show statistical significant difference between each other their final performance. Nonetheless, MB has higher learning speed during the initial 40 tasks. The results for the agents that learn the reward weights online (SFQL (O), MF & (O), and MB (O)) follow the same trend with MF (O) outperforming SFQL (O) slightly. Nonetheless, the E-agents have a much stronger learning speed during the initial 70 tasks compared to SFQL (O), due to the errors in the approximation of the weight vectors, especially at the beginning of new task. All agents can clearly outperform standard Q-learning.
Results:
E.2 TOTAL RETURN IN TRANSFER LEARNING EXPERIMENTS AND STATISTICAL SIGNIFICANT DIFFERENCES
Fig. shows for each of the transfer learning experiments in the object collection and the racer enviromment the total return that each agent accumulated over all tasks. Each dot besides the boxplot shows the total return for each of the 10 repetitions. The box ranges from the upper to the lower quartile. The whiskers represent the upper and lower fence. The mean and standard deviation are indicated by the dashed line and the median by the solid line. The tables in Fig. report the p-value of pairwise Mann-Whitney U test. A significant different total return can be expected if p < 0.05.
For the object collection environment (Fig a; b), E-learning outperforms SFQL in both conditions, in tasks with linear and general reward functions. However, the effect is stronger in tasks with general reward functions where SFQL has more problems to correctly approximate the reward function with its linear approach. For the condition, where the agents learn reward model online (O), the difference between the algorithms in the general reward case is not as strong due the effect of their poor approximated reward models for all agents.
28

---

Under review as a conference paper at ICLR 2022
In the racer environment (Fig c) SFQL has poor performance below standard Q-learning can not appropriately approximate the reward functions with a linear model. In difference CMF E-learning outperforms QL.
(a) Environment by Barreto et al. (2017) with Linear Reward Functions
Qr SFQU. MF Xi (0) MR XI (0)
SEQI
MF
MR XI
Average Retur
600
400
200
10
80
90
100
110
120
130
160
170 180
190
200
210
220
230
240
260
270
280
290
300
Tasks Trained
(b) Total Return and Statistical Significance Tests
SEQL (O)
(i)
250k
200k
Total Retum
150k
100k
50k
p-value QL.
XI
MB XTTO)
SFQL
MF
Xi
< 0.001
0.001
0.001
< 00.011
0.000
< 0.001
SFQL (0)
0.038
0.121
< 0.001
< 0.001
0.001
MF
0.623
0.001
0.001
0,001
MB Xi (0)
< 0.001
0.001
0.001
SFQL
MF Xi
0.001
Figure 4: MF E-learning outperforms SFQL in the object collection environment by Barreto et al. (2017), both in terms of asymptotic performance and learning speed. (a) The average over 10 runs of the average reward per task per algorithm and the standard eтror the mean depicted. (b) Total return over the 300 tasks in each evaluated condition. The table shows the p-values of pairwise Mann-Whitney Whitney tests between the agents.
29

---

Under review as a conference paper at ICLR 2022
(a) Object Collection Environment with Linear Reward Functions
QL
SPQL (O)
MF (i)
MB (i)
SPQL
MF Xi
MB Xi
200k
Total Return
100k
50k
p-value
SFQL (0)
MF Xi (O)
MB O
SFOL
MF Xi
MB Xi
QL
< 0.001
0.001
0.001
0,001
0.001
0.001
SFQL (O)
0.001
0.001
0.001
0.001
0.001
MF Xi (O)
0.054
< 0.001
0.001
< 0.001
(O)
SPQL
0.001
0,001
0.162
(b) Object Collection Environment with General Reward Functions
QL
SPQL (O)
MF XI (O)
MB Xi (O)
SPQL
MF XI
MB Xi
2504
Total Return
200k
150k
100k
p-value
SFQL (0)
MF (i)
MB Xi
SFQL
MF Xi
MB Xi
QL
< 0.001
0.001
0.001
0.001
< 0.001
< 0.001
SFQL (0)
0.045
0.009
0.001
0.001
0.001
MF Xi (O)
0.345
0.001
0.001
0.001
(0)
0.0001
SFOL
0.001
0.001
MF
0.001
(c) Racer Environment with General Reward Functions
SPQU
SFQI.
CMP Xi
3M
Total Return
2.5M
QL
SFOL (O)
< 0.001
<.000
SFQL
< 0.001
CMF Xi
SFQ
65
SFQL.
0.001
2M
1.5M
Figure 5: Total return over all tasks in each evaluated condition. The tables show the p-values of pairwise Mann- Whitney U tests between the agents. See the text for more information.
30


Reviews:
[{"review_id": "vwwcjGfDq21", "reviewer": "Reviewer_UrNN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The motivation of this paper is not described and supported well. Why the assumption of linear composition is bad, or not applicable to what domains? From the results, previous SF methods perform not that badly on the problem with a general reward function.\nIt is more suitable to say two environments with discrete state space and continuous state space.\n\n\nFor experiments, \nWhy not compare with [1], and some SOTA meta RL algorithms?\n\nFor the second environment, why discretize the state space, since deep SF can be applied to continuous domains?\n\nThe reason that why MF \\xi-learning outperforms MB \\xi-learning is not discussed. Since MF \\xi-learning is superior, what is the need to propose MB \\xi-learning? \n\nResults of MB \\xi-learning in the second environment are not provided.\n\nSome descriptions need to be improved. MB \\xi-learning refers to Model-based \\xi-learning, not Modle \\xi-learning.\n\n\n[1] Fast reinforcement learning with generalized policy updates.\n", "summary_of_the_paper": "This paper proposes a new successor feature learning algorithm, called \\xi-learning. Previous SF assumes reward function is a linear composition of SF and reward weights, this paper extends previous SF to a setting with a general reward function over SF. Based on this, \\xi-learning learns to estimate the cumulative discounted probability of SF, and provides two update operators, model-free \\xi-learning and model-based \\xi-learning. The convergence of \\xi-learning is proved, and empirical results on environments with discrete and continuous state spaces show it outperforms previous SF methods.", "summary_of_the_review": "I think this paper contributes to the SF research areas, but the motivation is not well discussed. Also, there are some details, analysis of results should be well discussed. Based on this, I give a borderline reject currently.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nhvc6P9CIp", "reviewer": "Reviewer_Ccnn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper tackles the important problem of transfer in reinforcement learning. However, the generalization it proposes is oddly positioned. The main premise is described as:\n\n> Nonetheless, this assumption also restricts successful application of\nSF&GPI only to problems where such a linear decomposition is possible. This paper investigates the\napplication of the SF&GPI framework to general reward functions: $r_i = R_i(\\phi)$.\n\nThis is true in the sense that the linearity of successor features prevents them from representing arbitrary sets of reward functions. (Any individual reward function is representable by setting $\\phi_t = r_t$.) One easy example of a reward set that SF could not represent (in continuous spaces) is that of Dirac delta functions over all states, as this would require feature vectors $\\phi$ of the same cardinality as the state space. While the proposed $\\xi$ variant is indeed more general than the linear SF, these rewards are still functions of $\\phi$, so the expressivity of $\\xi$ depends on the expressivity of the featurization $\\phi.$\n\nOne possible response to that criticism is to simply set $\\phi_t = \\text{concat}(s_t, a_t, s_{t+1})$, which really would fulfill the promise of handling general reward functions. However, this raises a serious practical issue, in that we must now recover the discounted occupancy of a policy in the original state space. This is a pretty difficult combination of dynamic programming and generative modeling (each in their own right hard problems), so is not straightforward to scale (though the paper cites a few attempts). I did not find the discussion of this difficulty too convincing; the main strategy here seemed to rely on a coarse discretization of continuous spaces. I would expect this to work well in the diagnostic environment studied, but mostly because the underlying state is low-dimensional enough that discretizing it and treating it as tabular is a reasonable strategy. (The actual observation is higher-dimensional, but the underlying state looks like position and velocity in $\\mathbb{R}^2$.)\n\nI think these two issues put the paper in a strange position:\n1. If the feature vectors $\\phi$ are unrestricted, then $\\xi$ really is more expressive than linear SF, but the method requires solving a long-horizon generative modeling problem to represent the state occupancy and does not propose a viable way of doing this.\n2. If the feature vectors $\\phi$ are restricted in such a way that $\\xi$ is learnable in practice, then $\\xi$ likely ends up in the same spot as SF: able to represent any _individual_ reward function, but not any arbitrary set of rewards due to limitations on the featurization itself.\n\nIf there were evidence presented that $\\xi$-learning is more scalable than I am giving it credit for (_e.g._, compared to SF, in environments for which the linearity assumption is limiting without artificial constraints on the featurization or observation space), then of course I would revise my review.\n\nI want to make sure to list the good too: the paper positions itself well in the SF and GPI literature. While the analysis mostly follows from analogous results in $Q$-learning, it is thorough. There is clean code provided.\n\n**Copy-editing**\n\n*Introduction*:\n1. \"intput\" --> \"input\"\n2. \"an cumulative\" --> \"a cumulative\"\n\n*Background*\n1. \"following following the Bellman equation \": repeated \"following\"\n2. \"a MDP\" --> \"an MDP\"\n3. \"proofed\" --> \"proved\n4. \"error in the approximation increase\" --> \"increases\"\n\n*Method*\n1. (Prop 1) \"quotien space\" --> \"quotient space\"\n2. (Corr 2) \"supreme norm\" --> \"supremum norm\"?\n3. (Prop 2) \"well-define\" --> \"well-defined\"\n\n*Experiments*\n1. \"which object as been already collected\" --> \"which object has\"\n2. \"number of Gaussian’s\": no apostrophe", "summary_of_the_paper": "This paper proposes a variant of successor features that represents the unnormalized cumulative discounted distribution of state features $\\xi(\\mathbf{s}, \\mathbf{a}) = \\sum_t \\gamma^t p(\\phi_t \\mid \\mathbf{s}_0 = \\mathbf{s}, \\mathbf{a}_0 = \\mathbf{a})$ instead of the standard expectation $\\psi(\\mathbf{s}, \\mathbf{a}) = \\mathbb{E} [ \\sum_t \\gamma^t \\phi_t \\mid \\mathbf{s}_0 = \\mathbf{s}, \\mathbf{a}_0 = \\mathbf{a}]$. It proposes both a model-free and model-based variant for training $\\xi$ and evaluates the effect of non-linear reward functions in two diagnostic environments.", "summary_of_the_review": "I am in the borderline mode for this paper. Due to the difficulty of the generative modeling problem associated with $\\xi$-learning, I am not quite convinced that the generalization presented in this paper would actually lead to something more expressive than SF in the types of setting where the linearity of SF proves limiting. That said, this paper does have merits and adds something new to the SF literature, as mentioned in the main review above. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "goPjWS8EJB", "reviewer": "Reviewer_wZjF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On the writing:\nThe paper is nicely written and well structured, making it a pleasant read. It also appears to be technically sounds. \n\nOn the proposed method:\nMy main concern is that authors didn't acknowledge previous efforts by Barreto et al. 2018 to address the problem of general reward functions. The authors state that the upper-bound in (7) is only interesting if the reward function can be decomposed linearly with respect to state-action features. Barreto et al. 2018 proposed a principled way of learning features (or even the reward function) such that the approximation error on the reward remains low, and the upper-bound on the optimal Q-value error remains low as well.\n\nWithout a comparison with Barreto et al. 2018 method, it is difficult to understand the necessity of introducing a new quantity to learn (the expected cumulative discounted probability of SF). While I like the formalism and the theoretical guarantees, to recommend the paper to be accepted, I would need more convincing on the necessity of the xi-function, specially given how difficult it will be to learn in large and stochastic environments.  \n\nOn the experiment:\n- Authors should specify how the weights \\tilde{w_i} were learned for SFQL in the general case. \n- Results including the learning of the reward function - or weights w - would be very interesting to appear in the main text, as in practice we rarely have access to the reward function. It would help build a case for practical use of the proposed method. \n- It seems to me that the continuous environment is rather a discretized environment? \n- I did not understand the proposed justification for xi-learning improving on SFQL (even slightly) in environments with linear reward function. What did the authors mean by \"xi-learning reduces the complexity for the function approximation of the xi-function compared to the phi-function in SFQL\"?\n\nMinor comment: I found the second sentence of Theorem 1 difficult to understand. \n", "summary_of_the_paper": "This paper addresses a limitation of the successor features (SF) which were introduced as a mechanism for transfer learning in reinforcement learning when the reward function changes across tasks. The authors claim that the original SF framework will only provide a good approximation of the true Q-function if the reward function in a task can be represented using a linear decomposition, and propose to address this linear decomposition of the reward function requirement by introducing a novel SF mechanism, Xi-learning, based on learning a cumulative discounted probability of successor features. The paper includes theoretical proofs of the convergence of xi-learning as well as transfer learning guarantees under generalized policy improvement (GPI). The authors compare the performance of their methods against standard Q-learning and SFQL in two different domains. ", "summary_of_the_review": "Despite enjoying the reading of the paper, the main reason for my recommendation is that due to the lack of discussion / comparison with another method that adresses the non-linearity of the reward function to learn accurate SF, I am not convinced by the necessity of the xi-function, which is the novelty of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XUDNtC1ooP5", "reviewer": "Reviewer_wA2K", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose an interesting extension to linear SF framework. The derivations look correct to me and are overall well written. \n\nJust a few suggestions on the presentation of the method. \n- It would be helpful if you reduce the new formulation to the special case linear SF around equation (13) and illustrate the key difference. \n- discuss the difficulty of simultaneously learning features and estimating the density of features.\n\nKudos to the authors conducted u-test for the significance of the results. Appreciated the effort in making more rigorous results. \n\nThe primary weakness of this work is the experimental section. \nThere seems to a number of different issues all conflated into one set of experiments.\n\nFirstly, in both collection and racer environments, the features are manually defined instead of learning from observations. This brings up a major concern that it hurts the flexibility and expressiveness of linear SF. \nWith learned features, even if we assume reward is a linear function of features, the resulting value function is not hindered and in fact the expressiveness of value function is not limited by the linearity, as features can be an expressive and arbitrarily non-linear function of observations. \n\nSecondly, it seems in all experiments the evaluation tasks set is the same as the training tasks set. This makes make it hard to interpret the experiments or understand how effective is the transfer. I encourage the authors to divide the tasks into training set and hold out test set, and show the average return on the hold out set. \n\nLastly, I believe the authors should compare again random feature baselines, where a neural network is randomly initialized and fixed during training, and then running linear SF and xi-learning on top it. Similar to Hansen et al. This would provide insights on how the SF works in these domains. \n\nThese issues are probably fixable by more careful experimentation and clarity on the exact setup.\n\nA general weakness is that the tasks are very simple, to the point that it is very unclear why the linear SF formulation does worse than xi-learning. I encourage the authors to try to evaluate the method at least on the Mujoco tasks from Barreto et al 2018. \n\nA question about the reward definition. The rewards for collection environment is defined as “The general reward functions are sampled by assigning a different reward to each possible combination of object properties ..., such that picking up an orange box might result in a reward of ...”, I don’t fully understand the rationality behind the particular choice, could the authors elaborate more? ", "summary_of_the_paper": "The authors extend the successor features (SF) linear formulation to general non-linear formulation. The new method allows the reward to be arbitrary composition of features. \n\nThe propose method is based on learning cumulative discounted probability of SF rather than cumulative discounted sum of SF as in linear SF framework. \nThe authors provide theoretical proofs about convergence and propose two practical methods. Experiments are conducted in two toy environments.\n", "summary_of_the_review": "In summary, the paper is an interesting effort in extending the linear SF framework. \n\nBut why the linear assumption in SF framework is bad is not well supported, especially when the features can be an expressive and arbitrarily non-linear function of observations. \n\nThe experiments failed to show very convincing results due to simple environments and problematic experimentation setup.  \n\nThe authors are encouraged to address these concerns. It would be a nice contribution if the concerns are resolved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers uniformly suggested rejecting the current paper.

I concur and remain especially somewhat unconvinced by the authors comments on learning features.  In particular, any argument based simply on (current) performance seems rather weak.  There are methodological reasons one might want to keep features fixed, and there are a small subset of problems with well-defined known useful features.  But in the long term surely we should want to be able to learn the features, and efficiently and elegantly handle the case where they are learnt continually.

I want to thank the authors for engaging.  This work has the potential to be improved and I would encourage the authors to carefully consider and incorporate the provided feedback by the reviewers into their work.
Decision: Reject