Second-Order Rewards For Successor Features
Authors: Norman L Tasfi
Miriam Capretz
Venue: ICLR 2022 Submitted (2022)

Abstract: Current Reinforcement Learning algorithms have reached new heights in performance. However, such algorithms often require hundreds of millions of samples, often resulting in policies that are unable to transfer between tasks without full retraining. Successor features aim to improve this situation by decomposing the policy into two components: one capturing environmental dynamics and the other modelling reward. Where the reward function is formulated as the linear combination of learned state features and a learned parameter vector. Under this form, transfer between related tasks now only requires training the reward component. In this paper, we propose a novel extension to the successor feature framework resulting in a natural second-order variant. After derivation of the new state-action value function, a second additive term emerges, this term predicts reward as a non-linear combination of state features while providing additional benefits. Experimentally, we show that this term explicitly models the environment's stochasticity and can also be used in place of $\epsilon$-greedy exploration methods during transfer. The performance of the proposed extension to the successor feature framework is validated empirically on a 2D navigation task, the control of a simulated robotic arm, and the Doom environment.

Under review as a conference paper at ICLR 2022
SECOND-ORDER REWARDS FOR SUCCESSOR FEA- TURES
Anonymous authors Paper under double-blind review
ABSTRACT
Current Reinforcement Learning algorithms have reached new heights in perfor- mance. However, such algorithms often require hundreds of millions of samples, often resulting in policies that are unable to transfer between tasks without full retraining. Successor features aim to improve this situation by decomposing the policy into two components: one capturing environmental dynamics and the other modelling reward. Where the reward function is formulated as the linear combi- nation of learned state features and learned parameter vector. Under this form, transfer between related tasks now only requires training the reward component. In this paper, propose novel extension to the successor feature framework resulting in a natural second-order variant. After derivation of the new state-action value function, second additive term emerges, this term predicts reward as non-linear combination of state features while providing additional benefits. Exper- imentally, we show that this term explicitly models the environment's stochasticity and can also be used in place of e-greedy exploration methods during transfer. The performance of the proposed extension to the successor feature framework is validated empirically on 2D navigation task, the control of simulated robotic arm, and the Doom environment.
INTRODUCTION
Recently, Reinforcement Learning (RL) algorithms have achieved superhuman performance sev- eral challenging domains, such as Atari (Mnih et al., 2015), Go (Silver et al.] 2016), and Starcraft [[(inyylls et al. 2019). The main driver of these successes has been the use of deep neural net- works, which are a class of powerful non-linear function approximators, with RI. algorithms(LeCCum al. 2015). However, this class of Deep Reinforcement Learning (Deep RL) algorithms require immense amounts of data within environment, often ranging from tens to hundreds of millions of Gamplle(Arulkumarcs et al. 2017). Furthermore, commonly used algorithms often have difficulty in transferring learned policy between related tasks, such as where the environmental dynamics remain constant, but the goal changes. In this case, the model must either be retrained completely fine-tuned on the new task, in both cases requiring millions of additional samples. If the state dynamics are constant, but the reward structure varies between tasks, it is wasteful to retrain the entire model.
A more pragmatic approach would be to decompose the RL agent's 's policy such that separate functions can learn the state dynamics and the reward structure: doing so enables reuse of the dynamics model and only requires learning the reward component. Successor features (Dayan 1993) do precisely this; model-free policy's action-value function is expressed as the dot product between vector of expected discounted future state occupancies, the successor features, and another vector representing the immediate reward each of those successor states. The factorization follows from the formulation of the reward the dot product between state representation vector and learned parameter vector, that is a linear product. Therefore, transfer to a new task requires relearning only the reward parameters instead of the entire model and amounts to the supervised learning problem of predicting the current state's ' immediate reward.
As the reward function of the successor feature framework is linear, it is fair to question whether the model can always accurately predict the reward. As no assumptions are made about the state representation, theoretically it is possible enable perfect recovery of any reward function if given

---

Under review as a conference paper at ICLR 2022
predictive state representation Barreto al. (2017). The state representation, within the successor feature framework, is learned end-to-end by a state encoder to perform well in state reconstruction and reward prediction tasks. The state encoder, given large enough set of parameters, should have enough representational power to disentangle the factors that are useful for reward prediction by utilized for both
linear model. However, because of how the encoder is trained, its parameters state reconstruction and reward prediction tasks; while the reward model parameters are only used for reward prediction. If the encoder learns sub-optimal state representation for reward prediction, say because of highly complex environment, the reward model might be unable to compensate with its limited of parameters correctly to predict the reward. Eysenbach et al. (2018) and Hansen et al.) (2019) have shown, within the successor feature framework, that there is no strong guarantee that the state encoder always able to learn features that enable accurate modelling of the reward.
In this paper, a novel extension to the successor feature framework is proposed, where the rewards are modelled with a second-order function. The second-order function, which follows naturally from the original linear variant, gives stronger guarantee on performance of the model due to both its representational structure and extra parameters. This especially true cases where the encoder learns state representations that are less than optimal where linear model does not have enough rep- resentational power to compensate. While, in cases where the encoder can learn sufficient represents for both reconstruction and reward tasks, the second-order function still provides many added benefits. In particular, the additional parameters of the reward model should lessen the representation load of the encoder with regards to the reward tasks allowing more of its representational capacity be dedicated towards modelling the environment in a task agnostic manner. Further benefits, via new term emerging after derivation, include a representational form of environmental stochasticity and the ability to use directed exploration during transfer instead of relying on purely random approach for exploration, such as e-greedy.
Following this, the contributions of this research are as follows:
A novel formulation of successor features that uses second-order reward function. This formulation increases the representational power of the reward function while decreasing the representational load the state encoder providing stronger guarantees on performance.
Under the reward formulation, second term appears that models the future expected auto-correlation matrix of the state features.
We provide preliminary results that show the second term can be used for guided exploration during transfer instead of relying on e-greedy exploration.
After the introduction of relevant background material Section we introduce the successor feature framework with non-linear reward function in Section 3 Section provides experimental support and provides an analysis the new term the decomposition. The paper concludes with final discussion and possible avenues for future work in Section
BACKGROUND
2.1 REINFORCEMENT LEARNING
Consider the interaction between agent and environment modelled by Markov decision process (MDP) (Puterman 2014). An MDP is defined as set states S, a set of actions A, a reward function R: S IR, a discount factor 7 [0, 1], and transition function T': A [0, 1]. The transition function gives the next-state distribution upon taking action in state and is often referred as the dynamics of the MDP.
The objective of the agent in RL is to find policy π. mapping from states to actions, which maximizes the expected discounted sum of rewards within the environment. One solution this problem is to rely on learning a value function, where the action-value function of policy π is defined as:
(= E
7 S( = =s, A, -
1=0
2

---

Under review as a conference paper at ICLR 2022
St
St
ENCODER
dof
Tt
St
ENODDER
Φt
A(or, ai)
W
p(br, at)
(a) Model Overview
(b) Reward Prediction
Figure 1: Model Overview a) The encoder transforms the raw state into an internal state represen- tation (s).. The state representation (r(s) is used by the decoder, A (= ac), and " (:.a))) The decoder tries to reconstruct the imput 9, from the state representation (s). A and 인 produce
one output per action, with the former predicting matrices and the latter predicting vectors. b) Reward prediction by dotting the current state (s)), produced by the encoder, and reward weight
where E= [...] denotes the expected value when following the policy и. The policy is learned using an alternating process of policy evaluation, given the action-value of a particular policy and policy improvement, which derives a new policy that is greedy with respect to Q= a)(Putemaan 2014)..
2.2 SUCCESSOR FEATURES
Successor Features (SF) offer decomposition of the Q-value function and have been mentioned under various names and interpretations (Dayan, 1993] Kulkarni et al., 2016; Barreto et al.) 2017 Machado et al. 2017). This decomposition follows from the assumption that the reward function can be approximately represented as linear combination of learned features ((s; 0g) extracted by a neural network with parameters 0g and reward weight vector As such, the expected one-step reward can be computed as: r(s, a) o(s, a, 0 Tw. Following from this, the Q function can be rewritten as:
w.
((s ET rt+1 + 7442 +... 1S, = =s =
E 101111, 0(a++2 8142: 0 Tw .... SS: = 8, Ar
Q(s.a) 2i os) as)
where (s, a) are referred as the successor features under policy æ. The 7t component of gi(s, a) provides the expected discounted sum of ob when following policy π starting from state s and action a. assumed that the features Φ(s; 0g) are representative of the state s, such that (b).) can be turned into function 20 (p(s;;;; az). For brevity, o(s;; 0%) is referred to simply as Φi and ei (s, a) as (ss,a).
The decomposition neatly separates the Q-function into two learning problems, for sith and wt estimating the features under the current policy dynamics and estimating the reward given state. Because the decomposition still has the same form as the Q-function, the successor features are computed using Bellman equation update in which the reward function is replaced by d.:
(or, az) = de + 7BB N0 (0141, a++1)
such that approximate successor features can be learned using an RI. method, such as Q- Learning(Szepesvánj 20099)
Following from this, the approximation of the reward vector w becomes supervised learning problem. Often, this weight is learned using ordinary least squares from the sampled environmental data. One benefit of having a decoupled representation is that only the relevant function must be relearned when either the dynamics the reward changes. Therefore, if the task changes, but the environmental dynamics remain constant, only the reward vector parameters w must be relearned, which are minimal compared to the total number of parameters in the full model.
3

---

Under review as a conference paper at ICLR 2022
3 MODEL, ARCHITECTURE, AND TRAINING
A natural extension to the Successor Feature framework begins by adjusting the fundamental structure of how the reward is represented. In this work, second-order extension proposed that improves the flexibility of the reward function while providing other benefits. This paper shows that by improving the representational power of the reward component, with a non-linear function of the state, it provides stronger guarantee of the framework's performance in such cases by developing more robust reward component.
This section discusses our change to the successor feature framework, which adjusts the reward function, from a linear function, to a non-linear function. First, a discussion of the new decomposition is given with the full derivation provided in Appendix [ Then experimental support for this change will be presented and analyzed to examine what the new term the decomposition learns.
3.1 NON-LINEAR REWARD FUNCTION
The successor feature framework builds upon functional representation of the current reward rt as a linear combination of the current state representation (s))) E I and a learned reward vector w E IR=, such that Ti = (x))) w. In this paper we extend the reward model by changing this linear reward model to one with the following form:
r2 (r(s) (s)
where {o) RF, and A R=x= Both and A are learnable parameters modelling the reward structure of the environment. E shows that the formulation introduces a non-linear transformation with respect to (s).. From here on, we use &i instead of (s) for brevity. With a slight abuse of notation, we can see the original formulation leads to this if w is replaced with Ad: Ti @ (o + AΦ). The state-action value function Q(s, a), under this new reward structure, can be derived to yield:
(" a) (st, a) To + Str(AA" a))
where B E {0, 1} controls the inclusion of A and tr is the trace operator. It can now be shown that v and A satisfy the Bellman qquation(Bellman 1966):
s)" (s: a) E" (o:41 +
A, a] = s, A, a
(3)
A E 7(stt)
(4)
where for e and A, & and ΦΦ respectively play the role of rewards. In addition to th it is now necessary to model A, which outputs an R=** matrix per action. The quantity Φx can be interpreted as an auto-correlation matrix of the state features. We can see that this form allows the A term to model some form of future expected stochasticity of the environment. For example. the diagonal of A will model a second order moment capturing each feature's 's change with respect to itself a We provide analysis and further discussion of A in Section 4.5
3.2 MODEL STRUCTURE AND TRAINING
The proposed model, shown in Figure uses encoder to produce a state embedding di consumed by downstream modelling tasks. Figure shows how the current reward 7r is predicted using w, with w AΦ, and current state representation Φ;; this process is defined in Equation [ Similar to previous work with successor features, the structure includes pathways for an incode-decode task and successor feature prediction ei(Machado et al., 2017] Kulkarni et al.) [0166] Zhang et al.) (007)) The decoder network ensures that the features learned by the encoder, which produces o, contain useful information for prediction. Furthermore, only the gradients from the state-dependent and reward prediction tasks modify the encoder parameters, and therefore é. An additional branch is added, by way of the non-linear reward function, model the quantity A A(s, a). This branch's output is a matrix, which differs from the vector predicting branches vé and Φ.
A quantity close the variance, but not zero mean.
4

---

Under review as a conference paper at ICLR 2022
The encode-decode task is trained by minimizing the mean squared difference between the input st and the decoder's 's reconstructed version it from Ø::
L (sr; 09, 00) [sr g(tr; )²)
where @ is the output of the encoder with parameters 8 and g(:: (0) produces the output of the decoder with parameters 8 As mentioned previously, we train ab and A, parametcrized with 0 and A respectively, using the Bellman equations to minimize the following losses:
C (s2, ar; 0%)
El(o, + ""0(s+t1,";;0"" () a,; a1: )
(6)
Ca ae; (A) Bi(o, क. "A($)+,, 0- - A(s a,; 8a) )²
(7)
where a" = m&X,' Q(s, (ss,a)), ). To help stabilize learning, we use lagged versions of 09, 08, and 0a as done by Myih al.] (2015): the lagged version is signified with the symbol in the exponent.
Unfortunately, as the dimensionality of a grows, the number of parameters needed by A grows quadratically. However, by identifying the ó term A(s, a) as a symmetric matrix, is possible to model only the upper triangular portion of the matrix ? requiring about half the number of parameters. To further reduce parameters, each v and pathways have two hidden layers before their outputs, reflected in Figure la In this way, the parameters are shared amongst pathways. which contrasts with other works with multiple sets of layers per action a E A (Kulkarni et al., 2016 Zhang al. 2017). To learn the reward parameters A and o, which are the parameters of the approximated non inear reward function, the following squared loss function is used:
("r o, A) = |r+ o, Bo] A
(8)
does not adjust the feature parameters involved in the prediction of Φt. We factorize matrix A to reduce its parameters, details are provided in Appendix Combining our losses, the composite loss function is the sum of the four losses given above:
C( 0z 00 0 o, A) C + +. +C"
In practice, to optimize Equation 9 with respect to its parameters, (0 0%) and (0%, 6o, o, A) are iteratively updated. Doing increases the stability of the approximations learned by the model and ensures that the branches modelling &b and A do not backpropagate gradients to affect 8% (Machado al.] 2017: Zhang et al., 2017, Kulkarni et al. 20166). Additionally, by training in this way, the state representation & can learn features that are both a good predictor of the reward rz and useful in discriminating between states (Kulkarni et al. 2016).
EXPERIMENTS
This section examines the properties of the proposed approach on Axes, a navigation task, on Reacher, robotic control task built using the MuJoCo engine (Todorov et al. (0122,, and a 3D) maze using the Doom game engine. The environments are shown Figure they each contain tasks specified by goal location and are split between training and test distributions, with the exception of Doom. The environments were chosen as both are similar to tasks in previous work on Successor Features (Barreto et al.) 2017) making comparison easier. The Axes and Reacher environments act test-bed for our method, allowing us to clearly show that the second-orde order function provides additional representation capacity the reward model. To do so, we purposely use weak encoder, represented by a single hidden layer, that can only learn a sub-optimal state representation.
Within the Axes and Reacher environments, two variants exist, easy and hard, where each refers to the difficulty of using the state for reward prediction. In both environments the reward function
takes the form - V VE (ge- a2)² where g is the current goal, a is the agent's perceived location, and C' the co-ordinate system. For Axes C' E (x,y) and E {ar, y, =} for Reacher. the easy variant, the state is represented as the distance between the agent and each possible goal within the
It would still be necessary manipulate this matrix that forms full matrix.
5

---

Under review as a conference paper at ICLR 2022
(d) Doom Environment.
(a) Axes Environment.
(b) Reacher Environment.
(c) Doom Map.
Figure 2: Environments A graphical representation of the Axes environment. The agent, shown as a red square, must traverse to various goal locations marked with the letter "G". The eight goal locations are split between training, shown blue boxes, and testing, shown green boxes. b) A rendering of the Reacher task. The agent controls the robotic Sawyer arm to move the end-effector to 3D point in space. The eight goal locations are shown balls. Training goals green, and test goals as red. c) The map layout of the Doom environment. The agent moves between rooms looking for goal point. d) Images of the Doom environment.
environment, making reward prediction easily accomplished by using a 1-hot reward vector. The easy variant is commonly used within other successor feature work Barreto et al. (2017). While, in work|Barreto
the hard variant, the state is simply the location of the agent and the location of the current goal. Therefore, the reward involves non-linear functions, square root and square powers, and the linear variant will have trouble modelling the reward of the hard environments with a sub-optimal state representation. The Axes and Reacher environments act as a test-bed to examine the various properties of the second-order function and to show its utility in controlled environment while examining the properties the second-order extension offers.
To demonstrate the general applicability of our proposed model to complex environments, we evaluate on a 3D navigation task the Doom environment from raw pixel This task challenging as the model must learn a state representation that is adequate for both reward prediction and use in the successor feature branch from raw pixels.
The primary set of experiments examined the performance between the proposed model and baselines over the training distribution tasks. As transfer to related tasks core benefit of the Successor Feature framework, we also evaluate how well the models transfer to unseen tasks from the test distribution on the Axes and Reacher environments. Next, we move to understand the new term that appears after derivation of proposed model. We examine the learned A function understand it captures environmental stochasticity and evaluate different guided exploration strategies using the A term tasks.
4.1 ENVIRONMENTS
Additional details each environment are provided Appendix C
Axes: In this environment, shown in Figure 2a the agent, shown by the red square, must traverse the map to reach a goal location using four actions: up, down, left, and right. The agent receives reward equal to the negative distance between itself and the target goal at each step. In the easy variant the state 8t is a set of distance between the agent and all available goals, such that 8; E R While in the hard variant, the state 5; is the agent's ss and the current active goal's 's 2D coordinates, such that 8, E 4
Reacher: The second environment is control task defined in the MuJoCo physics engine (ooooov ct al., 2012), shown in Figure (2b This environment was chosen to show that the proposed method scale to difficult control tasks. In this environment, the agent must move simulated robotic arm to a specific 3D point in space by activating four torque controlled motors. In the easy variant the of distance between the agent and all available goals, such that s; E R While in the hard variant, the state s, is the agent's 's and the current active goal's 2d coordinates, such that s; E g
state s;
Figures used with permission from the author Kulkarni al. 2016

---

Under review as a conference paper at ICLR 2022
Axes
Reacher
Easy
Hard
Easy
Hard
Model
Train
Transfer
Train
Transfer
Train
Transfer
Train
Transfer
Random
-3.43
-3.43
-3.43
-3.43
-78.48
-78.48
-78.48
-78.48
Linear SF
-1.28
-1.29
-1.58
-1.6
-6.29
-6.25
-10.25
-10.15
Second-Order SF
-1.3
-1.45
-1.33
-1.31
-6.31
-6.33
-6.87
-7.29
Figure 3: Performance in the Axes and Reacher environments during training and transfer over the last 1000 steps. Both variants are able solve the easy environment with essentially equal performance. In the hard environment the second-order model has higher performance than the linear model, and is much closer to the easy variants score.
2 (right), the agent must navigate between rooms trying collect an item from of the rooms after which the episode ends. We the same map as [Kulkarni et al. 20166) but with a slightly different environmental setup, which is detailed in the Appendix. The agent receives a small negative reward per step and large reward when it picks up the goal item.
Doom: In this environment, shown in Figure
4.2 EXPERIMENTAL SETUP
The agent is trained on a randomly sampled training tasks; then, during testing, we change to a unseen task. A single policy is trained over all tasks. During transfer to unseen tasks, the model re-learns only the reward parameters, with the remainder of the model frozen (Zhang et al.) 2017] Kulkarni et al. 2016). The newly learned policy will vary from the original but able to exploit previously learned environment knowledge for the new tasks. The training and testing methodology identical to those used in previous studies (Barreto al. 2017.. A uniform random action baseline was considered in all environments to act as floor.
Within the Doom environment, we are simply interested in the performance between the second-order model and that of the linear baseline. Therefore, we do not use a train and test split in this environment but instead rely on the randomness of the item and agent spawn locations.
Our method is compared against the successor feature framework with linear reward model; the architecture is similar to that of Kulkarni et al. (2016). This baseline is identical in all ways to the second-order model except for the exclusion of terms containing A, specifically Equations 2 and which can be obtained by setting 3 = More exactly, the linear baseline and our proposed model use the exact same code with only the 3 and hyperparameters adjusted.
In all environments we adjust the dimension of the latent representation of de. to ensure that the second-order variant has less than equal parameters to that of the linear variant. This ensure that performance from the second-order variant is not from the additional parameters alone. both the Axes and Reacher environments, the linear variant uses a hidden dimensionality of 24 while the second-order variant has In the Axes, the linear model has 5, 704 parameters and the second-order model has 2, 100 parameters. While, within the Reacher environment this led to 16, 710 and 7, 318 total parameters for the linear and second-order variants, respectively.
In the case of the Doom environment, to again ensure that the performance of the second-ordes variant is not due to additional parameters, the linear model uses a large three layer convolutional encoder and the second-order model uses much smaller of 343M and 276M parameters for the linear and second-ordes models, respectively. Each model's s mean performance is reported on all plots the average over three runs with varied seeds. Each plot includes the standard deviation over all runs as a shaded area. Additional details can be found in
layer encoder. This leads to a total
Appendix D
4.3 ENVIRONMENT PERFORMANCE
The primary point of comparison between the proposed method and the original formulation of the successor feature framework, which can be recovered exactly by setting = 0 in Equations 2 and of the proposed model. The result of these experiments shown in Table 3 for both Axes and
7

---

Under review as a conference paper at ICLR 2022
Reacher. In both environments, variants. This is expected as the reward, in the case of the linear variant, can be exactly recovered by using 1-hot encoded reward vector and even with weaker state representation the linear reward model has enough capacity to compensate. However, we see this is not true in the hard task, as the linear variant has weaker performance when compared to the second-order model.
that the easy task is solved by both the linear and second-order
Clearly, the second-orde variant provides extra representational capacity the reward model such that it can compensate on its own for non-ideal state representation which is shown by the greater performance on the hard tasks. The linear variant, not able appropriately model the environments reward structure as the
Doom
50
Ineer
second-order
40
Avg. Episade Reward
30
reward non-linear function of the state; in
this case, the agent's 's coordinates and the current goal location.
20
From the result within the Doom environment, shown in Figure 4 we can see that our proposed model is clearly able to out perform the base- line successor feature implementation. Not only does proposed method is near the ceiling performance of the environment, it converges rapidly. In comparison, the baseline method fails to achieve similar performance and also converges much slower rate. From the dif- ference in learning curves we can conclude that the extra representational power of the reward model has drastic impact on performance. As both variants have roughly equal parameters, with the linear variant containing more, it can be conclude that the extra representational power of better use in the reward component the model instead of the encoder.
20k
100K
Step
Figure 4: Performance of the baseline linear vari- ant and our proposed second-order model in the Doom environment.
4.4 TASK TRANSFER
An important property of the successor feature framework is the ability to adapt rapidly to tasks within the same environment. Adaption, or transfer, is accomplished by freezing the model's state- dependent components, such as v/, and quickly learning just the reward parameters w. Specifically, we minimize (s;; o. A) [r: o, + 36, A of Equation S with respect to the parameters and A only. Similarly, after training the models convergence, within the environment. During transfer, we scaled the learning rate by fixed factor, randomly initialize the learned weights, and re-decay the value so the method has a chance explore. Full details are provided in Appendix The results during transfer on both environments are provided in Table
change the task distribution
Als. gya.1
Ns.e 119.31
Als. 2y1.
Als. 3ya.
(a) Half-random Axes.
(b) Visualization Of A.
Figure 5: Visualization of Lambda Function on half-random Axes. The half-random variant of Axes. b) The learned expected future correlation of one feature with itself along A's diagonal is visualized over the entire state space. The first column is the max value of A over the actions. The remaining columns, from left to right, correspond to each action: left, up, right, and down. Red and blue correspond to maximal and minimal values.
8

---

Under review as a conference paper at ICLR 2022
4.5 MODELLING ENVIRONMENTAL STOCHASTICITY
This section examines the A function to determine whether it can capture stochasticity in the environ- ment in the Axes environment. We use a modified version, referred to half-random and shown in Figure 5al which is identical in all aspects to the base version except for a location-based conditional that affects the agent's actions. If the agent is within the positive z quadrant of the map. >> then actions are randomly perturbed with fixed probability. Otherwise, they are fully deterministic. After training to convergence on the half-random variant, we examine the A function that the model has learned. Because the A function is modelling the auto-correlation matrix, the future expected correlation of each feature with itself is found by looking along the diagonal. Figure 5b shows the result of plotting a diagonal value, in this case, feature (1, 1), of the A-matrix over the entire state space. We can see that one of the diagonal components of A did indeed learn to approximately model the conditional random field within the environment.
4.6 GUIDED EXPLORATION WITH A
Here we examine whether it is possible to use the A-function for guided exploration during transfer within the Axes and Reacher environments.
The Successor Features, given in Equation 2. can be interpreted as predicting the future expected path taken by the policy π an environment. Under this interpretation, & can be seen as capturing the expected features of the states and A the expected variance between state features along these pathways. Adding noise to the A component would then perturb around the expected path. Therefore, instead of using é-greedy exploration, it is possible to add noise to A during transfer, such that Á(s.a) a) A(s, a) + eA(s, a), where is sampled from some distribution. During learning, the variance of the sampling distribution, controlled by a, can be annealed to some final value. The actions are then sampled from the model at time t as: O₂ = argmaz.. { [15(5e, a²) aT) o + tr(AA(s;,a")) a²)]
From Figure 6 we see that using A for guided exploration is indeed possible alternative to e-greedy. Additionally, we found that using scalar value sampled from uniform noise, that is E ~ U (aa, a) where [-a, oi., provides the best performance.
Explonat on Modes on hoes
Exploration Modes Reacher
Tocal drg Boooor
E-gredd
unifom
fruntar
0.0
0.20
0.30
0.00
10
Steps 10
(a) Axes Environment.
(b) Reacher Environment.
Figure 6: Guided Exploration: The A component of the proposed model is used to guide exploration during transfer. By using A the agent explores in directions with large variance in the state space.
5 CONCLUSION & FUTURE WORK
In this paper, have derived novel formulation of successor features with non-linear reward. We have shown that the agent can perform well with a second-order reward structure, providing extra flexibility to the reward model. Further, we have shown the utility of the A term that appears in the derivation of the new state-action function. Experimentally, we have shown that the A term is able to capture the stochastic nature of an environment and can be used for directed exploration during transfer. In future work, we aim to explore the A function deeply and if a formulation exists that learns the future expected variance.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath, A brief survey of deep reinforcement learning. arXiv preprint arXiv:/708.055600 2017.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, pp. 6055-4065 2017.
Richard Bellman. Dynamic programming. Science, 153(3731):34-37, 1966.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613-62)) 1993.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is you need: Learning skills without reward function. arXiv preprint arXiv: 2018.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv: 1601. 1606.018677 01447, 2016.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley and Volodymyr Mnih, Fast task inference with variational intrinsic successor features. arXiv preprint 0rXiv:1906.05000 2019.
Tejas Kulkarni, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv:1606.0233 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-44, 2015.
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv: 1089, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-333, 2015.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam. Marc Lanctot, et Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Csaba Szepesvári. Algorithms for reinforcement learning. 2009.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. IEEE, 2012.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):550-354 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv: 19910.0097) 10897, 2019.
Zhou Yu. Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 1821-1830. 2017.
10

---

Under review as a conference paper at ICLR 2022
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. In 2017 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp- 2371-2378. IEEE, 2017.

---

Under review as a conference paper at ICLR 2022
APPENDIX
A NON-LINEAR REWARD DERIVATION
Here we provide the derivation for the non-lineat reward in the successor framework. First, we start by assuming the reward rt has the following form:
Tt o. o 0 A
(10)
where főr, oo E R=x1, and A E R and both o and are learnable parameters. Following from the definition of the state-action value function Q(s, a), the adjusted reward function can be substituted to yield:
(=(s.a) = E"Ire + 7244 t... ............ s, = a|
(11)
1 o Ad T0 70 +...
|S 15:= 8, A: a]
(12)
Dropping the conditional portion of the expectation for brevity, linearity of expectation can be used to split apart the terms containing A and o. Then is pulled out from the first term:
= E 2 + 70 + +...| + ET 1 AQi+ +
+...]
(13)
E [0 + 700+2 + 1 ET [+ Adet1 + 70 Ad +...]
(14)
By recognizing the first expectation term as the successor features (bss a). Equation rewritten as
14 can be
= b (s, ET
O ET A0t+1 + T 20++2A01+2 +...] (15)
Because o AØ results scalar, the trace function tr(-) can be used inside the right-hand term:
(s, ET [t (0 A + tr(10T+2 + +...] (16)
By exploiting the fact that tr(AB) = tr(BA), the terms inside the trace function can be swapped to yield:
2 (s, a) T o T tr A ( + tr tr(7AA42200H) + (17)
Because both (r(-) and A are linear, they can be pulled out of the expectation, giving:
-6 (s, a T + tr EE AQ4T10T + A + +...)) (18)
v a) T + tr(AET 00 + 701+20
T + + +...)) (19)
Finally, the remaining expectation can be expressed as a function:
a) = x (8, (((,)))= (tr(AAA
(20)
E {0, 1) is hyperparameter that controls the inclusion of the non-linear component. We define 20 and A as:
=E TT (s, a) = ET [6++1 =Ea( + yb
( T (8t+1))) = =, A₁ = a]
(21)
AT क (8, a) =
EE =ET ETTOLT1TET + 7 T ( (8411)) = 8, A, = a
(22)
12

---

Under review as a conference paper at ICLR 2022
B A MATRIX FACTORIZATION
Similar to A, as the dimensionality of increases, so does the number of parameters needed for modelling matrix E R=** Therefore, in the interest of reducing the number of parameters we use factorization that splits the matrix A E R into two parts with a smaller inner dimension f. A L S-R T where {L, R E R=x/_ By factoring the matrix in this way, we require 2 x parameters instead of X If we use values for smaller than 1.2 we reduce the number of parameters required by matrix A. A similar factorization was suggested in the context of visual question answering (Yu et al., [0177] Fukui et al. (016). The factorization of A was primarily done to reduce the total number parameters in our model.
C ENVIRONMENTS
C.I AXES
Within this environment eight separate goal locations exist split between train and test distributions. We used modified version of the robotic model provided by Metaworld (Yu et al. (019). An episode ends when either the agent reaches the goal or more than 25 steps have elapsed. The agent's starting location is randomly sampled from grid x step units, centered at (0, 0).
With this state space the agent must learn reward function that can approximate the distance between itself and the goal location, d(a, V(b2 a²)² + (by ay)², a non-linear function.
C.2 REACHER
Similarly to Axes, the environment has predefined tasks split between training and test distributions, with the eight goals shown in Figure 2b as green and red balls, respectively. In the Reacher environ- ment, an episode ends when 150 steps have elapsed or the agent within 7em of the goal. The agent receives reward equal the negative distance between the end-effector and the current target goal at each step.
We discretize the actions such that the agent has nine discrete actions that control the arms movements. Because the models can be used only with discrete actions, it was necessary to transform the environmental actions. Therefore, the four-dimensional continuous action space A was discretized using two values per dimension: the maximum positive and maximum negative torque for each actuator, An all-zero option was included that applies zero torque along all actuators, resulting in total of nine discrete actions.
C.3 DOOM
In the Doom environment the agent must traverse between four rooms looking for a goal. The rooms separated by doors that the agent must manually open. At each step the agent receives small negative reward of -0.01 and upon finding the goal receives +50. The agent perceives the state and the 4 stacked frames of RGB frames of shape (3, 84, 84), corresponding and height. The agent has 4 actions available: forward, rotate left, rotate right, and activate door. We action repeat of 5 across all actions, which differs from the original implementation that used selective action repeat.
color channels, width,
D EXPERIMENTS
Within both the Axes and Reacher environment, c-greedy annealed from 1.0 to final over the first 250k steps.
In the Axes and Reacher environments the encoder and decoder each respectively contain and two hidden layers with an embedding size equal to the double the raw state size. Initially, on the Axes environment, the models all used the raw features with encoder such that oi ((s)). We
13

---

Under review as a conference paper at ICLR 2022
found this led to worse performance for the linear model it now had chance to learn suitable encoding of the features for reward prediction. Both y and increase the hidden dimension & by fixed factor before output. This factor Sfactor depends the environment. All environments used discount factor of y = 0.99, À = 0.1, and updated the parameters every 25k steps.
D.1 AXES
An embedding size of was used for the second-order model and 24 for the linear model in the Axes environment. The final value used in e-greedy was 0.1 with learning rate of a 2.5e 4 was used. The encoder and decoder in this environment were a single fully connected layer with the same embedding size 2.
D.2 REACHER
An embedding size of = = 8 was used for the second-order model and = = 24 for the linear model in the Reacher environment. The final value used e-greedy was 0.05 with learning rate of 5e was used. The encoder and decoder in this environment were a single fully connected layer with the same embedding size z.
D.3 DOOM
An embedding size of 512 used for the linear model and 256 for the second-order model. encoder network with 3 layers of convolutional layers with hyperparameters (3c, 32o, 8s), (32c, 640, 4s), and (64c, 64o, 3s) where is the number of incoming "channels", is the number of filters, and s the filter size. The corresponding decoder had 5 layers of transposed convolutional layers: (64c, 2560, 4s), (256c, 128o, 4s), (128c, 64o, 4s), (64c, 32o, 4s), and (32c, 3o, 38).
While the second-order variant used 2 layer convolutional encoder with: (3c, 160, 8s), (16c, 32o, 48). And corresponding convolutional decoder with 2 layers: (32c, 160, 4s) and (16c, 30, 8s).
E
TRANSFER
During transfer reinitialize the reward specific parameters A and constant values of 0.1. All other model parameters are held frozen and do not change. The learning rate is increased by factor of 2x the Axes and Reacher environments. We anneal the exploration parameter from to 0.1 in Axes and to 0,05 on the Reacher environments over the first 200k steps.
14


Reviews:
[{"review_id": "ofJ2eOL6Z3B", "reviewer": "Reviewer_j97Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ the motivation of the paper is clear to me and the proposed method is straightforward and seems to be technically sound.  \n+ the algorithm was evaluated on a diverse set of environments, with different level of difficulties, including both vectorized state inputs and imagery state inputs.  \n+ the authors clearly stated and compared the number of parameters used in the proposed algorithm and in the baseline, which is helpful to better gauge the effectiveness of the algorithm. In addition, adjustments were made to reduce the number of total parameters of the proposed neural architecture.  \n\nConcerns:\n\n1.[Clarity] The clarity of the paper needs to be further improved in my opinion. Please see the list of items below for details. \n\n+ 1.a. Section 3.1 derives equations (3) and (4) following/conditioning a given policy $\\pi$. However, Section 3.2 directly moves to present an algorithm learning $\\psi$ and $\\Lambda$ in an off-policy Q-learning manner (equations (6) and (7)). It is not clear to me what kind of successor features would be learned with (6) and (7) when the behavior policy keeps changing during training. Could you elaborate a bit more on this? It would be better if the authors could add some explanations or insights in the paper when transitioning from 3.1 (on-policy) to 3.2 (off-policy).  \n\n+ 1.b. The analysis of the extra term $\\Lambda$ is not sufficient. Related to 1.a, I am not sure how to interpret $\\Lambda$ as “the expected variance between state features along these pathways” (described in Section 4.6). Therefore, the exploration strategy in Section 4.6 by randomly perturbing $\\Lambda$ is not convincing to me and I am not sure how the exploration is being “guided”. The performance also looks similar to random exploration $\\epsilon$-greedy. It could be helpful to compare empirically the trajectories traversed by the agent when using perturbed $\\Lambda$ against the trajectories by $\\epsilon$-greedy.    \n\n+ 1.c. In Section 2.2, first paragraph, it defines reward as a function of state-action pair: $r(s,a) = \\phi(s)^T w$, but the RHS and the network architecture Figure 1(b) both do not involve any actions. Please check accordingly in the next version. I believe the paper is modelling reward as a function of state $r(s)$, why not modelling reward as $r(s, a) = \\phi(s, a)^T w$ (where a separate feature vector is learned per action)?  \n\n2.[Significance + Baselines] Another concern is on the significance of the proposed method when compared with previous approaches in the literature. It would be very beneficial if the authors can add additional baselines to better demonstrate the improvements from the proposed 2nd-order SF framework.   \n\n+ 2.a. The paper improves the reward model to better learn and capture the dynamics of the environment via encoder-decoder. Another perspective/direction is improving the “encoder” itself to capture more meaningful state features without modifying the reward model. This may lead to having a more complex encoder module. Existing work [1] showed that adding auxiliary tasks, e.g., predicting the next state, is effective for representation learning, and the architecture in [1] performed well for Atari games. I suggest the authors try adding auxiliary tasks in the linear SF baseline and compare with the 2nd-order SF method. I also encourage the authors to compare against the algorithm proposed in [1] although the focus is on improving exploration with SF. For transfer, I believe [1] used a single FC layer to map features $\\phi$ to Q values. It suffices to only retrain the last FC layer when doing task transfer. Hence, the authors could compare transfer as well against [1].           \n\n3.[Experiments] \n\n+ 3.a. Why not evaluating the task transfer on the Doom environment? It would be interesting to see transfer results on Doom.  \n\n+ 3.b. The visualization of matrix $\\Lambda$ in Figure 5 is not very informative to me, because only one element in the matrix (1, 1) is being visualized. In Figure 5(b), does that mean element (1, 1) takes larger values in frequently-visited states? E.g., the region where $x > 0$. It may be helpful if the authors can relate matrix $\\Lambda$ or its norm to state visitation counts (previous work [1] has done this before for SF). \n\n+ 3.c. Section 4.6 guided exploration is not clear to me (see 1.b for more comments). The performance looks similar to $\\epsilon$-greedy, and it would be interesting to see some results as well on the Doom environment. \n\n\n\nMinor Comments: \n-\tEquation 8: should be $[r_t – A – B]^2$, two minus signs. \n-\tSection 4.5: feature (1, 1) instead of feature 1? \n\n\n\n[1] Machado, Marlos C., Marc G. Bellemare, and Michael Bowling. \"Count-based exploration with the successor representation.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020. \n\n", "summary_of_the_paper": "The paper provides a new idea for deep reinforcement learning with successor features (SF). The existing SF framework assumes a linear reward model which requires learning predictive and informative state representations to capture reward signals. To reduce the burden of the encoder for learning meaningful state features, this paper extends the linear framework by proposing an additional quadratic term in the reward model, leading to an SF framework with 2nd-order rewards. Since the representation power for modeling rewards is greatly improved, the encoder-decoder can focus more on capturing the dynamics of the environment. Empirically, the 2nd-order SF algorithm outperformed the linear SF baseline, especially when rewards are defined by applying nonlinear transformation of state observations. \n", "summary_of_the_review": "The paper presents an extension to deep successor reinforcement learning [2] which adds an extra quadratic term in the reward model, resulting in a new 2nd-order SF framework. The idea is interesting and the motivation is clear. My major concerns are about (1) the clarity of the paper, and (2) the significance of the proposed method, and I suggest the authors include additional baselines and experiments in the rebuttal period (see concerns above for details). Therefore, I don’t think the paper can be accepted with the current version and vote for weak reject.  \n\n\n[2] Kulkarni, Tejas D., et al. \"Deep successor reinforcement learning.\" arXiv preprint arXiv:1606.02396 (2016).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "eVsv2JfDPEz", "reviewer": "Reviewer_yWbZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the authors clearly wrote the paper and the derivation is relatively clear and appears to be correct. \n\nOne major concern I have is that even if we assume reward is a linear function of features, the resulting value function is not hindered and in fact the expressiveness of value function is not limited by the linearity, as features can be an expressive and arbitrarily non-linear function of observations. \n\nOf course, for a fixed trained SF and its value function, there are rewards that are non-linear function of SF, and then the trained model will find it difficult to generalize to such out-of-distribution rewards. But this seems not the focus of this work. \n\nThe experiments demonstrate that second-order SF outperforms linear SF on Reacher(Hard) and slighter worse than linear SF on Axes(Easy and Hard) and Reacher(Easy). In hard variants, the states encode less information than easy variants about the agent’s status and goal. \nThe authors also conducted experiments on pixel based Doom environment and show second-order SF is able to linear SF. \n\nHowever, I am wondering if feature learning of the linear SF baseline is well tuned and have a fair comparison. The appendix mention that linear SF uses embedding size 512 but second-order SF uses embedding size 256. \nThere should also have ablation study of the reconstruction auxiliary loss.\n\nThe exploration idea is interesting but seems incomplete. The authors wrote the variance term can be seen as “the expected variance between state features along these pathways.” is interesting, but why is adding noise to the variance term helps exploration? To support the claim, the paper would benefit from comparison with related methods on “state features” and “noisy input” e.g. random network distillation, active pretraining with successor features, and noisy network for exploration. ", "summary_of_the_paper": "The authors extend the successor features linear formulation to model a possibly non-linear relationship between features and rewards. \n\nThe new method is an instantiation of the successor feature that adds adds features that are dot products of other features, this adds quadratic nonlinearity which the authors argue is helpful to handle more general reward functions.", "summary_of_the_review": "In summary, the paper needs more clarifications and convincing experiments to support their main claims, especially around why is second order SF is better than linear SF.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SpzE0mJc-AO", "reviewer": "Reviewer_gjLL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed approach requires extra effort and parameters.\nNow a matrix needs to be evaluated per action.\nAlthough the authors present favourable results with a smaller number of\nparameters, in comparison with the linear model, this seems\nmisleading. It is relatively easy to increase the number of\nparameters in the linear model and still have the same performance. \n\nThe authors should include a comparison where the same architecture is\nused for both the linear and second order functions, and compare\nboth approaches in terms of training times, number of parameters, and\nperformance. \n\nIt is not clear why the \\beta parameter was introduced and what values\nof \\beta were used in the experiments.\n\nAnother important parameter is the value of zeta. Was it determined by\ntrial and error? how sensitive is the system to this value?\n\nIt is not clear what the authors are trying to prove with Fig. 5,\nplease clarify. \n\nTypos:\n- An extra parenthesis is included in formulation of the Q value\n  function in Sec. 2.2:\n  \\phi(s_{t+1});\\theta_{\\phi})^Tw => \\phi(s_{t+1}_{\\phi})^Tw\n- In Eq 8, I suppose the minus sign includes both terms: (r - (\\phi\n... + \\beta ...))^2, the same comment applies to Sec. 4.4\n- with the only the => with only the\n- proposed method near the ceiling => proposed method is near the ceiling\n", "summary_of_the_paper": "Successor features decomposes the policy into two components: one\nmodelling the environmental dynamics and the other, the rewards.\nThe paper describes an extension to the successor features framework\nwhere the rewards are modelled using a second order function.\nThe extension provides a more robust reward component.\nUnder this formulation a second term appears that models the expected\nauto-correlation matrix of the state features and can be used for\nguided exploration during transfer.\nThe authors provide experimental results is three domains with\ncomparison with the linear approach.\n", "summary_of_the_review": "The paper describes an extension to the successor feature framework.\nPros:\n- a new formulation with provides a more robust reward component\n- the approach can be used for exploration during transfer\n- a clear improvement over the linear approach with less parameters\nCons:\n- it can be seen as an incremental work\n- the experiments do not reflect, under the same conditions, the extra\neffort needed with the second-order rewards", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1visU3qPGY1", "reviewer": "Reviewer_xH2J", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- Prior work / the explanation on how successor feature works is well done. \n- The paper is written well and the math is easy to follow. The authors intuitions and assumptions are easy to understand and believable. Replacing the linear model with a quadratic one is intuitive and a natural next step for these kinds of problems.\n- The algorithm is evaluated on many different kinds of RL environments which is appreciated and highlights its strengths and weaknesses.\n\nWeaknesses \n- The secondary and tertiary contributions of the paper (modelling the auto-correlation matrix of state features and using the second term for guided exploration) seems underdeveloped / incomplete. In Figure 6, for example, neither method significantly outperforms the other and I am not convince that to use their method over epsilon sampling. \n- Standard RL model-free baselines would be interesting to see on the experimental setup i.e. where dynamics and reward modelling are not  decoupled. I would expect that model-free methods would outperform the algorithm on standard tasks but would under-perform when it came to transfer learning.\n", "summary_of_the_paper": "The authors take the successor feature framework that separately models enivronmental dynamics and reward to use a second order reward learning formulation over the standard linear model. This allows a non-linear relationship to more easily form between features and rewards lessening the burden of a the feature encoder to learn \"good\" features for the RL task at hand. The authors showcase experiments where the designed new quadratic reward modeler performs better than standard RL systems ", "summary_of_the_review": "Paper is well written and a natural extension to the linear formulation of successor features. Re-implementing the model based on the writing itself should be possible and I thank the authors for clear text. \nThe latter half of the paper is less well developed coming across as a little undercomplete. I wonder if the paper would benefit more from more significant transfer learning experiments uitlizing the second order model while relegating the auto-correlation matrix / alternative to epsilon exploration to the appendix / future work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submitted paper considers a form of second-order extension of successor features building on a second-order representation of the reward function in terms of state-features. The authors demonstrate that this approach can be useful for transfer learning and also show an application to exploration.
All reviewers gave borderline recommendations (2x weak accept, 2x weak reject). While most reviewers agree that the proposed approach can be sensible and that the paper is well written, there are concerns that experimental results do not fully support all claims and additional experiments are required to clearly demonstrate advantages over existing baselines. Also the proposed approach for exploration is rather incomplete and not well studied. The raised concerns were not fully refuted by the authors during the discussion period but rather made some reviewers more concerned about full validty of all claims. Thus, while I think the paper has potential and can be turned into a good paper, I am recommending rejection of the paper in its current form. I would like to encourage to authors to carefully address the reviewers' concerns in future versions of the paper.
Decision: Reject