Non-Linear Rewards For Successor Features
Authors: Norman L Tasfi
Miriam Capretz
Venue:  (2021)

Abstract: Reinforcement Learning algorithms have reached new heights in performance, often overtaking humans on several challenging tasks such as Atari and Go. However, the resulting models learn fragile policies that are unable to transfer between tasks without full retraining. Successor features aim to improve this situation by decomposing the policy into two components: one capturing environmental dynamics and the other modelling reward. Under this framework, transfer between related tasks requires only training the reward component. However, successor features builds upon the limiting assumption that the current reward can be predicted from a linear combination of state features. This paper proposes a novel improvement to the successor feature framework, where we instead assume that the reward function is a non-linear function of the state features, thereby increasing its representational power. After derivation of the new state-action value function, the decomposition includes a second term that learns the auto-correlation matrix between state features. Experimentally, we show this term explicitly models the environment's stochasticity and can also be used in place of $\epsilon$-greedy exploration methods during transfer. The performance of the proposed improvements to the successor feature framework is validated empirically on navigation tasks and control of a simulated robotic arm.

Under review as a conference paper at ICLR 2021
NON-LINEAR REWARDS FOR SUCCESSOR FEATURES
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement Learning algorithms have reached new heights in performance, of- ten overtaking humans on several challenging tasks such as Atari and Go. However, the resulting models typically learn fragile policies that are unable to transfer be- tween tasks without full retraining. Seattrrs aim to improve this situation by decomposing the policy into two components: one capturing environmental dynamics and the other modelling reward. Under this framework, transfer between related tasks requires only training the reward component. However, successor features builds upon the assumption that the current reward can be predicted from linear combination of state features, assumption with no guarantee. This paper proposes a novel improvement to the successor feature framework, where we instead assume that the reward function is non-linear function of the state features, thereby increasing its representational power. After derivation of the new acttinacioonn value function, the decomposition includes a second term that learns the auto-cor correelleeltion matrix between state features. Experimentally, we show this term explicitly models the environment's 's stochasticity and can also used place of e-greedy exploration methods during transfer. The performance of the proposed improvements to the successor feature framework validated empirically navigation tasks and control of a simulated robotic arm.
Recently, Reinforcement Learning (RL) algorithms have achieved superhuman performance in several challenging domains, such as Atari (Mnih et al., 2015), Go (Silver et al. 2016). and Starcraft II (Vinyals al. 2019). The main driver of these successes has been the use of deep neural networks, which are a class of powerful non-linear function approximators, with RL algorithms (LeCun al. 2015). However, this class of Deep Reinforcement Learning (Deep RL) algorithms require immense amounts of data within an environment, often ranging from tens to hundreds of millions of samples (Arulkumaran et al.) 2017]. Furthermore, commonly used algorithms often have difficulty in transferring a learned policy between related tasks, such as where the environmental dynamics remain constant, but the goal changes. In this case, the model must either be retrained completely or fine-tuned on the new task, in both cases requiring millions of additional samples. If the state dynamics are constant, but the reward structure varies between tasks, is wasteful to retrain the entire model.
more pragmatic approach would be to decompose the RL agent's 's policy such that separate functions can learn the state dynamics and the reward structure; doing so enables reuse of the dynamics model and only requires learning the reward component. Successor features (Dayan) 1993) do precisely this; model-free policy's action-value function is expressed as the dot product between vector of expected discounted future state occupancies, the successor features, and another vector representing the immediate reward in each of those successor states. The factorization follows from the assumption that reward can be predicted as the dot product between state representation vector and learned reward vector. Therefore, transfer to new task requires relearning only the reward parameters instead of the entire model and amounts to the supervised learning problem of predicting the current state's immediate reward.
This factorization can be limiting because it is assumed that the reward is a linear function of the current state, which might not always be the case as the encoded features might not capture the required quantity for accurate reward modelling (Eysenbach al. 2018 Hansen et al. 2019). Therefore, this paper introduces a new form for the reward function: non-linear with respect to the current state. We assume that the learned features are not optimal and the reward cannot be predicted directly from the raw features, which is not a strong assumption. This form increases the reward function's 's representational power and makes it possible incorporate the current state into reward

---

Under review as a conference paper at ICLR 2021
estimation; lessening the burden the encoder components. Under the new reward formulation, a secondary term emerges, which learns the future expected auto-correlation matrix of the state features. This new secondary term, referred to A, can be exploited as possible avenue for directed exploration. Exploring the environment using A allows us to exploit and reuse learned environmental knowledge instead of relying on purely random approach for exploration, such as e-greedy.
Following this, the contributions this research are as follows:
A novel formulation of successor features that uses non-linear reward function. This formulation increases the representational power of the reward function.
Under the new reward formulation, a second term appears that models the future expected auto-correlation matrix of the state features.
We provide preliminary results that show the second term can be used for guided exploration during transfer instead of relying on e-greedy exploration.
After the introduction of relevant background material in Section we introduce the successor feature framework with non-linear reward function in Section 2 Section provvddes experimental support and provides an analysis of the new term in the decomposition. The paper concludes with final discussion and possible avenues for future work Section
BACKGROUND
1.1 REINFORCEMENT LEARNING
Consider the interaction between an agent and an environment modelled by a Markov decision process (MDP) (Puterman 2014). An MDP defined set states S, set of actions A, reward function R: S + I a discount factor E [0, IJ. and transition function T S x A + [0, 1j. The transition function gives the next-state distribution upon taking action a in state and often referred to as the dynamics of the MDP.
The objective of the agent in RL is to find a policy æ, a mapping from states to actions, which maximizes the expected discounted sum of rewards within the environment. One solution to this problem to rely on learning a value function, where the action-value function of a policy æ is defined as:
o Σ ((((s)) = A, =
where ET [...] denotes the expected value when following the policy æ. The policy is learned using an alternating process of policy evaluation, given the action-valuc of particular policy and policy improvement, which derives a new policy that is greedy with respect to (*(ssa) (Puterman 2014).
1.2 SUCCESSOR FEATURES
Successor Features (SF) offer a decomposition the Q-value function and have been mentioned under various names and interpretations (Dayan) 1993 Kulkarni et al. [016 Barreto et al. 2017] Machado al. 2017). This decomposition follows from the assumption that the reward function can be approximately represented
linear combination of learned features ((s; 0x) extracted by
neural network with parameters 0g and reward weight vector w. As such, the expected one-step reward can be computed as: r(s,a) a) o(s; 0&) Following from this, the function can be rewritten as:
Q( R ET
rt+1 + 7742 .... |S, -.==88.
E
0(s(+1i00)) 1o) To w 0( 00) Tow w + ......1... A,
Q(s,a) + (s)
where (bss, a) are referred to as the successor features under policy л. The 1hh component of (i(s, a) provides the expected discounted of of when following policy starting from state and
2

---

Under review as a conference paper at ICLR 2021
action It is assumed that the features ((s; 0g) representative of the state s, such that ()).) can be turned into a function 1 (o(s;; ae). For brevity, ((5 06) is referred to simply as Φe and (s, a) a)),a))
The decomposition neatly separates the Q-function into two learning problems, for o and we estimating the features under the current policy dynamics, and estimating the reward given state. Because the decomposition still has the same form as the Q-function, the successor features are computed using Bellman equation update in which the reward function is replaced by o,;
(o,, ax) = de + 7B 16 (6++1, (01+1,a++1)
such that approximate successor features can be learned using an RL method, such as Q-Learning (Szepesvári, 2009).
Following from this, the approximation of the reward vector becomes supervised learning problem. Often, this weight is learned using ordinary least squares from the sampled environmental data. One benefit of having decoupled representation is that only the relevant function must be relearned when either the dynamics or the reward changes. Therefore, the task changes, but the environmental dynamics remain constant, only the reward vector parameters must be relearned, which are minimal compared to the total number of parameters in the full model.
MODEL, ARCHITECTURE, AND TRAINING
The Successor Feature framework has several limitations, primarily stemming from the assumptions around its derivation, such as constant environmental structure between tasks or that the reward can be linearly predicted from state features. Work towards solving the former has been developed by Zhang et al. (2017) whereby they learn a linear mapping between task state features. The latter assumption, whereby the reward assumed to a linear mapping state features, is not guaranteed and, as we show, the Successor Feature framework fails in such cases. Therefore, the method presented in this section aims to provide a stronger guarantee of the framework's performance such cases by developing more robust reward component.
This section discusses our change to the successor feature framework, which adjusts reward function, from a linear function, to non-linear function. First, discussion of the new decomposition is given with the full derivation provided in Appendix A Then experimental support for this change will be presented and analyzed examine what the new term in the decomposition learns.
2.1 NON-LINEAR REWARD FUNCTION
The successor feature framework builds upon the assumption that the current reward rt can be represented by the linear combination of the current state representation di E R and a learned reward vector w E R2, such that r, = 0, w. This form limiting because there no guarantee that the reward will be a linear combination of the state features that the required state features can be learned by the encoder (Eysenbach et al. 2018: Hansen et al. 2019). In practice the optimal state features are often which in itself is not strong assumption. To increase the flexibility of this reward model, let us consider the following form:
learned; therefore, build on the basis that the state features are sub-optimal,
Pt -== o Aø:
(I)
where ((),, 0) E R and A E R Both and A are learnable parameters modelling the reward structure of the environment. Equation[]sshi [hsoos that the formulation introduces a non-linear transformation with respect Φ. Comparing this with the original formulation, is evident that this is equivalent to setting w = + As. The state-action value function Q(s, a), under this new reward structure, can be derived to yield:
Qa (st ( T Str(AAT (A,AAssss,) (st,a))
where 3 E {0, 1} controls the inclusion of A and tr is the trace operator. can now be shown that and A satisfy the Bellman equation (Bellman 1966):
3/ (st, a) E. (d+11 (1 ( )){ S. A,
(3)

---

Under review as a conference paper at ICLR 2021
St
DECOUL
St
ENCODER
dof
Tt
St
ENODDER
Φt
A(or, ai)
W
v(dr, at)
(a) Model Overview
(b) Reward Prediction
Figure 1: Model Overview a) The encoder transforms the raw state into an internal state representa- tion dr. The state representation d, used by the decoder, A(., at), and eb(, ar). The decoder tries to reconstruct the raw imput 9, from the state representation dr. A and d produce with the former predicting matrices and the latter predicting vectors. Also, both and e use two hidden layers process d, before their final layers. b) Reward prediction by dotting the current state Φs. produced by the encoder, and reward weight w.
output per action,
A (s,,, a) = (T + (
A, = al
where for ei and A, é and oó respectively play the role of rewards. In addition v. it is now necessary to model A, which outputs an R*** matrix per action. The quantity Φ.Ø, can be interpreted auto-correlation matrix of the state features. We see that this form allows the A term to model some form of future expected stochasticity of the environment. For example, the diagonal of A will model second order moment capturing each feature's change with respect itself cf We provide analysis and further discussion of A in Section 3.5
2.2 MODEL STRUCTURE AND TRAINING
The proposed model, shown in Figure la uses an encoder produce a state embedding di, consumed by downstream modelling tasks. Figure shows how the current reward r; is predicted using w, with w/ AØ. and current state representation Φy; this process is defined in Equation [ Similar to previous work with successor features, the structure includes pathways for an encode-decode task and successor feature prediction vi (Machado et al., 2017 Kulkarni et al.) 2016 /Zang et al. 2017). The decoder network ensures that the features learned by the encoder, which produces é, contain useful information for prediction. Furthermore, only the gradients from the state-dependent and reward prediction tasks modify the encoder parameters, and therefore d. An additional branch is added, by way of the non- non-linear reward function, to model the quantity A A(s, a). This branch's output matrix, which differs from the vector predicting branches vó and Φ.
The encode-decode task is trained by minimizing the mean squared difference between the input st and the decoder's 's reconstructed version di from Φi:
(s;; 00, 00) [st g( )²)
(5)
where & is the output of the encoder with parameters 0 and g(: 0²) produces the output of the
decoder with parameters g As mentioned previously, we train db and A, parameterized with 0 and @ respectively, using the Bellman equations to minimize the following losses:
(st, a 08) B[(o, + T0( a'; (hs ari )²)
(6)
CA ay; 0*) E 7A($5+1, 0"; - A(si, a;; 0 1)
(7)
where a" = max,, Q (s, a²) To help stabilize learning, we use lagged versions of 0 0% and 0^ as done Mnih al, (2015); the lagged version is signified with the symbol in the exponent.
Unfortunately, as the dimensionality of z grows, the number of parameters needed by A grows quadratically, However, by identifying the or term A(s, a) as a symmetric matrix, is possible to model only the upper triangular portion of the matrix requiring about half the number of
A quantity close to the variance, but not zero mean.
It would still be necessary manipulate this matrix that forms full matrix.
4

---

Under review as a conference paper at ICLR 2021
parameters. To further reduce parameters, each e and A pathways have two hidden layers before their outputs, reflected in Figure Ia In this way, the parameters are shared amongst pathways. which contrasts with other works with multiple sets of layers per action eet al. 2017). To learn the reward parameters A and o, which are the parameters of the approximated non- innar reward function, the following squared loss function is used:
A (Kulkami et al. 2016 Zhang
C Os: in. 1 86, Adi]²
We found that the loss L is not enough to train the parameters A and o alone, and that also regressing towards the Q-function target to be more informative. Therefore, similar to Ma et al. (2020) use the additional loss:
(s,, OA) - (Ô: Vir,, ur; To (Ar,(A
where @ - + ((++),,a)) is the target term The additional L loss can be treated as an auxiliary loss that forces the agent learn values relevant to the actual quantity used for decision making. The L loss is additively included and scaled with a hyperparameter X, which, in this work, typically set between 0.01 and 0.1. It does not adjust the feature parameters involved in the prediction of di.
However, similar to as the dimensionality of increases, so does the number of parameters needed for modelling matrix A E R Therefore, in the interest of reducing the number of parameters we use factorization that splits the matrix A R=*= into two parts with a smaller inner dimension f. A = L R where (L, RJ E R=x By factoring the matrix in this way, require 2 x parameters instead of = X z. If we use values for smaller than 5. we reduce the number of suggested in the context of visual question answering (Yu et al., 2017] Fukui et al., 2016). The factorization of A was primarily done to reduce the total number of parameters in our model. In Appendix E we show that using the full matrix still a tractable learning problem and differs only slightly in overall performance. Combining our losses, the composite loss function is the sum of the four losses given above:
parameters required by matrix A. A similar factorization
C(00, 0 0 0r ., C LV + 3C + L" + +
(10)
In practice, optimize Equation 10 with respect its parameters, (0*, (*) and (0 0 o, A) are iteratively updated. Doing so increases the stability of the approximations learned by the model and ensures that the branches modelling and A do not backpropagate gradients to affect 8 (Machado et al.) 2017: IZhang et al., 2017, Kulkarni et al.) (0166). Additionally, by training in this way, the state representation e can learn features that are both good predictor of the reward ry and useful in discriminating between states (Kulkarni et al. 2016).
3 EXPERIMENTS
This section examines the properties of the proposed approach Axes, navigation task, and on Reacher, a robotic control task built using the MuJoCo engine (Toorov et al. 2012)). The environments are shown in Figure 2 they each contain several tasks specified by goal location and are split between training and test distributions. The experiments examine what the model learns and give preliminary examination of possible uses for the A component in the proposed model.
Specifically, we show that the non-linear assumption increases the representational power of the reward function, provide evidence that the model can learn non-linear rewards, examine the learned function to understand if it captures environmental stochasticity, and evaluate different guided exploration strategies using the A term on new tasks. Additional environment details are provided in Appendix B
This term is written to be generally applicable to different training methodologies. As we train with an A3C-type framework, we regress towards estimate of the return from the environment trajectories.
5

---

Under review as a conference paper at ICLR 2021
(a) Axes Environment.
Reacher Environment.
Figure 2: Environments A graphical representation of the Axes environment. The agent, shown as a red square, must traverse to various goal locations marked with the letter "G". The eight goal locations are split between training, shown blue boxes, and testing, shown green boxes. b) A rendering of the Reacher task. The agent controls the robotic Sawyer arm to move the end-effector to 3D point in space. The eight goal locations are shown balls. Training goals as green, and test goals as red.
3.1 ENVIRONMENTS
Axes: In this environment, shown in Figure 2a the agent, shown by the red square, must traverse the map to reach a goal location using four actions: up, down, left, and right. The agent receives a reward equal to the negative distance between itself and the target goal each step. The agent's 's state space contains the (I, y) location itself and the current target goal within the environment such that the state 8, Rd. With this state space the agent must learn reward function that approximate the distance between itself and the goal location, d(aa, b) /(b, a²)² (by ay)², 2 non-linear function. We expect that the linear variant will fail this task as it cannot model non-linear reward function. This experiment shows that the non-linear reward function can compensate for weak state representation.
Reacher: The second environment is control task defined in the MuloCo physics engine (Todoroy iet al. 2012), shown in Figure 2b A modified version of the robotic model provided by Metaworld (Yu et al. 2019) was used. This environment was chosen to show that the proposed method can scale to difiicutt control tasks. In this environment, the agent must move a simulated robotic arm specific 3D point in space by activating four torque controlled motors. Similarly to Axes, the environment has predefined tasks split between training and test distributions, with the eight goals shown in Figure 2b as green and balls, respectively. The state space is represented as the (Ir, y, =) location of the current goal and the end effector, such that E TR. Reacher is similar to the Axes environment, in that the agent must compute the distance between the two points. The agent receives reward equal the negative distance between the end-effector and the current target goal each step. We discretize the actions such that the agent has nine discrete actions that control the arms movements.
3.2 EXPERIMENTAL SETUP
The train and test setup was similar to those used in previous studies. The agent is trained on distribution of randomly sampled train tasks; then, during testing, we change to a distribution of unseen tasks. A single policy и is trained over all tasks. During transfer to unseen tasks, the model re-learns only reward parameters, with the remainder the model frozen (Zhang et al., 2017 7, Kulkarni et al. 2016). The newly learned policy will vary from the original but is able to exploit previously learned environment knowledge for the new tasks.
Our method is compared against bascline version of the successor feature framework, most similar to that of Kulkarni et al. (2016). This baseline is identical in all ways to the proposed method except for the exclusion of terms containing A, specifically Equations 2 and I0 which can be obtained by setting 3 = 0. uniform random action baseline was considered in environments to act as a floor.
The number of fully connected layers within the encoder of each model varied between environments. The Axes environment used one layer while the Reacher environment used two layers in their encoders. Initially, on the Axes environment, the models all used the raw features with encoder

---

Under review as a conference paper at ICLR 2021
Axes
Reacher
Total Aug. Roward
AmD
Total Avg. Reward
8m0
Random
Random
-10
-15
-30
-20
-40
-25
-50
0.3
0.25
0.5
0.75
1.0
3.25
1.5
Steps 10
Staps
(a) Axes Environment.
(b) Reacher Environment.
Figure 3: Performance in Axes and Reacher environments. The propose model, with the non-linear reward, is shown as the green line, while the linear variant is shown as the blue line. The non-linear variant outperforms the linear variant while differing by only the inclusion of the A component, controlled with the 3 hyperparameter. We assume the worse than random performance of the linear variant is due the model learning degenerate state representation and therefore policy.
such that Φb I(ss). We found this led to worse performance for the linear model as now had no chance to learn suitable encoding of the features for reward prediction. In Appendix []] we performed an ablation within the Axes environment that controls for the number of parameters in both variants of the model; we found even with equal parameters there is little impact on overall performance.
We report each model's mean performance all plots as the average over three runs with varied seeds. Each plot includes the standard deviation over all runs as a shaded area. Additional experiment and model details can be found in Appendix C
3.3 ENVIRONMENT PERFORMANCE
The first step was to examine the performance of the proposed method against various baselines. The primary point of comparison was between the proposed method and the original formulation of the successor feature framework, which can be recovered exactly by setting 3 0 in Equations 2 and I0 shown Figure for both Axes, on the
of the proposed model. The result of these experiments left, and Reacher, on the right. As shown, all versions of the proposed model using the non-linear reward outperformed the other baseline methods in either convergence speed overall performance. Clearly, using a non-linear reward function greatly improved flexibility and performance, enabling the proposed method learn more accurate reward model for the task.
The failure of the linear reward model, SF 3 = 0 of Figure4aaand 4b in both environments is expected. The model, with the linear reward function, not able appropriately model the environments reward structure as the reward is non-linear function of the state; in this case, the agent's coordinates and the current goal location. It was found that increasing the representational power of the encoder with additional fully connected layers helps but does not allow the model to match performance of the non- non-linear variant (see Appendix [E). We also hypothesize that the poor performance of the linear model, leading it to eventually perform worse than random, is due to the model learning degenerate features.
3.4 TASK TRANSFER
An important property of the successor feature framework is the ability to adapt rapidly to new tasks within the same environment. Adaption, or transfer, is accomplished by freezing the model's state-dependent components, such th, and learning only the reward parameters often a small vector equal to the embedding dimensionality of σ, can be quickly learned. This section examines how well the proposed method, with the additional branch for A and the reward parameter A, can transfer to new tasks in both the Axes and Reacher environment.
Because is
change the task distribution within the environment. After the task distribution change, denoted by the dashed vertical line in Figure 4 all model parameters frozen except those pertaining the reward functions, which were then trained using samples
After training the models to convergence,
7

---

Under review as a conference paper at ICLR 2021
Axes Transfer
SF P00
Total Aug, Rowardd
Total Avg. Reward
-1.
SF Bml
Ra -dam
Randorn
-20
-30
-20
-40
-50
0.25
0.75
1.55
Steps 10
Sceps 10*
(a) Axes Environment.
(b) Reacher Environment.
Figure 4: Transfer Performance on the Axes and Reacher environments. Performance on the Axes environments is lower than training the test tasks are average farther from the agent's 's initial position.
from this task. During transfer, scaled the learning rate by fixed factor, randomly initialize the learned weights, and re-decay the value so the method has chance to explore. Full details are provided in Appendix D
Figure clearly shows that our proposed method converged quickly to the new test tasks and does so faster than training from scratch both the Axes and Reacher environments. Faster learning indicates that the model is reusing previously learned task knowledge. We see that the linear variant still unable model the reward within the environment, leading to poor performance.
3.5 MODELLING ENVIRONMENTAL STOCHASTICCTY
This section examines the A funetion to determine whether it can capture stochasticity the en- vironment. We use the Axes environment as it is easy examine the representations learned by the model. However, to so, we modify the Axes environment to include some randomness. The modified version, referred to half-randaa nalf-random, is identical in all aspects to the base version except for a location- based conditional that affects the agent's ' actions. In other words, the agent is within the positive quadrant of the map, 0, then actions randomly perturbed with fixed probability. Otherwise, they are fully deterministic. This randomness is shown in Figure 5a by the red shaded area.
After training to convergence on the half-random variant, we examine the function that the model has learned. Because the A function is modelling the auto-correlation matrix, the future expected correlation of each feature with itself is found by looking along the diagonal. Figure 55) shows the result of plotting diagonal value, this case, feature 1, of the A-matrix over the entire state space. We can see that one of the diagonal components of A did indeed learn to approximately model the conditional random field within the environment.
monNs. 2t11
a 1)0.1
2y0.D
3)
(a) Half-random Axes.
(b) Visualization Of A.
Figure 5: Visualization of Lambda Function on half-random Axes. a) The half-random variant of Axes. b) The learned expected future correlation of one feature with itself along A's diagonal is visualized over the entire state space. The first column is the max value of A over the actions. The remaining columns, from left to right, correspond to each action: left, up, right, and down. Red and blue correspond to maximal and minimal values.
8

---

Under review as a conference paper at ICLR 2021
Exploration Mocies on Axes
Exoloration Modes Reacher
Total Aug. Roward
Total Avg. Reward
-10
-10
-20
-20
-30
e-greecy A-uniform
E-greedy
-10
-10
Avunttorm
6.1 Steps 10"
0.1
0.2
0.3
0.4
Staps
(a) Axes Environment.
(b) Reacher Environment.
Figure 6: Guided Exploration: The A component of the proposed model is used to guide exploration during transfer. By using the agent explores in directions with large variance in the state space.
3.6 GUIDED EXPLORATION WITH
This section presents preliminary results on possible uses for the A-function. Specifically, whether it possible to use the A-function for guided exploration during transfer within the Axes and Reacher environments.
The Successor Features, given Equation 2 can be interpreted as predicting the future expected path taken by the policy in an environment. Under this interpretation, t can be seen as capturing the expected features the states and the expected variance between state features along these pathways. Under e-greedy policy, the captured variance would be induced by the random actions taken. Adding noise to the A component would then perturb around the expected path, the one captured by v. to nearby states done by e-greedy policy. Therefore, instead of using e-greedy exploration, it possible add noise to A during transfer, such that Å(s, a) A(s, a) eA(s, a). where is sampled from some distribution. During learning, the variance of the sampling distribution, controlled by a, can be annealed to some final value. The actions are then sampled from the model at time as:
0₁ = argmaz (1b( a TT o tr (r(A a²)aa"))))
(11)
Various permutations of sampling distributions and structures were examined. Experimentally. it was found that using a scalar value sampled from uniform noise, that is e ~ u((aa, a.) where E [-a, o.). provides the best performance. From Figure we see that using for directed exploration alternative. In Figure 6a the directed exploration method manages to converge faster than e-greedy. We believe that using A lets the model efficiently explore in directions along expected state pathways and reuses previously gained knowledge towards new tasks.
viable
4 CONCLUSION & FUTURE WORK
In this paper, we have derived novel formulation of successor features with non-linear reward. We have shown that the agent can model reward non-linear reward structure, not possible under the old linear formulation. Further, we have shown the utility of the A term that appears in the derivation of the new state-action function. Experimentally, we have shown that the A term is able to capture the stochastic nature of an environment and can be used for directed exploration.
In future work, we aim to explore the A function deeply. Specifically, what A learns and we can find a formulation that learns the future expected variance. Other possible avenues of future work include improvements to directed exploration with different annealing and whether our finding with the A function can help improve the Q-function.
9

---

Under review as a conference paper at ICLR 2021
REFERENCES
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath, A brief survey of deep reinforcement learning. arXiv preprint arXiv: 05866, 2017.
André Barreto, Will Dabney. Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in neural information processing systems, pp. 4055-4065, 2017.
Richard Bellman. Dynamic programming. Science, 153(3731):34 37, 1966.
Peter Dayan. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613 3-4624 1993.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070, 2018.
Gregory Farquhar, Tim Rocktäschel, Maximilian Igl, and Shimon Whiteson. Treeqn and atreec: Differ- entiable tree-structured models for deep reinforcement learning, arXiv preprint arXiv:1710.11411 2017.
Akira Fukui, Dong Huk Park, Daylen Yang. Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:
2016.
Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. arXiv preprint 1rXiv::906.000300 2019.
Tejas D Kulkarni, Ardavan Sacedi, Simanta Gautam, and Samuel J Gershman. Deep successor reinforcement learning. arXiv preprint arXiv: 02396, 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015.
Chen Ma. Dylan R Ashley, Junfeng Wen, and Yoshua Bengio. Universal successor features for transfer reinforcement learning. arXiv preprint urXiv:2001.0400 044025, 2020.
Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. arXiv preprint arXiv:1710.11003 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Junhyuk Oh, Xiaoxiao Guo. Honglak Lee, Richard L Lewis, and Satinder Singh. Action-conditioe video prediction using deep networks atari games. In Advances neural information processing systems, pp. 2863-2871, 2015.
Martin L Puterman. Markov decision processes: discrete stochastic dynumic programming. John Wiley & Sons, 2014.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwiesser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484,, 2016.
Csaba Szepesvári. Algorithms for reinforcement learning. 2009.
Norman Tasfi and Miriam Capretz. Dynamic planning networks. arXiv preprint arXiv:1812.11244 2018.
10

---

Under review as a conference paper at ICLR 2021
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 0026-5033.
IEEE, 2012.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using agett reinforcement learning. Nature, 575(7782):350-354 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv: 1910. 10897, 2019.
Zhou Yu, Jun Yu, Jianping Fan, and Dacheng Tao. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 1821-1830, 2017.
Jingwei Zhang, Jost Tobias Springenberg, Joschka Boedecker, and Wolfram Burgard. Deep rein- forcement learning with successor features for navigation across similar environments. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2371 1378-. IEEE, 2017.

---

Under review as a conference paper at ICLR 2021
APPENDIX
A NON-LINEAR REWARD DERIVATION
Here we provide the derivation for the non-linear reward in the successor framework. First, we start by assuming the reward rt has the following form:
= o Ao:
(12)
where { oo R=x1, and A E Rax= and both and A are learnable parameters. Following from the definition of the state-action value function Q(s, a), the adjusted reward function can be substituted yield:
(=,,, = E" [ret + Y744 + ........... ................................ s, A. = a|
(13)
E 10 of Adt+ + 20 + 70 +... |S s, Ar = a]
(14)
Dropping the conditional portion of the expectation for brevity, linearity of expectation be used to split apart the terms containing A and o. Then o is pulled out from the first term:
E 10 70 ... E 0 Aou+1
yo 201+2Ar12 + +...]
(15)
E" (o+41 70442 + o E" (P+t
Adit1
7 Adit2 +...]
(16)
By recognizing the first expectation term as the successor features (bs a). Equation 16 can be rewritten as
as, E + A0142 +...]
(17)
Because T Aó results in scalar, the trace function tr(-) can be used inside the right-hand term:
o+ E (tr(0 AΦ1+1) + A01-2) +...]
(18)
By exploiting the fact that tr(AB) = tr(BA), the terms inside the trace function can be swapped to yield:
- o E [r [tr(A + (1)7 +...]
(19)
Because both tr(-) and are linear, they can be pulled out of the expectation, giving:
x (s,a)Te o + (A +
A + ....))
(20)
vid (s, a) tr(AE" 10 + 70.420T4 +....)
(21)
Finally, the remaining expectation be expressed as function:
(" = 30 (s, To (tr(AAA
(22)
8 e {0, 13 is hyperparameterr that controls the inclusion of the non-linear component. We define 2 and A as:
(s,a) = =E + 7b05541, (((+1)))) = A, = a]
(23)
A A²(8,a) a) 17 + ((48H) A, a)
(24)
12

---

Under review as a conference paper at ICLR 2021
B ENVIRONMENTS
AXES
In the Axes environment, each action moves the agent by 0.01 units in the desired direction with the traversable space defined by 15 x 15 unit box centered at the origin (0, 0). Within this environment eight separate goal locations exist split between train and test distributions. An episode ends when either the agent reaches the goal or more than 225 steps have elapsed. The agent's starting location is randomly sampled from grid of 3 step units, centered (0,0). 0).
B.2 REACHER
In the Reacher environment, an episode ends when 150 steps have elapsed or the agent is within 7cm of the goal.
Because the models can be used only with discrete actions, it was necessary to transform the environmental actions. Therefore, the four-dimensional continuous action space A was discretized using two values per dimension: the maximum positive and maximum negative torque for each actuator. An all-zero option was included that applies zero torque along all actuators, resulting in total of nine discrete actions.
C EXPERIMENTS
We use the factorization of matrix with the two separate matrices, L and R. with an inner dimension equal to f= 3 1. All models, unless specified otherwise, were trained using a synchronous version of A3C (Mnih al. 2016). We trained all components on n-step trajectories, with 5, generated by 16 separate threads. The targets for yi and A were estimated using n-step return across latent states é. All models used an annealed c-greedy method for exploration. Within both the and Reacher environment, e-greedy was annealed from 1.0 to final over the first 250k steps.
In the Axes and Reacher environments encoder and decoder each respectively contain one and two hidden layers with embedding size equal to the double the raw state size. Both & and A increase the hidden dimension z by a fixed factor before output. This factor Zfactor depends on the environment. All environments used discount factor of y 0.99, À 0.1, and updated the parameters every 25k steps.
C.I AXES
We used embedding size of 2 8 in the Axes environment. The final value used in e-greedy was 0.1 with a learning rate a = 2.5e was used.
C.2 REACHER
We used embedding size of z 12 in the Reacher environment. The final value used in e-greedy was 0.05 with a learning rate of a = 5e 4 was used.
D TRANSFER
During transfer reinitialize the reward specific parameters A and o using orthogonal initializa- tion with a gain of All other model parameters are held frozen and do not change. The learning rate is increased by factor of 2 and 10x on the Axes and Reacher environments respectively. We anneal the exploration parameter from to 0.1 in Axes and to 0.05 on the Reacher environments over the first 200k steps.
E ADDITIONAL EXPERIMENTS
In this section we discuss additional experiments performed within the Axes environment. The Axes environment was used as it has simple dynamics and representation such that the resulting
13

---

Under review as a conference paper at ICLR 2021
Axes With Additional Ercoder Loyers
Axes A Comparison
Total Ave. Roward
-10
Avrg
-20
pwo Bnc
-10
B-0
-15
-30
Random
Bel: L-R
Steps 10
Steps 10*
(a) Stronger State Encoder
(b) Factorization
Axes Equal Parameters
SF p-0 Equal
Tutal Ava. Reward
5f P-0
10
Random
ST pi-
-30
Steps
(c) Equal Parameters
Figure 7: Additional Experiments
performance is This primary motivation was to understand if there are any confounding factors in the performance of the linear and non-linear models.
E.1 STRONGER STATE ENCODER
In this experiment we add additional fully connected layers to the encoder of the linear model. From Figure 7a we see that as the strength of the encoder increases so does the model's overall performance. This is unsurprising with additional parameters. To near the performance four total, fully connected layers each with æ hidden units. This is significant increase in parameters when compared to the non-linear variant which only has single fully connected layer in its encoder. Even with the additional layers the linear version does match the performance of the non-linear model.
the encoder is able to learn a better encoding of the state for reward prediction
the non- non-linear model requires three additional,
E.2 A FACTORIZATION
In this experiment we show the factorization of the parameter has little effect on the model's performance. We examined using full matrix of parameters, such that A E g=*= and a factorization where A- L- R where [L, R.- E R=*/, From Figure 7b we see that the final performance of both methods are identical and they roughly converge at the same speed. The full matrix variant converges faster than the factorized version, but has nearly double the number of parameters. Learning the full matrix variant is tractable learning problem and learning matrices of such scale has been done in other work, such those in state-transition models (Oh et al., 2015 Farquhar et al. (2017] Tasfi & Capretz 2018). In the case of model, chose the factorized version to reduce the total number of parameters.
E.3 EQUAL PARAMETERS
To ensure that the non-linear model's performance is not simply due to a greater number of parameters, examine linear variant with approximately equal number of parameters. The parameters of the linear variant are increased by adjusting the number hidden units, controlled by the hyperparameter From Figure 7 we see that setting the number of parameters roughly equal to that of the non-linear model does help performance slightly but the linear model still cannot solve the task. We conclude that the reward structure introduced by paper, with its ability to model linear reward, is the primary reason for the increase in performance.
14


Reviews:
[{"review_id": "lAeMTHCp7gT", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a quadratic reformulation of successor features (SF), in which rewards are given by $r=\\phi^\\top \\mathbf{o} + \\phi^\\top A \\phi$ instead of the usual $r = \\phi^\\top w$. This generalization leads to learning a second-order term $\\Lambda = \\mathbb{E}[\\sum_{t} \\gamma^t \\phi^\\top \\phi]$ to augment the expected featurization $\\psi = \\mathbb{E}[\\sum_{t} \\gamma^t \\phi]$.\n\nWhile this is a more expressive parametrization, learning $\\Lambda$ is difficult due to its dimensionality. One of the strengths of SF is that it turns a high-dimensional prediction problem into a set of independent scalar Q-learning problems; incorporating $\\Lambda$ arguably misses out on that important strength by no longer being able to treat the dimensions of the state featurization as independent. It seems that the learning problem can be made tractable by a low-rank factorization, but this does somewhat call into question whether a quadratic form is really the benefit or just expressivity from having more parameters. This could be answered by an experiment controlling for the number of parameters in $\\phi$ (for the linear SF baseline) to match that in the quadratic variant.\n\nThe addition of $\\Lambda$ is motivated by the limitation of a linear reward model:\n> *The factorization follows from the assumption that reward can be predicted as the dot product between a state representation vector and a learned reward vector.*\n\nThis is not exactly an assumption as stated, since the featurization could be the reward itself ($\\phi(s) = r(s), w = 1$), meaning that this decomposition is possible for any reward function. (See the discussion under Equation 2 in [Barreto, 2016].) That is not to say there is no limitation from linearity: it is true that for a *given* featurization, it may not be possible to linearly solve for any reward function, and that it may not be possible to find any suitable featurization for a set of many reward functions, but this is not discussed in enough depth to know exactly what issue is being referenced.\n\nThe experimental evaluation shows that the quadratic variant outperforms linear SF in two toy tasks. This is explained by the fact that these environments require a nonlinear model:\n> *Finding a solution to the environmental reward structure is difficult as the reward is a\nnon-linear function of the features; in this case, the agent’s coordinates and the current goal location.*\n\nHowever, even in the case of SF, the reward should not depend linearly on the observations themselves, but on a featurization of the observations. This is why a linear parametrization is suitable for any single reward function: the featurization can be arbitrarily complicated. Are you using observations (coordinates and goal location) instead of $\\phi$ here?\n\nOther experiments focus on exploration by adding noise to $\\Lambda$ (instead of actions) and the structure of the learned $\\Lambda$. The exploration idea is interesting, but somewhat underdeveloped, with there not being much justification and no clear empirical win. For what it is worth, the two environments studied might be simple enough that naive $\\epsilon$-greedy exploration is good enough, and it is difficult to squeeze out much improvement without considering an environment which poses more of an exploration challenge. I did not understand the $\\Lambda$ visualization in figure 5.\n\nThough the major comparison of interest is between SF and the quadratic variant, it would be nice to explore a few more ablations of the method (like controlling for number of parameters, discussed above). It would also relieve a little bit of concern to have one or two more baselines on the standardized environments; the results in figure 3 look surprisingly sample-inefficient compared to modern Q-learning methods, so it could be worth figuring out why or showing that this is not the case by including another baseline.\n\n**Questions:**\n1. Where exactly does $\\beta$ come from? It seems to appear between Equations 21 and 22 in Appendix A, but I cannot find the reason. Can the constant just be subsumed into $A$ in the reward regression in Equation 8? Is there ever a reason to set $\\beta$ to anything besides 0 or 1?\n2. What happens when you remove the reconstruction objective on $\\phi$? Another advantage of the SF decomposition is that it can learn features informative for predicting values without having to rely on reconstruction, so can discard irrelevant parts of the state which would normally thwart a reconstruction objective. Incorporating a reconstruction term seems to miss out on that benefit. I realize prior work like [Kulkarni 2016] has used a similar auxiliary objective, so this isn't really a negative so much as a question as to why you think it is necessary.\n\n**Copy-editing:**\n1. Section 1.1 *The object* —> *the objective*\n\n**Summary:**\nThough this is an interesting generalization of SF, it does not seem quite ready for publication. I encourage the authors to more precisely motivate the quadratic form, since even after reading this paper its disadvantages (can no longer treat the dimensions of $\\phi$ as independent, intractable due to dimensionality so requires approximation anyway) seem to outweigh potential advantages due to the increased expressivity (especially given the note above about a linear model being sufficient for any single reward).\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "KZCZLzhomwf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Successor representations are an old idea that has seem recent interest in the ML community. The idea is conceptually straightforward, by assuming the rewards are linear in some space $r = \\vect{\\phi}(s, a) \\cdot \\vect{w}$ then we learn something analogous to an action-value function for the discounted expected features under a policy so that the action-value on task $\\vect{w}$ is $Q^\\pi_{\\vect{w}}(s, a) = \\psi^\\pi(s, a) \\cdot \\vect{w}$. This allows computing the action value for the policy under a new task $\\vect{w}'$ straight.\n\nOne limitation is the assumption that there is some feature space where the rewards of all tasks can be linear in this space.\n\nThe key idea is to extend the idea of successor features to the 2nd order relation between features and reward so now the assumption is the reward for all tasks is defined by eq 1 in the paper\nso in addition to computing the expected features in the future $\\psi^\\pi(s, a)$ (a vector value) the autocorrelation of the features $\\Lambda^\\pi(s, a)$ must also be track (a matrix).\n\nThey construct a method for learning features as an autoencoder of the state and experiment on some simple tasks such as reaching to different locations comparing this ``second order'' method against successor features.\n\nStrengths:\n- This does seem like the natural ``second order'' version of successor features. I'm not aware of any closely related prior work to extend successor features in this way and it clearly allows for a more expressive description of rewards.\n\n- Paper is mostly well-written and communicated, including try to help develop an intuition for the new quantities introduced.\n\n- The interpretations of $\\Lambda$ are interesting and ideas around how to use it for exploration are intriguing.\n\nWeakness:\n\nThe primary weakness of this work is the experimental section. There seems to a number of different issues all conflated into one set of experiments.\n\nFirstly, whether using successor features (SF) or 2nd order successor features (SF^2), $\\psi^\\pi$ and $\\Lambda^\\pi$ are a function of a particular policy. It is unclear if, for each training task a different $\\mathbf{w} = \\mathbf{o} + \\mathbf{A}$ was learned (as one might expect in a SF setup), there is never any indices indicating these are task specific so perhaps there is only one version of these learned for all training tasks? It seems that only a single policy is learned for all training tasks. The setup should be clarified and the reasoning for these choices explained.\n\nThis means that over both training and test tasks only a single policy is available. Therefore, any hope of solving an individual task must be due to computing the advantage under this policy which is quite limiting. Both Barreto et al., and Ma et al. estimate the successor features for a set of policy (indeed the main contribution of Barreto is to show how to use Generalized Policy Improvement (GPI) to construct a policy for a new task defined by $w'$ from a set of existing policies, typically one per training task, there seems no reason GPI cannot be used here).\n\nFinally, it would be helpful when introducing a conceptual idea as here to have a simple version where only that idea is needed. In this case, by (as in Barreto) experiments with fixed, pre-defined features. Here, in both experiments the features are learned using an autoencoder of the state space (which is not guaranteed to learn a representation which is ideal for use in successor features).\n\nThese limitations make it hard to interpret the experiments or compare with other work. For example, the two tasks appear quite similar to Barreto et al., yet here the SF baseline ($\\beta=0$ in the paper) performs worse than random, even on the baselines. It is hard to interpret why this: is it due to the learned features not being good for this task, using only one policy for all tasks or the non-linearity of reward? It seems clear that e.g. for the Axes task if there was a feature for being located on each potential goal and a policy per task then, at least on the training tasks, SF should perform much better than random (by learning to reach for the targets).\n\nIdeally, SF would also be compared against other methods for transfer such as meta-learning (e.g MAML). The existence of such methods, along with model-based approaches such as used in Go mentioned in the introduction probably means the claim in the introduction that \"current algorithms cannot transfer a learned policy between related tasks\" is too strong.\n\nThis issues are fixable by more careful experimentation and clarity on the exact setup of the problem. As part of that the captions for figures 3 and 4 could be extended.\n\nA more general weakness is that, while this paper improves the expressiveness of SF by allowing a 2nd order relationship between features and reward, it is not clear if this is the key limitation of successor features. Namely, it still does not allow transfer between tasks where the transition function has changed ([1] should probably be cited as an attempt to extend SF in this direction) and only supports estimating the feature occupancy under existing policies. I personally would regard these are the more limiting factors in SF rather than the linearity of reward. These issues should be discussed and the limitations and weakness compared to e.g. meta-learning approaches should be discussed (and as mentioned above, ideally compared).\n\nJust to be explicitly clear the rating given to the paper is the paper in it's current state. I think if the issues above are addressed this is an interesting paper and I would rate higher.\n\n[1] Zhang, Jingwei, et al. \"Deep reinforcement learning with successor features for navigation across similar environments.\" 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GfPCmr2RomY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to extend successful features by learning the second moments of cumulants in addition to the cumulants. They demonstrate that the resulting method performs better on 2D and 3D goal-reaching tasks (without obstacles) when the reward is the squared distance.\n\nThe paper is clearly written and extending successor features to non-linear rewards is an interesting problem. However, the framework does not so much provide a “novel formulation of successor features” but rather presents a specific instantiation. While this maybe be an interesting observation, the resulting experiments fail to compare to more appropriate baselines and are tested on rather simple domains that can be solved by a quadratic policy that is greedy with respect to the reward.\n\nIn detail:\n\nThe paper is effectively an instantiation of the successor feature that adds structure to the cumulants. In particular, it adds features that are products of other features, i.e. phi_new = [phi_1, phi_2, phi1 * phi_2, etc.]. Thus, the claim that this is a “novel formulation of successor features with a non-linear reward” seem inaccurate since the original successor feature formulation already handles the case where the reward is a non-linear function of the state.\n\nGiven the goal-directed nature of the tasks, the authors should compare to goal-conditioned methods [1,2,3] or at least demonstrate the method on tasks that cannot be solved by goal-conditioned methods. It is also an overstatement to say that the 3D reaching task is a particularly “difficult control task” given that much more challenging control tasks have been solved [4].\n\n\nMinor comments:\n\nIs it actually torque control? If so, how is it possible for the policy to learn given that it only has the XY location of the end effector? I’m under the impression that the meta-world tasks use end-effector velocity control.\n\n“as the dimensionality of z grows, the number of parameters needed by Lambda grows exponentially” Do you mean that as the dimensionality of phi grows (or as z grows) the dimensionality of Lambda grows quadratically?\n\n[1] Schaul, Tom, et al. \"Universal value function approximators.\" International conference on machine learning. 2015.\n[2] Andrychowicz, Marcin, et al. \"Hindsight experience replay.\" Advances in neural information processing systems. 2017.\n[3] Nair, Ashvin V., et al. \"Visual reinforcement learning with imagined goals.\" Advances in Neural Information Processing Systems. 2018.\n[4] Plappert, Matthias, et al. \"Multi-goal reinforcement learning: Challenging robotics environments and request for research.\" arXiv preprint arXiv:1802.09464 (2018).\n\n--- Post Rebuttal ---\n\nI've read the author response. However, I do not plan on changing my score as my main concerns have not be addressed. In particular:\n\n> the framework does not so much provide a “novel formulation of successor features” but rather presents a specific instantiation... Thus, the claim that this is a “novel formulation of successor features with a non-linear reward” seem inaccurate since the original successor feature formulation already handles the case where the reward is a non-linear function of the state.\n\nThe author response is\n\n> There is no guarantee that the learned state features are able to find appropriate values. In theory yes, we agree but practically we found this not to be the case if we explicitly test for it. As our environments do, where the reward is a non-linear function of the state.\n\nI understand that perhaps existing methods are not capable of learning good state features, and presenting a method for finding better state features (not the weights) would be interesting. However, the current paper simply hard-codes good features, which I do not find compelling.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1u8m0QocCb-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors extend the Successor Features (SF) framework to allow for a quadratic  relationship between the features and rewards, rather than the strictly linear one given in the initial formulation. The derivation is relatively clear and appears to be correct, with the nice result being that the additional term needed to account for the non-linearity can still be estimated via a Bellman equation.\n\nOne fact that the authors fail to address is that the linearity between the features and rewards doesn't limit the expressiveness of value functions computed by SF, since the features can be an arbitrarily non-linear function of the observations. Now, it might be the case that the ability to generalize to novel tasks is greater when making the rewards be non-linear in the features, but the author should make this distinction.\n\nIndeed, this ties into my larger issue when the paper: the Axes task (and maybe the Reacher task depending on the dynamics) is very simple, to the point that it is very unclear why the linear SF formulation does worse than not only the quadratic case, but than a random agent. As per my argument about expressivity, both SF agents should be equally able to solve the training tasks, with the only space for an advantage for the non-linear version coming from generalization.\n\nBut the Axes task appears too simple to even be used for comparing generalization performance (unless I'm mistaken about the dynamics). If the tasks terminate upon reaching the goal, and the start states are all in the middle of the goals, then couldn't an agent linear in the *observations* solve the tasks optimally? e.g. map the difference between the current and goal observations to the action most closely corresponding to the resulting direction.\n\nThe Reacher task might also suffer from these problems, but I'm unsure without further details. Do the actions act on the torques of the robot motors, or do they directly impact end-effector space? If it is the latter, then Reacher appears to just be Axes with an addition spatial dimension, which would still be readily solved without any non-linear function approximation.\n\nI apologize if I sound overly harsh. The extension is interesting, but currently its unclear if these experiments provide any support to your claims about its advantages. Unless I'm mistaken about the dynamics of these environments, I'd suggest showing results on tasks of known complexity, like the Doom game Deep SR uses or the Scavenger environment from the SF paper.\n\nThe exploration angle is quite interesting, but currently its hard to understand the motivation. Why is perturbing this expected variance-like object a good idea? Expanding upon that would be greatly appreciated, though for it to be the main claim of the paper there would also have to be some comparison to alternative exploration methods (e.g. Noisy Nets or Bootstrap DQN).", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the idea of successor representations. Typically the reward is compute linearly on top of states in this setting but the authors relax it to have a quadratic form. 

${\bf Pros}$:
1. A novel formulation of the successor representation where the reward does not follow the linearity assumption
2. The idea of using a second order term for the reward branch is interesting and could have meaningful implications for learning and exploration. 

${\bf Cons}$:
1. All authors agree that the experimental results do not clearly validate the advantage of this method. More work is needed to establish the effects of using this particular reward structure on a wide variety of tasks

2. Both R2 and R4 were unconvinced of the limitations of the linearity assumptions in the original successor representation formulation -- especially in the case when the state is represented by a non-linear function approximator.

The ideas presented in this paper are quite interesting and promising. But more experimental work is needed to show the benefits of this approach. 
Decision: Reject