Continual Auxiliary Task Learning
Authors: Matthew K McLeod
Chunlok Lo
Matthew Kyle Schlegel
Andrew Jacobsen
Raksha Kumaraswamy
Martha White
Adam M White
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Never-Ending Learning
Auxiliary Tasks
Intrinsic Motivation
Off-Policy Learning
Successor Features
General Value Functions

Abstract: Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems. A variety of off-policy learning algorithms have been developed to learn such predictions, but as yet there is little work on how to adapt the behavior to gather useful data for those off-policy predictions. In this work, we investigate a reinforcement learning system designed to learn a collection of auxiliary tasks, with a behavior policy learning to take actions to improve those auxiliary predictions. We highlight the inherent non-stationarity in this continual auxiliary task learning problem, for both prediction learners and the behavior learner. We develop an algorithm based on successor features that facilitates tracking under non-stationary rewards, and prove the separation into learning successor features and rewards provides convergence rate improvements. We conduct an in-depth study into the resulting multi-prediction learning system. 

Continual Auxiliary Task Learning
Matthew McLeod, Chunlok Lo, Matthew Schlegel, Andrew Jacobsen, Raksha Kumaraswamy Department Computing Science, University of Alberta (mmcleod2, chunlok, mwschllg, jacoba, htta:cobbbkuaaassst.... Qual
Martha White, Adam White
Department of Computing Science, University of Alberta CIFAR Canada AI Chair, Alberta Machine Intelligence Institute (Amii) {whitem, amw8 Qualberta.
Abstract
Learning auxiliary tasks, such as multiple predictions about the world, can provide many benefits to reinforcement learning systems. A variety of off- off-policy learning algorithms have been developed learn such predictions, work on how to adapt the behavior to gather useful data for those off-policy pre- dictions. In this work, we investigate a reinforcement learning system designed learn collection of auxiliary tasks, with behavior policy learning to take actions to improve those auxiliary predictions. We highlight the inherent non-stationarity in this continual auxiliary task learning problem, for both prediction learners and the behavior learner. We develop an algorithm based on successor features that facilitates tracking under stationny rewards, and prove the separation into learning successor features and rewards provides convergence rate improvements. We conduct in-depth study into the resulting multi-prediction learning system.
as yet there is little
Introduction
In never-ending learning systems, the agent often faces long periods of time when the external reward uninformative. smart agent should use this time practice reaching subgoals, learning new skills, and refining model predictions. Later, the agent should use this prior learning to efficiently maximize external reward. The agent engages in this self-directed learning during times when the primary drives of the agent (e.g., hunger) are satisfied. Other times, the agent might have to trade-off directly acting towards internal auxiliary learning objectives and taking actions that maximize reward.
In this paper we investigate how an agent should select actions to balance the needs of several auxiliary learning objectives no-reward setting where no external reward is present. In particular, we assume the agent's 's auxiliary objectives are to learn a diverse set of value functions corresponding set of fixed policies. Our solution at high-level is straightforward. Each auxiliary value function is learned in parallel and off-policy, and the behavior selects actions to maximize learning progress. Prior work investigated similar questions in state-less bandit like setting, where both off-policy learning and function approximation are not required [Linke et al., 2020].
Otherwise, the majority of prior work has focused how the agent could make use of auxiliary learning objectives, not how behavior could be used to improve auxiliary task learning. Some work has looked at defining (predictive) features, such successor features and basis of policies [Barreto et al., 2018, Borsa et al., 2019, Barreto et al., 2020, 2019]; universal value function approximators [Schaul et al., 2015]; and features based on value predictions [Schaul and Ring, 2013, Schlegel et al., 2021]. The other focus has been exploration, using auxiliary learning objectives to generate bonuses aid exploration on the main task [Pathak al., 2017, Stadie et al., 2015, Badia et al.. 2020. Burda et al.. 2019]; using a given set of policies in call-return fashion for scheduled auxiliary control
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

[Riedmiller et al., 2018]; and discovering subgoals in environments where it is difficult for the agent to reach particular parts of the state-action space [Machado et al., 2017. Colas et al., 2019, Zhang et al., 2020, Andrychowicz et al., 2017, Pong et al., 2019]. In all of these works, the behavior was either fixed or optimized for the main task.
The problem of adapting the behavior to optimize many auxiliary predictions in the absence of external reward is sufficiently complex merit study in isolation. It involves several inter-dependent learning mechanisms, multiple sources of non-stationarity, and high-variance due to off-policy updating. If we cannot design learning systems that efficiently learn their auxiliary objectives in isolation, then the agent is unlikely to learn its auxiliary tasks while additionally balancing external reward maximization.
Further, understanding how to efficiently learn a collection of auxiliary objectives is complementary to the goals of using those auxiliary objectives. It could amplify the auxiliary task effect in UNREAL (Jaderberg et al., 2017]. improve the efficiency and accuracy of learning successor features and universal value function approximators, and improve the quality of the sub-policies used in scheduled auxiliary control. It can also benefit the numerous systems that discover options, skills, and subgoals [Gregor al., 2017, Eysenbach et al., 2019a, Veeriah al., 2019, Pitis et al., 2020, Nair et al., 2020, Pertsch et al., 2020, Colas et al., 2019, Eysenbach et al., 2019b]. by providing improved algorithms learn the resulting auxiliary tasks. For example, for multiple discovered subgoals, the agent can adapt its behavior to efficiently learn policies to reach each subgoal.
In this paper introduce architecture for parallel auxiliary task learning. As the first such work to tackle this question in reinforcement learning with function approximation, numerous algorithmic challenges arise. We first formalize the problem of learning multiple predictions as reinforcement learning problem, and highlight that the rewards for the behavior policy are inherently non-stationary due changes in learning progress over time. We develop strategy use successor features to exploit the stationarity of the dynamics, whilst allowing for fast tracking of changes in the rewards, and prove that this separation provides a faster convergence rate than standard value function algorithms like temporal difference learning. We empirically show that this separation facilitates tracking both for prediction learners with non-stationary targets as well as the behavior.
2 Problem Formulation
We consider the multi-prediction problem, in which an agent continually interacts with an environment to obtain accurate predictions. This interaction is formalized as a Markov decision process (MDP), defined by set of states S, a set of actions A, and a transition probability function P(s, a, s'). The agent's s goal, when taking actions, is to gather data that is useful for learning N predictions, where each prediction corresponds to general value function (GVF) [Sutton et al., 2011].
A GVF question is formalized as three tuple (TT, (T,,YC), where the target is the expected return of the cumulant, defined by c: S x A x S - R. when following policy π : S x A + [0, 1]. discounted by y: x A x S [0, 1]. More precisely, the target is the action-value
(s, del E, [G 8, A. a) for G. def Ge+1 + 7++1
where C++1 def c(S,, Ar, 1 and 7+1 del = Ar. Ar. S.+1). The extension of y to transitions allows for a broader class of problems, including easily specifying termination, without complicating the theory [White, 2017]. The expectation is under policy क, with transitions according to P. The prediction targets could also be state-value functions; we assume the targets are action-values in this work to provide a unified discussion of successor features for both the GVF and behavior learners.
At each time step, the agent produces N predictions, a Q,") A.) for prediction j with true targets ol (St. At). We assume the GVF question can change over time, and so Q can change with time. The goal is to have low error in the prediction, in terms of the root mean-squared value error (RMSVE), under state-action weighting d S x A R:
RMSVE( Q Q) del
Σ #ES A
Ms ()( a()
(1)
The total error up to time step f, across all predictions, is TE del Σ EN RMSVE(Q,") .Q(").

---

The agent's ' goal is to gather data and update its predictions to make TE small. This goal can itself be formalized as a reinforcement learning problem, by defining rewards for the behavior policy that depend on the agent's 8 predictions. Such rewards are often called intrinsic rewards. For example, if we could directly measure the RMSVE, one potential intrin- sic reward would be the decrease in the RMSVE after taking action A,
Environment
+
State Obeerreeen Sm
Behavior
R
Intrinsic Reward
Weight Change
from state S, and transitioning to S1+1. This reflects the agent's 's learning progress-how much it was able to learn-due to that new experience. The reward is high if the action generated data that resulted in substan- tial learning. While the RMSVE is the most direct measure of learning progress, it cannot be calculated without the true values.
Task
Task
..*
Task N
Agent
Many intrinsic rewards have been considered to es- timate the learning progress of predictions. A recent work provided a thorough survey of different options, empirical study [Linke et al., 2020]. Their conclusion was that, for reasonable prediction learners, simple learning progress measures-like the change in weights-wee were effective for producing ef- fective data gathering. We rely on this conclusion here, and formalize the problem using the l norm on the change in weights. Other intrinsic rewards could be swapped into the framework; but, because focus is the non-stationarity in the system and because empirically we found this weight-change in- trinsic reward be effective, we opt for this simple choice upfront.
Algorithm Multi-Prediction Learning System
Input: N GVF questions
as well
Initialize behavior policy parameters On
and GVF learners w/ ............ w/ (.V)
Obtain initial observation So
for t = 0, I,... do
Choose action A, according тe, (-|S,) Observe next state vector Se+1 1 and 7++1 // Update predictions with new data for j = 1 to N do c ( AL, S4+1) Y +- (S, Ac, S(+1)
Update w with (Se, A1, S(+1,
// Compute intrinsic reward, update behavior Re+1 + Σx= [| - w " 111
We provide the generic pseudocode for a multi- prediction reinforcement learning system, in Algo- rithm 1. Note that the behavior agent also has sep- arate transition-basee
Update 0, with (S,, A,, R 1
7, which enables us to encode both continuing and episodic problems. For example, the eseudo-terminationn for GVF could be particular state in the environment, such as a doorway. The discount for the GVF would be zero in that state, though it is not true terminal state; the behavior discount "/+1 would not be zero
3 Non-stationarity Induced by Learning
On the surface, the multi-predieetee problem outlined the previous section a relatively straightfor- ward reinforcement learning problem. The behavior policy learns to maximize cumulative reward, and simultancously learns predictions about its environment. Many RL systems incorporate prediction learning, either auxiliary tasks or to learn model. However, unlike standard RL problems, the rewards for the behavior are non-stationary when using intrinsic rewards, even under stationary dynamics. Further, the prediction problems themselves non-stationary due to changing behavior.
To understand this more deeply, consider first the behavior rewards. On each time step, the predictions are updated. Progressively, they get more and more accurate. Imagine a scenario where they can become perfectly accurate, such as the tabular setting with stationary cumulants. The behavior rewards are high in early learning, when predictions are inaccurate. As predictions become more and more accurate, the change in weights gets smaller until eventually the behavior rewards are near zero. This means that when the behavior revisits state, the reward distribution has actually changed. More generally, in the function approximation setting, the behavior rewards will continue to change with time, not necessarily decay to zero.
The prediction problems are also con-stationary for two reasons. First, the cumulants themselves might be non- non-stationary, even if the transition dynamics are stationary. For example, the cumulant could correspond to the amount of food in a location in the environment, that slowly gets depleted. Or, the cumulant could depend on a hidden variable, that makes the outcome appear non-stationary. Even with stationary cumulant, the prediction learning problem can be non-stationary due to a changing behavior policy, As the behavior policy changes, the state distribution changes. Implicitly
3

---

when learning off off-pp the predictions are minimizing an objective weighted by the state visitation under the behavior policy, As the behavior changes, the underlying objective is actually changing. resulting in non-stationary prediction problem.
Though there has been some work learning under non-stationarity in RL and bandits, none to our knowledge has addressed the multi-predictiin setting in MDPs. There has been some work developing reinforcement learning algorithms for non-stationary MDPs, but largely for the tabular setting [Sutton and Barto, 2018, Da Silva et al., 2006, Abdallah and Kaisers, 2016, Cheung et al., 2020] or assuming periodic shifts [Chandak et al., 2020a,b, Padakandla et al., 2020]. There has also been some work in the non statioati multi-armed bandit setting [Garivier and Moulines, 2008, Koulouriotis and Xanthopoulos, 2008, Besbes et al., 2014]. The non-stationary rewards for the behavior, that decay over time. have been considered for the bandit setting, under rotting bandits [Levine et al., 2017, Seznec et al., 2019]; these algorithms do not obviously extend to the RI. setting.
Handling the Non-Stationarity in a Multi-prediction System
In this section, describe unified approach to handle non-stationarity in both the GVF and behavior learners, using successor features. We first discuss how to use successor features to learn under non-stationary cumulants, for prediction. Then we discuss using successor features for control, allows us to leverage this approach for non-stationary rewards for the behavior. We then discuss state-reweightings. and how mitigate stttsttttooitty due to a changing behavior.
4.1 Successor Features for Non-stationary Rewards
Successor features provide an elegant way to learn value functions under non-stationarity. The separation of learning stationary successor features and rewards enables more effective tracking of non-stationary rewards, as we explain in this section and formally prove in Section 5.
Assume that there is weight vector w' E Rd and features x(s,a) E IRd for each state and action (s, a) such that r(s, a) = (x(8, a): w*). Recursively define
(bss, a) = (x At) + (t+1
s, o]
<b( a) is called the successor features, the discounted cumulative sum of feature vectors, if we follow policy For vh, del (S,, A,) and x, def x(S,, A.), we can see Q(s a) (ıb(s, a), w*)
(#b(s, a), w") = Ex(,, w |S = A, = =a]+ E. (1 (Wh+11,
w //w.1////bbbb
8, A, a)
=r(s, a) + Ex(7++1 (x++1, w w')| = 8, A. --=00 =a= =a]+ E (7 2 w')[S, = 8, A: = a)
= =r(s, a) E2 S: 8. A, a] -- (2/44, w = 8, A, = a)
=
F. [r(s, + 7++1"++1 + +11417k+77++2 +... [S. = A. = a) = Q(+
If we have features x(s, a) E Rd which allow us to represent the immediate reward, then successor features provide a good representation to approximate the GVF. We simply learn another set of parameters W. E I that predict the immediate cumulant (or reward): c(s,a) s (x(s, a), we). These parameters w. are updated using a standard regression update, and Q(s, a) (vb(s, we).
The successor features (bs, a) themselves, however, also need to be approximated. In most cases, we cannot explicitly maintain separate (b(s, a) for each a), outside of the tabular setting. Notice that each element in 1b(s, a) corresponds to a true expected return: the cumulative dis- counted sum of a reward feature into the future. Therefore, (sss, a) can be approximated using any value function approximation method, such as temporal difference (TD) learning. We learn parameters we for the approximation (b,s, W )= [&9] (s, a: w.)...., Lba(s, wo)] E Rd where (s, a; w&) uin a). We can use any function approximator for L6ss, a; we), such as linear function approximation with tile coding with We, linearly weighting the tile coding features to produce (s, a; we), neural networks, where we are the parameters of the neural network.
We summarize the algorithm using successor features for non-stationary rewards/cumulants, called SF-NR, in Algorithm 2. We provide an update formula for the approximate SF using Expected Sarsa for prediction [Sutton and Barto, 2018] for simplicity, but note that any value learning algorithm can be used here. In our experiments, we use Tree- Bree-Bacuup [Precup, 2000] because it reduces variance

---

from off- off-policy learning; we provide the pseudocode in Appendix D. Algorithm 2 assumes that the reward features x(s, a) are given, but of course these can be learned well. Ideally, we would learn compact set of reward features that provide accurate estimates as linear function of these reward features. A compact (smaller) set of reward features preferred because it makes the SE more computationally efficient to learn.
Algorithm 2 Successor Features for Non-stationary Rewards (SF-NR) Wo. We
There are two key advantages from the separation into learning successor features and immediate cumulant estimates. First, it easily allows different or chang- ing cumulants to be used, for the same policy, using the same successor features. The transition dynamics summarized in the stationary successor features can be learned slowly to high accuracy and re-used. This re-use property is why these representations have been used for transfer [Barreto et al., 2017, 2018, 2020]. This property is pertinent for us, because it allows us
S1 x t x(S,, A.)
2b + (sS Ae; Wp))
eb' En Ta (aallS (alt:+1))(((((11t1 (w)
A <
for m 1 to dl do
dm f Xm + 7+ B Fm
A x- A + 5m Vabm
to more easily track changes in the cumulant. The re-
W + as
gression updates can quickly update the parameters W.. and exploit the already learned successor features to more quickly track value estimates. Small changes in the rewards can result in large changes in the values; without the separation, therefore, it can be more difficult to directly track the value estimates.
W. - W. + a(C++1 (x))
Second, the separation allows us to take advantage of online regression algorithms with strong convergence guarantees. Many optimizers and accelerations are designed for a supervised setting, rather than for temporal difference algorithms. Once the successor features are learned, the prediction problem reduces to a supervised learning problem. We can therefore even further improve tracking by leveraging these algorithms to learn and track the immediate cumulant. We formalize the convergence rate improvements, from this separation, in Section 5.
4.2
GPI with Successor Features for Control
In this section we outline a control algorithm under non-stationary rewards. SF-NR provides method for updating the value estimate due to changing rewards. The behavior for the multi-prediction problem has changing rewards, and so could benefit from SF-NR. But SF-NR only provides a mechanism to efficiently track action-values for a fixed policy, not for a changing policy. Instead, we turn to the idea of constraining the behavior to act greedily with respect to the values for a set of policies, introduced as Generalized Policy Improvement (GPI) Barreto al. [2018, 2020].
For our system, this is particularly natural, as we are already learning successor features for a collection of policies. Let us start there, where assume our of policies is IT (-.,,-.,, NN). Assume also that we have learned the successor features for these policies, (5(s, a; w!)) (i)), and that we have weights 0, E Rd such that (x(s, a), 0,) x E[Re+1 [S 8, A, a] for behavior reward Re+1- Then on each step, the behavior policy takes the following greedy action
(as) argmax max JE(1....,N
Q60 a) argmax max (1b(s, a; w ).)))
a
a
jE(1....,N)
The resulting policy guaranteed to be an improvement: in every state the new policy has a value at least as good as any of the policies in the set [Barreto et al., 2017, Theorem IJ. Later work also showed sampled efficiency of GPI when combining known reward weights to solve novel tasks [Barreto et al., 2020].
The use of successor features has similar benefits as discussed above, because the estimates can adapt more rapidly as the rewards change, due to learning progress changing over time. The separation even more critical here, as we know the rewards are constantly drifting, and tracking quickly is even more critical. We could even more aggressively adapt to these non-stationary rewards, by anticipating trends. For example, instead of a regression update, we can model the trend (up or down) in the reward for a state and action. If the reward has been decreasing over time, then likely it will continue to decrease. Stochastic gradient descent will put more weight on recent points, but would likely predict a higher expected reward than is actually observed. For simplicity here, we still choose to use stochastic gradient descent, as it is reasonably effective tracking algorithm, but note that performance improvements could likely be obtained by exploiting this structure in this problem.
5

---

We can consider different set of policies for GVFs and behavior. However, the two are naturally coupled. First, the GPI theory shows that greedifying over a larger collection of policies provides better policies. It is sensible then to at least include the GVF policies into the set for the behavior. Second, the behavior needs to learn the successor features for the additional policies. Arguably, should try to gather data to learn these well, so as to facilitate its own policy improvement. It should therefore also incorporate the learning progress for these successor features, into the intrinsic reward. For this work, therefore, we assume that the behavior uses the set of GVF policies. Note that the weight change intrinsic reward uses the concatenation of was and We.
4.3 Interest and prior corrections for the changing state distribution
The final source of non-stationarity is in the state distribution. As the behavior μ changes, the state-action visitation distribution d,,, S x + [0. 1] changes. The state distribution implicitly weights the relative importance of states in the GVF objective, called the projected Bellman error (PBE). Correspondingly, the optimal SF solution could be changing, since the objective changing. The impact of a changing state-weighting depends on function approximation capacity, because the weighting indicates how to trade-off function approximation error across states. When approximation error is low zero-suruee -uch as in the tabular setting- the weighting has little impact the solution. Generally, however, we expect some approximation error and so a non-negligible impact.
We can completely remove this source of non-stationary by using prior corrections. These are products of importance sampling ratios, that reweight the trajectory to match the probability of seeing that trajectory under the target policy т. Namely, it modifies the state weighting to d,, the state-action visitation distribution under т. We explicitly show this in Appendix C. Unfortunately, prior corrections can be highly problematic in a system where the behavior policy takes exploratory actions and target policies are nearly deterministic. is likely that these corrections will often either be zero, or near zero, resulting in almost no learning.
To overcome this inherent difficulty, we restrict which states are important for each predictive question. Likely, when creating a GVF, the agent is interested in predictions for that GVF only in certain parts of the space. This is similar to the idea of initiation sets for options, where an option is only executed from small set of relevant states. We can ask: what is the GVF answer, from this smaller set of states of interest? This can be encoded with a non-negative interest function, i(ss a), where some (or even many) states have an interest of zero. This interest is incorporated into the state-weighting in the objective, so the agent can focus function approximation resources on states of interest.
When using interest, it is sensible to use emphatic weights [Sutton et al., 2016]. Emphatic weightings are a prior correction method, used under the excursions model [Patterson et al., 2021]. They reweight to a discounted state-action visitation under т when starting from states proportionally to d,. Further, they ensure states inherit the interest of any states that bootstrap off of them. Even if a state has an interest of zero, we want to accurately estimate its value if an important states bootstraps off of its value. The combination of interest and emphatic weightings- -whic shift state-action weighting to visitation under r-means that we mitigate much of the non-stationarity in the state-action weighting. We provide the pseudocode for this Emphatic TB (ETB) algorithm in Appendix D.
5 Sample Efficiency of SF-NR
As suggested in Section 4.1, the use of successor features makes SF-NR particularly well- to our multi-prediction problem setting. The reason for this is simple: given access to an accurate SF matrix, value function estimation reduces to a fundamentally simpler linear prediction problem. Indeed, access to an accurate SF enables one to sidestep known lower-bounds on PBE estimation.
For simplicity, we prove the result for value functions; the result easily extends to action-valle Denote by v E IR S) the vector with entries v" (s), r E IR|S| the vector of expected immediate rewards in each state, and P E IR x(S| the matrix of transition probabilities. The following lemma, proven in Appendix 1. relates mean squared value error (VE) to one-step reward prediction error.
Lemma Assume there exists a w" E Rd such that r = Xw". Let i def Xw for some w E TRdd, and let D= Diag( (d((s) res) for distribution d fully supported on S, with ||.I|D the weighted norm
under D. Then the value estimate v dd v/w satisfies 1 ][lv $13 | -

---

Thus we can ensure that VE(v", v) I E by ensuring that ||r- S <(11))? This promising, as this latter expression readily expressed as the objective of linear regression problem. To illustrate the utility of this, let's look at a concrete example: suppose the agent has an accurate SF matrix Vi and that the reward function changes at some point in the agent's 's deployment. Suppose access to a batch of transitions DD (St, Ar, St, rt, Tt, )= with which we can correct our estimate of v", where each (8, (, s', E D such that 8 ~ du for some known behavior policy μ4, A, ~ T (-16), 8' ~ P(-]s, a) and r= r(s, a, s'). Assume for simplicity that Pt S Pmax [1o(S))ll S T,, rz S Rmax for some finite Pmax , Rmax L E Rt. Then we can get the following result, proven in Appendix A.2, that is a straightforward application of Orabona [2019, Theorem 7.26].
(r: - (a((S),, w)) Suppose we apply basic recursive least- squares estimator to minimize regret on this loss sequence, producing a sequence of iterates wt- Let Et= We denote the average iterate. For i(s) (s)(s), Tr) we have that
Proposition 1 Define f<(w)
dí 2
WT def
T
[Pmaax R log (1 + Pmnx L
|| vIB SO
- (1-
In contrast, without the SF we are faced with minimizing harder objective: the PBE. It can be shown that minimizing the PBE is equivalent to stochastic saddle-poin problem, and the convergence to the saddle-point of this problem has unimprovable rate of O the maximum eigenvalue of the covariance matrix and a bounds gradient stochasticity, and this conver- (V - + (1+r)) 1.dd
f + [1+>) l.dd
VT where is
gence rate translates into the performance bound |/ - vI < O
[Liu et al., 8a, Proposition 5]. Comparing with Equation 2, we observe an additional dependence of (VFTT) as well as the worse dependence of at least O(1/vT) 2 (log (T) /T) on all other
quantities of interest, reinforcing the intuition that access to the SF enables us to more efficiently re-evaluate the value function.
A First Experiment Testing the Multi-prediction System
In this section, we investigate the utility of using SF- NR under non-stationary cumulants and rewards, both for prediction and control. We conduct the experiment in a TMaze environment, inspired by the environments used to test animal cognition [Tolman and Honzik, 1930]. The environment, depicted in Figure 1, has four GVFs where each policy takes the fastest route to its corresponding goal. The cumulants are zero everywhere except for at the goals. The cumulant can be of three types: a constant fixed value (constant), a fixed-mean and high variance value (distractor), or a non-stationary zero-mean random walk process with low variance (drifter). Exact formulas for these cumulants are in Appendix E.
G1
Dritter
Constant
Distrnction
Timeslep Figure Tabular TMaze with GVFs, with cumulants of zero except in the goals. The right plot shows the cumulants in the goals. G2 and G4 have constant cumu- lants, G1 has a distractor cumulant and G4 a drifter.
Utility of SF-NR for Fixed Behavior Policy
We start by testing the utility SF-NR for GVF learning, under a fixed policy that provides good data coverage for every GVF. The Fixed-Behavics Policy is started from random states in the TMaze, and moves towards the closest goal, with a 50/50 chance of going either direction if there is a tic. This policy is like round robin policy, in that one of the GVF policies is executed each episode and, in expectation, all four policies are executed the same number of times.
We compare an agent that uses SF-NR and one that learns the approximate GVFs using Tree Back- (TB). TB is an off-policy temporal difference (TD) algorithm, that reduces variance in the eligibility trace. We also use TB to learn the successor features in SF-NR. Both use À = 0.9 and stepsize method called Auto [Mahmood et al., 2012] designed for online learning. We sweep the initial stepsize and meta stepsizes for Auto. For further details about the agents and optimizer, see Appendix D. We additionally compare to least squares TD (LSTD), with = 0.9, particularly as computes a matrix similar to the SF. but does not separate out cumulant learning (sce Appendix B for this connection).
7

---

Dellle
#(Sarsa) (IDD)
Freciar of Goul Valta
Faerbe of Vetts
A/Moedd, ((UST
2
Dealier
H(Tidd), +(78)
Distactoe
P(GPD) (SR)
Sanna),
Contzers 200
Steps 1000)
(per 1000)
EpGade Count
(a) Fixed-Behavior Policy
(b) Learned Behavior Policy
(c) Visitation Plot Comparison
Figure 2: Performance in Tabular TMaze, with averages over 30 runs. (a) and (b) show average off- policy prediction RMSE, with standard errors, where the error is weighted by (a) the state distribution dy for the Goal visitation plots for GPI with SF and TB.
policy and (b) uniform state weighting when learning the behavior. (c)
In Figure 2a, we can see SF-NR allows for much more effective learning, particularly later in learning when it more effectively tracks the non-stationary signals. LSTD performs much more poorly, likely because it corresponds to closed-form batch solution, which uses old cumulants that are no longer reflective of the current cumulant distribution.
Investigating GPI for Learning the Behavior
Next we investigate if SF-NR improves learning of the whole system, both for the GVFs and for the behavior policy. We use SF-NR and TB for the GVF learners, and Expected Sarsa (Sarsa) and GPI for the behavior. The GPI agent uses the GVF policies for its set of policies. The reward features for the behavior are likely different than those for the GVF learners, because the cumulants are zero in most states whereas intrinsic rewards are likely non-zero in most states. The GPI agent, therefore, learns its own SFs for each policy, also using TB. The reward weights that estimate the (changing) intrinsic rewards are learned using Auto, share the same meta-step namely only one shared parameter is swept.
the SFs. Note that the behavior and GVF learners all
The results highlight that SF for the GVFs is critical for effective learning, though GPI and Sarsa perform similarly, as shown in Figure 3b. The utility SF even greater here, with TB GVF learners inducing much worse performance than SF GVF learners. GPI and Sarsa are similar, which is likely due to the fact that Sarsa uses traces with tabular features, which allow states along the trajectory to the drifter goal to update quickly. In following sections, we find a bigger distinction between the two.
We visualize the goal visitation of GPI in Figure 2c. Once the GVF learners have a good estimate for the constant cumulant signals and the distractor cumulant signal, the agent behavior should switch to visiting only the drifter cumulant as that is the only goal where visiting would improve the GVF prediction. When using SF GVF learners, this behavior emerges, but under TB GVF learners the agent incorrectly focuses on the distractor. This is even more pronounced for Sarsa (see Appendix F.2).
Experiments under Function Approximation
We evaluate our system in a similar fashion to the last section, but now under function approximation. We use a benchmark problem at the end of this section, but start with experiments in the TMaze mod- ified to be continuous environment, with full details described in Appendix 1. The environment observation o, E IR22 corresponds to the xy coordinates of the agent. We use tile coded features of tilings of 8 tiles for the state representation, both for TB and to learn the SF.
The reward features for the GVF learners can be much simpler than the state-action features because they only need to estimate the cumulants, which are zero in every state except the goals. The reward features are a one-hot encoding indicating if in the tuple of a, is in the pseudo-termination goals of the GVFs. For the Continuous TMaze, this gives a 4 dimensional vector. The reward features for GPI is state aggregation applied along the one dimensional line components. Appendix E. contains more details on the reward features for the GVF and behavior learners.
Results for a Fixed Behavior and Learned Behaviors
Under function approximation, SF-NR continues to enable more effective tracking of the cumulants than the other methods. For control, GPI is notably better than Sarsa, potentially because under
8

---

o(Moed) FULSTO)
(SR))
RASE
MIGPD FTTD
RANSE
M(Preed) m7
p(Saroo) misnt
Piaac (red))
pfPPeed v/S10
Steps (per 1000)
Steps (per 1000)
Steps (per 1000)
(a) Exeed-Behavior Policy
(b) Learned Behavior Policy
(c) Replay with Fixed-Behaad Behavior
Figure 3: Performance in Continuous TMaze, with averages over 30 runs. (a) and (b) show average off-policy prediction RMSE, with standard errors, where the error is weighted by (a) the state distribution d for the Fixed-Behavior policy and (b) uniform state weighting when learning the behavior. (c) RMSE in Continuous TMaze with a Fixed Behavior when incorporating replay.
function approximation eligibility traces are not as effective at sweeping back changes in behavior rewards and so the separation is more important. We include visitations plots in Appendix F.1, which are similar to the tabular setting.
Note that the efficacy of SF-NR and GPI relied on having reward features that did not overly generalize. The SF learns the expected feature vector when following the target policy. For the GVF learners, states on the trajectory share features with states near the goal, then the value estimates will likely be higher for those states. The rewards are learned using squared error, which unlike other losses, is likely only bring cumulant estimates near zero. These small non-zero cumulant estimates are accumulated by the SF for the entire trajectory, resulting in higher error than TB. We demonstrate this in Appendix F.3. We designed reward features to avoid this problem for experiments, knowing that effective reward features can and have been learned for SF [Barreto et al., 2020].
Results using Replay
The above results uses completely online learning, with eligibility traces. natural question is if the more modern approach of using replay could significantly change the results. In fact, early versions of the system included replay but had surprisingly negative results, which we later realized was due to the inherent non-stationarity in the system. Replaying old cumulants and rewards, that have become outdated, actually harms performance of the system. Once have the separation with the SF, however, we can actually benefit from replay for this stationary component.
We demonstrate this result in Figure 3c. We use À 0 for this result, because settings are otherwise the same as above, and we resweep hyperparameters for this experiment. SF-NR benefits from replay, because it only uses it for its stationary component: the SF. TB, on the other hand, actually performs more poorly with replay. As before, LSTD which similarly uses old cumulants, also performs poorly.
replay. The
Incorporating Interest
13
([Saran) w/ Interest)
To study the effects of interest, more open world environment needed. The Open 2D World used analyze this problem as described in Appendix E.2. At the start of each episode, the agent begins in the center of the environment. The interest for each GVF in the states is one if the state is in the same quadrant as the GVF's respective goal, and zero otherwise. This enables the GVFs to focus their learning on subset of the entire space and thus use the function approximation resources more wisely and give a better weight change profile an intrinsic reward to the behavior learner. Each GVF prediction i is evaluated under is standard error over 30 runs. state-action weighting induced by running Tú with results in Figure 4.
12
Sarsa
SRR w/ ETB)
RMSE
of GGP), F(SR)
70
50
Steps (per 1000)
Figure 4: Using interest: shading
Both TB with interest and ETB reweight states to focus more on state visitation under the policy. Both significantly improve performance over not using interest, both allowing faster learning and reaching a lower error. The reweighting under ETB more closely matches state visitation under the policy, and accounts for the impacts of bootstrapping. We find that ETB does provide some initial learning benefits. The original ETD algorithm known to suffer from variance issues; we may find with variance reduction that the utility of ETB is even more pronounced.

---

Loft Hill Pscurloterm nntionnion
L/40)
Right Hill Pstudotermintton
100
1500
RMSE
RRndomm s(SR)
12500
Jr (Srrra) x(SR)
100
64
P(CPI), m(SR)
Steps (per 1000)
S0 9330
(nhundonnt w151)
(a) RMSE for each GVF
(b) GVF#2 goal visits: left hill top (c) GVF#1 goal visits: right hill top
Figure 5: Performance in Mountain Car averaged over 30 runs, with standard errors. (a) Learning
curves for RMSE, with uniform weighting over states and actions. (b), (c) show the number of times that the agent reached the termination for each GVF.
Validation of the Multi-Prediction System a Standard Benchmark Problem
Finally, we investigate multi-prediction learning in environment not obviously designed for this setting: Mountain Car. The goal here is to show that multi-prediction learning is natural in many problem settings, and to show results in standard benchmark not designed for our setting that has more complex transition dynamics. In the usual formulation the agent must learn to rock back and forth building up momentum reach the top of the hill on the right-a classic cost to goal problem. This is hard exploration task where a random agent requires thousands of steps to reach the top of the hill from the bottom of the valley. Here we use Mountain Car about more than just getting out of the valley quickly. We specified a GVF whose termination and policy focuses reaching top of the left hill, and second GVF about reaching the top of the other side. The full details of the GVFs and setup of this task can be found in the Appendix E.3.
if our approach can learn
Figure 5a shows how GPI and Sarsa compare against baseline random policy. GPI provides much better data for GVF learning than the random policy and Sarsa, significantly reducing the RMSE of the learned GVFs. The goal visitation plots show GPI explores the domain and visits both GVFs goal far more often than random, and more effectively than Sarsa.
8 Conclusion
In this work, we take the first few steps towards building an effective multi-prediction learning system. We highlight the inherent non-stationarity in the problem and design algorithms based on successor features (SF) to better adapt to this non-stationariit. We show that (1) temporally consistent behavior emerges from optimizing the amount of learning across diverse GVF questions; (2) successor features are useful for tracking nonstationary rewards and cumulants, both theory and empirically; (3) replay is well suited for learning the stationary components successor features while meta-learning works well for the non-stationary components; and (4) interest functions can improve the performance of the entire system, by focusing learning to subset of states for each prediction.
Our work also highlights several critical open questions. (1) The utility of SFs is tied the quality of the reward features; better understanding of how to learn these reward features is essential. (2) Continual Auxiliary Task Learning is an RL problem, and requires effective exploration approaches to find and maximize intrinsic rewards the intrinsic rewards do not provide a solution to exploration. Never-ending exploration is needed. (3) The interaction between discovering predictive questions and learning them effectively remains largely unexplored. In this work, we focused for a given set of GVFs. Other work has focused on discovering useful GVFs [Veeriah et al., 2019, 2021, Nair et al., 2020, Zahavy et al., 2021]. The interaction between the two is likely to introduce additional complexity in learning behavior, including producing automatic curricula observed in previous work [Oudeyer et al., 2007, Chentanez et al., 2005].
learning,
This work demonstrates the utility of several new ideas in RL that are conceptually compelling, but not widely used in RL systems, namely SF and GVFs, GPI with SF for control, meta- meta-descent step-size adaption, and interest functions. The trials and tribulations that lead this work involved many failures using classic algorithms in RL, like replay; and, in the end, providing evidence for utility in these newer ideas. Our journey highlights the importance of building and analyzing complete RI systems, where the interacting parts- paths-witth different timescales of learning and complex -eeeniiisnnennesttttteee
interdependencies necessitate incorporating these conceptually important ideas. Solving these integration problems represents the next big step for RL research.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by NSERC Discovery, IVADO, CIFAR through CCAI Chair funding and by the Alberta Machine Intelligence Institute (Amii).
References
Sherief Abdallah and Michael Kaisers. Addressing environment non-stationarity by repeating q-learning updates. The Journal of Machine Learning Research, 17(1):1822-1612 2016.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, 2017.
Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference on Learning Representations, 2020.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference Machine Learning, 2018.
Andre Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan hunt, Shibl Mourad, David Silver, and Doina Precup. The option keyboard: Combining skills in reinforcement learning. In Advances in Neural Information Processing Systems, 2019.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 117 (48):30079-3000) -00777 2020.
Omar Besbes, Yonatan Gur, and Assaf Zeevi. Stochastic multi-armed- problem with non- stationary rewards. In Advances in Neural Information Processing Systems, 2014.
Diana Borsa, André Barreto, John Quan, Daniel Mankowitz, Rémi Munos, Hado van Hasselt, David Silver, and Tom Schaul. Universal successor features upproximators. In International Conference on Learning Representations, 2019.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2019.
Yash Chandak, Scott M Jordan, Georgios Theocharous, Martha White, and Philip S Thomas. Towards safe policy improvement for non-stationary mdps. Advances Neural Information Processing Systems, 2020a.
Yash Chandak, Georgios Theocharous, Shiv Shanka, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps. International Conference on Machine Learning, 2020b.
Nuttapong Chentanez, Andrew Barto, and Satinder Singh. Intrinsically motivated reinforcement learning. In Advances in Neural Information Processing Systems, 2005.
Wang Chi Cheung, David Livchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. International Conference on Machine Learning, 2020.
Cédric Colas, Pierre Fournier, Mohamed Chetouani, Olivier Sigaud, and Pierre- Yies Oudeyer. Curious: intrinsically motivated modular multi-goal reinforcement learning. International conference on machine learning, 2019.

---

Bruno C Da Silva, Eduardo W Basso, Filipo S Perotto, Ana L C. Bazzan, and Paulo M Engel. Improving reinforcement learning with context detection. In Autonomous agents and multiagent systems, pages 810-812, 2006.
Benjamin Eysenbach. Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representutions, 2019a.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In Advances in Neural Information Processing Systems, 2019b.
Aurélien Garivier and Eric Moulines. On upper-confidence bound policies for non-stationary bandit problems. arXiv preprint MxXiv:0005.3415, 2008.
Sina Ghiassian, Andrew Patterson, Martha White, Richard $ Sutton, and Adam White. Online off-policy prediction. arXiv preprint arXiv: 0ee.11//1..0299 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. Interational
Conference on Learning Representations Workshop, 2017.
Assaf Hallak and Shie Mannor. Consistent on-line off-policy evaluation. International Conference on Machine Learning. 2017.
Andrew Jacobsen, Matthew Schlegel, Cameron Linke, Thomas Degris, Adam White, and Martha White. Meta-descent for online, continual prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
Max Jaderberg, Volodymyr Mnih, Wojeiech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. In International Conference Learning Representations, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International
Conference on Learning Representations, 2015.
Dimitris E Koulouriotis and Xanthopoulos. Reinforcement learning and evolutionary algorithms for non-stationary multi-armed bandit problems. Applied Mathematics and Computation, 196(2): 913-922, 2008.
Nir Levine, Koby Crammer, and Shie Mannor. Rotting bandits. Advances in Neural Information Processing Systems, 2017.
Cam Linke, Nadia M Ady, Martha White, Thomas Degris, and Adam White. Adapting behavior via intrinsic reward: survey and empirical study. Journal of Artificial Intelligence Research, 69: 1287-1332, 2020.
Bo Liu, Ian Gemp, Mohammad Ghavamzadeh, Ji Liu, Sridhar Mahadevan, and Marek Petrik. Proximal gradient temporal difference learning: Stable reinforcement learning with polynomial sample complexity. Journal of Artificial Intelligence Research, 201 8a.
Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou, Breaking the Curse of Horizon: Infinite- Horizon Off-Policy Estimation. Advances in Neural Information Processing Systems, 2018b.
Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-Policy Policy Gradient with Stationary Distribution Correction. Uncertainty in Artificial Intelligence, pages 1180-1190, 2020.
Marlos C Machado. Mare G Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. In International Conference on Machine Learning, 2017.
Ashique Mahmood, Huizhen Yu, and Richard Sutton. Multi-step off off-policy learning without impor- tance sampling ratios. 02 2017.
Ashique Rupam Mahmood, Richard S Sutton, Thomas Degris, and Patrick M Pilarski. Tuning- free step-size adaptation. In IEFE International Conference on Acoustics, Speech and Signal Processing, 2012.
12

---

Ashvin Nair, Shikhar Bahl, Alexander Khazatsky, Vitchyr Pong. Glen Berseth, and Sergey Levine. Contextual imagined goals for self-supervised robotic learning, Conference on Robot Learning. 2020.
Francesco Orabona. A modern introduction to online learning, 2019.
Pierre-Yves Oudeyer, Frédéric Kaplan, and Verena V. Hafner. Intrinsic motivation systems for
autonomous mental development. In IEEE Transactions on Evolutionary Computation, 2007.
Sindhu Padakandla, KJ Prabuchandran, and Shalabh Bhatnagar. Reinforcement learning algorithm for non-stationary environments. Applied Intelligence, 50(1 (0(11):3500-36 -366, 2020.
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driver exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
Andrew Patterson, Adam White, Sina Ghiassian, and Martha White. generalized projected bellman error for off-policy value estimation in reinforcement learning. arXiv preprint arXiv:21044 102.100411100 13844, 2021.
Karl Pertsch, Oleh Rybkin, Frederik Ebert, Shenghao Zhou, Dinesh Jayaraman, Chelsea Finn, and Sergey Levine. Long-horizer visual planning with goad-conditioned hierarchical predictors. In Advances in Neural Information Processing Systems. 2020.
Silviu Pitis, Harris Chan, Stephen Zhao, Bradly Stadie, and Jimmy Ba. Maximum entropy gain exploration for long horizon multi-goal reinforcement learning. In International Conference on Machine Leanning 2020.
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew-fit: State-covering supervuue reinforcement learning. arXiv preprint
2019.
Doina Precup. Eligibility traces for uff-policy policy evaluation. Computer Science Department Faculty Publication Series, page 80, 2000.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Wiele, Vlad Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. In International Conference on Machine Learning. 2018
Tom Schaul and Mark B Ring. Better generalization with forecasts. In International Joint Conferences on Artificial Intelligence, 2013.
Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International Conference on Machine Learning, 2015.
Bruno Scherrer. Improved and generalized upper bounds on the complexity of policy iteration.
Mathematics of Operations Reseurch, 2016.
Matthew Schlegel, Andrew Jacobsen, Zaheer Abbas, Andrew Patterson, Adam White, and Martha White. General value function networks. Journal of Artificial Intelligence Research, 70:497-543,
2021.
Julien Seznec, Andrea Locatelli, Alexandra Carpentier, Alessandro Lazaric, and Michal Valko, Rot- ting bandits are no harder than stochastic ones. International Conference on Artificial Intelligence and Statistics, 2019.
Bradly C Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint 1rXiv:1507.00814, 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018 ISBN 0262039249.
Richard Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsuper- vised sensorimotor interaction. In International Conference on Autonomous Agents and Multiagent Systems, 2011.
13

---

Richard S Sutton, A Rupam Mahmood, and Martha White. An emphatic approach to the problem of off policy camporal-different learning. The Journal of Machine Learning Research, 17(1): 2603-2631, 2016.
Edward Chace Tolman and Charles H Honzik. Introduction and removal of reward, and maze performance in rats. University of California publications in psychology, 1930.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Janarthanan Rajendran, Richard L. Lewis, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions auxiliary tasks. In Neural Information Processing Systems, 2019.
Vivek Veeriah, Tom Zahavy, Matteo Hessel, Zhongwen Xu, Junhyuk Oh, Iurii Kemaev, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of options via meta-learned subgoals. In Advances in Neural Information Processing Systems, 2021.
Martha White. Unifying task specification in reinforcement learning. In International Conference on Machine Learning, 2017.
Tom Zahavy, Brendan DDnoghuue, Andre Barreto, Volodymyr Mnih. Sebastian Flennerhag, and Satinder Singh. Discovering diverse nearly optimal policies with successor features. arXiv preprint arXiv:2106.00669, 2021.
Tianren Zhang, Shangqi Guo, Tian Tan, Xiaolin Hu, and Feng Chen. Generating adjacency- constrained subgoals in hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
14


Reviews:
[{"review_id": "yiouQBub3MD", "reviewer": "Reviewer_GBFf", "summary": "This work addresses the multi-prediction problem in MDPs, with key focus on how to generate and adapt the behavior to optimize auxiliary predictions in the absence of task rewards.  They show that the rewards for the behavior have inherent non-stationarity. The proposed method approximates successor features (via expected SARSA and Tree Backup) and then uses them to learn value predictions (via regression updates). This specific separation allows for a faster convergence rate as proved by the authors.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**:  \n\nThe work addresses an important problem of learning how to generate behavior such that multiple predictions can be learned, which remains under-explored in the literature.  The approach being fully online adds an additional challenge to tackle under non-stationarity - this is an important contribution. With theoretical and practical considerations, the proposed method seems an effective strategy to learn multiple predictions. This is a very valuable contribution as a lot of the existing literature has explored learning multiple predictions without paying much heed towards learning the behavior policy itself to be able to result in meaningful multiple predictions. While the paper can be significantly improved in writing, the contributions outweigh its flaws.\n\n**Weaknesses**:\n* One of the primary limitations of this work is that the cumulants are hand-specified and not being learned or discovered. There is no discussion about this. Any insights would be useful to add. For example how does the work on discovering useful cumulants might tie to the ideas presented in this paper for overcoming user designed cumulant equations?\n* The phrasing of the paper title is misleading. The word continual learning is typically used in the context of lifelong learning or never ending learning (also referred to as continual learning). But what the authors really imply here is online learning (e.g “continually interacts with the environment”), which warrants the current title misplaced and not suitable for this work. I would recommend updating it to reflect more clearly what the paper addresses. \n \n\n**Empirical Analysis**: \nThe empirical analysis provided support for the claims. The experiments cover a range of tasks. However, I have the following questions:  \n\n* In the current pipeline, the value estimates of the prediction have a huge dependence on the accurate learning of the successor features. What happens if the SF is not well learned? Would the method still work?\n* How does the change in weights impact the scalability of the algorithm? A potential failure case is that while the weights have changed a lot but nothing meaningful has been learned. How would one interpret this in the context of the proposed approach?\n* Besides, it would be interesting and useful to show the failure cases for the proposed ideas in general. Please add additional analysis if any towards this.\n* An interesting baseline would be an agent such as Horde where the behaviour policy is not being learned or adapted smartly. Would it be a good idea to add this? Any insights from the authors on this would be appreciated. \n \n\n**Writing and Presentation**:  \n* The paper is very well motivated. \n* The experiments section is less clear as compared to the other sections. It might be useful to revisit it to ensure better understanding.\n* The checklist has many links missing which show as ?\n* L359- the above results are obtained by learning online.\n* L375 fourth → forth\n* I recommend at least one pass of proofread to check the paper for typos and minor writing errors. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss limitations. I recommended adding this with the conclusion discussing failure modes and future work. As it stands, nothing along these lines have been covered in the manuscript.   \n\nI did not see a broader impact statement, although the work is abstract enough to have any direct concerns.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ozeWAMvGWoi", "reviewer": "Reviewer_WNTT", "summary": "This paper studies the continual RL setting in which an agent behaves according to a policy designed to improve predictions on a number of auxiliary tasks. It proposes an algorithm based on successor features that facilitates tracking under non-stationary rewards, demonstrating both empirical and theoretical benefits of this approach. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an important problem, namely that of learning about the environment without external rewards, in a very challenging setting, namely continual RL with inherent non-stationary for both the prediction learners and the behavior learner. The paper is well written, it introduces a new problem formulation, and proposes a new method, which is thoroughly evaluated empirically and analyzed from a theoretical perspective. I think this paper could be of value for the community and open up multiple directions for future work. \n\nI particularly appreciated that the authors report results over 30 different runs of the models, which increases confidence in the drawn conclusions. I also liked the fact that the experiments were designed in simple environments, where it is easier to understand the strengths and limitations of these methods. \n\nOne limitation of this work is that the approach seems quite similar to prior work which uses successor features for transfer to new tasks (e.g. Barreto et al. 2017, Barreto et al. 2018 Borsa et al. 2018, Ma et al. 2020). Hence, the algorithmic contribution is not particularly novel, but I don't think this should prevent acceptance given that the empirical evaluation is quite thorough, the problem setting is novel and important, and the results look promising. However, I do think the authors should discuss in greater detail how the approach differs from prior work. \n\nAnother potential limitation of this work is the reliance upon the choice of policies / tasks to learn. I think it would be valuable to analyze or discuss how the algorithm depends on this choice and how one should choose the set of auxiliary tasks or policies. What if one doesn't have access to such tasks or policies -- how could these be discovered so that they are (most) useful for learning about the environment or downstream tasks without prior knowledge? How could one combine this with skill discovery or other related research directions?\n\nGiven all the above, I recommend acceptance for this paper, but hope the authors can take into account the feedback to further improve the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "FA3pGAx32_", "reviewer": "Reviewer_NevW", "summary": "This paper considers the setting where an RL agent has to learn to make auxiliary predictions, and captures each prediction task as an RL task. In particular, the goal is to maximize the learning progress for each prediction task. Specifically, this is measured by the l1-norm of the change in weights and introduced as an intrinsic reward.\n\nHowever, this multi-prediction problem is difficult because of multiple sources of non-stationarity, i.e., from the changing rewards (which shrink as the agent makes better predictions) and from the changing agent behavior. To address this non-stationarity, the successor features framework is leveraged. That is, they can use learned successor features to efficiently track the non-stationary rewards and to correct state distribution changes by importance sampling.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- The use of successor features to track and handle non-stationarity is novel. Applying this to efficiently learn multiple auxiliary tasks is also new.\n\n- The introduction also provides a good overview of the related work in the literature, and describes how the ideas in this paper relate to the prior work.\n\nQuality:\n- The paper provides a theoretical analysis of the sample efficiency and shows that it is better compared to standard algorithms.\n\n- The algorithm is evaluated in a simple maze environment and in the Mountain Car task, which are pretty simple RL tasks. Learning multiple auxiliary tasks in simpler environments may be more inefficient than solving a main task, when given one. I’m curious how well this approach scales to more complex environments.\n\n- The number of auxiliary tasks studied in the experiments is also quite small (4 in the maze setting and 2 in Mountain Car). It’d be nice to study how a larger number of auxiliary tasks impacts learning.\n\nClarity:\n- Using the change in the weights as the RL reward seems to introduce an unnecessary source of non-stationarity. In the settings studied in the experiments, the auxiliary tasks involve reaching specific goal states. Could the intrinsic reward instead be defined in terms of reaching these goals which would then give stationary reward functions?\n\n- The different parts of Section 4 currently read a bit disconnected and could be better organized. Including a unified algorithm box would be helpful. Also, are the weights of the GVF learners the same weights as in Alg. 2?\n\n- There is a lot of notation and terms that get abbreviated, which gets difficult to keep track of. A table that summarizes all of the notation and abbreviations would be helpful.\n\nSignificance:\n- Learning a set of auxiliary tasks in the absence of external rewards ideally allows for efficient transfer when an external reward is actually specified, and is hence an important problem setting to study. \n\n- The successor features framework allows for transfer between these auxiliary reward functions, which are assumed to be linear in some features. Is this a limiting assumption when we might want to consider more complex auxiliary tasks that do not share a common feature set?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. See main review for other limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "BWcnrtV3vO", "reviewer": "Reviewer_8jZM", "summary": "The paper proposes a framework for continually learning multiple generalized value functions, where each value function is learned off-policy and the behaviour policy learns to maximize the learning progress. The paper identifies the non-stationarity of the action value function during the process of maximizing learning progress and proposes an algorithm that separately learns the immediate reward and the successor feature. The experimental results show that the proposed algorithm significantly improves sample efficiency and perfroms well with the use of replay.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n\nThe idea of using successor feature for MDPs with changing rewards are not new (Barreto et al.18'), but its application in learning auxiliary tasks is novel.\n\n### Quality\n\nThe motivation of addressing the non-stationary learning problem in using auxiliary tasks for exploration using successor feature (SF) is very convincing. The paper also shows theoretically that given accurate SF, better convergence can be achieved. The assumption of accurate SF fits the problem setup where rewards are stationary while the dynamics are static. The empirical experiments are also convincing to me.\n\n### Clarity\n\nThe paper is well written and easy to follow.\n\ntypo: line 277, equation 3 -> equation 2", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The formulation of a GVF question was confusing to me in the first read as it is described as a goal-conditioned value function in Sutton et al. 11'. It seems like the formulation in this paper follows Veeriah et al. 19'. it would be better to clarify this point.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have come to a strong consensus for acceptance of this work. Its strong points include novelty, where the important combination of multi-prediction and non-stationarity is well motivated, as well as pleasing theory and experiments to address the same. 

I would ask the authors to carefully consider the suggested writing improvements from GBFf and suggestions for clarity from NevW. I would also like to see additionally comentary about the number of tasks and the potential to discover "useful" tasks/skills in the context of your work (these are not things I expect a solution to today, but something where guidance will help the authors bound to study the topic in the future).
Decision: Accept (Poster)