Adaptive Interest for Emphatic Reinforcement Learning
Authors: Martin Klissarov
Rasool Fakoor
Jonas Mueller
Kavosh Asadi
Taesup Kim
Alex Smola
Venue: NeurIPS 2022 Accept (2022)
Keywords: emphatic temporal difference
interest function
meta gradients
meta learning

Abstract: Emphatic algorithms have shown great promise in stabilizing and improving reinforcement learning by selectively emphasizing the update rule. Although the emphasis fundamentally depends on an interest function which defines the intrinsic importance of each state, most approaches simply adopt a uniform interest over all states (except where a hand-designed interest is possible based on domain knowledge). In this paper, we investigate adaptive methods that allow the interest function to dynamically vary over states and iterations. In particular, we leverage meta-gradients to automatically discover online an interest function that would accelerate the agent’s learning process. Empirical evaluations on a wide range of environments show that adapting the interest is key to provide significant gains. Qualitative analysis indicates that the learned interest function emphasizes states of particular importance, such as bottlenecks, which can be especially useful in a transfer learning setting.

Adaptive Interest for Emphatic Reinforcement Learning
Martin Klissarov" Mila, McGill University
Rasool Fakoor Amazon Web Services
Jonas Mueller
Cleanlab
Kavosh Asadi Amazon Web Services
Taesup Kim
Alexander J. Smola Amazon Web Services
Scoul National University
Abstract
Emphatic algorithms have shown great promise in stabilizing and improving re- inforcement learning by selectively emphasizing the update rule. Although the emphasis fundamentally depends on an interest function which defines the intrinsic importance of each state, most approaches simply adopt uniform interest over all states (except where hand-designed interest is possible based on domain knowledge). In this paper, investigate adaptive methods that allow the interest function to dynamically vary over states and iterations. In particular, we leverage meta-grann eeta-graddents to automatically discover online an interest function that would accelerate the agent's 's learning process. Empirical evaluations on wide range of environments show that adapting the interest is key to provide significant gains. Qualitative analysis indicates that the learned interest function emphasizes states of particular importance, such as bottlenecks, which can be especially useful in a transfer learning setting.
1 Introduction
A fundamental challenge in reinforcement learning (RL) is to approximate key quantities such as value functions and optimal policies. Under the assumption that the world in which an RL agent interacts is large and the computational capacity is limited, natural trade-off emerges in which certain quantities are more accurately predicted than others over the course of learning. Standard RL algorithms, such temporal differences (TD) [39], perform updates at every state, thereby spending more resources on such frequent states at the expense of other potentially more useful ones. A possible solution could be to selectively emphasize certain updates, for example through state-dependent interest function. However, when combined with standard bootstrapping as in TD(A), such update rules known to be unstable [25].
Emphatic algorithms propose a solution in which state-dependent selective updating can be applied while maintaining stability under linear function approximation [44]. At their core, emphatic algorithms determine the emphasis to applied at each update by accounting for how much the current state is being bootstrapped from as well as intrinsic measure of its importance relative to other states. This intrinsic measure is encoded through an arbitrary state-dependent interest function which the practitioner comes a problematic question: how should one select the interest function?
set to any desired positive value. However, with this added flexibility
Previous works propose hand-crafted solutions for the interest function that showcase the usefulness of selective updating [27, 3, 28). However, when applying emphatic algorithms to complex environments
"Correspondence to: Martin [martin klissarov @ mail.meeiillco ca] and Rasool [fakoor @ amazon.com].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

where external domain knowledge may be too hard to encode, other than rare and specific exceptions [59], most practitioners use uniform interest over states [44, 53, 20, 60, 21, 17]. That is, they simply set the interest to 1 for all states. Building on the intuition that it can be beneficial to learn more from certain states than others, we argue that different emphases may be useful at various stages of the learning process. Indeed, as the RL learning process is inherently non-stationary. the relative importance of a particular state in the agent's updates should likely vary over training iterates as well.
In this work, we study how to adaptively learn the interest function in complex environments where hand-crafting an effective interest function is impractical. A good approach should allow for fast and flexible adaptation based on the agent's interactions with its environment. Considering the previous success of meta- mneta-graieent framework in discovering hyperparameters [55, 58], objective functions [54]. intrinsic rewards [62], and temporal abstractions [51], we here propose to learn and adapt the interest function based on meta- graddietsss in an online fashion. The interest function in our method is parameterized by parammtersss,,ss which are updated by gradient descent along with the parameters of the policy and value function.
We empirically investigate the merits of adapting the interest function on wide variety of envi- ronments and settings, ranging from prediction with linear function approximation control on vision-based tasks, Our contributions are the following, (1) In the off-policy setting, we see substan- tial gains in performance and sample efficiency when adapting the interest function. (2) We extend the traditional application of emphatic algorithms from the off-policy setting to on on-policy control, where we find it is crucial to adapt the interest function in order to observe consistent gains. (3) Qualitatively, our learned interest function appears to naturally discover states of importance, such as bottlenecks [38]. Such discovery is demonstrated to be very useful in transfer learning experiments. Our results highlight the general applicability of emphatic algorithms beyond the off-policy single-task setting considered in most previous studies of emphatic RL.
Background
We assume a Markov Decision Process M,, defined tuple (S, P) with finite state space S, finite action space A, transition probability distribution P: S x A AxS - [0, 1]. and a scalar reward function r(s, a) depending action E A in state E S. The policy AAxx x S - [0, 1] specifies the agent's 's behaviour and its expected discounted return starting from any state is represented as (*(s)= = E, 2
s]. where y E [0. 1) is the discount factor and R is the sampled reward after performing action A, in state S,. Under linear function approximation, the value function is defined with parameters 0 RR" and features (s)) E TR? that is V= (s; 0) = T ((s)?. An efficient family of algorithms for learning such functions builds on the Temporal Difference (TD) algorithm [39] where the value parameters, 9, are updated as follows:
the value function:
0c+1 = 0, + a (Be++1 + +0 6:+1 0, o.)o,
(1)
with a denoting the step size. In the control setting, the policy gradient theorem [45] for the episodic case provides the gradient of the expected discounted return from an initial state distribution d(sp) with respect to a stochastic policy T(- s; v) now parameterized by 2/:
aJ+(V) dv
Ox (als; v)
Σ dz ds(ss Σ
Q* (s,a)
(2)
Ov
where d)(s) = [d(s))>[(s))>>))> 80
2 pT (S. s|So so) is the discounted state occupancy measure
of the target policy и and (s. n) = E -Σ z R,+1 S, = s, A, = a] is the state-action value function. For a more detailed presentation of the notation please refer to App. C.
2.1 Emphatic Algorithms
Emphatic algorithms [44, provide a way to emphasize and de-emphasize the updates made at each iteration while preserving convergence. Their development was motivated by the challenges that arise under off off-policy learning when using function approximation and bootstrapping [50, 41]. In the off-policy setting, a behavior policy b(als) generates the data to learn value functions or
TTo simplify notation, drop in á (i.e. di instead of d(S:) where St is the sampled state at time t).
2

---

policies evaluated under the target policy π (a]s). Emphatic algorithms generalize TD in various ways, however of particular interest to our work is the added flexibility of arbitrarily defining the intrinsic importance of each state through the interest function. In the following we present emphatic algorithms in the general off-policy setting, as learning on-poliey is a special case.
Policy Evaluation: In its simplest one-step bootstrapping form, the Emphatic Temporal Difference (ETD) update rule for the value parameters takes the following form
0 = 0, + a: + 70, Φt+1 - 02 o1)o.
S.) is the importance sampling ratio at time and Fe, the followon trace, is defined where P: = ((A)))))
as,
F. = i(S.) + 700--1((5-1) ( +... =
i(S.) =... 1P
where i R is the arbitrary user-defined interest function. The specific form of this trace depends in part on the interest function, but also on how much state is bootstrapped from by previous states, discounted over time. This specific form is what confers stability and convergence to ETD [57], without introducing the full product of importance ratios used for prior correction [30].
Control: In the actor-critic setting, [10] proposed maximize the excursions objective Ju(v) E. Vd(ssV where (s(s) 9) is the stationary distribution of the policy b. We explain in App. C the reason why the stationary distribution appears instead of the discounted state occupancy measure of
(2). They proposed a way to approximate the policy gradient, where such approximation is only valid in the tabular case. [20] later derived the correct gradient for the more general objective that now includes the state dependent interest function,
Jo()) = Σ do(s))(s)V*(())
where the correct stochastic gradient update for the policy parameters v takes the following form,
Mt-1 = Vt + aPip V log T u.)
(5)
Interestingly, the same trace F; from the off-policy policy evaluation setting appears in the off-policy control setting. Note that A would be equal to 1 in the on-policy setting where target policy and behavior policy are the same.
3 Adaptive Interest
The interest function was designed a way to emphasize some states more than others and such can be an efficient way to encode useful inductive bias. Although it may be possible to find an interest function that is effective for specific and simple case (e.g. when additional knowledge about the task is readily available), it is not convenient to hand-desigh interest functions that effectively work for complex domains. For this reason, most previous works on emphatic algorithms consider simple uniform interest over states [44, 21].
Furthermore, we hypothesize that the usefulness of particular interest function vary through the learning process itself. This is obvious in the case of a changing environment, for example in continual learning, but is also relevant the single task setting where the agent's policy or bootstrapping targets vary in a non-stationary manner. In the next section, we further motivate the advantage of an adaptive interest function through the example of simple chain MDP.
3.1 Motivating Example
In Fig. 1, we consider the case of off- off-policy control in simple chain MDP made of four non-terminal states. Here our agent uses one-step SARSA [32] learn the action-value function (Q-value). The agent starts in state Sp and can reach either the terminal state on the right with reward of 1 or the more distant terminal state on the left with reward of 100. In our example, suppose the behavior policy is biased towards going right in the three rightmost states. The resulting target policy learned via SARSA without any emphatic weighting is misguided toward a suboptimal solution (Fig. I). App. B describes additional details of this experiment.
3

---

When designing a fixed interest function for SARSA with emphatic weighting, it would be advanta- geous to emphasize the states on the left and de-emphasize the states on the right, as a way to try and avoid the sub-optimal solution. However, Fig. 1 shows that although a well-designed fixed interest function can improve upon the this baseline, the resulting emphatic SARSA still unable to converge to good policy within 500 updates.
Finally we consider using an adaptive interest function. Here we leverage the same pattern as in the previous fixed interest function, but we only activate the interest in some states at cer- tain times. Particularly, at the start of training, only the left-most state is emphasized, and all other states have interest set to near-zero. As credit is propagated from the terminal left state towards the rest of the chain, the interest of the second left-most state is increased. This contin- ues until credit assignment reaches the starting state and the optimal action is selected. Emphatic SARSA with such a dynamic interest function is able to quickly converge to a good policy (Fig. 1).
+100
Fig. I empirically shows that, in our example MDP, a standard SARSA agent (without emphatic weight- ing) is outperformed by emphatic SARSA with fixed interest (supporting the general utility of em- phatic algorithms), which is in turn outperformed by emphatic SARSA with an adaptive interest (sup- porting the additional utility of adaptive interest). This example highlights that even in tabular settings without function approximation, the additional flex- ibility of an adaptive interest function can be quite beneficial. It is important to note that the fixed in- terest agent eventually finds the right solution in our example, but it has much worse sample complexity compared to our adaptive interest agent.
SD
Adaptive Interest
0.66
Fixed Interest
lett S=5g)
0.50
Base line
0.50
0.45
0.36
0.10
100
200
300
Number of steps
Figure I: Four State Chain MDP with off off-policy control where the behavior policy is biased towards going right. We plot the probability that the learned targel policy (implemented as a Boltzmann policy) goes left in the initial state Sp. which is the optimal action take. A baseline SARSA agent not using emphatic weighting will struggle to overcome this bias, whereas an emphatic agent with fixed inter- est ([1,, 10011 for each state from left to right) will require many samples to obtain the optimal policy. Using an adaptive interest inside emphatic updates guides credit assignment towards the start- ing state efficiently, converging towards the optimal policy early on. Results are averaged over 500 runs.
3.2 Meta-Gradient Interest (MINT)
As discussed in the previous section, we seek an adaptive interest function that would improve the learning process during training. However, automat- ically discovering such function is not straightfor- ward, in part as it can take any arbitrary value, and because we have evaluate the effect of a particular change in the interest function with regards to the agent's parameters. We explore adapting the interest function with a wide variety of heuristics such as the prediction error and find that such approaches do not generally provide improvements. Alter- natively, meta-gradients [55, 63] are a natural candidate as they can automatically discover such interest functions at each stage of learning through the interaction of an inner loop and an outer loop of optimization. We therefore propose to learn and adapt the interest function parameterized by meta- pett-parameters 17 in an online manner, within a single lifetime and within a single environment.
During the inner loop, the meta-parameters v), together with the agent's policy and value parameters {u, 0). appear in the base objective, J (0, 7)) Only the parameters are updated through this objective while the meta-parameters 1) remain fixed and influence the gradients. To illustrate the influence of the meta-parameters on the resulting parameters, we can write them as functions of n), i.e. (v'(n), (m))).
During the outer loop, the updated parameters are evaluated with respect to meta-objective, JM (n), (())), from which we derive the gradients with respect to 77. This is referred to as the meta- meta-gradien gradient, which evaluates how the values of the meta- peta--aarmettrss affected the performance of the updated parameters. By repeating this process, meta meta-gradients will adapt the meta- metam-ers in order more efficiently improve the parameters themselves.
We now describe the specific choices behind applying meta-gradients to emphatic algorithms when updating the policy. For simplicity, the derivation for the value function is relegated to App. H.3.

---

In the inner loop, the agent maximizes the following inner objective,
(") =
- - (()
(6)
where the interest function is parameterized by the peeta-aaraeeters and the policy ı is parameter- ized by v. This inner objective based on the excursions objective [20], that is, the future reward achieved by following the target policy т starting from the distribution of states generated by the behavior Another possibility would have been consider the counterfactual objective [59] or the alternative life objective [29], however these choices imply additional complexities which we leave for future work. We provide a more detailed discussion on the choice of the objective function in the App. H.1. The inner loop can be written by using (5), obtaining:
v t v an Ft,,
where Fi., = n) + 7Pt-1 emphasizes the current state according to the current meta interest and the followon trace the previous timestep.
When considering the meta-objective, practitioners usually employ the same form as the inner objective. In our case this would be written
JM - ( (s(s)
(8)
where is defined by the updated parameters v1)).. Recently, [14] argue that such an approach may lead to poor meta-optimisation landscape, as both objectives share the same curvature. In App. we verify different meta-objectives, such as the variance of the reward-to-go [47], and report no increase in performance when compared to (8) in our setting. It is likely that obtaining their improvements also relies tackling myopia in meta- meta-gradie geaddggrdsents. From (8) we obtain the following meta- aata-graddeet
r/ 1l am V = 17 + am J V
where v,,e' encodes how the meta-parameters affected the new parameters. A stochastic sample of this quantity at time can be expanded as (see also App. H.2),
Σ 1=0
n (S.))..)) V. (A, (S) At)
where Piu is a product of importance sampling ratios. Pseudocode for our approach, which we call MINT (Meta-gue (Meta-gradient Interest), is presented in Algorithm of App. A.
Performing the update rule in (9) would require a new set of samples. In practice, the same samples are re-used for both loops [63, 62, 51] through sliding window of experience. In our work we opt to
use the importance sampling ratio method of [63]. Finally in the present derivation we only consider 1 1-step meta-gradient [52] as greatly simplifies the exposition see App. H.4 for derivations). We also show in App. H.4 why the sampling correction term [2] does not appear in the off-policy setting.
4 Experiments
We now validate our method on a wide range of scenarios to assess the following questions: 1) Can we automatically learn an interest function on complex environment order to improve performance? 2) How robust is the meta interest with respect the agent's 's hyperparameters? 3) Is possible to leverage the information encoded by the learned interest function for downstream tasks?
We first conduct experiments in the off oof-po policy evaluation setting under linear function approx- imation. Next, we extend the usual field of study of emphatic algorithms and verify their general utility. In particular, we study how they can improve the performance of on-policy algorithms under control setting, where considerable gains are witnessed only under an adaptive interest. Finally, we further extend our investigation to the transfer learning setting by leveraging the interest function learned previous task in order to greatly speed up the learning process in a second task. All hyperparameters available in the App. E.
5

---

0.7
0.6
0.5
MSVE
MSVE
0.3
0.2
0.1
0.30 3.75
2.00
Number steps
Learning Rate (2*)
(b)
(c)
0.7
MIXT
0.6
ETDLB Error Interest
0.5
MSVE
TDRC
MSVE
0.4-
Virace
0.3
0.2
0.1
47 44
Number stops
Learning Rate (2")
(d)
(I)
Figure 2: Off-Policy Evaluation under linear function approximation where we build on the empirical setup of [16]. The top row presents results for the 4Rooms- 8Tasks (4R8T) domain, while the bottom row presents for the High Variance--ccR -Tasks (H4R8T) domain, where states in blue present high variance. We compare emphatic algorithms high performing off-policy basclines such as TDRC and Vtrace. For both environments we notice that by adapting the interest function, either through meta gradients as MINT through absolute value of TD error, the final prediction error is significantly improved, especially in the second environment where MINT reduces by half the error when compared to non-emphatic methods.
4.1 Linear Function Approximation
Setup. Our first experiments are done in the off-policy policy evaluation setting with linear function approximation using tile coding [41]. We adopt the setup from [16], who considered variations of the classical Four Rooms domain, shown in Fig. 2. We name these variations 4Rooms-8Taa -Tasks (4R8T) and High Variance-4Rooms-B8 -Tasks (H4R8T) highlight their characteristies (See App. D for their details).
Quantitative Results. Fig. 2 shows the results for the 4R8T (top) and H4R8T (bottom row). The y- axis indicates the mean squared value error (MSVE) averaged across all policies. In Fig. 2c and 2f, we vary the learning rate the x-axis and for each value report each algorithm's best final performance chosen across all values of the bootstrapping coefficient X (and other possible hyperparameters). Fig. 2b and 2e show the best learning curves for each algorithm.
In these two domains, we compare learning the interest through meta gradients, MINT, to bascline that adapts the interest with respect to the absolute value of the TD error (ETDLB TD Error Interest, where ETDLB refers to the generalized version emphatic TD [19]). We compare these adaptive methods the standard emphatic baseline ETDLB. Finally, also compare to the recent TD with Regularized Corrections (TDRC) algorithm [15], which follows the line of work on Gradient TD [43], well the V-trace [1]], which is representative of the performance of methods that truncated importance sampling ratios [26]. We notice that across both tasks as shown in Fig. 2e, leveraging and especially H4R8T, our method almost halves the error when compared to non-emphatic methods.
adaptive interest leads better final value
In App. D we present additional figures that take into account different metric: the area under the curve (instead of the final performance) and notice a similar pattern. When looking at the learning curves the left, we notice that ETDLB pays price in terms of slower convergence in order to achieve a better final performance. However, when using an adaptive interest, the difference with non-emphatic methods is greatly reduced, especially in the 4R8T domain.
Examining closely Fig. 2c and 2f, we notice that leveraging an adaptive interest moves the bottom of the U-shaped curve to the right. By selectively emphasizing some states, emphatic algorithms using an adaptive interest are able to learn on higher learning rate. However, the shape of the U curve tends to cut drastically after certain threshold, which point the updates become unstable.

---

When we compare adapting the interest through meta gradients to the one defined as the absolute value of the TD error, we notice that their performance is almost equal. It can perhaps seem surprising that this would be case, as the meta gradients method can in theory learn any function. For our particular choice of objectives, we show in App. H.3 the form of the meta gradients under linear function approximation. For simplicity, if we further assume that features are tabular and we are under the on on-policy setting, we get that the stochastic sample at time t of the gradient is,
V v MM =6))
(10)
where e₁ indicates the one-hot vector represent the current state at time t. (10) shows that the meta gradient updates in the direction of the squared TD error. Although the convergence of the meta-parameters will not be to the sampled squared TD error, at each iteration the meta-parameters affected in similar way when compared to the absolute value of the TD error heuristic. In practice, the features are not exactly tabular and therefore the updates made on one state may affect another, which would also explain why MINT is slower to converge.
Qualitative Results. In Fig. 3, we inspect the learned interest functions obtained by MINT on four of the eight tasks (one per room). The top row shows the learning process for 4R8T, while the bottom shows the process for H4R8T. In both domains, the first state be highlighted is the one next the goal in the hallway. Indeed this state is highly influential since all states bootstrap from directly or indirectly.
(a) 4Rooms 8Tasks (4R8T)
As training progresses, the interest in the 4R8T dif- fuses to neighbouring states (this bears close resem- blance to the example presented in Fig. ). In the H4R8T, different pattern emerges where the state with high variance is being highlighted early on in training. As this state is visited by many trajectories that the target policy would take, it influences the val- ues of many other states that need bootstrap from it. However, since it exhibits higher variance than neighbouring states, requires more computation be correctly estimated.
(b) High Vawaaaaa--eaoooooee -Tasks (H4R8T)
Figure 3: Visualization of the interest func- tion across iterations. These show results the start (left column), mid-training (middle column) and at the the end (right column). Depending the environment, different patterns are being encoded in the interest function. In the H4R8T environment, the high variant state is being em- phasized as it requires more resources order to be estimated accurately.
At the end of training, we observe that some states are particularly less important, like state near the opposite corners of the hallways. As the target policy does not visit them often and not many states bootstrap from them, it is less important for them to be accurate. Moreover, it is interesting to note that the diffusion of interest observed in the 4R8T domain is not perfectly uniform. Since tile coding [41] is used the function approximator, specific pattern of the interest function may be needed at different states to lower the overall prediction error.
4.2 Experiments at Scale
Emphatic algorithms were initially derived for the off-policy setting. However, the flexibility given by the interest function is generally applicable, even in the on-policy case. To showcase this flexibility, in this section we investigate the performance of emphatic algorithms in the on-policy setting using non-linear function approximation.
4.2.1 MinAtar
Setup. We verify the generality of the proposed method by considering the MinAtar domain [56].
which is miniaturized version of some of the games from the classic Atari 2600 testbed [6].
The environment provides 10 x 10 x n state representations, where n varies for each game. The environments implemented using sticky actions and randomization [24]. For all games use 10 random seeds and report the mean and standard deviation after 10M timesteps.
Results. Fig. 4 shows that MINT provides good gains when compared to the two baselines. A standard PPO agent [37] and a meta-gradien approach [54] that meta-learns the target function which appcars in RL update rules. We explain in detail this baseline in the App. F and theoretically show
7

---

25
50
Rewards
20
Rewards
Rewards
40-
18
10
20
Interest
Meta Learned Target
10
PPO
0.0
0.2
0.4
0.6
0.6
1,0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
02
04
0.6
08
1,
Iterations
Iterations
Iterations
(a) Asterix-v0
(b) Breakout-v0
(c) Freeway-v0
250
Rewards
100
0.0
02
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1,3
Iterations
Iterations
(d) Spacelnvaders-v0
(c) Seaquest-vO
Figure 4: Performance on MinAtar. Adapting the interest provides consistent gains in sample efficiency well as final performance. Meta-lcamingg the target function more general approach than ours, but also requires samples provide improvements over the standard baseline.
that learning the interest function is not simply special case of their approach. Results Fig clearly demonstrate that our method outperforms both baselines. Our findings also agree with the experiment in [54] (Fig 3a in their paper) that meta-learning the target function requires many more samples before it can match the baseline's performance. [54] also compares to approach that meta-learns the complete loss function (which could recover our update rule) and find that the agent is not able to learn in the online, single-lifetime setting (which is setting).
These results highlight the difficulties more general meta-learning formulations and their impact on sample efficiency. This suggests that meta-learning the interest-function may be good trade-off' between generality and the amount of inductive bias. We additionally present results in Fig. where we vary the learning rate and present at the U shaped curves of performance, which seem to behave similarly to the linear function approximation case.
4.2.2 Continuous Control
Setup. We perform experiments the MuJoCo domain [49, 7], where states and actions continuous. We report the mean and standard error averaged across 10 random seeds. We include several emphatic baselines where we explore using fixed interest and various heuristics for adapting the interest function. Additionally, we investigate the usefulness of adapting the learning rate itself using hypergradient descent [5]. We provide description of all the baselines in App. E.
Results. As Fig. shows, utilizing adaptive interest function is key to get consistent improvement over PPO (across almost all environments). Interestingly, the adaptive heuristic based on the TD error that worked well in the prediction setting does not generalize to this one. One way to understand this to consider that low TD error may not be indicative of high performing policy in control.
We also compare our method to hypergradient descent (HD) [5] which dynamically updates the learning rate during training. We notice that HD does not seem to provide gains, except in Humanoid- v3 where reaches the performance of MINT the cost of a slower learning process. This is in contrast to our method which generally does not suffer from increased sample complexity to achieve better performance. We highlight that an important difference between HD and MINT that the interest function is state-dependent quantity, which can provide additional flexibility. In the App. E we compare to additional baselines, such as meta-learning the reward function [63] and various adaptive heuristics for the interest function.
8

---

4000
2000
3500
2000
Keward
2500
1500
2000
1000
1000
500
0.00 0.26 0.50
0.75
1.00
1.25 1.30 1.75
0.00 0.25 0.30 6.75
1.00
1:26
1.50 1.75
0.00
0.20
0.30
0.75
1,00
1.25
1,00
1.75
herations
Herations
herations
(a) Ant-v3
(b) HalfChectah-v3
(c) Walker2d-v3
1.0
030
@ 480
I Fixed Interest
MINT
420
TO Error Intereet
Hypergradient Descent PPO
0.2
350
5.0
025 0.50
0.75 1,00 torations
122
1.50
1.75
Beralions
(d) Humanoid-v3
(c) FetchReach-vi
Figure 5: Results on continuous control. We compare MINT to various baselines including an emphatic variant of PPO using fixed interest, well interest based the absolute value of the TD error. We also verify whether updating the learning rate via hypergradient descent match the performance of MINT. Across environments, we notice adapting interest via meta- gradd-ggassetss key to obtain consistent gains.
1.0
Interest Function
0.8
From scratch
Rewards
0.6
0.4
0.2
0.0
2
Interest Function
Reward Function
Iterations
1e4
(b)
(c)
Figure Transfer experiment where the interest the reward function is transferred to help an agent policy from scratch a variant of the task. In b) and c) we visualize the values of each function, where brighter color means higher value. Interestingly, the interest function highlights states near the hallways of the starting state (top-left), also referred bottleneck states.
4.3 Transfer Learning across RI. Tasks
Setup. As the interest function automatically learned, we observe that it likely encodes knowledge and information that be useful later. In particular, this knowledge may serve speed up learning in new environment over learning from scratch. We investigate this hypothesis in the FourRoomsTranne Transfer environment [9], shown Fig. 6a where we show mean and standard deviation across 30 seeds. The agent starts in the top left corner and has to get to the goal location in green. The state highlighted orange is a distractor state that provides random reward, R No, 1).
For the transfer setting, we change the location of each of these entities (see App. G). In this setting, the agent leamss in the first environment and only transfers the interest function to the second one, which remains fixed thereafter. We also compare to an agent that uses meta gradients to learn an intrinsic reward function before transferring it the second task, similarly [62].
Results. Fig. shows that the learned interest function provides significant speed-up when compared to the actor-critic baseline that learss from scratch without an interest function.

---

Here, transferring the intrinsic reward does not seem to help. Note we do nor claim that using meta-gradients to learn an intrinsic function always better choice than learn an intrinsic reward function, as it likely depends on the exact transfer learning setup".
To understand how the interest function helps in transfer, present in Fig. 6b and Fig. 6c the learned interest function and the learned reward function. We notice that the interest function highlights states that are near the goal, but also the hallways of the starting room (top left), which are usually referred to as bottleneck states [38]. Such states are of particular importance as they influence the trajectory an agent takes as well as many of the predictions it makes during such trajectory. As we notice, the interest function highlights the hallway leading to the goal, but also the one leading to the distractor. On the opposite, the reward function naturally highlights the path the goal and de-emphasizes the one leading to the distractor state. This illustrates one useful property of the interest function: it highlights the location of rewards, whether they are positive or negative. This kind of invariance is key for a better transfer performance in our setting, and could be used more generally in continual learning [31 This experiment also points to an interesting future direction where universal interest function could be defined similarly to universal value functions [33].
5 Related Work
Initially derived by [44, 27] as a stable and simple one-time-step solution to the problem of off-policy prediction. Its convergence is shown in [57] when employing the full trace and later [61] show convergence for the Truncated ETD algorithm. The ideas prediction were extended to control by [20. 18, 60]. Emphatic algorithms have been shown to be strong baseline in many benchmarks under linear function approximation [16, 17], even in the on-policy case [53, 3]. Recently, emphatic algorithms have been extended the deep RL by building on variant of the IMPALA agent [11] with auxiliary heads and have shown superior performance on Atari [21, 22].
Emphatic Methods.
Selective Updating. Emphatic methods can be more generally as performing selective updating, whereby through a scalar we emphasize or de-emphasize the updates to each state. This has previously been investigated in the context of model free learning [34] and model based planning [1]. Recently, [8] provide a unifying view of various algorithms as a form of selective credit assignment. In hierarchical reinforcement learning, selective updating is integral part of the options framework either through initiation sets [46] or interest functions [23].
Meta Gradients. Meta learning [35, 48, 13] is class of methods that have better capabilities in adapting to new tasks by learning prior from previously seen related tasks in the past [36, 4]. While these methods mainly focus multi-task learning [12], meta meta-gradients based methods [62, 63] focus on learning the meta parameters online within a single task, based on online cross validation [40].
Conclusion
We propose learn and adapt the interest function utilizing meta-gradients in an online fashion where hand-coded solutions are not feasible. Comprehensive experiments on various settings suggest that automatically adapting the interest function from stream of data leads to improved perfor- mance. Although certain heuristics for adapting the interest function are occasionally beneficial, our experiments point that consistency and general usefulness are achieved through our adaptive method.
'Note, we do not consider using recurrent neural networks or a lifetime value function here, which were found critical for meta-learned intrinsic rewards to work well in transfer [62]. Also, the interest/reward functions trained on single environment at first, before being transferred new one, without further training.
10

---

References
[1] Z. Abbas, S. Sokota, E. J. Talvitie, and M. White. Selective dyna-style planning under limited model capacity. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 11436-1 1856-11446 11446. PMLR, 13-18 Jul 2020.
[2] M. Al-Shedivat, T. Bansal, Y. Burda, Sutskever, I. Mordatch, and P. Abbeel. Continuous adaptation via meta-learning in nonstationary and competitive environments. In International Conference on Learning Representations, 2018.
[3] N. Anand and D. Precup. Preferential temporal difference learning. In Proceedings of the 38th International Conference Machine Learning, volume 139, pages 286-296. PMLR, 18-24 Jul 2021.
[4] J. Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12:149-199 2000.
[5] A. G. Baydin, R. Cornish, D. M. Rubio, M. Schmidt, and Wood. Online learning rate adap- tation with hypergradient descent. In International Conference on Learning Representations, 2018.
[6] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. CoRR, abs/1207.4700 2012.
[7] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym. CoRR, abs/1606.01544 2016.
[8] V. Chelu, D. Borsa, D. Precup, and H. van Hasselt. Selective credit assignment, 2022.
[9] M. Chevalier-Boisvert, L.. Willems, and S. Pal. Minimalistic gridworld environment for openai
gym. https: htttps://githut cwm/co///////oe httmm//aaimec///eememmmmm beeeeec//yymmmiie gym-minigrid, 2018.
[10] T. Degris, M. White, and R. S. Sutton. Off-policy actor-critic. In Proceedings of the 29th International Coference on International Conference on Machine Learning, page 179-186. 2012.
[11] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning. S. Legg. and K. Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. CORR, abs/1802.01561. 2018
[12] R. Fakoor, P. Chaudhari, Soatto, and A. J. Smola. Meta-q-learning. International Conference Learning Representations, 2020.
[13] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1126-1135. PMLR, 06-11 11 Aug 2017.
[14] S. Flennerhag, Y. Schroecker, T. Zahavy, H. van Hasselt, D. Silver, and S. Singh. Bootstrapped meta-learning. In International Conference on Learning Representations, 2022.
[15] S. Ghiassian, A. Patterson, S. Garg, D. Gupta, A. White, and M. White. Gradient temporal- difference learning with regularized corrections. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 3524-3534. PMLR, 13-18 Jul 2020.
[16] S. Ghiassian and R. S. Sutton. An empirical comparison of off-policy prediction learning algorithms in the four rooms environment. CoRR, abs/2109.051 10, 2021.
[17] S. Ghiassian and S. Sutton. An empirical comparison of off off-policy prediction learning algorithms on the collision task. CoRR, abs/2106.00922, 2021.
[18] E. Graves, E. Imani, R. Kumaraswamy, and M. White. Off- Off-policy actor-critic with emphatic weightings. CoRR, 0abs22111081111 072, 2021.
[19] A. Hallak, A. Tamar, R. Munos, and S. Mannor. Generalized emphatic temporal difference learning: Bias-variance analysis, 2015.

---

[20] E. Imani, E. Graves, and M. White. An pollcy policy gradient theorem using emphatic weightings. In Advances in Neural Information Processing Systems, volume 31, 2018.
[21] R. Jiang, T. Zahavy, Z. Xu, A. White, M. Hessel, C. Blundell, and H. Van Hasselt. Emphatic algorithms for deep reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 5023-5033 PMLR, 18-24 Jul 2021.
[22] R. Jiang, S. Zhang, V. Chelu, A. White, and H. van Hasselt. Learning expected emphatic traces for deep RL. In AAAL, pages 7015-7023, AAAI Press, 2022.
[23] K. Khetarpal, M. Klissarov, M. Chevalier-Boisse Temporal abstraction with interest functions. CoRR, abs/2001. abs//200.00027,, 2020.
P. Bacon, and D. Precup. Options of interest:
[24] M. C. Machado, M. G. Bellemare, E. Talvitie, Veness, M. J. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. CoRR, abs//700.0660 2017.
[25] A. Mahmood.
Incremental off- off-policy reinforcement learning algorithms, 2017.
[26] A. R. Mahmood, H. Yu, and R. S. Sutton. Multi-step off-policg learning without importance sampling ratios. CoRR, abs/1702.03006, 2017.
[27] A. R. Mahmood, H. Yu, M. White, and R. Sutton. Emphatic temporal-difference learning. CoRR, abs/1507.01569 2015.
[28] M. K. McLeod, C. Lo, M. K. Schlegel, A. Jacobsen, R. Kumaraswamy, M. White, and A. M. White. Continual auxiliary task learning. In Advances in Neural Information Processing Systems, 2021.
[29] A. Patterson, A. White, S. Ghiassian, and M. White. A generalized projected bellman error for off-policy value estimation in reinforcement learning. CoRR, 0ttss/104.138880 2021.
[30] D. Precup, Sutton, and Dasgupta. Off-policy lemporal-difference learning with function approximation. Proceedings of the 18th International Conference on Machine Learning, 06 2001.
[31] D. Rolnick, A. Ahuja, J. Schwarz, T. Lillicrap, and G. Wayne. Experience replay for continual learning. In Advances in Neural Information Processing Systems, volume 32, 2019.
[32] G. A. Rummery and M. Niranjan. On-line Q-learnig using connectionist systems. Technical Report TR 166, Cambridge, England, 1994.
[33] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pages 1312-1320. PMLR, 07-09 Jul 2015.
[34] T. Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay, 2015. cite Published at ICLR 2016.
[35] J. Schmidhuber. A neural network that embeds its own meta-levels. IEEE International Conference on Neural Networks, pages 407-412 vol.1, 1993.
[36] J. Schmidhuber, J. Zhao, and M. Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-improvement. Machine Learning, 28(1): Jul 1997.
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint 1rxXiv:107..6333 2017.
[38] A. Solway, C. Diuk, N. Córdova, D. M. Yee, A. G. Barto, Y. Niv, and M. M. Botvinick. Optimal behavioral hierarchy. PLoS Computational Biology. 10, 2014.
[39] R. S. Sutton. Learning to predict by the methods of temporal differences. Mach. Learn., 3(1):9 aug 1988.
12

---

[40] R. S. Sutton. Adapting bias by gradient descent: An incremental version of batt-baar-detta. In Proceedings of the Tenth National Conference on Artificial Intelligence. 92 page 171-176. AAAI Press, 1992.
[41] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
[42] R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018.
[43] R. S. Sutton, H. Maei, and C. Szepesvári. A convergent o(n) temporal -enforal-differrenee algorithm for off ooflcy learning with linear function approximation. In Advances in Neural Information Processing Systems, volume 21, 2009.
[44] R. S. Sutton, A. R. Mahmood, and M. White. An emphatic approach to the problem of off-policy temporal-difference learning. J. Mach. Learn. Res., 17(1):2003-6311jjj) jan 2016.
[45] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforce- ment learning with function approximation. In Advances in Neural Information Processing Systems, volume 12. MIT Press, 2000.
[46] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):11-2111 1999.
[47] A. Tamar, D. D. Castro, and S. Mannor. Learning the variance of the reward-to-ge go. Journal of Machine Learning Research, 17(13):1-3,, 2016.
[48] S. Thrun and L. Pratt, editors. Learning to Learn. Kluwer Academic Publishers, USA, 1998.
[49] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5066 5533. IEEE, 2012.
[50] H. van Hasselt, Y. Doron, F. Strub, M. Hessel, N. Sonnerat, and J. Modayil. Deep reinforcement learning and the deadly triad, 2018.
[51] V. Veeriah, T. Zahavy, M. Hessel, Z. Xu, J. Oh, I. Kemaev, H. van Hasselt, D. Silver, and S. Singh. Discovery of options via meta-learned subgoals. In Advances in Neural Information Processing Systems, 2021.
[52] R. Vuorio, J. A. Beck, G. Farquhar, J. N. Foerster, and S. Whiteson. No DICE: An investigation of the bias- tradeoff in greaingr In Deep RL Workshop NeurlPS 2021, 2021.
[53] A. M. White and M. White. Investigating practical, linear temporal difference learning. CoRR, abs/ Abs//602.08771, 2016.
[54] Z. Xu, H. P. van Hasselt, M. Hessel, Oh. S. Singh, and D. Silver. Meta-gradient reinforcement learning with an objective discovered online. In Advances in Neural Information Processing Systems, volume 33, pages 15254-15264, 2020.
[55] Z. Xu, H. P. van Hasselt, and D. Silver. Meta- deta-ggradien reinforcement learning. In Advances in Neural Information Processing Systems, volume 31, 2018
[56] K. Young and T. Tian. Minatar: An atari-inspired testhed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv:1933.03 09900.337766 2019.
[57] H. Yu. On convergence of emphatic temporal-diffeee -iffference learning. In Proceedings of The 28th Conference on Learning Theory, volume 40 of Proceedings of Machine Learning Research, pages 1724-1756 1751. PMLR, 03-06 Jul 2015.
[58] T. Zahavy, Z. Xu, V. Veeriah, M. Hessel, J. Oh, H. P. van Hasselt, D. Silver, and S. Singh. self-tuning actor-critic algorithm. In Advances in Neural Information Processing Systems, volume 33, pages 20913-20924, 2020.
13

---

[59] S. Zhang, W. Bochmer, and S. Whiteson. Generalized off- off-policy actor-critic. In Advances in Neural Information Processing Systems, volume 32, 2019.
[60] Zhang, B. Liu, H. Yao, and S. Whiteson. Provably convergent off-policy actor-critic with function approximation. CoRR, abs/1911.04384, 2019.
[61] S. Zhang and S. Whiteson. Truncated emphatic temporal difference methods for prediction and control. CoRR, 0ass2208.0533. 2021.
[62] 7. Zheng, J. Oh, M. Hessel, 7. Xu, M. Kroiss, H. Van Hasselt, D. Silver, and S. Singh. What can learned intrinsic rewards capture? Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 11436-11446. PMLR, 13-18 Jul 2020.
[63] Z. Zheng, J. Oh, and S. Singh. On learning intrinsic rewards for policy gradient methods. In Advances Neural Information Processing Systems, volume 31, 2018.
14


Reviews:
[{"review_id": "pNN-QISIcb5", "reviewer": "Reviewer_UBwo", "summary": "This paper extends work focusing on emphatic TD learning. While the original formulation of emphatic TD learning included an interest function that prioritises updates to certain states, in practice this was always made uniform or required expert knowledge to set. The idea here is to (meta) learn the interest function as training progresses - an inner loop follows regular emphatic TD learning with the learned interest function, while an outer loop next runs to update the interest function's parameters. Empirical results are provided in several domains, including pixel-based ones, continuous control tasks and linear function approximation settings. Results indicate that an adaptive interest function learned over time seems key to applying these approaches to on-policy learning, while the method also outperforms a fixed interest function, as well as other meta-parameter learning approaches. \n", "questions": "1. In the MinAtar experiments, I was expecting to see the TD-error heuristic which was used in the chain walk (and popped up again in MuJoCo). Is there a reason that it wasn't included in this set of experiments?\n\n2. On Line 292 it states: \"We explain in detail this baseline in the App. F and theoretically show that learning the interest function is not simply (sic) special case of their approach\". However, I cannot find a discussion of this anywhere. \n\n3. In the motivating example, I'm not entirely clear on why the choice of base learning algorithm was SARSA. Is this because the focus, later on, is on-policy learning? Otherwise, it seems strange to apply the various interest strategies on top of a learning algorithm that doesn't quite fit. \n\n4. Is there any intuition for why the MSVE should diverge for larger learning rates with the TD error interest on H4R8T, but not 4R8T?\n\nAs an aside (certainly not necessary to include here), the focus on the robustness of hyperparameters like learning rate could potentially benefit from applying the methodology of [1].\n\nMinor comments:\n\n1. It would be helpful to move Figure 4 to the top of the next page. Currently, it appears before Fig 3 and you have to jump between pages to read the associated text. \n2. Without knowledge of the exact reward function, Figure 3 takes some puzzling out. My guess is that it is sparse (1 at the goal, 0 elsewhere) but that is simply a guess because I do not see it listed anywhere\n\n3. Many of the references list the arxiv or preprint versions of papers. Please check these all, since many have been published. For one example: [59] appeared at NeurIPS 2019. \n\nMinor typos:\nL184: \"as THE inner objective\"\nL239: algorithm -> algorithms\n\n[1] Jordan, Scott, et al. \"Evaluating the performance of reinforcement learning algorithms.\" International Conference on Machine Learning. PMLR, 2020.", "limitations": "N/A", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\\+ I found the paper to be extremely well-written and well-motivated. Every time a sentence raised a question in my mind, the very next sentence explained or justified it to me! \n\n\\+ The main idea is conceptually simple (in a good way) and the paper does a good job of evaluating it in several different kinds of domains. \n\n\\- I realise that space is a large issue here, but I found that I had to jump back and forth between the appendix and the main paper quite often. While much of this is extra discussion, a few times it was for key information, such as the details of the domain. In particular, I'm still not 100% certain if the reward function for the four rooms domain is a sparse one (and what the exact values are), even having consulted the appendix.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "amOfptpHnqA", "reviewer": "Reviewer_ndyW", "summary": "The paper proposes a method to learn interest state function via meta-gradients in order to do emphatic RL. This interest function essentially simplifies the credit assignment problem and highlights 'more important / interesting' states.\n\nThe authors study their method in multiple settings: toy domain, continuous control and discrete domains as well as in the transfer learning setting. The signal is quite clear, the method which authors propose seems to always provide the benefit compared to standard RL approaches.\n\nThe authors study different ways of constructing interest function and provide informative ablations over these, with somewhat limited explanations.\n\nOverall, the paper is clearly written, the ideas are easy to understand and the results look good.", "questions": "* How does the TD error interest method perform on MinAtar?\n* Why do the authors claim that meta-gradient performs better on Figure 2, where it is clear that TD error interest converges faster and to a better value?", "limitations": "Limitations are clear from the paper", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Clearly written paper\n* The authors studied a novel approach for RL and credit assignment via meta-learning of the interest function and provided solid experimental evidence that the approach can improve RL\n* The experiments are quite broad, demonstrating the applicability of the method in different scenarios: discrete control, continuous control, transfer learning.\n\nWeaknesses:\n* It is not clear why the authors didn't provide `TD error interest` for MinAtar. Given the results on the toy domain, this approach may actually perform better than meta-gradients.\n* Minor writing / visualization details. The legend colours on the Figure 2 are very hard to read. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "18hAl_4ryO-", "reviewer": "Reviewer_xj8y", "summary": "This paper studies the problem of learning the interest function, which emphasizes the importance of different states differently in RL objectives. The interests of different states are typically set to be all 1 in existing approaches. And this paper seems to pioneer the study of the adaptive interest function (the paper didn't claim it but from its discussion of existing works it appears to me that the authors believe that no other works have studied this problem before).\n\nThe paper proposed a meta-learning approach to the problem. Specifically, the interest function is learned to maximize an objective, which is an off-policy control objective. The paper proposed an off-policy prediction algorithm and an on-policy control algorithm. Empirically, the off-policy prediction algorithm almost matches the performance of an ETD algorithm with TD error as the interest. The on-policy control algorithm is significantly better than the PPO algorithm and the other Meta-learning algorithm.", "questions": "Overall, could you explain how different interests influence the learning process in prediction and control settings respectively?\n\nline 21 A possible solution could be to selectively emphasize certain updates, for example through a state-dependent interest function.\nWhy is this a possible solution?\n\nAre there any existing methods that adaptively choose the interest? Is your work the first one studying the adaptive interest function?\n\nIt is not clear from the introduction section the prediction or the control setting or both is considered in the paper.\n\nWhen designing a fixed interest function for SARSA with emphatic weighting, it would be advantageous to emphasize the states on the left and de-emphasize the states on the right, as a way to try and avoid the sub-optimal solution.\nWhy?\n\nCan interests go arbitrarily large?\n\nIn figure 1, no algorithm eventually converges to the optimal policy (left), right? Why is that?\n\nIn figure 2, how did you compute the MSVE?\n\nIsn't figure 2 showing that the proposed method is slightly worse than the ETDLB method with the TD error as the interest in the tested problem (c.f. subfigure c)? \n\nI didn't find in the paper a description of the off-policy policy evaluation algorithm (the one you used to produce figure 2).\n\nIt seems that the interest function is shared across all sub-tasks? If so, why is it reasonably to share it? Why do you consider the multi-task setting instead of the single one?\n\nIs your off-policy policy evaluation algorithm a batch algorithm? Are TDRC, Vtrace, and ETDLB online algorithms or batch algorithms?\n\nIn figure 3, I see that during the entire training period, cells near the hallway cells have high interests. I can also see that, at the end of training, the further a state is from from the hallway state, the lower interest the state is assigned. The paper explains that \"as the target policy does not visit them often and not many states bootstrap from them, it is less important for them to be accurate\". This is a reasonable explanation. But given that the pattern looks pretty close to the value function, it can also be hypothesized that higher state value results in higher interest. If a state has a high value but is not frequently visited by the target policy, would the interest be high?\n\nWhat is the on-policy control algorithm? \n\nWhen comparing with PPO and the Meta Learned Target algorithm, do you use the same amount of buffer to store past experience and the same amount of computation?", "limitations": "Overall I would need the questions listed above to be addressed to tell the paper's limitations. The paper does not have a potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nIn general, the paper is easy to follow.\nAdapting the interest function seems to be interesting and important, and is not well understood in the literature.\nThe proposed algorithm uses an elegant idea.\nEmpirical results show some interesting patterns, which would be helpful for understanding the proposed algorithm.\n\nWeakness:\nThe motivation of the idea, the description of the algorithm, and the explanation of the empirical results are not clear enough. Some important questions need to be addressed to evaluate the paper's contribution. See the Questions section for details.\n\n-------------------------------------------------------------------------------\n\nI have read the authors' responses and feel that most of my questions were answered, including the key question about explaining how interest function influences the learning process. Therefore I decided to increase my score to 6.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0YnxfdkMEb", "reviewer": "Reviewer_2sr2", "summary": "This paper proposes to use meta-gradient to adapt the interest function in emphatic RL algorithms. Empirical study covering off-policy policy evaluation with linear function approximation, on-policy control with nonlinear function approximation, as well as transfer learning confirm the efficacy of the proposed method.", "questions": "See \"Weaknesses\"", "limitations": "n/a", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. This paper is well motivated and targets an important problem in RL. The use of meta-gradient is straightforward but appears novel to me.\n2. The paper is well organized and easy to follow. Empirical study covers a wide rage of settings. \n\nWeaknesses:\n1. The jump from off-policy prediction to on-policy control seems a bit abrupt to me. Emphatic methods are originally designed for off-policy prediction so starting with Section 4.1 seems natural to me. But is there any particular reason that the control experiments are in the on-policy setting, especially given the success of [22, 23], the motivating example in Section 3.1, and the derivation in Section 3.2. I feel adding more off-policy control experiments would significantly strengthen this work.\n2. The meta-objective in Equation (8) seems not well motivated. Why not simply use the episodic return \\sum_s p_0(s) v_\\pi(s) where p_0 denotes the initial distribution? After all all the domains used for control is episodic instead of continuing and the agents are evaluated against episodic return. Is the ease of optimization the main consideration for the use of the excursion objective? I feel more discussion about this would also strengthen this work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept