What Can Learned Intrinsic Rewards Capture?
Authors: Zeyu Zheng
Junhyuk Oh
Matteo Hessel
Zhongwen Xu
Manuel Kroiss
Hado van Hasselt
David Silver
Satinder Singh
Venue:  (2020)
Keywords: reinforcement learning
deep reinforcement learning
intrinsic movitation

Abstract: Reinforcement learning agents can include different components, such as policies, value functions, state representations, and environment models. Any or all of these can be the loci of knowledge, i.e., structures where knowledge, whether given or learned, can be deposited and reused. Regardless of its composition, the objective of an agent is behave so as to maximise the sum of suitable scalar functions of state: the rewards. As far as the learning algorithm is concerned, these rewards are typically given and immutable. In this paper we instead consider the proposition that the reward function itself may be a good locus of knowledge. This is consistent with a common use, in the literature, of hand-designed intrinsic rewards to improve the learning dynamics of an agent. We adopt a multi-lifetime setting of the Optimal Rewards Framework, and investigate how meta-learning can be used to find good reward functions in a data-driven way. To this end, we propose to meta-learn an intrinsic reward function that allows agents to maximise their extrinsic rewards accumulated until the end of their lifetimes. This long-term lifetime objective allows our learned intrinsic reward to generate systematic multi-episode exploratory behaviour. Through proof-of-concept experiments, we elucidate interesting forms of knowledge that may be captured by a suitably trained intrinsic reward such as the usefulness of exploring uncertain states and rewards.

Under review as a conference paper at ICLR 2020
WHAT CAN LEARNED INTRINSIC REWARDS CAPTURE?
Anonymous authors Paper under double-blind review
ABSTRACT
Reinforcement learning agents can include different components, such as policies, value functions, state representations, and environment models. Any or all of these can be the loci of knowledge, i.e., structures where knowledge, whether given Iearned, can be deposited and reused. Regardless of its composition, the objective of an agent is behave as to maximise the sum of suitable scalar function of state: the reward. As far as the learning algorithm is concerned, these rewards are typically given and immutable. In this paper we instead consider the proposition that the reward function itself may be good locus of knowledge. This is con- sistent with common use, in the literature, of hand-designed intrinsic rewards improve the learning dynamics of an agent. We adopt the multi-lifetime setting of the Optimal Rewards Framework, and propose to meta-learn an intrinsic reward function from experience that allows agents to maximise their extrinsic rewards accumulated until the end of their lifetimes. Rewards as a locus of knowledge provide guidance on "what" the agent should strive to do rather than on knowl- edge of "how" the agent should behave that is more directly captured policies or value functions for example. Thus, our focus here is on demonstrating the follow- ing: (1) that it is feasible to meta-learn good reward functions, (2) to show that the learned reward functions can capture interesting kinds of "what" knowledge, and (3) that because of the indirectness this form of knowledge the learned reward functions can generalise other kinds of agents and to changes in the dynamics of the environment.
Reinforcement learning agents can store knowledge in their policies, value functions, state represen- tations, and models of the environment dynamics. These components can be the loci of knowledge in the sense that they are structures in which knowledge, either learned from experience by the agent's s algorithm or given by the agent-designer, can be deposited and reused. The objective of the agent defined by reward function, and the goal is to learn to act so as to optimise cumulative rewards. In this paper we consider the proposition that the reward function itself is good locus of knowledge. This is unusual in that most prior work treats the reward given and immutable, at least as far the learning algorithm is concerned. At the same time, especially in challenging ceinforeeeent-earming problems, agent designers do find it convenient to modify the reward function given to the agent to facilitate learning. It is therefore useful to distinguish between two kinds of reward functions (Singh al. 2010): extrinsic rewards define the task and capture the designer's preferences over agent be- haviour, whereas intrinsic rewards serve as helpful signals to improve the learning dynamics of the agent. Intrinsic rewards are typically hand-designed and then often added to the immutable extrinsic rewards to form the reward optimised by the agent.
Most existing work on intrinsic rewards falls into two broad categories: task-dependent and task- independent. Both are typically designed by hand. Hand-designing mask-dependen rewards can be fraught with difficulty as even minor misalignment between the actual reward and the intended bias can lead to unintended and sometimes catastrophic consequences (Clark & Amodei] 2016)- Tusk-independent intrinsic rewards are also typically hand-designed, often based on an intuitive understanding of animal/human behaviour or on heuristics desired exploratory behaviour. It can, however, be hard to match such inskiindepnndent intrinsic rewards to the specific learning dynamics induced by the interaction between agent and environment. The motivation for this paper is our interest in the comparatively under-explored possibility of learned (not hand (enddee)g task- dependent intrinsic rewards (see Zheng et al. 2018 for previous work).
We emphasise that it is not our objective to show that rewards are a better locus of learned knowledge than others; the best locus likely depends on the kind of knowledge that is most useful in given

---

Under review as a conference paper at ICLR 2020
task. In particular, knowledge captured in rewards provides guidance on "what" the agent should strive to do while knowledge captured in policies provides guidance on "how" an agent should behave. Knowledge about "what" captured in rewards is indirect and thus slower to make impact on behaviour because it takes effect through learning. while knowledge about "how" can directly have immediate impact on behaviour. At the same time, because of its indirectness the former can generalise better to changes in dynamics and learning architectures. Therefore, instead of comparing different loci of knowledge, the purpose of this paper to show that it is feasible to capture useful learned knowledge in rewards and to study the kinds of knowledge that can be captured.
How should we measure the usefulness of learned reward function? Ideally, we would like to measure the effect the learned reward function has on the learning dynamics. Of course, learning happens over multiple episodes, indeed happens over an entire lifetime. Therefore, we choose lifetime return, the cumulative extrinsic reward obtained by the agent over its entire lifetime, as the main objective. To this end, we adopt the multi-lifetime setting of the Optimal Rewards Frame- work (Singh et al. 2009) in which agent is initialised randomly at the start of each lifetime and then faces a stationary or non-stationary task drawn from some distribution. In this setting, the only knowledge that is transferred across lifetimes is the reward instead of the policy. Specifically, the goal is to learn a single intrinsic reward function that, when used to adapt the agent's policy using a standard episodic RL algorithm, ends up optimising the cumulative extrinsic reward over its lifetime.
In previous work, good reward functions were found via exhaustive search, limiting the range of applicability of the framework. Here, we develop more scalable gradient-based method (Xu al.) 2018c) for learning the intrinsic rewards by exploiting the fact the interaction between the pol- icy update and the reward function differentiable (Zheng al.. 2018). Since it is infeasible to backpropgate through the full computation graph that spans across the entire lifetime, we truncate the unrolled computation graph of learning updates up to some horizon. However, we handle the long-term credit assignment by using a lifetime value function that estimates the remaining lifetime return, which needs to take into account changing policies. Our main scientific contributions are a sequence of empirical studies on carefully designed environments that show how our learned in- trinsic rewards can capture interesting regularities in the interaction between a learning agent and an environment sampled from a distribution, and how the learned intrinsic reward can generalise to changed dynamics and agent architectures. Collectively, our contributions present an effective approach to the discovery of intrinsic rewards that can help agent optimise the extrinsic rewards collected in a lifetime.
RELATED WORK
Hand-designed Rewards There is a long history of work on designing rewards to accelerate learn- ing in reinforcement learning (RI.). Reward shaping aims to design task-specific rewards towards known optimal behaviours, typically requiring domain knowledge. Both the benefits (Randlov & Alstrm 1998, Ng et al. 1999. Harutyunyan et al. 2015) and the difficulty (Clark & Amodei| 2016) of task-specific reward shaping have been studied. On the other hand, many intrinsic rewards have been proposed to encourage exploration, inspired by animal behaviours. Examples include predic-
tion error (Schmidhuber 1991b Gordon & Ahissar 2011
Mirolli & Baldassarre 2013 Pathak et al.)
2017
Schmidhuber] 1991a), surprise (Itti & Baldi 2006). weight change (Linke et al. 2019)), and
state-visitation counts (Sutton| 1990 Poupart et al.
2006 Strehl & Littman| 22088) Bellemare al.)
2016 Ostrovski et al. 2017). Although these kinds of intrinsic rewards are not domain-specific,
they are often not well-aligned with the task that the agent tries solve, and ignores the effect on the agent's 's learning dynamics. In contrast, our work aims to learn intrinsic rewards from data that take into account the agent's learning dynamics without requiring prior knowledge from a human.
Rewards Learned from Data There have been a few attempts to learn useful intrinsic rewards from data. The optimal reward framework (Singh et al. 2009) proposed to learn an optimal reward function that allows agents to solve a distribution of tasks quickly using random search. We revisit this problem in this paper and propose a more scalable gradient-based approach. Although there have been follow-up works (Sorg et al. 2010 Guo et al. (016)) that uses a gradient-basee method, they consider non- non-parameters policy using Monte- Carlo Tree Search (MCTS). Our work is closely related to LIRPG (Zheng et al., 2018) which proposed a meta meta-gradien method to learn intrinsic rewards. However, LIRPG considers single task in single lifetime with a myopic episode return
2

---

Under review as a conference paper at ICLR 2020
Lifetime with tusk T p(T)
Intrinsic Reward
7/
0₁ Ppicodes
O2
11
1
ON
Gliso
Fppiodde
Figure I: Illustration of the proposed intrinsic reward learning framework. The intrinsic reward y' is used to update the agent's parameter 0g throughout its lifetime which consists of many episodes. The goal is to find the optimal intrinsic reward parameters 7)" across many lifetimes that maximises the lifetime return (Gllfe given any randomly initialised agents and possibly non-stationary tasks drawn from some distrihution p(T).
objective, which limited in that does not allow exploration across episodes or generalisation to different agents.
Meta-learning for Exploration Meta-learning (Schmidhuber al., 1996; Thrun & Pratt 1998) has recently received considerable attention in RL. Recent advances include few-shot adapta- tion (Finn et al.] 2017a). few-shot imitation (Finn et al. 201 7b
Duan al., 2017]. model adap- tation (Clavera et al. 2018), and inverse RL (Xu et al. 201 8a). In particular, our work is closely related to the prior work on meta-learning good exploration strategies (Wang et al.] 2016: Duan et al.) 2016 Stadie et al., 2018 Xu et al., 2018b) in that both perform temporal credit assignment across episode boundaries by maximising rewards accumulated beyond an episode. Unlike the prior work that aims learn exploratory policy, our framework indirectly drives exploration via reward function which can be reused by different learning agents as we show in this paper (Section
Meta-learning of Agent Update There have been few studies that directly meta-learn how to update the agent's s parameters via meta-pare meta-parameters including discount factor and returns (Xu et al.] 2018c), auxiliary tasks (Schlegel et al. [2018] Veeriah et al.) (019), unsupervised learning rules (Metz et al. 2019), and RL objectives (Chebotar et al. (2019). Our work also belongs to this category in that our preta-parameters are the reward function used the agent's update. In particular, our multi- lifetime formulation is similar to ML (Chebotar et al. 2019). However, we consider the long term lifetime return as objective perform cross-episode temporal credit assignment as opposed to the myopic episodic objective in 3L
2
THE OPTIMAL REWARD PROBLEM
We first introduce some terminology.
Agent: A learning system interacting with an environment. On each step the agent seleets an action a and receives from the environment an observation 81+1 and an extrinsic reward r(+1 defined by task T. The agent chooses actions based on a policy aa (se) parameterised by 0. Episode: A finite sequence of agent-environment interactions until the end of the episode defined by the task. An episode return defined as: G = >'in=" Tip-1 7'r.-1, where is a discount factor, and the random variable Tep gives the finite number of steps until the end of the episode.
Lifetime: finite sequence of agent-environment interactions until the end of training de- fined by an agent-designer which can include multiple episodes. The lifetime return is Chife
TT-1 7're+1. where 7 is discount factor, and T is the number of steps in the lifetime.
Intrinsic reward: A reward function (T(+++)) parameterised by 7) where T (5o, Go, F1, d1, Tt, de, 8t) is lifetime history with (binary) episode terminations de-
The Optimal Reward Problem (Singh et al. 2010), illustrated in Figure T aims to learn the param- eters of the intrinsic reward such that the resulting rewards achieve a learning dynamic for an RI. agent that maximises the lifetime (extrinsic) return on tasks drawn from some distribution. Formally, the optimal reward function is defined as:
ry = arg max (n) arg max Be,~0.TTP(T) (7))), [Glife]]]
(1)
"
"
where θ and p(T) are an initial policy distribution and a distribution over possibly non-stationary tasks respectively, and Glife Σx- r'reta lifetime return. The likelihood of lifetime history
3

---

Under review as a conference paper at ICLR 2020
Algorithm Learning intrinsic rewards across multiple lifetimes via meta-gradient
Input: p(T): Task distribution, G: Candomly-initialin policy distribution
Initialise intrinsic reward function y and lifetime value function Ø repeat
Initialise task T ~ p(T) and policy 0 ~ θ while lifetime ended do
Oo 4- 0
for k = 1......,.. do
Generate trajectory using Th.-1 Update policy 02 Oz- aVa. ,0 using intrinsic rewards rn (Eq. end for
Update intrinsic reward function 7 using Eq Update lifetime value function @ using Eq. 4
0 ON
end while until 17 converges
Te, (dr(s))p(de++ Tr+ 81411 18,, az), where 0, - f(; 1,:0-1,70 7) is a policy parameter as updated with update function f. which is policy gradient in this paper Note that the optimisation of 77 spans multiple lifetimes, each of which can span multiple episodes.
is (T)00) p(50) IT p(50)((("""
Using the lifetime return Glife as the objective instead of the conventional episodic return G allows exploration across multiple episodes as long the lifetime return is maximised in the long run. In particular, when the lifetime is defined as a fixed number of episodes, we find that the lifetime return objective sometimes more beneficial than the episodic return objective even in terms of the episodic return performance measure. However, different objectives (e.g., final episode return) can be considered depending on the definition of what good reward function is.
3 META- INTRINSIC REWARD
We propose a meta-gradient approach Xu et al. 80 Zheng et al. 2018) to solve the optimal reward problem. At high-level, we sample new task and a new random policy parameter at each lifetime iteration. We then simulate an agent's 's lifetime by updating the parameter 0 using an intrinsic reward function ry (Section 3.T) with policy gradient (Section 3.2). the meantime, we compute the meta- grtadggradiett& by taking into account the effect of the intrinsic rewards on the pol- icy parameters to update the intrinsic reward function with a lifetime value function (Section 3.3. Algorithm gives an overview of our algorithm. The following sections describe the details.
3.1 INTRINSIC REWARD AND LIFETIME VALUE FUNCTION ARCHITECTURES
The intrinsic reward function is a recurrent neural network (RNN) parameterised by 7. which pro- duces scalar reward on arriving in state s, by taking account the history an agent's lifetime (sp, 4g, T1, d1, 81, Tz, de, 8.). We claim that giving the lifetime history across episodes as input is crucial for balancing exploration and exploitation, for instance by capturing how frequently certain state is visited to determine an exploration bonus reward. The lifetime value function is a separate recurrent neural network parameterised by ó, which takes the same inputs as the intrin- sic reward function and produces scalar value estimation of the expected future return within the lifetime.
3.2 POLICY UPDATE (0)
Each agent interacts with environment and a task sampled from a distribution T p(T). How- ever, instead of directly maximising the extrinsic rewards defined by the task, the agent maximises
We assume that the policy parameter is updated after each time-step throughout the paper for brevity. However, the parameter he updated less frequently in practice.
4

---

Under review as a conference paper at ICLR 2020
the intrinsic rewards (ri,) by using policy gradient (Williams 1992
Sutton et al. 2000):
T
(0) = Eg
7 (re((Te1))
VoJ,(0) = Es
GoP n.e
Vo log (o(as)))
(2)
where ry (T++1) the intrinsic reward at time t, and :
94 =
T-p-1
k=t " (T<+1) is the return of
the intrinsic rewards accumulated over episode with discount factor 7
3.3 INTRINSIC REWARD (77) AND LIFETIME VALUE FUNCTION (Ø) UPDATE
To update the intrinsic reward parameters 7, directly take meta-gradient ascent step using the overall objective (Equation]) D Specifically, the gradient is (see Appendix for derivation):
(n) B0,~0,7 Txpp)(
E ET.p(TT1100))
G. Glife log Te, (1 V,,
(3)
where Glife= Σ1= k= " rk+1 is a lifetime return based the extrinsic rewards of task T with discount factor y. The chain rule is used to get the meta-gradient (V,00) as in previous work (Zheng jet al. 2018). The computation graph of this procedure is illustrated in Figure [
Computing the true meta-gradient in Equation 3 requires backpropagation through the entire life- time, which is infeasible as each lifetime can involve more than thousands of policy updates. To partially address this issue, we truncate the meta-gradient after N policy updates but approximate the lifetime return Glife, x Gife using lifetime value function V²(T) parameterised by ø, which is learned using a temporal difference learning from n-step trajectory:
Glite d Σ * + (o((rn)
δ Φ + (G Vo((T)) (4)
In our empirical work, we found that the lifetime value estimates were crucial to allow the intrinsic reward to perform long-term credit assignments across episodes.
3.4 CONNECTION TO STANDARD RL FRAMEWORKS
The policy learning problem specified in Section 3.2 deviates from the standard Markov Decision Process (MDP) framework because the intrinsic reward function is function of the lifetime history rather than function of states or state-action pairs. Consequently, from the memoryless policy's perspective, the rewards are non non-stationary. However, we can also view the combination of the intrinsic reward function and the policy as a joint lifetime-history-baase and 0 (see derivation in Appendix A). From this perspective, the overall learning problem specified in Scction 3.3 can be formulated as an MDP with history as state (recall, we use RNNs for the intrinsic reward function). As result, standard camporal-differenet learning methods applicable to learning lifetime value functions.
policy parameterised by
EMPIRICAL INVESTIGATIONS: FEASIBILITY AND USEFULNESS
We present the results from empirical investigations in sections. For the results in this section, the experiments and domains are designed to answer the following research questions:
What kind of knowledge can learned by the intrinsic reward?
How does the distribution of tasks drive the form of the intrinsic reward?
What is the benefit of the lifetime return objective over the episode return?
When is it important to provide the lifetime history as input to the intrinsic reward?
We investigate these research questions in various grid-world domains illustrated Figure 2 For each domain, we trained an intrinsic reward function across many lifetimes and evaluated it by training an agent using the learned reward. We implemented the following baselines.
Extrinsic-EP: A policy is trained with extrinsic rewards maximise the episode return.

---

Under review as a conference paper at ICLR 2020
A
(a) Empty Rooms
(b) ABC
(c) Key-Box
Figure 2: Illustration of domains. (a) The agent needs find the goal location which gives positive reward, but the goal is not visible to the agent, (b) Each object (A, B, and C) gives rewards. (c) The agent is required to first collect the key and visit one of the boxes (A, B, and C) to receive the corresponding reward. All objects are placed to random locations after every episode.
Emply Rooms
Random ABC
Key- Bex-BBxx
Non-stationary ABC
Episode return
0.8
0.4
0.5
0.6
0.3
0.4
1.5
1.0
M
Learned (ours)
Extrinsic: EP Extrinsic-LIFE Count-based
0.3
0.2
0.5
0.6
0.2
0.4
0.1
ICM Near-optimal
0.2
0.1
100 150 200
10
20
40
2500
5000
250 500 7501000
Num episodes
Figure 3: Evaluation of different reward functions averaged over 30 seeds. The learning curves show agents trained with intrinsic reward (blue), with the extrinsic reward with the episodic return objective (orange) the lifetime return objective (brown), with count- count-based exploration reward (green). The dashed line corresponds to a hand- designed near-optimal exploration strategy.
Extrinsic-LIFE: A policy is trained with extrinsic rewards to maximise the lifetime return.
Count-based (Strehl & Littman 2008 A policy is trained with extrinsic rewards and count- based exploration bonus rewards.
ICM (Pathak et al. 2017): p: A policy is trained with extrinsic rewards and curiosity rewards based inverse dynamics model.
Note that these baselines, unlike the learned intrinsic rewards, do not transfer any knowledge across different lifetimes. Throughout Sections [4. [.44] we focus on analysing what kind of knowledge is learned by the intrinsic reward depending the nature of environments. We diseuss the benefit of using the lifetime return and considering the lifetime history when learning the intrinsic reward in Section 4.5
The details of implementation and hyperparameters described in Appendix B
4.1 EXPLORING UNCERTAIN STATES
We designed 'Empty Rooms' (Figure 2a whether the intrinsic reward learn to encourage exploration of uncertain states like novelty-based exploration methods. The goal is to visit an invisi- ble goal location, which is fixed within each lifetime but varies across lifetimes. Episode terminates when the goal is reached. Each lifetime consists of 200 episodes. From the agent's perspective, its policy should visit the locations suggested by the intrinsic reward. From the intrinsic reward's perspective. it should encourage the agent to go to unvisited locations to locate the goal, and once the goal located to exploit that knowledge for the rest of that lifetime.
Figure shows that our learned intrinsic reward was more efficient than extrinsic rewards and count- based exploration when training new agent. We observed that the intrinsic reward learned two interesting strategies visualised in Figure 4 While the goal is not found, it encourages explo- ration of unvisited locations, because it learned the prior that there exists a rewarding goal location somewhere. Once the goal is found the intrinsic reward encourages the agent to exploit it without further exploration, because it learned that there is only one goal. This result shows that curiosity about uncertain states naturally emerge when various states can be rewarding in domain, even when the rewarding states are fixed within an agent's 's lifetime.

---

Under review as a conference paper at ICLR 2020
Room instance
(b) Intrinsic (ours)
Extrinsic
(d) Count-based
ICM
Figure 4: Visualisation of the first 3000 steps of an agent trained with different reward functions in Empty Rooms. (a) The blue and yellow squares represent the agent and the hidden goal, respectively. (b) The learned reward encourages the agent to visit many locations if goal not found (top). However, when goal found early, the intrinsic reward makes the agent exploit it without further exploration (bottom). (c) An agent trained only with extrinsic rewards explores poorly. (d-c) Both the count-based and ICM rewards tend to encourage exploration (top) but hinders exploitation when the goal found (bottom).
Visit A
A-
Episode
Episode
Hist
Episade
Visit
B-4L
Episode
Episode 3
Figure 5: Visualisation the learned intrinsic reward in Random ABC, where extrinsic rewards B, and C are 0.2, -0.5, and 0.1 respectively. Each figure shows the sum of intrinsic rewards for a trajectory towards each object (A, B, and C). In the first episode, the intrinsic reward encourages the agent to explore A. In the second episode, the intrinsic reward encourages exploring C if A is visited (top) vice versa (bottom). In episode 3, after both A and are explored, the intrinsic reward encourages to revisit (both top and bottom).
4.2 EXPLORING UNCERTAIN OBJECTS AND AVOIDING HARMFUL OBJECTS
In the previous domain, considered uncertainty of where the reward (or goal location) is. We now consider dealing with uncertainty about the value different objects. the 'Random ABC' environment (see Figure 2b), for each lifetime the rewards for objects A, B, and C uniformly sampled from [- 1, 1]. 0.5, 0), and [0, 0.5] respectively but are held fixed within the lifetime. A and C have uncertain rewards. hence require systematic exploration (first go to one and then the other), and 3) once is determined better, exploit that knowledge by encouraging the agent to repeatedly go
good intrinsic reward should learn that: 1) should be avoided,
which the two A to that object for the rest of the lifetime.
ssoows that the agent learned near-optimal suploration-and--ee--e-pppp the learned intrinsic reward. Note that the agent cannot pass information about the reward for ob- jects across episodes, usual in reinforcement learning. The intrinsic reward can propagate such information across episodes and help the agent explore exploit appropriately. We visualised the learned intrinsic reward different actions sequences Figure The intrinsic rewards encourage the agent to explore towards A and C in the first few episodes. Once A and C are explored. the agent exploits the largest rewarding object. Throughout training, the agent is discouraged to visit B through negative intrinsic rewards. These results show that avoidance and curiosity about uncertain objects can potentially emerge the environment has various or fixed rewarding objects.
method with

---

Under review as a conference paper at ICLR 2020
1.2
Episode
0.6
Intrinsic Extrinsic
0.4
0.0
470
480
490
500
51.0
520
530
470
480
490
500
510
520
530
Episodes
Episodes
Figure 6: Visualisation the agent's intrinsic and extrinsic rewards (left) and the entropy its policy (right) on Non-stationary ABC. The task changes at 500th episode (dashed vertical line). The intrinsic reward gives a negative reward even hefore the task changes (green rectangle) and makes the policy less peaky (entropy increases). As result, the agent quickly adapts to the change.
Empty Rooms
Random ABC
Key-Box
Non-stationary ABC
ABC
Episode retum
0.6
1.0
0.8
0.4
0.2
0.3
0.10-
05
0
0.30
0.05
0.5
0.0
LS LSTM-Episode FF-Episede
LS LSTM-Lifetime
0.2
-0.
0.00
-0.5
-0.
10.10
-1.0
50
100 150 200
0
10
20
30
40
50
2500
5000
250
500
750 1000
Num episodes
Figure 7: Evaluation of different intrinsic reward architectures and objectives. For 'LSTM' the reward network has an LSTM taking the lifetime history as input. For "FF' a feed forward reward network takes only the current time-step. 'Lifetime' and 'Episode' means the lifetime and episodic return as objective respectively.
4.3 EXPLOITING INVARIANT CAUSAL RELATIONSHIP
To see how the intrinsic reward deals with causal relationship between objects, we designed 'Key- Box', which is similar to Random ABC except that there is key in the room (sce Figure 2c). The agent needs to collect the key first to open one of the boxes (A, B, and C) and receive the corre- sponding reward. The rewards for the objects are sampled from the same distribution as Random ABC. The key itself gives a neutral reward of O. Moreover, the locations of the agent, the key, and the boxes randomly sampled for each episode. As result, the state space contains more than 3 billion distinct states and thus is infeasible to fully enumerate. Figure 3 shows that learned intrinsic reward leads to near-optimal exploration. The agent trained with extrinsic rewards did not learn to open any box. The intrinsic reward captures that the key is necessary to open any box, which is true across many lifetimes of training. This demonstrates that the intrinsic reward can capture causal relationships between objects when the domain has this kind of invariant dynamics.
4.4 DEALING WITH NON-STATIONARITY
We investigated how the intrinsic reward deals with non-stationarity of tasks within a lifetime in our NNnn-statonaayy ABC' environment. Rewards are as follows: for A is either 1 or -1, for B is -0.5, for C is the negative value of the reward for A. The rewards of A and C are swapped every 250 episodes. Each lifetime lasts 1000 episodes. Figure 3 shows that the agent with the learned intrinsic reward quickly recovered its performance when the task changes, whereas the baselines take more time to recover. Figure agent react to the changing rewards. Interestingly, the intrinsic reward has learned to prepare for the change by giving negative rewards to the exploitation policy of the agent a few episodes before the task changes. other words, the intrinsic reward starts to discourage the agent to commit to the current best rewarding object, thereby increasing entropy in the current policy in anticipation of the change, eventually making it casier to adapt quickly. This shows that the intrinsic reward can cap- ture the (regularly) repeated non-stationarity across many lifetimes and make the agent intrinsically motivated not to commit too firmly to policy, in anticipation of changes in the environment.
6 shows how the learned intrinsic reward encourages the learning
4.5 ABLATION STUDY
To study relative benefits of the proposed technical ideas, we conducted an ablation study 1) by replacing the long-term lifetime return objective (Glife) with the episodic return (GT) and 2) by re- stricting the input of the reward network to the current time-step instead of the entire lifetime history. ssowss Fisurs//shows that the lifetime history was crucial to achieve good performance. This is reasonable because all domains require some past information (e.g., current object rewards in Random ABC,
8

---

Under review as a conference paper at ICLR 2020
Empty Rooms
Random ABC
Key-Box
Non-stationary ABC
Episode return
1.0
0.4
1.0
1.5
Intrinsic (ours)
0.8
0.3
0.2
0.6
0.4
0.1
0.1
-0.5
0.0
MAML
0.2
-0.1
0.0
-1.0-
-0.2
0.1
-1.5
150
2500
5000
250
7500
Num episodes
Figure 8: Comparison policy transfer methods.
visited locations in Empty Rooms) to provide useful exploration strategies. the lifetime return objective was beneficial on Random ABC, Non-stationary ABC, and Key-Box. These domains require exploration across multiple episodes in order to find the optimal policy. For example, collecting uncertain object (e.g.. object A in Random ABC) is necessary episode terminates with a negative reward. The episodic value function would directly penalise such an under-performed exploratory episode when computing meta-gradient which prevents the intrinsic reward from learning to encourage exploration across episodes. On the other hand, such behaviour can be encouraged by the lifetime value function as long as it provides useful information to maximise the lifetime return in the long term.
also shown that
if the
5 EMPIRICAL INVESTIGATIONS: GENERALISATION VIA REWARDS
As noted above, rewards capture knowledge about what an agent's goals should be rather than how it should behave. At the same time, transferring the latter in the form of policies is also feasible in our domains presented above. Here we confirm that by implementing and presenting results for the following two meta-learning methods:
MAML (Finn et al., 2017a): A policy meta-learned from a distributions of tasks such that it can adapt quickly to the given task after few parameter updates.
RL2 (Duan et al. 2016 Wang et al. 2016): An LSTM policy unrolled over the entire
lifetime to maximise the lifetime return, which is pre-trained on a distributions of tasks.
Although all the methods we implemented including ours are designed to learn useful knowledge from distribution of tasks, they have different objectives. Specifically, the objective of our method to learn knowledge that is useful for training "randomly-initiaaiee" policies" by capturing "what to do", whereas the goal of policy transfer methods is to directly transfer useful policy for fast task adaptation by transferring "how to do" knowledge. reuse pre-trained policies instead of restarting from random policy and learning using the learned rewards given a new task. Figure 8 indeed shows that RL.² performs better than our intrinsic reward approach. It is also shown that MAML and RL achieve good performance from the beginning, as they have already learned how to navigate the grid worlds and how to achieve the goals of the tasks. In our method, the other hand, the agent starts from random policy and relies the learned intrinsic reward which indirectly tells it what to do. Nevertheless, our method outperforms MAML and achieves comparable asymptotic performance to RL?
fact, it can be more efficient to transfer and
5.1 GENERALISATION TO DIFFERENT AGENT- THENT-ENVTRONMENE INTERFACES
In fact, our method can be interpreted as an instance of RL2 with a particular decomposition of pa- rameters (O and 17), which uses policy gradient as a recurrent update (see Figure While this modu- lar structure may not be more beneficial than RL when evaluated with the same agent-enviromment interface, such a decomposition provides clear semantics of each module: the policy (0) captures "how to do" while the intrinsic reward (1)) captures "what to do". and this enables interesting kinds of generalisations as we show below. Specifically, we show that "what" knowledge captured by the intrinsic reward can be reused by many different learning agents as follows.
Generalisation to unseen action spaces We first evaluated the learned intrinsic reward on new action spaces. Specifically, the intrinsic reward was used to train new agents with either 1) permuted actions, where the semantics of left/right and up/down are reversed, or 2) extended actions, with additional actions that move diagonally. Figure 9a shows that the intrinsic reward provided useful

---

Under review as a conference paper at ICLR 2020
Randorn ABC
Random ABC
Permuted Random ABC
0.5
0.5
0.5
0.4
0.4
0.4
Episode Return
Episode Return
0.2
0.1
0.0
Original
0.1
AC-Intrinsic Q Intrinsic Extrinsic
0.1
Ours
Permuted
0.0
0.0
RL'
Extended
-0.2
-0.1
-0.1
10
20
30
40
50
50
100
150
200
10
20
30
40
50
Action space
(b) Algorithm
(c) Comparison policy baselines
Figure 9: Generalisation to new agent-environment interfaces in Random ABC. (a) 'Permuted" agents have different action semantics. 'Extended' agents have additional actions. (b) 'AC-Intrinsic' the original actor- critic agent trained with the intrinsic reward. "Q-Intrinsic" is a Q-Ieaming agent with the intrinsic reward learned from actor-critic agents. 'Q- "Q-Extrinsic' is the Q-laarnin agent with the extrinsic reward. (c) shows the performances of the policy transfer baselines with permuted actions during evaluation.
rewards to new agents with different actions, even when these were not trained with those actions. This is possible because the intrinsic reward assigns rewards to the agent's state changes rather than its actions. In other words, the intrinsic reward captures "what to do", which makes it possible to generalise to new actions, as long as the goal remains the same. On the other hand, it is unclear how to generalise RL2 and MAML in this way.
Generalisation to unseen learning algorithms We further investigated how general the knowl- edge captured by the intrinsic reward is by evaluating the learned intrinsic reward on agents with dif- ferent learning algorithms. In particular, after training the intrinsic reward from actor-critic agents, we evaluated it by training new agents through Q-learning while using the learned intrinsic reward as denoted by 'Q-Intrinsic' in Figure Interestingly, it turns out that the learned intrinsic reward is general enough to be useful for Q-learning agents, even though it was trained for actor-critic agents. Again, it is unclear how to generalise RL and MAML in this way.
Comparison to policy transfer While it wasn't possible to apply the learned policy from RL and MAML when extended the action space and when changed the learning algorithm, can do so when we keep the same number of actions and just permute them. As shown in Figure 9c| both RL and MAML generalise poorly when the action space is permuted for Random ABC, because the transferred policies are highly biased to the original action space. Again, this result highlights the difference between "what to do" knowledge captured by our approach and "how do" knowledge captured by policies.
6 CONCLUSION
We revisited the optimal reward problem (Singh et al. 2009) and proposed a more scalable gradient- based method for learning intrinsic rewards. Through several proof-of-coneee concetept experiments, showed that the learned non-stationary intrinsic reward can capture regularities within a distribution of environments over time, within non-stationary environment. As result, they were capable of encouraging both exploratory and exploitative behaviour across multiple episodes. In addition, some lask-independent notions of intrinsic motivation such curiosity emerged when they were effective for the distribution over tasks across lifetimes the agent was trained on. We also showed that the learned intrinsie rewards can generalise to different agent-environment interfaces such as different action spaces and different learning algorithms, whereas policy transfer methods fail to generalise. This highlights the difference between the "what" kind knowledge captured by rewards and the "how" kind of knowledge captured by policies. The flexibility and range of knowledge captured by intrinsic rewards in our proof-of -o-ccctt experiments encourages further work towards combining different loci of knowledge to achieve greater practical benefits.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. CoRR, wwsp/6/0.001e.. 868, 2016.
/tRpptpp//axxii.oot hbsg://bs//6666 01868
Yevgen Chebotar, Artem Molchanov, Sarah Bechtle, Ludovic Righetti, Franziska Meier, and Gau- rav S. Sukhatme, Meta-leaming via learned loss. CoRR, abs/1906.05374, 2019. URL http:
//arxiv..rggaab///1333 05374]
Jack Clark and Dario Amodei. Faulty reward functions in the wild. CoRR, 2016. URL https: //blog.openal. com/
Ignasi Clavera, Anusha Nagabandi, Ronald S. Fearing, Picter Abbecl, Sergey Levine, and Chelsea Finn. Learning to adapt: Meta-learning for model-based control. ArXiv, abs/1 803. 1347, 2018.
abs/1803.11
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.0277,, 2016.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in nearal information processing systems, pp. 1087-1098, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126-1135. JMLR. org, 2017a.
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imita- tion learning via meta-learning. arXiv preprint arXiv: 1709.04905 2017b.
G. Gordon and E. Ahissar. Reinforcement active learning hierarchical loops, 2011.
Xiaoxiao Guo, Satinder Singh, Richard Lewis, and Honglak Lee. Deep learning for reward design to improve monte carlo tree search in atari games. arXiv preprint arXiv: 1604.00055, 2016.
Anna Harutyunyan, Sam Devlin, Peter Vrancx, and Ann Nowé. Expressing arbitrary reward func- tions as potential-based advice. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Laurent Itti and Pierre F. Baldi. Bayesian surprise attracts human attention. In Y. Weiss, B. Schölkopf, and J. C. Platt (eds.). Advances in Neural Information Processing Sys- tems 18, pp. 547-554. MIT Press, 2006. URL http::/ nips cttccc///pppe paper/ 2822 bayes an- surpris se
wwtrrriseetttttttteee attracts
autan-attetiionnn pdd
Cam Linke, Nadia M. Ady, Martha White, Thomas Degris, and Adam White. Adapting behaviour
via intrinsic reward: A survey and empirical study. CoRR, abs/1906.07865, 2019. URL http:
//axxiv.oq//bbe wttp////aaaaaaaaaoaeaaaa/
Luke Metz, Niru Maheswaranathan, Brian Cheung, and Jascha Sohl-Dickstein. Meta-lcarning up- date rules for unsupervised representation learning. In International Conference Learning Representations, volume 8, 2019.
Marco Mirolli and Gianluca Baldassarre. Functions and Mechanisms of Intrinsic Motiva- tions, pp. 49-72. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013. ISBN 978-3- 642-32375-1.
doi:
URL httts:://oi.org/100
978-3- 22332225222222
Andrew Y. Ng. Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Confer- Machine Learning, ICML '99, pp. 278-287, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc. ISBN 1-55860-612-2. URL http:pp://ddl..ccooor/rtttttteee
cfm?id=645522 657613
Georg Ostrovski, Mare G. Bellemare, Aäron den Oord, and Rémi Munos. Count-based explo- ration with neural density models. CoRR, abs/1703.013/ 2017. "ollattp://axxxi
abs 1103. 01310
11

---

Under review as a conference paper at ICLR 2020
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosityydrrive explorraai by self-supervised prediction. CoRR, foss/1005.053633 2017. URL bttp://arxiv.orx/ abs/
1705. 1705.05363
Pascal Poupart, Nikos Vlassis, Jesse Hoey, and Kevin Regan. An analytic solution to discrete bayesian reinforcement learning. In Proceedings of the 23rd International Conference Ma- chine Leurning, ICML '06, pp. 697-704, New York, NY, USA, 2006. ACM. ISBN 1-59593-383 URL http: /oooi doi amm.
2. doi: 10.1145/1 143844.1 1143932.
1143932
Jette Randlov and Preben Alstrm. Learning to drive bicycle using reinforcement learning and shaping. pp. 463-471, 01 1998.
Matthew Schlegel, Andrew Patterson, Adam White, and Martha White. Discovery of predictive rep- resentations with network of general value functions, 2018. URL /RLpsstss////peeeeeaa de GZ0Z
Jüergen Schmidhuber, Jieyu Zhao, and MA Wiering. Simple principles of metalearning. Technical report IDSIA, 19:1-23, 1996.
Jürgen Schmidhuber, Curious model-building control systems. In In Proc. International Joint Con- ference on Neural Networks, Singapore, pp. 1458-1463. IEEE, 1991a.
Jürgen Schmidhuber, A possibility for implementing curiosity and boredom in model-building neu- ral controllers, 1991b.
Satinder Singh, Richard L. Lewis, and Andrew G. Barto. Where do rewards come from?, 2009.
Satinder Singh, Richard I.. Lewis, and Andrew G. Barto. Intrinsically motivated reinforcement learning: An evolutionary perspective. In L. K. Saul, Y. Weiss, and L. Bottou (eds.), IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT. 2010.
Jonathan Sorg, Richard L Lewis, and Satinder Singh. Reward design via online gra- dient ascent. In D. Lafferty, C. K. I. Williams, J. Shawe-Taylor, R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp- 2190-2198. Curran Associates. Inc., 2010. URL http::/papers..r peappers.nisss. htpps.ccppppp.//////e r146-rewarddder
onlinenine gradient ascent. .dd pddf
Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcement learning. arXiv preprint arXiv:1803 01 118, 2018
Alexander L Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 1408):1309-1331, 2008.
Richard S. Sutton. Integrated architectures for learning, planning, and reacting based on approx- imating dynamic programming. In In Proceedings of the Seventh International Conference on Machine Learning, pp. 216-224. Morgan Kaufmann, 1990.
Richard S Sutton, David A McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057-1063, 2000.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. In Learning to learn, pp. 3-17. Springer, 1998.
Vivek Veeriah, Matteo Hessel, Zhongwen Xu, Richard Lewis, Janarthanan Rajendran, Junhyuk Oh, Hado van Hasselt, David Silver, and Satinder Singh. Discovery of useful questions auxiliary tasks. CoRR, abss1909.044.. 2019.
wrg. abs 1909. 04607
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew M Botvinick. Learning to reinforcement learn. ArXiv, abs/161 1.05663, 2016.
12

---

Under review as a conference paper at ICLR 2020
Ronald J. Williams. Simple statistical trudient-following algorithms for connectionist reinforce-
ment learning. Mach. Learn., 8(3-4)2229-2) May 1992. ISSN 0885-6125. doi: 10.1007/ BF00992696.
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning prior over intent via meta-inverse reinforcement learning. arXiv preprint arXiv: 1805. 2018a.
Tianbing Xu, Qiang Liu, Liang Zhao, and Jian Peng. Learning to explore with meta-policy gradient. arXiv preprint / 1803.8800044 2018b.
Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gradient reinforeement learning. In
Advances in Neural Information Processing Systems, pp. 2396-24000 2018c.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pp. 4644-4654, 2018
13

---

Under review as a conference paper at ICLR 2020
A DERIVATION OF INTRINSIC REWARD UPDATE
Following the conventional notation in RL, we define (Tel7) fo) as the state-value function that estimates the expected future lifetime return given the lifetime history Tr, the task T, initial policy parameters Og and the intrinsic reward parameters Specially, by (Tol 0o) denotes the expected lifetime return at the starting state, i.e.
"T n. 0o) (T-p)(r166) [Chie]]
where Glife denotes the lifetime return in task T. We also define the action-value function qT (Tt, arl7, fo) accordingly as the expected future lifetime return given the lifetime history т and an action dz.
The objective function of the optimal reward problem defined as:
J(n) = EB T- [E>rp,((18o)) [G
(5)
O.0x7) [eT (Toll) 0o)],
(6)
where O and p(7) are an initial policy distribution and a task distribution respectively.
Assuming the task T and the initial policy parameters Og are given, we omit T and dg for the rest of equations for simplicity. Let TT, (- = Te, (-1se) be the probability distribution over actions at time t given the history Tr, where 02 fy (Te, 0p) the policy parameters time é in the lifetime. We can derive the meta-gradient with respect to by the following:
((n) V (Tool)
V,
Σ To (aol (aglTa))(TT)
V To, (ao (aoll))((T))).))m anln) To, aoln)]
V To (oo + noo (aol Vn Σ p(r. roltoro + + u(T) e(T1l))
dg
tu.ro
Σ 0a
V (a0)) anln) + no (colro)
Σ p(Tilm ao) Vne(Tin))))
E,,
v (e
E., [V, log u
17. (1) [nl)
Er, [G& log 7 m
EE, [G. a, log To. (a, V 0cl.
k F& is the lifetime return given the history and assume the discount factor 7 = 1 for brevity. Thus, the derivative of the overall objective is:
where G, = T-1
(n) =.T-0, [Erpp((rim..0] [G, Vg, log тØ, (a, st)) v,
(7)
14

---

Under review as a conference paper at ICLR 2020
B EXPERIMENTAL DETAILS
B.1 IMPLEMENTATION DETAILS
We used mini-batch update to reduce the variance of meta-gradient estimation. Specifically, ran 64 lifetimes in parallel, each with a randomly sample task and randomly initialised policy param- eters. We took the average of the meta- graddinrsdnttss from each lifetime to compute the update to the intrinsic reward parameters(n). We ran 2 10 updates to at training time. We used arctan activa- tion the output of the intrinsic reward. The hyperparameters used for each domain are described in T
Table I: Hyperparameters.
Hyperparameters
Empty Rooms
Random ABC
Key-Box
Non-stationary ABC
Time limit per episode
100
10
100
10
Number episodes per lifetime
200
50
4 5000
1000
Inner unroll length
8
16
Entropy regularisation
0.01
0.01
0.01
0.05
Policy architecture
Conv(16)-FC(64)
Policy optimiser
SGD
SGD
Adam
SGD
Policy learning rate
0.1
0.1
0.001
0.1
Reward architecture
Con((16)FCC(44)LSTMMG
Reward optimiser
Adam
Reward learning rate
0.001
Outer unroll length
5
Inner discount factor
0.9
Outer discounter factor
0.99
B.2 DOMAINS
We will consider five task distributions, instantiated within one of the three main gridworld domains shown in Figure 2 In all cases the agent has four actions available, corresponding to moving up, down, left and right. However the topology of the gridworld and the reward structure may vary.
B.2.1 EMPTY ROOMS
Figure 2a shows the layout of the Empty Rooms domain. There four rooms in this domain. The agent always starts at the centre of the top-left room. One and only one cell is rewarding, which is called the goal. The goal is invisible. The goal location sampled uniformly from all cells the beginning of each lifetime. An episode terminates when the agent reaches the goal location time limit of 100 steps is reached. Each lifetime consists of 200 episodes. The agent needs explore all rooms to find the goal and then goes to the goal afterwards.
B.2.2 ABC WORLD
Figure 2b shows the layout of the ABC World domain. There is single 5 by 5 room, with three objects (denoted by A, B. C). All object provides reward upon reaching them. An episode terminates when the agent reaches an object or a time limit of 10 steps is reached. We consider three different versions of this environment: Fixed ABC, Random ABC and Non-stationary ABC. In the Fixed ABC environment, each lifetime has 200 episodes. The reward associated with each object is fixed across lifetimes. Specifically, the rewards for objects A, B, and C are 1, -0.5, and 0.5 respectively. The optimal policy is to always collect A. In the Random ABC environment, each lifetime has 50 episodes. The reward associated with each object randomly sampled for each lifetime and is held fixed within lifetime. Thus, the environment is stationary from agent's perspective but non-stationary from the reward function's perspective. Specifically, the rewards for A, B, and C uniformly sampled from [-1, 1], [-0.5, 0), and [0.0.5] respectively. The optimal behaviour is to explore A and C at the beginning of a lifetime to assess which is the better, and then commits to the better one for all subsequent episode. In the non-stationary ABC environment, each lifetime has
15

---

Under review as a conference paper at ICLR 2020
Fixed ABC
1.2
1.0
Learnid (ours)
Episode Return
Extrinsic Count based
0.8
0.6
0.2
-0.2
100
150
200
Evaluation of different rewards in the Fixed ABC domain. The x-axis shows the number of episodes within single lifetime; the y-axis measures the episode return
Figure 10:
1000 episodes. The rewards for A, B, and are 1, -0.5, and -1 respectively. The rewards for A and C swap every 250 episodes.
B.2.3 KEY Box WORLD
Figure 2c shows the Key Box World domain. In this domain, there is key and three boxes, A, B, and C. In order to open any box, the agent must pick up the key first. The rewards for A, B, and are uniformly sampled from 1-1, 1], 0.5, O], and [0, 0.5] respectively for each lifetime. An episode terminates when the agent opens box or a time limit of 100 steps is reached. Each lifetime consists of 5000 episodes.
B.3 HAND-DESIGNED NEAR -PAR-OLTIMAL EXPLORATION STRATEGY FOR RANDOM ABC
We hand-designed a heuristic strategy for the Random ABC domain. We assume the agent has the prior knowledge that B is always bad and A and have uncertain rewards. Therefore, the heuristic to go to A in the first episode, go C in the second episode, and then go the better one in the remaining episodes in the lifetime. We view this heuristic as an upper-bound because it always finds the best object and arbitrarily control the agent's behaviour.
C ADDITIONAL EMPIRICAL RESULT
C.1 EXPLOITING OPTIMAL BEHAVIOUR ON A FIXED TASK
To investigate what the intrinsic reward learns in a fixed task, we designed the 'Fied ABC' en- vironment (see Figure 2b). The reward for each object (A, B, and C) fixed within and across lifetimes. When the agent collects an object, it receives the corresponding reward of 1, -0.5, or 0.5 for object A, B, or C respectively, and the episode terminates. Each lifetime contains 200 episodes. The optimal policy is to always collect A. The optimal reward should capture the regularity of the environment that object A has the highest reward and drive the agent towards object A.
Figure shows that agents trained with the learned intrinsic reward learn optimal policies within few episodes. This indicates that the intrinsic reward memorises the fixed optimal behaviour during training and assigns rewards accordingly to aid learning during evaluation. The result Fixed ABC not particularly surprising. In fixed task a stationary environment, an optimal reward function does not need encourage exploration, and helps the agent to directly learn the optimal behaviour as quickly as possible, similarly to reward shaping.
16


Reviews:
[{"review_id": "HkeUwwNKYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary \nThe paper evaluates the intrinsic reward as a way of storing information about episodes. It adopts the optimal intrinsic reward setting (Singh'09), and extends its recent policy gradient implementation, LIRPG, to lifetime settings. The task in the lifetime setting is to learn an intrinsic reward such that when trained with it, the agent maximizes its total return over its lifetime. A lifetime is defined as a sequence of episodes, where the agent does not have memory of previous episodes, however, the function computing the intrinsic reward does. In proof-of-concept experiments, the paper demonstrates that the learned intrinsic reward captures properties of several gridworld environments and induces meaningful behavior in the agent, successfully transferring information from previous episodes. Interestingly, a state-based reward function also generalizes to agents with perturbed action spaces, showing that this way of storing information is agnostic to the agent’s action space.\n\nDecision\nThe paper proposal is interesting and adequately evaluated, however, the impact of the paper might be limited by its limited technical novelty and lack of comparisons to strong baselines. I recommend marginal accept. \n\nPros\n- The paper is well-motivated.\n- The paper is well-written and the method is clearly explained. The literature review is thorough.\n- The experimental evaluation demonstrates several interesting and potentially promising phenomena.\n\nCons\n- The novelty of the paper is limited as it is a somewhat straightforward extension of prior work.\n- The impact of the paper is hard to judge as the experimental evaluation does not focus on potential usecases. \n\nQuestions. Here, I will focus on scientific questions, answering which would significantly improve the quality of the paper.\n- The biggest drawback of the paper is that the proposed method has an unfair advantage as it has a way of transmitting information across episodes, which the baselines do not (as stated on the bottom of page 5). While the findings of this paper are interesting, it is unclear how it compares to methods that have memory of previous episodes, such as agents with non-episodic recurrent policies, or meta-learning agents such as Duan’16, Finn’17. Is it possible that the proposed method e.g. scales better than recurrent policies due to compact representations or provides better generalization to things like action space changes?\n- How does the method compare to hand-designed intrinsic rewards on hard exploration games (such as montezuma’s revenge or pitfall Atari games)? Since it can only learn to explore on games that it previously successfully solved, it is possible that a hand-designed intrinsic reward such as RND (Burda’19) would perform better on these hard games. On the other hand, it is possible that the method will in fact perform better on these games due to more directed exploration.\n- How does the method compare to hand-designed intrinsic reward on out-of-distribution tasks? Intuitively, the method should perform the worse the further from the training distribution the task is, while the hand-designed rewards will always perform similarly. However, what is the extent to which the proposed method generalizes? It is possible that this method would be very useful in practice if it generalized well.\n\nOther potentially related work.\n- Xu’18, Learning to Explore with Meta-Policy Gradient, is a relevant work that proposes a meta-learning framework for training an exploration policy.\n- Metz’19, Meta-Learning Update Rules for Unsupervised Representation Learning, is a conceptually relevant work that proposes to meta-learn loss functions for unsupervised learning (and there is more recent related work on this topic too).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlH6qc0YS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "(Originally my score was a weak reject.)\n\nThis paper aims to study whether a learned reward function can serve as a locus of knowledge about the environment, that can be used to accelerate training of new agents. The authors create an algorithm that learns an intrinsic reward function, that when used to train a new agent over a “lifetime” (which consists of multiple episodes), leads to the best cumulative reward over the lifetime. As a result, the learned intrinsic reward is incentivized to quickly “teach” the agent when and where to explore to find out as-yet unknown information, and then exploit that information once there is no more to be had. Experiments on gridworlds demonstrate that these learned intrinsic rewards: 1. switch between early exploration and later exploitation, 2. explore only for information that is relevant for optimal behavior, 3. capture invariant causal relationships, and 4. can anticipate and adapt to changes in the extrinsic reward within a lifetime.\n\nI very much appreciated the design of the environments to test for specific properties within the learning algorithm: I think these experiments provide a very useful conceptual analysis of what learned intrinsic rewards can do.\n\nMy main qualm with the paper is with its significance -- the authors claim that the goal is to find out whether reward functions can be loci of knowledge, but we already know the answer is yes: the whole point of reward shaping is to improve training dynamics by building in knowledge into the reward function. It is not a surprise that learned reward functions can be loci of knowledge if our hand-designed reward functions already do so.\n\nTo me, the more interesting aspect of this paper is how much benefit we can get by learning intrinsic reward functions, relative to other ways of improving training dynamics. The authors do show that by allowing the intrinsic reward to be recurrent (and so dependent on past episodes), it is able to first incentivize exploration and later exploitation, which standard reward shaping cannot do (since usually reward shaping still maintains the assumption that the reward is a function of the state). However, given this motivation, it would be important to see comparisons between the proposed method of learning intrinsic rewards, and other methods for fast adaptation in the literature, such as MAML, which as I understand also has many of the properties highlighted in this paper.\n\nIdeally there would also be experiments on more complex environments: the environments in the paper have 104, 25, and 49 states. If we in the ABC environments if you count “whether or not reward(object) is known” as part of the state, that multiplies it by 2^3 = 8 giving 200 and 392 states, if you then further add the ordering of r(A), r(B), and r(C), that multiplies by a factor of 3! = 6 giving 1200 and 2352 states. These environments are excellent for demonstrating the properties of learned intrinsic rewards and I am glad the authors have done these experiments and analyzed the results. However, given that the paper aims to scale the optimal reward problem, it would have been useful to see examples where the state space cannot be fully enumerated to evaluate scalability.\n\nQuestions:\n\nIn Figure 5, in episode 1, why is the learned intrinsic reward heavily penalizing the path to C, but not penalizing the path to B? In the initial episode, the intrinsic reward should only know that B is to be avoided; it doesn’t yet know whether A or C is the better object.  I would expect the learned intrinsic reward to put similar positive rewards on the path to C and the path to A, and negative reward on the path to B. (It is slightly more likely that C is the best object. This probably changes things slightly, but not significantly.)\n\nAlso in Figure 5, by episode 3, shouldn’t the final states (A or C) have intrinsic rewards of larger magnitude? Otherwise the agent can go back and forth on the path to collect lots of intrinsic reward without terminating the episode, even though this wouldn’t get extrinsic reward.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxcM8b5FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper proposes a meta-learning approach to learn reward functions for reinforcement learning agents. It defines an algorithm to optimize an intrinsic reward function for a distribution of tasks in order to maximise the agent’s lifetime rewards. The properties of this reward function and meta-learning algorithm  are investigated through a number of proof-of-concept experiments. \n\nThe meta-learning algorithm and the corresponding empirical investigation are the main contributions of the paper. The algorithm seems to be similar to previous meta-learning approaches, but differs by introducing a lifetime value function. While I thought the paper raises some interesting possibilities, I am currently leaning towards rejection. The proposed algorithm does not seem like a major innovation over cited previous work. The empirical evaluation provides a number of proof-of-concept ideas, but no in depth investigation of the properties of the approach. The theoretical properties of the approach are barely discussed.\n\nDetailed remarks:\n\n* The main addition to the meta-learning algorithm is the lifetime value function. The authors mention multiple times that this is crucial to learning, but the properties of this value function are not really investigated or discussed in depth:\n\n- The authors mention that the value function must take into account changing future policies, but do not discuss this further. The value function update seems to be a standard on-policy TD update with the lifetime return and the complete history as input. The policy for this value function, however, is still a standard policy with only state as input (but it will be non-stationary over the agent lifetime). It would be good to discuss this learning problem in more detail.\n- The algorithm uses an n-step return. Is this important? What effect does n have on learning?\n\n* Another issue which I would have liked being discussed in more detail is the non-stationarity of the learning problem in general. Most of the approaches discussed in related work (e.g. shaping)  are aimed at learning/designing more informative reward functions. These reward functions still fit in the MDP framework, however, and map from states and actions to rewards. In the case of shaping approaches guarantees can be given that this does not alter the learning problem. The intrinsic reward functions used in this paper map the full life-time history of the agent to rewards. While this is a richer framework that can express more complicated tasks (like exploration over multiple episodes), it also invalidates many of the basic assumptions of reinforcement learning. The rewards are now no longer Markovian when only observing the current state. Moreover, the reward function will change over time. To what extent does this require non-stationary / history-based policy and value function learning to solve these issues? While some of these issues also apply to count based exploration strategies, (Strehl and Littman,2008 ) provided results that the  exploration bonuses result a Bellman Equation that accounts for uncertainties. No real guarantees seem to exist here.\n\n* The empirical contribution focuses on trying to answer a number of questions regarding the properties of the learnt intrinsic rewards. I found these questions to be very broad, while the answers are mostly anecdotal evidence through proof-of-concept examples.  These examples do show potential benefits of meta-learning intrinsic rewards, but I was somewhat disappointed that there was no more systematic investigation. For example, questions like ‘how does the distribution of tasks affect intrinsic rewards’ or ‘does intrinsic reward generalise’ are not really answered by providing metrics of performance or generalisation in controlled experiments, but by providing some example cases. Several of these questions (including optimising exploration and dealing with non-stationarity) also seem to have been investigated to some extent in the original Optimal reward papers (Singh, 2009/2010). It would be good to clearly indicate what we have learned beyond these previous results. \n\n* There seems to be a bit of a mismatch between the learning objective for intrinsic rewards in the optimal reward framework and the results shown in the experiments. The learning objective aims to optimise lifetime rewards for a distribution of tasks. Most of the experiments seem to analyse episodic reward performance and compare against single-task (or task agnostic) methods.\n\nMinor comments:\n\n- The architecture / parameterization of the lifetime value function does not seem to be defined anywhere. Given that it takes histories as input I assume this is another RNN?\n- There seems to be some small overloading in the notation with \\eta occasionally being used to denote the parameters of the reward function r_eta or the reward function itself.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a metalearning-based approach to learning intrinsic rewards that improve RL performance across distributions of problems.  This is essentially a more computationally efficient approach to approaches suggested by Singh (2009/10).  The reviewers agreed that the core idea was good, if a bit incremental, but were also concerned about the similarity to the Singh et al. work, the simplicity of the toy domains tested, and comparison to relevant methods.  The reviewers felt that the authors addressed their main concerns and significantly improved the paper; however the similarity to Singh et al. remains, and thus the concerns about incrementalism.   Thus, I recommend this paper for rejection at this time.
Decision: Reject