What Can Learned Intrinsic Rewards Capture?

Reinforcement learning agents can store knowledge in their policies , value functions , state representations , and models of the environment dynamics . These components can be the loci of knowledge in the sense that they are structures in which knowledge , either learned from experience by the agent ’ s algorithm or given by the agent-designer , can be deposited and reused . The objective of the agent is defined by a reward function , and the goal is to learn to act so as to optimise cumulative rewards . In this paper we consider the proposition that the reward function itself is a good locus of knowledge . This is unusual in that most prior work treats the reward as given and immutable , at least as far as the learning algorithm is concerned . At the same time , especially in challenging reinforcement-learning problems , agent designers do find it convenient to modify the reward function given to the agent to facilitate learning . It is therefore useful to distinguish between two kinds of reward functions ( Singh et al. , 2010 ) : extrinsic rewards define the task and capture the designer ’ s preferences over agent behaviour , whereas intrinsic rewards serve as helpful signals to improve the learning dynamics of the agent . Intrinsic rewards are typically hand-designed and then often added to the immutable extrinsic rewards to form the reward optimised by the agent . Most existing work on intrinsic rewards falls into two broad categories : task-dependent and taskindependent . Both are typically designed by hand . Hand-designing task-dependent rewards can be fraught with difficulty as even minor misalignment between the actual reward and the intended bias can lead to unintended and sometimes catastrophic consequences ( Clark & Amodei , 2016 ) . Task-independent intrinsic rewards are also typically hand-designed , often based on an intuitive understanding of animal/human behaviour or on heuristics on desired exploratory behaviour . It can , however , be hard to match such task-independent intrinsic rewards to the specific learning dynamics induced by the interaction between agent and environment . The motivation for this paper is our interest in the comparatively under-explored possibility of learned ( not hand-designed ) taskdependent intrinsic rewards ( see Zheng et al. , 2018 , for previous work ) . We emphasise that it is not our objective to show that rewards are a better locus of learned knowledge than others ; the best locus likely depends on the kind of knowledge that is most useful in a given task . In particular , knowledge captured in rewards provides guidance on “ what ” the agent should strive to do while knowledge captured in policies provides guidance on “ how ” an agent should behave . Knowledge about “ what ” captured in rewards is indirect and thus slower to make an impact on behaviour because it takes effect through learning , while knowledge about “ how ” can directly have an immediate impact on behaviour . At the same time , because of its indirectness the former can generalise better to changes in dynamics and learning architectures . Therefore , instead of comparing different loci of knowledge , the purpose of this paper is to show that it is feasible to capture useful learned knowledge in rewards and to study the kinds of knowledge that can be captured . How should we measure the usefulness of a learned reward function ? Ideally , we would like to measure the effect the learned reward function has on the learning dynamics . Of course , learning happens over multiple episodes , indeed it happens over an entire lifetime . Therefore , we choose lifetime return , the cumulative extrinsic reward obtained by the agent over its entire lifetime , as the main objective . To this end , we adopt the multi-lifetime setting of the Optimal Rewards Framework ( Singh et al. , 2009 ) in which an agent is initialised randomly at the start of each lifetime and then faces a stationary or non-stationary task drawn from some distribution . In this setting , the only knowledge that is transferred across lifetimes is the reward instead of the policy . Specifically , the goal is to learn a single intrinsic reward function that , when used to adapt the agent ’ s policy using a standard episodic RL algorithm , ends up optimising the cumulative extrinsic reward over its lifetime . In previous work , good reward functions were found via exhaustive search , limiting the range of applicability of the framework . Here , we develop a more scalable gradient-based method ( Xu et al. , 2018c ) for learning the intrinsic rewards by exploiting the fact the interaction between the policy update and the reward function is differentiable ( Zheng et al. , 2018 ) . Since it is infeasible to backpropgate through the full computation graph that spans across the entire lifetime , we truncate the unrolled computation graph of learning updates up to some horizon . However , we handle the long-term credit assignment by using a lifetime value function that estimates the remaining lifetime return , which needs to take into account changing policies . Our main scientific contributions are a sequence of empirical studies on carefully designed environments that show how our learned intrinsic rewards can capture interesting regularities in the interaction between a learning agent and an environment sampled from a distribution , and how the learned intrinsic reward can generalise to changed dynamics and agent architectures . Collectively , our contributions present an effective approach to the discovery of intrinsic rewards that can help an agent optimise the extrinsic rewards collected in a lifetime . 1 RELATED WORK . Hand-designed Rewards There is a long history of work on designing rewards to accelerate learning in reinforcement learning ( RL ) . Reward shaping aims to design task-specific rewards towards known optimal behaviours , typically requiring domain knowledge . Both the benefits ( Randlov & Alstrm , 1998 ; Ng et al. , 1999 ; Harutyunyan et al. , 2015 ) and the difficulty ( Clark & Amodei , 2016 ) of task-specific reward shaping have been studied . On the other hand , many intrinsic rewards have been proposed to encourage exploration , inspired by animal behaviours . Examples include prediction error ( Schmidhuber , 1991b ; Gordon & Ahissar , 2011 ; Mirolli & Baldassarre , 2013 ; Pathak et al. , 2017 ; Schmidhuber , 1991a ) , surprise ( Itti & Baldi , 2006 ) , weight change ( Linke et al. , 2019 ) , and state-visitation counts ( Sutton , 1990 ; Poupart et al. , 2006 ; Strehl & Littman , 2008 ; Bellemare et al. , 2016 ; Ostrovski et al. , 2017 ) . Although these kinds of intrinsic rewards are not domain-specific , they are often not well-aligned with the task that the agent tries to solve , and ignores the effect on the agent ’ s learning dynamics . In contrast , our work aims to learn intrinsic rewards from data that take into account the agent ’ s learning dynamics without requiring prior knowledge from a human . Rewards Learned from Data There have been a few attempts to learn useful intrinsic rewards from data . The optimal reward framework ( Singh et al. , 2009 ) proposed to learn an optimal reward function that allows agents to solve a distribution of tasks quickly using random search . We revisit this problem in this paper and propose a more scalable gradient-based approach . Although there have been follow-up works ( Sorg et al. , 2010 ; Guo et al. , 2016 ) that uses a gradient-based method , they consider a non-parameteric policy using Monte-Carlo Tree Search ( MCTS ) . Our work is closely related to LIRPG ( Zheng et al. , 2018 ) which proposed a meta-gradient method to learn intrinsic rewards . However , LIRPG considers a single task in a single lifetime with a myopic episode return objective , which is limited in that it does not allow exploration across episodes or generalisation to different agents . Meta-learning for Exploration Meta-learning ( Schmidhuber et al. , 1996 ; Thrun & Pratt , 1998 ) has recently received considerable attention in RL . Recent advances include few-shot adaptation ( Finn et al. , 2017a ) , few-shot imitation ( Finn et al. , 2017b ; Duan et al. , 2017 ) , model adaptation ( Clavera et al. , 2018 ) , and inverse RL ( Xu et al. , 2018a ) . In particular , our work is closely related to the prior work on meta-learning good exploration strategies ( Wang et al. , 2016 ; Duan et al. , 2016 ; Stadie et al. , 2018 ; Xu et al. , 2018b ) in that both perform temporal credit assignment across episode boundaries by maximising rewards accumulated beyond an episode . Unlike the prior work that aims to learn an exploratory policy , our framework indirectly drives exploration via a reward function which can be reused by different learning agents as we show in this paper ( Section 5.1 ) . Meta-learning of Agent Update There have been a few studies that directly meta-learn how to update the agent ’ s parameters via meta-parameters including discount factor and returns ( Xu et al. , 2018c ) , auxiliary tasks ( Schlegel et al. , 2018 ; Veeriah et al. , 2019 ) , unsupervised learning rules ( Metz et al. , 2019 ) , and RL objectives ( Chebotar et al. , 2019 ) . Our work also belongs to this category in that our meta-parameters are the reward function used in the agent ’ s update . In particular , our multilifetime formulation is similar to ML3 ( Chebotar et al. , 2019 ) . However , we consider the long-term lifetime return as objective to perform cross-episode temporal credit assignment as opposed to the myopic episodic objective in ML3 . 2 THE OPTIMAL REWARD PROBLEM . We first introduce some terminology . • Agent : A learning system interacting with an environment . On each step t the agent selects an action at and receives from the environment an observation st+1 and an extrinsic reward rt+1 defined by a task T . The agent chooses actions based on a policy πθ ( at|st ) parameterised by θ . • Episode : A finite sequence of agent-environment interactions until the end of the episode defined by the task . An episode return is defined as : Gep = ∑Tep−1 t=0 γ trt+1 , where γ is a discount factor , and the random variable Tep gives the finite number of steps until the end of the episode . • Lifetime : A finite sequence of agent-environment interactions until the end of training defined by an agent-designer , which can include multiple episodes . The lifetime return is Glife =∑T−1 t=0 γ trt+1 , where γ is a discount factor , and T is the number of steps in the lifetime . • Intrinsic reward : A reward function rη ( τt+1 ) parameterised by η , where τt = ( s0 , a0 , r1 , d1 , s1 , . . . , rt , dt , st ) is a lifetime history with ( binary ) episode terminations di . The Optimal Reward Problem ( Singh et al. , 2010 ) , illustrated in Figure 1 , aims to learn the parameters of the intrinsic reward such that the resulting rewards achieve a learning dynamic for an RL agent that maximises the lifetime ( extrinsic ) return on tasks drawn from some distribution . Formally , the optimal reward function is defined as : η∗ = arg max η J ( η ) = arg max η Eθ0∼Θ , T ∼p ( T ) [ Eτ∼pη ( τ |θ0 ) [ Glife ] ] , ( 1 ) where Θ and p ( T ) are an initial policy distribution and a distribution over possibly non-stationary tasks respectively , andGlife = ∑T−1 t=0 γ trt+1 is a lifetime return . The likelihood of a lifetime history Algorithm 1 Learning intrinsic rewards across multiple lifetimes via meta-gradient Input : p ( T ) : Task distribution , Θ : Randomly-initialised policy distribution Initialise intrinsic reward function η and lifetime value function φ repeat Initialise task T ∼ p ( T ) and policy θ ∼ Θ while lifetime not ended do θ0 ← θ for k = 1 , 2 , . . . , N do Generate a trajectory using πθk−1 Update policy θk ← θk−1 + α∇θk−1Jη ( θk−1 ) using intrinsic rewards rη ( Eq . 2 ) end for Update intrinsic reward function η using Eq . 3 Update lifetime value function φ using Eq . 4 θ ← θN end while until η converges τ is pη ( τ |θ0 ) = p ( s0 ) ∏T−1 t=0 πθt ( at|st ) p ( dt+1 , rt+1 , st+1|st , at ) , where θt = f ( θt−1 , η ) is a policy parameter as updated with update function f , which is policy gradient in this paper.1 Note that the optimisation of η spans multiple lifetimes , each of which can span multiple episodes . Using the lifetime return Glife as the objective instead of the conventional episodic returnGep allows exploration across multiple episodes as long as the lifetime return is maximised in the long run . In particular , when the lifetime is defined as a fixed number of episodes , we find that the lifetime return objective is sometimes more beneficial than the episodic return objective even in terms of the episodic return performance measure . However , different objectives ( e.g. , final episode return ) can be considered depending on the definition of what a good reward function is .

Summary: The paper evaluates the intrinsic reward as a way of storing information about episodes. It adopts the optimal intrinsic reward setting (Singh'09), and extends its recent policy gradient implementation, LIRPG, to lifetime settings. The task in the lifetime setting is to learn an intrinsic reward such that when trained with it, the agent maximizes its total return over its lifetime. A lifetime is defined as a sequence of episodes, where the agent does not have memory of previous episodes, however, the function computing the intrinsic reward does. In proof-of-concept experiments, the paper demonstrates that the learned intrinsic reward captures properties of several gridworld environments and induces meaningful behavior in the agent, successfully transferring information from previous episodes. Interestingly, a state-based reward function also generalizes to agents with perturbed action spaces, showing that this way of storing information is agnostic to the agent’s action space.