Intrinsically Guided Exploration in Meta Reinforcement Learning

1 INTRODUCTION . Human intelligence is able to transfer knowledge across tasks and acquire new skills within limited experiences . However , most reinforcement learning ( RL ) agents still require large amounts of data to achieve human-level performance ( Silver et al. , 2017 ; Hessel et al. , 2018 ; Vinyals et al. , 2019 ) . Meta reinforcement learning ( meta-RL ) makes a step toward such efficient learning by extracting prior knowledge from a set of tasks to quickly adapt to new tasks . However , efficient exploration of meta-RL needs to consider both training and adaptation phases simultaneously ( Ishii et al. , 2002 ) , which becomes a key challenge for meta-RL . One branch of meta-RL algorithms ( Finn et al. , 2017 ; Stadie et al. , 2018 ; Rothfuss et al. , 2019 ; Gurumurthy et al. , 2019 ) utilizes policies injected with time-irrelevant random noise for meta-training exploration , while another branch of methods ( Duan et al. , 2016 ; Mishra et al. , 2017 ; Gupta et al. , 2018 ; Zintgraf et al. , 2019 ; Rakelly et al. , 2019 ) introduces memories or latent variables that enable temporally-extended exploration behaviors . EPI ( Zhou et al. , 2018 ) introduces intrinsic rewards based on the improvement of dynamics prediction . However , these exploration mechanisms are still inefficient in either meta-training or adaptation and underperform in complex sparse-reward tasks . To address this challenge of meta-RL , we introduce information-theoretic intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation . Inspired by the common task-inference component in context-based meta-RL algorithms ( such as PEARL ( Rakelly et al. , 2019 ) and VariBAD ( Zintgraf et al. , 2019 ) ) , we leverage an insight that exploration behaviors should collect trajectories that contain rich information gain about the current task , and design an exploration objective to maximize the information gain for inferring taskss . Based this objective , we derive an intrinsic reward for learning an effective exploration policy . To reduce variance from estimating information-gain intrinsic rewards in complex domains , we simplify and derive an intrinsic reward based on prediction errors to achieve superior stability and scalability . We develop a novel off-policy meta-RL algorithm , Meta-RL with effiCient Uncertainty Reduction Exploration ( MetaCURE ) , that incorporates our intrinsic rewards and separates exploration and exploitation policies . MetaCURE learns to perform sequential exploration behaviors to reduce task uncertainty across adaptation episodes and maximizes the expected extrinsic return in the last episode of the adaptation phase . During meta-training , MetaCURE collects training data from both exploration and exploitation policies . As for adaptation , the exploration policy collects informative trajectories , and then the exploitation policy utilizes gained experiences to maximize final performance . We evaluate our algorithm on various sparse-reward MuJoCo locomotion tasks as well as sparse-reward Meta-World tasks . Empirical results show that it outperforms baseline algorithms by a large margin . We also visualize how our algorithm explores in novel tasks and discuss the pros and cons of the two proposed intrinsic rewards . 2 BACKGROUND . In meta-RL , we consider a distribution of tasks p ( T ) , with each task T modelled as a Markov Decision Process ( MDP ) , which consists of a state space , an action space , a transition function and a reward function . In common meta-RL settings ( Duan et al. , 2016 ; Finn et al. , 2017 ; Zintgraf et al. , 2019 ; Rakelly et al. , 2019 ) , tasks differ in the transition and/or reward function , and we can describe a task T with a tuple 〈pT0 ( s0 ) , pT ( s′|s , a ) , rT ( s , a ) 〉 , with pT0 ( s0 ) the initial state distribution , pT ( s′|s , a ) the transition probability and rT ( s , a ) the reward function . We denote context cTn = ( an , rn , sn+1 ) as the experience tuple collected at the n-th step of adaptation in task T , and we use cT−1 : T−1 = 〈cT−1 , cT0 , ... , cTT−1〉1 to denote all trajectories collected in the T timesteps A common objective for meta-RL is to optimize final performance after few-shot adaptation ( Finn et al. , 2017 ; Gupta et al. , 2018 ; Stadie et al. , 2018 ; Rothfuss et al. , 2019 ) . During adaptation , an agent first utilizes some exploration policy πe to explore for a few episodes , and then updates an exploitation policy π to maximize expected return . Such a meta-RL objective can be formulated as : max π , πe ET [ R ( T , π ( cTπe ) ) ] ( 1 ) where cTπe is a set of experiences collected by π e , and R is the last episode ’ s expected return with π . The exploitation policy π is adapted with cTπe for optimizing final performance . 3 METACURE . To support efficient exploration in both meta-training and adaptation , we propose MetaCURE , a novel off-policy meta-RL algorithm , that learns separate exploration and exploitation policies . The exploration policy aims to collect trajectories that maximize the agent ’ s information gain to reduce uncertainty of task inference . The exploitation policy maximizes the expected extrinsic return in the last episode of adaptation . In this section , we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation . 3.1 THE METACURE FRAMEWORK . As shown in Figure 1 , MetaCURE is composed of three main components : ( i ) a task encoder qφ ( z|c ) that extracts information from context c and estimates the posterior of the task belief z , ( ii ) an Explorer to learn an behavior or exploration policy , and ( iii ) an Exploiter to learn the target or exploitation policy . We utilize variational inference methods ( Kingma & Welling , 2013 ; Alemi et al. , 2016 ; Rakelly et al. , 2019 ) to train the task encoder qφ . In order to learn effective task encodings , its decoder is designed to recover the action value function of the exploiter , which captures rich and temporally-extended information about the current task . The algorithm utilizes two separate replay buffers , B and Benc . Buffer B is used to train both the Explorer and the Exploiter , while bufferBenc is used to train the task encoder . During meta-training , MetaCURE iteratively infers the posterior task belief with contexts and performs both exploration and exploitation policies to collect data . B stores experiences from both policies , while Benc only stores exploration trajectories . During the adaptation phase , only the explorer is used to collect informative trajectories for inferring the posterior task belief , and the exploiter utilizes this posterior belief to maximize final performance . 1for n = −1 , we define cT−1 = ( ~0 , ~0 , s0 ) . In following derivations , we may drop T for brevity . To learn efficient exploration in sparse-reward tasks , the reward signal for the explorer is defined as follows : rexplorer ( c−1 : i−1 , ai ) = γ nr ( si , ai ) + λr ′ intr ( c−1 : i−1 , ai ) ( 2 ) where r is the extrinsic reward , r′intr is an intrinsic reward , λ > 0 is a hyper-parameter and γ ∈ [ 0 , 1 ] is a discount factor , and n is the number of training iterations . The intrinsic reward is conditioned on past trajectories c , which direct the explorer to collect relevant information for task inference . Empirically , we utilize the task belief qφ ( z|c ) instead of context c as explorer ’ s input , because it extracts useful task-specific information . This modification reuses learned knowledge from raw experiences and accelerates learning . We will discuss in detail the design of intrinsic reward r′intr in the next section . As training proceeds , the extrinsic reward gradually diminishes , and the explorer converges to behaviors that collects useful experiences for adaptation . The exploiter π is expected to utilize collected experiences and optimize exploitation behavior . We design it to take state s and the latent code z as input , and maximize the expected extrinsic return . Finally , we utilize SAC ( Haarnoja et al. , 2018 ) , an off-policy RL algorithm to train the policies . Algorithm pseudo-codes are available in Appendix A.3 . 3.2 INTRINSIC REWARDS FOR METACURE . As shown in Figure 1 , to support efficient exploration in both meta-training and adaptation , we maximize agent ’ s information gain during exploration , and introduce an information-theoretic intrinsic reward in the following form : r′IG ( c−1 : i−1 , ai ) = E ( ri , si+1 ) | ( c−1 : i−1 , ai ) [ DKL [ p ( z|c−1 : i−1 ) ||p ( z|c−1 : i ) ] ] ( 3 ) The left-hand term p ( z|c−1 : i−1 ) in the KL distance is the agent ’ s task belief before observing ci , while the right-hand term is the posterior task belief . This term reflects how much the agent ’ s belief has changed after collecting an experience . We also observe similar intrinsic rewards proposed in VIME ( Houthooft et al. , 2016 ) , in which they explain the rewards as compression improvement . Further derivation ( see Appendix A.1 for details ) shows that this intrinsic reward can be estimated by the difference of two prediction errors : r′IG ( c−1 : i−1 , ai ) = E ( ri , si+1 ) | ( c−1 : i−1 , ai ) [ DKL [ p ( z|c−1 : i−1 ) ||p ( z|c−1 : i ) ] ] = E ( z , ri , si+1 ) | ( c−1 : i−1 , ai ) [ log p ( ri , si+1|c−1 : i−1 , ai ) − log p ( ri , si+1|c−1 : i−1 , ai , z ) ] ( 4 ) Although sharing some similarities with EPI ( Zhou et al. , 2018 ) , our intrinsic reward possesses several key differences : first , it maximizes both information gains in reward and dynamics prediction , while EPI ignores reward signals that are often critical for task inference in meta-RL ; secondly , our intrinsic reward supports end-to-end learning and collect online data for effective task inference , while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration . Empirically , we find that this intrinsic reward suffers form high variance , sometimes causing instability in training . Inspired by curiosity-driven exploration in traditional RL ( Still & Precup , 2012 ; Pathak et al. , 2017 ; Burda et al. , 2018 ) , we remove the first term in r′IG and use prediction errors as the intrinsic reward to reduce the estimation variance : r′PE ( c−1 : i−1 , ai ) = E ( z , ri , si+1 ) | ( c−1 : i−1 , ai ) [ − log p ( ri , si+1|c−1 : i−1 , ai , z ) ] ( 5 ) Unlike the prediction-based curiosity in traditional RL , this intrinsic reward is estimated on multiple tasks , and does not diminish during meta-training . 4 EXPERIMENTS . In this section , we aim at answering the following questions : 1 . Can MetaCURE achieve good final adaptation performance in sparse-reward tasks that require efficient exploration in both metatraining and adaptation ? 2 . Do the explorer and exploiter obtain desirable behaviors ? 3 . What ’ s the pros and cons of the two proposed intrinsic rewards ? 4 . Is MetaCURE ’ s components vital for efficient exploration ? Besides , we are also interested in its sample efficiency in meta-training and adaptation .

Summary: This paper proposes a novel meta-learning method, aimed at solving the following problem: at test time, the agent has N episodes to gather information [exploration phase], and we care about its return in the N+1th episode [exploitation phase]. To this end, the authors propose to learn a separate exploration and exploitation policy. The core of the algorithm is to use an exploration bonus for the exploration policy that rewards finding novel trajectories. This should help it to collect valuable information during meta training and meta testing. The exploration policy is only used in the N+1th episode, and  is conditioned on a context vector which is computed from the exploration trajectories.