Intrinsically Guided Exploration in Meta Reinforcement Learning
Authors: Jin Zhang
Jianhao Wang
Hao Hu
Tong Chen
Yingfeng Chen
Changjie Fan
Chongjie Zhang
Venue:  (2021)
Keywords: Meta reinforcement learning
Exploration
Information gain

Abstract: Deep reinforcement learning algorithms generally require large amounts of data to solve a single task. Meta reinforcement learning (meta-RL) agents learn to adapt to novel unseen tasks with high sample efficiency by extracting useful prior knowledge from previous tasks. Despite recent progress, efficient exploration in meta-training and adaptation remains a key challenge in sparse-reward meta-RL tasks. We propose a novel off-policy meta-RL algorithm to address this problem, which disentangles exploration and exploitation policies and learns intrinsically motivated exploration behaviors. We design novel intrinsic rewards derived from information gain to reduce task uncertainty and encourage the explorer to collect informative trajectories about the current task. Experimental evaluation shows that our algorithm achieves state-of-the-art performance on various sparse-reward MuJoCo locomotion tasks and more complex Meta-World tasks.

Under review as a conference paper at ICLR 2021
INTRINSICALLY GUIDED EXPLORATION IN META RE- IEFORTEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep reinforcement learning algorithms generally require large amounts of data to solve single task. Meta reinforcement learning (meta-RL) agents learn to adapt to novel unseen tasks with high sample efficiency by extracting useful prior knowledge from previous tasks. Despite recent progress, efficient exploration in meta-training and adaptation remains key challenge in sparse-rewar reward meta-RL tasks. We propose a novel off- offlicl meta-RL algorithm to address this problem, which disentangles exploration and exploitation policies and learns intrinsically motivated exploration behaviors. We design novel intrinsic rewards derived from information gain to reduce task uncertainty and encourage the explorer to collect informative trajectories about the current task. Experimental evaluation shows that our algorithm achieves state-of sitee--ffthearrt the-art performance on various sparse-reward MuJoCo locomotion tasks and more complex Meta-World tasks.
INTRODUCTION
Human intelligence able to transfer knowledge across tasks and acquire new skills within limited experiences. However, most reinforcement learning (RL) agents still require large amounts of data to achieve human-level performance (Silver et al. 2017 Hessel et al. [2018] Vinyals et al. 2019). Meta reinforcement learning (meta-RL) makes step toward such efficient learning by extracting prior knowledge from a set of tasks to quickly adapt to new tasks. However, efficient exploration of
meta-RL needs consider both training and adaptation phases simultaneously (Ishii et al.. 2002). which becomes a key challenge for Meta-RI.
One branch of meta-RI. algorithms (Finn al., 2017; Stadie et al. 2018 Rothfuss al. 2019 Gu-
rumurthy et al. 2019) utilizes policies injected with time-irrelevant random noise for meta-training exploration, while another branch methods (Duan et al. 2016 Mishra et al. 2017 Gupta et al. 2018 Zintgraf et al. 2019 Rakelly et al. 2019) introduces memories or latent variables that enable exploration behaviors. EPI (Zhou et al. 2018) introduces intrinsic rewards based on the improvement of dynamics prediction. However, these exploration mechanisms still inefficient in either meta-training or adaptation and underperform in complex sparse-reward tasks.
temporally-extente
To address this challenge of meta-RL., we introduce information-the -tteorettc intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation. Inspired by the common task-inference component in context-based meta-RL algorithms (such as PEARL (Rakelly et al. (2019) and VariBAD (Zintgraf et al. 2019)), we leverage an insight that exploration behaviors should collect trajectories that contain rich information gain about the current task, and design an exploration objective to maximize the information gain for inferring taskss. Based this objective, we derive an intrinsic reward for learning an effective exploration policy. To reduce variance from estimating information intrinsic rewards in complex domains, we simplify and derive an intrinsic reward based on prediction errors achieve superior stability and scalability.
We develop a novel off-policy meta-RL algorithm, Meta-RL with effiCient Uncertainty Reduction Exploration (MetaCURE), that incorporates our intrinsic rewards and separates exploration and ex- ploitation policies. MetaCURE learns to perform sequential exploration behaviors to reduce task un- certainty across adaptation episodes and maximizes the expected extrinsic return in the last episode of the adaptation phase. During meta-training, MetaCURE collects training data from both ex- ploration and exploitation policies. As for adaptation, the exploration policy collects informative

---

Under review as a conference paper at ICLR 2021
trajectories, and then the exploitation policy utilizes gained experiences to maximize final perfor- mance. We evaluate our algorithm on various sparse-reward MuJoCo locomotion tasks as well as sparse-reward Meta-Worke tasks. Empirical results show that it outperforms baseline algorithms by a large margin. We also visualize how our algorithm explores in novel tasks and discuss the pros and cons of the two proposed intrinsic rewards.
2 BACKGROUND
In meta-RL, we consider distribution of tasks p(T), with each task T modelled as a Markov De- cision Process (MDP), which consists of a state space, an action space, a transition function and reward function. In common meta-RL settings (Duan al. 2016; Finn et al. 20177] Zintgraf ctt al., 2019; Rakelly et al. 2019). tasks differ in the transition and/or reward function, and we can (s).))) with pa (so) the initial state distribu- tion, pT (s'|s,a) a) the transition probability and rT (s,a) a) the reward function.
describe task T with tuple (pb (so). pp' (s'|ss (ssls.))) r 7
We denote context of (an Iwy Sn+1) 1) as the experience tuple collected at the n-th step of adapta- tion task T, and we use cT c-1-T-1 = co ..... - to denote all trajectories collected in the T timesteps
common objective for meta-RL. is to optimize final performance after few-shot adaptation (Finn al. 2017, Gupta al.) 2018 Stadie et al. 2018: Roothuus al.] (019).. During adaptation, agent first utilizes some exploration policy л" to explore for few episodes, and then updates an
exploitation policy maximize expected return. Such meta-RL objective can be formulated as:
max Ey [R(T, n (cf-))
(1)
where c²= is a set of experiences collected by π², and R is the last episode's expected return with m. The exploitation policy π is adapted with o- for optimizing final performance.
METACURE
To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference. The exploitation policy maximizes the expected extrinsic return in the last episode of adaptation. In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation.
3.1 THE METACURE FRAMEWORK
As shown in Figure I MetaCURE is composed of three main components: (i) task encoder (2(e) that extracts information from context e and estimates the posterior the task belief 2, (ii) an Explorer learn an behavior exploration policy, and (iii) Exploiter to learn the target or exploitation policy. We utilize variational inference methods (Kingma & Welling 2013 20133 Alemi et al.) 2016 Rakelly et al., (219)) train the task encoder gz- In order to learn effective task encodings, its decoder is designed to recover the action value function of the exploiter, which captures rich and temporally-extendee information about the current task.
The algorithm utilizes two separate replay buffers, B and Bene Buffer B is used to train both the Explorer and the Exploiter, while buffer Bane is used to train the task encoder. During meta-training. MetaCURE iteratively infers the posterior task belief with contexts and performs both exploration and exploitation policies collect data. B stores experiences from both policies, while B only stores exploration trajectories. During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.
for -1, define cT, (0, 0, so). In following derivations, may drop T for brevity.
2

---

Under review as a conference paper at ICLR 2021
To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows:
Texploer(C-1si111: = r ai) + X (c-1s-1::;)
(2)
where r the extrinsic reward, yar an intrinsic reward, A>0 hyper-parameter and y E [0, is discount factor, and is the number of training iterations. The intrinsic reward is conditioned on past trajectories c, which direct the explorer to collect relevant information for task inference. Empirically, utilize the task belief 4s (z|c) instead of context extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning. We will discuss in detail the design of intrinsic reward r in the next section. As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges behaviors that collects useful experiences for adaptation.
explorer's input, because it
The exploiter is expected to utilize collected experiences and optimize exploitation behavior. We design take state s and the latent code z as input, and maximize the expected extrinsic return. Finally, we utilize SAC (Haarnoja et al., 2018). an off-policy RL algorithm to train the policies. Algorithm pseudo-codes are available in Appendix A.3
Buffers
Explorer
B
Sample 9p a,
S
Meta- Predictor
prred
Tu 5te1
Benc
St
Lyd'
rr, + Ar mnr
dt
Sample context
Context encader
Exploiter
St
L
(x(c)))~
Posterior Belief
Replay Buffers
DEL
St
Q
LQu
de
Networks
Forward Pass
Loss Functions
O
Backward Pass
Figure 1: MetaCURE's meta- meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency both policies be off-policy trained with history experiences. Batches from B and Bene are sampled from the same task. x Q function is omitted for brevity.
3.2 INTRINSIC REWARDS FOR METACURE
As shown in Figure support efficient exploration in both meta-training and adaptation, we max- imize agent's 's information gain during exploration, and introduce an information-theoreti intrinsic reward in the following form:
a) = bet
The left-hand term p(z(c_1;1-1 the KL distance the agent's task belief before observing c₁ while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience. We also observe similar intrinsic rewards proposed in VIME (Houthooft et al. 2016)) in which they explain the rewards as compression improvement.
Further derivation (see Appendix A.I for details) shows that this intrinsic reward can be estimated by the difference of two prediction emors:
r(o(c_1--194))
(4)
Caarmauulle-----e log 8:41 10-111 814 11---

---

Under review as a conference paper at ICLR 2021
Although sharing some similarities with EPI (Zhou et al. 2018), our intrinsic reward possesses sev- eral key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.
Empirically, we find that this intrinsic reward suffers form high variance, sometimes causing insta- bility in training. Inspired by curiosity-driven exploration in traditional RL (Still & Precup 2012
Pathak et al., 2017: Burda et al. 2018). we remove the first term in TíG and use prediction errors as the intrinsic reward reduce the estimation variance:
r'PE(C-N-1,,)) a;) E (z.r.. myjee (ifl)) log p(r,, $141le-18-11 ag a4,
(5)
Unlike the prediction- curiosity in traditional RL., this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.
4 EXPERIMENTS
In this section, we aim at answering the following questions: 1. Can MetaCURE achieve good final adaptation performance in sparse-reward tasks that require efficient exploration in both meta- training and adaptation? 2. Do the explorer and exploiter obtain desirable behaviors? 3. What's s the pros and cons of the two proposed intrinsic rewards? 4. Is MctaCURE's components vital for efficient exploration? Besides, we are also interested in its sample efficiency in meta-training and adaptation.
4.1 ADAPTATION PERFORMANCE ON CONTINUOUS CONTROL
Environment Setup We evaluate our algorithm on various continuous control task sets with sparse rewards, which exploration is vital for good performance. These tasks vary either the reward function (goal location in Point-Robot-Spare Vel-Sparse and Walker- See--arr or the transition function (Walker-Paranl -arasss-Spars).. These tasks (except for Point-Robot-Spars)) are simulated via MuJoCo (Todorov et al. 2012) and are bench- marks commonly used by current meta-learning algorithms (Mishra et al.j 2017] et al., 2017] Rothfuss et al., 2019; Rakelly et al. 2019). Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration. Also, we do not provide dense rewards while meta-training, which is different from the setting of PEARL. Typically, each adaptation phase consists of 2- episodes, each episode 32~-64 steps, varying with the specific task set. Detailed parameters and reward function settings are available in Appendix
and Reacher-Gool--parse target velocity in Cheetah-
Algorithm setup We compare our algorithm against several representative meta-Ri algorithms, in- cluding RL (Duan et al.) 2016), MAML (Finn et al. 2017). E-RL (Stadie al. 2018), ProMP (Rothfuss et al., 2019). MAME (Gurumurthy et al. 2019). PEARL (Rakelly et al.] 2019) and VariBAD Zintgraf et al. 2019). We also compare with EPI (Zhou et al.) 2018).
modify 2 We use open-source codes provided by the original pa- averaged over three random seeds. We implement two variants of
to make it consider reward prediction
pers, and the performance MetaCURE with PyTorch (Paszke al., (019): MetaCURE-IG utilizes the information-gain intrin- sic in Equation 4 and MctaCURE-PE utilizes the prediction error intrinsic in Equation [S] For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.
Results and analyses As shown in Tigure[aand Appendix A.5 we plot the algorithms' meta-testing performance as a function of number of experiences collected during meta- meta-training. Performance is evaluated by the last adaptation episode's return averaged over all meta-testing tasks. MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better. PEARL fails effectively explore within
"EPT is not trained end-to-end,and plot final performance in dash line.
4

---

Under review as a conference paper at ICLR 2021
Walker-Vel-Sparse
Cheetah-Vel-Spane
SointRobbot-Sparce
Average Return
Average Return
Average Return
Million Environment Samples
Million Environment Samples
Million Environment Samples
Walker-Vel-Sparse
Cheetah Seattah-Vel-Sparss.
Point-Robot-
Average Return
Averoge Return
Iwereee Return
Million Environment Samples
Million Environment Samples
Million Environment Samples
MetaCURE-PE (Durs)
ProMP
MAML
Final performance
MelaCURE-IG (Ours)
RL
PEARL
VariBAD
E
Evaluation of MetaCURE and several meta-RL baselines various sparse-reward con- tinuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.
Figure 2:
short adaptation, as posterior sampling (Osband & Van Roy 2017) ) is only optimal asymptotic per- formance. RL2 achieves good performance in tasks with simple dynamics like Point-Robot-Sparss and Cheetah- Seretah- but fails in more complex tasks. MAML acquires good performance in Walker-Parrms-Sparee but fails to learn tasks with different reward functions. The rest of the base- lines also perform poorly with sparse rewards. As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off off-policy trained, needing far fewer samples to converge.
4.2 ADAPTATION PERFORMANCE ON META- META-WORLD
World (Yu et al. 2019) is recently proposed challenging evaluation benchmark for meta-RL, including a variety of robot arm control tasks. We test MetaCURE-PE as well as baselines on two Meta-Worke task sets: Push and Reach. Although these task sets are not very distinct, they still very difficult to learn due to their complexity in dynamics and long time-horizon. To address make the rewards sparse, which makes the tasks Following the setting in Yu et al. (2019). we evaluate algorithms by their final success rates, and results are shown in Table MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration.
the importance of exploration,
harder.
4.3 ADAPTATION VISUALIZATION
To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visual- ize MetaCURE-PE' adaptation phase in Point-Robot-Sparye and Walker- Vel-Sparse, shown in Figure 3 and Figure 4 We compare against PEARL, which explores via posterior sampling. While MetaCURE effectively explores before carrying good exploitation behaviors, PEARL keeps its
5

---

Under review as a conference paper at ICLR 2021
Table 1: Final success rates the hard Meta- World tasks.
ENVIRONMENTS
METACURE- PE
PEARL
MAML
PROMP
RL.
E-RL
META-WORLD
0.46+00110
0.28+0.11
0+:0
0+:0
0+:0
0+:0
PUSH
META-WORLD
0.25++0.07
0.074+0.04
0+:0
0+:0
0+:0
0+00
REACH
15
35
155
10
10
14
1.0
55-
02
05
00-
05
00
-0.
1.2
1.5
15-
1.0
02
05
08
010
(a) MetaCURE-PE
(h) PEARI.
Figure Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparee The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate re- warding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PP PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.
belief about current task throughout an entire episode, and explores less effectively. We additional visualize adaptation in Cheetah- Seetahh-ee--Spasee and results are available in Appendix A.6
4.4 GAFIR VERSUS PREDICTION ERROR
In Figure MetaCURE-PE. outperforms MetaCURF-IG, as intrinsic reward of lower variance. However, the prediction errors may lead to bad exploration behaviors under certain scenarios. In this subsection we discuss about irrelevant dynamics, a problem that MetaCURE-PE may fail to solve while MetaCURE-IG manages to deal with.
We discuss a 2D navigation task set called Satt--Rooo--SSSS-Noiiii The agent's state is a 3D vec- tor composed of its current position (x, y) and a random noise 2, which is sampled from uniform distribution [0. d). with the L2 distance between the agent's 's current position and the original point. Goals uniformly distributed
semicircle of radius and only sparse reward is provided. We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown F While MetaCURE-PE interested distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics. We also provide another example in Appendix A.7
4.5
ABLATION STUDY
In this section, discuss the necessity of MetaCURE's components for its efficient exploration.

---

Under review as a conference paper at ICLR 2021
Selkerr-el--ppasee MetaCURE-PE
Walker-Vel-Sparses PEARL
"piiode 1 steps
Episode steps
Epliode stops
Epliode 2 stops
Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker- Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get in- formative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCURE- PE first efficiently explores the goal velocity before performing good exploitation policies.
Point Robot Sparse-Noii Noise
Average Return
HuSiC RC
Millon Environment Samples
(a) Learning curve
(b) MetaCURE-IG with average
(c) MetaCURE-PE with average return 7.3
return 20.2
Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point Soint-Roott-Spare--Wee Noise. (b) Adaptation visualization of MetaCURE-IG, It is motivated to gain task information, and is not disturbed by noises in dynamics. Adaptation visualization of MetaCURE-PE. good exploration behaviors, distinct regions obtain nosier dynamics and larger prediction losses.
fails to obtain
Hyper-parameters We conduct ablation studies MetaCURE's hyper- saperr-arameters, and results shown in Appendix A.8 Results show that MetaCURE is generally robust to hyper-parameters and does not need careful fine-tuning.
Intrinsic rewards The intrinsic rewards are vital for exploration in both meta-training and adapta- tion. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.I We also show algorithms' performance on another task set, in which adaptation short and the agent cannot perform good adaptation exploration by maximizing average return during adaptation. As shown in Appendix A.9.2 while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.
Baselines with intrinsic rewards We additionally test baselines that simply add our intrinsic re- ward (curiosity intrinsic) to PEARL and E-RL", respectively (for E-RL2, the intrinsic is provided to the exploration episodes only). On the Cheetah-Vel-Spaan a performance of 104.5 2.2, PEARL with intrinsic rewards only achieves 96.6 4.5, and E-RL2 with intrinsic rewards only achieves 92.3 + 9.3. These results show that trivially combining intrinsic rewards with meta-RL does not lead to good performance. MetaCURE outperforms these two baselines by performing task inference and separating exploration and exploitation.
task set, while MetaCURE achieves

---

Under review as a conference paper at ICLR 2021
5 RELATED WORKS
Exploration in meta-RL Compared meta supervised learning, in meta-RL the agent not given a task sask-specific dataset to adapt to, and it must explore the environment to collect useful information.
This exploration part is vital for both meta-training and adaptation (Finn & Levine 2019).
Adaptation exploration in gradient-based meta-RL. Raaa-RL: is mainly addressed by computing the gradient to the pre-update policy's state distribution (Stadie et al. 2018 Rothluss al. 2019). MAME (Guru- murthy et a 2019) learns separate explorer, and adapts via self-supervised losses. These methods utilize policy injected with random noise for meta-traine veeta-training exploration, which ineffective and empirically fail with sparse rewards. MAESN (Gupta et al., 2018) introduces temporally-extende exploration during adaptation with latent variables. Most context-based methods automatically bal- ances between exploration and exploitation during adaptation by maximizing average adaptation performance (Duan et al. 2016 Zintgraf et al. 2019). and E-RL (Stadie et al. 2018) directly op- timizes for final performance. T These methods also fail meta-train policies with sparse rewards. PEARL (Rakelly et al. 2019), a recently proposed approach, utilizes posterior sampling (Osband & Van Roy] 2017) to explore, and is less efficient with short adaptation, posterior sampling is only optimal is asymptotic performance. EPI (Zhou et al. 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement dynamics. However, it ignores useful reward signals, and lacks efficient exploration mechanisms to collect the dataset for intrinsic reward estimation, especially in sparse-rewes settings.
Exploration with cntrrmmtion-thhorree intrinsic rewards Measuring information gain is an im- portant way of designing intrinsic rewards for exploration. VIME (Houthooft al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim al. 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space. [Sun et al. (201T) discusses exploration with information gain, but is restricted to planning problems and requires oracle estimating the posterior.
Prediction loss as intrinsic reward Prediction loss can serve as a kind of curiosity- driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore. OOh et al. (2015) directly predicts the image observation, and [Stadie al. (2015) utilize prediction loss in the latent space, extracting useful features from observations. To avoid trivial solutions in learning the latent space, Pathak et al. (017)) introduces inverse model guide the learning latent space, only predicting things that the agent can control. BBrrda et al. (2018) utilizes random neural networks as projections onto the latent space and achieved good performance on Atari games.
CONCLUSION
In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adap- tation exploration, which becomes critical for sparse reward signals. By introducing information- theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-arr performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.
REFERENCES
Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv: 2016.
Mohammad Babaeizadeh, Chelsea Finn, Dumitru Erhan, Roy Campbell, and Sergey Levine. Stochastic variational video prediction. arXiv preprint arXiv: 1710. 1252, 2017.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. arXiv preprint arXiv: 1808 04355, 2018.
8

---

Under review as a conference paper at ICLR 2021
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances in neural information processing systems, pp. 2980-2988, 2015.
Yan Duan, John Schulman, Xi Chen, Peter Bartlett, llya Sutskever, and Pieter Abbeel. RI 2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:/611.0277 2016.
Chelsea Finn and Sergey Levine. Meta-learning from few-shot learning to rapid reinforcement learning.
e
cml 9meta Iettaaigarnn 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pp. 1126-1135. JML.R org, 2017.
Abhishek Gupta, Russell Mendonea, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pp. 5302-5311, 2018.
Swaminathan Gurumurthy, Sumit Kumar, and Katia Sycara. Mame: Model-agnostic meta- exploration. arXiv preprint arXiv:1911.04024 2019.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, pp. 1856-1865, 2018.
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, pp. 1109-1117, 2016.
Shin Ishii, Wako Yoshida, and Junichiro Yoshimoto. Control of parameter in reinforcement learning. Neural networks, 15(4-6):665-63) 687, 2002.
exploration meta-
Hyoungseok Kim, Jaekyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Ex- ploration with mutual information. In International Conference on Machine Learning, pp. 3360- 3369, 2019.
Diederik P Kingma and Max Welling. arXiv:1312.6114 2013.
Auto-encoding variational bayes
arXiv preprint
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. arXiv preprint arXivv 2017.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder Singh. Action-conditin video prediction using deep networks in atari games. In Advances neural information process- ing systems, pp. 2863-2871, 2015.
Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforce- ment learning? In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2701-2710. JMLR. org, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp- 8024-8035 2019.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 16-17, 2017.

---

Under review as a conference paper at ICLR 2021
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off off-co meta-reiinorremmnt learning via probabilistic context variables. In International Conference on Machine Learning, pp. 5331-5340, 2019.
Neale Ratzlaff, Qinxun Bai, Li Fuxin, and Wei Xu. Implicit generative modeling for efficient explo- 2019.
ration. arXiv preprint 1911.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, Pieter Abbeel, Dmitriy Shingarey, Lukas Kaul, Tamim Asfour, Dometios Athanasios, You Zhou, et al. Promp: Proximal meta- policy search. In International Conference on Learning Representations, volume 3, pp. 4007- 4014, 2019.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self- self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv: 2015.
Bradly C. Stadie, Ge Yang. Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforreement learning. ArXiv, abs/1803.01141 2018.
Susanne Still and Doina Precup. information theeoreii approach to curiosity-driven reinforce- ment learning. Theory in Biosciences, 131(3):139-144 2012.
Yi Sun, Faustino Gomez, and Jürgen Schmidhuber. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International Conference Artificial General Intelli- gence, pp. 41-51. Springer, 2011.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameekki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pp. 1-5, 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-tasi and meta reinforcement learning. arXiv preprint arXiv: 1910. 10897, 2019.
Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. In International Conference Learning Representations, 2018.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep via meta- learning. arXiv preprint 1910v0 2019.
A APPENDIX
A.1 PROOF OF EQUATION
4
We prove the following equation:
B(ress 1a- n.,,) (
pf - 0 (6)
10

---

Under review as a conference paper at ICLR 2021
Proof. We first state an important fact before providing proof. Knowing action without
knowing subsequent observations and rewards does not bring information about the task, or say:
p(=|c-1n-1) = p(2]c-1n-111.;
This fact also used in Sun et al.
2011 and Houthooft et al. 2016). Then
(1r,,44))lleee [DL pllco p=
-fʃ p(ele-
peele-ns--DD((lll--a- dr, dr;ds;41dz
log
plalc-uw)plelee-nn-ss
p stlc--n--)ppaalc-----11 dr,ds;+1]
Esjcmmil/ pttp:///tt..ee....-..-.-. log
(8)
Eeje-m- 01 log P(c](c11--1,4)) dr,ds++1]
ia-1 t TatpaaaDaaaai...aaaaaa..a....... p(c|c-1:-11,;;
One may concern that in Section the explorer takes qs (z]c) instead of as input, and may violate the fact proposed above, the action may obtain extra information about However, actually (=(=]c) is Gaussian distribution parameterized by μæ and o., both of which are deterministic functions of So knowing (=(2)c) does not introduce extra information about 2, and does not violate the proposed fact.
A.2 ESTIMATING INFORMATION- THERRRTTC INTRINSIC REWARDS
The intrinsic rewards proposed in Equation and Equation 5 requires estimating density functions log p(ri, 81+ 1 1: di, =) and logp(ri, 854 1:--:: az), which is difficult continuous state and action spaces. In implementation. we estimate them with prediction errors, which is commonly
used in density estimation (Chung al. 2015 Babaeizadeh al. 2017 Ratzlaff et al. 2019):
rho (c
~ Beenssppl s141) (rpeedl--is-1, 0s (mal spre(CC
(9)
-1(1,8141) - (rmrade- --a.--a--aaa...aa----------a)aaa
(pp(c-1s-1,0)
(10)
B( -n ma)alll 8141) "0 ais 2), Sprol(C
A.3 ALGORITHM PSEUDO-CODES
In this subsection we provide pscudo-codes for MctaCURE's meta trata-traning and adaptation, as shown in Algorithm and Algorithm 2
A.4 ENVIRONMENT AND PHYPER-PRRAMEER SETTINGS
In this subsection, provide detailed settings of reward functions and apper-parameters, which shown Table 2 and All tasks obtain sparse reward functions, providing zero rewards if the agent is outside the range of goals. The agent is also penalized with control costs.
As for the Meta- World tasks, adaptation phase consists of 4 episodes, with 150 steps in each episode. The agent gets non-zen non-zero rewards only if it "successes" in the task, which is given as binary signal by the environment.
A.5 ADAPTATION PERFORMANCE
In this subsection we plot learning curves on Reacher-Goal Spaal-Sprrse and Walker-Rand as shown in Figure
11

---

Under review as a conference paper at ICLR 2021
Algorithm MetaCURE: Meta-training Phase
Input: A set of meta-training tasks {T') "i=11........ drawn from p(7) Initialize replay buffers B¹, , .ne for each task T
while not done do for each task T do
Initialize experience context C; = 0 for cpisodes=I.....K do
for steps=I...,II do
Take action according to T² gg(2(c;))
Add collected experience (8, 8') to ca B and B
end for end for for episodes=1.....3 do
Sample (2(cc)
Collect a trajectory with (als, 2), add collected experience to B end for end for for steps in training steps do
for each T do
Sample encoder batch b'ne and policy training batch 6²
(=]bbne)) compute r with Equation 5 Sample (s
Train encoder yá. π and ne using SAC, b and r train the predictor to minimize prediction loss end for end for end while
Algorithm MetaCURE: Adaptation Phase
Input: Meta-test task T drawn from p(T) Initialize experience context c = { for episodes=1,....N do
for steps=1...,,I do
Take action according n (a)s; Add collected experience (s, a, s') to end for end for Sample 2 ( for steps=1,2..... do
Take action according to xaa(8, 2) end for
A.6 VISUALIZATIONS
We show additional visualization results Cheetah- Cheetah-Vel-Sparse, Vel-Sparse, shown in Figure 7
A.7 ANOTHER EXAMPLE OF IRRELEVANT DYNAMICS
In this subsection we discuss another task set that MetaCURE-PE fails on, called Pparse-Point Robot-Random. In this task set, there exists another kind of noise that is irrelevant to optimal policies: the agent get close to (0, 11), the agent will be transmitted to point randomly sampled on circle of radius 5. As these points task-specific knowledge, optimal policies will never get close to (0.11).
faraway from possible goals and does not provide any
Algorithms' performance is shown in Figure 8 information, MetaCURE-PP keeps going to the green circle to get transmitted, and fails to gain useful knowledge for adaptation.
As MetaCURE-IG insists probing task-specific
12

---

Under review as a conference paper at ICLR 2021
Table 2: Adaptation length and goal settings for environments used evaluate algorithm. Goals are uniformly distributed in Goal range and non-zero informative reward is provided only when the distance between the agent's position/speed and the goal is smaller than Goal radius.
ENVIRONMENT OF ADAP- TATION EPISODES
MAX STEPS PER EPISODE
GOAL TYPE
GOAL RANGE
GOAL
RA-
DIUS
CHEETAH-
2
64
VELOCITY
[0,3]
0.5
VEL-SPARSE
WALKER-
2
64
VELOCITY
[0,2]
0.5
VEL-SPARSE
REACHER-
2
64
POSITION
SEMICIRCLE
0.09
GOAL-
WITH DIUS 0.25
SPARSE
POINT-
4
32
POSITION
SEMICIRCLE
0.3
ROBOT-
WITH
RA-
SPARSE
WALKER-
4
64
VELOCITY
1.5
0.5
PARAMS-
SPARSE
Table 3: Hyperparameter settings for MetaCURE on these tasks. We use 3e-4 for all learning rates.
ENVIRONMENT
LATENT SIZE
B
A
CHEETAH- VEL-
2
0.1
0.1
SPARSE
WALKER- VEL-
2
0.2
SPARSE
REACHER-GOAL.
5
3
SPARSE
POINT-ROPE
SPARSE
WALKER-PARAMS:
2
3
SPARSE
META-WORLD WORLD
4
0.1
3
PUSH
META WORLD
0.1
REACH
A.8 HYPER- PARAMERRR ABLATION STUDIES
We test MetaCURE-PE with a wide range of hyper- hyper-paramese parameters on the Cheetah-Vel-Sppe task set, and results are shown in Table 4 is a hyper- parameter controlling the weight of KL divergence in the encoder's loss, similar to Rakelly et al. (2019)
A.9 ABLATION STUDY: INTRINSIC REWARDS
A.9.1 INTRINSIC REWARDS FOR META-TRAINIE EXPLORATION
and results shown in Fig- ure 9 Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.
We test MetaCURE without intrinsic rewards
13

---

Under review as a conference paper at ICLR 2021
Reacher-Goal-Sparse
Walker-Params-Sparee
Average Return
Average Return
1
100
0.2
0.2
6.75
1.22
1.50
175
Million Environment Samples
Million Environment Samples
Reacher-Goal-Sparse
Walker-Params-Sparss
Average Return
Average Return
>>
682
025
250
6is
Ldc
150
1755
Million Environment Samples
Million Environment Samples
MetaCURE-PE (Durs)
ProMP
E performance
MetaCURE-IG (Ours)
RL
PEARL
VariBAD
Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goud Goaal-Sparse and Walker-Rand-Parte
MetaCURE-PE
Cheetah-Ver Seeetah-Vel-Sparsee PEARL
steps
Episode steps
Episode stspps
Figure 7: Visualization of MetaCURE and PEARL on the Cheetah- Veletah-Vel-Sparse tasks.
14

---

Under review as a conference paper at ICLR 2021
Seaet-Rooot--pprrae Seboot-Sarre-Randdme
NesacuRt-Ca NeoOUUE--
Aver
Samples
Learning
(b) MetaCURE-IG with MetaCURE-PF with average return 3.2
age return 23.5
Figure (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Doint-Robot-Spant
Random. (b) Adaptation visualization of MetaCURE-IG, The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MctaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.
Table 4: MetaCURE's parameer ablation studies the Cheetah- Cheetah-Vel-Sparte Vel-Sparse task set.
LEARNING RATE
B
A
PERFORMANCE
3E-4
1E-1
1E-1
104.5+2.2
1E-3
1F-1
1E-1
106.8:16.5
3E-4
1E-2
1E-1
105.5433.7
3E-4
1E-1
SB-2
107.916.2
3E-4
1E-2
5B-2
105.8
A.9.2 INTRINSIC REWARDS FOR ADAPTATION EXPLORATION
To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Robtt-ooboe and has sub-optimal goal that keeps unchanged across all tasks, as shown Figure TO While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.
15

---

Under review as a conference paper at ICLR 2021
Point-Robot Spare-
Average Return
NasaCURE-R PE
NasaCURE rewurds
Million Environment Somples
Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta- veta-training and performs poorly.
Point-Robot-Sparre-SSu
15
10
Average Return
WP
005
-1.0
intrinsic reward
-1.
is
-0.5
Million Environment Samples
(a) Environment illustration
(b) Learning
15
10
10
35
05-
-3
-03
-05
-05
-10
-18
-15
-15
15
15
10
10
55
85-
30
08
-25
-05
LB
18
-15
(c) MetaCURE with intrinsic rewards achieves average return of 23.8
(d) MctaCURE without intrinsic rewards achieves average return of 19.2
Figure 10: (a) Point-Robot-Sparse with sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration offers lower returns. The cross indicates the agent's initial position. Blue circles indicate optimal goals.
(b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher fi- nal performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading inefficient exploration.
16


Reviews:
[{"review_id": "wDxSVmaHS_d", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*SUMMARY*\n\nThe paper presents a method for efficient task identification to improve adaptation in a meta RL setting. The approach is based on learning an exploration policy to quickly discriminate the task at hand, so that to leverage a task-specific policy for exploitation. To do so, it employs an intrinsic reward proportional to the information gain (or prediction error) over both the transition and reward models. Finally, the algorithm is evaluated over a set of continuous control domains with sparse rewards.\n\n*EVALUATION*\n\nThe idea of focusing the exploration on identifying the current task, instead of collecting good samples for policy improvements, is quite interesting, albeit not completely novel. Unfortunately, the lacking presentation does not help assessing the real quality of the contributions and the robustness of the experimental analysis, and it make me leaning towards a negative evaluation. However, I believe that a partial overhaul of the paper structure, along with a sharper experimental analysis, would make for a nice contribution to the meta RL community.\n\n*DETAILED COMMENTS*\n\nC1) As noted by the authors, the idea of maximizing information gain over the transition model in a single-environment setting has been presented in VIME (Houthooft et al. 2016), while Zhou et al. (2018) employs information gain to fast identification of the transition model in multi-environment setting. Thus, the main contribution of this work seems to account also for the reward model in the information gain computation. However, it is not clear to me how this can lead to a significant improvement, especially in sparse-reward settings, where rewards are zero almost everywhere.\n\nC2) An information theoretic approach to leverage the reward structure in a multi-goal scenario has been previously proposed in (Goyal et al. InfoBot: Transfer and exploration via the information bottleneck. ICLR 2019). Though the setting is not equivalent, I am  wondering how their concept of *decision state* relates to the idea of fast identification of the reward model.\n\nC3) The paper propose two alternatives reward function, one that essentially normalizes the prediction error with the predictability of the transition (IG), and the other that solely consider the prediction error (PE). While the experimental analysis shows a slight edge for the latter, I would say that this might be negligible with respect to the ability of the former to deal with noisy environments (as explained in Section 4.4).\n\n*QUESTIONS*\n\n- Could the authors address the comments above in their response?\n- The last line of Section 3 says that the proposed intrinsic reward does not vanish over time: Can the authors explain why?\n- The experimental setup describes a very short adaptation phase (hundreds of samples) but the subsequent plots are functions of millions of samples. Could the authors clarify the experimental setup, especially the number of samples employed in the meta-training and meta-testing phases?\n\n\n*ADDITIONAL FEEDBACK*\n\n- I would suggest to revise the structure of the paper by focusing on the core ideas first, and to the implementation details later.\n- I did not find the notation crystal clear, especially regarding the role of the context c, and the indiscriminate use of the words trajectory, episode, experience. To me it would be better to refer as sample every (s,a,r,s) tuple and trajectory every sequence of samples from the initial state to the end of an episode.\n- In my opinion the experimental section could be sharper. I would suggest to focus on a smaller set of domains (such as one with changing dynamics, one with changing rewards and one with both for the Mujoco set, in addition to the two Meta-World) and a smaller set of comparisons, and to seek for a clearer illustrations of the benefits of MetaCURE with respect to other methods.\n- I believe the ablation study to be central in the evaluation of MetaCURE. Especially, I would have preferred to see a convincing assessment of the benefit of computing the information gain over both the reward and transition models, with respect to accounting just for the former or the latter.\n- I would not claim the performance of Meta-World Push (Table 1) to be significantly higher than PEARL, since the confidence intervals are overlapping. Averaging over more trials might help.\n- I think the main text relies too much on the Appendix: It is fine to have proofs and derivations out of the paper, but I would not directly comment in the experimental section figures that only appear in the Appendix.\n\n#############\n\nAFTER RESPONSE\n\nI would like to thank the authors for the detailed response. I encourage them to keep working on MetaCURE: With some improvements I believe it will provide a valuable contribution to the meta-RL field.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hWv8sjaOCJx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper aims to facilitate meta-reinforcement learning by improving sample efficiency during meta-training as well as adaptation. Towards this goal, the paper proposes to use an intrinsic reward (two versions) and to learn a separate exploration policy to collect data for faster task inference. \n\nStrengths - \n\nApproaches to improve sample efficiency in meta-RL would be of interest to the community and are potentially useful to RL practitioners.\n\nThe proposed technique does not utilise dense rewards during meta-training. The use of intrinsic rewards in this approach possibly helps to overcome an important weakness of some previous methods (like PEARL or MAESN), which find it difficult to use sparse rewards during meta-training and assume access to dense reward in the meta-training phase.\n\nThe approach outperforms other meta-RL techniques within the given experimental setup. Additionally, the visualisations of adaptation behaviour (like in Figures 3 and 4) were quite helpful in understanding the method. However, I will discuss my concerns regarding the experiments in the subsequent sections. \n\nWeaknesses and concerns - \n\nThe writing and clarity of the paper can be improved. It could be useful to provide additional details to motivate the problem and choice of experimental setting. In the introduction, providing further comments (or citations) could justify this claim: “However, these exploration mechanisms are still insufficient in either meta-training or adaptation and underperform in complex sparse-reward tasks.” It would be helpful to provide more details about the proposed method beyond Figure 1. Is the encoder a recurrent network? And the meta-predictor is another feedforward neural network?\n\nRegarding the experiments, I would like to better understand the motivation behind the choice of restricting the adaptation phase to 2-4 episodes and then evaluating on the last episode. Is this choice realistic in applications? It could be worth considering that such a choice does not take into account potentially catastrophic behaviour by the exploratory policy and is closer to best arm identification (rather than regret minimization) in the multi-armed bandit setting. \n\nIt appears that a crucial reason for the better performance of the proposed approach is that it can explore within an episode. The off-policy competitor, PEARL, which uses posterior sampling, keeps its belief fixed for one episode and takes actions accordingly. While the experiments illustrate this weakness of the specifics of PEARL, in principle, there is no requirement in posterior sampling to update and sample from the posterior only once per episode. Since using an intrinsic reward introduces new hyperparameters, it could also be possible to introduce a hyperparameter for the number of steps (say T) between which PEARL updates and samples from the posterior. Tuning such a hyperparameter will also allow PEARL to explore within an episode and could make it much harder to beat. It could be informative to see such a comparison.\n\nAnother main concern is regarding the novelty of the proposed approach since the major off-policy meta-RL component appears similar to PEARL. And as the authors mention in the related work section, similar intrinsic rewards and maintaining a separate exploration policy (Gurumurthy et al., 2019) have also been studied earlier.\n\nRelated work -\n\nThe related work section is missing the original references. For example, the paper refers to recent work related to the use of prediction error as an intrinsic reward: Oh et al. (2015), Stadie et al. (2015) and Pathak et al. (2017). However, prediction error-based intrinsic rewards were introduced in 1990 [b]. Using information gain as a reward for RL agents to better explore the environment was introduced in 1995 [a]. Meta-RL was introduced in 1994 [c]. Intrinsically guided exploration in Meta-RL was introduced in 1997-2002 [d]. Please correct and describe what's really novel in your approach. \n\n[a] J. Storck et al. \"Reinforcement driven information acquisition in non-deterministic environments.\" Proceedings of ICANN'95, Paris. Vol. 2. 1995.\n\n[b] J. Schmidhuber. \"A possibility for implementing curiosity and boredom in model-building neural controllers.\" Proc. of SAB 1991. \n\n[c]          ---.          \"On learning how to learn learning strategies.\" TR FKI-198-94, TUM, 1994. http://people.idsia.ch/~juergen/FKI-198-94ocr.pdf \n\n[d]          ---.          \"What's interesting?\" TR IDSIA-35-97, IDSIA, July 1997. Also in: Exploring the Predictable. In Ghosh, S. Tsutsui, eds., Advances in Evolutionary Computing, p. 579-612, Springer, 2002. ftp://ftp.idsia.ch/pub/juergen/explorepredictable.pdf \n\n\nOther minor comments - \n\n- At first glance, it was slightly unclear what the curves in Figure 2 represent. I would suggest moving something like the statement on Page 4, “meta-testing performance as a function of number of experiences collected during meta-training” to the caption of Figure 2. \n\n\nSome typos - \n\n- Page 1, paragraph 3 in the Introduction:\n\ntaskss -> tasks\nBased this objective -> Based on\n\n- Page 2, section 3.1, 1st paragraph\n\nExplorer policy to learn an behavior -> a behavior\n\n- Page 3, last paragraph\n\nKL distance -> divergence\n\n- Page 7, Figure 5 caption\n\nnosier -> noisier\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LWOnlG9lDrp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel meta-learning method, aimed at solving the following problem: at test time, the agent has N episodes to gather information [exploration phase], and we care about its return in the N+1th episode [exploitation phase]. To this end, the authors propose to learn a separate exploration and exploitation policy. The core of the algorithm is to use an exploration bonus for the exploration policy that rewards finding novel trajectories. This should help it to collect valuable information during meta training and meta testing. The exploration policy is only used in the N+1th episode, and  is conditioned on a context vector which is computed from the exploration trajectories. \n\nPro: \n- Very relevant subject.\n- The authors bring up an interesting point, that metaRL methods should consider both exploration during meta training and meta testing.\n- I think it makes sense to separate exploration and exploitation policy explicitly.\n- The authors propose to sparsify common metaRL benchmarks. Evaluating your algorithm on environments that are proposed in the same paper can be tricky (and often means that the empirical results are evaluated more critically), but I think the suggestion of looking at sparse metaRL tasks is useful for the field and should be considered by other researchers as well.\n- The authors provide illustrations of the behaviours of the learned policy at meta-test time, which I think is really nice!\n\nConcerns:\n- I fee like there's a mismatch between the reward that the explorer gets, and the problem setting that the authors consider. The way I understand it, the two terms in the explorer's reward (Eq (2)) can be helpful during meta-training but I don't see how that translates to meta-test time. Let me try to explain:\n - [First term] The objective of the agent is to maximise the return in the *last* episode (the exploitation episode). Then why is the explorer not maximising the return of the exploitation phase, but instead gets a reward that corresponds to the exploration episodes? Doesn't that contradict the premise that we're not interested at all in the reward that is collected during exploration? Also, what is n (you write \"training iterations\" - is that just the number of environment steps per task? number of policy updates over the entirety of meta-training? Does this mean this is annealed to zero over training? Maybe then it shouldn't be called $\\gamma$ so that it's not confused with the per-episode discount usually used in RL.). \n - [Second term] The reward bonus can be seen as a type of novelty bonus for finding new trajectories. Can the authors clarify why this would help (a) during meta-training and (b) during meta-testing? I can see how it helps during training, but I don't understand why it also helps at test time. At the end of Sec 3.2, you write \"the intrinsic reward is estimated on multiple tasks and does not diminish during meta-training\". Why not? In Eq 5, does $c_{-1:i-1}$ come from multiple tasks or just one? Why can't this diminish over time, shouldn't our model $p(r,s|x,a,z)$ (dropping the subscripts for brevity) get better over time? Or do you just mean that the weight $\\lambda$ is not annealed? And doesn't it have all the information to, in principle, predict r and s exactly (except for aleatoric noise) so then this term reduces to the noise in the environment? Or should the terms in the subscript not match the ones within the expectation? It's also a bit unclear what the expectation is over, the subscript just shows some tuples. What's the probability distribution? Where do the terms in the subscript come from? Are $q$ and $p^\\mathcal{T}$ mixed in here? It would help my understanding if these things could be made more explicit. \n - [Missing term?] I understand the above two terms might help during training, but they are only auxiliary tasks in a way. And if the first term is annealed to zero (if that's indeed what's happening?), and the second term reduces to aleatoric noise (is that's indeed what's happening?), what is the exploration policy even doing at the end of training? Should there be a third term, which is the return in the *exploitation* phase, which is the only thing that the exploration policy should try to maximise at meta-test time?\n- I think the mathematical formulation has to be fleshed out. The paper is sometimes hard to follow because the notation is unclear, and not all terms are explained. I have a few open questions because of this: \n - What are the expectations in (3), (4), (5) over?\n - Not all terms in Figure 1 are defined or mentioned in the text. Are $L_\\pi$ and $L_{Q_\\pi}$ loss terms from SAC, so they correspond to equation (1)? And $L_{\\pi_e}$ corresponds to equation (2)? And what is $L_{pred}$? All these should be explicitly defined in the text with an equation. \n - Sec 2, directly after Eq (1): $c^\\mathcal{T}_{\\pi_e}$ is not defined.\n- In Figure 2, PEARL (Walker-Vel-Sparse) and VariBAD (Point-Robot-Sparse) seem to first learn and then diverge. Do the authors have an explanation as to why this is? It seems to me that maybe the hyperparameters just weren't set right, and makes me a bit sceptical of the results. When PEARL/VariBAD *do* learn, their (end-) performance is much closer to that of MetaCURE.\n- The authors propose two versions of MetaCURE (IG and PE). On the continuous control tasks PE is better, but then on the noisy point robot PE completely fails and IG succeeds. I think the authors are not doing themselves a favour by setting up the presentation of their methods like this. It would be better to propose *one* method, and then fully focus on this. The way it is now, there are two methods where neither of them is clearly better. I think that's confusing to the reader, and less convincing. Shouldn't there be a way to develop an exploration bonus that can do well across all environments? \n- The \"noisy\" environment is a POMDP (the agent gets noisy state observations). So doesn't that mean that there is simply a mismatch between the problem setting and the way the policy is set up? I'm not sure that this is a good experiment to add, because the solution (using the PE explorer) does not directly address the problem (partial observability). Maybe a noisy reward function would fit better here? \n\nOther questions:\n- Can you explain why the same exploration mechanism useful during meta-training and meta-testing? Aren't we mixing across-task and within-task information gain, and if so, why is that okay?\n- What is the difference between row 1 and row 2 in Figure 2? This is not explained in the text. Is that separation just because everything in one figure would've been too much? And how exactly were the baselines evaluated, always after the same number of episodes?\n- Why is the loss of the explorer not backpropagated through the context encoder? How does that fit together with \"our intrinsic reward supports end-to-end learning and collect online data for effective task inference\"?\n- Why does the exploiter get a sample $z$ and not the entire distribution $q$? Is that because we assume that all information is there and $q$ is concentrated anyway?\n- You mention that the intrinsic reward of PE is of lower variance. Could you explain why? Is that a hypothesis (if so, say so explicit) or do you have any empirical/theoretical evidence for this (again, if so, make it explicit).\n\nSuggestions:\n- You write that \"efficient exploration of meta-RL needs to consider both training and adaptation phases simultaneously\". This is a good point and I agree with it, and I think the paper can be improved by fleshing this argument out more. Help the reader understand why this is the case, and what the (different) challenges in exploration are during training and testing. One open question I still have for example (see above) is why the *same* exploration mechanism is useful during meta-training and meta-testing. Why not separate them? This should be explained in the paper. In the related work section, you write that prior methods \"utilise policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards\". This point can be made about almost all prior methods, and should come in the introduction. Then explain intuitively *why* this fails, maybe which an example. Another question I have is, how does the Ishii et al. 2002 paper back up this point that this distinction exists?\n- The experiments section relies too much on the appendix, and it's quite unnatural to read it like this. Either I have to jump back and forth between the appendix and the main text, or I have to take your word for what is shown in the results in the appendix. You introduce five continuous control tasks at the start, but then actually only show 3 and the other 2 are in the appendix. I think this section can be improved by focusing on the main important points in the main text, explain this well and mention that more results can be found in the appendix. \n- Mention in the main text that the sparse setting for Meta-World means you use the success signal only. This is a very difficult setting and is worth highlighting instead of burying it in the appendix.\n- I think a discussion about what an good/optimal exploration strategy would look like can help the reader understand the problem setting, and what you are trying to achieve. Right now, in Sec 4.3 you write for example that PEARL \"explored less effectively\" because it keeps the belief about the current task throughout an entire episode. That's not a fundamental problem with Thompson-sampling like methods though, but is quite problem-specific. If the number of adaptation episodes that are available corresponds to the number of possible tasks, then Thompson sampling always does optimal exploration. But, if the number of possible tasks exceeds the number of exploration episodes Thompson sampling fails. Adding discussions like these to understand what type exploration *is* necessary and why your method can achieve it, could strengthen the paper.\n\nTypos:\n- Intro, Paragraph 3: taskss -> tasks; Based -> Based on\n- Background, Paragraph 2: Full stop missing at end\n- 3.2, second paragraph + last paragraph: starts with free space\n- Figure 2, caption, writes \"five\" tasks but there are only three (other two in appendix)\n- Sec 4 (intro): 3. What's -> What are; 4. Is MetaCURE's -> Are MetaCURE's\n\n\n\n------------------------------------------------------------------------------------\nUPDATE\n\nThanks for your detailed response!\n\nA1: I see, that makes more sense to me now. I'm still not 100% convinced that it might not be better to let the policy entirely meta-learn what a good exploration policy is, and only use the inductive bias (terms 1 and 2) for how to do so for meta-training and anneal those terms over time. This would mean that at test time the only thing that the exploration policy should do is maximise the return in the *exploitation* phase - and basically figure out what the best way to do so is (so you'd need the \"missing term\" I describe above). But that's just a hunch, not sure if that would actually work better.\n\nA2-A8: Thanks a lot for clarifying. \n\nI'll keep my current score given all reviewers agree that the work is unpolished in its current form, and because the authors plan to propose a new version of MetaCURE soon. I think this is very interesting and promising work and look forward to reading an updated version in the future!", "extra_scores": {}, "extra_text": {}}, {"review_id": "4Lm7Gl7D2X0", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the use of intrinsically motivated exploration in meta-training and adaptation phase in meta Reinforcement Learning. The proposed framework consists of using task-inference, already used in other works like PEARL and VariBAD, and two deep neural networks, trained with an arbitrary off-policy algorithm, respectively for exploitation and exploration phases.\n\nI think the paper is well-written and discusses the related literature sufficiently well. The empirical results are significantly better than the considered baselines.\nDespite these good qualities, I think the major weakness of this paper is the lack of contribution. It looks to me that the major contribution discussed by the authors is the use of intrinsically motivated exploration in meta-training and adaptation, while the use of task-inference has been already proposed in other works and the use of different policy networks for exploitation and exploration is trivial. Considering the intrinsically motivated exploration, I miss the importance of the contribution, since it is already well-known that this technique can be beneficial, especially in sparse MDPs. The intrinsic reward makes sense, but it is not novel, as similar intrinsic rewards have been proposed in VIME and Pathak et al. (2017).\nExperimental results are good, but not well described. For example, I don't find a clear explanation of the tasks used in meta-training. The random seeds are only three, which makes the results less reliable.\nConsidering this, I don't consider this paper significant enough to be accepted.\n\nMinor issues:\n* Pag. 1, typo: \"tasks\" -> tasks\n* Pag. 1, \"Based this objective\" -> Based on this objective\n* the R function in (1) is not formally defined.\n\nPros\n------\n* Well-written;\n* Good discussion of related literature;\n* Good experimental results.\n\nCons\n-------\n* Lack of contribution;\n* Few, but important, missing details about the experimental setting, and too few seeds.\n\nPost-rebuttal feedback\n-------------------------------\nI thank the author for their reply and I encourage them to make the suggested improvements to the paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel off-policy meta-RL algorithm able to achieve efficient exploration in meta-training able to perform a fast task identification.
Although the reviewers agree that this paper has merits (relevant topic, interesting idea, nice experimental analysis), they have raised several concerns about the clarity, the novelty, and the soundness of the proposed approach, which make the paper not ready for publication. However, the authors are encouraged in improving their approach since the direction is considered promising by the reviewers.
Decision: Reject