First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs

Authors: ben  norman
jeff  clune
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 00:02:36

Abstract: Standard reinforcement learning (RL) agents never intelligently explore like a human (i.e. taking into account complex domain priors and adapting quickly based on previous exploration). Across episodes, RL agents struggle to perform even simple exploration strategies, for example systematic search that avoids exploring the same location multiple times. This poor exploration limits performance on challenging domains. Meta-RL is a potential solution, as unlike standard RL, meta-RL can *learn* to explore, and potentially learn highly complex strategies far beyond those of standard RL, strategies such as experimenting in early episodes to learn new skills, or conducting experiments to learn about the current environment.
Traditional meta-RL focuses on the problem of learning to optimally balance exploration and exploitation to maximize the *cumulative reward* of the episode sequence (e.g., aiming to maximize the total wins in a tournament -- while also improving as a player).
We identify a new challenge with state-of-the-art cumulative-reward meta-RL methods.
When optimal behavior requires exploration that sacrifices immediate reward to enable higher subsequent reward, existing state-of-the-art cumulative-reward meta-RL methods become stuck on the local optimum of failing to explore.
Our method, First-Explore, overcomes this limitation by learning two policies: one to solely explore, and one to solely exploit. When exploring requires forgoing early-episode reward, First-Explore significantly outperforms existing cumulative meta-RL methods. By identifying and solving the previously unrecognized problem of forgoing reward in early episodes, First-Explore represents a significant step towards developing meta-RL algorithms capable of human-like exploration on a broader range of domains.

Main paper: 
 introduction: 
 Reinforcement learning (RL) [1] can perform challenging tasks, such as plasma control [2] , molecule design [3] , game playing [4] , and robotic control [5] . However, RL is sample inefficient (taking thousands of episodes to learn tasks humans master in a few tries) [6] , limiting its application. Meta-RL [7] [8] [9] [10] [11] [12] circumvents this issue by enabling an agent to adapt to new environments based solely on prior experience (i.e., remembering what occurred in previous episodes and using that to inform subsequent behavior). By replacing slow weight-based RL updates with memory-based meta-RL adaption, human-like sample efficiency [12] can be achieved. This paper focuses on cumulative-reward meta-RL, which aims to optimize performance across a sequence of episodes, τ 1 , . . . , τ n (e.g., games in a tournament). The objective is to maximize the total reward accumulated over all episodes (e.g., the number of games won), expressed as n i=1 G(τ i ), where the episode return G(τ i ) is the total reward of episode τ i . To maximize this sum, the agent should optimally balance exploration and exploitation across the sequence, e.g., prioritizing exploration in early episodes so as to better exploit in later ones.Cumulative reward meta-RL has an unrecognized failure mode, where state-of-the-art (SOTA) methods achieve low cumulative-reward regardless of how long they are trained. This dynamic occurs in domains with the following properties: A. Maximizing the expected total reward requires exploratory actions that forgo immediate reward, and B. The benefit of these exploratory actions only occurs when they are reliably followed by good exploitation (i.e., if exploitation is too inconsistent, then exploration results in lower total reward).An example is a bandit domain where the first arm always provides a reward that is better than the average arm, but not the highest possible. To maximize cumulative reward over many pulls, the agent must explore the other arms, and then repeatedly exploit the best one. Property A holds, as this optimal policy forgoes immediate reward by not sampling the first arm and its above-average reward. Property B holds, as exploration (sampling arms other than the first) is only valuable when it is followed by sufficiently consistent exploitation (reliably re-sampling the best arm).The failure occurs as follows: 1. At the start of training, the agent, being randomly initialized, lacks the ability to reliably exploit learned information. 2. As a result, the domain properties A and B cause exploratory actions to lead to lower total reward than the other actions do. 3. This lower reward trains the agent to actively avoid exploration. 4. This avoidance then locks the agent into poor performance, as it cannot learn effective exploitation without exploration. This process occurs in the bandit example. Initially, the agent cannot exploit (e.g., when it finds the best arm it does not reliably resample it). The associated negative expectation of exploration then trains the agent to avoid exploration by only sampling the first arm, with its above-average, but sub-optimal, arm reward.Current SOTA meta-RL algorithms such as RL 2 [8, 9] , VariBAD [7] , and HyperX [10] attempt to train a single policy to maximize the cumulative reward of the whole episode sequence. This optimization causes step 3 of the above-described failure process, and thus these methods suffer from the issue of failing to properly learn (e.g., converging rapidly to a policy of not exploring), which we demonstrate on multiple domains (Section 6). The issue is especially insidious because distributions of simple environments (each trivially solved by standard-RL) can stymie these methods. Surprisingly, domains such as bandits can be too hard for SOTA meta-RL.We introduce a new approach, First-Explore (visualized in Figure 1 ), which overcomes this problem associated with directly optimizing for cumulative reward. Rather than training a single policy to maximize cumulative reward, First-Explore learns two policies: an exploit policy, and an explore policy. The exploit policy maximizes episode return, without attempting to explore. In contrast, the explore policy explores to best inform the exploit policy, without attempting to maximize its own episode return. Only after training are the two policies combined to achieve high cumulative reward.Because the explore policy is trained solely to inform the exploit policy, poor current exploitation no longer causes immediate rewards (property A) to actively discourage exploration. This change eliminates step 3 of the failure process, and enables First-Explore to perform well in domains where SOTA meta-RL methods fail. By identifying and solving this previously unrecognized issue, First-Explore represents a substantial contribution to meta-RL, paving the way for human-like exploration on a broader range of domains.2 Background RL Terminology: environments are formally defined as partially observable Markov decision processes (POMDPs, [1] ). Each POMDP E is specified by a tuple E = (S, A, p, p 0 , R, Ω, O, γ), where S is the state space, A the action space, p : S × A → S a probabilistic transition function mapping from the current state and action to the next state, p 0 a distribution over starting states, R : S × A → R a stochastic reward function, Ω the space of environment observations, O : S → Ω a stochastic function mapping from states to observations, and γ the discount factor. The environment starts (at t = 0) in a start state s 0 according to p 0 , s 0 ∼ p 0 . Each subsequent time-step, the agent receives the current state's observations o i = O(s i ), takes an action a i , and the transition function p updates the environment state s i+1 = p(s i , a i ). An episode τ of length h is then a sequence of time-steps starting from t = 0 to t = h. The sum of an episode's γ-discounted rewards is called its Figure 1 : First-Explore aims to maximize the cumulative reward of a sequence of n episodes on a target environment distribution. This optimization is achieved by first training two separate policies, and then combining them after training to maximize the total reward obtained. A. First, two separate policies are trained on the distribution of environments: one to explore (produce informative episodes), and one to exploit (maximize current episode return). During training, the explore policy π explore provides all the context c i = τ 1 , . . . , τ i for both policies. This flow of context is visualized by solid arrows →. The exploit policy π exploit takes a context of episodes, and produces a single episode of exploitation. The return of this exploit episode is then used to train both policies, with the feedback to the explore policy visualized by the dotted green arrows . B. After the two policies are trained, different combinations of them are evaluated to find the combination that maximizes total reward. Each combination involves first exploring for k episodes, and then repeatedly exploiting for the remaining n -k episodes. C. The best combination is then used at inference time: exploring for a fixed number of episodes on new environments, and then exploiting for the remaining episodes. return G(τ ) = h t=0 γ t R(s t , a t ). Standard RL generally aims to learn a stochastic policy π : Ω → A that maximizes the expected episode return E [G(τ )].Unlike standard RL, meta-RL trains an agent to: a) perform well in a distribution D of environments (e.g., a collection of mazes) and b) dynamically tailor itself to new environments (e.g. memorize a new maze over successive episodes). The agent completes meta-rollouts, each a sequence of episodes τ 1 , . . . , τ n on a new environment E ∼ D, with each episode within a meta-rollout beginning from a newly sampled start-state. The agent can remember information from earlier episodes in the same meta-rollout (e.g., using a sequence model such as a transformer [13] ), allowing it to adapt its behavior based on that information. By training (via weight updates) on large batches of meta-rollouts, the agent then learns to leverage its memory mechanism, enabling sample-efficient memory-based adaptation and high performance on new environments sampled from D.Meta-RL methods can be split into two approaches that each solve a different problem, with specific algorithms designed and used for each approach [14] 1 . Cumulative-reward meta-RL trains to maximize cumulative reward n i=1 G(τ i ). Examples include RL 2 [8, 9] , VariBAD [7] and Hy-perX [10] . Final-episode-reward meta-RL aims to instead optimize solely for final episode reward G(τ n ). Examples include DREAM [11] and MetaCURE [15] . In this paper we compare and analyze First-Explore in a cumulative-reward setting. However, for the sake of completeness, we discuss how First-Explore relates to final-episode-reward meta-RL methods in Appendix I.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: the claims in the abstract and introduction establish the claims of the paper and these claims are demonstrated empirically in section 6.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: there is a limitation section (section 7) which identifies multiple limitations, and suggests potential areas of future work to address these.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
related work: 
 Current cumulative-reward meta-RL methods train a single policy to maximize the cumulative reward of the whole episode sequence. However, optimizing directly for cumulative reward can prevent effective learning even on simple domains (e.g., bandits), as Section 1 describes and Section 6 demonstrates.RL 2 [8, 9] is one of the first (cumulative-reward) meta-RL methods. It uses an RNN to provide acrossepisode memory, and standard RL algorithms to train the agent. RL 2 has the advantage of simplicity. By training standard RL algorithms with the capacity for across-episode memory, the agent may learn to optimally balance exploration and exploitation across successive episodes. However, training dynamics can hinder achieving optimal performance (e.g., due to the requirement of sacrificing immediate reward (Section 1 and Section 6) or because the reward signal is too sparse [10] ).Subsequently, various cumulative-reward meta-RL works have been produced. VariBAD [7] outperforms RL 2 in certain domains. VariBAD achieves this improvement via splitting training into producing a posterior belief of the current task (task inference), and a task-posterior conditioned policy. HyperX [10] improves exploration during policy training by adding an incentive to visit novel states. This exploration incentive is gradually decreased throughout training, starting high and ending at zero. The issue is that the cumulative-reward term can still actively discourage exploration, as described in Section 1. Therefore, HyperX primarily tackles the issue of sparse exploration rewards, rather than overcoming the challenge of domains that actively disincentivize exploration.As Section 6 shows, when good early exploration requires forgoing immediate rewards, all these methods can fail to learn good behavior (e.g., converging immediately to a policy of never exploring). This failure arises not from reward sparsity (the problem HyperX tackles) or a lack of sophisticated posterior-belief-conditioned optimization (VariBAD's focus), but because the reward dynamics actively train the agent to avoid exploration. As such, First-Explore can achieve substantial total reward in domains in which RL 2 , VariBAD, and HyperX all perform poorly.AdA [12] demonstrates that meta-RL scales to complex domains and that training meta-RL on a curriculum of tailored learning challenges can produce agents capable of human-like adaption on complex held-out tasks. However, AdA may struggle to forgo reward, as the authors note that their agent always maximized current episode return (rather than the total reward of the episode sequence). This behavior suggests AdA's performance might depend on training and testing on environments that do not require such sacrificial exploration. Unfortunately, investigating AdA's dynamics is outside the scope of this paper.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: the paper makes several important conceptual contributions, such as identifying that cumulative meta-RL algorithms can be deceived by simple environment distributions (that standard-RL algorithms would easily solve) (section 1 and section 6). This problem is clearly identified and argued. However, these contributions are in the identification of this phenomenon, the empirical demonstration of it, and creating a novel algorithm (First-Explore) that can explore effectively despite these deceptive environments. As such, there are no mathematical proofs. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
first-explore: 
 First-Explore is a general framework for meta-RL that can be implemented in various ways. The framework trains two distinct policies, one policy to explore and one policy to exploit. Individually neither policy can achieve high cumulative reward, but after weight-update training, the policies are combined to produce an inference policy that does so. Figure 1 visualizes this process. By not directly training a single policy to maximize the total reward of the whole episode sequence, First-Explore avoids the failure mode of earlier approaches.The explore policy π explore performs successive episodes. During each episode, it has access to a context containing all previous actions, observations, and rewards within the current exploration sequence. In contrast, the exploit policy π exploit is trained to take the context c from the explore policy that has explored for a number of episodes between 1 and n, and to then run one further episode of exploitation. During this episode, the exploit policy has access to the actions, observations, and rewards from the current exploitation episode, as well as from the preceding exploration episodes (Figure 1 ).The explore and exploit policies are incentivized differently. The exploit policy is incentivized to produce high-return episodes (based on the provided context of previous explorations). The explore policy is instead incentivized to produce episodes that each, when added to the current context, result in subsequent high-return exploit-policy episodes. Training the exploit policy requires context from the explore policy, and training the explore policy requires the returns of subsequent exploits. This is efficiently achieved by interleaving the policies as depicted in Figure 1-A , where each rollout from the explore policy is followed by a rollout from the exploit policy.After the two policies are trained, First-Explore searches for a combination of them that maximizes the expected cumulative reward on newly sampled environments. Each candidate combination, first explores (via π explore ) for a set number of episodes k, and then exploits (via π exploit ) for the remaining n -k episodes. These combination policies are evaluated on independent environments sampled from the target distribution, with the combination that maximizes mean cumulative reward being chosen (see Appendix H for an example). This process of selecting k does not involve retraining the policies, and thus it is comparatively computationally cheap. As such, k should not be thought of as a hyperparameter, as unlike hyperparameters, the majority of the training compute expenditure is on policy weight updates that are performed before k is chosen.The resulting combination policy -first exploring for k episodes and then exploiting for n -k episodes -is then used as the inference policy π inference at test time, as shown in Equation 1.EQUATIONQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: code is provided for the full architecture, as well as for training and analysis. This code includes the main algorithm presented in the paper (First-Explore), as well code to replicate the controls that First-Explore was compared against. Furthermore, one of the main contributions is high level architecture which is clearly described in section 4. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
experimental setup: 
 First-Explore is implemented with a GPT-2-style causal transformer architecture [13] , chosen for its strong temporal sequence modeling capabilities. To simplify the design, the explore and exploit policies share parameters, differing only in their final linear-layer head.A novel sequence-modeling approach improved training stability in initial experiments, and was thus used for all First-Explore results. The method is described here and in the pseudocode provided in Appendix B. While this training method outperformed others in preliminary experiments, we believe the First-Explore meta-RL framework will work with alternate training approaches, such as PPO [16] , Q-learning [6] and other RL algorithms.The exploit policy π exploit is trained to generate episodes that match or surpass the highest return achieved in prior episodes within the meta-rollout sequence. The explore policy π explore is trained to produce episodes that are followed by the exploit policy achieving higher episode returns than those seen so far. These training incentives implement the First-Explore framework: the exploit policy maximizes immediate episode returns, while the explore policy generates episodes that increase subsequent exploit-episode returns.Training involves periodically updating the rollout policies (π explore , π exploit ) with successor versions. These successor policies (ϕ explore , ϕ exploit ) are trained to model the actions taken in the good rolloutpolicy episodes, defined as 1. good exploit episodes meet or surpass previous exploit returns in the meta-rollout sequence, and 2. good explore episodes are followed by the exploit policy achieving higher episode returns than those seen so far. We term these good exploit episodes 'maximal,' and the good explore episodes 'informative2 '. Since the first exploit episode in each meta-rollout has no previous episodes for comparison, a baseline reward b initializes the list of prior returns. This value is set as a domain-specific hyperparameter (but could easily be set automatically via heuristics).At the start of training, both rollout policies are initialized with random weights. They are then copied to produce the initial successor policies.Training is structured into iterated epochs. Each epoch, the current exploit and explore policies (π explore , π exploit ) produce batched meta-rollouts. In each batch, the exploit episodes τ exploit ∼ π exploit that are maximal and the explore episodes τ explore ∼ π explore that are informative are recorded. For these criteria-satisfying episodes, a cross-entropy loss is calculated between the successor policies (ϕ explore , ϕ exploit ) and the action distributions in the associated maximal or informative episodes. The successor policy weights are then updated, using gradient descent on the loss. In this way the successor policies learn to emulate the conditioned rollout policies (Equation 2). Finally, every T epochs, the rollout policies π explore , π exploit are updated to match the successor policies ϕ explore , ϕ exploit , ensuring continuous improvement. This hyperparameter T manages the trade-off between preserving behavioral diversity and accelerating learning iteration.EQUATIONDuring training, actions are sampled from their predicted probability distributions a ∼ π exploit or a ∼ π explore . In contrast, during inference, actions are selected greedily. When obtaining maximum cumulative reward requires forgoing early-episode reward, cumulativereward meta-RL algorithms fail to learn optimal behavior regardless of how long they are trained (Appendix F), as they become stuck on a local optimum of not exploring well. Even simple environments such as bandits flummox them. Three varied domains empirically demonstrate this dynamic. On all three a) the meta-RL controls perform poorly, and b) First-Explore significantly outperforms the controls. Further, two of the domains have modified versions that do not require forgoing immediate rewards, and this change causes significant control policy improvement 3 .When forgoing immediate reward is required, First-Explore achieves 2x more total reward than the meta-RL controls in the first domain, 10x in the second, and 6x in the last (Table 1 ). These significant reward differentials reflect substantive behavioral differences, e.g., First-Explore exploring well (at the cost of reward) and then exploiting vs. the meta-RL methods converging to a policy of minimal exploration. First-Explore outperforming the other methods is also statistically significant, with p-values less than 10 -5 for each comparison, as calculated by two-tailed Mann-Whitney U tests (MWU).Domain 1: Bandits with One Fixed Arm: The first domain is a multi-arm bandit problem designed to require forgoing immediate reward, where pulling the first bandit arm is immediately rewarding while also having no exploratory value. Pulling the first-arm yields a guaranteed reward of µ 1 , unlike the other arms, which -averaging across environments within the domain -yield expected reward 0. We consider two cases: µ 1 = 0.5 (the deceptive case) and µ 1 = 0 (the non-deceptive case).For both values, the greatest cumulative reward can be reliably achieved by first exploring to find the highest reward arm (with reward greater than µ 1 ) and then exploiting by repeatedly sampling it. However, µ 1 = 0.5 creates the deceptive challenge described in Section 1, where before reliable exploitation has been learnt, exploration (i.e., not sampling arm 1 and obtaining its guaranteed reward) leads to a lower total reward. See Appendix C.1 for further details on this domain.Bandit Results (Figure 2 ): First-Explore is evaluated against 3 baselines, with oracle performance (grey) and random action selection (black) plotted for additional reference.Deceptive Case: The worst-performing control, RL 2 [8, 9] (fuchsia) is stymied by the deceptive domain dynamic. Four of five RL 2 training runs (overlapping bold fuchsia) learn to only sample the guaranteed reward, achieving a constant 0.5 reward each pull (Figure 2 -B1), and exactly 50 reward after 100 pulls (Figure 2-A1 ). The remaining run (faint fuchsia) does better but still poorly. The best returns are achieved by methods that balance exploration (finding the most rewarding arm), and exploitation (pulling the best arm found so far). First-Explore (green) and the non-meta-RL bandit and non-deceptive (right) versions of the bandit domain. Each method trained 5 independent times, and each such run is plotted individually, so as to faithfully represent the variance between runs (e.g., that multiple of the bandit-domain RL 2 training runs achieve exactly the same reward). Appendix C provides alternative plots with mean reward ± standard deviation. The top figures plot the cumulative reward against the number of arm pulls, while the bottom figures illustrate the reward dynamics by plotting the individual pull rewards against the same. When the domain is deceptive, the cumulativereward meta-RL method, RL 2 (fuchsia), performs extremely poorly, despite the deceptive domain giving strictly higher rewards than the non-deceptive version. In contrast, First-Explore (green) impressively outperforms UCB (purple) and Thompson Sampling (orange) despite them being specialized bandit algorithms, in both the deceptive and non-deceptive settings, with p < 10 -5 . controls, Thompson Sampling [17] (orange) and UCB [8] (purple), both exhibit this behavior. Interestingly, First-Explore achieves the balance differently from the bandit algorithms, with the average pull rewards transitioning from close to zero, to close to optimal sharply around 10 pulls (Figure 2-B1 ). Impressively, First-Explore achieves greater reward than even the specialized bandit algorithms despite being applicable to more general domains (p < 10 -5 , as calculated by MWU). HyperX and VariBAD are not included as baselines because their performance was overly poor even on non-deceptive versions of this domain 4 . Non-Deceptive Case: When the domain is not deceptive (i.e., when exploration does not require forgoing immediate reward), RL 2 achieves high total-reward (Figure 2 -A2), and samples arms similarly to the bandit algorithms (Figure 2 -B2). All other algorithms perform well regardless of deception. This dynamic validates that the need to forgo immediate reward is what stymies cumulative-reward meta-RL performance.Illustrating that SOTA cumulative-reward meta-RL can fail on such simple domains (i.e., where standard-RL can easily solve each individual environment) is a key contribution of this paper.The second domain is a grid world environment, where the agent cannot see its surroundings. In each environment, there are multiple randomly positioned reward locations. When each location is encountered, the agent consumes it for that episode, receiving a reward ∼ U [ρ, 2]. The agent receives only its current coordinate as an observation, and to explore the agent must move blindly into unobserved grid locations that it has not visited before.We consider two values of ρ: ρ = -4 (the deceptive case) and ρ = 0 (the non-deceptive case).For both values, high total reward is achieved by first exploring (visiting unseen grid locations to find positive reward locations) and then consistently exploiting (revisiting discovered positive rewards locations each episode). However, when ρ = -4 this process is challenging, as only 1 3 of the reward locations are positive, and the expected reward of a positive location is only 1. These expectations create the challenge described in Section 1, where one must consistently exploit (by revisiting discovered positive reward objects more than three times) for any initial exploration to be worthwhile. See Appendix C.2 for an explanation of these properties, along with further domain details. In contrast, when ρ = 0, visiting unseen locations provides positive expected reward causing immediate rewards to incentivize (rather than deter) initial exploration. Each method trained 5 independent times, and each such run is plotted individually. The top figures plot the cumulative reward obtained against step and episode number, while the bottom figures provide a proxy for exploration by plotting the number of times agents move against the same. When the domain is deceptive, the cumulative-reward meta-RL methods, RL 2 (fuchsia), HyperX (brown), and VariBAD (purple) achieve low total-reward, as the policies learnt to minimize exploration. In contrast, First-Explore (green) performs well on both the deceptive and non-deceptive domains.Dark Treasure Room Results (Figure 3 ): First-Explore is evaluated against three meta-RL controls, with random action selection (grey) plotted for additional reference. Deceptive Case: The meta-RL algorithms RL 2 (fuchsia) and VariBAD (pale lilac) all achieve close to zero reward (Figure 3 -A1), and rarely move (Figure 3 -B1). HyperX explores more but minimally, as reflected by the modest negative first episode return 5 . However, the method fails to then exploit, and thus under-performs the other methods, despite this exploration.By first exploring and then exploiting, First-Explore (green) achieves 10x more total reward in the ten-episode setting than the meta-RL controls. Impressively, despite the high cost of exploration (with a single episode of exploration taking more than three subsequent exploitations to yield positive expected reward), First-Explore is sufficiently skilled that its policy of exploring twice and then exploiting achieves more total-reward than a hand-coded policy of optimally exploring in the first episode, and then optimally exploiting conditioned on this initial exploration 6 (2.0 > 1.78). Non-Deceptive Case: the meta-RL controls achieve reasonable total-reward (Figure 3-A2 ) and move consistently (Figure 3 -B2), demonstrating that it is the difficulty of forgoing immediate rewards that challenges SOTA methods. However, First-Explore still outperforms them, with p < 10 -5 (MWU).The final domain is significantly more complex than the previous ones, and demonstrates that First-Explore can scale beyond bandit and grid-world problems. The domain is composed of randomly generated mazes with 3 reward locations. The agent must parse a large number of lidar observations (see Figure 4 ) to navigate around walls, identify reward locations, and move onto them. Each episode, the first reward location visited gives a reward of either -1 or +1, each reward location having an independent 30% chance of positive reward. The agent has three discrete actions: rotate right, rotate left, and move forward. One complexity is that (unlike the grid-world domain), actions do not commute (e.g., rotating left and then moving forward is different from moving forward and then rotating left). The agent can see the reward locations, but cannot tell by looking whether they give positive or negative reward. Optimal behavior requires first exploring (navigating to visit un-visited reward locations, and obtaining their negative expected reward), and then reliably exploiting (re-visiting a reward location if it gave positive reward). The need to sacrifice immediate reward (by visiting the negative reward locations) challenges the SOTA cumulative-reward baselines. Ray Maze Results (Figure 5 ): On this more complex domain, First-Explore similarly achieves significantly more reward than the meta-RL controls, with all controls failing to perform well. Two of the three meta-RL controls, RL 2 (fuchsia) and VariBAD (purple), learn to never move (and instead spin in-place). This behavior avoids encountering any reward location, and thus results in exactly 0 total reward in every evaluation of all five independent training runs. For the final meta-RL control, four of five HyperX (brown) runs move little (achieving near zero reward), with the remaining run achieving only modest cumulative reward.First-Explore (green) significantly outperforms the controls, achieving 0.47 mean reward (Table 1 ), by exploring to find a reward location, remembering it and reliably exploiting by navigating to that location if it gave positive reward. Figure 5 reflects this process, illustrating that First-Explore's cumulative reward goes down substantially before then going up. For comparison, the optimum possible behavior of exploring and exploiting perfectly achieves at most 0.64 average reward 7 .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: the submitted code contains sufficient instruction to (along with the paper) faithfully reproduce the main experimental result. Guidelines:• The answer NA means that paper does not include experiments requiring code. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: there are no direct societal implications of First-Explore, however we discuss First-Explore being an exciting opportunity for the field of meta-RL (and so RL more generally). We believe the field as a whole has great potential to create powerful algorithm that if used responsibly could greatly benefit all of humanity. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
limitations and future work: 
 As presented, First-Explore does not actively explore to enable future exploration, because the explore policy π explore only trains to increase the expected reward of the subsequent exploit policy episode. Unfortunately, a sequence of optimal myopic explorations is not necessarily an optimal exploration sequence (Appendix G). A potential solution is to reward exploration episodes based on a weighted sum of rewards from all subsequent exploitation episodes, analogous to summing discounted future rewards in standard RL.Another limitation is that First-Explore could be unsafe in certain environments. The risk is that First-Explore's explore policy does not avoid negative rewards, and so penalizing unsafe action has minimal effect on the policy. For example, a chef robot attempting a new recipe in a physical home might explore mistakes that endanger humans or damage the kitchen (despite negative rewards telling it not to). This concern only applies to a subset of environments, as many environments are safe, e.g., simulated ones. That said, addressing this concern is a valuable direction of future work. One potential solution is to add a safety penalty to the explore policy, i.e., train the explore policy to maximize subsequent exploit episode-returns while also avoiding safety risks. This proposed version of First-Explore could actually produce in-context adaption that is safer than standard RL training, as the meta-RL policies would have learnt a strong prior of avoiding potentially dangerous actions.This paper makes no claims regarding First-Explore's meta-training efficiency, and future work may substantially improve meta-training time. Instead, this paper a) identifies a challenge that SOTA meta-RL algorithms fail on regardless of training time (the need to forgo immediate rewards), and b) provides a framework, First-Explore, that can solve these challenging domains.A final problem is the challenge of long sequence modelling, with certain environments requiring a huge context and high compute (e.g., can one have a large enough context, and enough compute to allow First-Explore to generalize and act as a replacement for standard RL?). AdA [12] suggests such a project might be possible. Progress on efficient long-context sequence modelling [18, 19] , research on RL transformer applications [20, 21] , and Moore's Law all make this possibility more feasible.Additionally, given that First-Explore learns a dedicated explore policy and a dedicated exploit policy, and both have been shown to work well (e.g., Figure 2 ), the method may be applicable to final-episode-reward meta-RL settings (Appendix I). Further, First-Explore always switches from exploration to exploitation after a fixed number of episodes. Future work could replace this fixed number with a learnt classifier that determines when to switch from exploration to exploitation, e.g., repeatedly exploring until sufficient information is obtained. However, despite these applications being highly exciting directions of future work, a proper investigation of either would require its own paper, including many different specialized controls and environments pertinent to the setting. 
discussion: 
 Given that First-Explore uses RL algorithms to train the meta-RL policy, how might it solve hardexploration problems that standard RL cannot, e.g. design a rocket for the first time? We believe that given a suitably advanced curriculum, and sufficient compute, First-Explore could learn powerful exploration heuristics (i.e., develop intrinsic motivations such as an analogue of curiosity) and that these heuristics would enable sample-efficient performance on hard sparse-reward problems. On a curriculum, initially First-Explore would explore randomly, and learn to exploit based on that random exploration. Once it has learnt rudimentary exploitation, the agent can learn rudimentary exploration. Then it would learn better exploitation and exploration, and so on, each time relying on there being 'goldilocks zone' tasks [22] that are not too hard and not too easy.Further, while curricula can aid all of meta-RL, e.g., RL 2 and AdA, First-Explore can have a significant training advantage on certain problems (e.g., in the ten-episode Dark Treasure-Room, First-Explore achieves positive cumulative reward while the standard cumulative-reward meta-RL methods catastrophically fail). This advantage could potentially allow far greater compute efficiency, and allow training on otherwise infeasible curricula. 
conclusion: 
 We have theorized and demonstrated that SOTA cumulative-reward meta-RL fail to train when exploration requires forgoing immediate reward. Even simple problems such as bandits (Figure 1 ) can stymie these methods. To overcome this challenge, we introduce a novel meta-RL framework, First-Explore, that learns two separate interleaved policies: one to first explore, another to then exploit. By combining the policies at inference time, First-Explore is able to explore effectively and achieve high cumulative reward on problems that hamstring SOTA methods.Meta-RL shows the promise of finally fixing the main problem in RL -that it is extremely sample inefficient -even producing human-level sample efficiency [12] . However, the promise of this approach is limited, as we have shown, on a large set of important problems. We can only take advantage of this approach if we can harness the benefits of meta-RL even on such problems, and First-Explore enables us to do so, thus offering an important and exciting opportunity for the field. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: none of the models or data involved in this paper have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To ensure full replicability, we are releasing the code used to train First-Explore and the controls, along with the environments trained on. We are also releasing the weights of a trained model for each domain. Each model contains both the explore and exploit policies as separate heads on the shared trunk. The code is available at https://github.com/btnorman/First-Explore. Problem Description: Dark Treasure-Rooms (inspired by the Darkroom in [20] ) are 9 × 9 grids full of treasures and traps. The agent starts in the middle of the grid, navigates (up, left, down, or right) to find treasure, and cannot see its surroundings. The agent can take five actions: four for cardinal movement, and one to not move. Only its current (x, y) coordinates are observed. Each environment has 8 objects (treasures or traps), and when the agent encounters a treasure or trap it consumes/activates it, and receives an associated reward (positive or negative). The treasure and trap values v i are uniformly distributed in the range ρ to 2, v i ∼ U [ρ, 2], with ρ being the maximum trap penalty. The locations of the objects are sampled uniformly, with overlapping objects having their rewards/penalties summed. Each episode has 9 steps, and the objective is to maximize the expected cumulative episode returns on a newly sampled Dark Treasure-Room over multiple episodes.See Figure 7 for a version of Figure 3 that plots mean ± standard deviation instead of each individual training run. How difficult the Dark-Treasure room domain is strongly depends on the value of ρ. The dynamics are as follows:• The fraction of positive rewards equals 2 2-ρ . • The average value of a positive reward is always E(U [0, 2]) = 1 (assuming ρ < 0).• The average value of a negative reward isE(U [ρ, 0]) = ρ 2 . • The expected reward of visiting random reward location (treasure or trap) is E(U [ρ, 2]) = 2+ρ 2 .Using these, we can calculate the expected reward of visiting a random reward location and then returning to it n times if and only if the location gave a positive reward. This expectation is equal to the expected reward of the first visit plus n times the chance of the visited reward location giving positive reward multiplied by the expected positive reward (Equation 3).EQUATIONWe can then calculate the number of revisits required for exploration (visiting a random reward location for the first time) to be worthwhile (i.e., lead to positive expected reward). Equation 4illustrates this calculation, for values of ρ < -2. This number of revisits corresponds to how reliable the exploitation policy must be for exploration to yield higher cumulative reward. For example, when ρ = -4 on the ten horizon setting, the agent needs to exploit correctly more than 1 3 of the time, as (after exploring) it has at most nine episodes to exploit, and must exploit correctly more than three times. If exploitation reliability is lower than this value, then exploration will be actively discouraged.EQUATIONWhen ρ = -4, n > (-4) 2 -4 4 = 3. Thus, ρ = -4 corresponds to exploration (visiting a new reward location) requiring more than three potential revisits to be worthwhile (have positive exploration). As (Section 1) describes, this dynamic challenges cumulative-reward meta-RL methods, as exploration thus requires reliable exploitation to be worthwhile. This value of ρ = -4 was chosen as it a) requires a reasonably high number of revisits, and b) is an integer number that corresponds exactly to exploration being worthwhile only when the exploitation policy on average performs more than an integer number of revisits.In contrast, as Section 6 describes, when ρ = 0, exploration is directly incentivized (and no revisits are required for exploration to increase the expected total reward).For completeness, we also calculate the dynamics that occur when ρ = -2 √ 2 and when ρ = -2 √ 3, which corresponds to requiring more than 1 and more than 2 revisits respectively 8 (Figure 8 and Table 2 ). Even when the domain is made significantly less challenging (i.e., for more positive values of ρ), First-Explore signficantly outperforms the controls. x corresponds to exploration (visiting a random treasure location) requiring more than x -1 revisits. Even when the domain is made significantly less challenging (i.e., for more-positive values of ρ), First-Explore signficantly outperforms the controls.ρ = -2 √ 2 ρ = -2 √ 3 ρ = -2 √ 4 First-Explore 4.4 ± 0.3 3.1 ± 0.3 2.0 ± 0.3 RL 2 0.4 ± 0.1 0.2 ± 0.1 0.2 ± 0.1 VariBAD 0.7 ± 0.3 0.3 ± 0.1 0.2 ± 0.1 HyperX 0.8 ± 0.5 -0.1 ± 0.1 -0.2 ± 0.2In this domain, perfect exploration in the first episode corresponds to visiting 9 unique coordinates, as each coordinate has an equal chance of being a reward location (a treasure or a trap). This process discovers 9 * 8 81 = 8 9 reward locations on average, as there are 8 reward locations spread over 81 coordinates, and the agent visits 9 of the coordinates.Assuming the agent then optimally myopically exploits (i.e., only maximizing current-episode reward and not attempting to explore more), the total reward is at most 8 9 times the expression in Equation 3 (the expected number of reward locations multiplied by the expected value of perfectly exploiting each individually). Equation 7calculates this expectation for ρ = -4 and n = 9 (the values of the deceptive Dark Treasure Room domain in Section 6). Note, n is the number of subsequent exploitations, after visiting a new reward location, and so the n = 9 calculation corresponds to the ten episode setting. E(explore optimally then optimally myopically exploit) ≤EQUATIONEQUATION≤ 1.77 ≈ 1.78This calculation is an upper bound, as it assumes the agent can effectively teleport to discovered positive reward locations. In reality, the agent may not be able to revisit some discovered positivereward locations without moving into discovered negative-reward locations.Problem Description: Ray Maze challenges the algorithms with a more complex domain. The agent must navigate a randomly generated maze of impassible walls (the maze layout is different for each sampled Ray Maze) to find one of three goal locations. Each goal location is either a trap or a reward, with a 0.3 probability of the goal being a treasure, and a 0.7 probability of it being a trap. Treasures give reward 1, while traps give penalty -1. In this domain, the agent can only receive one goal reward per episode, as triggering the first prevents others activating. To perceive the maze, the agent receives 15 lidar observations (see Figure 4 ). Each lidar observation reports the distance to the nearest wall along an angle (relative to the agent's orientation). It further tells the agent whether the lidar ray hit a goal location, as well as the wall orientation (east-west or north-south). However, the lidar measurements do not show whether a goal is a trap or a treasure. The agent has 3 actions, turn left, go forward, and turn right, with rotation turning the agent's field of view.Ray Maze is a challenging domain for several reasons. It has a high-dimensional observation space (15 separate lidar measurements), complex action dynamics (with actions not commuting, e.g. turn left then move forward is different from move forward then turn left) and a randomly generated maze that interacts with both movement and observation. Furthermore, similarly to earlier environments, the agent must learn from experience, and risk the traps in early episodes, so as to exploit and consistently find treasure in later ones. Because of how frequent traps are, the agent can only obtain positive cumulative reward by a) searching for treasures in early episodes (at the cost of expected reward, as the average value of a goal is negative) and b) reliably exploiting in later episodes (navigate to identified treasures, while avoiding potential traps).See Figure 9 for a version of Figure 5 that plots mean ± standard deviation instead of each individual training run. Despite being a complicated environment, Ray Maze has simple reward dynamics.• Each goal location has an independent 0.3 chance of yielding +1 reward, otherwise it gives -1 reward.• All goal locations are independent, and the agent can only receive one goal reward each episode Thus, perfect exploration within an episode involves always visiting a goal location (potentially finding a location that gives positive reward), and perfect exploitation involves always visiting a positive reward location if one is known, while avoiding all negative and unknown goal locations.Assuming the agent always revisits positive goal locations, the expected reward of visiting an new goal location with n episodes remaining is -0.4 + 0.3n (the expected value of the first visit, plus the expected value of revisiting positive rewards n times).• In the first episode the expected value of visiting a new goal location is -0.4 + 0.3 × 3 = 0.5.• In the second episode, the expected value is -0.4 + 0.3 × 2 = 0.2.• In the third episode, the expected value is -0.4 + 0.3 × 1 = -0.1, and so not worthwhile.Thus the optimal policy is to visit a goal location in the first episode (and then repeatedly revisit it if there was a positive reward), and then if the first goal location was negative (70% chance), visit a goal location in the second episode.This policy achieves 0.5 + 0.7 * 0.2 = 0.64 expected total reward. This value is an upper bound, as it is possible that the agent cannot always find a goal location.D Tabulated Results: Each training run commanded a single GPU, specifically a Nvidia T4, and up to 8 cpu cores. Table 6 gives the approximate walltime of each run. Notably, in the Dark Treasure-Room for ρ = -4, VariBAD and RL 2 rapidly converges to a policy of staying still, and while HyperX seems to slowly improve rewards, the reward increase is an artifact caused by the HyperX having an exploration incentive that is gradually attenuated to zero. This attenuation creates a slow convergence from negative reward (due to moving into traps and not exploiting) to a higher near zero reward (obtained by mostly staying still). Because the attenuation is designed to occur throughout the entire run, scaling the run length merely scales how long HyperX takes to converge to close to zero reward.Due to a desire to not waste compute on converged policies, once this behavior was verified, the control runs on this setting were limited to 10 hours. In contrast, First-Explore was run for longer, as it continued to improve with additional training. This is a fair comparison, as due to the controls having converged, increasing the control run training time would not yield better policies, as Figure 10 demonstrates. To this end, as RL 2 on the bandit environment still showed improvement after 10 hours, those runs were also extended.Total compute used for the experiments would then be around 1100 hours (5 runs for each treatment). However, there was also hyperparameter search, e.g., for the RL 2 bandit parameters. As such, total compute may be over 2000 GPU hours. Furthermore, there were many preliminary experiments to iterate on the First-Explore architecture as well as to research and identify cumulative-meta-RL deception.The issue is not that the methods are slow to converge, and that with more training they could perform well. As Section 1 describes, these controls achieve low cumulative reward regardless of how long the methods are trained. Figure 10 demonstrates this phenomenon on the deceptive treasure room. Figure 11 : Well-planned sequential exploration can sometimes significantly outperform a sequence of optimal myopic explorations. For example, consider exploring a plain over four days, where each day one must explore by walking from the plain's center. Sequence of Optimal Myopic Explorations one 'optimal' way of exploring is to perform a spiral from the center (e.g., the red spiral on the right). This strategy achieves the optimal amount of exploration on day 1 as one never retraces one's steps. However, if one does a spiral on day 1 then on day 2, one must retread old ground -wasting time otherwise spent exploring new locations. Each day bee-lining to unseen areas and then spiralling from there is also optimal for that day, however it increases the amount of retreading tomorrow.Optimal Sequence of Explorations: another optimal way of exploring on day 1 is to explore a quadrant, visualized in red on the left. Again, as one does not backtrack, this strategy is optimal on day 1. However, unlike the spiral strategy, this strategy is also part of an optimal sequence of four explorations, as one can explore a new quadrant each of the four days, without ever retreading the same ground. Methods such as MetaCURE [15] , EPI [23] and CCM [24] learn an exploration policy that aims to extract maximum environment information (independent of whether such information informs good exploitation). These approaches discard grounding exploration in (maximizing) future reward. Not grounding exploration in future exploitation reward means that the policy may learn (via weight updates) to spend meta-rollouts acquiring irrelevant information (e.g., the exact penalty of bad actions). This distraction potentially prevents optimal exploration from ever being learnt. E-RL 2 [25] modifies RL 2 to ignore the first-k episode rewards. This modification enables pure exploration (that is not dissuaded by negative rewards). However, E-RL 2 introduces an acrossepisode value assignment problem: identifying which exploration episodes enabled good subsequent exploitation. This problem potentially limits training sample efficiency. Further, the exploratory episodes number k is set as a hyperparameter and constant across all tasks (both at training and at inference), preventing efficient combination with a curriculum that contains different difficulty tasks (as hard tasks may need significantly more exploration episodes than easy ones). Finally, hard coding k limits the flexibility and usefulness of E-RL 2 because one cannot explore until a satisfactory policy quality is reached, preventing meta-RL in-context adaptation from off-the-shelf replacing standard RL.DREAM [11] also separately optimizes exploration and exploitation policies (and grounds exploration in exploitation), but has four complex, manually designed, interacting components and a reliance on knowing unique problem IDs during meta-training. This complexity enables increased sampleefficiency by avoiding the chicken and egg problem of simultaneously learning explore and exploit policies. Unlike E-RL 2 , because a part of DREAM's machinery must learn to produce the right information per problem based on the (unique, random) problem ID only, it is unable to generalize or handle never-seen-before challenges during meta-training, raising questions about its scalability and generality. For example, DREAM may potentially be difficult to apply to problems where each training environment is unique (e.g., for environments with continuous variables, or samples from otherwise vast search spaces). It may also struggle when each environment is a hard-exploration challenge, as it may be difficult for the model to explore enough to learn which information is required to solve the problem. We believe curricula are necessary to solve such environments. However, because DREAM cannot generalize during meta-training (as described above), it cannot take advantage of a curriculum to build an exploration skill set to tackle harder and harder exploration challenges.Applying First-Explore to a final-episode-reward meta-RL setting is a promising direction of future work, as First-Explore i) learns grounded exploration ii) can explore until sufficient information is obtained ( rather than having a fixed number of explorations), and iii) does not rely on privileged information (problem IDS), allowing generalization during meta-training.Evaluation (sampling the multiple evaluation environments and performing iterated rollouts) was with a single GPU. For the Bandit Results, each of the First-Explore evaluations sampled 128 × 100 bandit environments, indepedent from those trained on. For the 5 RL 2 bandit evaluations batch size was reduced to 2, 000 (due to taking longer to evaluate). Since there is no meta-RL training variance for UCB1 and TS, five independent evaluations were done, each with an independently sampled 10, 000 bandits. UCB: UCB was implemented according to the description in [8] , with c = 1. Namely, each pull, UCB picks the arm that maximizes ucb i (t) = μi (t -1) + 2 log t Ti(t-1) , where μi (t -1) is the estimated mean reward of the ith arm, T i (t -1) is the number of times the ith arm has been pulledFor the Dark Treasure-Room and the Ray Maze domains, all policies were evaluated on a batch of 1, 000 environments sampled independently from those trained on.The official VariBAD [7] (VariBAD and RL 2 ) and HyperX [10] (HyperX) codebase ran the meta-RL controls. Dark Treasure-Room trained with the default hyperparameters of the coded bases gridworld environments. These were found to perform well, with variations tried not yielding improvement. To provide a strong control on the Bandits with One Fixed Arm problem, the controls were advantaged by having individual hyperparameter gridsearches for each µ 1 value (unlike First-Explore). See the SI attached configuration file for the exact hyperparameters. Both of these codebases are licensed under a MIT license.The architecture for both domains is a GPT-2 transformer architecture [13] specifically the Jax framework [26] implementation provided by Hugging Face [27] , with the code being modified so that token embeddings could be passed rather than token IDs. The different hyperparameters for the two domains are given in Table 7 . The code being provided with a Modified MIT License (allowing free use with attribution).For all domains each token embedding is the sum of a linear embedding of an action, a linear embedding of the observations that followed that action, a linear embedding of the reward that followed that action, a positional encoding of the current timestep, and a positional encoding of the episode number. See the provided code for details. For the dark treasure-room environments a reset token was added between episodes that contained the initial observations of the environment, and a unique action embedding corresponding to a non-action. The bandit domain had no such reset token. For training we use AdamW [28] with a piece-wise linear warm up schedule that interpolates linearly from an initial rate of 0 to the full learning rate in the first 10% training steps, and then interpolates linearly back to zero in the remaining 90% of training steps. Table 8 gives the optimization hyperparameters. Hyperparameters were chosen based on a relatively modest amount of preliminary experimentation. Finally, for efficiency, all episode rollouts and training was done on GPU using the Jax framework [26] .For evaluation, we then sample by taking the argmax over actions, and do not add the ϵ-noise. Here are example iterated First-Explore rollouts of the two trained policies, π explore , π exploit , visualized for a single sampled darkroom.Figure 14 : The first (First-Explore) explore episode. Top left visualizes the last step of a First-Explore explore episode, with the locations that are not in the cumulative context being coloured white, as the agent is blind to them (having no observations or memory of those locations). This figure plots the end of the first exploration, and shows a reward has been found. Bottom left visualizes the coverage of the cumulative context by plotting the total number of unique locations visited by the exploration against the cumulative episode step count. In this explore, the agent never doubled back on itself, which is good as it is optimal to have as many unique locations visited as possible. Top right visualizes a step in a First-Explore exploit episode, with the locations that are in context visualized.The agent can effectively 'see' those locations in its memory. Bottom right plots the exploit reward against the exploit episode timestep. As this figure plots before the start of the exploit episode, the agent has yet to move and encounter rewards, but will have done so in the subsequent visualizations. The end of the fifth (First-Explore) exploit episode. After consuming the nearby reward the agent has reached the newly discovered reward at the top of the room and consumed it. This success required making no mistakes and navigating first to the nearby reward then to the top one on the first try. This inference is possible because the quickest the agent can reach both rewards is exactly the length of the episode (9 steps). The navigation in this episode is an example of intelligent exploitation, as after the information reveal (the reward at the top) of a single episode the agent appropriately changes its policy based on the context and using the learnt environment prior (e.g., how to navigate), produces a highly structured behavior (navigating with no mistakes).NeurIPS Paper Checklist[14] use different terminology, calling cumulative-reward meta-RL "zero-shot meta-RL", and final-episodereward meta-RL "few-shot meta-RL."As in all RL, improvement is a noisy process, (e.g., an episode labeled 'informative' might not be genuinely informative, due to random noise in the exploit reward). However, in expectation the labeling is correct.Surprisingly, First-Explore outperforms the controls even on the non-deceptive domains, p < 10 -5 (MWU). While promising grounds for future work, this paper only claims that First-Explore outperforms other methods in domains that require forgoing immediate reward.Despite hyperparameter tuning, good performance on either distribution (deceptive and non-deceptive) could not be achieved. Though bandits seem simple, the distribution of environment arm means is large and continuous. This may cause problems for VariBAD and HyperX, which were designed for and evaluated on small discrete task spaces, e.g. a goal being in one of 25 locations.The amount of exploration in the first episode is equal to the number of unique locations visited which is exactly proportional to the average negative first-episode return. As such, while HyperX moves a lot more than VariBAD and RL 2 , it mostly moves into locations previously visited.See Appendix C.2 for a mathematical derivation.See Appendix C.3 for an upper bound on optimal performance.As found by solving Equation 3, ρ = -2 √x corresponds to exploration (visiting a random treasure location) requiring more than x -1 revisits to yield increased total reward. 


Decision: The paper presents an interesting and valuable contribution to the Meta-RL field by addressing the challenge of balancing exploration and exploitation, particularly in the presence of deceptive rewards. 
However, the concerns regarding mathematical clarity, computational cost, and the limited complexity of the tested environments suggest that there is still room for improvement. The paper would benefit from further refinement, particularly in the explicit formulation of its objectives and broader experimentation. 
Given the overall positive trajectory of the reviews post-rebuttal, I recommend accepting this paper.
The authors need to incorporate all the improvements proposed during the rebuttals and all the suggestions provided by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenge in meta-reinforcement learning, where agents struggle to perform intelligent exploration across episodes, often failing to avoid repetitive exploration of the same locations. The authors propose a method called First-Explore, which learns two distinct policies: one focused on exploration and another on exploitation. By separating these policies, First-Explore aims to overcome the limitations of existing cumulative-reward meta-RL methods, which maximize cumulative rewards rather than maximizing the rewards of the final episode and, hence, not exploring adequately. The proposed method improves performance in environments like deceiving bandits and dark treasure rooms.
strength_weakness: strength_weakness: 1. The idea of separating exploration and exploitation in meta RL is not new. Several prior works, like MetaCURE[1] and DREAM [2], proposed similar ideas, but they are not properly discussed or compared.

2. The experiment is sufficient. The proposed approach is tested only on two tabular enironments, namely Meta-RL-Deceiving Bandits and Dark Treasure-Rooms. It is unclear how would the proposed method performance on more complex tasks like continuous control.

3. The writing of the method section can be improved. It needs proper formulas or an algorithm box to clearly demonstrate the methods for training the exploration and exploitation policies.


[1] Zhang, Jin, et al. "Metacure: Meta reinforcement learning with empowerment-driven exploration." International Conference on Machine Learning. PMLR, 2021.

[2] Liu, Evan Z., et al. "Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices." International conference on machine learning. PMLR, 2021.
questions: questions: 1. Can you give explicit formulas of the objectives for training exploration and exploitation policies?

2. Can you explain why the proposed method takes so long to run on the tabular environments (50 GPU hours, according to Appendix C.1)?
  
3. How does the proposed perform on more complex tasks, like continuous control tasks (e.g., Meta-World)? How does the proposed method compare with MetaCURE and DREAM?
limitations: limitations: The authors discussed limitations of their work.
paper_summary: paper_summary: This work proposes a novel Meta-RL framework called First-Explore to address the balance between exploration and exploitation. The method learns two distinct policies: one for exploration and one for exploitation. These two policies are then combined to form the final inference policy. The effectiveness of the method has been demonstrated in bandit tasks.
strength_weakness: strength_weakness: 1. The writing of the paper needs more careful organization. Figures 1 and 2 are not cited in the main text, which makes it difficult to understand their relevance. Additionally, the training process for the explore and exploit policies is not well explained in Section 4.

2. The inference policy appears to require exhaustive hyperparameter tuning (k), which limits the method's generalization and makes it challenging to apply to other tasks.

3. The compared baselines seem outdated. It would be beneficial to provide comparison results with more recent methods to better demonstrate the effectiveness of the proposed approach. Additionally, the experimental results are not significant, as the proposed method does not outperform other baselines by a substantial margin in most tasks.
questions: questions: 1. What is the practical objective of the exploration policy, and how can the informativeness of an episode be defined?

2. I am curious about the training efficiency of the proposed method. Is it more efficient than learning an individual policy for each task?
limitations: limitations: The authors have discussed the limitations.
paper_summary: paper_summary: This paper proposes a method to mitigate the issue of deceptive rewards in Meta-RL, I.e., rewards that may impede further exploration leading to higher cumulative rewards. The authors propose to learn two policies: an exploration and exploitation policy that are conditioned on a common context. During policy training the reward from the exploitation policy is fed through the exploration policy via the context as feedback. In the next phase the amount of necessary exploration iterations are determined through the cumulative reward. When the optimal cutoff is found the final policy is the combination of the exploration and exploitation policy based on the optimal cutoff $k$. The authors test on a bandit problem and dark-treasure room environment, mitigating the issue of deceptive rewards compared to other known Meta-RL approaches.
strength_weakness: strength_weakness: * The choice of $\rho=4$ seems a bit arbitrary. I think $\rho$ should be ablated to see what the effect of different penalties is on the proposed method and other methods is.
* The environments chosen for the experiments seem very simplistic. It is difficult to assess whether the method would succeed when having to learn long-horizon and more complex policies. In this case it might become intractable to get feedback from the exploitation policy  to decide on the right amount of exploration steps.
* I think a good test would also have been environments such as MiniGrid or Minihack for example, which have also been used for hierarchical reinforcement learning.  The required sequence of actions is more complex than the proposed environments, while still being observationally simple.
questions: questions: * How do you think your method scales to more complex environments? Would it be possible to maybe make a more complex dark-treasure-room?

* In the caption of Figure 4 I don’t understand the following statement: 
	> By first exploring for two episodes using πexplore, and then exploiting 	using πexploit, First-Explore’s inference policy πinference achieves significant cumulative reward...”

* I thought this cutoff was determined automatically and also considering the environments are procedural why does the value of 2 work for all the configurations? Shouldn’t this value vary depending on where the different objects are spawned?

* Why does the performance of your method go down first in Figure 4 while this does not happen for the other methods?
limitations: limitations: I think the authors have properly addressed the limitation of their work.

Comments: title: Thank You
Thank You
Response
One More Question
Question Answers Part 2
Question Answers Part 1
Response to Author's Rebuttal
comment: Thanks for the detailed comments. My concerns about the novelty part are resolved. However, I still find that the paper lacks an explicit mathematical formula for the objective of explorative and exploitive policies and that the computation cost can be too high. Considering the above, I have changed my score to 5.
We are sincerely grateful for your questions and the time you invested engaging with our work, and we are delighted that you value the additional experiments and explanations. We greatly appreciate your score increase, and hope that (in tandem with the other reviewers and the AC) it leads to this work being published and thus shared with the NeurIPS community.
I appreciate the authors efforts in addressing my questions.

- Regarding Minihack and Minigrid, I didn't think your method was Hierarchical Reinforcement Learning based, but these environments lend themselves to test your scenario as well and have been quite established in the literature. However, this is a minor point for me. 

- Overall, with the added experiments and explanations, I think this is a decent paper that addresses a gap when deceptive rewards have to be foregone Meta-RL. Overall, it seems to me this is a still unexplored issue in RL in general.

I will raise my score to 6 and keep my confidence at 3
We sincerely value your questions and the time you invested engaging with our work. Your feedback led to revisions that significantly strengthened our paper. We are delighted that our responses have clarified your concerns and appreciate your support for sharing our work with the NeurIPS community.
Thank you for addressing my questions. I appreciate the effort put into clarifying my concerns, and based on your responses, I am willing to raise my score to 'weak accept'.
Thank you for your question. First-Explore is an example of meta-reinforcement learning (meta-RL), which is an instance of meta-learning. Similar to RL$^2$, First-Explore operates within the meta-RL framework (see the definition below) by using machine learning to develop an inference policy. Once produced, this policy then functions as a “reinforcement learning algorithm,” because, as stated in lines 120-122 of Section 2, the policy adapts to new environments by leveraging memory of previous episodes (and associated rewards) to enhance performance in successive episodes (e.g., learning to pull the bandit arm that yields the highest mean reward). We will modify the text to further emphasize this process of adaptation, and how it enables “learning to (reinforcement) learn.”

The bidirectional relationship between $\pi_{explore}$ and $\pi_{exploit}$ (with each relying on the other to train) is a unique feature of our approach but does not affect its nature as meta-RL (and thus meta-learning).

To clarify terms:

Meta-learning is a term used broadly across various areas of machine learning, where there is some sense of “learning to learn.” For instance, Oreshkin et al. [2020] describes meta-learning as “*usually linked* to being able to (i) accumulate knowledge across tasks (i.e. transfer learning, multi-task learning) and (ii) quickly adapt the accumulated knowledge to the new task (task adaptation)”

Meta-RL is a subset of meta-learning (corresponding to “learning to *reinforcement* learn”) and is more concretely defined. Beck et al. [2023] write, “meta-RL uses sample-inefficient ML to learn sample-efficient RL algorithms, or components thereof.” To elaborate, the “learning” is separated into two phases, a sample inefficient initial phase that produces an RL algorithm, and the RL algorithm (which can then “learn” by performing successive rollouts in environments). This structure is in First-Explore, as it trains the inference policy (by learning both $\pi_{explore}$ and $\pi_{exploit}$ and then combining them). The resulting inference policy then acts as a highly sample-efficient reinforcement learning algorithm (see above, along with the results in Section 5, and the main rebuttal).

Oreshkin, B. N., Carpov, D., Chapados, N., & Bengio, Y. (2020). Meta-learning framework with applications to zero-shot time-series forecasting. arXiv. https://arxiv.org/abs/2002.02887

Beck, J., Vuorio, R., Liu, E. Z., Xiong, Z., Zintgraf, L., Finn, C., & Whiteson, S. (2023). A Survey of Meta-Reinforcement Learning. arXiv. https://arxiv.org/abs/2301.08028
Thank you for your quick and clear response. I have one last question: Does First-Explore belong to the meta-learning framework? 

To my knowledge, the meta-learning framework operates on the principle of "learning to learn." For example, one learner acquires knowledge that helps another learner to perform a specific task. Typically, the relationship between these two learners is unidirectional. However, the two policies, $\pi_{explore}$ and $\pi_{exploit}$, in your approach seem to learn from each other, making their relationship bidirectional. 

Is my understanding incorrect?
> Is "informative" defined as achieving a higher return? 

“Informative” is defined for the exploration policy (only). It means those exploration episodes that (when the *exploit* policy gets to condition on them) are followed by the exploit policy achieving a higher return (than it would have otherwise, i.e. the exploit policy gets a higher return than it gets without adding this new explore episode to its context, meaning there is valuable information in that explore episode, hence it being “informative”). 

> What if the agent receives negative rewards before reaching the objective but receives positive rewards when deviating from the objective? Does the concept of "informative" still work in this context?

We are not sure what you are asking. Are you asking if the episodes labeled as “informative” are always truly informative? 

The process is noisy. On average, genuinely informative episodes (e.g., ones that do meaningfully inform the exploit policy) are most likely to be followed by an increase in reward and hence be labeled as “informative.” However, not all “informative” episodes will contain useful information. For example, the exploit policy might happen to achieve high reward (e.g., from the bandit noise term), despite not receiving new information from the explore policy, and the code would still label the explore-policy episode “informative,” and incentivize more such explorations.

It should be noted that this type of problem applies to all of RL. For example, a value function can erroneously identify an action as being high-value due to noise in the reward signal. Despite this noise, the concept is still valid, and as our results demonstrate, there is sufficient signal for the explore policy to learn effectively.

We will add text to Section 3 explaining this dynamic.

> [Figures 3 and 4 should display the mean and standard deviation]

Due to the non-normal distribution of our training run returns, representing variability with standard deviation could potentially mislead readers. For example, in the deceptive bandit domain, four out of five $RL^2$ runs consistently select the same arm, leading to a highly skewed distribution. To accurately represent the variability across runs, the individual training runs are plotted instead. However, for completeness, we will include alternative plots with mean and standard deviation ranges in the appendix, accompanied by an explanation of their limitations.
Thank you for your continued feedback. We are delighted you are open to raising your scores to support the paper's publication. We hope our responses below (in this and Part 2) fully address your concerns. Please feel free to reach out if you have further questions.
# Your questions:
> [How are $\pi_{explore}$ and $\pi_{exploit}$ different, as they both load from $\theta$?]

$\theta$ is a container holding the parameters of both policies. The two policies, $\pi_{explore}$ and $\pi_{exploit}$, differ because, while they share some parameters, they each have unique parameters. In particular, our implementation shares all parameters in the same neural network except for the last layer, see lines 161-162 in Section 3.2.

We will update the pseudocode to clarify that $\theta$ represents a combination of parameters for both $\pi_{explore}$ and $\pi_{exploit}$. A comment will also be added to the load_policies function stating that each policy is constructed using its relevant subset of $\theta$.
> Why is l_explore added twice to the loss when r_exploit is greater than or equal to best_r?

To clarify, l_explore and l_exploit are distinct terms in the loss function, and each is added only once under specific conditions related to r_exploit and best_r. Is it possible you mistook l_explore as l_exploit at some point?

Quoting from the main rebuttal,
```python
if r_exploit > best_r:
	loss += l_explore
if r_exploit >= best_r:
	loss += l_exploit
```
To clarify, l_explore is added to the loss function if the exploit return (r_exploit) exceeds the best exploit return observed so far. This addition encourages the explore policy to produce episodes that enhance the exploit policy's performance.

Similarly, l_exploit is added if the exploit return meets or exceeds the best exploit episode return in the sequence. Here, the loss condition incentivizes the exploit policy to achieve high episode returns. Notably, both conditions check the return of the exploit policy, but add different terms to the overall loss. We will emphasise this detail in the pseudocode.

> [Is $k$ chosen automatically?]

After training, the optimal value of $k$ is determined by automatically evaluating each candidate $k$ within the interval $[1,n−1]$ where $n$ is the total number of episodes. To evaluate each of these values of $k$, the associated inference policy (explore $k$ times, then exploit $n-k$ times) is run on a batch of target-distribution environments, and the $k$ that achieves the highest mean cumulative reward is selected. This selected $k$ is then used for inference at test time.

We will rephrase the quoted text to clarify the automatic nature of the $k$ selection process, eliminating any implication of manual tuning. We will also make sure to clarify that this automated selection of the value of $k$ happens after training, and is computationally extremely inexpensive vs. training.

> [What is the role of the predictor?]

In our implementation, the agent uses its policy parameters $\theta$ to perform rollouts. These parameters are only periodically updated. A second set of parameters, the predictor's parameters $\phi$ learn an improved policy and are updated every batch of environments. Every $T$ learning updates, the agent policy is set equal to the predictor policy. This setup increases behavioral diversity, and enhances training stability. 

We will expand the pseudocode and methods section to clarify the distinct roles of the agent and predictor parameters.
Thank you for the clarifications provided. However, I still have some confusion regarding the paper:

**1. Training Process:**
- **Distinction between Policies:** I don't see the difference between $\pi_{explore}$ and $\pi_{exploit}$, as both seem to load from $\theta$. Could you please clarify this?
- **Loss Calculation:** Why is $l_{explore}$ added twice to the loss when $r\underline{\text{}}exploit$ is greater than or equal to $best\underline{\text{}}r$?
- **Role of Predictor:** What is the role of the predictor? Is it similar to the target policy?
- **Definition of "Informative":** Is "informative" defined as achieving a higher return? What if the agent receives negative rewards before reaching the objective but receives positive rewards when deviating from the objective? Does the concept of "informative" still work in this context?

**2. Inference:**
I am unclear on how $k$ can be automatically chosen. Figure 2B states that "after the two policies are trained in the previous phase, the optimal number of explorations is estimated by selecting the number of explorations $k$ that leads to the highest mean cumulative-episode reward on a large batch of new (unseen) environments sampled from the target distribution." This suggests to me that $k$ is selected through tuning. If I have misunderstood something, please correct me.

**3. Experimental Results:**
I am not an expert in meta-RL and may not be up-to-date with the latest research. If other reviewers do not consider it necessary to compare with newer methods, I am fine with that. However, Figures 3 and 4 could be improved by plotting the mean and standard deviation ranges. Multiple lines for one method make it somewhat difficult to follow the intended message. I can see that First-Explore outperforms other Meta-RL methods. If the authors can clarify my questions regarding the training and inference processes, I am willing to raise my score.