First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitation Trade-Offs
Authors: Ben Norman
Jeff Clune
Venue: NeurIPS 2024 poster (2024)
Keywords: Meta Reinforcement Learning
Meta-RL
Reinforcement Learning
RL
Primary area: reinforcement_learning

Abstract: Standard reinforcement learning (RL) agents never intelligently explore like a human (i.e. taking into account complex domain priors and adapting quickly based on previous exploration). Across episodes, RL agents struggle to perform even simple exploration strategies, for example systematic search that avoids exploring the same location multiple times. This poor exploration limits performance on challenging domains. Meta-RL is a potential solution, as unlike standard RL, meta-RL can *learn* to explore, and potentially learn highly complex strategies far beyond those of standard RL, strategies such as experimenting in early episodes to learn new skills, or conducting experiments to learn about the current environment.
Traditional meta-RL focuses on the problem of learning to optimally balance exploration and exploitation to maximize the *cumulative reward* of the episode sequence (e.g., aiming to maximize the total wins in a tournament -- while also improving as a player).
We identify a new challenge with state-of-the-art cumulative-reward meta-RL methods.
When optimal behavior requires exploration that sacrifices immediate reward to enable higher subsequent reward, existing state-of-the-art cumulative-reward meta-RL methods become stuck on the local optimum of failing to explore.
Our method, First-Explore, overcomes this limitation by learning two policies: one to solely explore, and one to solely exploit. When exploring requires forgoing early-episode reward, First-Explore significantly outperforms existing cumulative meta-RL methods. By identifying and solving the previously unrecognized problem of forgoing reward in early episodes, First-Explore represents a significant step towards developing meta-RL algorithms capable of human-like exploration on a broader range of domains.

First-Explore, then Exploit: Meta-Learning to Solve Hard Exploration-Exploitaiin Trade-Offs
Ben Norman 1.2
Jeff Clune jclune@gmail. com
btnorman&cs. ubc. ca
Department of Computer Science, University of British Columbia Vector Institute
Canada CIFAR Al Chair
Abstract
Standard reinforcement learning (RL) agents never intelligently explore like a human (i.e. taking into account complex domain priors and adapting quickly based on previous exploration). Across episodes, RL agents struggle to perform even simple exploration strategies, for example systematic search that avoids exploring the same location multiple times. This poor exploration limits performance on challenging domains. Meta-RL
potential solution, as unlike standard RL, meta-RL. RL can learn to explore, and potentially learn highly complex strategies far beyond those of standard RL, strategies such as experimenting early episodes to learn new skills, or conducting experiments to learn about the current environment. Traditional meta-RI. focuses on the problem of learning to optimally balance explo- ration and exploitation to maximize the cumulative reward of the episode sequence (c.g,, aiming to maximize the total wins in a tournament while also improving player). We identify meta-RL methods. When optimal behavior requires exploration that sacrifices immediate reward to enable higher subsequent reward, existing state-of-th off-the-ue art cumulative-reward meta-RL methods become stuck on the local optimum failing explore. Our method, First-Explore, overcomes this limitation by learning two policies: one to solely explore, and one to solely exploit. When exploring requires forgoing early-episode reward, First-Explore significantly outperforms existing cumulative meta-RI. methods. By identifying and solving the previously unrec- ognized problem of forgoing reward in early episodes, First-Explore represents a significant step towards developing meta-RL algorithms capable of human-like exploration on broader range of domains.
challenge with ffute-of-thh-art cumulative-reward
Introduction
Reinforcement learning (RL) [T] can perform challenging tasks, such as plasma control [2), molecule design 3 game playing (4, and robotic control (5). However, RL sample inefficient (taking thousands of episodes to learn tasks humans master in a few tries) (6. limiting its application. Meta- RL 12 circumvents this issue by enabling an agent to adapt new environments based solely on prior experience (i.e., remembering what occurred in previous episodes and using that to inform subsequent behavior). By replacing slow weight- based RL updates with memory-bat meta-RL adaption, human-like sample efficiency [12] can be achieved.
This paper focuses cumulative-reward meta-RL, which aims optimize performance across a sequence episodes, Ti,..., Ta (e.g., games in a tournament). The objective to maximize the total reward accumulated over all episodes (e.g., the number of games won), expressed as
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

G(T,,, where the episode return G(T,) is the total reward of episode T. To maximize this sum, the agent should optimally balance exploration and exploitation across the sequence, e.g., prioritizing exploration in early episodes so as to better exploit in later ones.
Cumulative reward meta-RL. has unrecognized failure mode, where state-of-the-art (SOTA) methods achieve low cumulative-reward regardless of how long they are trained. This dynamic occurs in domains with the following properties: A. Maximizing the expected total reward requires exploratory actions that forgo immediate reward, and B. The benefit of these exploratory actions only occurs when they are reliably followed by good exploitation (i.e., if exploitation is too inconsistent, then exploration results in lower total reward).
An example is bandit domain where the first arm always provides reward that is better than the average arm, but not the highest possible. To maximize cumulative reward over many pulls, the agent must explore the other arms, and then repeatedly exploit the best one. Property A holds, as this optimal policy forgoes immediate reward by not sampling the first arm and its above-average reward. Property B holds, as exploration (sampling arms other than the first) is only valuable when is followed by sufficiently consistent exploitation (reliably re-sampling the best arm).
The failure occurs as follows: 1. At the start of training, the agent, being randomly initialized, lacks the ability to reliably exploit learned information. 2. As a result, the domain properties A and B cause exploratory actions to lead to lower total reward than the other actions do. 3. This lower reward trains the agent to actively avoid exploration. 4. This avoidance then locks the agent into poor performance, as it cannot learn effective exploitation without exploration. This process occurs in the bandit example. Initially, the agent cannot exploit (e.g., when finds the best arm it does not reliably resample it). The associated negative expectation of exploration then trains the agent avoid exploration by only sampling the first arm, with its above-average, but sub-optimal, arm reward.
Current SOTA meta-RL algorithms such as RL g 91. VariBAD 77 and HyperX [10] attempt to train a single policy to maximize the cumulative reward of the whole episode sequence. This optimization causes step 3 of the above-described failure process. and thus these methods suffer from the issue of failing to properly learn (c.g., converging rapidly to a policy of not exploring), which we demonstrate distributions of simple environments (each trivially solved by standard-RL, can stymie these methods. Surprisingly, domains such bandits can be too hard for SOTA meta-RL.
multiple domains (Section 6). The issue is especially insidious because
We introduce a new approach, First-Explore (visualized in Figure which overcomes this problem associated with directly optimizing for cumulative reward. Rather than training single policy to maximize cumulative reward, First-Explore learns two policies: an exploit policy, and an explore policy. The exploit policy maximizes episode return, without attempting to explore. In contrast, the explore policy explores to best inform the exploit policy, without attempting to maximize its own episode return. Only after training are the two policies combined to achieve high cumulative reward.
Because the explore policy is trained solely inform the exploit policy, poor current exploitation no longer causes immediate rewards (property A) to actively discourage exploration. This change eliminates step 3 of the failure process, and enables First-Explore to perform well in domains where SOTA meta-RL. RL. methods fail. By identifying and solving this previously unrecognized issue, First- Explore represents a substantial contribution to meta-RL, RL, paving the way for human-like exploration on a broader range of domains.
2 Background
environments are formally defined as partially observable Markov decision processes (POMDPs, (.). Each POMDP E' is specified by tuple E' = (S.A, Po, R, Ω, O,7). where S is the state space, A the action space, p mapping from the current state and action to the next state, Pu distribution over starting states, R: S - R. stochastic reward function, S the space of environment observations, O: S + Ω stochastic function mapping from states to observations, and the discount factor. The environment starts (at / = 0) in a start state 8n according to Po. 80 ~ Po- Each subsequent time-step, the agent receives the current state's observations
RL Terminology:
A + S a probabilistie transition function
0(s;), takes an action a,, and the transition function p updates the environment state 8;+1 = p(s;, a.). An episode 7 of length h is then a sequence of time-steps starting from t 0 to t = h. The sum of an episode's 's -ddicouute rewards is called its
2

---

A: Policy Training Phase
B: k- k-Selection Phase
Final Policy
1:
T1
T2
T2
T3
T4
T1
T2
T3
C2
Cp
C2
C3
k 2:
22
T
Explore Episodes via explore
Exploit Episodes via TT exploit
Return of Subsequent Exploit
k 3:
Flow of Information (Context)
Episode used Feedback
T1
T2
T3 T4
for Exploration Training
Figure First-Explore aims to maximize the cumulative reward of a sequence of n episodes on target environment distribution. This optimization is achieved by first training two separate policies, and then combining them after training to maximize the total reward obtained. A. First, two separate policies are trained on the distribution of environments: one to explore (produce informative episodes), and one to exploit (maximize current episode return). During training, the explore policy Texplore provides all the context G; . for both policies. This flow of context is visualized by solid arrows >. The exploit policy Texploit takes a context of episodes, and produces a single episode of exploitation. The return of this exploit episode is then used train both policies, with the feedback to the explore policy visualized by the dotted green arrows ....>. > B. After the two policies trained, different combinations of them evaluated to find the combination that maximizes total reward. Each combination involves first exploring for & episodes, and then repeatedly exploiting for the remaining k episodes. C. The best combination is then used inference time: exploring for a fixed number of episodes on new environments, and then exploiting for the remaining episodes.
return G(T)= R(s(s az). Standard RL generally aims learn a stochastic policy π ΩQ A that maximizes the expected episode return E [G(()].
Unlike standard RI., meta-RI. trains an agent to: perform well in distribution 'D of environments (e.g., collection of mazes) and b) dynamically tailor itself to new environments (e-g. memorize new maze over successive episodes). The agent completes meta-rollouts, each sequence of episodes Ti.... Tn on new environment E ~ D, with each episode within a meta-rollout beginning from newly sampled start-state. The agent can remember information from earlier episodes in the same meta-rollout (e.g., using sequence model such as a transformer [3]), ), allowing to adapt its behavior based that information. By training (via weight updates) large batches of meta-rollouts, the agent then learns to leverage its memory mechanism, enabling environments sampled from D.
memory- based
adaptation and high performance
Meta-RI. methods can be split into two approaches that each solve a different problem, with specific algorithms designed and used for each approach [14]]] Cumulative-reward meta-RI. trains to maximize cumulative reward D G(T.). Examples include RL [8] Di VariBAD [7] and Hy- meta-RL aims to instead optimize solely for final episode reward G(T,). Examples include DREAM MTT) and MetaCURE [15. In this paper we compare and analyze
perX [10). Final-episode-rewan
First-Explore in cumulative-rewart setting. However, for the sake of completeness, discuss how First-Explore relates to manl-eepsode-reeee reward meta-RL RL methods Appendix[]
Related Work
Current cumulative-reward meta-RL methods train a single policy maximize the cumulative reward of the whole episode sequence. However, optimizing directly for cumulative reward can prevent effective learning even on simple domains (e.g., bandits), as Section describes and Section demonstrates.
14 use different terminology, calling cumulative reward meta-RL RL "zero-shot meta--L", and final -inal-episode- reward meta-RL "few-shot meta-RL.
3

---

RL 8 9 is one of the first mennulatvvereward meta-RL. RL methods. It uses an RNN to provide across- episode memory, and standard RL algorithms train the agent. RL has the advantage of simplicity, By training standard RI. algorithms with the capacity for across-episode memory. the agent may learn to optimally balance exploration and exploitation across successive episodes. However, training dynamics can hinder achieving optimal performance (e.g., due to the requirement of sacrificing immediate reward (Section | and Section 6 or because the reward signal is too sparse (TO).
Subsequently, various cumulative-reward meta-RL works have been produced. VariBAD 7 out- performs RL in certain domains. VariBAD achieves this improvement via splitting training into producing posterior belief of the current task (task inference). and a task-posterior conditioned policy. HyperX [10] improves exploration during policy training by adding an incentive to visit novel states. This exploration incentive is gradually decreased throughout training, starting high and ending at zero. The issue is that the cumulative-reward term can still actively discourage exploration, as described in Section [ Therefore, HyperX primarily tackles the issue of sparse exploration rewards, rather than overcoming the challenge of domains that actively disincentivize exploration.
As Section 6 shows, when good early exploration requires forgoing immediate rewards, all these methods can fail learn good behavior (e.g., converging immediately to a policy of never exploring). This failure arises not from reward sparsity (the problem HyperX tackles) or a lack of sophisticated posterior- welief-ccodiiioone actively train the agent to avoid exploration. As such, First- First-Explore can achieve substantial total reward in domains in which RL VariBAD, and HyperX all perform poorly.
optimization (VariBAD's focus), but because the reward dynamics
AdA 12 demonstrates that me scales to complex domains and that training meta-RI. on a curriculum of tailored learning challenges can produce agents capable of human-like adaption on complex held-out tasks. However, AdA may struggle to forgo reward, as the authors note that their agent always maximized current episode return (rather than the total reward of the episode sequence). This behavior suggests AdA's performance might depend on training and testing on environments that do not require such sacrificial exploration. Unfortunately, investigating AdA's dynamics is outside the scope of this paper.
First-Explore
First-Explore is general framework for meta-RL that be implemented in various ways. The framework trains two distinct policies, one policy to explore and one policy to exploit. Individually neither policy can achieve high cumulative reward, but after weight-update training, the policies are combined to produce an inference policy that does so. Figure[vvil visualizes this process. By not directly training a single policy maximize the total reward of the whole episode sequence, First-Explore avoids the failure mode of earlier approaches.
The explore policy Texplore performs successive episodes. During each episode. it has access to context containing all previous actions, observations, and rewards within the current exploration sequence. In contrast, the exploit policy Texploit is trained to take the context from the explore policy that has explored for a number of episodes between and n, and to then run one further episode of exploitation. During this episode, the exploit policy has access to the actions, observations, and rewards from the current exploitation episode, as well as from the preceding exploration episodes (Figure I..
The explore and exploit policies are incentivized differently. The exploit policy is incentivized to produce high-return episodes (based on the provided context of previous explorations). The explore policy is instead incentivized to produce episodes that cach, when added to the current context, result in subsequent high-return exploit-polic episodes. Training the exploit policy requires context from the explore policy, and training the explore policy requires the returns of subsequent exploits. This is efficiently achieved by interleaving the policies as depicted in Figure A where each rollout from the explore policy is followed by a rollout from the exploit policy.
After the two policies are trained, First-Explore searches for combination of them that maximizes the expected cumulative reward on newly sampled environments. Each candidate combination, first explores (via Tesplore) for set number of episodes k, and then exploits (via Tesploit) for the remaining n episodes. These combination policies are evaluated on independent environments sampled from the target distribution, with the combination that maximizes mean cumulative reward being

---

chosen (see Appendix for an example). This process of selecting & does not involve retraining the policies, and thus is comparatively computationally cheap. As such, &o should not thought as hyperparameter, unlike hyperparameters, the majority of the training compute expenditure is on policy weight updates that are performed before k chosen.
The resulting combination policy first exploring for k episodes and then exploiting for n k episodes is then used as the inference policy Tinference at test time, as shown in Equation
Texplore
if S k. where is the episode number.
Winference
(1)
Texploii:
otherwise
Experimental Setup
First- First-Explore is implemented with GPT-2-style causal transformer architecture [33 chosen for its strong temporal sequence modeling capabilities. To simplify the design, the explore and exploit policies share parameters, differing only in their final linear-layer head.
novel sequence- sequence-modeling approach improved training stability in initial experiments, and was thus used for all First-Explore results. The method is described here and in the pseudocode provided in Appendix B While this training method outperformed others in preliminary experiments, we believe the First-Explore meta-RL framework will work with alternate training approaches, such as PPO and other RL algorithms.
[6.. Q-learning (6)
5.1 Training Method
The exploit policy Texploit is trained to generate episodes that match or surpass the highest return achieved in prior episodes within the meta-rollout sequence. The explore policy eeppore is trained to produce episodes that are followed by the exploit policy achieving higher episode returns than those seen far. These training incentives implement the First-Explore framework: the exploit policy maximizes immediate episode returns, while the explore policy generates episodes that increase subsequent exploit-episee episode returns.
Training involves periodically updating the rollout policies (Texplore: Tesploit) with successor versions. These successor policies (Dexplore: @esploit) trained model the actions taken in the good rollout- policy episodes, defined as 1. good exploit episodes meet or surpass previous exploit returns in the meta-rollout sequence, and 2. good explore episodes
followed by the exploit policy achieving higher episode returns than those seen so far. We term these good exploit episodes 'maximal,' and the good explore episodes "informative"). Since the first exploit episode in each meta-rollout has no previous episodes for comparison, a baseline reward b initializes the list of prior returns. This value is set as a domain-specific hyperparameter (but could easily be set automatically via heuristies).
At the start of training, both rollout policies are initialized with random weights. They are then copied to produce the initial successor policies.
Training structured into iterated epochs. Each epoch, the current exploit and explore policies (Texplore, Texploit) produce batched meta-rollouts. In each batch, the exploit episodes Texploit Texploit that are maximal and the explore episodes Texplore ~ Texplore that are informative are recorded. For episodes, cross-entropy loss is calculated between the successor policies (dexplore Φesploit) and the action distributions in the associated maximal or informative episodes. The successor policy weights are then updated, using gradient descent on the loss. In this way the successor policies learn to emulate the conditioned rollout policies (Equation 2). Finally, every T epochs, the rollout policies Texplore Texploit updated to match the successor policies Pesplore Pesploit ensuring continuous improvement, This hyperparameter manages the trade-off between preserving behavioral diversity and accelerating learning iteration.
these criteri-satisffyii
Desplore Wesplore
'informative episodes'
Pexploit 2 Tepploit
'maximal episodes'
(2)
*As in all RL, improvement is a noisy process, (e.g., an episode labeled 'informative" might not be genuinely informative, due random noise in the exploit reward). However, in expectation the labeling is correct.
5

---

During training, actions are sampled from their predicted probability distributions a ~ Texploit Or Tepporre In contrast, during inference, actions are selected greedily.
Results
Table 1: Mean cumulative reward + standard deviation of First-Explore compared against control algorithms in hard-to-exppe terd-loe domains, with random action (picking actions uniformly random at each timestep) added for additional reference. In each domain, First-Explore significantly outperforms
meta-RL controls. The bandit domain compares to two non- mon--met--Ree RL. baselines, marked t.
Bandits with One Fixed Arm
Dark Treasure Rooms
Ray Maze
First-Explore
127.7 + 2.0
First-Explore
2.0 + 0.3
First-Explore
0.47 = 0.01
RL
56.1 + 12.2
RL
0.2 0.1
RL
+ 0
UCB-1*
116.8 = 0.5
VariBAD
0.2 0.1
VariBAD
0 + 0
TSt
123.3 + 2.5
HyperX
-0.2 0.2
HyperX
0.07 0.07
Random Action
5.2 + 0.2
Random Action
5..5 + 0.1
Random Action
- -0.23 + 0.1
When obtaining maximum cumulative reward requires forgoing early-episode reward, cumulative- reward meta-RL algorithms fail to learn optimal behavior regardless of how long they are trained (Appendix F), as they become stuck on local optimum of not exploring well. Even simple environments such as bandits flummox them. Three varied domains empirically demonstrate this dynamic. On all three a) the meta-RL controls perform poorly, and b) First-Explore significantly outperforms the controls. Further, two the domains have modified versions that do not require forgoing immediate rewards, and this change causes significant control policy improvemen f
When forgoing immediate reward is required, First-Explore achieves 2x more total reward than the meta-RL controls in the first domain, 10x in the second, and 6x in the last [ These significant reward differentials reflect substantive behavioral differences, e.g., First-Explore exploring well (at the cost of reward) and then exploiting vs. the meta-RL methods converging to a policy of minimal exploration. First-Exxy Eess--Exllore outperforming the other methods is also statistically significant, with p-values less than 10 for each comparison, as calculated by two-tailed Mann-Whne Whitney U tests (MWU).
The first domain is multi-arm bandit problem designed to require forgoing immediate reward, where pulling the first bandit arm is immediately rewarding while also having no exploratory value. Pulling the first-arm yields guaranteed reward of μл, unlike the other arms, which - averaging across environments within the domain - yield expected reward O. We consider two cases: 111 = 0.5 (the deceptive case) and # = 0 (the non-deceptive case).
Domain 1: Bandits with One Fixed Arm:
For both values, the greatest cumulative reward can be reliably achieved by first exploring to find the highest reward arm (with reward greater than (1) and then exploiting by repeatedly sampling it. However, 141 = 0.5 creates the deceptive challenge described in Section [[] where before reliahle exploitation has been learnt, exploration (i.e., not sampling arm 1 and obtaining its guaranteed reward) leads to a lower total reward. See Appendix [.T for further details on this domain.
Bandit Results (Figure 2 First-Explore is evaluated against 3 baselines, with oracle performance (grey) and random action selection (black) plotted for additional reference.
Deceptive Case: The worst-pe control, RL22
(fuchsia) is stymied by the deceptive domain dynamic. Four of five RL² training runs (overlapping bold fuchsia) learn to only sample the guaranteed reward, achieving a constant 0.5 reward each pull (Figure B1), and exactly 50 reward after 100 pulls (Figure AI). The remaining run (faint fuchsia) does better but still poorly. The best returns are achieved by methods that balance exploration (finding the most rewarding arm), and exploitation (pulling the best arm found so far). First-Exae Eersl-E (green) and the non-meta-Re RL bandit
'Surprisingly, First-Explore outperforms the controls even on the dec-deceetiee domains, < 10-5 (MWU). While promising grounds for future work, this paper only claims that Etrst-Explore outperforms other methods in domains that require forgoing immediate reward.

---

Rrst-Explore
RL
Oracle Best Arm
Thompson Sampling
Random Action-Selection
A1
= 0.5 the deceptive case)
A2
= Ithe nor-deceppive case)
1.50
Cumuiatvve rovars
1.0
1.5
Sverge
0.5
0.0
of puls
A puis
Figure 2: Mean performance (averaged across sampled bandits) of algorithms for deceptive (left) and non-deceptive (right) versions of the bandit domain. Each method trained independent times, and each such run is plotted individually, so as to faithfully represent the variance between runs (e.g.. that multiple of the bandit-domain RL training runs achieve exactly the provides alternative plots with mean reward standard deviation. The top figures plot the cumulative reward against the number of arm pulls, while the bottom figures illustrate the reward dynamies by plotting the individual pull rewards against the same. When the domain is deceptive, the cumulative- reward meta-RL method, RL (fuchsia), performs extremely poorly, despite the deceptive domain giving strictly higher rewards than the non-deceptive version. In contrast, First- Explore (green) impressively outperforms UCB (purple) and Thompson Sampling (orange) despite them being specialized bandit algorithms, in both the deceptive and non-deceptive settings, with 10-5.
reward). Appendix C
controls, Thompson Sampling 17 (orange) and UCB 8 (purple), both exhibit this behavior. Interestingly, First-Explore achieves the balance differently from the bandit algorithms, with the average pull rewards transitioning from close to to close to optimal sharply around 10 pulls (Figure II).). Impressively, First-Explore achieves greater reward than even the specialized bandit algorithms despite being applicable to more general domains (p < 10-3, as calculated by MWU). HyperX and VariBAD are not included as baselines because their performance was overly poor on non-deceptive versions of this domain Non-Deceptive Case: When the domain is not deceptive (i.c., when exploration does not require forgoing immediate reward), RL² achieves high total-reward (Figure 2 A2), and samples arms similarly to the bandit algorithms (Figure 2aarre2BB B2). All other algorithms perform well regardless of deception. This dynamic validates that the need to forgo immediate reward is what stymies cumulative-reward meta-RL performance.
Illustrating that SOTA cumulative-reward meta-RL can fail on such simple domains (i.e., where standard-RL. RL can easily solve each individual environment) is key contribution of this paper.
Domain 2: Dark Treasure Rooms:
The second domain grid world environment, where the agent cannot see its surroundings. In each environment, there are multiple randomly positioned reward locations. When each location encountered, the agent consumes it for that episode, receiving a reward ~ U[p, 2]. The agent receives only its current coordinate as an observation, and to explore the agent must move blindly into unobserved grid locations that has not visited before.
We consider two values gt = -4 (the deceptive case) and p 0 (the non-deceptive case). For both values, high total reward is achieved by first exploring (visiting unseen grid locations to find positive reward locations) and then consistenily exploiting (revisiting discovered positive rewards locations each episode). However, when p -4 this process is challenging, only 1 of the reward locations are positive, and the expected reward positive location is only 1. These
DDepptte hyperparameter tuning. good performance on either distribution (deceptive and non-deceptive) could not be achieved. Though bandits seem simple, the distribution environment arm means is large and continuous. This may cause problems for VariBAD and HyperX, which were designed for and evaluated on small discrete task spaces, e.g. goal being of 25 locations.

---

expectations create the challenge described in Section 1 where one must consistently exploit (by revisiting discovered positive reward objects more than three times) for any initial exploration be worthwhile. See Appendix C.2 for explanation of these properties, along with further domain details. In contrast, when p O, visiting unseen locations provides positive expected reward causing immediate rewards to incentivize (rather than deter) initial exploration.
First Explxx
RL2
HyperX
VariBAD
Random Action Selion-Selection
episodes Treasure Room for p- (the decoptive case)
cpisodes Treasure Room p- D non deceptive case)
apsoda number
epieed number
arDON0 epinnte (RO0r1
METOSS spands
30.0
7.5
2.5
80
40
1004 Linte
40
20
2000
stop
Figure 3: Mean performance (averaged across sampled treasure rooms) of algorithms for deceptive (left) and non-deceptive (right) versions of the Dark Treasure Room domain. Each method trained 5 independent times, and each such is plotted individually. The top figures plot the cumulative reward obtained against step and episode number, while the bottom figures provide a proxy for exploration by plotting the number of times agents deceptive, the cumulative-reward meta-RL methods, RL² (fuchsia), HyperX (brown), and VariBAD (purple) achieve low total-reward, as the policies learnt minimize exploration. In contrast, First- Explore (green) performs well on both the deceptive and non-deceptive domains.
against the same. When the domain is
First-Explore evaluated against three meta-RI. controls, with random action selection (grey) plotted for additional reference. Deceptive Case: The meta-RL. algorithms RL (fuchsia) and VariBAD (pale lilac) all achieve close to zero reward (Figure A I), and rarely move (Figure BBB.). HyperX explores more but minimally, reflected by the modest negative first episode return However, the method fails to then exploit, and thus under-performs the other methods, despite this exploration.
Dark Treasure Room Results (Figure 3):
By first exploring and then exploiting, First-Explore (green) achieves 10x more total reward in the ten-episode setting than the meta-RI. controls. Impressively, despite the high cost of exploration (with single episode of exploration taking more than three subsequent exploitations to yield positive expected reward), First-Explore is sufficiently skilled that its policy of exploring twice and then exploiting achieves more total-reward than a hand-coded policy of optimally exploring in the first episode, and then optimally exploiting conditioned Deceptive Case: the meta-RL controls achieve reasonable total-reward (Figure A2) and move consistently (Figure B-B2) demonstrating that it is the difficulty of forgoing immediate rewards that challenges SOTA methods. However, First-Explore still outperforms them, with p < 10 (MWU).
this initial exploratior (2.0 > 1.78). Non-
The final domain is significantly more complex than the previous ones, and demonstrates that First-Explore can scale beyond bandit and grid-world problems. The domain is composed of randomly generated mazes with 3 reward locations. The agent must parse large number of lidar observations (see Figure 4 to navigate around walls, identify reward locations, and move onto them. Each episode, the first reward location visited gives reward of either -1 or +1, each reward location having an independent 30% chance of positive reward. The agent has three discrete actions: rotate right, rotate left, and move forward. One complexity is that (unlike the
Domain 3: Ray Maze:
'The amount of exploration in the first episode is equal to the number of unique locations visited which is exactly proportional the average negative first-episode return. such, while HyperX moves more than VariBAD and RI.". it mostly moves into locations previously visited.
"See Appendix C.2 for mathematical derivation.
8

---

grid-world domain). actions do not commute (e.g., rotating left and then moving forward is different from moving forward and then rotating left). The agent can see the reward locations, but cannot tell by looking whether they give positive or negative reward. Optimal behavior requires first exploring (navigating to visit un-visited reward locations, and obtaining their negative expected reward), and then reliably exploiting (re-visiting reward location if it gave positive reward). The need to sacrifice immediate reward (by visiting the negative reward locations) challenges the SOTA cumulative-reward baselines.
Exrs-Explore
RL2
Hyperx
VariBAD
Random Action
cumulative reward
episode number
1
2
3
4
0.5
0.0
-0.5
20
40 step #
60
80
Figure 5: Mean performance averaged across 1000 Ray Mazes for five of each treatment. First- Explore strongly outperforms the SOTA meta-RI. baselines on this complex environment, achieving mean 0.47 reward, only slightly worse than the expected total-reward of behaving optimally, 0.64.
Ray Maze Results (Figure 5 On this more complex domain, First-Explore similarly achieves significantly more reward than the meta-RL controls, with all con- trols failing to perform well. Two of the three meta-RL controls, RL² (fuchsia) and VariBAD (purple), learn never move (and instead spin in-place). This behav- ior avoids encountering any reward location, and thus results in exactly 0 total reward in every evaluation of all five independent training runs. For the final meta-RL control, four of five HyperX (brown) runs move little (achieving near zero reward), with the remaining run achieving only modest cumulative reward.
15 Rays
201 Rays
Figure 4: Left; Raw agent observations from sampled ray maze converted to an image. The agent receives the wall dis- tances and the wall types. Portraying this numerical data as an image, goal locations are green, and the wall orientations are distinguished (east-west teal, and north- south navy). To aid the eye, the floor has been coloured in dark purple, and the sky yellow. Although the goal is visible, could be treasure (positive reward) or trap (negative reward). Right: The image pro- duced with direct ray casting (large num- ber of processed lidar measurements) rather than the 15 the agent receives.
First-Explore (green) significantly outperforms the controls, achieving 0.47 mean reward (Table by ex- ploring to find a reward location, remembering it and reliably exploiting by navigating to that location if it gave positive reward. Figure 5 reflects this process, illustrating that First-Explore's cumulative reward goes down substantially before then going up. For com- parison, the optimum possible behavior of exploring and exploiting perfectly achieves at most 0.64 average reward"
Limitations and Future Work
As presented, First-Explore does not actively explore to enable future exploration, because the explore policy Texplore only trains to increase the expected reward of the subsequent exploit policy episode. Unfortunately, sequence of optimal myopic explorations is not necessarily optimal exploration sequence (Appendix G A potential solution is to reward exploration episodes based on a weighted sum of rewards from all subsequent exploitation episodes, analogous to summing discounted future rewards standard RL.
Another limitation is that First-Explore could be unsafe in certain environments. The risk is that First-Explore ' explore policy does not avoid negative rewards, and so penalizing unsafe action has minimal effect on the policy. For example, a chef robot attempting new recipe in a physical home
See Appendix C.3 for upper bound optimal performance.

---

might explore mistakes that endanger humans or damage the kitchen (despite negative rewards telling it not to). This concern only applies subset of environments, as many environments are safe, e.g., simulated ones. That said, addressing this concern is a valuable direction of future work. One potential solution to add a safety penalty to the explore policy, i.e., train the explore policy to maximize subsequent exploit episode-retums while also avoiding safety risks. This proposed version of First-Explore could actually produce in-context adaption that is safer than standard RL training, as the meta-RL policies would have learnt strong prior of avoiding potentially dangerous actions.
This paper makes no claims regarding First-Explore's meta- veeaantainnng efficiency, and future work may substantially improve meta-training time. Instead, this paper a) identifies a challenge that SOTA meta-RL algorithms fail on regardless of training time (the need to forgo immediate rewards), and b) provides a framework, First-Explore, that can solve these challenging domains.
A final problem is the challenge of long sequence modelling, with certain environments requiring huge context and high compute (e.g., can one have a large enough context, and enough compute to
allow First-Explore to generalize and act as replacement for standard RL?). AdA [12] suggests such
a project might be possible. Progress on efficient long-contexI sequence modelling [18 19] research
on RL transformer applications 20 211 and Moore's ' Law all make this possibility more feasible.
Additionally. given that First-Explore learns a dedicated explore policy and a dedicated exploit policy, and both have been shown to work well (e.g., Figure 2). the method may be applicable to fial--ppxode-rewarrt meta-RL settings (Appendix[]] II.. Further, First- Soirst-Explore always switches from exploration exploitation after fixed number of episodes. Future work could replace this fixed number with a learnt classifier that determines when to switch from exploration to exploitation, e.g., repeatedly exploring until sufficient information is obtained. However, despite these applications being highly exciting directions of future work, a proper investigation of either would require its own paper, including many different specialized controls and environments pertinent to the setting.
8 Discussion
Given that First-Explore uses RL algorithms to train the meta-RL policy, how might it solve hard- exploration problems that standard RL cannot, e.g. design a rocket for the first time? We believe that given a suitably advanced curriculum, and sufficient compute, First- trsst-e could learn powerful exploration heuristics (i.e., develop intrinsic motivations such as an analogue of curiosity) and that these heuristics would enable sample-efficien performance on hard pparse-reward problems. On a curriculum, initially First-Explor exploration. Once it has learnt rudimentary exploitation, the agent can learn rudimentary exploration. Then it would learn better exploitation and exploration, and so on, each time relying on there being 'goldilocks zone' tasks (22 that are not too hard and not too easy.
would explore randomly, and learn to exploit based on that random
Further. while curricula can aid all of meta-RL, e.g., RL2 and AdA, First-Explore can have a significant training advantage on certain problems (c.g., in the ten-episode Dark Treasure-Rowe Room, First- Explore achieves positive cumulative reward while the standard cumulative-rewan me methods catastrophically fail). This advantage could potentially allow far greater compute efficiency, and allow training on otherwise infeasible curricula.
9 Conclusion
We have theorized and demonstrated that SOTA cumularive-reward meta-RI. fail to train when exploration requires forgoing immediate reward. Even simple problems such as bandits (Figure I can stymie these methods. To overcome this challenge, we introduce novel meta-RL framework. First-Explore, that learns two separate interleaved policies: one to first explore, another to then exploit. By combining the policies at inference time, First-Explore is able to explore effectively and achieve high cumulative reward on problems that hamstring SOTA methods.
Meta-RL- shows the promise of finally fixing the main problem in RL - that it is extremely sample inefficient even producing human-level sample efficiency
(2). However, the promise of this approach is limited, as we have shown, on a large set of important problems. We can only take advantage of this approach we can harness the benefits of meta-RI. even on such problems, and First- Eirsl-e enables us to do so, thus offering an important and exciting opportunity for the field.
10

---

Acknowledgments and Disclosure of Funding
Resources used in preparing this research the Government of Canada through CIFAR, and companies sponsoring the Vector institute www vectorinstitute ai/ ai/#partners This work further supported by an NSERC Discovery Grant, and generous donation from Rafael Cosman.
provided, in part, by the Province of Ontario,
Thanks to Michiel van de Panne, Mark Schmidt, Ken Stanley and Shimon Whiteson for discussions, and to Yuni Fuchioka, Ryan Fayyazi and Nick Ioannidis for feedback on the writing. We also thank Aaron Dharn,, Cong Lu, Shengran Hu, and Jenny Zhang (sorted alphabetically) in our lab the University of British Columbia for extensive discussions and feedback.
References
[1] Richard S. Sutton, Francis Bach, and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press Ltd, 2018.
[2] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, and et al. Magnetic control of tokamak plasmas through deep reinforcement learning, Feb 2022. URL
https://wwv /ttp:://www.nature... nature.
com/ wtt.aaattiileeeeeeeeeeeeeeeeeeet 9416666-22-00000--93
[3] Zhenpeng Zhou, Steven Kearnes, Li Li, Richard N. Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning, Jul 2019. URL https: /// http:/wwwnaaurre com/ articcee//41199-01-4477777
[4] David Silver, Thomas Hubert, Julian Schrittieseer, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 2018
(22264419:1140-111
[5] OpenAL, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, arXiv preprint, 2019.
[6] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 13 2013.
[7] Luisa Zintgraf. Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning. In International Conference on Learning Representation (ICLR), 2020.
[8] Yan Duan, John Schulman, Xi Chen, Peter I. Bartlett, Ilya Sutskever, and Pieter Abbeel. ri?: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 161.07777 2016.
[9] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: /6/ 1.05663, 2016.
[10] Luisa Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning (ICML), 2021.
[11] Evan 7. Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinnorreement learning without sacrifices. In International conference machine learning, pages 6925-6935, PMLR, 2021.
[12] Adaptive Agent Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopehand, Nathalie Bradley-Schmiee Michael Chang, Natalie Clay, Adrian Collister, et Human-timescale adaptation an open-ended task space, arXiv preprint 200102230,.07608, 2023.
11

---

[13] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, 1lya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
[14] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson, survey of meta-reinforcemeet learning. arXiv preprint WxXiv:331108002. 2023.
[15] Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang. Metacure: Meta reinforcement learning with impowerment-driveer exploration. In International Conference on Machine Learning. pages 12600-12610. PMLR, 2021.
[16] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXiv:/77.00344/ 2017.
[17] William R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3/4):255-94,, 1933. ISSN 00063444. URL
jstor wrtp http///aaeee233225555m
[18] Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena: A benchmark for efficient transformers. arXiv preprint arXiv:201 1.04006, 2020.
[19] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00390 2021.
[20] Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse. Steven Hansen, Angelos Filos, Ethan Brooks, al. In-context reinforcement learning with algorithm distillation. arXiv preprint arXiv:2110.12215, 2022.
[21] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 14:50044-15097, 2021.
[22] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley, Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: 01755,, 2019.
[23] Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. arXiv preprint arXiv: /907 1740, 2019.
[24] Haotian Fu, Hongyao Tang. Jianye Hao, Chen Chen, Xidong Feng, Dong Li, and Wulong Liu. Towards effective context for meta-reinforcement learning: an approach based on contrastive learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7457-7465, 2021.
[25] Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via meta-reinforcemmen learning. arXiv preprint arXiv: 1803 01 18, 2018
[26] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Mill and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
http://github.. wtt. wowgg//oe///aa...
[27] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush, Huggingface's 'ss transformers: State-of-te State-of-the-ar the-arterttt art natural language processing, 2020.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
12

---

Appendix
Table of Contents
Replicability
14
Training Pseudocode
14
Detailed Domains:
16
Bandits with One Fixed Arm
16
C.2
Dark Prasure-Room:
17
C.2.1
The p Domain Dynamics
17
C.2.2
One Optimal Exploit episode, and then Subsequent Optimal Myopic Ex- ploitation
19
Ray Maze
20
C.3.1
Optimal Rotal-Reward
20
D Tabulated Results:
22
Compute Usage
23
Poor Performance Regardless of Train Time:
23
Myopic Exploration
25
H K-Selection Phase
25
Final-Episode-Rewee Reward Meta-RL
26
Evaluation Details
26
Training Details
27
K. I Controls:
27
2 First-Explore:
27
Dark Treasure-Room Visualizations
28
13

---

A Replicability
To ensure full replicability, we are releasing the code used train First-Explood and the controls, along with the environments trained on. We are also releasing the weights of trained model for each domain. Each model contains both the explore and exploit policies as separate heads on the shared trunk. The code is available at https:://g github....... com combbtnorman/ First- First-Explore
B Training Pseudocode
def rollout TT v Cr, ca):
### """pfrorrm a single episode inputs: the environment (env), the agent policy n, the successor policy 1h, and the current policies' contents Cr. Gp
### returns the episode return, temp_ loss, and updated contents""
temp_loss
0,
n. b. temp_ loss is only used if the episode meets condition see (+) in the conditional_acti action loss function s env reverest() # state $ for in renge episode__ength):
(env
# calculate action probabilities Pr, Pe for both polictes # and update content Pr. Cr π m...ctioon actbioo.probbbiitiieeea,, cx)
Pps Co ww..aiionn.rbabiiitttte
Gp)
sample_action(p..
temp_ _mop_ cross_ enteroyrrpaa, Ps pe) hadamard product
# Pa ensures action diversity by weighting against likely actions
8 env.ste a)
+= s.reward return I, temp_loss, Cz cy
def conditionalacctinn__
0, D, b)
### """calculates First-Eaplord loss for both exploring and emploiting on domain using the agent and successor parameters 9, ### and baseline reward 6"n"
env = sample_env(D)
т T_explore, =_exploit load_policiess!) / v_explore, v_exploit Ca Gp set(), set() the agent and successor contents loss, best_r = 0, for in range (D. episode_num)
r_explore, 1_explore, cx, co rollout (env, æ r_explore, th exexplore, Cr, Gp) r_ r_exploit, 1_exploit, # "emploit content not kept
rollout T_exploit, v __eplooit, Go ox)
# (+) accumulate loss if:
if r_exploit >= best_r: # exploit episode is 'énformative'
loss += 1_epploit
if r_exploit best_r: explore episode is 'mazimal'
loss += 1_xxplor
best_r r_exploit
return loss
Algorithm I: Example First-Explore Cross-Entropy Loss.
14

---

def train(epoch_nuug _um, num, T, D, b):
"""mmpp le First-Eaplore training (ignoring batchsize)
runs meta-rollouts, accumulating loss
this loss is then outo-dafferemttaeed""" T_counter
0 init_params() ()
for i in
# 0 is the agent behavior parameters
# yp is the successor parameter (learning an improved agent policy)
Δup 3Φ aconditionl__atiin__Joo (vp, 0, D, b))
8 step_size Avp
# Update 0 every T epochs
T_ counter
if T_coutter == T:
= 4
T_counter 0
return 0
Algorithm 2: Example of Training First-Explore using the Cross-Entropp loss and Auto- Differentiation.
15

---

Detailed Domains:
C.1 Bandits with One Fixed Arm
Domain Description: Each bandit has ten arms, and in the environment the agent acts by selecting an arm pull E [1 10). The first arm always yields the reward μl1 while the other arms yield their environment specific arm mean 2a ~ NO 1) plus a normally distributed noise term No0, 2). added make the environments more challenging. The arm mean is fixed once the environment sampled, but the noise term is resampled each arm pull. The objective is to maximize the expected reward obtained on newly sampled bandit over 100 pulls.
Additional Treatments: Two specialized bandit algorithms were evaluated on this domain. The bandit algorithms: UCB-1 estimates upper confidence bound and selects the that maximizes it, see Appendix for details. Thompson Sampling (TS) [17] samples arm means from the posterior distribution, and chooses the arm with the best sampled mean.
See Figure for version of Figure that plots mean + standard deviation instead of each individual run.
Firt-Expplre
UCB
Thompson Sampling
Random Action-Selection
A1
A/ m 0.5 (the deceptive case)
X2
(the non-deceptive case)
comdetive pull reseord
150
100
Svemg5 put POND
Figure 6: Alternative -ithtts--wwthOO---e-eeee Plots with Mean + Standard Deviation.
16

---

C.2 Dark Roomsure-Rooms
Dark Ceasure-RRomm (inspired by the Darkroom in (20] are grids full of treasures and traps. The agent starts in the middle of the grid, navigates (up, left, down, or right) to find treasure, and cannot see its surroundings. The agent can take five actions: four for cardinal movement, and one to not move. Only its current (x, v) coordinates are observed. Each environment has 8 objects (treasures or traps), and when the agent encounters a treasure or trap consumes/activates it, and receives an associated reward (positive or negative). The treasure and trap values U, are uniformly distributed in the range p to 2, 02 ~ U [p. 2/. with p being the maximum trap penalty. The locations of the objects are sampled uniformly, with overlapping objects having their rewards/penalties summed. Each episode has 9 steps, and the objective to maximize the expected cumulative episode returns newly sampled Dark Sreasure-Roon over multiple episodes.
Problem Description:
See Figure for version of Figure that plots mean + standard deviation instead of each individual training run.
First Explore
RL2
Hyperx
VariBAD
Random Action Selection
10 episodes of Treasure Room for p= (the deceptive case) upsode number
episodes Treasure Room for p= 0 (the non-deceptive case) episdde number
arTO apleota cumunitie
7.5
2.5
0.0
1224 limes mosed
60
120.4 moveed
40
40-
20
stop
Figure 7: Alternative
Plots with Mean +'2 Standard Deviation.
C.2.1 The p Domain Dynamics
How diffieult the are as follows:
room domain is strongly depends on the value of p. The dynamics
The fraction of positive rewards equals 2-p"
The average value positive reward is always [UU 2)) = (assuming p < 0).
The average value negative reward is [(U 0)) =
The expected reward of visiting random reward location (treasure or trap) EUU 2)) 2t..
Using these, we can calculate the expected reward of visiting a random reward location and then returning to it n times if and only if the location gave positive reward. This expectation is equal to the expected reward of the first visit plus n times the chance of the visited reward location giving positive reward multiplied by the expected positive reward (Equation 3).
E(U 2)) + " B(U = p+2 + n1
2
2
2
can then calculate the number of revisits required for exploration (visiting random reward location for the first time) to be worthwhile (i.e., lead to positive expected reward). Equation illustrates this calculation, for values of p < -2. This number of revisits corresponds how reliable the exploitation policy must be for exploration to yield higher cumulative reward. For example, when
17

---

Fist--
Hyperx
VariBAD
2/4 opisode number
action wdhande cumusbyv mamuaayyeeewa raward
10
WcrONN aphcnte cumulbttcc row0rd
-2
SC022 SETOOS
18
Figure 8: Behavior of First-Explore and the meta-RL controls on the 10 Horizon Dark Treasure Room for various values of p.
P 4 on the ten horizon setting, the agent needs exploit correctly than 3 of the time, (after exploring) it has at most nine episodes to exploit, and must exploit correctly more than three times. If exploitation reliability is lower than this value, then exploration will be actively discouraged.
2+P
4 p +
4n
8² 4
When p -. [-4 3. Thus, p = -4 corresponds to exploration (visiting a new reward 4 location) requiring more than three potential revisits to be worthwhile (have positive exploration). As (Section [ describes, this dynamic challenges cumulative-rewart meta-RL methods, as exploration thus requires reliable exploitation be worthwhile.
This value of p = -4 was chosen as it a) requires a reasonably high number of revisits, and b) an integer number that corresponds exactly to exploration being worthwhile only when the exploitation policy on average performs more than an integer number of revisits.
In contrast, as Section 6 describes, when 0, exploration is directly incentivized (and no revisits are required for exploration to increase the expected total reward).
For completeness, we also calculate the dynamics that occur when p 22 and when -2 233 which corresponds to requiring more than and more than revisits respectively (Figure and Table 2 Even when the domain is made significantly less challenging (i.e., for more positive values of p), First-Explore signficantly outperforms the controls.
"A found by solving Equation 3 22 corresponds to exploration (visiting random treasure location) requiring than revisits yield increased total reward.
18

---

Table 2: Mean cumulative reward + standard deviation on 10 Episode Dark Treasure Rooms, for different values of p = -2VI corresponds to exploration (visiting a random treasure location) requiring more than æ revisits. Even when the domain is made significantly less challenging (i.e., for more-positive values of p). First-Explore signficantly outperforms the controls.
p 22
p
22
p 22
First-Explore
4.4 0.3
3.1 0.3
2.0 0.3
RL
0.4 0.1
0.2 0.1
0.2 + 0.1
VariBAD
0.7 + 0.3
0.3 + 0.1
0.2 + 0.1
HyperX
0.8 + 0.5
-0.1 = 0.1
-0.2 + 0.2
C.2.2 One Optimal Exploit episode, and then Subsequent Optimal Myopic Exploitation
In this domain, perfect exploration in the first episode corresponds to visiting 9 unique coordinates, each coordinate has equal chance of being reward location (a treasure trap). This process discovers 9 * ST coordinates, and the agent visits 9 of the coordinates.
& reward locations on average, as there are 8 reward locations spread over 81
Assuming the agent then optimally myopically exploits (i.e., only maximizing current-episode reward and not attempting explore more), the total reward is at most times the expression in Equation (the expected number of reward locations multiplied by the expected value perfectly exploiting each individually).
Equation 7 calculates this expectation for / = -4 and = (the values of the deceptive Dark Treasure Room domain in Section(6). Note, n is the number of subsequent exploitations, after visiting a new reward location, and so the n calculation corresponds the ten episode setting.
8
/+2
2
2
E(explore optimally then optimally myopically exploit)
+
(7)
9
2
2
P
91
(-1
1
+
3
(8)
1.77 ~ 1.78
(9)
This calculation an upper bound, it assumes the agent can effectively teleport to discovered positive reward locations. In reality, the agent may not be able to revisit some discovered positive- reward locations without moving into discovered negative-reward locations.
19

---

C.3
Ray Maze
Problem Description: Ray Maze challenges the algorithms with a more complex domain. The agent must navigate randomly generated maze of impassible walls (the maze layout is different for each sampled Ray Maze) to find one of three goal locations. Each goal location either a trap or a reward, with 0.3 probability of the goal being treasure, and 0.7 probability of it being trap. Treasures give reward 1, while traps give penalty -1. In this domain, the agent can only receive one goal reward per episode, triggering the first prevents others activating. To perceive the maze, the agent receives 15 lidar observations (see Figure 4.. Each lidar observation reports the distance to the nearest wall along an angle (relative to the agent's s orientation). It further tells the agent whether the lidar ray hit a goal location, as well as the wall orientation (east-west or north-south). However, the lidar measurements do show whether goal is trap or treasure. The agent has 3 actions, turn left, go forward, and turn right, with rotation turning the agent's 's field of view.
Ray Maze is a challenging domain for several reasons. It has nigh-dimensional observation space (15 separate lidar measurements), complex action dynamics (with actions not commuting, c.g. turn left then move forward is different from move forward then turn left) and randomly generated maze that interacts with both movement and observation. Furthermore, similarly to earlier environments, the agent must learn from experience, and risk the traps in early episodes, consistently find treasure in later ones. Because of how frequent traps are, the agent can only obtain positive cumulative reward by a) searching for treasures in early episodes (at the cost of expected reward, as the average value of goal is negative) and b) reliably exploiting in later episodes (navigate to identified treasures, while avoiding potential traps).
to exploit and
D for version of Figure that plots mean + standard deviation instead of each individual training run.
See Figure
First-Explor
RL2
HyperX
VariBAD
Random Actions
cumulative reward
episode number
1
2
3
0.5
0.0
20
40 step #
60
80
Figure 9: Alternative Ray-Maze Plot with Mean + Standard Deviation.
C.3.1 Optimal Total-Reward
Despite being a complicated environment, Ray Maze has simple reward dynamics.
Each goal location has an independent 0.3 chance of yielding +1 reward, otherwise it gives reward.
All goal locations are independent, and the agent can only receive one goal reward each episode
Thus, perfect exploration within episode involves always visiting goal location (potentially finding a location that gives positive reward), and perfect exploitation involves always visiting positive reward location if one known, while avoiding all negative and unknown goal locations.
Assuming the agent always revisits positive goal locations, the expected reward visiting an new goal location with n episodes remaining is -0.4 0.3n (the expected value of the first visit, plus the expected value of revisiting positive rewards n times).
In the first episode the expected value of visiting a new goal location is 0.4 0.5.
In the second episode, the expected value is - 0.4 + 0.3 x 2 = 0.2.
In the third episode, the expected value is -0.4 + 0.3 x -0.1, and so not worthwhile.
20

---

Thus the optimal policy is to visit a goal location in the first episode (and then repeatedly revisit it if there was positive reward), and then if the first goal location was negative (70% chance), visit goal location in the second episode.
This policy achieves 0.5 0.7 0.2 0.64 expected total reward. This value is upper bound, it is possible that the agent cannot always find goal location.
21

---

D Tabulated Results:
Table 3: Bandits with One Fixed Arm Results, The bandit domain compares to two non-meta-Re RL baselines, marked f.
J1 = 0.5 (the deceptive setting)
A11 = (the non-deceptive setting)
mean = std dev
median
mean + std dev
median
First- Explore
127.7 2.0
128.4
128.4 + 0.7
128.6
RL
56.1 + 12.2
50
117.9 3.9
117.0
UCB-1'
116.8 + 0.5
116.7
116.1 + 0.6
115.8
TS
123.3 2.5
122.5
122.7 2.4
121.9
Table 4: Dark Treasure-Room Results
4 (the deceptive setting)
p 0 (the non-deceptive setting)
+ std dev
median
mean + std dev
median
First-Explore
2.0 0.3
1.8
11.1 0.2
11.0
RL
0.2 +: 0.1
0.2
7.2 = 0.7
7.4
VariBAD
0.2 +: 0.1
0.2
9.6 + 0.4
9.8
HyperX
-0.2 = 0.2
-0.2
7.5 = 1.2
8.0
Table 5: Ray Maze Results
mean + std dev
median
First-Explore
0.5 + 0.01
0.47
RL2
0 0+0
0
VariBAD
0
0
HyperX
0.07 + 0.07
0.06
22

---

E
Compute Usage
Each training run commanded single GPU, specifically a Nvidia T4, and up to cpu cores. Table gives the approximate walltime of each run.
Table 6: Compute Usage Per Training Run. Many of the meta-RL controls converged early, and did not improve with longer periods training time (see Appendix F Domains where this occurred are marked t. To save compute, these were not trained as long as First-Explore
Run
Runtime
Meta-RI RL Deceptive Bandits First-Explors
20 hours
Meta-RL Deceptive Bandits RL
40 hours (extended, see below)
Dark Treasure-Roon First-Explore
50 hours
Dark Treasure-Room HyperX & VariBAD & RL3
10 hourst
Ray Maze First-Explore
75 hours
Ray Maze HyperX, VariBAD, and RI,2
10 hourst
Notably, in the Dark Treasure-Room Room for = -4, VariBAD and RL rapidly converges to policy of staying still, and while HyperX seems to slowly improve rewards, the reward increase an artifact caused by the HyperX having an exploration incentive that is gradually attenuated to zero. This attenuation creates a slow convergence from negative reward (due to moving into traps and not exploiting) to a higher near zero reward (obtained by mostly staying still). Because the attenuation is designed to occur throughout the entire run, scaling the run length merely scales how long HyperX takes to converge to close zero reward.
Due desire control runs on this setting were limited to 10 hours. In contrast, First-Explore was run for longer, as continued to improve with additional training. This is fair comparison, due the controls having converged, increasing the control run training time would not yield better policies, as Figure [0 demonstrates. To this end, as RL on the bandit environment still showed improvement after TO hours, those runs were also extended.
waste compute converged policies,
this behavior was verified, the
Total compute used for the experiments would then be around 1100 hours (5 runs for each treatment). However, there was also hyperparameter search, e.g., for the RL bandit parameters. As such, total compute may be over 2000 GPU hours. Furthermore, there were many preliminary experiments to iterate on the First-Explore architecture as well as to research and identify cumulative-meta-RR deception.
F Poor Performance Regardless of Train Time:
The issue is not that the methods are slow to converge, and that with more training they could perform well. As Section describes, these controls achieve low cumulative reward regardless of how long the methods are trained. Figure 10 demonstrates this phenomenon on the deceptive treasure room.
Figure 10 Top: RL2, VariBAD, and HyperX average cumulative reward plotted against training time. RL (yellow) and VariBAD (gray) converge to zero reward almost immediately. This transition corresponds to the policies learning to stay still. HyperX (teal) reward increases (toward zero) throughout meta-training. However, this increase in reward comes not from HyperX learning an increasingly sophisticated policy, but instead is the result of the HyperX algorithm'' 's meta-training exploration bonus being linearly reduced from the start to the end of meta-tran veeta-training. Thus, once that bonus is near zero, HyperX also learns to stay still.
Figure 10 Bottom: HyperX with different training lengths (specified by number of episode steps). When HyperX is run for ten times long (orange) or ten times less long (blue) than the default training time (light blue) the same behavior is observed (of slow convergence to (slightly below) zero reward). This behavior demonstrates the improvement in reward comes from the HyperX algorithm reducing the exploration incentive during the meta-training. It also implies that changing the length of training runs (including running for much longer) would not change the final performance results.
23

---

Average Cumulative Reward
Control Training Reward
-2
RL2
VariBAD
-4
HyperX
20
40
60
80
100
% Training Progress
Average Cumulative Reward
HyperX ran for different lengths
0.0
-2.
-5.0
2e6 Episode Step Training
2e7 Episode Step Training
-7.5
2e8 Episode Step Training
20
60
80
100
% Training Progress
Figure 10: Regardless of training time, the meta-RI. controls perform poorly.
24

---

explore a new quadrant each of the four days, without ever retreading the G Myopic Exploration
Optimal Sequence Explorations
myopically Optimal Explorations
M 0
5
Figure 11: Well-planned sequential exploration can sometimes significantly outperform a sequence of optimal myopic explorations. For example, consider exploring plain over four days, where each day one must explore by walking from the plain's center. Sequence of Optimal Myopic Explorations one 'optimal' way of exploring is to perform spiral from the center (e.g., the red spiral on the right). This strategy achieves However, if one does spiral on day then on day 2, one must retread old ground - wasting time otherwise spent exploring new locations. Each day bee-lining to unseen areas and then spiralling from there is also optimal for that day, however increases the amount of retreading tomorrow. Optimal Sequence of Explorations: another optimal way of exploring on day to explore quadrant, visualized in red on the left. Again, one does not backtrack, this strategy is optimal on day 1. However, unlike the spiral strategy, this strategy also part an optimal sequence of four explorations, same ground.
optimal amount of exploration on day as one never retraces one's 's steps.
H K-Selection Phase
Automatic selection Phase μl
120
80
Meta -ool-Roootuttto
60
20
0-
-20
Figure 12: Demonstration of First-Explore's k-selection phase, for the bandit distribution, with
N1 = O. Five separate First-Explore runs are plotted. The training runs select different values of (due to the relative strengths of each runs explore and exploit policy), with the associated selected corresponding to the peak of each curve (marked by a cross).
25

---

I Final-Episode-Rewarrd Meta-RL
Methods such as MetaCURE [5]. EPI 23 and CCM [24] learn an exploration policy that aims to extract maximum environment information (independent of whether such information informs good exploitation). These approaches discard grounding exploration in (maximizing) future reward. Not grounding exploration in future exploitation reward means that the policy may learn (via weight updates) to spend meta-rollouts acquiring irrelevant information (e.g., the exact penalty of bad actions). This distraction potentially prevents optimal exploration from ever being learnt.
E-RL (25 modifies RL to ignore the first-k episode rewards. This modification enables pure exploration (that is not dissuaded by negative rewards). However, E-RL² introduces an across- episode value assignment problem: identifying which exploration episodes enabled good subsequent exploitation. This problem potentially limits training sample efficiency. Further, the exploratory episodes number is set as a hyperparameter and constant across all tasks (both at training and at inference), preventing efficient combination with curriculum that contains different difficulty tasks (as hard tasks may need significantly more exploration episodes than easy ones). Finally, hard coding k limits the flexibility and usefulness of E-RL because one cannot explore until satisfactory policy quality is reached, preventing meta-RL. in-context adaptation from sff-the-shelf replacing standard RL.
DREAM I| also separately optimizes exploration and exploitation policies (and grounds exploration in exploitation), but has four complex, manually designed, interacting components and reliance on knowing unique problem IDs during meta- veeta-raiiiing. This complexity enables increased sample- efficiency by avoiding the chicken and egg problem of simultaneously learning explore and exploit policies. Unlike E-RL?, because a part of DREAM's machinery must learn to produce the right information per problem based on the (unique, random) problem ID only, is unable to generalize or handle never-seen-befie and generality. For example, DREAM may potentially be difficult apply to problems where each training environment is unique (e.g.. for environments with continuous variables, or samples from otherwise vast search spaces). may also struggle when each environment is a hard-exploration challenge, as it may be difficult for the model to explore enough to learn which information is required to solve the problem. We believe curricula necessary solve such environments. However, because DREAM cannot generalize during meta-training (as described above). it cannot take advantage of a curriculum build an exploration skill set to tackle harder and harder exploration challenges.
challenges during meta-training, raising questions about its scalability
Applying First-Explore
final-episdderreward meta-RL setting is promising direction of future work, as First-Explore i) learns grounded exploration ii) can explore until sufficient information is obtained rather than having fixed number of explorations), and iii) does not rely privileged information (problem IDS), allowing generalization during meta-training.
J Evaluation Details
Evaluation (sampling the multiple evaluation environments and performing iterated rollouts) was with single GPU. For the Bandit Results, each of the First-Explore evaluations sampled 128 100 bandit environments, indepedent from those trained on. For the RL. bandit evaluations batch size was reduced to 2, 000 (due taking longer evaluate). Since there is meta-RL training variance for UCB1 I and TS, five independent evaluations were done, each with an independently sampled 10, 000 bandits.
UCB: UCB was implemented according to the description in [8) with e = Namely, each pull, 2log UCB picks the arm that maximizes ucb, (t) (t(t - 1)+ + V (1-1) where (t is the estimated
mean reward of the ith arm, Te(l- 1) is the number of times the ith arm has been pulled
For the Dark Freasure-Room and the Ray Maze domains, all policies were evaluated on batch of 1, 000 environments sampled independently from those trained on.
26

---

K
Training Details
K.1
Controls:
The official VariBAD [7] (VariBAD and RL²) and HyperX [10] (HyperX) codebase ran the meta- RL controls. Dark Preasure-Room trained with the default hyperparameters of the coded bases gridworld environments. These were found to perform well, with variations tried not yielding improvement. To provide a strong control on the Bandits with One Fixed Arm problem, the controls were advantaged by having individual hyperparameter gridsearches for each Ai value (unlike First-Explore). See the SI attached configuration file for the exact hyperparameters. Both of these codebases are licensed under MIT license.
K.2 First-Explore:
The architecture for both domains GPT-2 transformer architecture [3] specifically the Jax framework [26] implementation provided by Hugging Face (27), with the code being modified so that token embeddings could be passed rather than token IDs. The different hyperparameters for the two domains are given in Table 7 The code being provided with Modified MIT License (allowing free use with attribution).
For all domains each token embedding is the sum of a linear embedding of an action, a linear embedding of the observations that followed that action, linear embedding of the reward that followed that action, a positional encoding of the current timestep, and a positional encoding of the episode number. See the provided code for details. For the dark treasure-room environments reset token was added between episodes that contained the initial observations of the environment, and unique action embedding corresponding non-action. The bandit domain had such reset token.
Table 7: Model Hyperparameters
Hyperparameter
Bandit
Dark
Hidden Size
128
128
Number of Heads
4
Number of Layers
4
For training AdamW 28
with piece- wise linear warm up schedule that interpolates
linearly from an initial rate of 0 to the full learning rate in the first 10% training steps, and then interpolates linearly back to in the remaining 90% of training steps. Tabeeeggives the optimization hyperparameters.
Table 8: Optimization Hyperparameters
Hyperparameter
Value
Batch Size
128
Optimizer
Adam
Weight Decay
le-4
Learning Rate
3e-4
Hyperparameters chosen based on relatively modest amount of preliminary experimentation. Finally, for efficiency, all episode rollouts and training was done on GPU using the Jax framework 166
For evaluation, we then sample by taking the argmax over actions, and not add the e-noise.
27

---

Table Training Rollout Hyperparameters
Hyperparameter
Bandit
Darkroom
Ray Maze
Exploit Sampling Temperature
1
Explore Sampling Temperature
1
1
1
Policy Update Frequency
every training update
every
10, 000
every 5, 000
e chance of random action selection
0.05
0
0
Baseline Reward
0
0
Training Updates
200,000
1,000,000
1,000,000
L Dark Treosure-Room Visualizations
Figure 13: A visualization of the dark treasure-room The agent's position is visualized by the blue square, positive rewards are in green, and negative rewards in red, with the magnitude of reward being visualized by the colour intensity. When the agent enters a reward location it consumes the reward, and for that timestep is visualized having the additive mixture of the two colours.
Here are example iterated First- First-Explore rollouts of the two trained policies, Texplore Txploitt visualized for single sampled darkroom.
28

---

Explore
Exploit
e9
Explore Coverage
Exploit Reward
3.5
2.5
coveroge
2.0
1.5
1.0
0.5
0.0
0
18
27
36
45
9
context
episode
Figure 14: The first (First-Explore) explore episode. Top left visualizes the last step of a First-Explore explore episode, with the locations that are not the cumulative context being coloured white, as the agent is blind to them (having observations memory of those locations). This figure plots end of the first exploration, and shows a reward has been found. Bottom left visualizes coverage of the cumulative context by plotting the total number of unique locations visited by the exploration against the cumulative episode step count. In this explore, the agent never doubled back on itself, which good is optimal have many unique locations visited possible. Top right visualizes a step in a First- First-Explore exploit episode, with the locations that are context visualized. The agent effectively 'see" those locations in its memory. Bottom right plots the exploit reward against the exploit episode timestep. As this figure plots before the start of the exploit episode, the agent has yet to move and encounter rewards, but will have done so in the subsequent visualizations.
29

---

Exploit
Explore Coverage
Exploit Reward
3.5
3.0
18
coverage
stops context
cpicodo stop
Figure 15: The first (First-Explore) exploit episode. This figure uses the same visualization design as Figure 14 Left top and bottom the same in Figure Figure 14, and of the explore context, not the current exploit episode. Right top, the agent (the light blue square) has found the reward in the first two steps. Consuming the reward is visualized by the agent colour and the reward colour being combined. Right bottom, the associated episode reward is shown.
30

---

Exolore
Exploit
Explore Coverage
Exploit Reward
18
coverege
1.5
1.0
0.5
0.0
0
27
36 45
step
Figure 16: The fifth (First-Explore) explore episode. At the end of the 5th explore episode the agent has discovered a new positive reward at the top of the room, and can now 'see" it in memory. The new information presents opportunity for the exploit policy to obtain both rewards, but it only has exactly enough time-steps in an episode to navigate to do so, and thus cannot make mistake navigating.
Exolore
Exploit
Explore Coverage
Exploit Reward
3.5
3.0
19
coverage
stops
context
episode step
Figure 17: The first reward of the fifth (First-Explore) exploit episode. Two steps into the episode the agent (in consuming, light blue) has consumed the nearby reward.
31

---

Exolore
Exploit
e9
Explore Coverage
Exploit Reward
18
boverege
reward
1.5
1.0
0.5
0.0
0
27
36
45
step
Figure 18: The end of the fifth (First-Exploree exploit episode. After consuming the nearby reward the agent has reached the newly discovered reward at the top of the room and consumed it. This success required making no mistakes and navigating first the nearby reward then the top one on the first try. This inference is possible because the quickest the agent can reach both rewards is exactly the length of the episode steps). The navigation in this episode is an example intelligent exploitation, as after the information reveal (the reward at the top) of single episode the agent appropriately changes its policy based on the context and using the learnt environment prior (e.g.. how to navigate), produces highly structured behavior (navigating with no mistakes).
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: the claims in the abstract and introduction establish the claims of the paper and these claims are demonstrated empirically in section 6
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: there is limitation section (section 7 which identifies multiple limitations, and suggests potential areas of future work address these.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
33

---

Answer: [NA]
Justification: the paper makes several important conceptual contributions, such as identifying that cumulative meta-RL algorithms can be deceived by simple environment distributions (that standard-Re RLL algorithms would easily solve) (section and section ( This problem is clearly identified and argued. However, these contributions are in the identification of this phenomenon, the empirical demonstration of it, and creating a novel algorithm (First- Explore) that can explore effectively despite these deceptive environments. As such, there are no mathematical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: code is provided for the full architecture, well as for training and analysis. This code includes the main algorithm presented in the paper (First-Explore), as well code to replicate the controls that First-Explore was compared against. Furthermore, one of the main contributions is high level architecture which is clearly described in section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
34

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: the submitted code contains sufficient instruction to (along with the paper) faithfully reproduce the main experimental result.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cm ec/
public/guides: htts///oduumisssooooooo.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
7 //nips.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: the experimental setting is detailed in section 6
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: The statistical significance of several key comparisons are evaluated using Mann-Whitney
two-tailed Mann- Whitney test. Furthermore, all result plots include training runs for each experimental treatment, which are each individually plotted. This approach (combined
35

---

with stating the statistical significance) was preferred over providing error bars, as for low numbers of runs such bootstrapped confidence intervals can statistically misleading. The results also benefit from having a high effect size (e.g..
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: all experiments have their wall-clock runtime recorded in Table 3, along with utilized computer resources, and an estimate of total compute hours spent.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpshhhrppppppppppppp..//ppp//p
Answer: [Yes]
Justification: this paper conforms in every respect with the NeurIPS Code of Ethics. There are no Data-related concerns as all data purely synthetic (and generated by algorithms interacting with RL-environments). No human subjects were involved. The research presented has no capacity to cause harm. Furthermore, we flag the safety concerns of potentially training agent an agent outside of simulation.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
36

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: there are no direct societal implications of First-Explore, however we discuss First-Explore being exciting opportunity for the field of meta-RL (and so RL more generally). We believe the field as a whole has great potential to create powerful algorithm that if used responsibly could greatly benefit all of humanity.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: none of the models or data involved this paper have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
37

---

Answer: [Yes]
Justification: First-Explors uses 27 Hugging Face implementation of [3] GPT-2 transformer (under modified MIT license). Further, this paper uses two existing code bases (licensed under a MIT license) to implement VariBAD, RL and HyperX. This asset usage is also detailed in appendix
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcode com/coo//oaoooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: while code is provided to reproduce the experimental results, no assets are being released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper involves no crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
38

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "cmFafJvecz", "reviewer": "Reviewer_Vncg", "summary": "This paper addresses the challenge in meta-reinforcement learning, where agents struggle to perform intelligent exploration across episodes, often failing to avoid repetitive exploration of the same locations. The authors propose a method called First-Explore, which learns two distinct policies: one focused on exploration and another on exploitation. By separating these policies, First-Explore aims to overcome the limitations of existing cumulative-reward meta-RL methods, which maximize cumulative rewards rather than maximizing the rewards of the final episode and, hence, not exploring adequately. The proposed method improves performance in environments like deceiving bandits and dark treasure rooms.", "questions": "1. Can you give explicit formulas of the objectives for training exploration and exploitation policies?\n\n2. Can you explain why the proposed method takes so long to run on the tabular environments (50 GPU hours, according to Appendix C.1)?\n  \n3. How does the proposed perform on more complex tasks, like continuous control tasks (e.g., Meta-World)? How does the proposed method compare with MetaCURE and DREAM?", "limitations": "The authors discussed limitations of their work.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper is motivated by an important problem in traditional meta-RL approaches, where they maximize the cumulative returns across episodes rather than maximizing the final episodes, hence leading to insufficient exploration.\n\n2. The authors proposed an interesting idea that uses the performance of the exploitation policy as a direct objective for the exploration policy to separate exploration and exploitation in meta-adaption.", "weaknesses": "1. The idea of separating exploration and exploitation in meta RL is not new. Several prior works, like MetaCURE[1] and DREAM [2], proposed similar ideas, but they are not properly discussed or compared.\n\n2. The experiment is sufficient. The proposed approach is tested only on two tabular enironments, namely Meta-RL-Deceiving Bandits and Dark Treasure-Rooms. It is unclear how would the proposed method performance on more complex tasks like continuous control.\n\n3. The writing of the method section can be improved. It needs proper formulas or an algorithm box to clearly demonstrate the methods for training the exploration and exploitation policies.\n\n\n[1] Zhang, Jin, et al. \"Metacure: Meta reinforcement learning with empowerment-driven exploration.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Liu, Evan Z., et al. \"Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices.\" International conference on machine learning. PMLR, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading, engaging with, and critiquing our work. We greatly appreciate your time and feedback, which has significantly improved the paper. We are delighted that you consider First-Explore to be “motivated by an important problem” and “an interesting idea.”\n\nWe have addressed your concerns below, and by doing so, we have substantially strengthened the paper. We hope you will consider our improvements and increase your score by 2 to 3 points. Without this score increase, the paper may not be selected for publication.\n\n# Weaknesses Response\n> [What about, for example, MetaCURE and DREAM? These methods separate exploration and exploitation]\n\nDREAM and MetaCURE are final-episode-reward meta-RL methods. In contrast, this paper and algorithm address cumulative-reward Meta-RL (see lines 41, 71, 126, 293).  An extensive discussion of MetaCure and DREAM is provided in Appendix F (see Section 2, lines 126-128, 'for the sake of completeness, we discuss final-episode-reward meta-RL and its connection to First-Explore in Appendix F'). Please see that Appendix for the answer to your question. \n\nTo make it clearer, we will add text to Section 2 identifying DREAM and MetaCure  as final-episode-reward meta-RL methods and their difference to First-Explore.\n> [More complex environments are needed]\n\nWe have added a more complex domain, Ray Maze (see the main response to all reviewers for details). This new domain is significantly more complex than the Dark Treasure Room; however, the same pattern of performance holds: First-Explore outperforms all controls, which fail catastrophically due to the deceptive nature of the domain.\n\nFurthermore, although the tabular environments are less complex than Ray Maze, the tabular results are highly informative. First-Explore identifies a challenge where state-of-the-art methods perform abysmally, often learning to remain stationary. Identifying this issue is a significant contribution of the paper, and is best demonstrated in the simplest environments possible (as is usually the case in science). It is surprising, and worth sharing with the NeurIPS community, how even simple environments can be unsolvable by existing state-of-the-art cumulative-reward meta-RL approaches.\n\n> [The methods section needs proper formulas or an algorithm box]\n> [What are the explicit formulas of the training objectives?]\n\nWe have now included full pseudocode for the algorithm in the paper (see the main rebuttal response) and will edit the methods section for clarity.\n# Questions\n> [Why does First-Explore take so long to run on the tabular environments?]\n\nThe hyperparameters and choice of architecture are not optimized for short meta-training durations. The transformer size, for example, prioritizes sufficient model capacity for any task over training efficiency on simpler tasks. It was also motivated by the desire to use a standard implementation (GPT2) of sufficient capacity to learn complex relationships. Similarly, the hyperparameter T (see line 184) enables exceptionally reliable training; however, it also increases meta-training time. If a shorter training time was desired, then these choices could be reconsidered, and future work could focus on optimizing First-Explore for such efficiency.\n\nHowever, the paper makes no claims on First-Explore’s meta-training efficiency. The paper identifies a new challenge with cumulative-reward meta-RL (the problem of needing to forgo rewards to explore), demonstrates the issue empirically, and provides a new method (First-Explore) that solves the challenge and achieves state-of-the-art results. We will summarize these details in the paper, including the exciting opportunity for future work to focus on substantially improving meta-training time.\n> How does First-Explore perform on more complex tasks (e.g., Meta-World)?\n\nWe have added a new more complex domain - Ray Maze. First-Explore performs excellently in this domain (see the main reply for details).\n\nMeta-World’s environments do not require exploration that sacrifices rewards in early episodes, which is the central challenge this paper addresses. Evaluating First-Explore on such a benchmark would be interesting for future work, but it is beyond the scope of this paper.\n> How does the proposed method compare with MetaCURE and DREAM?\n\nMetaCURE and DREAM are final-episode-reward meta-RL methods designed for and applicable to the final-episode reward meta-RL setting. This paper exclusively makes claims about cumulative-reward meta-RL dynamics (see lines 41, 71, 126, 293, etc.). While comparing First-Explore to these (or any) methods is interesting, it is not essential to the scientific questions addressed here and is beyond this paper’s scope. That said, we do provide a discussion of this issue in Appendix F (please see it for the most direct answer to your question) and we speculate in the future works section that First-Explore may be applicable to the final-episode-reward setting.\n\n# Conclusion\nThank you again for your feedback. We hope you will support a score that would lead to our work being shared with the NeurIPS community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OqRbhMmXNT", "reviewer": "Reviewer_Erp8", "summary": "This work proposes a novel Meta-RL framework called First-Explore to address the balance between exploration and exploitation. The method learns two distinct policies: one for exploration and one for exploitation. These two policies are then combined to form the final inference policy. The effectiveness of the method has been demonstrated in bandit tasks.", "questions": "1. What is the practical objective of the exploration policy, and how can the informativeness of an episode be defined?\n\n2. I am curious about the training efficiency of the proposed method. Is it more efficient than learning an individual policy for each task?", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem of balancing exploration and exploitation in RL is highly relevant, and the proposed solution provides valuable insights.\n\n2. The experimental results are comprehensive, demonstrating the effectiveness of the proposed method across bandit tasks.", "weaknesses": "1. The writing of the paper needs more careful organization. Figures 1 and 2 are not cited in the main text, which makes it difficult to understand their relevance. Additionally, the training process for the explore and exploit policies is not well explained in Section 4.\n\n2. The inference policy appears to require exhaustive hyperparameter tuning (k), which limits the method's generalization and makes it challenging to apply to other tasks.\n\n3. The compared baselines seem outdated. It would be beneficial to provide comparison results with more recent methods to better demonstrate the effectiveness of the proposed approach. Additionally, the experimental results are not significant, as the proposed method does not outperform other baselines by a substantial margin in most tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading, engaging with, and critiquing our work. We greatly appreciate the time you have spent doing so, and your feedback has enabled us to strengthen the paper. We are delighted that First-Explore tackles “a highly relevant” problem, that the “proposed solution provides valuable insights,” and that the experimental results are “comprehensive.”\n\nWe have addressed your concerns and substantially improved the paper. Furthermore, we believe your current positive comments support a higher score. We hope you will consider our improvements and increase your score by 2 to 3 points. Without this score increase, the paper may not be selected for publication.\n\n# Weaknesses Responses\n> [Figures 1 and 2 are not referenced]\n\nThe paper now references Figure 1 and Figure 2.\n> The training process for the explore and exploit policies is not well explained in Section 4.\n\nDid you mean Section 3.2? Section 4 discusses related work. To address this issue, we have added pseudocode detailing the training process (see main reviewer response for details) and improved the writing in Section 3.2.\n> The inference policy appears to require exhaustive hyperparameter tuning (k), which limits the method's generalization and makes it challenging to apply to other tasks.\n\nAs mentioned in Section 3.2, lines 154-157, “k is not a hyperparameter as, unlike hyperparameters, all policy-weight updates are performed independently of k, precluding the need to train the explore and exploit policies multiple times (which is the majority of the training compute-expenditure).” As illustrated in Figure 2, after training, k is automatically chosen (with minimal compute) via a quick interval search on a validation set. It can thus be quickly updated if the test distribution shifts.\n> [The compared baselines seem outdated.]\n\nVariBAD and HyperX are state-of-the-art cumulative-reward meta-RL methods. This paper exclusively makes claims on cumulative-reward meta-RL behavior, as opposed to final-episode-reward meta-RL. Similarly, RL$^2$ is still frequently used for cutting edge research (e.g., AdA) and remains an important baseline due to its simplicity and power.\n> [the results are not significant]\n\nThe results have a large effect size, are extremely statistically significant, and fully support the claims of the paper. In short, the results are highly significant in every sense.\n\nThe paper’s central thesis is that in deceptive domains existing cumulative-reward meta-RL methods fail, and that First-Explore overcomes this challenge. In the deceptive version of all three domains – the bandits when $\\mu_1=0.5$ (Figure 3 A1), the Dark Treasure Room when $\\rho=-4$ (Figure 4 A1), and the Ray Maze (see main reviewer response) – the meta-RL controls achieve abysmal performance. In these deceptive domains, First-Explore achieves significantly higher rewards than the best performing meta-RL control: 128.36 vs. 56.12 (> 2x improvement), 1.99 vs. 0.16 (> 10x improvement), 0.46 vs. 0.06 (> 7x improvement) for the Bandit results, Dark Treasure Room results, and Ray Maze results respectively. This large effect size reflects a significant and meaningful difference in agent behavior; the controls have learnt to minimize movement and exploration (Figure 4 B1), while First-Explore explores initially at the cost of reward (Figure 3 B1 and Figure 4 A1). All the results are also extremely statistically significant ($p < 2 \\times 10^{-5}$). We shall modify the paper to make the results’ significance clearer.\n\n# Question Responses\n> What is the practical objective of the exploration policy, and how can the informativeness of an episode be defined?\n\nRegarding the practical objective of the exploration policy and the definition of informativeness of an episode: From Section 3.2, “the explore policy π explore is trained to produce episodes that are followed by the exploit policy achieving higher episode returns than those seen so far. These episodes are termed ‘informative.’”\n\nTraining the explore policy to produce these ‘informative’ episodes (ones that are followed by the exploit policy achieving higher episode returns than it would have otherwise) is detailed in the pseudocode now added to the paper (see the main response). We will also improve the writing throughout the paper to clarify these issues.\n\n> [Is First-Explore efficient?]\n\nIs it more efficient than standard-RL applied to each environment?\n\nAs the introduction states, “Meta-RL can potentially [be more sample efficient than standard-RL by] expending a large amount of compute to train a single agent that can then rapidly adapt to new environments, even showcasing human-like sample efficiency.” Once the initial compute expenditure is made then the trained algorithm can be extremely efficient on new (in-distribution) tasks at inference time (i.e., when adapting to new tasks).\n\nIs learning two policies more efficient than learning a single policy?\n\nLearning two policies enables First-Explore to avoid catastrophic failure in the deceptive domains. In this sense, First-Explore is far more efficient than the controls, as they fail to learn good behavior regardless of how long they are trained (e.g., in the hostile Dark Treasure Room Domain, RL2 and VariBAD converge to not moving, and HyperX exhibits the same poor performance regardless of training time, see Figure 5 in Appendix C).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kZNILHBY0q", "reviewer": "Reviewer_zDt1", "summary": "This paper proposes a method to mitigate the issue of deceptive rewards in Meta-RL, I.e., rewards that may impede further exploration leading to higher cumulative rewards. The authors propose to learn two policies: an exploration and exploitation policy that are conditioned on a common context. During policy training the reward from the exploitation policy is fed through the exploration policy via the context as feedback. In the next phase the amount of necessary exploration iterations are determined through the cumulative reward. When the optimal cutoff is found the final policy is the combination of the exploration and exploitation policy based on the optimal cutoff $k$. The authors test on a bandit problem and dark-treasure room environment, mitigating the issue of deceptive rewards compared to other known Meta-RL approaches.", "questions": "* How do you think your method scales to more complex environments? Would it be possible to maybe make a more complex dark-treasure-room?\n\n* In the caption of Figure 4 I don’t understand the following statement: \n\t> By first exploring for two episodes using πexplore, and then exploiting \tusing πexploit, First-Explore’s inference policy πinference achieves significant cumulative reward...”\n\n* I thought this cutoff was determined automatically and also considering the environments are procedural why does the value of 2 work for all the configurations? Shouldn’t this value vary depending on where the different objects are spawned?\n\n* Why does the performance of your method go down first in Figure 4 while this does not happen for the other methods?", "limitations": "I think the authors have properly addressed the limitation of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Meta-RL is a challenging and important problem in reinforcement learning. The problem of deceptive rewards in Meta-RL seems to be an inherent problem of Meta-RL worth solving.\n* The idea of conditioning on different exploration contexts to determine the exploration cutoff is a simple but elegant idea.\n* The proposed method is effective at mitigating the problem of deceptive rewards for the tested environments.\n* The paper is generally well written and I understood the motivation behind tackling this problem.", "weaknesses": "* The choice of $\\rho=4$ seems a bit arbitrary. I think $\\rho$ should be ablated to see what the effect of different penalties is on the proposed method and other methods is.\n* The environments chosen for the experiments seem very simplistic. It is difficult to assess whether the method would succeed when having to learn long-horizon and more complex policies. In this case it might become intractable to get feedback from the exploitation policy  to decide on the right amount of exploration steps.\n* I think a good test would also have been environments such as MiniGrid or Minihack for example, which have also been used for hierarchical reinforcement learning.  The required sequence of actions is more complex than the proposed environments, while still being observationally simple.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reading, engaging with, and critiquing our work. We greatly appreciate the time you have spent, and your feedback has enabled us to improve the paper. We are delighted that you support sharing First-Explore with the NeurIPS community and consider it well-motivated and addressing “a problem worth solving.”\n\nWe have addressed your concerns and substantially strengthened the paper. We hope you will consider our improvements and raise your score to further support its acceptance.\n\n# Weakness Response:\n> [The choice of $\\rho$ seems arbitrary.]\n\nSetting $\\rho=0$ ensures all objects have non-negative rewards, so exploring (visiting new locations) only increases rewards. This setting provides the simplest non-deceptive control, and demonstrates that the baselines (and First-Explore) can handle problems without local optima.\n\nSetting $\\rho = -4$ ensures that repeated and consistent exploitation is necessary for exploration to be worthwhile, thus creating local optima. Evaluating in this domain examines First-Explore’s central claim: that these exploration dynamics are deceptive and challenging to existing cumulative-reward meta-RL methods, and that First-Explore overcomes this challenge.\n\nWhen $\\rho = -4$, the expected reward of an object is -1 (as the rewards are distributed U[-4, 2]). Furthermore, the expected reward of an object with positive reward is only 1 (as they are distributed U[0, 2]), and only $\\frac{1}{3}$ of objects have positive reward. This means multiple exploitations are required before exploration pays off. If an object is found while exploring, the expected reward is -1. However, optimal exploitation yields an expected $\\frac{1}{3}$ reward (by avoiding traps and navigating to positive reward objects). Thus, it takes at least four exploitation episodes for exploration to be worthwhile, as $4 \\times \\frac{1}{3} - 1 = \\frac{1}{3} > 0$. \n\nWe will include an explanation of these choices in the paper.\n\n > [how would behavior vary with $\\rho$?]\n\nBetween $\\rho=-4$ and $\\rho=0$ there will be a transition point between the two results shown in the paper. When $\\rho=-4$, the baseline fails catastrophically while First-Explore performs well (see Figure 4 A1). When $\\rho=0$, the baselines perform reasonably, although First-Explore still outperforms them (see Figure 4 B1). While exciting (and a promising direction of future work), First-Explore outperforming the baselines in the non-deceptive case is not central to the paper’s claims. Increasing $\\rho$ from 0 would not qualitatively affect the results since the environment cannot become more non-deceptive. Decreasing $\\rho$ cannot worsen baseline performance since they already fail, but it might eventually harm First-Explore due to the sparsity of rewarding objects. We will add text to the paper noting this, along with supporting plots.\n\n> [What about more complex environments?]\n\nWe have added a more complex domain, Ray Maze (see the main response to all reviewers for details). This new domain is significantly more complex than the Dark Treasure Room. However, the same pattern of performance holds: First-Explore outperforms all controls, which fail catastrophically due to the deceptive nature of the domain.\n\n>[How about environments such as MiniGrid or Minihack for example, which have also been used for Hierarchical Reinforcement Learning?]\n\nThank you for the suggestion. We do not consider First-Explore an example of Hierarchical Reinforcement Learning, which typically involves high-level and low-level policies (e.g., a policy setting subgoals and another achieving them). First-Explore operates by exploring first, then exploiting, without one policy setting subgoals for another. Therefore, MiniGrid and MiniHack would not test the main scientific hypotheses of this paper.\n# Question Response:\n> [Is a more complex dark treasure room possible?]\n\nIndeed it is, as we now show! Please see the main PDF, and our main response for details on the new Ray Maze domain.\n> Why does the value of 2 work for all the configurations? Shouldn’t this value vary[...]?\n\nThere may be confusion about when First-Explore switches from exploring to exploiting. After training the explore and exploit policies, the number of exploration episodes $k$ is determined by evaluating on a new batch of environments. Given $k$, the inference policy explores for the first $k$ episodes (regardless of exploration performance) before switching to the exploit policy. Surprisingly, in deceptive domains, this simple approach outperforms all controls. We will clarify this in the text.\n\nWhat you describe (the exploit policy deciding on the right amount of exploration steps) is a promising direction of future work that we will add to Section 6.\n\n> Why does [First-Explore performance] go down first in Figure 4 [and not for] other methods?\n\nWhen $\\rho = -4$, the environments are such that the expected reward of an unseen location (one not visited in a prior episode) is negative, and thus moving in the first episode leads to negative expected reward. Positive cumulative rewards can be achieved because, over multiple episodes, the agent can exploit any positive rewards it has found while avoiding negative ones. Unfortunately, the other methods fail to learn this strategy and instead learn to mostly stay still (so avoiding negative reward in the first two episodes, but also preventing the significant cumulative reward First-Explore achieves by the end). \n\nThe reward goes down for 2 episodes (vs. another number), because $k$ was automatically determined to be 2 on this domain. This setting causes the resulting inference policy to explore for the first 2 episodes (and so seek unseen negative-expected-value locations) before then exploiting for the remaining 8 episodes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Many thanks for reading our work. We are grateful for the feedback provided and for the time you spent engaging with and critiquing First-Explore. We are delighted that First-Explore addresses an "important" [Vncg], "highly relevant" [Erp8] problem "worth solving" [zDt1], and that it is an "interesting" [Vncg], "elegant" [zDt1] idea that provides "valuable insights" [Erp8]. Like you, we feel that First-Explore addresses (and identifies) an important problem of significant relevance to the meta-RL community, and that the paper has great potential to spark future works that build on it. 

Using your feedback, we have strengthened the paper by adding a more challenging domain, Ray Maze, and by providing detailed training pseudocode (see below). We have also made several writing improvements. Please see our individual reviewer responses for replies to your questions and comments.

The current scores are low and prevent publication. In light of these substantial improvements, which we believe fully address your comments and concerns, we kindly request you consider raising your scores by 2-3 points to support sharing First-Explore with the NeurIPS community. We would appreciate it, and believe publishing this work will inform the community, improve understanding of RL, and inspire future research.

# Ray Maze

As requested, we have added a significantly more challenging domain, Ray Maze, demonstrating that First-Explore works beyond tabular and bandit environments (see rebuttal PDF for plots).

In Ray Maze, the agent must navigate a randomly generated maze of impassable walls, with the maze layout different for each sampled Ray Maze, to find one of three goal locations. Each goal location is a trap (70% probability, -1 penalty) or a treasure (30% probability, +1 reward). The agent can only receive one goal reward per episode. To perceive the maze, the agent receives 15 lidar observations (see PDF). Each lidar observation reports the distance to the nearest wall at a specific angle relative to the agent's orientation. It also details the orientation of the intercepted wall (east-west or north-south), and whether the lidar ray hit a goal location (but not whether it is a trap or a treasure). The agent has three actions: turn left, go forward, and turn right.

Ray Maze is a challenging domain for several reasons. It has a high-dimensional observation space (15 separate lidar measurements), complex action dynamics (with actions not commuting; for example, turning left then moving forward is different from moving forward then turning left), and a randomly generated maze that interacts with both movement and observation. Furthermore, the agent must learn from experience, and risk the traps in early episodes to enable exploiting and consistently finding treasure in later ones. Because goal locations are frequently traps, the agent can only obtain positive cumulative reward by first a) constantly searching for treasures in early episodes (despite this leading to negative expected reward) and then b) repeatedly exploiting in later episodes (navigating to identified treasures, while avoiding identified traps).

In this challenging domain, First-Explore significantly outperforms all other treatments (green lines are above all other lines in the rebuttal PDF), with First-Explore achieving over seven times the mean cumulative reward of the best control (0.46 vs. 0.06). The difference is statistically significant, with ($p < 6 \times 10^{-8}$) for all pairwise comparisons.

# Added Training Pseudocode
The cross-entropy loss calculation:
```python
def rollout(env, π, ψ, c_π, c_ψ):
	"""perform a single episode
	inputs: the environment (env),
	the agent policy π, the prediction policy ψ,
	and the current policies' contexts c_π, c_ψ
	returns the episode return, temp_loss, and updated contexts"""
	# n.b. temp_loss is only used if the episode meets a condition
	# see (*) in the conditional_action_loss function
	temp_loss, r = 0, 0
	s = env.reset() # state s
	for i in range(env.episode_length):
		# calculate action probabilities p_π, p_ψ for both policies
		# and update context
		p_π, c_π = π.action_probabilities(s, c_π)
		p_ψ, c_ψ = ψ.action_probabilities(s, c_ψ)
		a = sample_action(p_π)
		temp_loss += cross_entropy(a, p_π * p_ψ) # hardamard product
		# * p_π ensures action diversity by weighting against likely actions
		s = env.step(s, a)
		r += s.reward
	return r, temp_loss, c_π, c_ψ

def conditional_action_loss(φ, θ, D, b):
	"""calculates First-Explore loss for both exploring and exploiting on domain D
	using the agent and predictor parameters φ, θ
	and baseline reward b"""
	env = sample_env(D)
	π_explore, π_exploit = load_policies(θ)
	ψ_explore, ψ_exploit = load_policies(φ)
	c_π, c_ψ = set(), set() # the agent and predictor contexts
	loss, best_r = 0, b
	for i in range(D.episode_num):
        r_explore, l_explore, c_π, c_ψ = rollout(env, π_explore,
    	                                         ψ_explore, c_π, c_ψ)
    	r_exploit, l_exploit, _, _ = rollout(env, π_exploit, ψ_exploit, c_π, c_ψ)
        # ^exploit context not kept
    	# (*) accumulate loss if:
    	if r_exploit > best_r: # explore episode is 'maximal'
        	loss += l_explore
    	if r_exploit >= best_r: # exploit episode is 'informative'
        	loss += l_exploit
    	if r_exploit > best_r:
        	best_r = r_exploit
	return loss
```

Training with the above loss:
```python
def train(epoch_num, T, D, b):
	"""example First-Explore training (ignoring batchsize)
    runs the meta-rollouts, accumulating a loss
	this loss is then auto-differentiated"""
	T_counter = 0
	φ = θ = init_params()
	for i in range(epoch_num):
    	# θ is the agent behavior parameters
    	# φ is the prediction parameter (for double-DQN-style updates)
    	Δφ = ∂/∂φ(conditional_action_loss(φ, θ, D, b))
    	φ -= step_size * Δφ
    	# Update θ every T epochs
    	T_counter += 1
    	if T_counter == T:
        	θ = φ
        	T_counter = 0
	return θ
```

Meta-review: The paper presents an interesting and valuable contribution to the Meta-RL field by addressing the challenge of balancing exploration and exploitation, particularly in the presence of deceptive rewards. 
However, the concerns regarding mathematical clarity, computational cost, and the limited complexity of the tested environments suggest that there is still room for improvement. The paper would benefit from further refinement, particularly in the explicit formulation of its objectives and broader experimentation. 
Given the overall positive trajectory of the reviews post-rebuttal, I recommend accepting this paper.
The authors need to incorporate all the improvements proposed during the rebuttals and all the suggestions provided by the reviewers.
Decision: Accept (poster)