SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jacob Beck
Department of Computer Science  
University of Oxford, United Kingdom  
jacobbeck@alumni.brown.edu
Zheng Xiong
Department of Computer Science  
University of Oxford, United Kingdom  
zheng.xiong@cs.ox.ac.uk
Risto Vuorio
Department of Computer Science  
University of Oxford, United Kingdom  
risto.vuorio@keble.ox.ac.uk
Shimon Whiteson
Department of Computer Science  
University of Oxford, United Kingdom  
shimon.whiteson@cs.ox.ac.uk
Deep reinforcement learning (RL) is notoriously impractical to deploy due to sample inefficiency. Meta-RL directly addresses this sample inefficiency by learning to perform few-shot learning when a distribution of related tasks is available for meta-training. While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shelf sequential model, such as a recurrent network, is a surprisingly strong baseline. However, such claims have been controversial due to limited supporting evidence, particularly in the face of prior work establishing precisely the opposite. In this paper, we conduct an empirical investigation. While we likewise find that a recurrent network can achieve strong performance, we demonstrate that the use of hypernetworks is crucial to maximizing their potential. Surprisingly, when combined with hypernetworks, the recurrent baselines that are far simpler than existing specialized methods actually achieve the strongest performance of all methods evaluated. We provide code at https://github.com/jacooba/hyper.
Meta-reinforcement learning [Beck et al., 2023] uses sample-inefficient reinforcement learning (RL) to learn a sample-efficient reinforcement learning algorithm. The sample-efficient algorithm maps the data an agent has gathered so far to a policy based on that experience. To this end, any sequential model such as a recurrent neural network (RNN), can be deployed to learn this mapping end-to-end [Duan et al., 2016, Wang et al., 2016]. Such methods are also called black-box [Beck et al., 2023].
Alternatively, much prior work has focused on a category of task-inference methods that are specialized for meta-RL. A meta-RL algorithm learns to reinforcement learn over a distribution of MDPs, or tasks. By explicitly learning to infer the task, many methods have shown improved performance relative to the recurrent baseline Humplik et al. [2019], Zintgraf et al. [2020], Kamienny et al. [2020], Liu et al. [2021], Beck et al. [2022].
Recent work has shown the simpler recurrent methods to be a competitive baseline relative to task-inference methods [Ni et al., 2022]. However, such claims are contentious, as the supporting experiments compare only to one task-inference method designed for meta-RL, the experiments provide additional compute to the recurrent baseline, and the results still show similar or inferior performance to more complicated methods on the majority of difficult domains. In particular, they consider two toy domains and four challenging domains, with RNNs significantly outperformed on two of the four challenging domains, and superior to the single task-inference baseline on only one.
Figure 1: In some environments, recurrent neural networks fail to learn meta-RL tasks, whereas recurrent hypernetworks achieve strong performance.
In this paper, we conduct a far more extensive empirical investigation with stronger and carefully designed baselines in meta-RL specifically. In addition, we afford equal computation in terms of number of samples for hyper-parameter tuning to all existing baselines. We present the key insight that the use of a hypernetwork architecture [Ha et al., 2017] is crucial to maximizing the potential of recurrent networks. For an illustration of the potential magnitude of improvement, see Figure 1. While the use of a hypernetwork with RNNs is not a novel idea, they have never been evaluated in meta-RL beyond a single environment, let alone shown to outperform contemporary task-inference methods [Beck et al., 2022]. We additionally provide preliminary evidence that the robust performance hypernetworks achieve such is in part due to how they condition on the current state and history. Finally, our results establish recurrent hypernetworks as an exceedingly strong method on meta-RL benchmarks that is also far simpler than alternatives, providing significant ease of use for practitioners in meta-RL.
Recurrent Meta-RL. Many meta-RL methods structure the learned RL algorithm as a black box using a neural network as a general purpose sequence model [Duan et al., 2016, Wang et al., 2016, Mishra et al., 2018, Fortunato et al., 2019, Ritter et al., 2021, Wang et al., 2021, Ni et al., 2022]. While any sequence model can be used, often the model is structured as an RNN [Duan et al., 2016, Wang et al., 2016, Ni et al., 2022]. Such models [Duan et al., 2016, Wang et al., 2016] are commonly used as simple meta-RL baselines.
One study has shown RNNs to be a competitive baseline in meta-RL [Ni et al., 2022]; however, the scope of the study was broader than meta-RL and the evidence specific to meta-RL is inconclusive. First, the study evaluates only a single specialized meta-RL method [Zintgraf et al., 2020], which was, but is not currently, state-of-the-art [Beck et al., 2022]. Second, the experiments use results or hyperparameters from the original papers, while affording extra computation to tune the RNNs on each benchmark, including dimensions that were not tuned for the other baselines. This computation includes tuning architecture choices, the context length, the RL algorithm used, and the inputs [Ni et al., 2022]. And third, the study does not show particularly strong performance of recurrent methods relative to the chosen specialized baseline. On the MuJoCo domains, the recurrent baseline outperforms the specialized method on only one of these four domains, performs similarly on another, and is significantly outperformed on the other remaining two [Ni et al., 2022]. In contrast, our work compares against four specialized baselines; affords equal computation to all methods, defaulting to hyper-parameters that favor existing task-inference methods for parameters that are not tuned; and still establishes recurrent hypernetworks as the strongest method evaluated.
Task Inference Meta-RL. In addition to recurrent meta-RL methods, task-inference methods [Humplik et al., 2019, Zintgraf et al., 2020, Kamienny et al., 2020, Liu et al., 2021, Beck et al., 2022] and policy-gradient methods [Yoon et al., 2018, Finn et al., 2017, Vuorio et al., 2019, Zintgraf et al., 2019] constitute a significant bulk of existing work. We exclude the latter methods from
Figure 2: The standard RNN policy (a) and the RNN policy with a hypernetwork (b).
comparison since the estimation of a policy gradient in policy-gradient approaches requires more data than in our benchmarks [Zintgraf et al., 2019, Beck et al., 2023]. Task inference methods are a strong baseline for our benchmark, but are generally more complicated than recurrent meta-RL methods. For example, such methods typically add a task inference objective [Humplik et al., 2019], and may also add a variational inference component [Zintgraf et al., 2021], or pre-training of embeddings with privileged information [Liu et al., 2021]. In this paper, we ablate each of these components to create the strongest task-inference baselines possible. In the end, we find the more complicated task inference methods are still inferior to the recurrent baseline with hypernetworks.
Hypernetworks. A hypernetwork [Ha et al., 2017] is a neural network that produces the parameters (weights and biases) for another neural network, called the base network. Hypernetworks have been used in supervised learning (SL) [Ha et al., 2017, Chang et al., 2020], Meta-SL [Rusu et al., 2019, Munkhdalai and Yu, 2017, Przewiezlikowski et al., 2022], and meta-RL [Beck et al., 2022, Xian et al., 2021, Peng et al., 2021, Sarafian et al., 2021]. While these networks are complicated, and can fail to work out-of-the-box, simple initialization methods can be sufficient to enable stable learning [Beck et al., 2022, Chang et al., 2020]. In meta-RL, only Beck et al. [2022] have investigated training a hypernetwork end-to-end to arbitrarily modify the weights of a policy. This study suggests that hypernetworks are particularly useful in preventing interference between different tasks and enable greater returns as the number of parameters increases. However, their study shows a task-inference method to be superior, and the recurrent hypernetwork is evaluated only on a single task with results that are statistically insignificant. Recurrent hypernetworks have never been widely evaluated in meta-RL, let alone shown to outperform contemporary task-inference methods.
A task in RL is formalized as a Markov Decision Processes (MDP), defined as a tuple of  $(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{P},\gamma)$ . At each time-step  $t$ , the agent inhabits a state,  $s_t\in S$ , which it can observe. The agent then performs an action  $a_{t}\in \mathcal{A}$ . The MDP subsequently transitions to the next state  $s_{t + 1}\sim \mathcal{P}(s_{t + 1}|s_t,a_t)\colon \mathcal{S}\times \mathcal{A}\times \mathcal{S}\to \mathbb{R}_+$ , and the agent receives reward  $r_t = \mathcal{R}(s_t,a_t)\colon \mathcal{S}\times \mathcal{A}\to \mathbb{R}$  upon entering  $s_{t + 1}$ . The agent acts to maximize the expected future discounted reward,  $R(\tau) = \sum_{r_t\in \tau}\gamma^t r_t$ , where  $\tau$  denotes the agent's trajectory throughout an episode in the MDP, and  $\gamma \in [0,1)$  is a discount factor. The agent takes actions sampled from a learned policy,  $\pi (a|s):\mathcal{S}\times \mathcal{A}\to \mathbb{R}_{+}$ .
Meta-RL algorithms learn an RL algorithm,  $f(\tau)$ , that maps from the data,  $\tau$ , sampled from a single MDP,  $\mathcal{M} \sim p(\mathcal{M})$ , to policy parameters  $\phi$ . As in a single RL task,  $\tau$  is a sequence up to time-step  $t$  forming a trajectory  $\tau_t \in (\mathcal{S} \times \mathcal{A} \times \mathbb{R})^t$ . Here, however,  $\tau$  may span multiple episodes within a single MDP, since multiple episodes of interaction may be necessary to produce a reasonable policy. We use the same symbol,  $\tau$ , but refer to it as a meta-episode. The policy,  $\pi_{\theta}(a|\phi = f_{\theta}(\tau))$ , is parameterized by  $\phi$ .  $f$  is itself parameterized by  $\theta$ , which are referred to as the meta-parameters.
Figure 3: Task-inference baselines. Naive task (a), additional multi-task pre-training (b).
The objective in meta-RL is to find meta-parameters  $\theta$  that maximize the sum of the returns in the meta-episode across a distribution of tasks (MDPs):
$$
\underset {\theta} {\arg \max } \mathbb {E} _ {\mathcal {M} \sim p (\mathcal {M})} \left[ \mathbb {E} _ {\tau} \left[ R (\tau) \mid \pi_ {\theta} (\cdot | f _ {\theta} (\tau)), \mathcal {M} \right] \right]. \tag {1}
$$
$f_{\theta}$  is referred to as the inner-loop, which produces  $\phi$  , in contrast to the outer-loop, which produces  $\theta$
3.2 Recurrent Methods
Recurrent methods are perhaps the simplest and most common meta-RL baseline. Recurrent methods use an RNN to encode history and train all meta-parameters end-to-end on Equation 1. These methods are depicted in Figure 2. While neither recurrent networks (RNN below), nor the combination of a hypernetwork with recurrent networks (RNN+HN below) is a novel, the combination has never been widely evaluated in meta-RL [Beck et al., 2022], but we will show that the combination actually achieves the strongest results.
RNN. Our first recurrent baseline is the simplest and is equivalent to RL2 [Duan et al., 2016] and L2RL [Wang et al., 2016]. In this case  $\pi_{\theta}(a|\phi = f_{\theta}(\tau))$ , where  $f$  is a recurrent network,  $\pi$  is a feed-forward network, and  $f$  and  $\pi$  each use distinct subsets of the meta-parameters,  $\theta$ .
$\mathbf{RNN} + \mathbf{HN}$ . Our second recurrent model is the recurrent hypernet and achieves the strongest results. Here, the recurrent network produces the weights and biases for the policy directly:  $\pi_{\phi}(a|s)$ . The state must be passed as input again to this policy for the feed-forward policy to condition on an input, and we follow the initialization method for hypernetworks, Bias-HyperInit, suggested by Beck et al. [2022]. In this initialization, the hypernetwork's final linear layer is initialized with a zero weight matrix and a non-zero bias, so that the hypernetwork produces the same base-network parameters for any trajectory at the start of training.
3.3 Task-Inference Methods
Task-inference methods [Beck et al., 2023] constitute the main category of meta-RL methods capable of adaptation as quickly as recurrent (i.e., black-box) methods [Humplik et al., 2019, Zintgraf et al., 2020, Kamienny et al., 2020, Liu et al., 2021, Beck et al., 2022]. These methods train the inner-loop not end-to-end but rather to identify the task, within the given task distribution. One perspective on these methods is that they attempt to shift the problem from the more difficult meta-RL setting to the easier multi-task setting by learning to explicitly infer the task [Beck et al., 2023]. Here we define relevant task-inference methods used as baselines (Figures 3 and 4). For additional ablations, summary, and details on the method selection process, see the appendix.
TI Naive. The inner-loop of a meta-RL method must take in the trajectory,  $\tau$ , and produce the policy parameters,  $\phi$ . In task inference methods, the inner-loop additionally produces an estimate
Figure 4: Task-inference baselines. Task inference with additional parameter reuse and hypernetwork (a), an existing contemporary task-inference algorithm [Beck et al., 2022] (b).
of the current task,  $\hat{c}_{\mathcal{M}}$ , given a known task representation,  $c_{\mathcal{M}}$ . While it is possible to represent  $\phi$  as  $\hat{c}_{\mathcal{M}}$  directly, i.e. pass  $\hat{c}_{\mathcal{M}}$  to the policy, it is common to compute  $\phi$  from an intermediate layer of the network that predicts  $\hat{c}_{\mathcal{M}}$ , which contains more information about the belief state [Humplik et al., 2019, Zintgraf et al., 2020]. It is also common to use an information bottleneck to remove information not useful for task inference from the trajectory [Humplik et al., 2019, Zintgraf et al., 2020]. Following Zintgraf et al. [2020], we condition  $\phi$  on the mean and variance of the bottleneck layer in order to explicitly condition the policy on task uncertainty for more efficient exploration (Figure 3). Putting these together, we can write a task inference method as follows:
$$
\mu = P ^ {\mu} (R N N (\tau))
$$
$$
\sigma = P ^ {\sigma} (R N N (\tau))
$$
$$
I B = \mathcal {N} (z; \mu , \sigma)
$$
$$
\hat {c} _ {\mathcal {M}} = P ^ {c} (z \sim I B)
$$
$$
\phi = R e L U \left(P ^ {\phi} \bot (\mu , \sigma)\right)
$$
$$
J _ {i n f e r} (\theta) = \mathbb {E} _ {\mathcal {M}} \left[ \mathbb {E} _ {\tau | \pi} [ - | | c _ {\mathcal {M}} - \hat {c} _ {\mathcal {M}} | | _ {2} ^ {2} ] \right]
$$
$$
J _ {p r i o r} (\theta) = \mathbb {E} _ {\mathcal {M}} \left[ \mathbb {E} _ {\tau | \pi} [ D (I B | | \mathcal {N} (z; \mu = 0, \sigma = I)) ] \right],
$$
where  $P^c$ ,  $P^\phi$ ,  $P_\mu$ , and  $P_\sigma$  are all linear projections,  $\perp$  represents a stop-gradient,  $P^\phi \perp (\mu, \sigma)$  is the matrix multiplication of  $P^\phi$  with stop-gradient of a concatenation of  $(\mu, \sigma)$ , and  $D$  is the KL-divergence. Here,  $J_{infer} + J_{prior}$  constitutes the evidence lower bound from Zintgraf et al. [2020] and is used to train  $IB$ , whereas all other parameters  $(P^\phi$  and  $\pi(\cdot|\phi))$  are trained via Equation 1.
TI. As presented, the TI Naive baseline may suffer from a known issue where the given task representation contains too little or too much information [Beck et al., 2023]. When too little information is present, the policy may miss information crucial to the task. When too much information is present, the policy may be presented with the difficult problem of separating the useful information from irrelevant task features. Toward this end, it is possible to pre-train the task representation end-to-end using an additional policy [Humplik et al., 2019, Kamienny et al., 2020, Liu et al., 2021]. Our TI baseline (Figure 3) is the same as TI Naive, except that an additional multi-task policy,  $\pi^{\prime}$ , is pre-trained to learn a representation of the task,  $g_{\theta}(c_{\mathcal{M}})$ . Given a linear projection,  $P^{g}$ :
$$
\hat {g} = P ^ {g} (z \sim I B)
$$
$$
J _ {m u l t i} (\theta) = \mathbb {E} _ {\mathcal {M} \sim p (\mathcal {M})} [ \mathbb {E} _ {\tau} [ R (\tau) | \pi_ {\theta} ^ {\prime} (\cdot | g _ {\theta} (c _ {\mathcal {M}})), \mathcal {M} ]
$$
$$
J _ {i n f e r} (\theta) = \mathbb {E} _ {\mathcal {M}} \left[ \mathbb {E} _ {\tau | \pi (\cdot | \phi)} [ - \| g _ {\theta} (c _ {\mathcal {M}}) - \hat {g} \| _ {2} ^ {2} ] \right].
$$
For a fair comparison, training of the multi-task policy,  $\pi^{\prime}$ , occurs at the expense of training the meta-learned policy  $\pi$ , with the total number of samples remaining constant. Instead of fully training the multi-task policy, we experiment with different amounts of pre-training in the appendix, finding significant benefits already from less than  $5\%$  of total training allocation for the pre-training.
$\mathbf{TI} + + \mathbf{HN}$ . The  $\mathrm{TI} + + \mathrm{HN}$  baseline is the same as TI, with three additions that we found to strengthen task inference (Figure 4). The first two additions  $(++)$  are novel and are 1) initializing the parameters of the meta-policy,  $\pi$ , to that of the pre-trained multi-task policy,  $\pi'$ , to encourage transfer and 2)
(a) Grid-World
(b) Grid Show
Figure 5: Evaluation on grid-world benchmarks. RNN+HN and TI++HN improve return. RNN+HN achieves the greatest asymptotic return and sample efficiency.
(c) Grid Dense
training of the task-inference  $(J_{infer})$  over trajectories from the initial multi-task training phase in addition to the meta-learning phase, since the former tend to be more informative and simply provide extra data. The third addition (HN) uses a hypernetwork to condition the policy on  $\phi$  [Beck et al., 2022]. We write this as  $\pi_{\phi}(\cdot |s)$  to show that  $\phi$  represents the weights and biases of  $\pi$ , just as in the recurrent baseline. The output of the hypernetwork is  $\phi$ , and the input to the hypernetwork is a the projection of  $\mu$  and  $\sigma$ ,  $\phi = h(ReLU(P^{\phi}\bot (\mu ,\sigma)))$ . When using a hypernetwork with the first two additions  $(++)$ , the parameters of the hypernetwork for the meta-learned policy are initialized to the parameters of hypernetwork for the multi-task policy, instead of sharing policy parameters directly.
$\mathbf{VI} + \mathbf{HN}$ . While task-inference methods rely on known task representations, it is also possible to design methods that can infer the MDP more directly. This can be done by inferring transitions and rewards in full trajectories, since the transition function and reward function collectively define the MDP. In particular, such a method, called VariBAD, is proposed by Zintgraf et al. [2020], and extended with the use of hypernetworks by [Beck et al., 2022]. Here, we call this method  $\mathrm{VI} + \mathrm{HN}$ , and it is a contemporary task-inference method (Figure 4). Precisely, this model reconstructs full trajectories including future transitions for meta-episodes,  $\tau_{0:T}$ , instead of task embeddings, from  $\tau$ , the current trajectory:
$$
J _ {i n f e r} (\theta) = \mathbb {E} _ {\mathcal {M}} \left[ \mathbb {E} _ {\tau | \pi (\cdot | \phi)} [ - | | \tau_ {0: T} - \hat {\tau} _ {0: T} | | _ {2} ^ {2} ] \right].
$$
See the appendix for ablations with VI alone.
In this section we compare recurrent hypernetworks (RNN+HN) to task inference baselines. We evaluate over three simple navigation domains [Zintgraf et al., 2020, Humplik et al., 2019, Rakelly et al., 2019], designed to test learning of exploration and memory, in addition to four more difficult tasks using MuJoCo [Todorov et al., 2012], and one task testing long-term memory from visual observations in MineCraft [Beck et al., 2020]. Results show meta-episode return, optimized over five learning rates and averaged over three seeds (four in MineCraft), with a  $68\%$  confidence interval using bootstrapping. Additional details and results on Meta-World [Yu et al., 2020] are available in the appendix. Our experiments demonstrate that while our task inference methods are strong baselines, RNN+HN is able to outperform them and achieve the highest returns.
4.1 Grid-Worlds
Navigation tasks are a common benchmark in meta-RL [Zintgraf et al., 2020, Humplik et al., 2019, Rakelly et al., 2019]. Here we evaluate on the grid-world variant from Zintgraf et al. [2020], in addition to two of our own variants. The first environment, Grid-World, consists of a five by five grid with a goal location in one of the grid cells. The agent starts in the bottom left corner of the grid, and then must navigate to a goal location, which is held constant throughout the meta-episode. (Details are in the appendix.) This environment is useful for testing how well a meta-learning algorithm learns to efficiently explore in the gridworld as it searches for the goal. Additionally, our Grid-World Show environment was designed to be relatively harder for end-to-end methods, in order to provide a challenge for our proposed method. In this environment, the goal position is visible to the agent
Figure 6: Models evaluated on MuJoCo benchmark. RNN+HN matches returns on Walker and Cheetah-Vel, and exceeds returns on Cheetah-Dir and Ant-Dir.
at the first timestep of each episode. Task inference methods will directly encourage the storage of this information in memory, whereas the end-to-end recurrent methods must learn to store this information through its effect on the policy. In contrast, our Grid-World Dense environment provides dense rewards and may be easier for end-to-end methods. In this environment, the agent receives and observes a reward equal to the Manhattan distance to the goal location. Instead of inferring the task explicitly, the agent can simply move up or to the right until the reward stops increasing.
Surprisingly, on all three grid-worlds, RNN+HN achieves both the greatest asymptotic return and greatest sample efficiency (Figure 5). The recurrent hypernetwork achieves the fastest learning on Gridworld Show, despite the environment being specifically designed to be harder for end-to-end methods.  $\mathrm{TI} + + \mathrm{HN}$  dominates all other task-inference baselines on these grid-worlds, suggesting that it is a relatively strong task-inference method. Collectively, these grid-worlds demonstrate that end-to-end learning with hypernetworks can learn to store the task in memory and to explore optimally with this information directly from return, just as well as task-inference methods.
4.2 MuJoCo
Here we evaluate baselines on more challenging domains. We evaluate on all four MuJoCo variants proposed by Zintgraf et al. [2020], which is known to be a common and more challenging meta-RL benchmark involving distributions of tasks requiring legged locomotion [Zintgraf et al., 2020, Humplik et al., 2019, Rakelly et al., 2019, Beck et al., 2023]. Ant-Dir and Cheetah-Dir both involve non-parametric task variation, whereas Cheetah-Vel and Walker include parametric variation of the target velocity and physics coefficients respectively. For environment details, see the appendix. We expect Walker in particular to be difficult for end-to-end methods since it has the largest space of tasks with the dynamics defined by 65 different parameters. Assuming the values of these parameters are important for the optimal policy, task inference methods may learn the optimal policy faster.
On Cheetah-Dir and Ant-Dir, RNN+HN achieves greater returns than all other baselines by a wide margin (Figure 6). On Cheetah-Vel, all methods achieve fairly similar results, with RNN+HN
Figure 7: RNN+HN outperforms VI+HN on MC-LS (MineCraft) environment.