SlowGuess/NeurIPS_2023_OCR huggingface dataset:
still achieving the greatest asymptotic return by a small margin. As expected, RNN+HN does not outperform task inference on Walker. For Walker, only TI outperforms RNN+HN in terms of efficiency, and only TI Naive outperforms RNN+HN in terms of asymptotic return; however, the effect size is small and both TI and TI Naive have among the worst performance on Cheetah-Dir and the grid-worlds. Still, RNN+HN achieves similar performance on Walker, which is notable in a high dimensional task space. And, RNN+HN achieves greater returns overall.
4.3 MineCraft
We additionally evaluate on the MC-LS environment from Beck et al. [2020], designed to test long-term memory from visual observations in MineCraft. Here, the agent navigates through a series of 16 rooms. In each room, the agent navigates left or right around a column, depending on whether the column is made of diamond or iron. Discrete actions allow for a finite set of observations. Correct behavior can be computed from the observation and receives a reward of 0.1. At the end, the agent moves right or left depending on a signal (red or green) that defines the task and is shown before the first room. Correct and incorrect behavior receives a reward of 4 and -3, respectively. We allow the agent to adapt over two consecutive episodes, forming a single meta-episode.
On MC-LS we compare RNN+HN to VI+HN alone, given a limited compute budget and since VI+HN is an established contemporary task-inference baseline Beck et al. [2022]. Additionally, we add an extra seed (four in total) and a linear learning rate decay due to high variance in the environment. In Figure 7, we see that RNN+HN significantly outperforms VI+HN. While VI+HN learns to navigate through all rooms, it does not reliably learn the correct long-term memory behavior. In contrast, RNN+HN is able to adapt reliably within two episodes, and one seed even learns to adapt reliably within a single episode. While further work is needed to learn the optimal policy, these experiments demonstrate that RNN+HN outperforms VI+HN, even on more challenging domains.
Here we investigate why the recurrent hypernetworks have such robust performance on meta-RL benchmarks. First, we observe that the state processing differs between the RNN and RNN+HN
Figure 8: The RNN+S policy (a) and the RNN policy with a hypernetwork (b).
(a) Grid
(b) Grid Show
Figure 9: RNN+HN matches or exceeds RNN+S and RNN, but RNN+S is also strong on grid-worlds.
(b) Cheetah-Vel
(c) Cheetah-Dir
Figure 10: RNN+HN matches or exceeds RNN+S on MuJoCo tasks. RNN alone is a weak baseline.
(d) Ant-Dir
basielines. In particular, RNN conditions on the current state only through its dependence on  $\phi$ , whereas hypernetworks pass in the state again to the policy. Thus, we investigate whether the difference in inputs alone could be the cause of the improvement in performance. To this end, we introduce a new ablation to test the effect of just passing in the state again. Details are below. Second, we inspect how sensitivity to latent variables encoding the trajectory affects performance.
$\mathbf{RNN} + \mathbf{S}$ . Hypernetworks condition on the current state both through  $\phi$ , which contains information about trajectory, including the current state, and by directly conditioning on the state. Since the hypernetwork conditions on state twice, we test to see the effect of conditioning on the state twice without hypernetworks. We call this ablation  $\mathrm{RNN} + \mathrm{S}$ , which we write as  $\pi_{\theta}(a|s,\phi)$  (Figure 8). In an empirical evaluation, we see that while  $\mathrm{RNN} + \mathrm{S}$  does perform favorably relative to RNN alone,  $\mathrm{RNN} + \mathrm{HN}$  still outperforms  $\mathrm{RNN} + \mathrm{S}$  (Figures 9 and 10). In particular,  $\mathrm{RNN} + \mathrm{HN}$  achieves similar returns to  $\mathrm{RNN} + \mathrm{S}$  on Ant-Dir and Cheetah-Vel, and outperforms  $\mathrm{RNN} + \mathrm{S}$  on all other environments, in terms of asymptotic return and sample efficiency. Taken together, we see that the advantage of  $\mathrm{RNN} + \mathrm{HN}$  comes both from the ability to re-condition on state directly and from the hypernetwork architecture. These results confirm that to achieve the strongest performance, re-conditioning on state directly is not sufficient, and that the hypernetwork architecture itself is still critical.
Figure 11: Returns (a) and Gradient norms on Walker (b). The RNN method must increase this norm to condition on state, whereas others do not. Lower gradient norms seem important to performance.
Latent Gradients. We also investigate how the hypernetworks condition on the trajectory. In particular, we investigate the sensitivity of the output of the network to an intermediate latent representation of the trajectory. For this purpose, we chose to measure the gradient norm of the first hidden layer of the hypernetwork on the Walker environment. We perform this investigation both with the initialization method designed for hypernetworks that we used throughout our experiments, Bias-HyperInit [Beck et al., 2022], and with Kaiming initialization [He et al., 2015], not designed for hypernetworks. We add Kaiming initialization, since Bias-HyperInit ignores trajectories at the start of training [Beck et al., 2022]. First, we confirm the finding of Beck et al. [2022] that Bias-HyperInit is crucial for performance (Figure 11). Second, we see the two models that performs worst, RNN and RNN+HN Kaiming, also have the greatest norm. Moreover, we find that both RNN+HN and RNN+S start with a low gradient norm and then further decrease this norm throughout training, whereas the RNN model increases this norm. We hypothesize that a low norm, i.e., low sensitivity to the latent variable, is crucial for stable training and that the RNN model increases this norm to remain sensitive to the state, since the state is only encoded in the latent for this model.
As an empirical study of meta-RL, we cannot guarantee that recurrent hypernetworks will improve over every baseline nor on every environment. However, we mitigate this issue by comparing to many baselines and performing many ablations. In particular, we compare to a contemporary task-inference method (VI+HN), design our own baseline which we show to be stronger than others on all grid-worlds (TI++HN), and also include standard methods (TI and TI Naive), in addition to further ablations in the appendix. In as much as an empirical study can, we believe our study demonstrates a significant improvement of the RNN+HN method over existing baselines.
In this paper, we establish recurrent hypernetworks as a surprisingly strong method in meta-RL. While much effort has gone into designing specialized task-inference methods for meta-RL, we present the surprising result that the simpler recurrent methods can be easily adapted to outperform the task-inference methods. By combining recurrent methods with the hypernetwork architecture, we achieve a new strong baseline in meta-RL that is both robust and easy to implement. In comparison to existing evidence, we provide much stronger empirical results, afford equivalent computation for tuning to all baselines, and establish recurrent hypernetworks as a strong method. We additionally show that passing the state variable to the policy is a crucial component of this method. Finally, we presented gradient analysis suggesting lower latent gradient norms to play an important role in the performance of meta-RL methods. Since the gradient analysis is preliminary and investigates state and latent variables in isolation, future work could investigate the interaction between these variables. Future work could also analyze the interaction between hypernetworks and other sequence models, such as transformers. We hope these insights, along with a simple and robust method, open the way for the broader use of sample-efficient learning in meta-RL and beyond.
We would like to thank Luisa Zintgraf for her help with the VariBAD code-base along with general advice and discussion. Jacob Beck is supported by the Oxford-Google DeepMind Doctoral Scholarship. Risto Vuorio is supported by EPSRC Doctoral Training Partnership Scholarship, Department of Computer Science Scholarship, and Scatcherd European Scholarship. Zheng Xiong is supported by UK EPSRC CDT in Autonomous Intelligent Machines and Systems (grant number EP/S024050/1) and AWS.
Jacob Beck, Kamil Ciosek, Sam Devlin, Sebastian Tschiatschek, Cheng Zhang, and Katja Hofmann. Amr: Aggregated memory for reinforcement learning. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=Bk17bREtDr.
Jacob Beck, Matthew Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta-reinforcement learning. CoRL, 2022.
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028, 2023.
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2020.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. R12: Fast reinforcement learning via slow reinforcement learning. arXiv, 2016.
Chelsea Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charlie Deck, Joel Z. Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. NeurIPS, 2019.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In International Conference on Learning Representation (ICLR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv, 2019.
Pierre-Alexandre Kamienny, Matteo Pirotta, Alessandro Lazaric, Thibault Lavril, Nicolas Usunier, and Ludovic Denoyer. Learning adaptive exploration strategies in dynamic environments through informed policy regularization. arXiv preprint arXiv:2005.02934, 2020.
Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In International Conference on Machine Learning, pages 6925-6935. PMLR, 2021.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=B1DmUzWAW.
Tsendsuren Munkhdalai and Hong Yu. Meta networks. ICML, 2017.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In Proceedings of the 39th International Conference on Machine Learning. PMLR, 2022.
Matt Peng, Banghua Zhu, and Jiantao Jiao. Linear representation meta-reinforcement learning for instant adaptation. arxiv, 2021.
8.1 Additional Task-Inference Ablations
In order to select task-inference methods for ultimate comparison in the main body, we evaluated those methods, in addition to further task-inference ablations, on the Grid-World and Walker environments, and then selected the best performing methods. From these results, we chose  $\mathrm{TI} + + \mathrm{HN}$  (on Grid-World), and TI and TI Naive (on Walker). See Figure 12. We additionally chose to evaluate  $\mathrm{VI} + \mathrm{HN}$  in the main body, as it is a prior strong method Beck et al. [2022]. A table summarizing the components in all methods can be seen in Table 1. The additional ablations are depicted in Figure 13, with details given below:
VI. This baseline, called VariBAD, is proposed in Zintgraf et al. [2020] and is the same as  $\mathrm{VI + HN}$  in the main body, but without the hypernetwork.
$\mathbf{TI} + +$  . This baseline is the same as  $\mathrm{TI} + + \mathrm{HN}$  in the main body, but without the hypernetwork. The two additions over TI (denoted  $^ { \text{十十} }$  ) are re-use of the multi-task policy parameters as an initialization for the meta-RL policy, and training of task inference over trajectories from pre-training.
TI+HN. This baseline is the same as TI in the main body, but with the addition of a hypernetwork.
$\mathbf{BI} + + \mathbf{HN}$ . Here, the base-network for the task is inferred (denoted "BI"), instead of the task representation directly. This method is novel and is the same as  $\mathrm{TI} + + \mathrm{HN}$  in the main body, but task inference is trained to reconstruct the parameters for the base-network produced by the multi-task hypernet,  $\phi^{\prime}$ , rather than the task representation ( $c_{\mathcal{M}}$  or  $g$ ):
$$
\hat {\phi} ^ {\prime} = P ^ {\phi^ {\prime}} (z \sim I B)
$$
$$
J _ {i n f e r} (\theta) = \mathbb {E} _ {\mathcal {M}} [ \mathbb {E} _ {\tau | \pi} [ - | | \phi^ {\prime} - \hat {\phi} ^ {\prime} | | _ {2} ^ {2} ] ].
$$
Note that while the performance of  $\mathrm{BI}++\mathrm{HN}$  is similar to that of  $\mathrm{TI}++\mathrm{HN}$  on Grid-World,  $\mathrm{TI}++\mathrm{HN}$  was chosen as a representative baseline since it is more common to existing methods and  $\mathrm{BI}++\mathrm{HN}$  requires significantly more compute.
Figure 12: TI++HN was chosen as a baseline given results from Grid-World (a). TI and TI Naive were chosen as baselines given results from Walker (b).
Figure 13: Additional Task-inference baselines.
Inference Target	Policy Conditions on State	Hypernetwork	Inference Training in Multi-Task Phase and Parameter Reuse
RNN	None	No	No	N/A
RNN+S	None	Yes	No	N/A
RNN+HN	None	Yes	Yes	N/A
TI Naive	Given	Yes	No	N/A
TI	Learned	Yes	No	No
TI++	Learned	Yes	No	Yes
TI+HN	Learned	Yes	Yes	No
TI++HN	Learned	Yes	Yes	Yes
VI	Transitions	Yes	No	N/A
VI+HN	Transitions	Yes	Yes	N/A
BI++HN	Base Net	Yes	Yes	Yes
Table 1: Summary of the components in each method
8.2 Hyperparameter Tuning
We tune each baseline over five learning rates for the policy, [3e-3, 1e-3, 3e-4, 1e-4, 3e-5], for three seeds each. We use a learning rate for the task inference modules of 0.001, both since varying the learning rate seemed to have little effect (Figure 14) and to provide a fair comparison in terms of computation. Error bars report a  $68\%$  confidence interval using bootstrapping.
Before being passed to the hypernetwork or to the policy, depending on the model, the task  $(c$  or  $g)$  are projected down to size 25, as are the concatenation of  $(\mu ,\sigma)$ . This size was chosen to be in the range of the values used by Zintgraf et al. [2020] across environments, but a single value was chosen to be consistent. Additionally, 25 is sufficiently large for a one-hot representation of all discrete task distributions used. However, on Ant-Dir, the projection size is 10. For the state embedding size (passed to the policy) we chose 256 and for the sizes of the MLP policy we chose 256, followed by 128. This corresponds to the "XL" model size in Beck et al. [2022], which was optimal or near optimal where reported. However, for the comparison of RNN, RNN+S, and RNN+HN on Grid-World, these results are reported with a state embedding size for the policy tuned over [5, 25, 125, 625]. For the state embedding size passed to the trajectory encoder, we used 32 for all
(a) Grid-World Pre-Train Time
(b) Grid-World LR
Figure 14: Training for 100 updates yielded optimal performance despite not fully training the multi-task policy, Multi+HN (a). Varying the learning rate for task inference module had little effect (b).
experiments, as the default value in Zintgraf et al. [2020]. And for all RNNs, we use a single GRU layer of size 256, as in Beck et al. [2022].
As discussed in the main body, pre-training of the multi-task policy occurs at the expense of the meta-RL policy training time, in order to ensure the same number of samples for training each method. While selecting the time for pre-training does add a hyperparameter that must be selected, we tune this once on Grid-World and then transfer to all other environments, in order to afford as little additional computation as possible to these baselines. Note that while this cutoff affords some extra computation to task inference methods, the recurrent hypernetwork method (RNN+HN), which we show to be the strongest, does not require any such additional tuning.
On Grid-World, we find 100 PPO updates for training the multi-task policy to be optimal. In this environment, each update consists of 960 frames. Interestingly, while the multi-task policy (Multi+HN) requires approximately 500 updates to fully train, training for 100 yielded optimal performance for  $\mathrm{TI} + + \mathrm{HN}$ . (We saw similar results for TI, not reported here, except that the random projection provided by 0 updates already gave similar performance to 100.) Note that 100 updates is equivalent to only  $20\%$  of the total frames for training the multi-task policy, and it is only  $2.4\%$  of the total frames for training the meta-RL policy. This suggests that very little amount of computation is required to get a good task representation, and delaying the start of meta-training significantly is generally not worthwhile. For MuJoCo experiments in the main body, we use 563 updates, since that is the same percent of total frames  $(2.4\%)$  for Walker and Ant-Dir, and  $(4.8\%)$  on Cheetah-Vel and Cheetah-Dir), which transferred well on initial experiments.
For all other hyperparameters, we default to those in Beck et al. [2022].
8.3 End-to-End and Task Inference Together
While it is possible to combine the end-to-end objective and task inference objective, prior work suggests that interference from competing objectives may be detrimental [Humphlik et al., 2019]. Moreover, separating the training objectives can be practically useful, as the combined objectives may require tuning of the relative weighting of the two objectives. Still, we perform investigatory experiments on Grid-World. See Figure 15. We find that adding task-inference objectives to the end-to-end meta-RL objective decreased return compared to the end-to-end objective alone (RNN+HN) for every method tested. The same effect can be seen regardless of the architecture (RNN+HN or RNN+S). Unsurprisingly, the performance of the combined end-to-end and task-inference methods falls in between each type of method used independently. Additionally, tuning the relative relative objective weighting had little effect. We experiment with giving  $10\%$  (BI10p, TI10p) and  $50\%$  (BI50p) weighting to task inference methods, but still observing the same decrease in return. In all these experiments, we use the RNN architecture from the end-to-end RNN methods, meaning that we did not use an information bottleneck, nor the linear encoding layer. In other words, the output of the RNN passed to the policy,  $\phi$ , was directly used as  $\hat{g}$  or  $\hat{c}$  and compared to the respective label  $g$  or  $c$ .
(a) Grid-World Combined Objectives
(b) Tuning Combined Weighting
Figure 15: Adding task inference objectives in combination with the end-to-end meta-RL objective decreases performance relative to end-to-end alone, and results in performance between the two types of methods (a). Varying the relative objective weighting had little effect (b).
8.4 Meta-World Experiments
We additionally evaluate on Meta-World (ML10) Yu et al. [2020] experiments and show RNN+HN is able to match the performance of the more complicated VI+HN method (Figure 16). Meta-World is a meta-RL benchmark for robotic manipulation. There are 10 non-parametric training tasks and 5 distinct non-parametric test tasks, ranging from pushing a ball to opening a window. Additionally, there is non-parametric variation, e.g. of the goal location, within each task. In this environment, the variance of returns and the computation requirements for training are dramatically greater, so the evidence is not strong is any direction. However, we include the results here since ML10 is a standard benchmark Beck et al. [2023], and since the only experiment ever reported in meta-RL with RNN+HN prior to our paper is on ML10 Beck et al. [2022].
Contrary to our findings, Beck et al. [2022] demonstrated RNN+HN to underperform VI+HN on ML10. However, since that result was not statistically significant, as calculated by Beck et al. [2022], we conduct further experiments on ML10. First, we run more seeds for the VI+HN model in their experiment. (Their experiment is the same as ours, but we use a latent encoding size of 25, instead of 10, for the projection of  $(\mu, \sigma)$ , to be consistent with the majority of our experiments, and we run for slightly fewer frames due to the intensive computation requirements.) Using different seeds from Beck et al. [2022] resulted in worse performance, indicating that the results truly were insignificant due to variance. Beck et al. [2022] calculate a mean return of 23.9 for VI+HN from the returns: [12.48, 25.69, 33.61]. In contrast, we calculate the final mean return is 18.35 for VI+HN, with an average over three seeds, with final returns: [32.80, 3.32, 18.93]. Comparing to the returns for RNN+HN from Beck et al. [2022], [11.31, 3.37, 27.77], it is difficult to draw strong conclusion from this experiment.
Additionally, we run our own experiments (with latent size 25) and compare RNN+HN to VI+HN. Contrary to Beck et al. [2022], we find that the final return of RNN+HN is higher than VI+HN. RNN+HN has a final average test success percent of 17.22, compared to 13.87 for VI+HN. However, the variance is still too large to draw firm conclusions. Additionally, we find that the learning curves for RNN+HN and VI+HN overlap consistently throughout training, and the confidence interval for RNN+HN lies almost entirely within that of VI+HN. Note that while tuning hyper-parameters, one of the seeds for the largest learning rate (3e-3) of RNN+HN encountered numerical instability and was ignored. However, 1e-4 was selected as the optimal learning rate, so this is unlikely to affect results, and if it did, would only increase the returns of RNN+HN.
While these results are not sufficient to conclusively show superiority on ML10, we do show superiority on many other environments. Additionally, these results do show RNN+HN to match the performance of VI+HN. Matching existing task-inference baselines on ML10 is significant, since it demonstrates that complicated task-inference methods are not necessary to be competitive.
(a) Additional Seeds on Original ML10 Experiment
(b) Our ML10 Results
Figure 16: Running three additional seeds of  $\mathrm{VI + HN}$  on the ML10 experiment from Beck et al. [2022] (with a latent size 10) shows the result is not strong due to high variance. Note that the initial sharp decline in one seed can occur due to smoothing with valid padding on a binary reward (a). While also not a strong result, running our own experiment on ML10 (with a latent size 25) shows that the RNN+HN is able to match the performance of the more complicated  $\mathrm{VI + HN}$  method (b).
8.5 Gridworld Details
Grid-World consists of a five by five grid with a goal location in one of the grid cells. The agent starts in the bottom left corner of the grid, and then has 15 steps in an episode to find the goal and remain there. The agent receives 1 reward for each timestep at the goal and -0.1 reward for all other timesteps. The goal location is held constant for each meta-episode, which consists of 4 regular episodes. In Grid-World Show, the goal position is visible to the agent at the first timestep of each episode. In Grid-World Dense, the agent receives and observes a reward equal to the Manhattan distance to the goal location.
8.6 MuJoCo Details
We evaluate on all four MuJoCo Todorov et al. [2012] environments used by Zintgraf et al. [2020]. All environments require legged locomotion. All episodes terminate after 200 timesteps. Meta-episodes in Walker consist of two regular episodes, whereas meta-episodes in the rest consist of one regular episode.