Hypernetworks in Meta-Reinforcement Learning
Authors: Jacob Beck
Matthew Thomas Jackson
Risto Vuorio
Shimon Whiteson
Venue: CoRL 2022 Poster (2022)
Keywords: Hypernetwork
Deep Learning
Reinforcement Learning
Reinforcement
RL
Meta-Learning
Meta
Meta-RL
Meta-World
Robotics

Abstract: Training a reinforcement learning (RL) agent on a real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim to improve sample efficiency by generalizing over a distribution of related tasks. However, doing so is difficult in practice: In multi-task RL, state of the art methods often fail to outperform a degenerate solution that simply learns each task separately. Hypernetworks are a promising path forward since they replicate the separate policies of the degenerate solution while also allowing for generalization across tasks, and are applicable to meta-RL. However, evidence from supervised learning suggests hypernetwork performance is highly sensitive to the initialization. In this paper, we 1) show that hypernetwork initialization is also a critical factor in meta-RL, and that naive initializations yield poor performance; 2) propose a novel hypernetwork initialization scheme that matches or exceeds the performance of a state-of-the-art approach proposed for supervised settings, as well as being simpler and more general; and 3) use this method to show that hypernetworks can improve performance in meta-RL by evaluating on multiple simulated robotics benchmarks.

Hypernetworks in Meta-Reinforcement Learning
Jacob Beck
Matthew Jackson
Department of Computer Science University Oxford, United Kingdom
Department of Engineering Science University of Oxford, United Kingdom j jackson@robots. ox. ac
jacob broo.beek@llum. brown.
Risto Vuorio
Shimon Whiteson
Department of Computer Science University of Oxford, United Kingdom risto. www.wokkere....e.oooooooo. ox. ac. uk
Department of Computer Science University of Oxford, United Kingdom shimon. whiteson@cs ox. ac uk
Abstract:
Training reinforcement learning (RL) agent real-world robotics task remains generally impractical due to sample inefficiency. Multi-task RL and meta-RL aim improve sample efficiency by generalizing over distribution of related tasks. However, doing so is difficult in practice: In multi-task RL, state of the methods often fail outperform degenerate solution that simply learns each task separately. Hypernetworks are a promising path forward since they replicate the separate policies of the degenerate solution while also allowing for generalization across tasks, and are applicable to meta-RI.. However, evidence from supervised learning suggests hypernetwork performance highly sensitive to the initialization. In this paper, we 1) show that hypernetwork initialization is also a critical factor in meta-RL, RL, and that naive initializations yield poor per- formance; 2) propose novel hypernetwork initialization scheme that matches exceeds the performance of state-of-the-an approach proposed for supervised settings, as well as being simpler and more general; and 3) use this method to show that hypernetworks can improve performance in me by evaluating on multiple simulated robotics benchmarks.
Keywords: Meta-Learning, Reinforcement, Hypernetwork
1
Introduction
Deep reinforcement learning (RL) has helped solve previously intractable problems but still remains highly sample inefficient. This sample inefficiency makes it impractical, particularly in settings where data collection happens in the real world. For example, robot's actions have the potential to inflict damage on both itself and its surroundings. Multi-task RL and meta-RL aim to improve sample efficiency on novel tasks by generalizing over distribution of related tasks. However, such generalization has proven difficult in practice. In fact, multi-task RL methods often fail to outperform degenerate solution that simply trains separate policy for each task [1].
One promising way to improve generalization is with hypernetwork, a neural network that pro- duces the parameters for another network, called the base network [2]. In multi-task RI, using hypernetwork that conditions on the task ID to generate task-specific parameters can replicate the separate policies of the degenerate solution, while also allowing generalization across tasks. Fur- thermore, unlike the degenerate solution, hypernetworks can also be applied to meta-RL, where task IDs provided and test tasks may be novel, by conditioning them the output of task encoder.
with their challenges. Since hypernttwrks generate base net- work parameters, the initialization parameters in the hypernetwork determines the initialization of the base network it produces. Evidence suggests hypernetwork performance is highly sensitive to the initialization scheme in supervised learning [3). However, to our knowledge this question has
However, hypernetoorks
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Cheetah-Dir
Pick-Place
70
Architecture
60
Hypernetwork
VariBAD
1500
Rotur
1000
40
500
30
Hypernetwork Init
-500
Bias-Hyperlnit HFI
10
Kalming
1000
10000
20000
30000
40000
50000
60000
70000
10000
20000
30000
40000
Step
Step (e)
(a) Hypemetwork initialization methods
(b) Architectures
Figure 1: Naive initializations such as Kaiming [4] fail for hypernetworks, whereas our proposed Bias-Hyperlnit does not and matches the state of the art, HFI [3] (claims 1, 2). Adding hypemet- works with the proposed Bias-Hyperlnit significantly improves the state-of-the-an meta-RI. method, VariBAD [5] (claim 3).
not been considered in meta-RL. In this paper, show that hypernetwork initialization is also critical factor in meta-RL, and that naive initializations yield poor performance.
Furthermore, we propose two novel initialization schemes: Bias- HaussHyyprlnit and Both produce strong results, with the former matching or exceeding the performance of the state-of- the-art hypernetwork initialization method designed for supervised learning [3]. Moreover, both pro- posed methods are simpler and more general than this existing method, that they may applied to arbitrary base network architectures and target base network initializations without additional derivation. Using Bias- Bias-Hyperlnit, we present results that substantially improve the a statc-of-the- tf-th--rr method range of meta-RL benchmarks.
Applying hypernetworks meta-RL, we make the following contributions (examples in Figure 1):
1. We empirically demonstrate that initialization is critical factor in the performance of hypernetworks in meta-RL, and that naive initializations fail learn reliably;
2. We propose novel hypernetwor initialization scheme that matches exceeds the per- formance state-of-the-an approach proposed for supervised settings, as well as being simpler and more general; and
3. We use this method to show that hypernetworks improve a state-of-the-an method on a range meta-RL benchmarks (grid-world [5], MuJoCo [6], and Meta- World [1]).
Related Work
Meta-RL. Despite the advantages of hypemetworks [2], they remain relatively unexplored in meta-RL. We use hypernetworks to arbitrarily update policy's parameters at every time-step. whereas all prior work we are aware of restrict this procedure in some way. Many procedures in few-shot meta-RL build off of MAML [7] to adapt the parameters of a policy network using pol- icy gradient [7, 8, 9]. Such methods require the estimation of policy gradient, which reduces sample-efficiency when faster adaptation is possible, as in our benchmarks [5]. Most meta-learning procedures capable of zero-shot adaption using an RNN (or convolutions) that can represent an ar- bitrary update function [5, 10, 11]. These methods generally update a set of activations on which a fixed policy is then conditioned. whereas hypernetworks update all policy parameters. We include a state-of of the-art method from this class in our evaluations [5]. There are also unsupervised meth- ods in zero-shot meta-RL. for weight updates [12, 13] but none can produce a fully general learning procedure since they make use of local and unsupervised heuristics. Sarafian et al. [14] use hyper- networks in the context of meta-RL, but the policy network, not the hypernetwork, conditioned on the RNN used for adaptation, preventing the hypernetwork from representing general learning pro- cedure. Finally, FLAP [155 learns to infer set of weights trained the multi-task setting; however since the adaptation procedure is not trained
me objective, it is constrained. For example,

---

FLAP cannot learn to explore to reduce uncertainty. Finally, Xian et al. [16] use hypernetworks to predict model dynamics then use model predictive control. However, this model still requires planning to make use of an uncertain model, whereas model-free RI. learns a policy that explores optimally in order to attain data for adaptation. Using a general procedure trained to arbitrarily modify the weights of model-free policy has never been tried in RI., to the best of our knowledge.
Hypernetworks. Hypernetworks, or similar architectures, have been used in supervised learning (SL), multi-task RL, and meta-SL. Hypernetworks have been used in the supervised learning litera- ture for sequence modelling [2], as well as in continual learning and image classification [3], where it was shown that the hypernetwork initialization scheme was crucial for performance. Similar mod- els have also been used in multi-task RI. and meta-SI., but not meta-RI.. For instance, in multi-task RL, Yu et al. [17] use a network conditioned on a task encoding to produce the weights and biases for every other layer in another network conditioned on state. In meta-SL, there have also been attempts to use one network to adapt weights of another, both as a general function of the dataset [18, 19, 20], conditioned on an embedding adapted by gradient descent [21]. and by adding deltas in way framed as learning to optimize [22, 23]. The abundance of representations in meta-SL. suggest there is similarly large space of representation based methods explore in meta-RL. Our work getting hypernetorks to work in practice for meta-RL- can be seen a first step towards applying all of these methods in meta-RI.
3 Background
3.1 Problem Setting
An RL task is formalized as Markov Decision Processes (MDP). We define MDP as a tuple of (S, A, R, P,r) 7). At time-step t, the agent inhabits a state, S; E S, observable by the agent. The agent takes action a E A. The MDP then transitions to state 1 P(s a;): S x A x S R20, and the agent receives reward r = R(s ae): S x A + R upon entering $1+1- Given discount factor, y E [0, 1), the agent acts to maximize the expected future discounted reward, R(T) = ((r)=Erer 7'T,, where r is the agent's 's trajectory over an episode in the MDP. To maximize this return, the action takes actions sampled from a learned policy, π (als)) S x A + R+.
Meta-RL algorithms learn an RL algorithm, i.e., a mapping from the data sampled from a single MDP, MM- ~ p(M), to a policy. Since an RL algorithm generally needs multiple episodes of inter- action to produce reasonable policy, the algorithm conditions on T. which is the entire sequence of states, actions, and rewards within M. As in the RL setting, this sequence up to time-step forms trajectory Te E (S A x R). Here, however, may span multiple episodes, and same symbol, but refer to it as a meta-episode. The policy is then a meta-episode dependent policy, Ta(a|s, T), parameterized by the meta-parameters, 6.
the
We define the objective in meta-RL. as finding meta- meta-parameters 0 that maximize the sum of the returns in the meta meta-episode across a distribution of tasks (MDPs):
arg max E EM-P(M) E, [((T) Te(:) (),M))
3.2 Policy Architecture
We consider meta-RL. agents capable of adaptation at every time-step, and adaptation within one episode is required to solve some of our benchmarks. In such methods 15, 10, 11]. the history is generally summarized by a function, g. into an embedding that represents relevant task information. We write this embedding as g(T), and call the task encoder. The policy, represented multi-layer perceptron, then conditions on this task embedding as an input, instead of on the history directly, which we write (a s, e). We call this the standard architecture, shown in Figure 2.
In this paper, we primarily build off of VariBAD [5], which can be seen as an instance of the standard architecture where the task encoder is the mean and variance from a recurrent variational auto- encoder (VAE) [24] trained using a self-supervised loss. other words, the task is inferred as a latent variable optimized for reconstructing a meta-episode. See Zintgraf et al. [5] for details. Additionally, evaluate the addition of hypernetworks to RL.2 [11] on the most challenging benchmark. (See section 5.2.) In RI.2, the task encoder is recurrent neural network trained end-to-end on equation 1.
3

---

To a s e)
T (alst)
Hypernet
T
π
ho
State Encoder
Task Encoder
State Encoder
Task Encoder
1
1
St
Tt
St
Tt
Figure 2: A standard architecture (left) and hypernetwork model (right).
3.3 Hypernetwork Initialization
Chang et al. [3] show that applying existing initialization methods for neural networks to hyper- networks produces unstable base network initializations with exploding or vanishing activations. Furthermore, they empirically demonstrate reduction in training stability for Kaiming initializa- tion [4]. We corroborate this failure for Kaiming initialization on meta-RL, as well as for Orthogonal initialization [25] and Norme initialization [26], which collectively refer as default initializa- tions.
As solution, Chang et al. [3] propose the first initialization designed for hypernetworks and show be effective in supervised learning. Their approach based on Kaiming initialization, which samples network parameters such that the activations of the network at each layer maintain the same variance as the previous layer. Chang al. [3] extend this variance analysis to hypernetworks [2]. They propose two methods: Hyperfan-in (HFI) sets the variance of the initial parameter distribution of the hypernetwork to maintain constant variance activations the base network in the forward pass, and hyperfan-out (HFO) does the same for the backward pass. Since these are equally compet- itive, and produce state-of ofeee-of-hharrr results, we include HFI as a baseline for comparison. However, this variance analysis is involved and requires modification for specific activation function and whether or not the network produces weights or biases the base network. This motivates the need for simpler and more general initialization method, which we propose in this paper.
cases depending the
Methods
In this section we introduce our proposed architecture using hypernetworks and proposed hy- pernetwork initialization. At high level, the hypernetwork conditions on a task representation to generate all of the parameters for base policy. The initialization provides simple and general way to sample parameters for this hypernetwork at the start of training.
4.1 Policy Architecture
We propose the use of hypernetworks in meta-RI, instead of the standard architecture described carlier. In this setting, we use a hypernetorr to arbitrarily adapt the parameters of the base policy. We still use task encoder, use a hypernetwork, heg to generate policy parameters, ø = he(e). These parameters, i.c., weights and biases, are then used directly for the base network, which we write: (alss)) This is also depicted Figure 2.
9(7), but instead of conditioning a policy on these activations,
In this case, the hyperneworr can arbitrarily adapt all the the parameters of based on history. In comparison, in the standard architecture, the shared fixed parameters of the base network (0) are still required to generalize between all of the tasks. Since training separate policies for each task often performs better than state-of-the-an methods for generalizing across all tasks [1], this motivates the ability to produce base policies with no or few shared parameters. Hypernetworks allow for shared

---

parameters when possible, but also provide the ability to have no shared parameters in the base policy when diverse policies are necessary and little generalization between tasks is possible.
In fact, hypernetworks are capable of replicating the training of separate policies for each task under certain conditions. To see this, consider the case where the hypernetwork is linear and has bias. Then, the hypernetwork consists only of weight matrix, W. (That is, = W.) If this hypernetwork conditions on one-hot task ID for task this hypernetwork, Φ are separate set of parameters for each task: o' = h(() WL, In other words, training individual networks for each task is equivalent to training hypernetwork when that hypernetwork is: 1) linear, 2) has no bias, and 3) is conditioned on one-hot task ID.
4, then the parameters selected by
relax these assumptions and still retain the ability to produce distinct parameters, while also enabling generalization. If we add a bias, we reintroduce shared parameters in the hyper- network, but they can still produce separate base networks for each task when little transfer between the policies is required. If we relax the one-hot assumption, the network is still capable of producing one-hot encoding when the tasks are discrete and the task embedding is sufficiently large. Relaxing these restrictions allows for both generalization and the application of hypernetworks to meta-RL.
However,
4.2 Hypernetwork Initialization
Default initialization methods fail for hypernetworks. However, given that hypernetworks generalize training separate networks for each task, it must be possible to initialize them as if each of the cor- responding base networks were initialized independently, from some given initialization function, f, known to train reliably. Using this insight, we propose and evaluate two initialization schemes for the hypernetwork. We propose one where (under some assumptions) each base network is inde- pendently initialized from f. We also propose one where (under assumptions) all base networks share an initialization sampled from f.
Our first method is Weight-Hyperrnit.. Let W and b be the weights and bias in the last layer of our hypernetwork, h(e), respectively, (These parameters are both contained in 0.) We define this weight-only initialization as follows:
W. (()) Vi,
b b==0 0,
where f is an arbitrary initialization scheme for the base network specifying a distribution over vector of parameters, 6, and W., is the i-th column of W.
Weight- Seight-Hyperlnit reproduces any given initialization for each base network (πa), under the assump- tions that e = 1, and the hypemetwork is linear. In this case, each column of W is simply sample from the base scheme, one which is selected for each task via the one-hot encoding. For example, given the task embedding e 11, the following base network initialization ónni is produced:
0
i
Ami = We
+ 0 = 33
(2)
0
Moreover, in the case that there is also no bias, it is also equivalent to training separate networks for each task. Although these assumptions are not met for meta-RL, and so do not hold for our experiments, we find this still an improvement over default neural network initialization schemes.
Additionally, we propose Bias-Hyperlnit. This bias-only initialization is defined as follows:
Was : Vi,j. b == @shared f()).
Bias- Stas-Hyperlnit achieves an arbitrary initialization for the base network without any assumptions, by setting the parameters for any task to be the same at initialization. This encourages parameter sharing between base networks at the beginning of training, where possible, but also allows for separate base network parameters to be learned, if necessary. Under any set of assumptions, the
5

---

base network is initialised to the following:
201
0 0
302
Pnnt Wx b
0
23
@shared
Pshareds
(3)
22
where x is the final hidden layer of the hypernetwork. (X in the of a linear hypernetwork.) Both methods initialize only W and b, which define the head of the network. All other layers in may be initialized by any default initialization scheme. All such choices are detailed in supplemen- tary materials.
4.3 Baselines
VariBAD & RL.2. See Sec. 3.2.
FiLM. FiLM [27] is a convenient baseline situated between hypernetworks and the standard ar- chitecture. In FiLM, the hypemetwork generates biases for each layer, but only point- wait scales the activations instead of generating weight matrices. In this case, the base network has its own weights. Hyperlylrl can easily be adapted to FiLM; details presented in supplementary materials.
HFL. HFI [3] is state-of-the-an initialization method developed tested in the supervised learning setting. While we do compare to HFI, our methods are simpler and more general. Specifically, our methods work with arbitrary base network target initialization (as opposed to being tied to Kaiming) and our methods work with arbitrary base network architectures (without additional variance analy- approach is straightforward to apply additional methods, which show by
sis). Moreover, applying to FiLM. Finally, although both HFI and Weight-Hyperlnit do make assumptions about the input to the hyperntworr,, strongest method, Bias-Hyperlnit, does not.
Experiments
In this section, compare hypernetwork architecture and initialization methods to baselines 2D navigation [5], MuloCo [1], ML1 [1], and MI.10 [1] benchmarks. MuloCo is common meta- RL benchmark [7, 5, 28, 29, 12], are toy 2D navigation tasks 5, 28, 29]. These two benchmarks allow us to demonstrate that hypernetworss with default initialization methods fail to learn, whereas our proposed methods learn reliably. ML1 and MLI0 benchmarks in Meta-Worlh [1]. These two benchmarks have greater room for improvement with state-of-the-art methods [5], which allow demonstrate improvement over the baseline architectures. Finally, ronments to investigate the performance of hypernetworks against standard architectures in terms of the number of parameters the model overall.
MuJoCo envi-
Throughout our evaluation, we use two-tailed t-tests with p = 0.05 to determine significance. De- tails model tuning and implementation are presented in supplementary materials.
5.1 Navigation and MuJoCo
Here we evaluate on the grid-world variant from Zintgraf et al. [5] as our 2D navigation task and MuJoCo [6]. Note respectively, while the other MuJoCo enviromments have parametric variation between the tasks.
and Cheetah-Dir contain twenty-four and two non-parametric tasks
In Table reliably. Specifically, Kaiming and Norme initializations used with hyperneworrss achieve far lower returns than all other methods. Orthogonal initialization is competitive, however it is still significantly outperformed by Bias Bias-Hyperlnit in every environment.
that default initializations frequently fail to learn while Bias- Bias-Hyperlnit learns
We also compare hypernetworks with Bias-Hyperlnit to the standard architecture and Bias-Hyperlng HFI. We see that Bips-Hyperlnit matches or exceeds the performance of HFI, with signifi- cant improvement in Walker. Hypernetworks with Bias- Bipe-Hyyerlnit also significantly outperform the

---

Table 1: Comparison return on grid-world and MuJoCo tasks over five random seeds (mean = standard error). Entries in bold have insignificant difference from the highest- perffrsnp result.
Method
Grid-World
Cheetah-Dir
Walker
Ant-Dir
Humanoid
Standard
35.5 + 0.4
2104 + 87
1828 + 38
1167 + 16
1842 + 233
Kaiming
32.1 + 1.2
378 + 169
331 + 37
253 + 86
266 + 23
Norme
32.2 0.5
356 134
357 = 60
264 + 106
249 +
18
Orthogonal
34.9 = 0.5
1379 = 310
1687 = 98
1127 + 93
1126 + 76
HFI
36.8 0.2
2218 + 80
1618 130
1370 9
1323 + 57
36.1
2066 119
1748 + 57
1346
7
1048 +
Bias- Hlas-Hyperlnit
36.7 + 0.2
2300 + 32
1994 + 67
1328 + 23
1678 + 162
standard architecture on grid-world and Ant-Dir. In fact, Bias-Hyperlnit not significantly out- performed by any other method. However, the standard architecture, HFI, and Bias- Bias-Hyperlnit all achieve near optimal performance, motivating an evaluation on Mcta-World on which the standard architecture has greater room for improvement [5].
5.2 ML1 and ML10
Here evaluate the more challenging Meta-World World MLI and ML10 benchmarks [1]. ML1 and ML10 have one and ten non-parametric training tasks respectively (e.g. pushing a ball or opening window). ML10 additionally has five distinet test tasks. Within each task, there exists parametric variation, such variation the goal location. Note that we only test on the Pick-Place task from ML1, since VariBAD already achieves 100% success rate on all other tasks [5].
Table 2: Comparison meta-test success percentage on the Pick-Place MI.1 task (ten seeds) and ML10 (three seeds).
Pick-Place VariBAD
ML.10
Method
VariBAD
RL2
Standard
4.4 2.4'
10.2
3.0
7.2 = 5.0
Norme
5.5 4.8
FiLM
Bias-lyperlnit
34.2 + 15.9
HFI
25.5 + 14.5
28.4 - 6.0
7.1 + 2.4
Hypernetwork
Bias- Hay--Hyperlntt
42.9 + 16.3
23.9
6.2
14.2 + 7.2
In Table 2 we see significant improvement from hypernetworks with Bias-Hyperlnit over the stan- dard architecture, as well the efficacy of Bias-Hyperlnit FiLM. On Pick-Place, Bias-Hyperllii outperforms the standard architecture with a 9-fold increase in test success percentage. Additionally, Bias- Haperlyperl improves the FiLM architecture and exceeds the performance of HFL. On ML10, hy- pernetworks with both Byperlit and HFI yield a 2-fold increase in test success percentage compared the standard architecture. Finally, we evaluated Bipe-Hyperlnit when applied RI.2 on MI.10, finding 2-fold increase over both HFT and the standard architecture. Taken together, these results show a clear improvement from the application of hypernetworks with Bias-Hyperlnitt regardless of the baseline method they are applied to.
'Zintgraf et al. [5] report success percentage of 29% for Pick-Place however, we were not able to replicate this result.

---

Cheetah-Dir
Walker
2400
2000
MM
XL
XL
Retum
S
15
XXL
1700
M
1600
X5
XS
1s
1500
XS
XXL
1600
1400
Standard
Hypemet
10
10a
10
10a
10a
10
paremeters
perameters
Figure 3: Performance of standard and hypernetwork models range of base policy architecture sizes on Cheetah-Din Dir and Walker. Architectures are presented in supplementary materials.
5.3 Network Size Comparison
Because hypernetworks learn mapping from task embedding to the parameters of a base network, they require significantly more parameters in the entire model than a standard architecture with the equivalent base network. For fair comparison, network sizes and total number of parameters in the model on the Cheetah-Dir and Walker tasks (Figure 3). We find that hypernetworks consistently equal or outperform the standard architectures with the equivalent base network size. We also find that hypernetworks likewise outperform standard architectures for equivalent number of parameters in the entire model, i.e. for given value the x-axis, when the total number of parameters in the model is sufficiently large.
evaluate return over both range of base
Limitations
While our proposed methods are general, cannot guarantee improvement for all meta-RL methods. To mitigate this limitation, we build on top of VariBAD, which is state of the art, and additionally evaluate our method applied RL2 on ML10. Furthermore, as in any empirical study, there is no guarantee that our results hold on real robots or other meta-RL benchmarks. However, we have tested on seven standard meta-RL. environments in total, including Meta- Woold, which was proposed specifically for addressing robotics. As much as is possible from simulated meta-RL. experiments, these results give us confidence in a significant improvement over previous methods.
Conclusion
We used hypernetoorss to improve state-of-the-art method in meta-RL, evaluating over range of benchmarks. In doing so, we demonstrated that hypernetworks are promising path forward for meta-RL. Moreover, showed that the initialization of the hypernetwork crucial, default initialization methods fail. To overcome this difficulty, we presented two novel initialization meth- ods: Bias-Iyperllit and Weight-Hyperllii. Bias-Hyperlnit matched exceeded the performance of existing methods from the supervised learning setting, while also being simpler and more general applying arbitrary base network initializations, base network architectures, and also improv- ing FiL.M. Using Bias-Hyperlnit, we showed that hypernetwork performance improves substantially over standard architectures. Finally, we demonstrated that hypernetworks outperform the standard architecture for equivalently sized base policies, and outperform it at any size given sufficiently many parameters in the entire model. This paper additionally opens the path for future research extending meta-SL methods using hypernetworks [18, 22] and multi-task RL methods with separate parameters for each task [30, 31, 32] meta-RL.

---

Acknowledgments
We would like to thank Luisa Zintgraf for her help with the VariBAD code-base along with general advice and discussion. Jacob Beck is supported by the Oxford- Oxford-Google DeepMind Doctoral Schol- arship. Matthew Jackson is supported by the UK EPSRC CDT in Autonomous Intelligent Machines and Systems, with funding from AWS in collaboration with the Oxford -Snggpppe HMC Initiative. Risto Vuorio is supported by EPSRC Doctoral Training Partnership Scholarship and Department of Computer Science Scholarship.
References
[1] Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRI, 2020. (cited on 1, 2, 4, 6, 7)
[2] D. Ha, A. Dai, and Q. V. Le. Hypernetworks. In International Conference on Learning Repre- sentation (ICLR), 2017. (cited on p. 3,
[3] O. Chang, L. Flokas, and Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2020. (cited on p. 1, 3, 4, 6)
[4] K. He, X. Zhang. S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages
2015. (cited on p. 2, 4)
[5] L. Zintgraf, S. Schulze, C. Lu, L. Feng, M. Igl. K. Shiarlis, Y. Gal, K. Hofmann, and S. White- son. Varibad: Variational bayes-adaptive deep rl via meta-learning Journal of Machine Learn- ing Research, 22(289):1 -3,, 2021. (cited on p. 2, 3, 6,
[6] E. Todorov, T. Erez, and Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033. IEEE, 2012. (cited on p. 2, 6)
[7] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017. (cited on p. 2, 6)
[8] R. Vuorio, S. Sun, H. Hu, and Lim. Multimodal model-agnostic meta-learning via task- aware modulation. NeurIPS, 2019. (cited on p. 2)
[9] Z. Li, F. Zhou, F. Chen, and H. Li. Meta-sgd: Learning to learn quickly for few shot learning. arxiv, 2017. (cited on p. 2)
[10] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-learner. ICLR, 2018. (cited on p. 2, 3)
[11] Y. Duan, Schulman, X. Chen, P. L. Bartlett, Sutskever, and P. Abbeel. RIS Fast reinforcement learning via slow reinforcement learning. arXiv, 2016. (cited p. 2, 3)
[12] E. Najarro and S. Risi. Meta-learning through hebbian plasticity in random networks. NeurIPS, 2020. (cited on p. 2, 6)
[13] T. Miconi, K. Stanley, and J. Clune. Differentiable plasticity: training plastic neural net- works with backpropagation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3559-3568, Stock- holmsmässan, Stockholm Sweden, 10-15 Jul 2018, PMLR. (cited on p. 2)
[14] E. Sarafian, S. Keynan, and S. Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In Proceedings of the 38th International Conference on Machine Learn- ing. volume 139 of Proceedings of Machine Learning Research, pages 9301-9312. PMLR, 18-24 Jul 2021. (cited on p. 2)
[15] M. Peng, B. Zhu, and J. Jiao. Linear representation meta-reinfforeemen learning for instant adaptation. arxiv, 2021. (cited on p. 2)

---

[16] Z. Xian, S. Lal, H.-Y. Tung, E. A. Platanios, and K. Fragkiadaki. Hyperdynamics: Meta- learning object and agent dynamics with hypernetworks. In International Conference on Learning Representations, 2021. (cited on p)
[17] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Multi-task reinforcement learning without interference. NewrIPS, 2019. (cited on
[18] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019. (cited on 3, 8)
[19] T. Munkhdalai and H. Yu. Meta networks. ICML, 2017. (cited on p. 3)
[20] M. Przewiezlikowski, P. Przybysz, J. Tabor, M. Zieba, and P. Spurek. Hypermaml: Few-shot adaptation of deep models with hypernetworks, 2022, (cited on p.
[21] D. Zhao, S. Kobayashi, Sacramento, and J. von Oswald. Meta-learning via hypernetworks. In MeurlPS Workshop Meta Meta-Learning, 2020. (cited on 3)
[22] S. Ravi and H. Larochelle. Optimization as model for few-shot learning. In ICLR, 2017.
(cited on p. 3, 8)
[23] K. Li and J. Malik. Learning to optimize. ICLR, 2017. (cited on p. 3)
[24] D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2013. (cited on p. 3)
[25] A. M. Saxe, J. L. Mcclelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural network. In International Conference on Learning Represen- lations, 2014. (cited on 4)
[26] P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov. Openai baselines, 2017. (cited on p. 4)
[27] E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. C. Courville, Film: Visual reasoning with a general conditioning layer. AAAI, 2018. (cited on p. 6)
[28] Humplik, A. Galashov, L. Hasenclever, A. Ortega, W. Teh, and N. Heess. Meta rein- forcement learning as task inference. arXiv, 2019. (cited on 6)
[29] K. Rakelly, A. Zhou, C. Finn, S. Levine, and D. Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. International conference on ma- chine learning, pages 5331-5340. PMLR, 2019. (cited p. 6)
[30] Y. Teh, V. Bapst, W. Czamecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. NeurlPS, 2017. (cited on p. 8)
[31] E. Parisotto, J. Ba, and R. Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforce- ment learning, ICLR, 2016. (cited on p. 8)
[32] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task leamn ing. CVPR, 2016. (cited on p. 8)
10


Reviews:
[{"review_id": "DnBz8LHMB6s", "reviewer": "Reviewer_jmaN", "summary": "Meta-RL is an important direction to apply RL to real-world applications which require an ability to adapt to new tasks/environments. This paper tackles meta-RL using hypernetworks that produces the parameters of main agent (i.e., another neural networks). First, the authors show that performance can be very sensitive to initialization of hypernetworks and propose two initialization schemes. The first one is not sharing parameters between tasks (i.e., no bias) and independently initializing parameter per each task and the second one is only utilizing the shared one. The authors evaluate the proposed methods on several meta-rl benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths\n\n1. Interesting research direction: the idea of utilizing hypernetwork for meta-RL sounds interesting (but I also expect that it also has some limitations; see `Weaknesses`)\n\n* Weaknesses\n\n1. writing: I may miss something but the learning components for hyper-network are not clear. At the beginning, I thought that this framework is training NNs (hypernetworks parameterized by $\\theta$) which generates parameters ($\\phi$) of main agents. However, from Section 4.2, $\\theta$ is gone and parameters of hypernetwork ($W,b$) become the parameters of main agent ($\\phi$). Overall, current description is bit confusing and hard to understand.\n\n2. soundness: several components are not convincing. First, the authors mentioned that task id is not given in meta-rl but they assume that task id (one-hot vector) is given as inputs to hypernetworks. Also, the proposed initializations just look like independent initialization or one shared parameters. Because of that, it is bit hard to say that the proposed method is convincing and novel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'd like to recommend \"weak reject\" due to the weaknesses mentioned above.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "* As mentioned in Weaknesses, it would be nice if the authors could clarify the confusing parts (e.g., structure of hypernetworks, problem formulation and so on).\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "7zCrbdMJj7R", "reviewer": "Reviewer_R1kq", "summary": "The paper has two contributions: 1. it demonstrates the use of hypernetworks for meta reinforcement learning and shows that such approach outperforms VariBAD; 2. it proposes a novel initialization scheme for hypernetworks which makes training possible.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is clearly written and easy to follow. Its main downside is that it is unclear whether the authors want to make a contribution to meta-RL or to research on hypernetworks. In its current form, it doesn't present enough experimental evidence to be considered a significant contribution to either of the two fields.\n\nSpecifically, given that the main novelty is the proposed initialization scheme for hypernetworks, I would expect to see supervised learning experiments since those are the benchmarks that the baselines were evaluated on (see e.g. the HFI paper).\n\nRegarding meta-RL, I would expect to see comparisons to more baselines than just VariBAD. In particular, you report 19% meta-test success rate for ML10. Looking at the meta-world paper, they report 35.8% meta-test success rate for ML10. You report 5.6% for VariBAD so why is it considered the state-of-the-art?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Paper needs more thorough experiments evaluating the proposed methods.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See the strengths&weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "1DslL3UGi4", "reviewer": "Reviewer_7NSX", "summary": "This paper proposed using hypernetwork in Meta Reinforcement Learning problem. The authors stated that initialization is crucial for the hypernetwork performance. Thus, they proposed and tested two initialization methods: Weight-HyperInit and Bias-HyperInit. In this work, the author run experiment on 2D navigation, MuJoCo, ML1 and ML10 benchmarks in Meta RL setting. Using VariBAD algorithm as baselines, they showed the effectiveness of their method on those baselines. They also compared the performance of using hypernetwork and VariBAD baseline with the equivalent base network size. Also, they pointed out that this method still has some limitations and needs to be tested on real robots.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. Well stated motivation in using hypernetwork for meta reinforcement learning.\n2. Good performance in control problems.\n3. Clear analysis in the scalability of the hyper-network.\n\nWeakness\n1. The meta-RL problems are variants between different RL task. Evaluating in meta-RL where each tasks differ in the goal context would make this paper stronger.\n2. Specifying how many tasks are used for meta-training stage and meta-test stage will make this paper stronger.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak-accept of this paper. The proposed method is efficient and the paper is well written. The experiment showed the effectiveness of the proposed method although some of the details might be missed.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. In section 5.2, the variance of Bias-HyperInit is big. Does that mean using hypernetwork is still sensitive to initialization in Pick-Place tasks?\n2. Section 5.3, the author mentioned comparing performance for equivalent number of parameters in the entire model. But I could not find the related results.\n3. The proposed hypernetwork architecture is quite straightforward. Does that mean a near-optimal policy for each meta-RL problems in the experiment can be considered of having a linear transform in the policy parameter space?\n"}}]

Author rebuttal:

Decision: Accept (poster)