Recurrent Hypernetworks are Surprisingly Strong in Meta-RL
Authors: Jacob Beck
Risto Vuorio
Zheng Xiong
Shimon Whiteson
Venue: NeurIPS 2023 poster (2023)
Keywords: meta-RL
RL
reinforcement learning
memory
rnn
recurrent
hypernetwork
few-shot
in-context
TL;DR: Recurrent neural networks trained end-to-end outperform more specialized and complicated meta-RL methods, but only if hypernetworks are used.

Abstract: Deep reinforcement learning (RL) is notoriously impractical to deploy due to sample inefficiency. Meta-RL directly addresses this sample inefficiency by learning to perform few-shot learning when a distribution of related tasks is available for meta-training. While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shelf sequential model, such as a recurrent network, is a surprisingly strong baseline. However, such claims have been controversial due to limited supporting evidence, particularly in the face of prior work establishing precisely the opposite. In this paper, we conduct an empirical investigation. While we likewise find that a recurrent network can achieve strong performance, we demonstrate that the use of hypernetworks is crucial to maximizing their potential. Surprisingly, when combined with hypernetworks, the recurrent baselines that are far simpler than existing specialized methods actually achieve the strongest performance of all methods evaluated. We provide code at https://github.com/jacooba/hyper.

Recurrent Hypernetworks are Surprisingly Strong in Meta-RL
Jacob Beck
Risto Vuorio
Department of Computer Science University Oxford, United Kingdom jacob_ beck@alumni. brown. edu
Department of Computer Science University of Oxford, United Kingdom risto. vuori soorlekeelle ox ac uk
Zheng Xiong
Shimon Whiteson
Department of Computer Science University of Oxford, United Kingdom zheng. xiongêcs ox. ac. uk
Department Computer Science University of Oxford, United Kingdom shimon. whiteson@cs ox ac uk
Abstract
Deep reinforcement learning (RL) is notoriously impractical to deploy due to sam- ple inefficiency. Me directly addresses this sample inefficiency by learning to perform few-shot learning when distribution of related tasks is available for meta-training. While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shhil sequential model, such recurrent network, surprisingly strong baseline. However, such claims have been controversial due to limited supporting evidence, particularly the face of prior work establishing precisely the opposite. In this pa- per, we conduct an empirical investigation. While we likewise find that recurrent network can achieve strong performance, we demonstrate that the use of hypernet- works is crucial to maximizing their potential. Surprisingly, when combined with hypernetworks, the recurrent baselines that are far simpler than existing specialized methods actually achieve the strongest performance of all methods evaluated. We provide code at hhttps : //github. com/jaccobbahhype
Introduction
Meta-reiinfrreemett learning [Beck al. 2023 uses sample-inefficiet reinforcement learning (RL) to learn a sample-efficient reinforcement learning algorithm. The sample-efficient algorithm maps the data agent has gathered so far policy based that experience. To this end, any sequential model such as recurrent neural network (RNN), can be deployed to learn this mapping end-to-end [Duan al. 2016 Wang al. 2016). Such methods also called black-box (Beck et al. 2023
Alternatively, much prior work has focused on a category of lask-inference methods that are special- ized for meta-RL. A meta-RL algorithm learns to reinforcement learn
distribution of MDPs,
or tasks. By explicitly learning to infer the task, many methods have shown improved performance relative to the recurrent baseline Humplik et al. 2019). Zintgraf et al. 2020 Kamienny et al. 2020
Liu et al. 2021 Beck et al. 2022].
Recent work has shown the simpler recurrent methods to be competitive baseline relative to task-inference methods [Ni et al. 2022). However, such claims are contentious, as the supporting experiments compare only to one task-inference method designed for meta-RL, the experiments provide additional compute to the recurrent baseline, and the results still show similar or inferior performance to more complicated methods on the majority of difficult domains. In particular, they consider two toy domains and four challenging domains, with RNNs significantly outperformed on two of the four challenging domains, and superior to the single task-inference baseline only
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Walker
(b) Cheetah Dir
2000
2500
1750
1500
1250
1500
1000
1000
RNN
Model
RNN HN
RNN+HN
0 10000 20000
30000 40000 50000 60000 70000
Frames (k)
0
10000 20000 30000 40000
50000 60000
70000
Frames
Figure 1: In some environments, recurrent neural networks fail to learn meta-RL tasks, whereas recurrent hypernetworks achieve strong performance.
In this paper, we conduct a far more extensive empirical investigation with stronger and carefully designed baselines in meta-RI. specifically. In addition, we afford equal computation terms of number of samples for hyper-parameter tuning to all existing baselines. We present the key insight that the use of a hypernetwork architecture (Ha al.) 2017] is crucial maximizing the potential of recurrent networks. For illustration of the potential magnitude of improvement, see Figure I While the use hypernetwork with RNNs not novel idea, they have never been evaluated in meta-RL. beyond a single environment, let alone shown to outperform contemporary lask-inference methods [Beck et al. 2022]. We additionally provide preliminary evidence that the robust performance hypernetworks achieve such is in part due to how they condition on the current state and history. Finally, our results establish recurrent hypernetworks as an exceedingly strong method meta-RL. benchmarks that is also far simpler than alternatives, providing significant of use for practitioners in meta-RL.
2 Related Work
Recurrent Meta-RL. Many meta-RL methods structure the learned RL algorithm black box
using a neural network as general purpose sequence model [Duan et al., 2016 Wang et al. 2016
Mishra et al. 2018 Fortunato et al. 2019 Ritter al. 2021 Wang al. 2021 Ni al. 2022
While any sequence model can be used, often the model is structured as an RNN Duan et al. 2016
Wang al., 2016 Ni et al. 2022]. Such models [Duan et al. 2016 Wang al.
2016
are commonly
used as simple meta- Re baselines.
One study has shown RNNs to be competitive baseline in meta-RI. Ni et al. 2022 however, the scope of the study was broader than meta-RI. and the evidence specific to meta-RI. is inconclusive. First, the study evaluates only single specialized meta-RI. RI. method (Zintgraf et al. 2020). which was, but is not currently, state-of-the-an [Beck et al. [2022]. Second, the experiments use results or hyperparameters from the original papers, while affording extra computation to tune the RNNs on each benchmark, including dimensions that were not tuned for the other baselines. This computation includes tuning architecture choices, the context length, the RL algorithm used, and the inputs et al., 2022]. And third, the study does not show particularly strong performance of recurrent methods relative the chosen specialized baseline. On the MuJoCo domains, the recurrent baseline outperforms the specialized method on only one of these four domains, performs similarly on another, and is significantly outperformed on the other remaining two [Ni al. 2002]. In contrast, our work compares against four specialized baselines; affords equal computation to all methods, defaulting to hyper-pane parameerss that favor existing task- iss-inferencc methods for parameters that are not tuned; and still establishes recurrent hypernetworks the strongest method evaluated.
Task Inference Meta-RL.
In addition to recurrent meta-RL. methods, task-inference methods Hummlii et al. 2019 Zintgraf et al.) 2020 Kamienny et al. 2020 Liu et al.] 2021 Beck et al.
2022 and policy-gradiem methods Yoon al. [2018] et al. 2017 Vuorio al. 201 Zintgraf et al. 2019 constitute significant bulk of existing work. We exclude the latter methods from

---

(a) RNN
(b) RNN+HN
T(ap
{a(a)
f
RNN
RNN
Figure 2: The standard RNN policy (a) and the RNN policy with hypernetwork (b).
comparison since the estimation of a policy gradient in policy-gradien approaches requires more data
than in our benchmarks [Zintgraf et al.) 2019 Beck et al. [023]. Task inference methods are a strong baseline for our benchmark, but are generally more complicated than recurrent meta-RL. methods. For example, such methods typically add task inference objective (Humplik et al. [019]. and may also add a variational inference component (Zintgraf et al., [021]. or pre-training of embeddings with privileged information [Liu al., 2021]. In this paper, we ablate each of these components to create the strongest task-inference baselines possible. In the end, we find the more complicated task inference methods are still inferior to the recurrent baseline with hypernetworss.
Hypernetworks. hypernetwork (Ha al. 2017] is neural networks that produces the parameters
(weights and biases) for another neural network, called the base network. Hypernetworks have been
used supervised learning (SI.) (Ha et al. 2017, Chang et al. 2020 Meta-SI. [Rusu al. 2019
Munkhdalai la and Yu 2017 Przzeiezlikowski et al. 2022], and meta-RL. [Beck et al. 2022 Xian
al. [021 Peng et al. [022 Sarafian et al. 2021]. While these networks are complicated, and can fail to work out-of-the-box, simple initialization methods can be sufficient to enable stable learning
Beck et al., 2022 Chang et al. 2020]. meta-RL, only Beek et al. 2022 have investigated training hypernetwork end-to-end to arbitrarily modify the weights of policy. This study suggests that
hypernetworks are particularly useful in preventing interference between different tasks and enable greater returns the the number of parameters increases. However, their study shows task-inference method to be superior, and the recurrent hy hyprenetwork is evaluated only on a single task with results that are statistically insignificant. Recurrent hypernetworks have never been widely evaluated in meta-RI, let alone shown to outperform contemporary task-inference methods.
Methods
3.1 Problem Setting
task in RL is formalized as Markov Decision Processes (MDP), defined as a tuple of AS,A, R, P,7). At each time-step t, the agent inhabits state, 8, E S, which it can observe. The agent then performs an action 0; E A. The MDP subsequently transitions to the next state 8141 P(8 St, a.): AxAxS + 4. and the agent receives reward = R(s,, ax): SxA + IR upon entering $1+1- The agent acts to maximize the expected future discounted reward, R(T) = Ener I'T,, where denotes the agent's trajectory throughout episode in the MDP, and y E [0, 1) a discount factor. The agent takes actions sampled from learned policy, ((ass): S A + IR
Meta-RL algorithms learn an RL algorithm, fT),, that maps from the data, т, sampled from a single MDP, M ~ p(M), to policy parameters é. As in single RL task, T is a sequence up to time-step t forming a trajectory 7: E (S x X x R. Here, however, T may span multiple episodes within single MDP, since multiple episodes of interaction may be necessary produce reasonable policy. We use the same symbol, т, but refer to as meta-episode. The policy, T = fo(r)), parameterized by ø. if is itself parameterized by 0, which are referred to the meta- meta-parameters.
3

---

(a) TI Naive
(b) TI
n(a|s,
r(als, Φ)
π' (' g)
Nu n)
[4.e IB Iw
1B
9
RNN
RNN
Figure 3: Task-inference baselines. Naive task (a), additional multi-task pre- training (b).
The objective in meta-RL. RL is to find meta-parameters that maximize the sum of the returns in the meta-episode across a distribution of tasks (MDPs):
arg max BM
Ex R(( (l()))(())(())))))))))))))))))))
(1)
fe is referred to as the inner-loop, which produces o, in contrast to the outer-loop, which produces
Recurrent Methods
Recurrent methods are perhaps the simplest and most common meta-RL baseline. Recurrent methods use an RNN to encode history and train all meta-parameters end-to-end on Equation T These methods depicted in Figure 2 While neither recurrent networks (RNN below), nor the the combination of a hypernetoork with recurrent networks (RNN+HN below) a novel, the combination has never been widely evaluated in meta-RL [Beck et al. 2022). but we will show that the combination actually achieves the strongest results.
RNN. Our first recurrent baseline is the simplest and is equivalent to RL2 [Duan et al. 2016 and L2RL [Wang et al. 2016). In this case To (al & = fo(T)), where f is a recurrent network, क is feed-fo network, and and each distinct subsets of the meta-parameters, 0.
Our second recurrent model is the recurrent hypernet and achieves the strongest results. Here, the recurrent network produces the weights and biases for the policy directly: (a)s)). The state must be passed as input again to this policy for the feed-forward policy to condition on an input, and follow the initialization method for hypernetworks, Bias- Heas-n,,,,,,it, suggested by |Beck et al.) [2022]. In this initialization, the hypernetwork 's final linear layer is initialized with zero weight matrix and non-zero bias, so that the hypernetwork produces the same base-network parameters for any trajectory at the start of training.
RNN+HN.
3.3
Methods
Task-inference methods [Beck al. 2023 constitute the main category of meta-RL methods capable
of adaptation quickly as recurrett (i.c.. black-box) methods (Humplik et al. 2019 Zintgraf et al. 2020, Kamienny al. 2020 Liu al. 2021 Beck et al.) [2022). These methods train the inner-loop not end-to-end but rather to Identity the task, within the given task distribution. One perspective on these methods is that they attempt to shift the problem from the more difficult meta-RL setting to the easier multi-task setting by learning to explicitly infer the task BBec et al. 2003). Here we define relevant rask-inference methods used as baselines (Figures aand[4] For additional ablations, summary, and details on the method selection process, see the appendix.
TI Naive. The inner-loop of a meta-RI. method must take in the trajectory, т, and produce the policy parameters, é. In task inference methods, the inner-loop additionally produces an estimate

---

(a) TI++HN
(b) VI+HN
T (ala
o
To (als)
h
9
(n
RNN
RVN
Figure 4: Task-inference baselines. Task inference with additional parameter reuse and hypernetwork (a), an existing contemporary task-inference algorithm BBek et al. 2022 (b).
of the current task, ÊMs given a known task representation, cM- While it is possible to represent ø as Cы directly, i.c. pass CM to the policy, it is common to compute from an intermediate layer of the network that predicts CM which contains more information about the belief state (Humplik] et al. 2019 Zintgraf et al., 2020). It is also common use information bottleneck remove information not useful for task inference from the trajectory [Humplik et al. 2019 Zintgraf et al.) condition á the mean and variance of the bottleneck layer in order to explicitly condition the policy on task uncertainty for more efficient exploration (Figure 3). Putting these together, we can write task inference method follows:
2020). Following Zintgraf et al. (2020),
μ P" (*(RN((F)
0 P" (RNN(TT)
IB = =N(Z;
êM (z IB)
o = ReLU (P (P* L(H,)))
Jinfer (0) = EM E Hl|cM 2m
Jyrior (0) = EM [E (DLB
0, 1))Il-
where Pt, _e, PR., and Po are all linear projections. 1 represents a stop stop-gradient, pd pP1[(,, σ) is the matrix multiplication of pe with stop-gradient of a concatenation of (p. (). and 1) is the KL- divergence. Here, Jinfer + Jprior constitutes the evidence lower bound from Zintgraf et al. (2020) and is used to train IB, whereas all other parameters (Po and it (-(o)) are trained via Equation
TI. As presented, the TI Naive baseline may suffer from a known issue where the given task repre- sentation contains too little or too much information [Beck al. 2023]. When too little information is present, the policy may miss information crucial to the task. When too much information is present, the policy may be presented with the difficult problem of separating the useful information from irrelevant task features. Toward this end, it is possible to pre-train the task representation end-to-end using an additional policy [Humplik et al.) 2019 Kamienny et al. 2020 Liu al., [021]. Our TI baseline (Figure is the same as TI Naive, except that an additional multi-task policy, n', is pre-trained learn representation of the task, go(c)) Given linear projection, P9:
( IB)
Jmulti (0) EM [E, (B
Jinfer (0)
11g0 011
For fair comparison, training of the multi-task policy, n', occurs at the expense of training the meta-learned policy π,, with the total number of samples remaining constant. Instead of fully training the multi-task policy, experiment with different amounts of pre-training in the appendix, finding significant benefits already from less than 5% of total training allocation for the pre-training.
TI++HN. The TI++HN baseline is the same as TI, with three additions that we found to strengthen task inference (Figure 4. The first two additions (+++ are novel and are 1) initializing the parameters of the meta-policy, to that of the pre-trat pre-trained multi-task policy, n', to encourage transfer and 2)
5

---

(a) Grid- Grid-World World
(b) Grid Show
(c) Grid Dense
Medel
Nedel
Medel Make
Veiih
Figure 5:
Evaluation on grid-world benchmarks. RNN+HN and TI++HN improve return. RNN+HN achieves the greatest asymptotic return and sample efficiecy..
training of the task-inference (Jinfer) over trajectories from the initial multi-task training phase in addition to the meta-learning phase, since the former tend to be more informative and simply provide extra data. The third addition (HN) uses a hypernetwork to condition the policy on Φ [Beck et al. 2022]. We write this as T-(-(s) to show that ø represents the weights and biases of т, just as in the recurrent baseline. The output of the hypernetwokk is ø, and the input the hypernetwork is a the projection of j and a, b h( Re L P (1, o))). When using hypernetwork with the first two additions (++), the parameters the hypernetwork for the meta-learned policy are initialized the parameters of hypernetwork for the multi-task policy, instead of sharing policy parameters directly.
While rask-inference methods rely on known task representations, is also possible to design methods that infer the MDP more directly. This can be done by inferring transitions and rewards in full trajectories, since the transition function and reward function collectively define the MDP. In particular, such method, called VariBAD, is proposed by Zintgraf et al. (2020), and extended with the use of hypernetworks by [Beck et al. 2022]. Here, we call this method VI+HN, and it is contemporary task-inference method (Figure 4. Precisely, this model reconstructs full trajectories including future transitions for meta-cpisodes, To:T, instead of task embeddings, from the current trajectory:
VI+HN.
Jinfer (0) (Ex(E fo.r 1o.r112)
See the appendix for ablations with VI alone.
4 Experiments
In this section we compare recurrent hypernetworks (RNN+HN) to task inference baselines. We evaluate over three simple navigation domains (Zintgraf et al.) 2020 Humplik et al. 2019 Rakelly et al. 2019). designed to test learning of exploration and memory, in addition four more difficult tasks using MuJoCo [Todorov et al. [012]. and one task testing long-term memory from visual observations in MineCraft [Beck et al. 2020]. Results show meta-episode return, optimized over five learning rates and averaged over three seeds (four in MineCraft), with a 68% confidence interval using bootstrapping. Additional details and results Meta-World Yuu et al. 2020) available in the appendix. Our experiments demonstrate that while our task inference methods are strong baselines, RNN+HN is able outperform them and achieve the highest returns.
4.1 Grid-Worlds
Navigation tasks are a common benchmark in meta-RL [Zintgraf et al. 2020 Humplik et al. 2019 Rakelly et al. [019]. Here evaluate on the grid-world variant from Zintgraf' et al. [2020], in addition to two of our own variants. The first environment, Frrd-World, consists of a five by five grid with goal location one of the grid cells. The agent starts the bottom left corner of the grid, and then must navigate to goal location, which is held constant throughout the meta- meta-episode. (Details are in the appendix.) This environment is useful for testing how well meta-learning algorithm learns to efficiently explore in the gridworld as it searches for the goal. Additionally, our Grid-Worlls Show environment was designed to be relatively harder for end-to-end methods, order to provide challenge for our proposed method. In this environment, the goal position is visible to the agent

---

(a) Walker
(b) Cheetah Vel
2000
-50
-100
-150
Retum
200
1000
Model
220
Model
T1 Nalve
VI+HN
300
Vi HN
fa.Nalve
T
330
+I+ RNN HN
400
10000
20000
30000 40000 s0000
60000
70000
0
10000
20000
30000
40000 50000 60000 70000 Frames (k)
Frames (K)
(c) Cheetah-Dir
(d) Ant-Dir
2500
1500
1000
1500
Retur
1000
Model
VI+HN
250
Na.vvvv
T
TI++HN
RNN-HHN
10000 20000
30000 40000 Frames lk]
50000
60000
70000
10000
20000
30000
50000 60000
70000
Figure Models evaluated on MuJoCo benchmark. RNN+HN matches returns on Walker and Cheetah- Vel, and exceeds returns on Cheetah-DDi and Ant-Dir.
at the first timestep of each episode. Task inference methods will directly encourage the storage of this information in memory, whereas the end-to-end recurrent methods must learn to store this information through its effect on the policy. In contrast, our Grid-Worlds Dense environment provides dense rewards and may be casier for end-to-end methods. this environment, the agent receives and observes reward equal to the Manhattan distance the goal location. Instead of inferring the task explicitly, the agent can simply move up or to the right until the reward stops increasing.
Surprisingly, on all three gord-wsrlds, RNN+HN achieves both the greatest asymptotic return and greatest sample efficiency (Figure The recurrent hypernetwork achieves the fastest learning on Gridworld Show, despite the environment being specifically designed to be harder for end-to-end methods. TI++HN dominates other task-inference baselines on these grid-worlds, suggesting that it is relatively strong task-inference method. Collectively, these grid-worlds demonstrate that end-to-end learning with hypernetworks can learn to store the task memory and explore optimally with this information directly from return, just as well as task-inference methods.
4.2 MuJoCo
Here we evaluate baselines more challenging domains. We evaluate on all four MuloCo variants proposed by Zinngraff et al. 22020), which known to be a common and more challenging meta- RL benchmark involving distributions of tasks requiring legged locomotion [Zintgraf al. 2020 Humplik et al. 2019, Rakelly et al. 2019 Beck al. 2023]. Ant-Dir and Cheetah- Dir both involve non-parametric task variation, whereas Cheetah- Vel and Walker include parametric variation of the target velocity and physics coefficients respectively. For environment details, see the appendix. We expect Walker in particular to be difficult for end-to-end methods since it has the largest space of tasks with the dynamics defined by 65 different parameters. Assuming the values of these parameters important for the optimal policy, task inference methods may learn the optimal policy faster.
Cheetah-Dir and Ant-Dir, RNN+HN achieves greater returns than all other baselines by wide margin (Figure 6). On Cheetah- Vel, all methods achieve fairly similar results, with RNN+HN

---

Model VI+HI HMN
20000
30000
40000
50000
50000
Frames
Figure 7: RNN+HN outperforms VI+HN on MC-LS (MineCraft) environment.
still achieving the greatest asymptotic return by a small margin. As expected, RNN+HN does outperform task inference on Walker. For Walker, only TI outperforms RNN+HN in terms of efficiency, and only TI Naive outperforms RNN+HN in terms of asymptotic return; however, the effect size is small and both TI and TI Naive have among the worst performance on Cheetah-Din and the grid-worlds. Still, RNN+HN achieves similar performance on Walker, which is notable in a high dimensional task space. And, RNN+HN achieves greater returns overall.
4.3 MineCraft
We additionally evaluate on the MC-LS environment from Beck et al. 2202)) designed to test long- term memory from visual observations MineCraft. Here, the agent navigates through series of 16 rooms. In each room, the agent navigates left or right around column, depending column is made of diamond or iron. Discrete actions allow for finite set of observations. Correct behavior can be computed from the observation and receives reward of 0.1. At the end, the agent moves right or left depending on a signal (red or green) that defines the task and is shown before the first room. Correct and incorrect behavior receives reward of 4 and -3, respectively. We allow the agent to adapt over two consecutive episodes, forming a single meta-episode.
whether the
On MC-LS we compare RNN+HN to VI+HN alone, given limited compute budget and since VI+HN an established contemporary task-inference baseline Beck et al. 2022]. Additionally, we add an extra seed (four in total) and linear learning rate decay due to high variance in the environment. In Figure 7 we see that RNN+HN significantly outperforms VI+HN. While VI+HN learns to navigate through all rooms, it does not reliably learn the correct long-term memory behavior. In contrast, RNN+HN is able to adapt reliably within two episodes, and one seed even learns to adapt reliably within single episode. While further work is needed to learn the optimal policy, these experiments demonstrate that RNN+HN outperforms VI+HN, even on more challenging domains.
Discussion
Here we investigate why the recurrent hypernetworks have such robust performance on meta-RL benchmarks. First, observe that the state processing differs between the RNN and RNN+HN
(a) RNN+S
(b) RNN+HN
<als, Φ)
(als)
RNN
RNN
Figure 8: The RNN+S policy (a) and the RNN policy with hypernetwork (b).

---

(a) Grid
(b) Grid Show
(c) Grid Dense
Modal
Nocel
Medel
Figure 9: RNN+HN matches or exceeds RNN+S and RNN, but RNN+S is also strong on grid-worlds
(a) Walker
(b) Cheetah Vel
2000
-50
1500
-100
1250
-150
Reeurn
220
1000
750
500
Model
250
RNN
RMN 5
RNN-HM
400
RNN HN
10000
20000
30000 40000
50000 60000
70000
0
10000
20000
30000
40000 50000 60000
70000
Frames (k)
Fremes
(c) Cheetah Dir
(d) Ant-Dir
2500
1400
1200
1000
1500
Return
500
Model
200
Model
RMN
RNN+
RMN HN
10000 20000
30000 40000 Frames (K)
50000
60000
70000
60000
70000
Frames (K)
Figure 10: RNN+HN matches or exceeds RNN+S on MuJoCo tasks. RNN alone a weak baseline.
baselines. In particular, RNN conditions the current state only through its dependence d whereas hypernetworks pass in the state again to the policy. Thus, we investigate whether the difference in inputs alone could be the cause of the improvement in performance. To this end, we introduce a new ablation to test the effect of just passing in the state again. Details are below. Second, we inspect how sensitivity latent variables encoding the trajectory affects performance.
Hypernetworks condition on the current state both through d which contains information about trajectory, including the current state, and by directly conditioning on the state. Since the hypernetwork conditions on state twice, we test to see the effect of conditioning on the state twice without hypernetworks. We call this ablation RNN+S, which we write as πt (als, Φ) (Figure 8) In an empirical evaluation, we see that while RNN+S does perform favorably relative to RNN alone, RNN+HN still outperforms RNN+S (Figures 9 and TO]. particular, RNN+HN achieves similar returns to RNN+S on Ant-Dir and Cheetah- Vel, and outperforms RNN+S on all other environments, in terms asymptotic return and sample efficiency. Taken together, we see that the advantage of RNN+HN comes both from the ability to re-condition on state directly and from the hypernetwork architecture. These results confirm that to achieve the strongest performance, re-conditioning on state directly is not sufficient, and that the hypernetwork architecture itself is still critical.
RNN+S.

---

(a) Walker Return
(b) Walker Latent Gradient Norm
1750
Model
20
RNN
1500
RNN
1250
Model
15
-HN, Kalming
Retur
1000
RNN
Norm
RNI--5
RNN
Kaiming
5000
10000
15000
20000
25000
30000
Frarees
Updates
Figure 11: Returns (a) and Gradient norms on Walker (b). The RNN method must increase this norm to condition on state, whereas others do not. Lower gradient norms seem important to performance.
Latent Gradients. We also investigate how the hypernetworks condition on the trajectory. In particular, we investigate the sensitivity of the output of the network to an intermediate latent representation of the trajectory. For this purpose, we chose to measure the gradient norm of the first hidden layer of the hypernetwork on the Walker environment. We perform this investigation both with the initialization method designed for hypernetworks that we used throughout our experiments, Bias-Hyperlnit (Beck et al. 2022], and with Kaiming initialization (He et al., 2015). not designed for ignores trajectories at the start of training [Beck et al. 2022]. First, we confirm the finding of Beck et al. [022] that Bias-Hyperlnit is crucial for performance (Figure I. Second, we see the two models that performs worst, RNN and RNN+HN Kaiming, also have the greatest norm. Moreover, we find that both RNN+HN and RNN+S start with a low gradient norm and then further decrease this norm throughout training, whereas the RNN model increases this norm. We hypothesize that a low norm, i.c., low sensitivity to the latent variable. is crucial for stable training and that the RNN model increases this norm to remain sensitive to the state, since the state is only encoded in the latent for this model.
hypernetworks. We add Kaiming initialization, since Bias-Hyperie
Limitations
As an empirical study of DLt we cannot guarantee that recurrent hypernetwrks will improve over every baseline nor on every environment. However, we mitigate this issue by comparing to many baselines and performing many ablations. In particular, we compare to a contemporary task- inference method (VI+HN), design our own baseline which we show to be stronger than others on all grid-worlds (TI++HN), and also include standard methods (TI and TI Naive), in addition further ablations in the appendix. In as much as empirical study can, we believe our study demonstrates significant improvement of the RNN+HN method over existing baselines.
Conclusion
In this paper, we establish recurrent hypernetworks as surprisingly strong method in meta-RL. While much effort has gone into designing specialized task-inference methods for meta-RI, we present the surprising result that the simpler recurrent methods can be easily adapted to outperform the task-inference methods. By combining recurrent methods with the hypernetwork architecture, we achieve new strong baseline in meta-RL that is both robust and easy to implement. In comparison to existing evidence, we provide much stronger empirical results, afford equivalent computation for tuning to all baselines, and establish recurrent hypernetworks show that passing the state variable to the policy is crucial component of this method. Finally, we presented gradient analysis suggesting lower latent gradient norms to play an important role in the performance of meta-RL methods. Since the gradient analysis is preliminary and investigates state and latent variables in isolation, future work could investigate the interaction between these variables. Future work could also analyze the interaction between hypernetworks and other sequence models, such as transformers. We hope these insights, along with a simple and robust method, open the way for the broader use of sample-efficient learning in meta-RL RL and beyond.
strong method. We additionally
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Luisa Zintgraf for her help with the VariBAD code-base along with general advice and discussion. Jacob Beck is supported by the Oxford-Google DeepMind Doctoral Scholar- ship. Risto Vuorio supported by EPSRC Doctoral Training Partnership Scholarship, Department of Computer Science Scholarship, and Scatcherd European Scholarship. Zheng Xiong is supported by UK EPSRC CDT in Autonomous Intelligent Machines and Systems (grant number EP/S024050/1) and AWS.
References
Jacob Beck, Kamil Ciosek, Sam Devlin, Sebastian Tschiatschek, Cheng Zhang, and Katja Hofmann. Amrl: Aggregated memory for reinforcement learning. In International Conference on Learning Representations, 2020. URL URL/ittpp://ooperrriisi
wtw.:/rrwwrereeeeeoruuuuu..aaaee www.:eeenonnmee.m http//orrmmdddBB117BBEEEEE Bk17bREtDr'
Jacob Beek, Matthew Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta- reinforcement learning. CoRL, 2022.
Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf. Chelsea Finn, and Shimon Whiteson. A survey of metuureeiforreement learning. arXiv preprint arXiv:2301.08028 2023.
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2020.
Yan Duan, John Schulman, Xi Chen, Peter L.. Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv, 2016.
Chelsea Finn, Abbeel, and Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICMI., 2017.
Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomênech Badia, Gavin Buttimore. Charlie Deck, Joel Z. Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. NeurlPS, 2019.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In International Conference on Learning Representation (ICLR), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Jan Humplik, Alexandre Galashov, Leonard Hasenelever, Pedro Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv, 2019.
Pierre-Alexandre Kamienny, Matteo Pirotta, Alessandro Lazaric, Thibault Lavril, Nicolas Usunier, and Ludovie Denoyer. Learning adaptive exploration strategies in dynamic environments through informed policy regularization. arXiv preprint arXiv:2005.00 02934,, 2020.
Evan Z. Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for rentrreeinoore learning without sacrifices. In International Conference Machine Learning, pages 6925-6935 PMLR, 2021.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Picter Abbeel. A simple neural attentive
meta-learner. In International Conference on Learning Representations, 2018. URL https
Tsendsuren Munkhdalai and Hong Yu. Meta networks. ICML, 2017.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be strong baseline for many POMDPs. In Proceedings of the 39th International Conference Machine Learning. PMLR, 2022.
Matt Peng, Banghua Zhu, and Jiantao Jiao. Linear representation meta-reinforceemen learning for instant adaptation. arxiv, 2021.

---

M. Przewiezlikowski, P. Przybysz, J. Tabor, M. Zieba, and P. Spurek. Hypermaml: Few-shot adaptation of deep models with hypemetworks.. 2022.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforceemet learning via probabilistic context variables. In International conference machine learning, pages 5331-5340, PMLR, 2019.
Samuel Ritter, Ryan Faulkner, Laurent Sartran, Adam Santoro, Matthew Botvinick, and David Raposo. Rapid task-solving in novel environments. In International Conference Learning Representations, 2021. URL http:p////ppeereeee...eeee wttp://wwwi..eee/tommidedeeeeee.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9301-9312. PMLR, 18-24 Jul 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference intelligent robots and systems, pages 5026-5033 IEEE, 2012.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal model-agnostic meta- learning via task-aware modulation. NeurIPS, 2019.
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. arXiv, abs//611.05766 1.05763, 2016.
Jane X. Wang, Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, H. Francis Song, Gavin Buttimore, David P. Reichert, Neil C. Rabinowitz, Loic Matthey, Demis Hassabis, Alexander Lerchner, and Matthew M. Botvinick. Alchemy: A structured task distribution for mete-reiinforcement learning. NeurlPS, 2021.
Zhou Xian, Shamit Lal, Hsiao-Y Yu Tung, Emmanouil Antonios Platanios, and Katerina Fragkiadaki. Hyperdynamics: Meta-learning object and agent dynamics with hypernetworks. In International Conference on Learning Representations, 2021.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-aggostic meta-learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. -esachBinchh, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 31. Curran Associates, Inc., 2018. URL httpstss:: http///oceeeinn..eeeee neurips.cc/paper/ cc/paper/ Paper. pdf
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL, 2020.
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep via meta-learning. In International Conference on Learning Representation (ICLR), 2020.
Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep via meta-learning. Journal of Machine Learning Research, 22(289):1 2021.
Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. ICLR, 2019.
12

---

8 Appendix
8.1 Additional Task-enference Ablations
In order to select lask-inference methods for ultimate comparison in the main body, we evaluated those methods, in addition further task-inference ablations, on the Grid-World and Walker environments, and then selected the best performing methods. From these results, we chose TI++HN (on Grid- World), and TI and Naive (on Walker). See Figure We additionally chose evaluate VI+HN in the main body, as it is prior strong method Beck et al. [2022). A table summarizing the components in all methods can be seen in Table [ The additional ablations are depicted in Figure [3] given below:
with details
VI.
This baseline, called VariBAD, proposed in Zintgraf al. [2020] and is the same as VI+HN in the main body, but without the hypernetwork.
TI++. This baseline the same as TI++HN in the main body, but without the hypernetwork. The two additions over TI (denoted "++") are re-use of the multi-task policy parameters as an initialization for the meta-RI. policy, and training of task inference over trajectories from pre-training.
TI+HN. This baseline is the same TI in the main body, but with the addition of a hypernetwork.
BI++HN. Here, the base-network for the task is inferred (denoted "BI"), instead of the task representation directly. This method is novel and is the same as Tl++HN in the main body, but task inference trained reconstruct the parameters for the base-network produced by the multi-task hypernet, &', rather than the task representation (cM or g):
S po' (= IB)
Jinfer (0) EM E 11 0'1
Note that while the performance BI++HN is similar to that of TI++HN on forld, TI++HN chosen representative baseline since it is more common existing methods and BI++HN requires significantly more compute.
(a) Grid-Worke
(b) Walker
2000
20
Model
Vi
VI+HN
10
TI Naive
TI Naive
T1
TI
Th+
Ti+-
TD+HH
Ti+HN The HN HHN
TI+
Bie HHN
500
1000
1500
2000
2500
3000
3300
4000
10000 20000
30000 40000
50000
60000
70000
Frames (4)
Frames
Figure 12: TN++HN was chosen baseline given results from Grid-World (a). TI and TI Naive were chosen as baselines given results from Walker (b).
13

---

(a) Tl+
(b) VI
T(a|s, 6)
π a(a 8,
Taa Φ)
77:21 vi MOTT
f
9
9
IB
RNN
RNN
(-10), (1119)
(c) TI+HN
(d) BI++HN
Ty (ala)
xy (ale)
R
h
(ale)
(ple)
RVN
RNN
Wea Ty
Figure 13: Additional Task-inference baselines.
Inference Training in Multi-Tase Phase and
Inference
Policy Condi- tions on State
Hypernetwork
Target
Parameter Reuse
RNN
None
No
No
N/A
RNN+S
None
Yes
No
N/A
RNN+HN
None
Yes
Yes
N/A
TI Naive
Given
Yes
No
N/A
TI
Learned
Yes
No
No
TI+
Learndd
Yes
No
Yes
TI+HN
Learned
Yes
Yes
No
TI+HN
Learned
Yes
Yes
Yes
VI
Transitions
Yes
No
N/A
VI+HN
Transitions
Yes
Yes
N/A
BI++NN
Base Net
Yes
Yes
Yes
Table I: Summary of the components each method
8.2 Hyperparameter Tuning
We tune each baseline over five learning rates for the policy. [3e-3, 1e-3, 3e-4, 1e-4, 3e-5], for three seeds each. We use learning rate for the task inference modules of 0.001, both since varying the learning rate seemed to have little effect (Figure 14p and to provide a fair comparison in terms of computation. Error bars report 68% confidence interval using bootstrapping.
Before being passed the hypernetwork or the policy, depending on the model, the task or g) are projected down to size 25, as are the concatenation of (H, ar.. This size was chosen to be in the range of the values used by Zintgraf' was chosen to be consistent. Additionally. 25 is sufficiently large for one-hot representation of all discrete task distributions used. However, on Ant-Dir, the projection size is 10. For the state embedding size (passed to the policy) we chose 256 and for the sizes of the MLP policy we chose 256, followed by 128. This corresponds to the "XL" model size Beck et al. 2022 which was optimal or near optimal where reported. However, for the comparison of RNN, RNN+S, and RNN+HN on Grid-World these results are reported with state embedding size for the policy tuned over [5, 25, 125, 625]. For the state embedding size passed to the trajectory encoder, we used 32 for all
(2020) across environments, but single value
14

---

(a) Grid Woold Pre-Train Time
(b) Grid- World LR
40
30
20
20
Model
TI++HN Oupdates
10
100updates
Model
300updates
THN
T1+ HHN 500updates
vaeLR1e-02
1000updates
TI++HN vael-Ree-0d
[1++HHl_veeLRBe
500
1000
1500
2000
3500
4000
500
1000
1500
2000
2500
3000
3500
4000
Fremes (k)
Figure 14: Training for 100 updates yielded optimal performance despite not fully training the multi-task policy, Multi+HN (a). Varying the learning rate for task inference module had little effect (b).
experiments, as the default value in Zintgraf et al. [0220] And for all RNNs, we use single GRU layer of size 256, as in Beck et al. 2022).
As discussed in the main body, pre-training of the multi-task policy occurs at the expense of the meta-RL policy training time, in order ensure the same number of samples for training each method. While selecting the time for pre-training does add a hyperparameter that must be selected, we tune this once on World and then transfer to all other environments, order to afford as little additional computation as possible to these baselines. Note that while this cutoff affords some extra computation task inference methods, the recurrent hypernetwork method (RNN+HN), which we show to be the strongest, does not require any such additional tuning.
On Grid-World, we find 100 PPO updates for training the multi-task policy to be optimal. In this environment, each update consists of 960 frames. Interestingly, while the multi-task policy (Multi+HN) requires approximately 500 updates to fully train, training for 100 yielded optimal performance for TI++HN. (We saw similar results for TI, not reported here, except that the random projection provided by 0 updates already gave similar performance to 100.) Note that 100 updates is equivalent to only 20% of the total frames for training the multi-task policy, and it is only 2.4% of the total frames for training the meta-RL policy. This suggests that very little amount of computation is required to get a good task representation, and delaying the start of meta-training significantly is generally not worthwhile. For MuloCo experiments in the main body, we use 563 updates, since that is the same percent of total frames (2.4%) for Walker and Ant-Dir, and (4.8% on Cheetah-Ve Vel and Cheetah-Dir), which transferred well on initial experiments.
For all other hyperparameters, we default to those in Beck et al. 2022
8.3 End-to-End and Task Inference Together
While it is possible to combine the end-to-end objective and task inference objective, prior work suggests that interference from competing objectives may be detrimental (Humplik et al. 2019). Moreover, separating the training objectives can be practically useful, the combined objectives may require tuning of the relative weighting of the two objectives. Still, we perform investigatory experiments on Grid-World. See Figure 15] We find that adding task-inference objectives the end- to-end meta-RL objective decreased return compared to the end-to-end objective alone (RNN+HN) for every method tested. The same effect can be seen regardless of the architecture (RNN+HN or RNN+S). Unsurprisingly, the performance of the combined end-to-ent moddt and task-inference methods falls in between each type of method used independently. Additionally, tuning the relative relative objective weighting had little effect. We experiment with giving 10% (BI10p, TI10p) and 50% (B150p) weighting to task inference methods, but still observing the same decrease in return. In all these experiments, we use the RNN architecture from the end-to-end RNN methods, meaning that we did not use an information bottleneck, nor the linear encoding layer. In other words, the output the RNN passed to the policy, é, was directly used as ğ or & and compared to the respective label g
15

---

(a) Grid- World Combined Objectives
(b) Tuning Combined Weighting
30
Model
20
RNNN-S
RINN HIN
20
RNN-HNN
RNN+S
HHN
RNNN-HM BB100p
RNN HIN+ Naive
10
RNN HI B150p -
RNN HN +8
Naive
-110pp Naiva
1000
1500
4000
500
1000
1500
2000
2500
3000
3500
4000
Fremes (k)
Figure 15: Adding task inference objectives in combination with the end-to-end meta-RI. objective decreases performance relative to end-to-end alone, and results in performance between the two types of methods (a). Varying the relative objective weighting had little effect (b).
8.4 Meta-World Experiments
We additionally evaluate on Meta-World World (ML10) Yu al. 2020
experiments and show RNN+HN
able to match the performance of the more complicated VI+HN method (Figure [6). SeradWorld is a meta-RL. benchmark for robotic manipulation. There are 10 non- non-parametric training tasks and distinct non-parametric test tasks, ranging from pushing ball to opening a window. Additionally, there is non-parametric variation, e.g. of the goal location, within each task. this environment, the variance of returns and the computation requirements for training are dramatically greater, so the evidence is not strong is any direction. However, we include the results here since ML10 is a standard benchmark Beck et al. [2023), and since the only experiment ever reported in meta- RL with RNN+HN prior to our paper is on ML Beck et al. [2022].
Contrary to our findings, Beck et al. 2022 demonstrated RNN+HN to underperform VI+HN on ML10. However, since that result was not statistically significant, calculated by Beck et al. [2022]. we conduct further experiments on ML10. First, we run more seeds for the VI+HN model in their experiment. (Their experiment is the same as ours, but we use latent encoding size of 25, instead of 10, for the projection of (n,, o7) to be consistent with the majority of our experiments, and we run for slightly fewer frames due to the intensive computation requirements.) Using different seeds from Beck et al. [2022] resulted in worse performance, indicating that the results truly were insignificant due to variance, Beck et al. 2022] calculate a mean return 23.9 for VI+HN from the returns: [12.48, 25.69, 33.61]. In contrast, we calculate the final mean return is 18.35 for VI+HN, with is an average over three seeds, with final returns: [32.80, 3.32, 18.93J. Comparing to the returns for RNN+HN from Beck et al. 2022). [11.31, 3.37, 27.77), is difficult to draw strong conclusion from this experiment.
Additionally, we run our own experiments (with latent size 25) and compare RNN+HN to VI+HN. Contrary to Beck et al. 22022) we find that the final return of RNN+HN is higher than VI+HN. RNN+HN has a final average test success percent of 17.22, compared to 13.87 for VI+HN. However, the variance is still too large to draw firm conclusions. Additionally, we find that the learning curves for RNN+HN and VI+HN overlap consistently throughout training, and the confidence interval for one
RNN+HN lies almost entirely within that of VI+HN. Note that while tuning of the seeds for the largest learning rate (3e-3) of RNN+HN encountered numerical instability and was ignored. However, le-4 was selected as the optimal learning rate, so this is unlikely to affect results, and if it did, would only increase the returns of RNN+HN.
While these results are not sufficient to conclusively show superiority on ML10, we do show superiority many other environments. Additionally, these results do show RNN+HN to match the performance of VI+HN. Matching existing task-inference baselines on MI.10 significant, since demonstrates that complicated lask-inference methods are not necessary to be competitive.
16

---

(a) Additional Seeds on Original MLI0 Experiment
(b) Our ML10 Results
0.35
Model
Model
0.30
SUcCO2e Rate (bos)
0.20
Success Raze
0.15
0.10
0.10
0.00
0.00
20000
40000
Frames
60000
80000
100000
10000 20000
30000
Frames (k)
40000 50000
60000
70000
Figure 16: Running three additional seeds of VI+HN on the ML10 experiment from Beck et al. 2022 (with a latent size 10) shows the result strong due to high variance. Note that the initial sharp decline in one seed can occur due to smoothing with valid padding on a binary reward (a). While also not a strong result, running
experiment ML10 (with latent size 25) shows that the RNN+HN is able to match the performance of the more complicated VI+HN method (b).
8.5 Gridworld Details
Frid-World consists of five by five grid with a goal location in one of the grid cells. The agent starts in the bottom left corner of the grid, and then has 15 steps in an episode to find the goal and remain there. The agent receives reward for each timestep at the goal and -0.1 reward for all other timesteps. The goal location is held constant for each meta-episode, which consists of regular episodes. In Grid-World Show, the goal position is visible to the agent at the first timestep of each episode. In Worl-World Dense, the agent receives and observes a reward equal to the Manhattan distance to the goal location.
8.6 MuJoCo Details
We evaluate on all four MuJoCo Todorov et al. [2012] environments used by Zintgraf al. (2020]. All environments require legged locomotion. All episodes terminate after 200 timesteps. Meta-episodes in Walker consist of two regular episodes, whereas meta-episodes in the rest consist of one regular episode.
In Cheetah-Dir, the agent controls robotic cheetah morphology by outputting a control torque for each of six joints on the morphology. The task is to run forward or backward with as large a velocity possible. The agent observes the position, angle, and velocity of each body part (17 dimensions in total) and is given a positive reward for its velocity in the direction given by the task and a negative reward for control costs (specifically, 5% of the magnitude of the action vector).
In Cheetah- Vel, the agent controls the same robot, but instead of the the positive reward component for running the selected direction, the agent receives negative reward component equal to the absolute difference to a target velocity, which is sampled uniformly from [0.3].
In And-Dir, the agent controls robotic ant morphology by outputting control torque for each of eight joints on the morphology. The task is run forward or backward, with as large a velocity as possible. The observations are 27 dimensional in this case, and the agent is given a positive reward for its velocity in the direction given by the task and a negative reward for control costs (specifically, 5% of the magnitude of the action vector). Additionally, the agent is given negative contact reward (.05% of the magnitude of the external forces on each joint) and a positive survival reward (1 per timestep). Episodes in this environment additionally terminate if the body falls outside of predefined height.
In Walker, the agent controls two-legged torso morphology and outputs control torque for each of six joints on the morphology. The observations are 17 dimensional for this environment. The tasks are defined as uniform samples of 65 different physics coefficients (e.g. body mass and friction) for the simulation. The agent given a positive reward for the forward velocity, a positive reward of one reward per timestep, and negative reward for the control costs (specifically, 0.1 of the magnitude
17

---

of the action vector). Episodes in this environment additionally terminate if the body falls outside of a predefined height or rotation range.
8.7 Compute
Experiments were run on four to eight machines simultaneously each with eight GPUs, ranging from NVIDIA GeForce GTX 1080Ti NVIDIA RTX A5000. Each model was tuned over five learning rates with three seeds per learning rate. Each experiment took between 4 hours for grid-worlds and 3 days for MuJoCo experiments. In total, there were approximately 360 experiments for MuJoCo and 270 for grid-worlds in the main results. With about 5 experiments on average per GPU, this constitutes roughly 5,400 GPU hours.
18


Reviews:
[{"review_id": "Ig8Bbd1Ub0", "reviewer": "Reviewer_YdBr", "summary": "This paper is an empirical investigation that tests the performance of different variants of three methods for meta-RL: RNN that provides task information implicitly from previous trajectories on the same MDP, TI that trains a task representation with VAE manner, and VI that is the baseline Varibad [1]. On mujoco environment, the paper finds that RNN with output being a hypernetwork that outputs the parameter for the policy works the best, and thus re-establish [2] RNN+hypernetwork as the state-of-the-art method for meta-RL. \n\n**References:**\n\n[1] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representation (ICLR), 2020.\n\n[2] Jacob Beck, Matthew Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta-reinforcement learning. In CoRL, 2022.", "questions": "Besides those listed in the weakness section, I have one question: why the performance of RNN+HN in Fig. 9a and Fig. 10a is different?\n\nBelow are my suggestions for this paper:\n\n1. Modify the introduction and method section of the paper such that it is more clear that this is an empirical study instead of proposing a novel method, and add contribution summary at the end of the introduction. Add a table that summarizes the methods tested in the paper.\n\n2. Add environment details of the mujoco environments in the appendix.\n\n3. Compare RNN+HN to more methods summarized in [1];\n\n4. Compare RNN+HN with VI+HN (and other baselines) on metaworld and other more challenging environment;\n\n5. Fix other minor issues mentioned in the weakness section.\n\n6. I strongly advice the authors to open source their code upon acceptance / next submission. The author could also consider adding license statement in the appendix.\n\n**References:**\n\n[1] J. Beck et al. A survey of meta-reinforcement learning. In arXiv, 2023.", "limitations": "There is an independent limitation section in the paper, which I think generally discusses an important limitation of the work. Though, as suggested in the weakness section, I think currently the limitation is not sufficiently mitigated. There is no negative societal impact discussed in the work, which I would encourage the readers to add; though the work is still far from application, automated control itself brings potential challenge, such as job loss, to the human society.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Many variants are tested in the environment. There are 12 variants tested in the paper, which gives a thorough analysis on the effect of using hypernetwork for each method; in addition, the effect of \"conditioning on the state twice\" is also considered, which makes the conclusion that hypernetwork is crucial more rigorous.\n\n2. The figures appended in the paper does a well job in conveying the architecture of each method.", "weaknesses": "**1. The experiment results are not convincing enough.** \n\na) The method proposed (line 184) in the paper, RNN+hypernetwork, is already proposed in prior work [1]. More specifically, RNN alone is equivalent to RL2 [3] as line 106 suggests, and RL2+hypernetwork is already tested in [1], which is inferior to Varibad [2]+hypernetwork. Thus, the key contribution of this paper is not proposing new method, but to overthrow the previous conclusion that Varibad+hypernetwork is better. However, this paper only tests gridworld and mujoco environments, and on both environments RNN+HN is only marginally better than VI+HN (which in [1] also has multiple methods that works similarly well, indicating that mujoco alone is not a strong benchmark). Most importantly, the metaworld (ML1 and ML10) environment, which is the decisive evidence that Varibad+hypernetwork is better than RL2+hypernetwork, is missing in this paper. With such result missing, it is hard for the readers to be convinced that the result is the other way around from that in [1].  \n\nb) The baselines, though with many variants, do not cover enough areas for the claim of the state-of-the-art. Reviewers of [1] have already pointed out that the state-of-the-art status of Varibad [2], even before the presence of [1], is questionable. Also, while it is true that the meta-RL method can be briefly summarized in policy gradient, implicit task representation and task inference as the paper suggests (line 61-63), in Table 2 of [4] there are many branches within each of the direction. For example, what if transformer instead of RNN is used [5]? Is policy gradient method necessarily worse than the other two branches? For a state-of-the-art method, those methods also need to be considered.\n\n**2. The delivery of the paper's idea can be improved.**\n\na)  There is no motivation stated about why hypernetwork is used. The paper only states that \"we present the key insight that the use of a hypernetwork architecture is crucial ...\" (line 36-37) and \"hypernetwork has never been widely evaluated in meta-RL\" (line 81-82). However, there is no intuitive explanation about why hypernetwork is useful for meta-RL and should be considered in the first place. While there is explanation in [1] about preventing degeneration of multi-tasking, the paper should be self-contained and inform the readers about why hypernetwork, the most important component in the finally proposed RNN+hypernetwork method, is considered for meta-RL.\n\nb) The method section can be modified to convey the idea more clearly. Currently, it is hard to identify which method is proposed (line 184) by the paper, and it is unclear for the first-time readers about why multiple methods are listed in the method section (and most of them is not proposed by this paper), instead of the usual paradigm where each key component is listed in a subsection. Since the methods are not novelly proposed by this paper, they can be put into experiment setup section (with more emphasis that this is an empirical study paper rather than conventional paper that proposes a new method, e.g., add a contribution summary at the end of introduction section); or, a summary could be added in the front of the method section, to tell the readers that all those methods listed below are tested in the experiment section, and in each section we discuss one type of method tested. A table summary would also be very helpful.\n\nc) There is no detail of the environment settings except gridworld. What is the cheetah-dir environment? What are the definitions of state, action and reward, and how are the MDPs in meta-learning differs? What reward can be considered expert-level performance? While they can be found in prior work, the paper should be self-contained and the settings should be attached in the appendix to help the reader form a better intuition about the environment.\n\n**3. Other minor problems:**\n\na) Some experiment results are missing, for example, RNN in Fig. 8b, 8c, 9b, 9c and 9d;\n\nb) Fig. 10 is never referred to in the paper; reference to Fig. 10 should be added in \"latent gradients\" paragraph.\n\nc) The x-axis of walker environment is not aligned with other mujoco environments.\n\nd) The name, VariBad, should be mentioned in VI and VI+HN to more clearly show the connection of the method tested to existing baselines.\n\n**References:**\n\n[1] J. Beck et al. Hypernetworks in meta-reinforcement learning. In CoRL, 2022.\n\n[2] L. Zintgraf et al. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In ICLR, 2020.\n\n[3] Y. Duan et al. $RL^2$: Fast reinforcement learning via slow reinforcement learning. In arXiv, 2016.\n\n[4] J. Beck et al. A survey of meta-reinforcement learning. In arXiv, 2023.\n\n[5] L. Melo. Transformers are Meta-Reinforcement Learners. In ICML, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate you noting the many benchmarks that we compare against and the rigorous conclusion that the hypernetwork is critical. We address your concerns below. Specifically, we add experiments (by running RNN on more domains and adding an additional domain) and add details (motivation for the hypernetwork and method details), as requested. Could you please raise the score correspondingly, or let us know if any topic remains unclear?\n* **1) Experimental Results**\n  * **a) Meta-World and Degree of Improvement** We do agree that more environments would make our claim stronger and have run more experiments. For this evidence, please see the global response above. Additionally, we also do feel strongly that our existing results show that RNN+HN is not just marginally better than HN+VI, but much better (and much simpler). For example, see the learning curves (blue vs orange) on Walker, Cheetah, and all of the gridworlds. Finally, there is no decisive evidence in the other direction from [1]: The results are not statistically significant in their table, and if you look at the appendix, the learning curves are highly unstable and difficult to draw conclusions from. This lack of evidence and baselines is the main reason for our paper.  \n  * **b) Baselines** We have now adjusted all of our claims throughout the paper to assert that recurrent hypernetworks are surprisingly strong instead of SOTA. For our baselines, we chose to ablate a large number of task-inference methods to show that simpler black-box methods could outperform them. We believe we have sufficiently covered this space. As for PPG methods, it is already known that such methods cannot solve the benchmarks we use, since they are meant to test rapid adaptation within 1 and 2 episodes, which is not sufficient for PPG methods [1] and also shown to be worse in [2]. We do intend transformers with hypernetworks as future work, but which sequence model the hypernetwork conditions on is an orthogonal problem. Moreover, we were simply looking to show that simpler end-to-end methods can surprisingly beat the contemporary task-inference methods, not that they all will.\n* **2) Delivery**\n  * **a) Hypernetwork Intuition** We agree that we can make the paper more self-contained by adding motivation for hypernetworks as in [1] and have now done so. In addition to preventing interference between different tasks and scaling trends suggested by Beck et al., 2022, we hypothesize that the low norm afforded by our models, i.e., low  sensitivity to the latent variable, and re-conditioning on state are crucial for stable training. We have added some text to expand on existing motivation for the use of hypernetworks in related work and have added some text on our analysis in the introduction to make this paper more self-contained.\n  * **b) Empirical Study, Contribution Summary in Introduction, Methods Table** We agree that this could be confusing and have added clarification to the introduction and methods to emphasize that this is an empirical study and RNN+HN is not a novel method, though it is the one we find gives the strongest results, and we add a table comparing method components to the appendix. Please see the global response above. \n  * **c) MuJoCo Details in Appendix** We have added details on the MuJoCo environment to the appendix as requested. See the global response above for details.\n* **3)  Minor problems**\n  * **a) Missing Results** We ran these experiments and results show that the RNN in fact always performs the same or worse than RNN+S and RNN+HN, as expected. Plots can be seen in the global response above. \n  * **b) Fig. 10 Reference** The reference has been added. \n  * **c) X-axis of Walker** We have run the limiting Walker experiment for longer and updated the plot with no significant changes to the results. Plots can be seen in the global response.\n  * **d) Calling VariBAD by Name** We have also now added the name Varibad, as requested.\n\n**Questions:**\n* **0) Performance Difference Between Figures** The difference in curves should be due to different seeds for each plot.\n* **1) Empirical Study, Contribution Summary in Introduction, Methods Table** We have added clarification to the introduction and methods to emphasize that this is an empirical study and not a novel method. We have likewise updated the contribution paragraph at the end of the introduction and created a table (see the global response above).\n* **2) MuJoCo Details in Appendix** We have added details on the MuJoCo environment to the appendix. The details can also be seen in the global response above.\n* **3) Methods from [1]** We do compare to the baselines in [1]. In [1], the authors only compare to VariBAD [2] and RL2 [3]. We added the many baseline variants that you mentioned precisely in order to address this limitation in [1]. Note that [1] Does evaluate VariBAD with many different initializations for the hypernetwork and FiLM, but [1] shows them all to be inferior, which we have also found to be true in our experience. (For other types of methods, see 1b above.)\n* **4) Challenging Environments** The comparison on Meta-World was already made by [1] and the results are not statistically significant in the table. If you look at the appendix in [1], the learning curves are highly unstable and difficult to draw conclusions from. For this reason, we chose to supplement our experiments with MineCraft from AMRL (Beck et al., 2020) instead. See the the global response for details.\n* **5) Minor Issues** (Addressed above.)\n* **6) Code** We plan to release code.\n"}, {"review_id": "FFA70dQMFR", "reviewer": "Reviewer_NnYJ", "summary": "The manuscript proposes to modify a common meta-RL baseline, in which a meta-learned RNN provides the policy network with an encoding of the trajectory. The proposed recurrent hypernetwork instead lets the RNN predict the weights and biases of the policy network. This is shown to be a strong model in comparison with several variations of RNN-based and task-inference meta-RL approaches.", "questions": "1. References to figures such as \"See Figures 3 and 4\" in line 123, I'd put in parentheses directly in the preceding sentence or say something like \"See Figure X for a diagram of Y\".\n\n## Acknowledgement of rebuttal\nI have read the rebuttal and reviews and engaged in discussion with the authors. The authors have mostly addressed my concerns by removing the general claims of the proposed method being state-of-the-art in meta-RL, by providing some clarification on the architecture and experimental setup, and by providing additional experimentation that support some of the design decisions. I have accordingly increased the score.", "limitations": "The authors mention that they cannot guarantee that the proposed method will improve over every baseline nor over every environment. This does not feel like a careful look at model details and how they might affect adaptation to different task variations. One general criticism of many meta-RL methods is that they are evaluated on very narrow task distributions. Maybe the authors could focus on that. The recent meta-RL survey by Beck et al. [1] might give some ideas for discussion.\n\n## References\n1. Beck, Jacob, et al. \"A survey of meta-reinforcement learning.\" arXiv preprint arXiv:2301.08028 (2023).", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "I think the proposed approach is novel and the idea is described in sufficient detail.\nThe experimentation affords equal computational resources for hyperparameter optimization of all models. The method seems to perform better in 6 out of 7 considered environments.\nThe ablation studies adequately address some questions that I thought of while reading the paper.", "weaknesses": "1. The implementation of the RNN and RNN+HN models could've been described in a bit more detail. It is for example not clear what the meta parameters $\\theta$ correspond to in the RNN baseline, since line 108 says that $f$ and $\\pi$ use distinct parameters from $\\theta$. Is this sentence wrong and $\\theta$ are the RNN's parameters? I could also not find all hyperparameters of the architectures, such as the type and layer sizes of the RNNs.\n1. For the strong claim that recurrent hypernetworks are SOTA in (all of) meta-RL, the presented experiments seem rather limited. What about experiments on RLBench or Meta-World? Also, there are many more categories of meta-RL methods to be considered than RNN-based and task-inference methods. The manuscript can benefit greatly from a clear justification of this strong claim.\n1. I could not find an indication of how exactly the performance curves were generated, e.g. how many seeds per model, what exactly does the shaded area convey, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We especially appreciate your time understanding that this paper is not in your field. We appreciate you noting that the ablation studies answered many questions, that the proposed method is stronger on 6/7 of the environments, and that our hyperparameters optimization affords equal computation to all baselines. Responses to your feedback are below. If we have addressed these concerns, please do raise our score, and if not, please let us know what remains unclear:\n* **1) “The implementation of the RNN and RNN+HN models could've been described in a bit more detail. It is for example not clear what the meta parameters  correspond to in the RNN baseline, since line 108 says that  and  use distinct parameters from . Is this sentence wrong and  are the RNN's parameters? I could also not find all hyperparameters of the architectures, such as the type and layer sizes of the RNNs.”** We use a single layer RNN of size 256, as in Beck et al., 2022. We have added this to the appendix. Additionally, we have clarified in the main body that “$f$ and $\\pi$ each use distinct subsets of the meta-parameters, $\\theta$.” We meant that the parameters are distinct from each other, not distinct from $\\theta$. Thank you for pointing out this ambiguity.\n* **2) “For the strong claim that recurrent hypernetworks are SOTA in (all of) meta-RL, the presented experiments seem rather limited. What about experiments on RLBench or Meta-World? Also, there are many more categories of meta-RL methods to be considered than RNN-based and task-inference methods. The manuscript can benefit greatly from a clear justification of this strong claim.”** For environments, we do agree that more environments would make our claim stronger and have done so. For details, please see the global response above. For baselines on the remaining experiments, we chose to ablate many task-inference methods to show that simpler black-box methods could outperform them. We believe we have sufficiently covered this space. As for PPG methods, it is already known that such methods cannot solve the benchmarks we use, since they are meant to test rapid adaptation within 1 and 2 episodes, which is not sufficient for PPG methods (Beck et al., 2023) and also shown to be worse in (Zintgraf et al., 2020). Moreover, we were simply looking to show that simpler end-to-end methods can beat the contemporary task-inference methods, not that they all will. \n* **3) “I could not find an indication of how exactly the performance curves were generated, e.g. how many seeds per model”** The learning curves show meta-episode return, optimized over five learning rates and averaged over three seeds, with a 68% confidence interval using bootstrapping. This information is in section 2 of the Appendix, but we have also now added it to the main body.\n\n**Questions:**\n* **1) “References to figures such as \"See Figures 3 and 4\" in line 123, I'd put in parentheses directly in the preceding sentence or say something like \"See Figure X for a diagram of Y\".”** We have updated these references to match your suggestion."}, {"review_id": "9R7VNojgDE", "reviewer": "Reviewer_rCPG", "summary": "This paper investigates the performance of recurrent neural networks in meta-RL. They suggested that a current neural network can achieve strong performance with hypernetwork. They compared this method with numerous baselines on several meta-RL benchmarks and found that the recurrent baselines along with hypernetwork could achieve SOTA performance.", "questions": "1. Why in the Cheetah environment, RNN+HN is worse than RNN+S?\n\n2. Please offer more ablation studies of the hypernetwork component.", "limitations": "The authors have discussed the limitations of this paper in the last section.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper introduced a novel assertion that a recurrent method, when combined with a hypernetwork, can achieve SOTA performance in meta-reinforcement learning. The authors substantiated this claim with rigorous empirical experiments demonstrating superior performance. Furthermore, they conducted an analysis elucidating the reasons behind the method's impressive performance. The paper is well-written and clearly structured, which makes it easily comprehensible for the readers.", "weaknesses": "1. The authors could enhance the comprehensiveness of the study by testing their algorithm on a wider range of environments.\n\n2. The final analysis segment of the paper would benefit from further development to unequivocally ascertain why the proposed method can achieve state-of-the-art performance.\n\n3. The paper lacks ablation studies for the different settings of hypernetwork component, leaving its individual contribution to the overall results unclear.\n\n4. The format of the reference is wrong", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback and for noting the rigorous empirical experiments, elucidating analysis, and clear structure. We address each of your points in turn below. Please do  adjust the score correspondingly, or let us know if anything remains unclear. Thank you!\n\n* **1) “The authors could enhance the comprehensiveness of the study by testing their algorithm on a wider range of environments.”** We do agree that more environments would make our claim stronger and have done so. For details, please see the global response above.\n* **2) “The final analysis segment of the paper would benefit from further development to unequivocally ascertain why the proposed method can achieve state-of-the-art performance.”** In addition to preventing interference between different tasks and scaling trends suggested by Beck et al., 2022, we hypothesize that the low norm afforded by our models, i.e., low  sensitivity to the latent variable, and re-conditioning on state are crucial for stable training. We have added information on existing motivation for the use of hypernetworks in related work and on our analysis in the introduction to make this paper more self-contained.\n* **3) “The paper lacks ablation studies for the different settings of hypernetwork component, leaving its individual contribution to the overall results unclear.”** In this paper we do perform ablation studies to separate the contribution of the hypernetwork from just conditioning on the state again. Additionally, we evaluate alternative initialization of the hypernetwork. However, we note that varying the size of the Hypernetwork was already ablated in (Beck et al., 2022), so we use the best size from that investigation. Thus, we feel we have properly covered the space of hypernetwork design. Most of this paper focuses on showing that a very simple recurrent hypernetwork (with few components to ablate) can outperform contemporary task-inference approaches. Thus, we spend the most effort ablating the design space of task-inference for comparison. \n* **4) “The format of the reference is wrong”** Could you elaborate on the reference format concern? As far as we are aware, according to the formatting instructions for 2023, “Citations may be author/year or numeric, as long as you maintain internal consistency. As to the format of the references themselves, any style is acceptable as long as it is used consistently.” Link: https://media.neurips.cc/Conferences/NeurIPS2023/Styles/neurips_2023.pdf\n\n**Questions:**\n* **1) “Why in the Cheetah environment, RNN+HN is worse than RNN+S?”** In Cheetah-Vel, RNN+HN has the same asymptotic return as RNN+S and higher initial returns at the start of training. For a brief period of time in the middle of training, RNN+S does achieve higher returns. Since this environment requires less action diversity than the rest (e.g. running at different velocities but not different directions), the curves can likely be explained as follows: In the beginning, RNN+HN learns behavior shared between the tasks more easily due to Bias-HyperInit that ignores the task, then RNN+S is able to learn the task adaptation marginally faster due to very small difference in actions between tasks, then RNN+HN catches up as it is just capable of learning small (and large) changes.\n* **2) “Please offer more ablation studies of the hypernetwork component.”** (Addressed above.)"}, {"review_id": "wfqDQcxN57", "reviewer": "Reviewer_8GEx", "summary": "This paper explores how to approach the problem of meta-reinforcement learning by using hypernetworks. In particular, the authors propose to employ an RL2-like scheme, where instead of producing a vector $\\phi$, the recurrent model outputs a set of neural network weights. They dub the method RNN+HN. They introduce a set of task inference-based baselines with and without the usage of hypernetworks. Subsequently, they show that the simple RNN+HN approach outperforms more sophisticated baselines in gridworld environments and in Mujoco. Finally, they perform an ablation study by using RL2 with state-conditioning.", "questions": "My main suggestion related to the first point in the Weaknesses section above, i.e. extending the empirical evaluation or reducing the state-of-the-art claims.", "limitations": "The author discuss limitations sufficiently.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Overall, I find the paper interesting and the results quite convincing. However, the authors' claims are very strong -- stating that the proposed method is state-of-the-art in the wide field of meta RL. Although the experiments do show that the method works quite well, I do not think it is enough to support this bold claim. As such, for now, I am going with \"borderline reject\", but I would be inclined to increase the score if the authors either extended the empirical evaluation or toned down the claims presented in the paper.\n\nStrengths:\n- The paper introduces a nice, conceptually simple idea that provides substantial improvements. Parameterizing the policy using a neural network with weights generated by a hypernetwork is an elegant idea.\n- The empirical results are in general quite good, the RNN+HN method outperforms most of the baselines in many cases.\n- The paper is well written.\n- The appendix includes additional ablation studies and more baselines.", "weaknesses": "- The critical weakness of the paper is the mismatch between the claims and the empirical evaluation. Namely, the authors state multiple times that their method is state-of-the-art in the field of meta-RL, but the experiments do not support that claim sufficiently:\n    - There are not enough external baseline methods. The baselines the authors use are inspired by existing methods but do not correspond exactly to what has been proposed in the literature previously (e.g. TI-naive, TI). Additionally, the authors omit existing established works such as MAML-based algorithms and PEARL. As they say, they \"exclude the latter methods since estimation of a policy gradient in policy-gradient approaches requires more data than in our benchmarks\". That is, in general, a reasonable assumption, but seems too strict when claiming state-of-the-art.\n    - The authors support their claim by saying that their method outperforms previous SOTA [1, 2], but I'm not convinced that these previous works are still SOTA as of now. Some of the recently published works also show very good results [3, 4].\n    - Currently, the authors only consider two sets of environments: gridworlds and Mujoco continuous control. I think the empirical evaluation should be extended to include more complex environments such as Meta-World, RLBench, Atari, or DeepMind Alchemy. I think that environments with high-dimensional state spaces (e.g. images) would be interesting to test the limits of the proposed method.\n    - There are some environments where RNN+HN falls behind (e.g. Walker, Cheetah-Vel). This is not a grave problem by itself, but it makes the problem of having relatively few benchmarks even more problematic -- how will this method scale up to other environments?\n    - Why the RNN baseline only appears in a single environment in Figure 8 and Figure 9? I think it's an important baseline that should be included.\n\n[1] Zintgraf, Luisa, et al. \"Varibad: A very good method for bayes-adaptive deep rl via meta-learning.\" arXiv preprint arXiv:1910.08348 (2019). \\\n[2] Beck, Jacob, et al. \"Hypernetworks in meta-reinforcement learning.\" Conference on Robot Learning. PMLR, 2023. \\\n[3] Melo, Luckeciano C. \"Transformers are meta-reinforcement learners.\" international conference on machine learning. PMLR, 2022. \\\n[4] Chalvidal, Mathieu, Thomas Serre, and Rufin VanRullen. \"Meta-Reinforcement Learning with Self-Modifying Networks.\" Advances in Neural Information Processing Systems 35 (2022): 7838-7851.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your feedback. We especially appreciate you noting the results are quite convincing and that you are open to increasing the score if we adjust our claims or add additional experimentation. We have now done both, and would very much appreciate a corresponding score increase. Answers to your specific questions are below. Please let us know if anything remains unclear:\n\n* **1) “There are not enough external baseline methods… the authors omit existing established works such as MAML-based algorithms and PEARL”** VariBAD (Zintgraf et al., 2020) already establishes a performance improvement over PEARL and MAML on the same domains we use. For this reason, we compare to VariBAD, a contemporary VariBAD with a hypernetwork (Beck et al., 2022), and direct task-inference methods. It is fair to note that the direct methods, e.g. TI and TI Naive, are not identical to previous methods. However, these were chosen to maintain the strengths of both VariBAD while also using the more direct supervision afforded by the Belief Agent (Humplik et al., 2019). We feel it is actually more fair to do so because the combination uses insights from both papers, and it makes hyperparameter tuning more fair.\n* **2) The authors support their claim by saying that their method outperforms previous SOTA [1, 2], but I'm not convinced that these previous works are still SOTA”** The difficulty in establishing SOTA with hundreds of meta-RL papers (Beck et al., 2023) is well taken. We have now adjusted all of our claims throughout the paper to assert that recurrent hypernetworks are surprisingly strong instead of SOTA. The goal of this paper is to show that simpler recurrent methods (trained end-to-end) can vastly outperform contemporary alternatives (often considered SOTA) in the task-inference design space, when hypernetworks are used. The use of transformers (Melo et al., 2022), as suggested, is an orthogonal consideration which we do agree could further improve the latent variable passed to the hypernetwork. This could make for great future work.  \n* **3) “Currently, the authors only consider two sets of environments: gridworlds and Mujoco continuous control… I think that environments with high-dimensional state spaces (e.g. images) would be interesting”** We do agree that more environments (including image observations) would make our claim stronger and have done so. Please see the global response above. \n* **4) “There are some environments where RNN+HN falls behind (e.g. Walker, Cheetah-Vel).…how will this method scale up to other environments?”** Note that on Cheetah-Vel, TNN+HN still has the greatest return at both the start and end of training. As for Walker, the only baselines (TI and TI_Naive) that outperform RNN+HN only do so marginally and are significantly outperformed by RNN+HN on the vast majority of environments. For scaling up, please see the global response above.\n* **5) “Why the RNN baseline only appears in a single environment in Figure 8 and Figure 9? I think it's an important baseline that should be included.”** We have since run these experiments and results confirm that the RNN is in fact always performs the same or worse than RNN+S and RNN+HN, as expected. Please see the global response above.\n"}, {"review_id": "GlMbEiszUm", "reviewer": "Reviewer_k7Er", "summary": "The paper investigate the performance of a specific type of architecture, RNNs coupled with hypernetworks, in meta-reinforcement learning. Meta-RL aims to address the sample inefficiency of RL algorithms by learning to perform few-shot learning when given a distribution of related tasks for meta-training.\n\nThe authors note that while specialized and complex meta-RL methods have been proposed in the literature, recent work suggests that using an off-the-shelf sequential model, such as an RNN, trained in an end-to-end manner can serve as a strong baseline. However, the supporting evidence for this claim has been limited. The paper presents an extensive empirical investigation to address this gap. While RNNs can achieve strong performance in meta-RL, the study finds that the use of hypernetworks is crucial in maximizing their potential. Interestingly, when combined with hypernetworks, the simpler recurrent baselines outperform existing specialized methods and establish themselves as state-of-the-art (SOTA) on standard meta-RL benchmarks. ", "questions": "Q1: Line 128, I could not understand why the predicted context is not directly used to condition the policy. From my understanding, the context should contain enough information to disambiguate the tasks, and the concatenation of the state and context should be Markovian. Could you provide an intuition why intermediate representations are used?\n\nQ2: The RNN-S is an interesting experiment. One additional hypothesis for why RNN+HN might work well is because it allows for some multiplicative interaction between the context and the state. What types of non-linearities were used in the network? Have you used activation units, such as Gated Linear Units (GLU) [1] which allows for multiplicative interactions between the neurons in the RNN-S?\n\n[1]: \"Language Modeling with Gated Convolutional Networks\", Dauphin 2016 ", "limitations": "The limitations I can think of have been addressed by the authors.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: The paper is extremely clear\n\nS2: The authors make sure to tune each methods, which is something I do not see often enough \n\nS3: Coupling RNNs and hypernetworks is an idea that is novel in the field of meta-RL, and the attempts at understanding why they might be outperforming other baselines are interesting", "weaknesses": "W1: While the hyperparameter tuning was fairly done in terms of computation budget, it is to me not clear this strategy is the most relevant. I can imagine that many of the complex methods have many more hyperparameters to tune than the simple methods that you propose, and an excessively small search grid could unfavorably disadvantage the former. \n\nW2: The empirical investigation is, as noted by the authors, quite limited. In that regard, more empirical evidence from different benchmarks would make the paper stronger.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate you noting the clarity and fair tuning. We address your comments below. If we have addressed these concerns, please do raise our score, and if not, please let us know what remains unclear:\n\n* **1) “While the hyperparameter tuning was fairly done in terms of computation budget, it is to me not clear this strategy is the most relevant. I can imagine that many of the complex methods have many more hyperparameters to tune than the simple methods that you propose, and an excessively small search grid could unfavorably disadvantage the former.”** That is a fair concern, but to mitigate this issue we do use the default parameters from prior work (Zintgraf et al., 2020 and Beck et al., 2022), and we perform additional tuning just for the baselines in the Appendix.\n* **2) “The empirical investigation is, as noted by the authors, quite limited. In that regard, more empirical evidence from different benchmarks would make the paper stronger.”** We do agree that more environments would make our claim stronger and have done so. Please see the global response above. \n\n**Questions:**\n\n* **1) “Line 128, I could not understand why the predicted context is not directly used to condition the policy.… Could you provide an intuition why intermediate representations are used?”** In Humplik et al., 2019 the authors suggest to condition on an earlier representation since the task can be inferred if necessary, but the earlier layer contains more information about the belief state. More specifically, we pass the inferred mean and variance from an information bottleneck layer, as in Zintgraf et al., 2020. This explicit representation of the task uncertainty enables more efficient exploration when the task is not yet known. For example, the uncertainty will inform the agent whether it has to explore more or not. We have now added some clarification of this point into the paper as well.\n* **2) “The RNN-S is an interesting experiment. One additional hypothesis for why RNN+HN might work well is because it allows for some multiplicative interaction between the context and the state. What types of non-linearities were used in the network? Have you used activation units, such as Gated Linear Units (GLU) [1] which allows for multiplicative interactions between the neurons in the RNN-S?”** Beck et al. 2022 did consider FiLM as an alternative and restricted type of multiplicative interactions and found the more general hypernetwork to be more capable. For this reason, we also use the hypernetwork. However, we appreciated the idea and think this could make for good future work!\n"}]

Author rebuttal:
In order to address reviewers requests, we have added additional experiments (environments and baselines), adjusted claims, and added details (in a table and text). Figures for new experiments are in an attached PDF, and specifics are below.

**TLDR:**
* We have removed claims to SOTA
* New experiments in MineCraft corroborate our claims
* New experiments on previous domains corroborate that RNN alone is a weak baseline
* We have added environment details and method details and motivation in text and a table

**Updated Title and Claim:**
Some reviewers felt that more experiments were needed to claim SOTA. To address this feedback, we have added experiments and removed claims to SOTA. For instance, our new title is *Recurrent Hypernetworks are Surprisingly Strong in Meta-RL*, and we have clarified our contributions at the end of the introduction. Here, we emphasize that we conduct an empirical investigation showing the value of hypernetworks in maximizing the potential of recurrent networks. We clarify upfront that while the use of a hypernetwork with RNNs is not a novel idea, they have never been evaluated in meta-RL beyond a single environment, let alone shown to outperform contemporary task-inference methods (Beck et al., 2022). 

**Additional MineCraft Environment:**
In addition to toning down our claims, we have added more experiments to demonstrate the strength of RNN+HN on more complex and challenging domains. Some reviewers have suggested Meta-World and Alchemy. Meta-World is more challenging; however, such benchmarks are generally too difficult to solve (Beck et al., 2023) and high variance, making them extremely difficult to get informative comparisons even after weeks of training (Beck et al., 2022). Similar issues exist for Alchemy (Beck et al., 2023). Instead, we add experiments with a MineCraft environment. 

Specifically, we supplement our results with experiments in the MC-LS environment from AMRL (Beck et al., 2020), since it can easily be adapted to meta-RL and was originally designed to test more challenging long-term memory from visual observations in MineCraft. In this environment, the agent must navigate through a series of 16 rooms. In each room, the agent must navigate left or right around a column, depending on whether the column is made of diamond or iron. Correct behavior receives a reward of 0.1. Finally, at the end, the agent must move right or left depending on a signal (red or green) that was shown before the first room. Correct behavior receives a reward of 4, while incorrect behavior receives a reward of -3. In our experiments, we allow the agent to adapt over two consecutive episodes, forming a single meta-episode, where each direction corresponds to a task.

On MC-LS we compare RNN+HN to VI+HN, since VI+HN is an established contemporary task-inference baseline (Beck et al., 2022). Additionally, we add an additional seed (four in total) and a linear learning rate decay due to high variance in the environment. In the MineCraft Figure in the PDF, we see that RNN+HN significantly outperforms VI+HN. While VI+HN learns to navigate through all rooms, it does not learn the correct long-term memory behavior. In contrast, RNN+HN is able to adapt reliably within two episodes, and one seed even learns to adapt reliably within a single episode. While further work is needed to learn the optimal policy, these experiments demonstrate that RNN+HN outperforms VI+HN, even on more challenging domains.

**Additional RNN Baselines:**
Some reviewers noted that the RNN baseline was not run on several MuJoCo experiments and that some results on Walker were not run as long. This was due to computational limitations. However, we have now added the RNN baseline to all MuJoCo experiments, and we have run all methods on the Walker experiment longer to match the duration of the other MuJoCo environments. Please see the PDF for figures. Results are consistent with previous claims: RNN is a weak baseline, RNN+S is a stronger baseline, RNN+HN outperforms both.

**Methods Table:**
A table summarizing the components in each all baseline methods has been added to the appendix:
| |Inference Target|Policy Conditions on State|Hypernetwork|Inference Pre-Training and Parameter Reuse|
|-|---|---|---|---|
|RNN|None|No|No|N/A|
|RNN+S|None|Yes|No|N/A|
|RNN+HN|None|Yes|Yes|N/A|
|TI Naive|Given|Yes|No|N/A|
|TI|Learned|Yes|No|No|
|TI++|Learned|Yes|No|Yes|
|TI+HN|Learned|Yes|Yes|No|
|TI++HN|Learned|Yes|Yes|Yes|
|VI|Transitions|Yes|No|N/A|
|VI+HN|Transitions|Yes|Yes|N/A|
|BI++HN|Base Net|Yes|Yes|N/A|

**MuJoCo Details:**
One reviewer noted that details on the MuJoCo environments should be included in order to make the paper more self-contained. We have added these details to the appendix. We share the details here for two environments (Cheetah-Dir and Walker) as examples.

We evaluate on all four MuJoCo environments used by Zintgraf et al., 2020. All environments require legged locomotion. 

In Cheetah-Dir, the agent controls a robotic cheetah morphology by outputting a control torque for each of six joints on the morphology. The task is to run forward or backward with as large a velocity as possible. The agent observes the position, angle, and velocity of each body part (17 dimensions in total) and is given a positive reward for its velocity in the direction given by the task and a negative reward for control costs (specifically, 5% of the magnitude of the action vector).

In Walker, the agent controls a two-legged torso morphology and outputs a control torque for each of six joints on the morphology. The observations are 17 dimensional for this environment. The tasks are defined as uniform samples of 65 different physics coefficients (e.g. body mass and friction) for the simulation. The agent is given a positive reward for the forward velocity, a positive reward of one reward per timestep, and a negative reward for the control costs (specifically, 0.1% of the magnitude of the action vector).

Meta-review: Initially, the paper received mixed scores. The reviewers liked the idea of revisiting hypernetwork for meta-RL tasks and noted the investigation of different variants of the proposed method. However, they were concerned about the limited evaluation environments, unconvincing performance improvement, and a lack of comparison with some baselines, which cannot support the strong claim of the proposed method being state of the art in meta-RL. Additionally, they were concerned about the limited novelty (being an empirical study of an existing method), the unclear motivation, the absence of explanation regarding some important design choices, implementation details, and ablation studies, and the quality of the writing and presentation.  


The rebuttal dealt partially with the reviewers’ points by toning down the state-of-the-art claim to be surprisingly strong, and offering additional explanation, experimental results (including more baselines and benchmarks, such as some Meta-World experiments), and discussion. 


After rebuttal, Reviewers rCPG, NnYJ, and 8GEx rated the weak accept or accept score, Reviewer k7Er rated the borderline accept score, and Reviewer YdBr rated the borderline reject score. In the subsequent discussion, Reviewer YdBr continued to express concerns regarding the major change of the paper’s claim and marginal performance improvement in experimental results, but Reviewer YdBr did not insist on rejecting the paper. On the other hand, Reviewers NnYJ and 8GEx championed the paper, pointing out that the changes made during the rebuttal can be incorporated into the camera ready version without the need for another review cycle. 


The AC has read the paper, reviews, and rebuttal, and discussed with the reviewers at length. The AC found the arguments of Reviewers NnYJ and 8GEx to be the most persuasive. While the experimental evaluation did not support the paper’s original claim of state-of-the-art performance in meta-RL, the conducted empirical study still demonstrated interesting findings and competitive performance, which could provide a different perspective and useful basis to develop simpler meta-RL methods for future work in this direction. Also, during rebuttal, the authors have committed to removing this claim and revising the title and narrative accordingly in the revision. These changes would not lead to a significant rewriting of the paper.


The authors are encouraged to improve the camera ready version by following reviewer recommendations, in particular precisely revising the claim and contributions as promised (i.e., not state of the art, being an empirical study rather than proposing a novel method), including new experimental results provided in the rebuttal and more in-depth discussion of limitations. While some results were incomplete due to time and computational constraints during rebuttal, it would be beneficial to conduct the full set of experiments and incorporate them in the camera ready version. This work would greatly benefit from such revision and its impact can be significantly improved. This decision was discussed with and approved by the AC and SAC.
Decision: Accept (poster)