Title: Recurrent Hypernetworks are Surprisingly Strong in Meta-RL
Venue: NeurIPS 2023
Accepted: yes
Keywords: meta-RL

RL

reinforcement learning

memory

rnn

recurrent

hypernetwork

few-shot

in-context
Abstract: Deep reinforcement learning (RL) is notoriously impractical to deploy due to sample inefficiency. Meta-RL directly addresses this sample inefficiency by learning to perform few-shot learning when a distribution of related tasks is available for meta-training. While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shelf sequential model, such as a recurrent network, is a surprisingly strong baseline. However, such claims have been controversial due to limited supporting evidence, particularly in the face of prior work establishing precisely the opposite. In this paper, we conduct an empirical investigation. While we likewise find that a recurrent network can achieve strong performance, we demonstrate that the use of hypernetworks is crucial to maximizing their potential. Surprisingly, when combined with hypernetworks, the recurrent baselines that are far simpler than existing specialized methods actually achieve the strongest performance of all methods evaluated. We provide code at https://github.com/jacooba/hyper.

Recurrent Hypernetworks are Surprisingly Strong in Meta-RL

Abstract

Deep reinforcement learning (RL) is notoriously impractical to deploy due to sample inefficiency. Meta-RL directly addresses this sample inefficiency by learning to perform few-shot learning when a distribution of related tasks is available for meta-training. While many specialized meta-RL methods have been proposed, recent work suggests that end-to-end learning in conjunction with an off-the-shelf sequential model, such as a recurrent network, is a surprisingly strong baseline. However, such claims have been controversial due to limited supporting evidence, particularly in the face of prior work establishing precisely the opposite. In this paper, we conduct an empirical investigation. While we likewise find that a recurrent network can achieve strong performance, we demonstrate that the use of hypernetworks is crucial to maximizing their potential. Surprisingly, when combined with hypernetworks, the recurrent baselines that are far simpler than existing specialized methods actually achieve the strongest performance of all methods evaluated. We provide code at https://github.com/jacooba/hyper .

1 Introduction

Meta-reinforcement learning [Beck et al., 2023] uses sample-inefficient reinforcement learning (RL) to learn a sample-efficient reinforcement learning algorithm. The sample-efficient algorithm maps the data an agent has gathered so far to a policy based on that experience. To this end, any sequential model such as a recurrent neural network (RNN), can be deployed to learn this mapping end-to-end [Duan et al., 2016, Wang et al., 2016]. Such methods are also called black-box [Beck et al., 2023].

Alternatively, much prior work has focused on a category of task-inference methods that are specialized for meta-RL. A meta-RL algorithm learns to reinforcement learn over a distribution of MDPs, or tasks . By explicitly learning to infer the task, many methods have shown improved performance relative to the recurrent baseline Humplik et al. [2019], Zintgraf et al. [2020], Kamienny et al. [2020], Liu et al. [2021], Beck et al. [2022].

Recent work has shown the simpler recurrent methods to be a competitive baseline relative to task-inference methods [Ni et al., 2022]. However, such claims are contentious, as the supporting experiments compare only to one task-inference method designed for meta-RL, the experiments provide additional compute to the recurrent baseline, and the results still show similar or inferior performance to more complicated methods on the majority of difficult domains. In particular, they consider two toy domains and four challenging domains, with RNNs significantly outperformed on two of the four challenging domains, and superior to the single task-inference baseline on only one.

(a) Walker

(b) Cheetah-Dir

2000

2500

1750

2000

1500

1250

1500

E

1000

Return

750

1000

500

Model

500

250

RNN

Model

0

RNN+HN

0

RNN

RNN+HN

0

10000200003000040000500006000070000

Frames (k)

0

10000200003000040000500006000070000

Frames (k)

In this paper, we conduct a far more extensive empirical investigation with stronger and carefully designed baselines in meta-RL specifically. In addition, we afford equal computation in terms of number of samples for hyper-parameter tuning to all existing baselines. We present the key insight that the use of a hypernetwork architecture [Ha et al., 2017] is crucial to maximizing the potential of recurrent networks. For an illustration of the potential magnitude of improvement, see Figure 1. While the use of a hypernetwork with RNNs is not a novel idea, they have never been evaluated in meta-RL beyond a single environment, let alone shown to outperform contemporary task-inference methods [Beck et al., 2022]. We additionally provide preliminary evidence that the robust performance hypernetworks achieve such is in part due to how they condition on the current state and history. Finally, our results establish recurrent hypernetworks as an exceedingly strong method on meta-RL benchmarks that is also far simpler than alternatives, providing significant ease of use for practitioners in meta-RL.

2 Related Work

Recurrent Meta-RL. Many meta-RL methods structure the learned RL algorithm as a black box using a neural network as a general purpose sequence model [Duan et al., 2016, Wang et al., 2016, Mishra et al., 2018, Fortunato et al., 2019, Ritter et al., 2021, Wang et al., 2021, Ni et al., 2022]. While any sequence model can be used, often the model is structured as an RNN [Duan et al., 2016, Wang et al., 2016, Ni et al., 2022]. Such models [Duan et al., 2016, Wang et al., 2016] are commonly used as simple meta-RL baselines.

One study has shown RNNs to be a competitive baseline in meta-RL [Ni et al., 2022]; however, the scope of the study was broader than meta-RL and the evidence specific to meta-RL is inconclusive. First, the study evaluates only a single specialized meta-RL method [Zintgraf et al., 2020], which was, but is not currently, state-of-the-art [Beck et al., 2022]. Second, the experiments use results or hyperparameters from the original papers, while affording extra computation to tune the RNNs on each benchmark, including dimensions that were not tuned for the other baselines. This computation includes tuning architecture choices, the context length, the RL algorithm used, and the inputs [Ni et al., 2022]. And third, the study does not show particularly strong performance of recurrent methods relative to the chosen specialized baseline. On the MuJoCo domains, the recurrent baseline outperforms the specialized method on only one of these four domains, performs similarly on another, and is significantly outperformed on the other remaining two [Ni et al., 2022]. In contrast, our work compares against four specialized baselines; affords equal computation to all methods, defaulting to hyper-parameters that favor existing task-inference methods for parameters that are not tuned; and still establishes recurrent hypernetworks as the strongest method evaluated.

Task Inference Meta-RL. In addition to recurrent meta-RL methods, task-inference methods [Humplik et al., 2019, Zintgraf et al., 2020, Kamienny et al., 2020, Liu et al., 2021, Beck et al., 2022] and policy-gradient methods [Yoon et al., 2018, Finn et al., 2017, Vuorio et al., 2019, Zintgraf et al., 2019] constitute a significant bulk of existing work. We exclude the latter methods from

(a) RNN

(b) RNN+HN

s

comparison since the estimation of a policy gradient in policy-gradient approaches requires more data than in our benchmarks [Zintgraf et al., 2019, Beck et al., 2023]. Task inference methods are a strong baseline for our benchmark, but are generally more complicated than recurrent meta-RL methods. For example, such methods typically add a task inference objective [Humplik et al., 2019], and may also add a variational inference component [Zintgraf et al., 2021], or pre-training of embeddings with privileged information [Liu et al., 2021]. In this paper, we ablate each of these components to create the strongest task-inference baselines possible. In the end, we find the more complicated task inference methods are still inferior to the recurrent baseline with hypernetworks.

Hypernetworks. Ahypernetwork [Ha et al., 2017] is a neural networks that produces the parameters (weights and biases) for another neural network, called the base network. Hypernetworks have been used in supervised learning (SL) [Ha et al., 2017, Chang et al., 2020], Meta-SL [Rusu et al., 2019, Munkhdalai and Yu, 2017, Przewiezlikowski et al., 2022], and meta-RL [Beck et al., 2022, Xian et al., 2021, Peng et al., 2021, Sarafian et al., 2021]. While these networks are complicated, and can fail to work out-of-the-box, simple initialization methods can be sufficient to enable stable learning [Beck et al., 2022, Chang et al., 2020]. In meta-RL, only Beck et al. [2022] have investigated training a hypernetwork end-to-end to arbitrarily modify the weights of a policy. This study suggests that hypernetworks are particularly useful in preventing interference between different tasks and enable greater returns as the the number of parameters increases. However, their study shows a task-inference method to be superior, and the recurrent hypernetwork is evaluated only on a single task with results that are statistically insignificant. Recurrent hypernetworks have never been widely evaluated in meta-RL, let alone shown to outperform contemporary task-inference methods.

3 Methods

3.1 Problem Setting

A task in RL is formalized as a Markov Decision Processes (MDP), defined as a tuple of ( S , A , R , P , γ ) . At each time-step t , the agent inhabits a state, s t ∈ S , which it can observe. The agent then performs an action a t ∈ A . The MDP subsequently transitions to the next state s t +1 ∼ P ( s t +1 | s t , a t ): S×A×S → R + , and the agent receives reward r t = R ( s t , a t ): S×A → R upon entering s t +1 . The agent acts to maximize the expected future discounted reward, R ( τ ) = ∑ r t ∈ τ γ t r t , where τ denotes the agent's trajectory throughout an episode in the MDP, and γ ∈ [0 , 1) is a discount factor. The agent takes actions sampled from a learned policy, π ( a | s ) : S × A → R + .

Meta-RL algorithms learn an RL algorithm, f ( τ ) , that maps from the data, τ , sampled from a single MDP, M∼ p ( M ) , to policy parameters ϕ . As in a single RL task, τ is a sequence up to time-step t forming a trajectory τ t ∈ ( S ×A× R ) t . Here, however, τ may span multiple episodes within a single MDP, since multiple episodes of interaction may be necessary to produce a reasonable policy. We use the same symbol, τ , but refer to it as a meta-episode . The policy, π θ ( a | ϕ = f θ ( τ )) , is parameterized by ϕ . f is itself parameterized by θ , which are referred to as the meta-parameters .

(a) TI Naive

IB

TI

(b) TI

TI

IB

The objective in meta-RL is to find meta-parameters θ that maximize the sum of the returns in the meta-episode across a distribution of tasks (MDPs):

f θ is referred to as the inner-loop , which produces ϕ , in contrast to the outer-loop , which produces θ .

3.2 Recurrent Methods

Recurrent methods are perhaps the simplest and most common meta-RL baseline. Recurrent methods use an RNN to encode history and train all meta-parameters end-to-end on Equation 1. These methods are depicted in Figure 2. While neither recurrent networks (RNN below), nor the the combination of a hypernetwork with recurrent networks (RNN+HN below) is a novel, the combination has never been widely evaluated in meta-RL [Beck et al., 2022], but we will show that the combination actually achieves the strongest results.

RNN. Our first recurrent baseline is the simplest and is equivalent to RL2 [Duan et al., 2016] and L2RL [Wang et al., 2016]. In this case π θ ( a | ϕ = f θ ( τ )) , where f is a recurrent network, π is a feed-forward network, and f and π each use distinct subsets of the meta-parameters, θ .

RNN+HN. Our second recurrent model is the recurrent hypernet and achieves the strongest results. Here, the recurrent network produces the weights and biases for the policy directly: π ϕ ( a | s ) . The state must be passed as input again to this policy for the feed-forward policy to condition on an input, and we follow the initialization method for hypernetworks, Bias-HyperInit, suggested by Beck et al. [2022]. In this initialization, the hypernetwork's final linear layer is initialized with a zero weight matrix and a non-zero bias, so that the hypernetwork produces the same base-network parameters for any trajectory at the start of training.

3.3 Task-Inference Methods

Task-inference methods [Beck et al., 2023] constitute the main category of meta-RL methods capable of adaptation as quickly as recurrent (i.e., black-box) methods [Humplik et al., 2019, Zintgraf et al., 2020, Kamienny et al., 2020, Liu et al., 2021, Beck et al., 2022]. These methods train the inner-loop not end-to-end but rather to identify the task, within the given task distribution. One perspective on these methods is that they attempt to shift the problem from the more difficult meta-RL setting to the easier multi-task setting by learning to explicitly infer the task [Beck et al., 2023]. Here we define relevant task-inference methods used as baselines (Figures 3 and 4). For additional ablations, summary, and details on the method selection process, see the appendix.

TI Naive. The inner-loop of a meta-RL method must take in the trajectory, τ , and produce the policy parameters, ϕ . In task inference methods, the inner-loop additionally produces an estimate

(a) TI++HN

s

IB

s

TI

(b) VI+HN

IB

s

VI

of the current task, ˆ c M , given a known task representation, c M . While it is possible to represent ϕ as ˆ c M directly, i.e. pass ˆ c M to the policy, it is common to compute ϕ from an intermediate layer of the network that predicts ˆ c M , which contains more information about the belief state [Humplik et al., 2019, Zintgraf et al., 2020]. It is also common to use an information bottleneck to remove information not useful for task inference from the trajectory [Humplik et al., 2019, Zintgraf et al., 2020]. Following Zintgraf et al. [2020], we condition ϕ on the mean and variance of the bottleneck layer in order to explicitly condition the policy on task uncertainty for more efficient exploration (Figure 3). Putting these together, we can write a task inference method as follows:

where P c , P ϕ , P µ , and P σ are all linear projections, ⊥ represents a stop-gradient, P ϕ ⊥ ( µ, σ ) is the matrix multiplication of P ϕ with stop-gradient of a concatenation of ( µ, σ ) , and D is the KLdivergence. Here, J infer + J prior constitutes the evidence lower bound from Zintgraf et al. [2020] and is used to train IB , whereas all other parameters ( P ϕ and π ( ·| ϕ ) ) are trained via Equation 1.

TI. As presented, the TI Naive baseline may suffer from a known issue where the given task representation contains too little or too much information [Beck et al., 2023]. When too little information is present, the policy may miss information crucial to the task. When too much information is present, the policy may be presented with the difficult problem of separating the useful information from irrelevant task features. Toward this end, it is possible to pre-train the task representation end-to-end using an additional policy [Humplik et al., 2019, Kamienny et al., 2020, Liu et al., 2021]. Our TI baseline (Figure 3) is the same as TI Naive, except that an additional multi-task policy, π ′ , is pre-trained to learn a representation of the task, g θ ( c M ) . Given a linear projection, P g :

For a fair comparison, training of the multi-task policy, π ′ , occurs at the expense of training the meta-learned policy π , with the total number of samples remaining constant. Instead of fully training the multi-task policy, we experiment with different amounts of pre-training in the appendix, finding significant benefits already from less than 5% of total training allocation for the pre-training.

TI++HN. The TI++HN baseline is the same as TI, with three additions that we found to strengthen task inference (Figure 4). The first two additions (++) are novel and are 1) initializing the parameters of the meta-policy, π , to that of the pre-trained multi-task policy, π ′ , to encourage transfer and 2)

(a) Grid-World

(b) Grid Show

(c) Grid Dense

40

40

30

20

30

20

0

Return

Return

20

Return

10

Model

Model

-20

Model

VI+HN

10

VI+HN

VI+HN

0

TI_Naive

TI

NH++1L

0

TI_Naive

TI

NH++11

-40

TI_Naive

NH++1L

T1

RNN+HN

RNN+HN

-60

RNN+HN

0

500

1000

1500200025003000

Frames (k)

3500

4000

0

500

1000

1500200025003000

Frames (k)

3500

4000

0

500

1000

1500200025003000

Frames (k)

3500

4000

training of the task-inference ( J infer ) over trajectories from the initial multi-task training phase in addition to the meta-learning phase, since the former tend to be more informative and simply provide extra data. The third addition (HN) uses a hypernetwork to condition the policy on ϕ [Beck et al., 2022]. We write this as π ϕ ( ·| s ) to show that ϕ represents the weights and biases of π , just as in the recurrent baseline. The output of the hypernetwork is ϕ , and the input to the hypernetwork is a the projection of µ and σ , ϕ = h ( ReLU ( P ϕ ⊥ ( µ, σ ))) . When using a hypernetwork with the first two additions (++), the parameters of the hypernetwork for the meta-learned policy are initialized to the parameters of hypernetwork for the multi-task policy, instead of sharing policy parameters directly.

VI+HN. While task-inference methods rely on known task representations, it is also possible to design methods that can infer the MDP more directly. This can be done by inferring transitions and rewards in full trajectories, since the transition function and reward function collectively define the MDP. In particular, such a method, called VariBAD, is proposed by Zintgraf et al. [2020], and extended with the use of hypernetworks by [Beck et al., 2022]. Here, we call this method VI+HN, and it is a contemporary task-inference method (Figure 4). Precisely, this model reconstructs full trajectories including future transitions for meta-episodes, τ 0: T , instead of task embeddings, from τ , the current trajectory:

See the appendix for ablations with VI alone.

4 Experiments

In this section we compare recurrent hypernetworks (RNN+HN) to task inference baselines. We evaluate over three simple navigation domains [Zintgraf et al., 2020, Humplik et al., 2019, Rakelly et al., 2019], designed to test learning of exploration and memory, in addition to four more difficult tasks using MuJoCo [Todorov et al., 2012], and one task testing long-term memory from visual observations in MineCraft [Beck et al., 2020]. Results show meta-episode return, optimized over five learning rates and averaged over three seeds (four in MineCraft), with a 68% confidence interval using bootstrapping. Additional details and results on Meta-World [Yu et al., 2020] are available in the appendix. Our experiments demonstrate that while our task inference methods are strong baselines, RNN+HN is able to outperform them and achieve the highest returns.

4.1 Grid-Worlds

Navigation tasks are a common benchmark in meta-RL [Zintgraf et al., 2020, Humplik et al., 2019, Rakelly et al., 2019]. Here we evaluate on the grid-world variant from Zintgraf et al. [2020], in addition to two of our own variants. The first environment, Grid-World, consists of a five by five grid with a goal location in one of the grid cells. The agent starts in the bottom left corner of the grid, and then must navigate to a goal location, which is held constant throughout the meta-episode. (Details are in the appendix.) This environment is useful for testing how well a meta-learning algorithm learns to efficiently explore in the gridworld as it searches for the goal. Additionally, our Grid-World Show environment was designed to be relatively harder for end-to-end methods, in order to provide a challenge for our proposed method. In this environment, the goal position is visible to the agent

(a) Walker

(b) Cheetah-Vel

(c) Cheetah-Dir

(d) Ant-Dir

2000

-50

-100

1500

-150

Return

-200

1000

Model

-250

Model

VI+HN

00-

VI+HN

500

TI_Naive

TI_Naive

TI

-350

TI

NH++I1

NH++11

RNN+HN

-400

NH+NN

0

0

0000000009000000000000000000700000

0

10000200003000040000500006000070000

Frames (k)

Frames (k)

2500

1500

1250

2000

1000

1500

750

Return

Return

500

1000

Model

Model

VI+HN

250

VI+HN

500

TI_Naive

TI_Naive

TI

0

TI

NH++1L

NH++IL

0

RNN+HN

-250

RNN+HN

0

0000000009000000000000000000000001

0

10000200003000040000500006000070000

Frames (k)

Frames (k)

at the first timestep of each episode. Task inference methods will directly encourage the storage of this information in memory, whereas the end-to-end recurrent methods must learn to store this information through its effect on the policy. In contrast, our Grid-World Dense environment provides dense rewards and may be easier for end-to-end methods. In this environment, the agent receives and observes a reward equal to the Manhattan distance to the goal location. Instead of inferring the task explicitly, the agent can simply move up or to the right until the reward stops increasing.

Surprisingly, on all three grid-worlds, RNN+HN achieves both the greatest asymptotic return and greatest sample efficiency (Figure 5). The recurrent hypernetwork achieves the fastest learning on Gridworld Show, despite the environment being specifically designed to be harder for end-to-end methods. TI++HN dominates all other task-inference baselines on these grid-worlds, suggesting that it is a relatively strong task-inference method. Collectively, these grid-worlds demonstrate that end-to-end learning with hypernetworks can learn to store the task in memory and to explore optimally with this information directly from return, just as well as task-inference methods.

4.2 MuJoCo

Here we evaluate baselines on more challenging domains. We evaluate on all four MuJoCo variants proposed by Zintgraf et al. [2020], which is known to be a common and more challenging metaRL benchmark involving distributions of tasks requiring legged locomotion [Zintgraf et al., 2020, Humplik et al., 2019, Rakelly et al., 2019, Beck et al., 2023]. Ant-Dir and Cheetah-Dir both involve non-parametric task variation, whereas Cheetah-Vel and Walker include parametric variation of the target velocity and physics coefficients respectively. For environment details, see the appendix. We expect Walker in particular to be difficult for end-to-end methods since it has the largest space of tasks with the dynamics defined by 65 different parameters. Assuming the values of these parameters are important for the optimal policy, task inference methods may learn the optimal policy faster.

On Cheetah-Dir and Ant-Dir, RNN+HN achieves greater returns than all other baselines by a wide margin (Figure 6). On Cheetah-Vel, all methods achieve fairly similar results, with RNN+HN

25

20

Return

15

10

5

Model

VI+HN

0

RNN+HN

0

10000

20000

30000

40000

50000

60000

Frames (k)

still achieving the greatest asymptotic return by a small margin. As expected, RNN+HN does not outperform task inference on Walker. For Walker, only TI outperforms RNN+HN in terms of efficiency, and only TI Naive outperforms RNN+HN in terms of asymptotic return; however, the effect size is small and both TI and TI Naive have among the worst performance on Cheetah-Dir and the grid-worlds. Still, RNN+HN achieves similar performance on Walker, which is notable in a high dimensional task space. And, RNN+HN achieves greater returns overall.

4.3 MineCraft

We additionally evaluate on the MC-LS environment from Beck et al. [2020], designed to test longterm memory from visual observations in MineCraft. Here, the agent navigates through a series of 16 rooms. In each room, the agent navigates left or right around a column, depending on whether the column is made of diamond or iron. Discrete actions allow for a finite set of observations. Correct behavior can be computed from the observation and receives a reward of 0.1. At the end, the agent moves right or left depending on a signal (red or green) that defines the task and is shown before the first room. Correct and incorrect behavior receives a reward of 4 and -3, respectively. We allow the agent to adapt over two consecutive episodes, forming a single meta-episode.

On MC-LS we compare RNN+HN to VI+HN alone, given a limited compute budget and since VI+HN is an established contemporary task-inference baseline Beck et al. [2022]. Additionally, we add an extra seed (four in total) and a linear learning rate decay due to high variance in the environment. In Figure 7, we see that RNN+HN significantly outperforms VI+HN. While VI+HN learns to navigate through all rooms, it does not reliably learn the correct long-term memory behavior. In contrast, RNN+HN is able to adapt reliably within two episodes, and one seed even learns to adapt reliably within a single episode. While further work is needed to learn the optimal policy, these experiments demonstrate that RNN+HN outperforms VI+HN, even on more challenging domains.

5 Discussion

Here we investigate why the recurrent hypernetworks have such robust performance on meta-RL benchmarks. First, we observe that the state processing differs between the RNN and RNN+HN

(a) RNN+S

s

(b) RNN+HN

s

(a) Grid

(b) Grid Show

(c) Grid Dense

40

40

30

20

30

Return

20

Return

20

Return

0

10

20

10

Model

Model

Model

0

RNN

RNN+S

0

RNN

RNN+S

-40

RNN

RNN+S

RNN+HN

RNN+HN

RNN+HN

0

500

1000

1500200025003000

Frames (k)

3500

4000

0

500

1000

1500200025003000

Frames (k)

3500

4000

0

500

10001500200025003000

Frames (k)

3500

4000

(a) Walker

(b) Cheetah-Vel

(c) Cheetah-Dir

(d) Ant-Dir

2000

-50

1750

-100

1500

-150

1250

Return

E

-200

1000

-250

750

-300

500

Model

Model

250

0

RNN

RNN+S

RNN+HN

-350

-400

RNN

RNN+S

RNN+HN

0

00000000090000000000000000000000001

Frames (k)

0

10000200003000040000500006000070000

Frames (k)

2500

1400

2000

1200

1000

1500

Return

Return

800

1000

600

400

500

Model

RNN

200

Model

RNN

RNN+S

RNN+S

0

RNN+HN

0

RNN+HN

0

00000000090000000000000000000000001

0

10000200003000040000500006000070000

Frames (k)

Frames (k)

baselines. In particular, RNN conditions on the current state only through its dependence on ϕ , whereas hypernetworks pass in the state again to the policy. Thus, we investigate whether the difference in inputs alone could be the cause of the improvement in performance. To this end, we introduce a new ablation to test the effect of just passing in the state again. Details are below. Second, we inspect how sensitivity to latent variables encoding the trajectory affects performance.

RNN+S. Hypernetworks condition on the current state both through ϕ , which contains information about trajectory, including the current state, and by directly conditioning on the state. Since the hypernetwork conditions on state twice, we test to see the effect of conditioning on the state twice without hypernetworks. We call this ablation RNN+S, which we write as π θ ( a | s, ϕ ) (Figure 8). In an empirical evaluation, we see that while RNN+S does perform favorably relative to RNN alone, RNN+HN still outperforms RNN+S (Figures 9 and 10). In particular, RNN+HN achieves similar returns to RNN+S on Ant-Dir and Cheetah-Vel, and outperforms RNN+S on all other environments, in terms of asymptotic return and sample efficiency. Taken together, we see that the advantage of RNN+HN comes both from the ability to re-condition on state directly and from the hypernetwork architecture. These results confirm that to achieve the strongest performance, re-conditioning on state directly is not sufficient, and that the hypernetwork architecture itself is still critical.

(a) Walker Return

(b) Walker Latent Gradient Norm

1750

Model

20

RNN

1500

RNN+S

RNN+HN

1250

15

RNN+HN_Kaiming

Model

Return

1000

RNN

Norm

RNN+S

R

750

RNN+HN

10

RNN+HN_Kaiming

500

5

250

0

0

0

5000

10000

15000

20000

25000

30000

0

2

4

6

8

Frames (k)

Updates

Latent Gradients. We also investigate how the hypernetworks condition on the trajectory. In particular, we investigate the sensitivity of the output of the network to an intermediate latent representation of the trajectory. For this purpose, we chose to measure the gradient norm of the first hidden layer of the hypernetwork on the Walker environment. We perform this investigation both with the initialization method designed for hypernetworks that we used throughout our experiments, Bias-HyperInit [Beck et al., 2022], and with Kaiming initialization [He et al., 2015], not designed for hypernetworks. We add Kaiming initialization, since Bias-HyperInit ignores trajectories at the start of training [Beck et al., 2022]. First, we confirm the finding of Beck et al. [2022] that Bias-HyperInit is crucial for performance (Figure 11). Second, we see the two models that performs worst, RNN and RNN+HN Kaiming, also have the greatest norm. Moreover, we find that both RNN+HN and RNN+S start with a low gradient norm and then further decrease this norm throughout training, whereas the RNN model increases this norm. We hypothesize that a low norm, i.e., low sensitivity to the latent variable, is crucial for stable training and that the RNN model increases this norm to remain sensitive to the state, since the state is only encoded in the latent for this model.

6 Limitations

As an empirical study of meta-RL, we cannot guarantee that recurrent hypernetworks will improve over every baseline nor on every environment. However, we mitigate this issue by comparing to many baselines and performing many ablations. In particular, we compare to a contemporary taskinference method (VI+HN), design our own baseline which we show to be stronger than others on all grid-worlds (TI++HN), and also include standard methods (TI and TI Naive), in addition to further ablations in the appendix. In as much as an empirical study can, we believe our study demonstrates a significant improvement of the RNN+HN method over existing baselines.

7 Conclusion

In this paper, we establish recurrent hypernetworks as a surprisingly strong method in meta-RL. While much effort has gone into designing specialized task-inference methods for meta-RL, we present the surprising result that the simpler recurrent methods can be easily adapted to outperform the task-inference methods. By combining recurrent methods with the hypernetwork architecture, we achieve a new strong baseline in meta-RL that is both robust and easy to implement. In comparison to existing evidence, we provide much stronger empirical results, afford equivalent computation for tuning to all baselines, and establish recurrent hypernetworks as a strong method. We additionally show that passing the state variable to the policy is a crucial component of this method. Finally, we presented gradient analysis suggesting lower latent gradient norms to play an important role in the performance of meta-RL methods. Since the gradient analysis is preliminary and investigates state and latent variables in isolation, future work could investigate the interaction between these variables. Future work could also analyze the interaction between hypernetworks and other sequence models, such as transformers. We hope these insights, along with a simple and robust method, open the way for the broader use of sample-efficient learning in meta-RL and beyond.

Acknowledgments and Disclosure of Funding

We would like to thank Luisa Zintgraf for her help with the VariBAD code-base along with general advice and discussion. Jacob Beck is supported by the Oxford-Google DeepMind Doctoral Scholarship. Risto Vuorio is supported by EPSRC Doctoral Training Partnership Scholarship, Department of Computer Science Scholarship, and Scatcherd European Scholarship. Zheng Xiong is supported by UK EPSRC CDT in Autonomous Intelligent Machines and Systems (grant number EP/S024050/1) and AWS.

References

Jacob Beck, Kamil Ciosek, Sam Devlin, Sebastian Tschiatschek, Cheng Zhang, and Katja Hofmann. Amrl: Aggregated memory for reinforcement learning. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=Bkl7bREtDr .

Jacob Beck, Matthew Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in metareinforcement learning. CoRL , 2022.

Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. arXiv preprint arXiv:2301.08028 , 2023.

Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations , 2020.

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl2: Fast reinforcement learning via slow reinforcement learning. arXiv , 2016.

Chelsea Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML , 2017.

Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adrià Puigdomènech Badia, Gavin Buttimore, Charlie Deck, Joel Z. Leibo, and Charles Blundell. Generalization of reinforcement learners with working and episodic memory. NeurIPS , 2019.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. In International Conference on Learning Representation (ICLR) , 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Jan Humplik, Alexandre Galashov, Leonard Hasenclever, Pedro A Ortega, Yee Whye Teh, and Nicolas Heess. Meta reinforcement learning as task inference. arXiv , 2019.

Pierre-Alexandre Kamienny, Matteo Pirotta, Alessandro Lazaric, Thibault Lavril, Nicolas Usunier, and Ludovic Denoyer. Learning adaptive exploration strategies in dynamic environments through informed policy regularization. arXiv preprint arXiv:2005.02934 , 2020.

Evan Z Liu, Aditi Raghunathan, Percy Liang, and Chelsea Finn. Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices. In International Conference on Machine Learning , pages 6925-6935. PMLR, 2021.

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations , 2018. URL https: //openreview.net/forum?id=B1DmUzWAW .

Tsendsuren Munkhdalai and Hong Yu. Meta networks. ICML , 2017.

Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can be a strong baseline for many POMDPs. In Proceedings of the 39th International Conference on Machine Learning . PMLR, 2022.

Matt Peng, Banghua Zhu, and Jiantao Jiao. Linear representation meta-reinforcement learning for instant adaptation. arxiv , 2021.

Przewiezlikowski, P. Przybysz, J. Tabor, M. Zieba, and P. Spurek. Hypermaml: Few-shot adaptation of deep models with hypernetworks, 2022.

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcement learning via probabilistic context variables. In International conference on machine learning , pages 5331-5340. PMLR, 2019.

Samuel Ritter, Ryan Faulkner, Laurent Sartran, Adam Santoro, Matthew Botvinick, and David Raposo. Rapid task-solving in novel environments. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=F-mvpFpn_0q .

Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations , 2019.

Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 9301-9312. PMLR, 18-24 Jul 2021.

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems , pages 5026-5033. IEEE, 2012.

Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J. Lim. Multimodal model-agnostic metalearning via task-aware modulation. NeurIPS , 2019.

Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. arXiv , abs/1611.05763, 2016.

Jane X. Wang, Michael King, Nicolas Porcel, Zeb Kurth-Nelson, Tina Zhu, Charlie Deck, Peter Choy, Mary Cassin, Malcolm Reynolds, H. Francis Song, Gavin Buttimore, David P. Reichert, Neil C. Rabinowitz, Loic Matthey, Demis Hassabis, Alexander Lerchner, and Matthew M. Botvinick. Alchemy: A structured task distribution for meta-reinforcement learning. NeurIPS , 2021.

Zhou Xian, Shamit Lal, Hsiao-Yu Tung, Emmanouil Antonios Platanios, and Katerina Fragkiadaki. Hyperdynamics: Meta-learning object and agent dynamics with hypernetworks. In International Conference on Learning Representations , 2021.

Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. URL https://proceedings.neurips.cc/paper/ 2018/file/e1021d43911ca2c1845910d84f40aeae-Paper.pdf .

Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In CoRL , 2020.

Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representation (ICLR) , 2020.

Luisa Zintgraf, Sebastian Schulze, Cong Lu, Leo Feng, Maximilian Igl, Kyriacos Shiarlis, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: Variational bayes-adaptive deep rl via meta-learning. Journal of Machine Learning Research , 22(289):1-39, 2021.

Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. ICLR , 2019.

8 Appendix

8.1 Additional Task-Inference Ablations

In order to select task-inference methods for ultimate comparison in the main body, we evaluated those methods, in addition to further task-inference ablations, on the Grid-World and Walker environments, and then selected the best performing methods. From these results, we chose TI++HN (on GridWorld), and TI and TI Naive (on Walker). See Figure 12. We additionally chose to evaluate VI+HN in the main body, as it is a prior strong method Beck et al. [2022]. A table summarizing the components in all methods can be seen in Table 1. The additional ablations are depicted in Figure 13, with details given below:

VI. This baseline, called VariBAD, is proposed in Zintgraf et al. [2020] and is the same as VI+HN in the main body, but without the hypernetwork.

TI++. This baseline is the same as TI++HN in the main body, but without the hypernetwork. The two additions over TI (denoted '++') are re-use of the multi-task policy parameters as an initialization for the meta-RL policy, and training of task inference over trajectories from pre-training.

TI+HN. This baseline is the same as TI in the main body, but with the addition of a hypernetwork.

BI++HN. Here, the base-network for the task is inferred (denoted 'BI'), instead of the task representation directly. This method is novel and is the same as TI++HN in the main body, but task inference is trained to reconstruct the parameters for the base-network produced by the multi-task hypernet, ϕ ′ , rather than the task representation ( c M or g ):

Note that while the performance of BI++HN is similar to that of TI++HN on Grid-World, TI++HN was chosen as a representative baseline since it is more common to existing methods and BI++HN requires significantly more compute.

(a) Grid-World

(b) Walker

2000

30

1500

Return

20

Model

Return

Model

VI

1000

VI

VI+HN

VI+HN

10

TI_Naive

TI_Naive

TI

TI

++11

500

++1L

0

NH+I1

TI+HN

NH++11

NH++11

NH++18

0

B1++HN

0

500

1000

150020002500

3000

3500

4000

0

10000200003000040000500006000070000

Frames(k)

Frames (k)

(a) TI++

TI

IB

(b) VI

IB

VI

(c) TI+HN

IB

s

s

TI

(d) BI++HN

IB

s

s

TI

8.2 Hyperparameter Tuning

We tune each baseline over five learning rates for the policy, [3e-3, 1e-3, 3e-4, 1e-4, 3e-5], for three seeds each. We use a learning rate for the task inference modules of 0.001, both since varying the learning rate seemed to have little effect (Figure 14) and to provide a fair comparison in terms of computation. Error bars report a 68% confidence interval using bootstrapping.

Before being passed to the hypernetwork or to the policy, depending on the model, the task ( c or g ) are projected down to size 25, as are the concatenation of ( µ, σ ) . This size was chosen to be in the range of the values used by Zintgraf et al. [2020] across environments, but a single value was chosen to be consistent. Additionally, 25 is sufficiently large for a one-hot representation of all discrete task distributions used. However, on Ant-Dir, the projection size is 10. For the state embedding size (passed to the policy) we chose 256 and for the sizes of the MLP policy we chose 256, followed by 128. This corresponds to the "XL" model size in Beck et al. [2022], which was optimal or near optimal where reported. However, for the comparison of RNN, RNN+S, and RNN+HN on Grid-World, these results are reported with a state embedding size for the policy tuned over [5, 25, 125, 625]. For the state embedding size passed to the trajectory encoder, we used 32 for all

(a) Grid-World Pre-Train Time

(b) Grid-World LR

40

30

30

20

Return

Return

20

Model

TI++HN_Oupdates

10

10

TI++HN_100updates

Model

TI++HN_300updates

NH++I1

0

TI++HN_1000updates

0

TI++HN_vaeLRle-02

Multi+HN

TI++HN_vaeLR1e-05

0

500

1000

150020002500

3000

3500

4000

0

500

1000

150020002500

3000

3500

4000

Frames (k)

Frames (k)

experiments, as the default value in Zintgraf et al. [2020]. And for all RNNs, we use a single GRU layer of size 256, as in Beck et al. [2022].

As discussed in the main body, pre-training of the multi-task policy occurs at the expense of the meta-RL policy training time, in order to ensure the same number of samples for training each method. While selecting the time for pre-training does add a hyperparameter that must be selected, we tune this once on Grid-World and then transfer to all other environments, in order to afford as little additional computation as possible to these baselines. Note that while this cutoff affords some extra computation to task inference methods, the recurrent hypernetwork method (RNN+HN), which we show to be the strongest, does not require any such additional tuning.

On Grid-World, we find 100 PPO updates for training the multi-task policy to be optimal. In this environment, each update consists of 960 frames. Interestingly, while the multi-task policy (Multi+HN) requires approximately 500 updates to fully train, training for 100 yielded optimal performance for TI++HN. (We saw similar results for TI, not reported here, except that the random projection provided by 0 updates already gave similar performance to 100.) Note that 100 updates is equivalent to only 20% of the total frames for training the multi-task policy, and it is only 2.4% of the total frames for training the meta-RL policy. This suggests that very little amount of computation is required to get a good task representation, and delaying the start of meta-training significantly is generally not worthwhile. For MuJoCo experiments in the main body, we use 563 updates, since that is the same percent of total frames (2.4%) for Walker and Ant-Dir, and (4.8% on Cheetah-Vel and Cheetah-Dir), which transferred well on initial experiments.

For all other hyperparameters, we default to those in Beck et al. [2022].

8.3 End-to-End and Task Inference Together

While it is possible to combine the end-to-end objective and task inference objective, prior work suggests that interference from competing objectives may be detrimental [Humplik et al., 2019]. Moreover, separating the training objectives can be practically useful, as the combined objectives may require tuning of the relative weighting of the two objectives. Still, we perform investigatory experiments on Grid-World. See Figure 15. We find that adding task-inference objectives to the endto-end meta-RL objective decreased return compared to the end-to-end objective alone (RNN+HN) for every method tested. The same effect can be seen regardless of the architecture (RNN+HN or RNN+S). Unsurprisingly, the performance of the combined end-to-end and task-inference methods falls in between each type of method used independently. Additionally, tuning the relative relative objective weighting had little effect. We experiment with giving 10% (BI10p, TI10p) and 50% (BI50p) weighting to task inference methods, but still observing the same decrease in return. In all these experiments, we use the RNN architecture from the end-to-end RNN methods, meaning that we did not use an information bottleneck, nor the linear encoding layer. In other words, the output of the RNN passed to the policy, ϕ , was directly used as ˆ g or ˆ c and compared to the respective label g or c .

(a) Grid-World Combined Objectives

(b) Tuning Combined Weighting

30

30

Model

Model

Return

20

RNN+HN+BI++

RNN+S

RNN+HN

Return

20

RNN+HN+BI++

RNN+S

RNN+HN

RNN+S+TI

RNN+HN+BI10p++

10

RNN+HN+TI_Naive

10

RNN+HN+BI5Op++

RNN+HN+BI

RNN+HN+TI_Naive

RNN+HN+TI1Op_Naive

0

0

0

500

1000

150020002500

300035004000

0

500

1000

1500

20002500

3000

3500

4000

Frames (k)

Frames (k)

8.4 Meta-World Experiments

We additionally evaluate on Meta-World (ML10) Yu et al. [2020] experiments and show RNN+HN is able to match the performance of the more complicated VI+HN method (Figure 16). Meta-World is a meta-RL benchmark for robotic manipulation. There are 10 non-parametric training tasks and 5 distinct non-parametric test tasks, ranging from pushing a ball to opening a window. Additionally, there is non-parametric variation, e.g. of the goal location, within each task. In this environment, the variance of returns and the computation requirements for training are dramatically greater, so the evidence is not strong is any direction. However, we include the results here since ML10 is a standard benchmark Beck et al. [2023], and since the only experiment ever reported in meta-RL with RNN+HN prior to our paper is on ML10 Beck et al. [2022].

Contrary to our findings, Beck et al. [2022] demonstrated RNN+HN to underperform VI+HN on ML10. However, since that result was not statistically significant, as calculated by Beck et al. [2022], we conduct further experiments on ML10. First, we run more seeds for the VI+HN model in their experiment. (Their experiment is the same as ours, but we use a latent encoding size of 25, instead of 10, for the projection of ( µ, σ ) , to be consistent with the majority of our experiments, and we run for slightly fewer frames due to the intensive computation requirements.) Using different seeds from Beck et al. [2022] resulted in worse performance, indicating that the results truly were insignificant due to variance. Beck et al. [2022] calculate a mean return of 23.9 for VI+HN from the returns: [12.48, 25.69, 33.61]. In contrast, we calculate the final mean return is 18.35 for VI+HN, with is an average over three seeds, with final returns: [32.80, 3.32, 18.93]. Comparing to the returns for RNN+HN from Beck et al. [2022], [11.31, 3.37, 27.77], it is difficult to draw strong conclusion from this experiment.

Additionally, we run our own experiments (with latent size 25) and compare RNN+HN to VI+HN. Contrary to Beck et al. [2022], we find that the final return of RNN+HN is higher than VI+HN. RNN+HN has a final average test success percent of 17.22, compared to 13.87 for VI+HN. However, the variance is still too large to draw firm conclusions. Additionally, we find that the learning curves for RNN+HN and VI+HN overlap consistently throughout training, and the confidence interval for RNN+HN lies almost entirely within that of VI+HN. Note that while tuning hyper-parameters, one of the seeds for the largest learning rate (3e-3) of RNN+HN encountered numerical instability and was ignored. However, 1e-4 was selected as the optimal learning rate, so this is unlikely to affect results, and if it did, would only increase the returns of RNN+HN.

While these results are not sufficient to conclusively show superiority on ML10, we do show superiority on many other environments. Additionally, these results do show RNN+HN to match the performance of VI+HN. Matching existing task-inference baselines on ML10 is significant, since it demonstrates that complicated task-inference methods are not necessary to be competitive.

(a) Additional Seeds on Original ML10 Experiment

(b) Our ML10 Results

0.35

Model

0.25

Model

0.30

VI+HN

VI+HN

0.20

RNN+HN

(test)

0.25

(test)

Rate

0.20

Rate

0.15

uccess

0.15

Success

0.10

0.10

S

0.05

0.05

0.00

0.00

0

20000

40000

Frames (k)

60000

80000

100000

0

10000

20000

3000040000

50000

60000

70000

Frames (k)

8.5 Gridworld Details

Grid-World consists of a five by five grid with a goal location in one of the grid cells. The agent starts in the bottom left corner of the grid, and then has 15 steps in an episode to find the goal and remain there. The agent receives 1 reward for each timestep at the goal and -0.1 reward for all other timesteps. The goal location is held constant for each meta-episode, which consists of 4 regular episodes. In Grid-World Show, the goal position is visible to the agent at the first timestep of each episode. In Grid-World Dense, the agent receives and observes a reward equal to the Manhattan distance to the goal location.

8.6 MuJoCo Details

We evaluate on all four MuJoCo Todorov et al. [2012] environments used by Zintgraf et al. [2020]. All environments require legged locomotion. All episodes terminate after 200 timesteps. Meta-episodes in Walker consist of two regular episodes, whereas meta-episodes in the rest consist of one regular episode.

In Cheetah-Dir, the agent controls a robotic cheetah morphology by outputting a control torque for each of six joints on the morphology. The task is to run forward or backward with as large a velocity as possible. The agent observes the position, angle, and velocity of each body part (17 dimensions in total) and is given a positive reward for its velocity in the direction given by the task and a negative reward for control costs (specifically, 5% of the magnitude of the action vector).

In Cheetah-Vel, the agent controls the same robot, but instead of the the positive reward component for running on the selected direction, the agent receives a negative reward component equal to the absolute difference to a target velocity, which is sampled uniformly from [0,3].

In And-Dir, the agent controls a robotic ant morphology by outputting a control torque for each of eight joints on the morphology. The task is to run forward or backward, with as large a velocity as possible. The observations are 27 dimensional in this case, and the agent is given a positive reward for its velocity in the direction given by the task and a negative reward for control costs (specifically, 5% of the magnitude of the action vector). Additionally, the agent is given a negative contact reward (.05% of the magnitude of the external forces on each joint) and a positive survival reward (1 per timestep). Episodes in this environment additionally terminate if the body falls outside of a predefined height.

In Walker, the agent controls a two-legged torso morphology and outputs a control torque for each of six joints on the morphology. The observations are 17 dimensional for this environment. The tasks are defined as uniform samples of 65 different physics coefficients (e.g. body mass and friction) for the simulation. The agent is given a positive reward for the forward velocity, a positive reward of one reward per timestep, and a negative reward for the control costs (specifically, 0.1% of the magnitude

of the action vector). Episodes in this environment additionally terminate if the body falls outside of a predefined height or rotation range.

8.7 Compute

Experiments were run on four to eight machines simultaneously each with eight GPUs, ranging from NVIDIA GeForce GTX 1080Ti to NVIDIA RTX A5000. Each model was tuned over five learning rates with three seeds per learning rate. Each experiment took between 4 hours for grid-worlds and 3 days for MuJoCo experiments. In total, there were approximately 360 experiments for MuJoCo and 270 for grid-worlds in the main results. With about 5 experiments on average per GPU, this constitutes roughly 5,400 GPU hours.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigate the performance of a specific type of architecture, RNNs coupled with hypernetworks, in meta-reinforcement learning. Meta-RL aims to address the sample inefficiency of RL algorithms by learning to perform few-shot learning when given a distribution of related tasks for meta-training.\n\nThe authors note that while specialized and complex meta-RL methods have been proposed in the literature, recent work suggests that using an off-the-shelf sequential model, such as an RNN, trained in an end-to-end manner can serve as a strong baseline. However, the supporting evidence for this claim has been limited. The paper presents an extensive empirical investigation to address this gap. While RNNs can achieve strong performance in meta-RL, the study finds that the use of hypernetworks is crucial in maximizing their potential. Interestingly, when combined with hypernetworks, the simpler recurrent baselines outperform existing specialized methods and establish themselves as state-of-the-art (SOTA) on standard meta-RL benchmarks.\n\nSTRENGTHS:\nS1: The paper is extremely clear\n\nS2: The authors make sure to tune each methods, which is something I do not see often enough \n\nS3: Coupling RNNs and hypernetworks is an idea that is novel in the field of meta-RL, and the attempts at understanding why they might be outperforming other baselines are interesting\n\nWEAKNESSES:\nW1: While the hyperparameter tuning was fairly done in terms of computation budget, it is to me not clear this strategy is the most relevant. I can imagine that many of the complex methods have many more hyperparameters to tune than the simple methods that you propose, and an excessively small search grid could unfavorably disadvantage the former. \n\nW2: The empirical investigation is, as noted by the authors, quite limited. In that regard, more empirical evidence from different benchmarks would make the paper stronger.\n\nQUESTIONS:\nQ1: Line 128, I could not understand why the predicted context is not directly used to condition the policy. From my understanding, the context should contain enough information to disambiguate the tasks, and the concatenation of the state and context should be Markovian. Could you provide an intuition why intermediate representations are used?\n\nQ2: The RNN-S is an interesting experiment. One additional hypothesis for why RNN+HN might work well is because it allows for some multiplicative interaction between the context and the state. What types of non-linearities were used in the network? Have you used activation units, such as Gated Linear Units (GLU) [1] which allows for multiplicative interactions between the neurons in the RNN-S?\n\n[1]: \"Language Modeling with Gated Convolutional Networks\", Dauphin 2016\n\nLIMITATIONS:\nThe limitations I can think of have been addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores how to approach the problem of meta-reinforcement learning by using hypernetworks. In particular, the authors propose to employ an RL2-like scheme, where instead of producing a vector $\\phi$, the recurrent model outputs a set of neural network weights. They dub the method RNN+HN. They introduce a set of task inference-based baselines with and without the usage of hypernetworks. Subsequently, they show that the simple RNN+HN approach outperforms more sophisticated baselines in gridworld environments and in Mujoco. Finally, they perform an ablation study by using RL2 with state-conditioning.\n\nSTRENGTHS:\nOverall, I find the paper interesting and the results quite convincing. However, the authors' claims are very strong -- stating that the proposed method is state-of-the-art in the wide field of meta RL. Although the experiments do show that the method works quite well, I do not think it is enough to support this bold claim. As such, for now, I am going with \"borderline reject\", but I would be inclined to increase the score if the authors either extended the empirical evaluation or toned down the claims presented in the paper.\n\nStrengths:\n- The paper introduces a nice, conceptually simple idea that provides substantial improvements. Parameterizing the policy using a neural network with weights generated by a hypernetwork is an elegant idea.\n- The empirical results are in general quite good, the RNN+HN method outperforms most of the baselines in many cases.\n- The paper is well written.\n- The appendix includes additional ablation studies and more baselines.\n\nWEAKNESSES:\n- The critical weakness of the paper is the mismatch between the claims and the empirical evaluation. Namely, the authors state multiple times that their method is state-of-the-art in the field of meta-RL, but the experiments do not support that claim sufficiently:\n    - There are not enough external baseline methods. The baselines the authors use are inspired by existing methods but do not correspond exactly to what has been proposed in the literature previously (e.g. TI-naive, TI). Additionally, the authors omit existing established works such as MAML-based algorithms and PEARL. As they say, they \"exclude the latter methods since estimation of a policy gradient in policy-gradient approaches requires more data than in our benchmarks\". That is, in general, a reasonable assumption, but seems too strict when claiming state-of-the-art.\n    - The authors support their claim by saying that their method outperforms previous SOTA [1, 2], but I'm not convinced that these previous works are still SOTA as of now. Some of the recently published works also show very good results [3, 4].\n    - Currently, the authors only consider two sets of environments: gridworlds and Mujoco continuous control. I think the empirical evaluation should be extended to include more complex environments such as Meta-World, RLBench, Atari, or DeepMind Alchemy. I think that environments with high-dimensional state spaces (e.g. images) would be interesting to test the limits of the proposed method.\n    - There are some environments where RNN+HN falls behind (e.g. Walker, Cheetah-Vel). This is not a grave problem by itself, but it makes the problem of having relatively few benchmarks even more problematic -- how will this method scale up to other environments?\n    - Why the RNN baseline only appears in a single environment in Figure 8 and Figure 9? I think it's an important baseline that should be included.\n\n[1] Zintgraf, Luisa, et al. \"Varibad: A very good method for bayes-adaptive deep rl via meta-learning.\" arXiv preprint arXiv:1910.08348 (2019). \\\n[2] Beck, Jacob, et al. \"Hypernetworks in meta-reinforcement learning.\" Conference on Robot Learning. PMLR, 2023. \\\n[3] Melo, Luckeciano C. \"Transformers are meta-reinforcement learners.\" international conference on machine learning. PMLR, 2022. \\\n[4] Chalvidal, Mathieu, Thomas Serre, and Rufin VanRullen. \"Meta-Reinforcement Learning with Self-Modifying Networks.\" Advances in Neural Information Processing Systems 35 (2022): 7838-7851.\n\nQUESTIONS:\nMy main suggestion related to the first point in the Weaknesses section above, i.e. extending the empirical evaluation or reducing the state-of-the-art claims.\n\nLIMITATIONS:\nThe author discuss limitations sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper investigates the performance of recurrent neural networks in meta-RL. They suggested that a current neural network can achieve strong performance with hypernetwork. They compared this method with numerous baselines on several meta-RL benchmarks and found that the recurrent baselines along with hypernetwork could achieve SOTA performance.

STRENGTHS:
This paper introduced a novel assertion that a recurrent method, when combined with a hypernetwork, can achieve SOTA performance in meta-reinforcement learning. The authors substantiated this claim with rigorous empirical experiments demonstrating superior performance. Furthermore, they conducted an analysis elucidating the reasons behind the method's impressive performance. The paper is well-written and clearly structured, which makes it easily comprehensible for the readers.

WEAKNESSES:
1. The authors could enhance the comprehensiveness of the study by testing their algorithm on a wider range of environments.

2. The final analysis segment of the paper would benefit from further development to unequivocally ascertain why the proposed method can achieve state-of-the-art performance.

3. The paper lacks ablation studies for the different settings of hypernetwork component, leaving its individual contribution to the overall results unclear.

4. The format of the reference is wrong

QUESTIONS:
1. Why in the Cheetah environment, RNN+HN is worse than RNN+S?

2. Please offer more ablation studies of the hypernetwork component.

LIMITATIONS:
The authors have discussed the limitations of this paper in the last section.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes to modify a common meta-RL baseline, in which a meta-learned RNN provides the policy network with an encoding of the trajectory. The proposed recurrent hypernetwork instead lets the RNN predict the weights and biases of the policy network. This is shown to be a strong model in comparison with several variations of RNN-based and task-inference meta-RL approaches.\n\nSTRENGTHS:\nI think the proposed approach is novel and the idea is described in sufficient detail.\nThe experimentation affords equal computational resources for hyperparameter optimization of all models. The method seems to perform better in 6 out of 7 considered environments.\nThe ablation studies adequately address some questions that I thought of while reading the paper.\n\nWEAKNESSES:\n1. The implementation of the RNN and RNN+HN models could've been described in a bit more detail. It is for example not clear what the meta parameters $\\theta$ correspond to in the RNN baseline, since line 108 says that $f$ and $\\pi$ use distinct parameters from $\\theta$. Is this sentence wrong and $\\theta$ are the RNN's parameters? I could also not find all hyperparameters of the architectures, such as the type and layer sizes of the RNNs.\n1. For the strong claim that recurrent hypernetworks are SOTA in (all of) meta-RL, the presented experiments seem rather limited. What about experiments on RLBench or Meta-World? Also, there are many more categories of meta-RL methods to be considered than RNN-based and task-inference methods. The manuscript can benefit greatly from a clear justification of this strong claim.\n1. I could not find an indication of how exactly the performance curves were generated, e.g. how many seeds per model, what exactly does the shaded area convey, etc.\n\nQUESTIONS:\n1. References to figures such as \"See Figures 3 and 4\" in line 123, I'd put in parentheses directly in the preceding sentence or say something like \"See Figure X for a diagram of Y\".\n\n## Acknowledgement of rebuttal\nI have read the rebuttal and reviews and engaged in discussion with the authors. The authors have mostly addressed my concerns by removing the general claims of the proposed method being state-of-the-art in meta-RL, by providing some clarification on the architecture and experimental setup, and by providing additional experimentation that support some of the design decisions. I have accordingly increased the score.\n\nLIMITATIONS:\nThe authors mention that they cannot guarantee that the proposed method will improve over every baseline nor over every environment. This does not feel like a careful look at model details and how they might affect adaptation to different task variations. One general criticism of many meta-RL methods is that they are evaluated on very narrow task distributions. Maybe the authors could focus on that. The recent meta-RL survey by Beck et al. [1] might give some ideas for discussion.\n\n## References\n1. Beck, Jacob, et al. \"A survey of meta-reinforcement learning.\" arXiv preprint arXiv:2301.08028 (2023).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is an empirical investigation that tests the performance of different variants of three methods for meta-RL: RNN that provides task information implicitly from previous trajectories on the same MDP, TI that trains a task representation with VAE manner, and VI that is the baseline Varibad [1]. On mujoco environment, the paper finds that RNN with output being a hypernetwork that outputs the parameter for the policy works the best, and thus re-establish [2] RNN+hypernetwork as the state-of-the-art method for meta-RL. \n\n**References:**\n\n[1] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In International Conference on Learning Representation (ICLR), 2020.\n\n[2] Jacob Beck, Matthew Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta-reinforcement learning. In CoRL, 2022.\n\nSTRENGTHS:\n1. Many variants are tested in the environment. There are 12 variants tested in the paper, which gives a thorough analysis on the effect of using hypernetwork for each method; in addition, the effect of \"conditioning on the state twice\" is also considered, which makes the conclusion that hypernetwork is crucial more rigorous.\n\n2. The figures appended in the paper does a well job in conveying the architecture of each method.\n\nWEAKNESSES:\n**1. The experiment results are not convincing enough.** \n\na) The method proposed (line 184) in the paper, RNN+hypernetwork, is already proposed in prior work [1]. More specifically, RNN alone is equivalent to RL2 [3] as line 106 suggests, and RL2+hypernetwork is already tested in [1], which is inferior to Varibad [2]+hypernetwork. Thus, the key contribution of this paper is not proposing new method, but to overthrow the previous conclusion that Varibad+hypernetwork is better. However, this paper only tests gridworld and mujoco environments, and on both environments RNN+HN is only marginally better than VI+HN (which in [1] also has multiple methods that works similarly well, indicating that mujoco alone is not a strong benchmark). Most importantly, the metaworld (ML1 and ML10) environment, which is the decisive evidence that Varibad+hypernetwork is better than RL2+hypernetwork, is missing in this paper. With such result missing, it is hard for the readers to be convinced that the result is the other way around from that in [1].  \n\nb) The baselines, though with many variants, do not cover enough areas for the claim of the state-of-the-art. Reviewers of [1] have already pointed out that the state-of-the-art status of Varibad [2], even before the presence of [1], is questionable. Also, while it is true that the meta-RL method can be briefly summarized in policy gradient, implicit task representation and task inference as the paper suggests (line 61-63), in Table 2 of [4] there are many branches within each of the direction. For example, what if transformer instead of RNN is used [5]? Is policy gradient method necessarily worse than the other two branches? For a state-of-the-art method, those methods also need to be considered.\n\n**2. The delivery of the paper's idea can be improved.**\n\na)  There is no motivation stated about why hypernetwork is used. The paper only states that \"we present the key insight that the use of a hypernetwork architecture is crucial ...\" (line 36-37) and \"hypernetwork has never been widely evaluated in meta-RL\" (line 81-82). However, there is no intuitive explanation about why hypernetwork is useful for meta-RL and should be considered in the first place. While there is explanation in [1] about preventing degeneration of multi-tasking, the paper should be self-contained and inform the readers about why hypernetwork, the most important component in the finally proposed RNN+hypernetwork method, is considered for meta-RL.\n\nb) The method section can be modified to convey the idea more clearly. Currently, it is hard to identify which method is proposed (line 184) by the paper, and it is unclear for the first-time readers about why multiple methods are listed in the method section (and most of them is not proposed by this paper), instead of the usual paradigm where each key component is listed in a subsection. Since the methods are not novelly proposed by this paper, they can be put into experiment setup section (with more emphasis that this is an empirical study paper rather than conventional paper that proposes a new method, e.g., add a contribution summary at the end of introduction section); or, a summary could be added in the front of the method section, to tell the readers that all those methods listed below are tested in the experiment section, and in each section we discuss one type of method tested. A table summary would also be very helpful.\n\nc) There is no detail of the environment settings except gridworld. What is the cheetah-dir environment? What are the definitions of state, action and reward, and how are the MDPs in meta-learning differs? What reward can be considered expert-level performance? While they can be found in prior work, the paper should be self-contained and the settings should be attached in the appendix to help the reader form a better intuition about the environment.\n\n**3. Other minor problems:**\n\na) Some experiment results are missing, for example, RNN in Fig. 8b, 8c, 9b, 9c and 9d;\n\nb) Fig. 10 is never referred to in the paper; reference to Fig. 10 should be added in \"latent gradients\" paragraph.\n\nc) The x-axis of walker environment is not aligned with other mujoco environments.\n\nd) The name, VariBad, should be mentioned in VI and VI+HN to more clearly show the connection of the method tested to existing baselines.\n\n**References:**\n\n[1] J. Beck et al. Hypernetworks in meta-reinforcement learning. In CoRL, 2022.\n\n[2] L. Zintgraf et al. Varibad: A very good method for bayes-adaptive deep rl via meta-learning. In ICLR, 2020.\n\n[3] Y. Duan et al. $RL^2$: Fast reinforcement learning via slow reinforcement learning. In arXiv, 2016.\n\n[4] J. Beck et al. A survey of meta-reinforcement learning. In arXiv, 2023.\n\n[5] L. Melo. Transformers are Meta-Reinforcement Learners. In ICML, 2022.\n\nQUESTIONS:\nBesides those listed in the weakness section, I have one question: why the performance of RNN+HN in Fig. 9a and Fig. 10a is different?\n\nBelow are my suggestions for this paper:\n\n1. Modify the introduction and method section of the paper such that it is more clear that this is an empirical study instead of proposing a novel method, and add contribution summary at the end of the introduction. Add a table that summarizes the methods tested in the paper.\n\n2. Add environment details of the mujoco environments in the appendix.\n\n3. Compare RNN+HN to more methods summarized in [1];\n\n4. Compare RNN+HN with VI+HN (and other baselines) on metaworld and other more challenging environment;\n\n5. Fix other minor issues mentioned in the weakness section.\n\n6. I strongly advice the authors to open source their code upon acceptance / next submission. The author could also consider adding license statement in the appendix.\n\n**References:**\n\n[1] J. Beck et al. A survey of meta-reinforcement learning. In arXiv, 2023.\n\nLIMITATIONS:\nThere is an independent limitation section in the paper, which I think generally discusses an important limitation of the work. Though, as suggested in the weakness section, I think currently the limitation is not sufficiently mitigated. There is no negative societal impact discussed in the work, which I would encourage the readers to add; though the work is still far from application, automated control itself brings potential challenge, such as job loss, to the human society.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}