On the Effectiveness of Fine-tuning Versus Meta-reinforcement Learning 
Authors: Zhao Mandi
Pieter Abbeel
Stephen James
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Fine-tuning
Multi-task Learning
Meta Reinforcement Learning
Meta-RL

Abstract: Intelligent agents should have the ability to leverage knowledge from previously learned tasks in order to learn new ones quickly and efficiently. Meta-learning approaches have emerged as a popular solution to achieve this. However, meta-reinforcement learning (meta-RL) algorithms have thus far been restricted to simple environments with narrow task distributions and have seen limited success. Moreover, the paradigm of pretraining followed by fine-tuning to adapt to new tasks has emerged as a simple yet effective solution in supervised learning. This calls into question the benefits of meta learning approaches also in reinforcement learning, which typically come at the cost of high complexity. We therefore investigate meta-RL approaches in a variety of vision-based benchmarks, including Procgen, RLBench, and Atari, where evaluations are made on completely novel tasks. Our findings show that when meta-learning approaches are evaluated on different tasks (rather than different variations of the same task), multi-task pretraining with fine-tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation. This is encouraging for future research, as multi-task pretraining tends to be simpler and computationally cheaper than meta-RL. From these findings, we advocate for evaluating future meta-RL methods on more challenging tasks and including multi-task pretraining with fine-tuning as a simple, yet strong baseline.

On the Effectiveness of Fine-tuning Versus
Meta-reinforcement Learning
Zhao Mandi
Pieter Abbeel
Stephen James
(mandi. zhao, pabbeel, stepjam)kberkeeleycco University of California, Berkeley
Abstract
Intelligent agents should have the ability to leverage knowledge from previously learned tasks in order to learn ones quickly and efficiently. Meta-learning approaches have emerged as a popular solution to achieve this. However, meta- reinforcement learning (meta-RL) algorithms have thus far been predominately validated on simple environments with narrow task distributions. Moreover, the paradigm of pretraining followed by fine-tuning to adapt to new tasks has emerged as simple yet effective solution in supervised and self-supervised learning. This calls into question the benefits of meta-learning approaches reinforcement learn- ing, which typically come at the cost of high complexity. We therefore investigate meta-RL approaches in variety of vision-base based benchmarks, including Proegen, RLBench, and Atari, where evaluations are made on completely novel tasks. Our findings show that when meta-learning approaches are evaluated on different tasks (rather than different variations of the same task), multi-task pretraining with fine- tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation. This is encouraging for future research, multi-task pretraining tends to simpler and computationally cheaper than meta-RL. From these findings, advocate for evaluating future meta-RL RL methods on more chal- lenging tasks, and including multi-task pretraining with fine-tuning as a simple yet strong baseline.
Procgen
RLBench
Atari
Train
Roward
60
+
0.05
Test
5K
10C
Endrrnmnnt Skk
Envicenent Stop
Finetune
Reptile
RL2
PEARL
Figure I: Meta reinforcement learning (meta-RL) algorithms have predominantly been validated on simple environments with narrow task distributions. Our study investigates meta-RL RL algorithms much wider task distributions across benchmarks, and concludes that multi-task pretraining with fine-tuning
tasks performs equally well, better, than meta-RL. Each of the three plots shows the average performance across tasks in the corresponding benchmark.
Project Website:
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Introduction
One of the major gaps between human and machine intelligence is the sample efficiency of learning. Whereas humans can leverage past knowledge to learn a new task from a few examples, current machine learning systems often require a large amount of data and supervision to achieve even single task. To bridge this gap, meta-learning has become a popular approach - it uses many tasks meta-train an optimal learning strategy, which enables few-shot generalization on a test task.
Meta-learning methods have had the most success in supervised learning settings [1-3], specifically few-shot image classification, where the goal is to learn a classifier to recognize unseen classes during a test-time training phase with limited labeled data. Recent work has found that variations of simple pretraining and fine-tuning can perform equally as well more complex meta-learning approaches [4-7]. However, within the realm of reinforcement learning, simple pretraining and fine-tuning is not known to out perform meta-reinforcement learning (meta-RL.. Our hypothesis for this discrepancy is simple: the computer vision (CV) community evaluates their approaches on distinet test tasks (e.g. classifying dogs, cats, and birds), while the meta-RL community evaluates on variations of the same train-time tasks; for example, varying transition dynamics (e.g. different friction parameters) or varying reward functions (e.g. running forward v.s. running backward) are better categorized as variations rather than different tasks, as discussed in recent work [8, 9]. Variation adaptation is inherently easier than task adaptation, and does not paint full picture of the shortcomings of meta-RI.. Moreover, most meta-RI. methods (with a few exceptions, discussed in 7) have been studied in fully observable settings or with shaped rewards [10-12], neglecting more realistic real-worl scenarios, where rewards are often sparse, and observations are high-dimensional (e.g. images, point-clouds, etc).
Evidently, there is gap in the literature, where a large-scale study would be well placed to analyze the setting of vision-based meta-RL across a fruly diverse set of tasks. We use 3 existing RL benchmarks: Proegen [13], RLBench [8], and Atari Learning Environment (ALE) [14, 15] each offers a diverse set of distinct tasks that we use for train and test. For example in RLBench [8], an agent could be trained to pick up cups, take a USB out of a computer, and reach target locations, while at test time, adaptation would be evaluated on completely unseen tasks, such as lifting blocks and pushing buttons.
We investigate three prominent meta-RL. algorithms of differing paradigms: Reptile [11] a gradient- based method, PEARL [12] context-based based method, and RL2 [16] an LSTM-based method. Results from this study are enlightening: multi-task pretraining, followed by fine-tuning on novel tasks, performs equally as well, or better, than all three meta-RL algorithms, while being much simpler and less computationally expensive to train. In light of this, we advocate for future research to shift towards more challenging benchmarks, and include multi-task pretraining with fine-tuning as a simple, yet strong baseline.
To summarize, the key contributions of our study are as follows:
We show that multi-task pretraining followed by fine-tuning on novel tasks performs equally as well, or better, than common meta-RL baselines on vision-based environments.
We present the first lcage-scale multi-tash and meta-RL study three existing benchmarks: cross-level adaptation on Procgen, cross-task adaptation on RLBench, and cross-game adaptation on Atari.
Within RI Bench, we show that large-scale multi-task pretraining can overcome sparse rewards on unseen test tasks and perform significantly better than training from scratch.
2 Preliminaries
Reinforcement Learning. Reinforcement learning (RL) assumes access to an agent that interacts with an environment which there are states S, actions a A. and reward function R(s, a.). where t represents the current time step. The agent must discover policy æ that maximizes the expectation of the sum of discounted rewards, i.e. Σ, 7' R(s, ar)], where future rewards are discounted by factor 7 E [o. 1). Each policy æ has a corresponding Q-value function Q(s, a), which defines the agents expected return when following the policy after taking action a in state s.
Meta- Reinforcement Learning. Meta-reinforcemeen learning (meta-RI.) aims to quickly adapt to any task via RI. Given distribution of tasks p(T), where each sampled task T. E p(T), is stand-alone Markov Decision Process (MDP) T. (S, A, P. 7, R), partially observable MDP

---

(POMDP) T: - (R. O), in the case where an agent receives observations E O rather than ground-trutt states s S. Given a task T, the agent allowed collect small amount of data DT and use it to adapt the policy to obtain γт with an RI. algorithm. Meta-RI. training aims to find an optimal initial policy or exploration strategy that maximizes the expected return of its adapted policies across all tasks within a limited number of steps: BT~p(T), (
[ R(s,, a,)].
In-distibulion Variation acieptation
Out-of-distibuton Variation adaptation
Task adaptation (this work)
Assumption on Task Distribution. We as- sume no ambiguity exists between any two tasks in the train or test sets. Namely, all tasks have distinct, non-overlapping state-action spaces: TT, E p( j. S; x A, x A, Ø. An agent should be able infer which task to perform based on single-step observation (e.g.
Train
whether the current task is playing Breakout or Pong is always immediately distinguishable from a single observation). As result, our multi-task pretraining baseline does not need a separate indicator to determine which task to complete. We remark that our assumption is dif- ferent from most prior work in meta-RL, which often assumes the difference in "tasks" only oc- curs in transition dynamics (e.g. two tasks are both half-cheetah running, but with different friction parameters) or the reward function (e.g. running forward v.s. running backward)- therefore categorize such assumptions as "vari- ation adaptation", and not "task adaptation", to be better distinguished from our work. Please see Figure for illustration.
Test
An illustrative comparison hetween the task assumptions from prior works' (in-distribution out-of- distribution adaptation) and from ours (lask adaptation). In our assumption, we emphasize both the high diversity of training tasks which should lead to generalizable skills, and strictly unscen test-time tasks which poses a more challenging adaptation problem than simple re- trieval trained skills.
Figure 2:
2.1 Overview of Compared Algorithms
Reptile [11] is first-order, gradient-based meta-learning algorithm. It finds initial model pa- rameters that can be quickly adapted to any task via gradient descent. Meta-training alternates between inner loop which performs multiple gradient steps single task, and an outer loop which mixes the updated parameters from multiple tasks. Reptile mathematically similar to first-order model- aodel--agostic meta-learning [3] (MAML) while being easier to implement, and achieves comparable results as MAML in both supervised [11] and imitation learning [17] settings. We choose Reptile for its simplicity and applicability to both on on-policy and off off-policy RL algorithms.
PEARL [12] is a context-based, off-policy meta-RL. algorithm. meta-trains a policy that con- ditions context variables, which are encoded from small amounts of an agent's 's experiences and are used to perform online probabilistic filtering to infer how to solve a new task. At test time, adapts to new task by alternating between 1) collecting new experiences to update the context and 2) adjusting its policy according to the updated context.
RL [16] is eecurrence-based policy-gradient meta-RL algorithm. It represents the RL policy with a recurrent neural network (RNN) that embeds agent experiences from multiple "trials" in single task. with the aim of encoding RL update rule into the RNN weights. At test time, it adapts by rolling out an agent in a new task, which updates the RNN hidden states and improves the policy to success after few trials.
Multi-task Training As a baseline, we run multi-tast reinforcement learning (MTRL), which simply learns to achieve all the training tasks simultancously with a single RL policy. At test time, the agent adapts by finetuning the policy on a new task using the same RL algorithm as training.
Base RL Algorithms Each of the above methods is applied to a base RL algorithm that are different across benchmarks. For our Progen experiments (Section 3), we use PPO [18]. For our RLBench experiments (Section 5), we use C2F-ARM [19]. For our Atari experiments (Section 7), we use Rainbow DQN [20] as the base algorithm. We choose the use the base algorithm that commonly used and well-tuned for each environment, and provide more detailed justifications for each choice
3

---

in their respective sections. See Algorithms 1 and 2 in Section 6 for unified overview of how the meta-RL algorithms are implemented.
All experiments in the study are trained and evaluated on a maximum of 8 RTX A5000 GPUs, each with 24GB of memory. Unless otherwise indicated, architectures as used in prior work; full details are given in the appendix.
the same hyperparameters and
3 Procgen Experiments
Procgen Benchmark [13] is suite of procedurally- generated game-like environments designed for studying generalization in deep RL, where agent is typically trained simultaneously on a set of levels and evaluated on unseen levels. We rank Procgen in this study, because both the embodiment and task ob- jects are consistent across tasks, while only the colors and layouts vary.
Train
10000 levels
1000 levels
the easiest benchmark
100 levels
Prior work has explored improving generalization Proc- gen by training on a large amount of levels [13], adding augmeettgtttttioonn to visual inputs [21, 22], knowledge transfer during training [23], and self-supervised world models [24]. However, beyond zero-shot generalization, few prior works use Procgen to evaluate adaptation meta-RI, with exception being Alver et al. [25], which showed that RI.² failed to generalize on simplified Procgen games.
Test
choose one representative Procgen game, namely Coin- run, where agent is tasked avoid obstacles and reach a golden coin. The game contains multiple "levels", each of which has unique color theme or layout. We up to 10,000 levels for training, and held-out 20 levels for testing. 100 million environment steps
Figure 3: In Procgen experiments, each agent trained on various number of Coinrun levels and tested on 20 unseen levels.
used for all pretraining runs, which amounts to 1 1500 PPO iterations.
Then the final model checkpoint is used test adaptation
the environment is run in parallel on 256 threads and each PPO iteration uses 256 steps.
3.1 Training Setup
We use PPO [18] as the base RL algorithm and use the same image encoder as IMPALA [26]. which has been shown achieve competitive performance on annle-game settings in Procgen [13, 27]. Note that PEARL is excluded from this section because it is based on off-policy RL algorithms.
Reptile-PPO incorporates the inner-outer loop in Reptile [11] with PPO. For each iteration in the inner loop, the environment fixed to one randomly sampled task level, where rollouts are collected and used to perform & iterations of batched gradient updates (we use k = in our experiments). Following that, the outer loop takes the updated policy parameters, and performs a soft parameter update. This process is repeated iteratively.
RL2-PPO combines an LSTM with PPO as proposed in [16]. We modify the Coinrun environment to re-sample the same level for multiple episodes, such that one original environment episode is treated as one "trial", and each RL" episode contains multiple "trials". Until each multi-trial episode terminated, the agent's 's past rollouts are concatenated and fed into LSTM layer. The image observations are first encoded by the IMPALA (following [13]) and flattened into 256-dimensional latent embeddings, then the latent embeddings concatenated with action, reward, and done signals. The concatenated vectors are embedded again before being fed into LSTM hidden layers.
MT-PPO jointly trains PPO multiple levels of Coinrun. The environment randomly samples from fixed number of training levels, and the policy updated with a mixture of rollouts that are collected from multiple environment levels.

---

3.2 Test-time Adaptation Setup
For Reptile-PPO and MT-PPO, trained agent is finetuned with vanilla PPO for 2 million envi- ronment steps on each test level. For RL'-PPO, the LSTM hidden states are reset, and the agent rolled-out each new level for 2 million environment steps without gradient updates. We also compare adaptation with training from scratch, where one agent trained for each test level with PPO for 2 million steps. We use total of 20 held-out levels for testing, and results reported by averaging across all 20 levels.
3.3 Results
Results for testing on unseen levels are shown in Figure 4. We first remark that finetuning achieves the best performance at adaptation new levels both in terms of sample efficiency and final performance, and is shown consistently across varying number of training levels. No- tably, Reptile-PPO is also able to improve per- formance on test tasks, but the finetuning perfor- mance is worse than MT-PPO, which suggests Reptile-learned parameters provides less adaptable initialization than the simpler multi- task trained parameters. Figure also shows that increasing the number of training levels im- proves zero-shot performance on MT-PPO (and less so for Reptile-PPO), which is consistent with the aforementioned prior work that studys generalization on Procgen. Expanding the training set also benefits finetuning and provides clear advantage over training from scratch, which is encouraging news for simple scaling of training tasks as an alternative to designing computationally complex meta-training algorithms.
Pretrain
Pretrain 1000
Pretnain 10000
10
10
19
Reaad
1M
2M
1M
2M
10
1M
7M
Enninommnt Step
Enn Scemarer Sap
Envirnmmnt Elep
Stathh
MT PPO
Regtie PPO
IRL PPO
Figure 4: pretrained Procgen(Coinnus agents on un- seen levels. Each sub-figure corresponds pretraining on different number of levels, ranging from 100, 1000, 10000. Solid lines are average over seeds, with shaded regions representing standard errors.
RL. does not improve performance on unseen levels, meaning that the updated hidden state of the RNN fails to adapt sufficiently while testing new levels. This is consistent with [25], where RL fails to adapt even after simplifying the level such that the coin is visible at the beginning of each trial.
4 RLBench Experiments
RLBench is a vision-based manipulation bench- mark and learning environment, with a focus on sparse rewards, multi-task learning, and meta- learning. The environment has a rich of more than 100 real-world inspired tasks in- volving diverse objects. The environment also provides easy access to expert demonstrations tasks, which vital for overcoming the exploration problem imposed by the bench- mark's 's sparse rewards. We classify RLBench as the medium difficulty benchmark in this study; while the embodiment is consistent across tasks (i.e., the Franka Panda robot arm), the task ob- jects and goals vary drastically.
Train
10 Task
Test
Figure In RLBench experiments, each agent is trained on 10 tasks and evaluated on unseen tasks.
To ensure the experiment results do not get af- fected by arbitrary task selection, we design comprehensive set of train-test task splits that resemble cross-validation in the supervised learning setting. Specifically, we use fixed set of RLBeecch tasks and create 5 splits. Each split uses (randomly selected) held-out task and trains agent on the remaining 10 tasks.
5

---

4.1 Training Setup
We use C2F-ARM [19] as the base off-policy RL algorithm. This was chosen because more widely- used RL algorithm, such as DDPG [28]. TD3 [29], SAC [30] and DrQ [22] are known to fail [31] in RLBench due to the challenging setup. C2F-ARM [19] is a vision-based robot manipulation algorithm that can learn sparse-reward reinforcement learning tasks by using small number of initial demonstrations. C2F-ARM is described in more detail in Appendix C. Note that RL² is excluded from this section because it is on-policy.
Reptile-C2F C2F-RMM modifies the off- off-policy batch update in C2F-ARM with an inner- and outer-loop proposed in Reptile [11]. At the beginning of training, each task is given a separate replay buffer, which is initialized with transitions collected from 5 demonstration trajectories and continuously appended with the agent's ' online experiences. During training, for multiple steps in the inner loop, the agent draws a batch from the replay buffer of a randomly sampled task and performs updates to the Q-attention. In the outer loop, the network gets soft update to mix the parameters from before and after the inner loop updates.
PEARL-C2F-ARI conditions a context embedding to the Q-attention network. To obtain the context for a task, a batch of transitions is drawn from a window of recent agent experiences, and separate convolution encoder is used to first encode the image observations individually. Then, each image embedding is concatenated with the action and reward, and together encoded into a single vector. Finally, the context embeddings are sampled as proposed in [12]. The context encoder is additionally trained with a KL. loss.
MT-C2F-ARM jointly trains C2F-ARM on all training tasks. During each replay batch update, both MT-C2F-ARM draw samples from multiple task replay buffers. During each replay update, fixed number of tasks (less or equal to the total number of available training tasks) are randomly selected, then an equal number samples are drawn for each task to construct the replay batch. 4.2
Test-time Adaptation Setup
use the same C2F-ARM update and adapt the agent parameters to the new task via gradient descent. Adaptation for PEARL-C2F-ARM is done by gathering rollout samples in the new environment and re-computing the context embeddings, hence running only inference on the agent's policy model.
Both MT-C2F-ARM and
4.3 Results
The first set of evaluations are the most challenging for adaptation: an unseen test-time task given 0 demonstrations. The agent is expected to leverage knowledge and skills gained in the 10 training tasks and perform intelligent exploration on the test task, without any guidance from demonstrations. Results for this setup are presented in the top row of Figure 6. Across all test tasks, multi-task fine-tuning performs equally well as Reptile while performing significantly better than both PEARL and training from scratch.
We next investigate the effect of reward sparsity on test-time performance. We now provide test-time demonstrations of each of the methods, as an aid for exploration under sparse reward. Results in the second and third row of Figure 6 show how the methods behave when given and 2 test-time demonstrations. The fact that increasing the number of demonstrations improves training from scratch performance is unsurprising, however, one intriguing observation is that this effect is less apparent for MT-C2F-ARM and Septile-CFFARR methods. This is encouraging evidence that fine-tuning significantly reduces (or even omit) the need for demonstrations in sparse rewarded tasks, with little loss to performance. We further investigate the various properties of fine-tuning C2F-ARM in Appendix C.
Apparent from Figure 6 is that PEARL does not seem equipped handle such disjoint train-test split. Recall that PEARL adapts without model parameter updates, and the only way to understand a new task is via aggregating new experiences into the context. However, the context encoder clearly fails at providing a useful context for the unseen tasks. we hypothesize this is due to our tasks setup: the training tasks are so visually disjoint that the agent never needs to learn high-quality context embeddings to infer which task it should do. This is different from the original experiment setup in PEARL., where variations are treated as "tasks", meaning that the observations from different "tasks" are similar or even identical; in order to disentangle the correct task to perform, the network

---

Beucepar:
up
Odens
TWo Lis Coucepant
Idemo Puen Dudion Ideno
Poccup come
Pirk Leane
Turm Lameer Idems
Pash
Pic
Fam
Enwronmeenr Stop
Enwiormatt Blss
Envisnmen Sxp
Ereroomtont Stop
Envirnment Eloa
MF-C2F ARM
Roptle C2F ARM
Scratch
FEARL CZF ARM
Figure 6: When varying the number of test-time demonstrations (from 0-2), does multi-task pretraining and fine-tuning outperform meta-RL methods unseen tasks? We perform "cross-validation style evaluation: from of RLBench tasks, is held out for test-time evaluation, while the other are used for training (and are given 5 demos). This is done for each of the 5 tasks above. Multi-task pretraining and fine- fine-tuning perform equally well or better than meta-Ru RL. Unsurprisingly fine-tuning (from either the Multi-task or Reptile agent) requires fewer demonstrations than training from scratch. Solid lines are average over seeds, with shaded regions representing standard
heavily motivated to read the context. Further evidence towards this hypothesis can be seen by looking at the zero-shot performance of the PEARL agent (i.e., environment steps = 0), where it starts with the same performance as multi-task and Reptile agents but doesn't 't improve. This suggests the meaningful performance that PEARL does achieve should be attributed to the pretraining and not test-time adaptation.
Atari Experiments
The Arcade Learning Environment (ALE, or the Atari benchmark) is one of the most commonly used environments within the RL community; however, rather than using the benchmark to train individual agents on each game, study due to the fact that the embodiment, objects, and goals, all vary across tasks; i.e., there is very little overlap between two different games.
use it to study meta-RL. We classify Atari as the hardest benchmark in this
Prior work has studied transfer between Atari games on relatively small scales, such as training an RL agent on one game and transfer to a visually similar game via fine-tuning [32, 33J. To the best of our knowledge, no prior meta-RI. algorithm has been meta-trained on a subset of Atari games and tested on other games. Recently, Oh et al. [34] demonstrated meta-training via discovering RL update rules can transfer from a mini-grid environment to Atari, but performance still falls short from state-of-the-art results on training Atari games from scratch.
5.1 Training Setup
Reptile-Rainbor incorporates Reptile [11] with the off-policy batch update in Rainbow. Similar PaptileeCCF-ABBRe transitions from each Atari game are stored in a separate replay buffer. For multiple steps in the inner loop, the agent draws batch from a randomly sampled buffer and performs updates to the distributional value network. In the outer loop, the value network gets soft update to mix parameters before and after the inner loop updates.
PEAR RL- Rainnaoboow conditions context embedding the value network. During each batch update, samples from each task are drawn from a recent window of agent experiences and get encoded as the context. The context encoder is additionally trained with a KL loss [12].
7

---

MT-Rainbow jointly trains RainbowDQQ on all training games/tasks. contrast to Reptile- Rainbow, the replay batch in both MT-Rainbow and PEARL-RRinbow contains samples from multiple task replay buffers. During each replay update, we randomly select fixed number of tasks (less or equal to the total number of avail- able training tasks), then draw an equal number of samples for each task and construct the replay batch.
Sequeai
Aslanee
Youi
Task Train
Dreakou
DemonnAtact
Rese
Tank Hleas
Scara
Note that, Atari games generally have differ- ent action dimensions, hence for pretrained agents, the action space is padded to be the max- imum possible size (discrete, 18-dimensional). The extra dimensions are all mapped to "No-op" for the games with smaller original action space, For training from scratch, the agents are trained on the original action dimensions on their corre- sponding test game.
Tasks
Phing
Acceet
Ratte
Me Parzzann
Beam Ricer
Figure 7: In Atari experiments, each agent is trained up 10 tasks and evaluated separately on 5 unseen tasks.
5.2 Test-time Adaptation Setup
We use test-time tasks, namely Pong, Assault, Battle Zone, Ms Pacman, and Beam Rider. For Reptile- Rainbow and MT MT-Rainbow, trained agent is finetuned with Rainbow for 100,000 envi- romment steps on each unseen game, following the data-efficient benchmark proposed in [35]. For PEARL-Rainhow, the agent is rolled-out updates the context embedding by encoding newly collected experiences.
new game for 100,000 environment steps, and only
We use the data-efficiem, version of Rain- bowDQN [20] as the base RL algorithm, i.e. we use the same training hyper-parameters and image encoder network architecture proposed in [35]. Note that RL3 is ex- cluded from this section because it is on- policy.
0.14
0.14
0.10
0.10
0.08
0.08
0.04
0.04
For both Reptile-Rainbow and MT- Rainbow, all network parameters are finetuned. We provide additional results in Appendix D for loading only the con- volutional encoder and re-initializing all MLP layers during finetuning. This setup was previously investigated in [33] at a smaller scale, i.c. training Breakout agent and fine-tuning on Pong, but the network architecture and base RL algorithms differ from ours.
22000 40300 62003 50220 103200
20005 40200
50035 80200 102000 Enstronment Step
Environment Blop
Scrwtsh
MT-Raentow
Figure 8: Adaptation unseen Atari games. Solid lines are average over 10 seeds, with shaded regions representing standard errors.
5.3 Results
We report the test-time adaptation performance for each pretrained agent in Figure 8. The results divided into sets, based on the number of training tasks provided to each compared algorithm. For each set of experiments, the same training from "scratch" baseline is used (colored in gray).
For 5-task- and 10-task- trained agents, we observe high variance across different test-time games despite being finetuned from the same pretrained agent. For example, finetuning the Reptile-Rainbon agent trained on 10 tasks provides a clear advantage over training from scratch on Assault, but falls short on Ms Paeman and Beam rider.

---

Consistent with results on RLBench, FEARL-Rainbow also fails to adapt to the unseen task despite continuously updating its context embedding for sufficiently long period. PEARL-Rainbow' zero-shot performance some of the games is clearly better than random (i.e. the first data point for training from scratch) however, this benefit comes more from the diversity of training tasks than from the meta-training algorithm itself, which is evident from observing that: (1) PEARL- Rainbow has similar zero-shot performance to that of MT-Rainbow and Reptile-Rainbon and (2) as the number of training task goes from 5 to 10, PEARL-Rainboo Overall, training from scratch has competitive performance across all test-time tasks, which is in contrast to previous results Procgen and RLBench. Previous work has also found that transfer between Atari games challenging [36, 32, 37J. and the common hypothesis is that the visuals and control strategies vary too much across games for positive transfer. We provide further empirical results in Appendix D to gain more insights into this hypothesis.
performance slightly improves.
6 Summary of Compared Algorithms
In order to allow casy side-by-side comparison of how different methods are implemented in our experiments, we provide unified overview of the algorithms that shown in Algorithms I and 2.
Algorithm Training On-policy Meta-RL
Require:
Learning rate a: Reptile step size
Algorithm 2 Training Off-policy Meta-RL.
Policy optimization objective L: Prajectory buffer D Input: Algorithm name algo Initialize policy LSTM embeilder d
Require: Learning rate Reptile step size
Context sampling heuristics S.. Value-function loss Jq:
while not converged do
KL Cx.L; L:
algo Reptile
Input: algorithm name algo
Initialize Q-function Q2. cuntext
Sample single Lask T p(7) Set golt
while not converged do
+-
Sample replay buffers draw samples from each construct batch 6 {0" b*}
for à to &
do Collect trajectories D' using fa g++1
a == Reptile then
a ai Df)
For getit +- 0
end for +- c(pold )²))
g+ +- 0¹ a (b())
else if algo == RL then
end for
<(gold 95))
Sample task To p(T) collect D' using Rgiid
end for
end for
Eeer (Fo D') Ei- (mo, 4. )
else if algo PEARL then
for = B do
Sample contexts of Sc():)
else algo Pretrain then
CkLL LKL( end for
for i = to do
Sample T: p(T); collect using We
a V Σ Eixt
end for
+ a oo (et
aV. E: C(TO. D)
else Pretrain then 0 + 0 a.
end
end while
end while
Related Work
Metf-Reinforeement Learning Meta-RL aims to find the best learning strategy that enables fast adaptation to new task via reinforcement learning. This often relies on meta-training with distribution of tasks and exploiting their shared structures. Two main approaches include context- based methods and gradient-based methods. Context-based methods experiences from a new task to form a context that can be used to distinguish what task the policy is solving. Previously, this context has been formed implicitly via LSTM [16, 38], explicitly, by passing trajectories through a separate encoder, whose output is given to a context-conditions policy [12, 39]. Gradient-bar based methods perform test-time optimisation of hyperparameters [40], loss functions [41, 42], or network parameters [3, 43, 44].
trained to recent rollout
The meta-RI. approaches above have only been studied in fully observable state settings with shaped rewards; neglecting more realistic real-world scenarios, where rewards are often sparse, and observations high-dimensional (c.g. images, point clouds, etc). There is limited work that study

---

these issues: for example, hindsight relabeling is used to aid in sparse reward setups e.g. [45], but uses fully observable states. Other approaches sparse reward and partial observability include HyperX [46]. DREAM [47], and MetaCure [48]. Out-of-distribution variation adaptation within the same task is another challenging setup, where recent methods include model-identification, experience relabeling, [49], and adding symmetries [50].
Beyond context-based and gradient- methods built on model-free RL algorithms, other lines of work include: model-basee meta-RL, via meta-training a dynamics model and has seen success in enabling adapting to different hardware or terrain conditions on legged millirobot [51]; meta- imitation learning, has been applied to vision-based robot manipulation [10, 52, 53]; meta-learn RL algorithms, which aims to discover RL objectives or update rules that can be transferred across different task environments [34, 54-57J.
Although in our experiments, we follow the original designs pf PEARL and RI.2 which don't 't allow gradient updates during test time, recent work [58] has looked into the theoretical limitations of context-based meta-RL. algorithms in out-of-distribution variation adaptation setting [58], and shown that adding gradient updates (i.e. finetuning) at test time helps improve adaptation.
Multi-task Reinforcement Learning The pretraining procedure in our experiments is multi-task reinforcement learning (MTRL), where the training objective is simply finding a single best policy across multiple tasks. The main challenge in multi-task learning in general lies in multi-objective optimization, and has been investigated in MTRL [59, 60] and applied to robotics [61]. Recently, Kurin et al. [62] demonstrated that joint training with proper regularization achieves competitive performance with the more complicated multi-task algorithms. This observation aligns with our multi-task training results.
Multi-variation pretraining followed by fine-tuning, also called domain random search (DRS), is also shown achieve comparable performance to meta-RL on existing state-based benchmarks [63]. Our work expands on this setup by training on more distinct tasks instead of variations, and excluding the test-time task from training. Notably, the meta-learning suite (e.g. ML10, ML45) in the Meta World [64] benchmark also poses such task generalization challenges, and finetuning is recently shown to be better than meta-RL RL algorithms such as RL and MAML [24].
Meta- v.s. Multi-task pretraining in RL
8 Conclusion
We perform large-scale study on vision- -ased meta-RL RL across truly diverse set of tasks. Our results show that when trained and tested on truly diverse reinforcement learning tasks, simple pretraining and fine-tuning can perform equally as well as well as more complex meta-learning approaches. This is consistent with the findings within the computer vision community [4-7], but in contrast to the large body of current literature, which shows that meta-RL. is effective when evaluated on variations of the same train-time task.
Our work is an initial step towards understanding the subtleties between meta-RL and multi-task pretraining, with plenty of room for further investigation. Despite the breadth of our experiments across 3 benchmarks. limitation of our study is the small number of available pre-training tasks for both RLBench and Atari. From the experimental results, we remark that training on such limited number of training tasks may not be sufficient to fully learn the representations required to enable the best adaptation performance on new tasks. However, note that the purpose of this study was not to show that multi-task pretraining followed by fine-tuning is better than training from scratch, but rather that multi-task pretraining can perform equally as well, or better, than meta-RI.
Acknowledgments and Disclosure of Funding
This work was supported by DARPA RACER and Hong Kong Centre for Logisties Robotics. The authors would like thank Shikun Liu, Danijar Hafnar, Olivia Watkins, Yuqing Du, and Luisa Zintgraf for their valuable discussions and helpful feedback on initial drafts of the paper.
References
[1] Vinyals, O., C. Blundell, T. Lillicrap, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29:3630-3633 2016.
10

---

[2] Ravi, S., H. Larochelle. Optimization as model for few-shot learning. Inil. Conference on Learning Representations, 2017.
[3] Finn, C., P. Abbeet, S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Intl. Conference on Machine Learning, pages 1126-11 1135. PMLR, 2017.
[4] Chen, W.-Y., Y.-C. Liu, Z. Kira, al. A closer look at few-shot classification. Intl. Conference on Learning Representations, 2019.
[5] Dhillon, G. S., P. Chaudhari, A. Ravichandran, et al. A baseline for few-shot image classification. Intl. Conference on Learning Representations, 2020.
[6] Tian, Y., Y. Wang, D. Krishnan, et al. Rethinking few-shot image classification: a good embedding is all you need? In European Conference Computer Vision, pages 266-282. Springer, 2020.
[7] Chen, Y., Z. Liu, H. Xu, al. Meta-baseline: exploring simple meta-lcarning for few-shot learning. In Intl. Conference on Computer Vision, pages 9062-9071. 2021.
James, S. Ma, D. Rovick Arrojo, et al. RLBench: The robot learning benchmark learning environment. IEEE Robotics and Automation Letters, 2020.
[9] Mandi, Z., F. Liu, K. Lee, al. Towards more generalizable one-shot visual imitation learning. arXiv preprint arXiv:21 10.01100110000 13423, 2021.
[10] Finn, C., T. Yu, T. Zhang, et al. One-shot visual imitation learning via meta-lcarning. In Conference on Robot Learning. pages 357-368. PMLR, 2017.
[11] Nichol, A., J. Achiam, J. Schulman. arXiv:18033 1803.0299,99 2018.
On first-order meta-learning algorithms.
arXiv preprint
[12] Rakelly, K., A. Zhou, D. Quillen, al. Efficient off-policy meta-reiiffrreement learning via probabilistic context variables, 2019.
[13] Cobbe, K., C. Hesse, J. Hilton, al. Leveraging procedural generation benchmark reinforcement learning. In International conference on machine learning, pages 2048 2056. PMLR, 2020.
[14] Bellemare, M. G.. Y. Naddaf. J. Veness, et al. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, 2013.
[15] Machado, M. C., M. G. Bellemare, E. Talvitie, et al. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-566 2018
[16] Duan, Y., Schulman, X. Chen, al. r12: Fast reinforcement leamiing via slow reinforcement learming. arXiv preprint /6. 2016.
[17] Cachet, T., J. Perez. Transformer-based mela-imitation learning for robotic manipulation. 2020.
[18] Schulman, J., F. Wolski, P. Dhariwal, et al. Proximal policy optimization algorithms. arXiv preprint 1XXv::7007.03347,0000 2017.
[19] James, S., K. Wada, T. Laidlow, et al. Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation. IEEE Conference Computer Vision and Pattern Recognition, 2022.
[20] Hessel, M., J. Modayil, H. Van Hasselt, et al. Rainbow: Combining improvements in deep reinforeement learning. In Thirty-second AAAI conference artificial intelligence. 2018.
[21] Wang, K., B. Kang, J. Shao, al. Improving generalization in reinforcement learning with mixture regularization. Advances Neural Information Processing Systems, 33:7968-7978, 2020.
[22] Yarats, Fergus, A. Lazaric, et al. Mastering visual continuous control: Improved data augmenteeeteed reinforcement learning. arXiv preprint arXiv:2107.09645. 2021.
[23] Igl, M., G. Farquhar, J. Luketina, et al. Transient non-stationarity and generalisation in deep reinforcement learning, 2020.
[24] Anand, J. Walker, Y. Li, et al. Procedural generalization by planning with self supervised world models. arXiv preprint arXiv:211.001587 2021.
11

---

[25] Alver. S.. D. Precup. A brief look at generalization in visual meta-reinforcemmnt learning. arXiv preprint urXiv:2006.07262, 2020.
[26] Espeholt, L., H. Soyer. R. Munos, et al. Impala: Scalable distributed deep-rl with importance weighted actor-leamer architectures. In International Conference Machine Learning, pages 1407-1416. PMLR, 2018.
[27] Lee, S., S.-Y. Chung. Improving generalization in meta-rl with imaginary tasks from latent dynamics mixture. ArXiv, abs/2105. 13524, 2021.
[28] Lillicrap. T. P., J. J. Hunt, A. Pritzel, al. Continuous control with deep reinforcement learning. Inil. Conference on Learning Representations, 2015.
[29] Fujimoto, S., H. Van Hoof, D. Meger. Addressing function approximation error in actor-critic methods. Intl. Conference Machine Learning, 2018.
[30] Haarnoja, T., A, Zhou, K. Hartikainen, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv: 181.05005 2018.
[31] James, S., A. J. Davison. Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation. IEEE Robotics and Automation Letters, 2022.
[32] Sobol, D., L. Wolf, Y. Taigman. Visual analogies between atari games for studying transfer learning in rl. arXiv preprint 1rXiv:180.11007. 2018.
[33] Du, Y., V. Gabriel, Irwin, et al. Initial progress in transfer for deep reinforeement learning algorithms. In Proceedings of deep reinforcement learning: frontiers and challenges workshop, New York City, NY, USA. 2016.
[34] Oh, J., M. Hessel, W. M. Czamecki, et al. Discovering reinforcement lcarning algorithms. Advances in Neurul Information Processing Systems, 33:1060-1070, 2020.
[35] van Hasselt, H. P., M. Hessel, J. Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019.
[36] Rusu, A. A., N. C. Rabinowitz, G. Desjardins, et al. Progressive neural networks. arXiv preprint arXiv:I606.00
2016.
[37] Mittel, A., P. Sowmya Munukutla. Visual transfer between atari games using competitive reinforcement lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 2019.
[38] Wang, J. X., Z. Kurth-Nelson, D. Tirumala, et al. Learning to reinforcement learn. arXiv preprint arXiv: 151.056111.576.. 2016.
[39] Fakoor, R., Chaudhari, S. Soatto, et al. Meta-q-learning 2020.
[40] Xu, Z., H. van Hasselt, D. Silver. Meta-gradient reinforcement leamiing arXiv preprint 100,00,80,,,9801, 2018.
[41] Sung, F., L. Zhang, T. Xiang, al. Learning learn: Meta-critic networks for sample efficient learning. arXiv preprint /706.09706099229 2017.
[42] Houthooft, R., R. Y. Chen, Isola, et al. Evolved policy gradients. arXiv preprint 2018
[43] Rothfuss, J., D. Lee, I. Clavera, et al. Promp: Proximal meta-policy search. Inil. Conference Learning Representartinss 2019.
[44] Raghu, A., M. Raghu, S. Bengio, al. Rapid learning feature reuse? towards understanding the effectiveness of maml, 2020.
[45] Packer, C., P. Abbeel, J. E. Gonzalez. Hindsight task relabelling: Experience replay for sparse reward meta-rl. Advances in Neural Information Processing Systems, 34, 2021.
[46] Zintgraf, L. M., L. Feng, C. Lu, et al. Exploration in approximate hyper-state space for meta reinforcement learning. In International Conference on Machine Learning, pages 12991-13001. PMLR, 2021.
[47] Liu, E. Z., A. Raghunathan, P. Liang, et al. Decoupling exploration and exploitation for meta reinforreeemen learning without sacrifices. arXiv preprint 2xXxi:2008.0279% 2020.
12

---

[48] Zhang, J., J. Wang. H. Hu, et al. Metacure: Meta reinforcement learning with empowerrment-driven exploration. In International Conference Machine Learning, pages 12600-12610. PMLR, 2021.
[49] Mendonea, R., X. Geng, C. Finn, et al. Meta-reinforcement learning robust to distributional shift via model identification and experience relabeling. arXiv preprint arXiv:2006.07172 2020.
[50] Kirsch, L., S. Fleneehaa, H. van Hasselt, et al. Introducing symmetries to black box meta reinforcement learning. arXiv preprint arXiv:2109. 10701, 2021.
[51] Nagabandi, A., I. Clavera, S. Liu, et al. Learning to adapt in dynamic, real-world environments through mefa-reinforcemment learning. arXiv preprint arXiv: 803. 1147, 2018.
[52] James, S., M. Bloesch, A. J. Davison. Task-embedded control networks for few-shot imitation learning. In Conference on Robot Learning, pages 783-795. PMLR, 2018.
[53] Bonardi, A., S. James, A. J. Davison, Learning one-shot imitation from humans without humans, IEEE Robotics and Automation Letters, 5(2):3533-3539) 2020.
[54] Kirsch, I.., S. van Steenkiste, J. Schmidhuher. Improving generalization in meta reinforcement learning using learned objectives. arXiv preprint 1rXiv:/910.04400 2019.
[55] Xu, Z.. H. P. van Hasselt, M. Hessel, al. Meta-gradient reinforcement learning with objective discovered online. Advances in Neural Information Processing Systems, 53:15254-1 2020.
[56] Co-Reyes, J. D., Y. Miao, D. Peng, et al. Evolving reinforcement learning algorithms. arXiv preprint arXiv:2101.0395: 2021.
[57] Houthooft, R., Y. Chen, P. Isola, et al. Evolved policy gradients. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett, eds., Advances Neural Information Processing Systems, vol. 31. Curran Associates, Inc., 2018.
[58] Xiong, L. M. Zintgraf. J. Beck, al. On the practical consistency of meta--cinforcement learning algorithms. ArXiv, abs/2112.00478,, 2021.
[59] Yang, R., H. Xu, Y. Wu, al. Multi-task reinforcement learning with soft modularization. ArXiv, Abs/2003.1366/ 2020.
[60] Sodhani, S., A. Zhang, J. Pineau. Multi-task reinforcement learning with context-based representations, 2021.
[61] Kalashnikov; J. Varlcy, Y. Chehotar, et al. Mt-opt: Continuous multi-task robotic reinforcement lcarning at scale, 2021.
[62] Kurin, V., A. De Palma, Kostrikov, In defense of the unitary scalarization for deep multi-task learning. arXiv preprint arXtv:2201.04122, 2022.
[63] Gao, K., Sener. Modeling and optimization trade-off in meta-lcarning. Advances Neural Information Processing Systems, 33:11154-111 11165, 2020.
[64] Yu. T., Quillen. Z. He, et al. Meta-world: A benchmark and evaluation multi tasti--tskk and meta reinforcement learning, 2021.
[65] Raffin, A., A. Hill, Gleave, al. Stable-baselinss; Reliable reinforcement leamning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021.
[66]
Arulkumaran, Pytorch implementation rainbowdqu. http:://gggtaab....////ttttt/t/t
2022.
[67] James, S., Abbeel. arXiv:2204.0157/ 2022.
Coarse-to-Fine Q-attention with Learned Path Ranking. arXiv preprint
[68] James, S., P. Abbeel. Coarse-to-Fine Q-attention with Tree Expansion. arXiv preprint arXiv:2204.124 12471, 2022.
13


Reviews:
[{"review_id": "f6OjxOIqiyB", "reviewer": "Reviewer_qVWu", "summary": "The paper presents a study comparing popular meta-learning approaches like Reptile, Pearl and RL^2 with standard multi-task pretraining + fine-tuning on 3 vision based benchmarks, namely Procgen, RLBench and Atari. On all three benchmarks, they test the generalization ability of the approaches on a completely novel task rather than variations of existing tasks from the distribution. They show that on all the tasks multi-task pretraining with fine-tuning performs equally or better than the meta-RL counterparts proposing multi-task pretraining + fine-tuning as a simple yet strong baseline for such tasks.\n", "questions": "I would appreciate the authors making amends on either of the points mentioned in the weaknesses section.\n\n", "limitations": "The authors discuss the limitation of the RLBench and Atari benchmarks in terms of lack of diversity of the tasks and the resulting effect of the adaptation behavior of the algorithms. However, I would have liked to see some discussion along the lines of above mentioned grievances even if not presented in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper is well written and easy to read. I also like the precise claims in the paper, ( e.g, “our findings show that meta-learning approaches are evaluated on different tasks (rather than different variations of the same task), multi-task pre-training with fine-tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation.”)\n2. I also like the selection of tasks and the experimental setup as they cover 3 broad categories of vision based RL tasks.\n**Weaknesses**\n1. The paper shows that multi-task pretraining + fine-tuning perform equally/better than meta-RL approaches when presented with completely novel tasks rather than variations of existing tasks. I would have liked to view some results on variations of the same (vision based?) task as well to get a better sense of how much of the performance difference in the two scenarios.  Furthermore, there’s probably a spectrum of diversity where the scales start tipping from meta-RL to MT finetuning. I would be interested in understanding where it tips and the factors that determine it. \n2. I would also like some additional baseline comparisons such as fine-tuning Pearl or RL^2 during adaptation. Currently the paper assumes the weights are fixed for these approaches during adaptation as in the original meta-learning setup. But the algorithm wasn’t designed to adapt to such completely novel tasks. So I think a fairer baseline to compare would be those algorithms with additional fine-tuning of their weights at test-time.\n\nOverall, I think although the results in the paper aren't very surprising, but can serve as a documentation of the implicitly understood tradeoffs between meta-RL and Multitask training + finetuning. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "A6xrHDdd0Bg", "reviewer": "Reviewer_J2Vz", "summary": "This paper mainly questions the benefits of meta learning approaches in reinforcement learning by comparing them with multi-task learning (pretraining) with fine-tuning. Vision-based benchmarks, including Procgen, RLBench, and Atari, are used to investigate this research question. In most cases, a multi-task pretraining approach with fine-tuning peforms equally or better than meta-learning approaches, although it is simpler and computationally cheaper than meta-learning approaches.  ", "questions": "- could we do this similar investigation with some non-vision-based benchmarks? could image representation learning be the main bottleneck of learning in these benchmarks?\n- in procgen experiments, why reptile is not working properly although it also fine-tunes the policy during adaptation as a multi-task pretraining approach does? (Other experiments show that reptile is similar to a multi-task pretraining approach)", "limitations": "As the authors mentioned in the paper,  the limited number of pre-training tasks for RLBench and Atari experiments are not appropriate for meta-RL, and therefore the results are not new and surprising. Moreover, the experiments are only conducted with vision-based environments, and therefore we could not easily generalize the results to other environments.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Meta-RL approaches are fairly compared with a simple multi-task pretraining approach, and it shows that meta-RL is not the best option to leverage knowledge from previously learned tasks.\n\n\nWeaknesses\n- The authors only investigated several approaches with vision-based benchmarks. Their conclusion may only apply to vision-based settings.\n- Meta-learning approaches generally requires a large number of tasks to be experienced during training, but the experiment settings don't have enough number of tasks. Therefore, the results are quite reasonable, but not surprising. \n- The authors haven't provided any meaningful research direction to improve existing meta-RL approaches. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7187daLRtXN", "reviewer": "Reviewer_DrBc", "summary": "The authors recap the pretrain-finetune framework for the generalisation RL problem and compare this simple baseline with many Meta-RL algorithms. They conduct the comparision on three vision-based RL tasks: Procgen, RLBench, and Atari. The final results show that in most case multi-task pretraining with fine-tuning can achieve comparable or even better performance than Meta-RL algorithms.", "questions": "(1) As far as I know, most Meta-RL algorithms are based on policy-based algorithms rather than Q-value based algorithms. In the experiments, 4 meta-rl algorithms are using Q-value based algorithms: Reptile-C2F-ARM, PEARL-C2F-ARM, Reptile-Rainbow, PEARL-Rainbow. For Reptile-C2F-ARM and Reptile-Rainbow. they are gradient-based Meta-RL and are meta-trained based on TD-error. However, in original MAML the meta-rl agent is trained with policy gradient. Though direct extension from policy-based MAML to value-based MAML seems to be technically sound, no further paper make it work empirically. For PEARL-C2F-ARM and PEARL-Rainbow, I also doubt whether there exist some papers make context-based value-based RL algorithm work on traditional meta-rl environments. Thus, I believe some meta-rl algorithms used in this paper are inappropriate.\n\n(2) Can you give more explainations on why you choose different task settings (described in Assumption on Task Distribution). If the state/action space vary a lot between different task, it more relies on the generalisation of observation space (as the author mentioned: \"An agent should be able to infer which task to perform based on a single-step observation\" rather than the generalisation of MDP. I think this is also the reason why RL generalisation and Meta-RL are sperate research branches because they are focusing on different problem.\n\n(3) It is not surprising to see that context-based Meta-RL algorithms such as Pearl and RL2 cannot work well for most experiments since the adaption steps are very large like 10k/1M. I believe on small RNN or simple MLP network cannot handle such large history information (especially your RNN hidden size is only 256). Have you tried more comlex network structure or more parameters to see if it can help increase the performance?\n\n(4) For vision-based meta-rl task, why not compare pretrain-finetune to the vision-bsed meta-rl algorithms like MELD [1]?\n\n(5) Can you also present the performance of the pretraining process? If the algorithms cannot be pretrained on the training distribution well, it is not surprising to see it cannot generalise on test distribution.\n\n\n[1] Zhao T Z, Nagabandi A, Rakelly K, et al. Meld: Meta-reinforcement learning from images via latent state models[J]. arXiv preprint arXiv:2010.13957, 2020.", "limitations": "Overall, this paper presents an interesting idea to reconsider the benchmark/evaluation protocol for Meta-RL. However, I think the main task setting and some baseline algorithms are inappropriate, which makes the final conclusion weak. I highly recommend the authors to try to start from some traditional meta-rl benchmarks. Because they tend to have well-tuned meta-rl baselines and can make the final comparision more fair. For instance, there also exists some complicated meta-rl benchmark such as Meta-World, where the task variation happens beyond reward/dynamic transitions. Some further experiments can be done there to strenghen your argument. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) Overall this paper offers a new view to consider a more realistic and reasonable evaluation protocol/benchmark for Meta-RL research.\n\n(2) Thorough experiments are conducted over different RL benchmarks comparing many different algorithms/baselines.\n\n(3) The writing is overall clear and easy to follow.\n\nWeaknesses:\n\n(1) Lack of more ablation studies analysing the reason why Meta-RL algorithms fail. (For instance, maybe because of the vision-based task's difficulty/it requires some many samples to adapt/lack of capacity of context network?)\n\n(2) Inappropriate meta-rl baseline extensions. (See Questions 1)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept