Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper mainly questions the benefits of meta learning approaches in reinforcement learning by comparing them with multi-task learning (pretraining) with fine-tuning. Vision-based benchmarks, including Procgen, RLBench, and Atari, are used to investigate this research question. In most cases, a multi-task pretraining approach with fine-tuning peforms equally or better than meta-learning approaches, although it is simpler and computationally cheaper than meta-learning approaches.  
strengths_and_weaknesses: Strengths
- Meta-RL approaches are fairly compared with a simple multi-task pretraining approach, and it shows that meta-RL is not the best option to leverage knowledge from previously learned tasks.


Weaknesses
- The authors only investigated several approaches with vision-based benchmarks. Their conclusion may only apply to vision-based settings.
- Meta-learning approaches generally requires a large number of tasks to be experienced during training, but the experiment settings don't have enough number of tasks. Therefore, the results are quite reasonable, but not surprising. 
- The authors haven't provided any meaningful research direction to improve existing meta-RL approaches. 
questions: - could we do this similar investigation with some non-vision-based benchmarks? could image representation learning be the main bottleneck of learning in these benchmarks?
- in procgen experiments, why reptile is not working properly although it also fine-tunes the policy during adaptation as a multi-task pretraining approach does? (Other experiments show that reptile is similar to a multi-task pretraining approach)
limitations: As the authors mentioned in the paper,  the limited number of pre-training tasks for RLBench and Atari experiments are not appropriate for meta-RL, and therefore the results are not new and surprising. Moreover, the experiments are only conducted with vision-based environments, and therefore we could not easily generalize the results to other environments.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a study comparing popular meta-learning approaches like Reptile, Pearl and RL^2 with standard multi-task pretraining + fine-tuning on 3 vision based benchmarks, namely Procgen, RLBench and Atari. On all three benchmarks, they test the generalization ability of the approaches on a completely novel task rather than variations of existing tasks from the distribution. They show that on all the tasks multi-task pretraining with fine-tuning performs equally or better than the meta-RL counterparts proposing multi-task pretraining + fine-tuning as a simple yet strong baseline for such tasks.

strengths_and_weaknesses: **Strengths**
1. The paper is well written and easy to read. I also like the precise claims in the paper, ( e.g, “our findings show that meta-learning approaches are evaluated on different tasks (rather than different variations of the same task), multi-task pre-training with fine-tuning on new tasks performs equally as well, or better, than meta-pretraining with meta test-time adaptation.”)
2. I also like the selection of tasks and the experimental setup as they cover 3 broad categories of vision based RL tasks.
**Weaknesses**
1. The paper shows that multi-task pretraining + fine-tuning perform equally/better than meta-RL approaches when presented with completely novel tasks rather than variations of existing tasks. I would have liked to view some results on variations of the same (vision based?) task as well to get a better sense of how much of the performance difference in the two scenarios.  Furthermore, there’s probably a spectrum of diversity where the scales start tipping from meta-RL to MT finetuning. I would be interested in understanding where it tips and the factors that determine it. 
2. I would also like some additional baseline comparisons such as fine-tuning Pearl or RL^2 during adaptation. Currently the paper assumes the weights are fixed for these approaches during adaptation as in the original meta-learning setup. But the algorithm wasn’t designed to adapt to such completely novel tasks. So I think a fairer baseline to compare would be those algorithms with additional fine-tuning of their weights at test-time.

Overall, I think although the results in the paper aren't very surprising, but can serve as a documentation of the implicitly understood tradeoffs between meta-RL and Multitask training + finetuning. 
questions: I would appreciate the authors making amends on either of the points mentioned in the weaknesses section.


limitations: The authors discuss the limitation of the RLBench and Atari benchmarks in terms of lack of diversity of the tasks and the resulting effect of the adaptation behavior of the algorithms. However, I would have liked to see some discussion along the lines of above mentioned grievances even if not presented in the paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors recap the pretrain-finetune framework for the generalisation RL problem and compare this simple baseline with many Meta-RL algorithms. They conduct the comparision on three vision-based RL tasks: Procgen, RLBench, and Atari. The final results show that in most case multi-task pretraining with fine-tuning can achieve comparable or even better performance than Meta-RL algorithms.
strengths_and_weaknesses: Strengths:

(1) Overall this paper offers a new view to consider a more realistic and reasonable evaluation protocol/benchmark for Meta-RL research.

(2) Thorough experiments are conducted over different RL benchmarks comparing many different algorithms/baselines.

(3) The writing is overall clear and easy to follow.

Weaknesses:

(1) Lack of more ablation studies analysing the reason why Meta-RL algorithms fail. (For instance, maybe because of the vision-based task's difficulty/it requires some many samples to adapt/lack of capacity of context network?)

(2) Inappropriate meta-rl baseline extensions. (See Questions 1)

questions: (1) As far as I know, most Meta-RL algorithms are based on policy-based algorithms rather than Q-value based algorithms. In the experiments, 4 meta-rl algorithms are using Q-value based algorithms: Reptile-C2F-ARM, PEARL-C2F-ARM, Reptile-Rainbow, PEARL-Rainbow. For Reptile-C2F-ARM and Reptile-Rainbow. they are gradient-based Meta-RL and are meta-trained based on TD-error. However, in original MAML the meta-rl agent is trained with policy gradient. Though direct extension from policy-based MAML to value-based MAML seems to be technically sound, no further paper make it work empirically. For PEARL-C2F-ARM and PEARL-Rainbow, I also doubt whether there exist some papers make context-based value-based RL algorithm work on traditional meta-rl environments. Thus, I believe some meta-rl algorithms used in this paper are inappropriate.

(2) Can you give more explainations on why you choose different task settings (described in Assumption on Task Distribution). If the state/action space vary a lot between different task, it more relies on the generalisation of observation space (as the author mentioned: "An agent should be able to infer which task to perform based on a single-step observation" rather than the generalisation of MDP. I think this is also the reason why RL generalisation and Meta-RL are sperate research branches because they are focusing on different problem.

(3) It is not surprising to see that context-based Meta-RL algorithms such as Pearl and RL2 cannot work well for most experiments since the adaption steps are very large like 10k/1M. I believe on small RNN or simple MLP network cannot handle such large history information (especially your RNN hidden size is only 256). Have you tried more comlex network structure or more parameters to see if it can help increase the performance?

(4) For vision-based meta-rl task, why not compare pretrain-finetune to the vision-bsed meta-rl algorithms like MELD [1]?

(5) Can you also present the performance of the pretraining process? If the algorithms cannot be pretrained on the training distribution well, it is not surprising to see it cannot generalise on test distribution.


[1] Zhao T Z, Nagabandi A, Rakelly K, et al. Meld: Meta-reinforcement learning from images via latent state models[J]. arXiv preprint arXiv:2010.13957, 2020.
limitations: Overall, this paper presents an interesting idea to reconsider the benchmark/evaluation protocol for Meta-RL. However, I think the main task setting and some baseline algorithms are inappropriate, which makes the final conclusion weak. I highly recommend the authors to try to start from some traditional meta-rl benchmarks. Because they tend to have well-tuned meta-rl baselines and can make the final comparision more fair. For instance, there also exists some complicated meta-rl benchmark such as Meta-World, where the task variation happens beyond reward/dynamic transitions. Some further experiments can be done there to strenghen your argument.
review_title: null

Meta-review: metareview: Reviewer qvwu summarizes the paper well: The paper presents a study comparing popular meta-learning approaches like Reptile, Pearl and RL^2 with standard multi-task pretraining + fine-tuning on 3 vision based benchmarks, namely Procgen, RLBench and Atari. On all three benchmarks, they test the generalization ability of the approaches on a completely novel task rather than variations of existing tasks from the distribution. They show that on all the tasks multi-task pretraining with fine-tuning performs equally or better than the meta-RL counterparts proposing multi-task pretraining + fine-tuning as a simple yet strong baseline for such tasks.

The other reviewers voted to reject the paper. Their main concerns were:
- Evaluation of vision-only benchmarks
- Issues in evaluation setup 

I believe the authors have sufficiently addressed these concerns, but unfortunately, the reviewers did not respond to the authors. In particular prior work already shows fine-tuning is competitive to meta-learning algorithms on state-based tasks. The main contribution of this work is in showing that these findings hold true in vision-based settings too and particularly in the scenario where tasks are different. This is a useful addition to the growing body of work comparing vanilla fine-tuning against meta-learning. I therefore recommend the paper be accepted. 
confidence: Certain
Decision: Accept