Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper proposes Meta Q-Learning (MQL), an algorithm for efficient off-policy meta-learning. The method relies on a simple multi-task objective which provides initial parameter values for the adaptation phase. Adaptation is performed by gradient descent, minimizing TD-error on the new validation task (regularizing towards initial parameter values). To make adaptation data efficient, the method makes heavy use of off-policy data generated during meta-training, by minimizing its importance weighted TD-error. Importance weights are estimated via a likelihood ratio estimator, and are also used to derive the effective sample size of the meta-training batch, which is used to adaptively weight the regularization term. Intuitively, this has the effect of turning off regularization when meta-training trajectories are “close” to validation trajectories. One important but somewhat orthogonal contribution of the paper is to highlight the importance of context in meta-learning and fast adaptation. Concretely, the authors show that a simple actor-critic algorithm (TD3), whose policy and value are conditioned on a context variable derived from a recurrent network performs surprisingly well in comparison to SoTA meta-learning algorithms like PEARL. MQL is evaluated on benchmark meta-RL environments from continuous control tasks and is shown to perform competitively with PEARL.

I have mixed opinions on this paper. On the positive side, and subject to further clarifications (see below), the paper seems to confirm that multi-task learning is almost sufficient to solve current meta-RL benchmarks in continuous control, without adaptation, as long as policy and critic are conditioned on a recurrent task context. This either highlights the strength of multi-task learning, or the inadequacies of current meta-RL benchmarks: either of which will be of interest to the community. On the other hand, the proposed MQL algorithm is only shown to significantly outperform this new baseline TD3-context agent on 1 of 6 tasks (Ant-Goal-2D), and furthermore the ablative analysis seems to suggest that the importance weighting and adaptive weighting of trust region are not very effective, and do not significantly change the performance of the method. The take-away seems to be that while context is crucial to generalization on these validation tasks, adaptation is not but can indeed be improved in a data-efficient manner with fine-tuning. MQL is only then a second thread to this story.


Clarifying Questions:

* The text and figures could be much clearer with respect to what is being measured/presented. Can you confirm that you report average validation returns as a function of meta-training steps, where validation performance is estimated after training on (at most) 2x200 transitions from the validation task (|D_new|=400)? This would match the protocol from PEARL. Fig 4(a) would then imply that one can get close to SoTA results on Ant-Fwd-Back + Half-Cheetah-Fwd-Back with no adaptation whatsoever (using TD3-context).
* Could the authors provide more details about how context is generated, in particular whether the GRU is reset on episode boundaries or not? If the recurrent context does span episode boundaries, are rewards being maximizing over a horizon greater than the episode length (similar to RL2)?
* How do you reconcile your result that a deterministic encoder is sufficient, compared to Figure 7 of PEARL which shows stochasticity is paramount for performing structured exploration during adaptation?
* Ablative analysis seems to suggest that off-policy learning plays a very minimal role during adaptation (beta=0). Can you confirm this interpretation? Would this not suggest that regularization via fine-tuning (regularized to multi-task prior) and context are sufficient to solve these meta-RL tasks? This would be a sufficient contribution by itself but unfortunately does little to validate the proposed method.
* Could you repeat the ablative analysis on all of the 6 tasks? Currently, this is performed on the two tasks for which TD3-context does best which leaves little room for improvement.
* Section 4.4. Propensity Score Estimation. “Use this model to sample transitions from the replay buffer that are similar to the new task”. Is this done via rejection sampling? This should be described more prominently in Section 3, along with a detailed description of the proposed MQL algorithm.

Detailed Comments:
* Paper desperately needs an algorithmic box, which clear and concise description of algorithm (how losses are interleaved, etc). Importantly, do the authors pretrain \theta_meta using the multi-task loss before doing adapation?
* Please add more informative labels to axes for all your figures: timesteps for [validation,training] ? Same for return. Please also augment captions to make figures as stand-alone as possible.
* MQL encompasses technique for off-policy training using a discriminator to estimate a likelihood ratio. It would be nice to evaluate this in standard off-policy learning setting, instead of it being limited to meta-learning.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary
-------------
The authors propose meta Q-learning, an algorithm for off-policy meta RL. The idea is to meta-train a context-dependent policy to maximize the expected return averaged over all training tasks, and then adapt this policy to any new task by leveraging both novel and past experience using importance sampling corrections. The proposed approach is evaluated on standard Mujoco benchmarks and compared to other relevant meta-rl algorithms.

Comments
--------------

Meta-rl is a relevant direction for mitigating the sample-complexity of rl agents and allowing them to scale to larger domains. This work proposes interesting ideas and overall it constitutes an nice contribution. In particular, I found interesting (and at the same time worrying) that a simple q-learning algorithm with hidden contexts compares favorably to state-of-art meta-rl approaches in standard benchmarks. The paper is well-organized and easy to read. Some comments/questions follow.

1. (15) is probably miss-written (theta is trained to maximize the TD error)

2. In the adaptation phase, are (18) and (19) performed one after the other? Could they be done at the same time by setting to 1 the importance weights of the new trajectories and sampling from the whole experience (new and old)?

3. Note that the ESS estimator (13) diverges to infinity when all weights are close to zero. Is it clipped to [0,1] in the experiments? See e.g. [1] or [2] for more robust estimators that are bounded.

4. Since many recent works try to improve the generalization capabilities of meta-rl algorithms, I was wondering how the proposed approach generalizes to out-of-distribution tasks (i.e., tasks that are unlikely to occur at meta-training). Though it is never mentioned in the paper, I believe the proposed method has the potential to be robust to negative transfer since the importance weights (which would be very small for very different tasks) should automatically discard old data and focus on new data alone. This is in contrast to many existing methods where the meta-trained model might negatively bias the learning of very different tasks. I think an experiment of this kind would be valuable to improve the paper.

[1] Elvira, V., Martino, L., & Robert, C. P. (2018). Rethinking the effective sample size. arXiv preprint arXiv:1809.04129.
[2] Tirinzoni, A., Salvini, M., & Restelli, M. (2019, May). Transfer of Samples in Policy Search via Multiple Importance Sampling. In International Conference on Machine Learning (pp. 6264-6274).
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors investigate meta-learning in reinforcement learning with respect to sample efficiency and the necessity of meta-learning an adaptation scheme. Based on their findings, they propose a new algorithm 'MQL' (Meta-Q-Learning) that is off-policy and has a fixed adaptation scheme but is still competitive on meta-RL benchmarks (a distribution of environments that differ slightly in their reward functions).

They motivate the paper by data-inefficiency of current meta-learning approaches and empirical results suggesting that meta-learning the adaptation scheme is less important than feature reuse. 

On the other hand, their introduction section would benefit from additional references to the kind of meta-learning they describe. In particular, their so-called "definition of meta-learning" is mostly about domain randomization (e.g. Tobin et al. 2017 https://arxiv.org/abs/1703.06907) and not about the broader 'learning to learn' RL methodology (in particular Schmidhuber 1994 "On learning how to learn learning strategies").

**The authors make the following contributions:**

1. They show that Q-Learning trained on multiple tasks with a context variable as an input (an RNN state summarizing previous transitions) is competitive to related work when evaluated on a test task even though no adaptation is performed

2. Based on these observations, they introduce a new method for off-policy RL that does not directly optimize for adaptation but instead uses a fixed adaptation scheme

3. The new method leverages data during meta-testing that was collected during meta-training using importance weights for increased sample efficiency

**Overall, we believe the contributions are significant and sufficiently empirically justified.**

There are strong similarities, however, to parallel work on analyzing whether MAML relies on feature reuse or rapid learning (Raghu et al. 2019 https://arxiv.org/abs/1909.09157).

This work and the present submission conclude that feature reuse is much more significant than meta-learning an adaptation scheme when evaluated on current meta-RL benchmarks. This is a significant result and supports the new method developed in this paper.

During meta-training, their proposed method maximizes only the average return across tasks, not the ability to adapt from the resulting parameters.

Their method introduces a fixed (non-learned) adaptation scheme that performs favorably compared to certain methods from the existing meta-learning literature and demonstrates that even dropping this adaptation still does well.

There are strong similarities to Nichol et al. 2018 (https://arxiv.org/abs/1803.02999). We encourage the authors to relate this work to Raghu et al. 2019 and Nichol et al. 2018.

**Despite these interesting results, we strongly disagree with the meta-learning narrative of their new method.**

Because the adaptation scheme is no longer optimized directly, instead a fixed adaptation scheme is assumed, hence the approach in this paper is no longer a meta-learning algorithm.

Instead, this method has strong similarities with transfer-learning and domain adaptation (first training on one distribution of tasks, then fine-tuning on another task).

The authors should discuss the links to these fields of research, and clarify what's really novel, already in the abstract.

For example, on page 2 the authors claim that optimizing the multi-task objective (the mean error across tasks) is the simplest form of meta-learning. This objective, however, is NOT meta-learning.

**Decision.**

The submission contains strong empirical results emphasizing the significance of feature reuse and the insignificance of learned adaptation on the tested meta-RL benchmarks.

The reuse of experience from meta-training during meta-testing by employing importance weights is also an interesting contribution.

In contrast, we are not satisfied with the presentation of their new approach as a meta-learning approach. This method should be introduced along the lines of: 'Transfer Learning / Feature Reuse in RL is competitive to meta-learning across similar tasks'.

In its current form, we tend to reject the paper because it further obscures what the term meta-learning refers to. The authors are confusing it with more limited transfer learning. 

Additionally, it was not clear to us whether the quadratic penalty they add to their adaptation scheme is only empirically valid or whether there is a theoretical reason.

For now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way - let us wait for the rebuttal.

Edit after rebuttal: score increased!
review_title: Official Blind Review #1

Meta-review: This paper’s contribution is twofold: 1) it proposes a new meta-RL method that leverages off-policy meta-learning by importance weighting, and 2) it demonstrates that current popular meta-RL benchmarks don’t necessarily require meta-learning, as a simple non-meta-learning algorithm (TD3) conditioned on a context variable of the trajectory is competitive with SoTA meta-learning approaches. 

The reviewers all agreed that the approach is interesting and the contributions are significant. I’d like to thank the reviewers for engaging in a spirited discussion about this paper, both with each other and with the authors. There was also a disagreement about the semantics of whether the approach can be classified as “meta-learning”, but in my opinion this argument is orthogonal to the practical contributions. After the revisions and rebuttal, reviewers agreed that the paper was improved and increased their ratings as a result, with all recommending accept.

There’s a good chance this work will make an impactful contribution to the field of meta-reinforcement learning and therefore I recommend it for an oral presentation.

Decision: Accept (Talk)