Meta-Q-Learning
Authors: Rasool Fakoor
Pratik Chaudhari
Stefano Soatto
Alexander J. Smola
Venue:  (2020)
Keywords: meta reinforcement learning
propensity estimation
off-policy

Abstract: This paper introduces Meta-Q-Learning (MQL), a new off-policy algorithm for meta-Reinforcement Learning (meta-RL). MQL builds upon three simple ideas. First, we show that Q-learning is competitive with state-of-the-art meta-RL algorithms if given access to a context variable that is a representation of the past trajectory. Second, a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies. Third, past data from the meta-training replay buffer can be recycled to adapt the policy on a new task using off-policy updates. MQL draws upon ideas in propensity estimation to do so and thereby amplifies the amount of available data for adaptation. Experiments on standard continuous-control benchmarks suggest that MQL compares favorably with the state of the art in meta-RL.

Published as a conference paper at ICLR 2020
META-Q-LEARNIH
Rasool Fakoor', Pratik Chaudhari Steffano Soatto', Alexander Smola'
1 Amazon Web Services
2 University of Pennsylvania
Email: {fakoor, soattos, smola) @ @amazon.com gratikac@seas.uueaea
ABSTRACT
This paper introduces Meta-Q-Learning (MQL), new off-policy algorithm for neta-Reinforcement Learning (meta-RL). MQL builds upon three simple ideas. First, we show that Q-learning competitive with state-of-the-ar meta-RL. algo- rithms if given access context variable that is representation of the past trajectory. Second, a multi-task objective to maximize the average reward across the training tasks is an effective method to meta-train RL policies. Third, past data from the meta- veta-training replay buffer can be recycled to adapt the policy on a new task using off- off-policy updates. MQL draws upon ideas in propensity estimation do so and thereby amplifies the amount of available data for adaptation. Experi- ments on standard continuous-contre benchmarks suggest that MQL compares favorably with the state of the art in meta-RL.
INTRODUCTION
Reinforcement Learning (RL) algorithms have demonstrated good performance on simulated data. There are however two main challenges in translat- ing this performance to real robots: (i) robots are complex and fragile which precludes extensive data collection, and (ii) real robot may face an environ- ment that is different than the simulated environment was trained in. This has fueled research into Meta- Reinforcement Learning (meta-RU RL) which develops algorithms that "meta-train" on large number of dif- ferent environments, e.g., simulated ones, and aim to adapt new environment with few data.
MAAI
TD0
PEAR
1000
Arsix
Charrh HhllCCeert
Figure How well does meta-RL work? Aver- age returns on validation tasks compared for two prototypical meta-RI. algorithms, MAMI. (Finn al., 2017) and PEARI. (Rakelly al., 2019), with those of vanilla Q-learning algorithm named TD3 (Fujimoto et al., 2018b) that was modified to incorporate context variable that is represen- tation of the trajectory from task (TD3-context). Even without any meta-training and adaptation on new task, TD3-context is competitive with these sophisticated a algorithms.
How well does meta-RL work today? Fig. 1 shows the performance of two prototypical meta-RL algo- rithms four standard continuous-control bench- marks.! We compared them to the following simple baseline: an off off-policy RL algorithm (TD3 by Fuji- moto al. (2018b)) and which was trained max- imize the average reward over all training tasks and modified to use a "context variable" that represents the trajectory. All algorithms in this figure use the same evaluation protocol. It is surprising that this simple haa--mttt-leannng-basssee
method is competitive with state-of-the-art meta-RL. algorithms. This is the first contribution of our paper: we demonstrate that it is not necessary to meta-train policies do well on existing benchmarks.
Our second contribution is an off-policy meta-RL algorithm named Meta-Q-Learning (MQL) that builds upon the above result. MQL uses a simple meta-training procedure: maximizes the average
Work done while at Amazon Web Services
We obtained the numbers for MAMI and PEARL from training logs published by Rakelly al. (2019).

---

Published as a conference paper at ICLR 2020
rewards across all meta- meta-rraining tasks using off-po updates to obtain
mmeta arg max
(0)
where (0) is the objective evaluated on the transition obtained from the task (*), e-g.. 1-step (8: r). This objective, which call the temporal-differenee (TD) enror would set e (0) TD²(0;
multi-task objective, is the simplest form of meta-training
For adapting the policy to new task, MQL samples transitions from the meta-training replay buffer that are similar to those from the new task. This amplifies the amount of data available for adaptation but is difficult to do because of the large potential bias. We use techniques from the propensity estimation literature for performing this adaptation and the off-policy updates of MQL are crucial to doing The adaptation phase of MQL solves
arg max
E Drr
Cnew (0)
E
[(T: Dew Pmeta) nrew (0) -(1 (1--ES)
ESS 110 @mett 2
(2)
where D Menu the meta-training replay buffer, the propensity score [T; Dnew Dmeea) the odds of transition r belonging to Doow versus Deeta and ESS is the Effective Sample Size between Doow and Deea that is a measure the similarly of the new task with the meta- reta-training tasks. The first term computes off- off-policy updates the new task, the second term performs B(-) 3(-)-weighted off aff-poolcy updates on old data, while the third term is an automatically adapting proximal term that prevents degradation of the policy during adaptation.
We perform extensive experiments in Sec. 4.2 including ablation studies using standard meta-RI. RL benchmarks that demonstrate that MQL policies obtain higher average returns on new tasks even they are meta-trained for fewer time-steps than state-of-the-arr algorithms.
2 BACKGROUND
This section introduces notation and formalizes the meta-RL problem. We discuss techniques for estimating the importance ratio between two probability distributions in Sec. 2.2.
Consider Markov Decision Processes (MDP) denoted by
Tt+1 f Uer Ee) 2o Po
where 2, C. Rd are the states and it E U Rº are the actions. The dynamics ft is parameterized by & E n? where each A corresponds to differrntt task. The domain of all these tasks, X for the states and U for the actions, is the same. The distribution p denotes the initial state distribution and & is the noise the dynamics. Given deterministic policy up (2:). the action- value function for 7-discounted future rewards r p (x2, (x(x)) over an infinite time-horizon is
d (x. u)
E
7
Fy = x. ug = u, u4 = us (1=)
(4)
Ec-)
t=
Note that we have assumed that different tasks have the same state and action space and may only differ in their dynamies f and reward function pk. Given one task k ..], the standard Reinforcement Learning (RL) formalism solves for
gh arg max et (8) where (0)
= E a--po
[g uo())).
(5)
Let us denote the dataset of all states, actions and rewards pertaining to task kt and policy uo by D* (0) {z,, (xx)),, r J++1 (*,, uo(xx))
=(0)~p =(0))))))ppp e(-)
we will often refer 1 as the "task" itself. The Deterministic Policy Gradient (DPG) algorithm (Sil- al., 2014) for solving (5) learns y-parameterized approximation q the optimal value func-
2

---

Published as a conference paper at ICLR 2020
tion q² by minimizing the Bellman error and the optimal policy ug that maximizes this approximation by solving the coupled optimization problem
[ (1o u) - -r - 2o u }.
arg min TNDE
(6)
6 arg max E TODD
9:32 (x. uz(()))
The 1-step temporal difference error (TD error) is defined as
TD² (0) = (g.(2, u) - - (a uo
where we keep the dependence of TD()) on implicit. DPG, or its deep newwrrk-bssee variant DDPG (Lillicrap et al., 2015), is an off-policy algorithm. This means that the expectations in (6) are computed using data that need not be generated by the policy being optimized (ug), this data can come from some other policy.
In the sequel, will focus the parameters 0 parameterizing the policy. The parameters yp of the value function are always updated to minimize the -D>error and are omitted for clarity.
2.1 META- REINFORCEMENT LEARNING (META- -R))
Meta-RL RL technique learn an inductive bias that accelerates the learning of new task by training on large of number of training tasks. Formally, meta-training tasks from the meta-training set Dmeea =1 [b)]
k= involves learning policy
m arg max
pk
(0)
(8)
meta
Y1
where eh (0) is meta-training loss that depends on the particular method. Gradient-based meta-RL, let us take MAML by Finn et al. (2017) as concrete example, sets
e (0) e (0 + a a Vo (
(9)
for a step-size > 0; e* (0) is the objective of non-meta-RL. (5). In this case gk meta is the objective obtained on the task h after one (or in general, more) updates of the policy on the task. The idea behind this is that even if the policy Qmeu does not perform well on all tasks in Dmeta it may be updated quickly on a new task new to obtain a well well-performing policy. This can either done using the same procedure that of meta-training time, i.c., by maximizing me),(0) with the policy mmeta as the initialization, or by some other adaptation procedure. The meta- veea-training method and the adaptation method in meta-RL, and meta-learning in general, can be different from each other.
2.2 LOGISTIC REGRESSION FOR ESTIMATING THE PROPENSITY SCORE
Consider standard supervised learning: given two distributions g(x) (say, train) and p(r) (say, test), we would like to estimate how model's predictions (Ix) change across them. This formally done using importance sampling:
E E [C(u, ((x)) 2-pp()) u/
E
E. [B(x) Ely, ((x)));
(10)
x>)((>))
where y| are the true labels of data, the predictions of the model are ú(x) and Ely, (()) is the loss for each datum (I, y). The importance ratio 3(x) de (z), also known as the propensity dy
score, is the Radon-Nikodym derivative (Resnick, 2013) of the two data densities and measures do
the odds of sample x coming from the distribution p versus the distribution q. In practice, not know the densities q(x) and p(x) and therefore need to estimate ((z) using some finite data X, { } drawn from g and X, ( Inm drawn from p. As Agarwal et al. (2011) show, this is easy to do using logistic regression. Set =
be the labels for data in X, and zs -1 to be the labels of the data in X, for & < and fit logistic classifier the combined 2m

---

Published as a conference paper at ICLR 2020
samples by solving
1
log (1+c==(()) + + tcllwl??
wº min
(11)
2m
(2,2)
This gives
((x) P( - 1/ T
(12)
P(: 1 [x)
Normalized Effective Sample Size (ESS): A related quantity to (x) is the normalized Effective Sample Size (ESS) which we define as the relative number of samples from the target distribution p(x) required to obtain an estimator with performance (say, variance) equal to that of the importance sampling estimator (10). It is not possible compute the ESS without knowing both densities ((x) and p(x) but there are many heuristics for estimating it. A popular one in the Monte Carlo literature (Kong, 1992; Smith, 2013; Elvira et al., 2018) is
1 - B(xx)) m Σx_ B(r
ESS
E [0,
(13)
where X = f= I is some finite batch of data. Observe that if two distributions & and are close then the ESS is close to one; if they are far apart the ESS is close to zero.
3 MQL
This section describes the MQL algorithm. We begin by describing the meta-training procedure of MQL including a discussion of multi-task training in Sec. 3.1. The adaptation procedure is described in Sec. 3.2.
3. MEAA-TRAINING
MQL performs meta-training using the multi-task objective. Note that if one sets
pkc meta 0 2 e*
E [a ux(x)) aーー
(14)
in (8) then the parameters mmesa are such that they maximize the average returns over all tasks from the meta-training set. We use off-policy algorithm named TD3 (Fujimoto al., 2018b) as the building block and solve for
Ometa = arg min-
E
[D):
(15)
where TD(-) is defined in (7), As is standard in TD3, by P and 22 and take their minimum to compute the target in (7). This trick known as "double- Q-learning" reduces the over-estimation bias. Let us emphasize that (14) is a special case of the procedure outlined in (8). The following remark explains why MQL uses the multi-task objective as opposed to the meta-training objective used, for instance, in existing gradient-based meta-RI. algorithms.
two action-value functions parameterized
Remark 1. Let compare the critical points of the m-step MAML objective (9) to those of the multi-task objective which uses (14). As is done by the authors in Nichol et al. (2018), we can perform Taylor series expansion around the parameters 0 to obtain
pl
meta (0) (0) + 2a(m - 1) (v² (0))))((()))
(16)
Further, note that pk meta in (16) is also the gradient of the loss
k (0) a(m 1)
(17)
4

---

Published as a conference paper at ICLR 2020
up to first order. This lends new interpretation that MAML is attracted towards regions in the loss landscape that under-fi fit on individual tasks: parameters with large || ll2 will be far from the local maxima of e* (0). The parameters a and m control this under-fitting. Larger the number of gradient steps, larger the under-fitting effect. This remark suggests that the adaptation speed of gradient-based meta-learning comes at the cost of under-fitting fitting on the tasks.
3.1.1 DESIGNING CONTEXT
As discussed in Sec. I and 4.4, the identity of the task in meta-RL can be thought of the hidden variable of an underlying partially-observabee MDP. The optimal policy on the entire trajectory of the states, actions and the rewards. We therefore design recurrent context variable 2, that depends on ((24, t4 r:)) We set 2/ to the hidden state time & of Gated Recurrent Unit (GRU by Cho al. (2014)) model. All the policies (x)) and value functions qo (2, u) in MQL are conditioned on the context and implemented as uo(x, 2) and (1, 2). Any other recurrent model can be used to design the context; we used a GRU because it offers a good trade-off between a rich representation and computational complexity.
Remark 2 (MQL uses deterministic context that is not permutation invariant). We have aimed for simplicity while designing the context. The context in MQL is built using an off-the-shelf model like GRU and is not permutation invariant. Indeed, the direction of time affords crucial in- formation about the dynamics of task to the agent, e.g., Half-Cheetal rumning forward versus backward has arguably the same state trajectory but in a different order. Further, the context in MQL a deterministic function of the trajectory. Both these aspects are different than the context used by Rakelly al. (2019) who design an inference network and sample a probabilistic context condi- tioned on a moving window. RI. algorithms are quite complex and challenging to reproduce. Current meta-RL techniques which build upon them further exacerbate this complexity. Our demonstration that simple context variable is enough is an important contribution.
3. ADAPTATION TO A NEW TASK
We next discuss the adaptation procedure which adapts the meta-trained policy Omets task Deew with few data. MQL optimizes the adaptation objective introduced in (2) into two steps.
1. Vanilla off-policy adaptation: The first step is update the policy using the new data
X 2 110 mmetall
arg max
E
gnow (0)
(18)
Tr
2
The quadratic penalty |10 mmemll 2 keeps the parameters close 0men. This is crucial to reducing the variance of the model that is adapted using few data from the new task (Reddi et al., 2015). Off-policy learning is critical in this step because of its sample efficiency. We initialize 0 to 0 while solving (18).
2. Importance-ratio corrected off-policy updates: The second step of MQL exploits the meta- training replay buffer. Meta-training tasks Dinta are disjoint from neew but because they are expected to come from the same task distribution, transitions collected during meta-training can potentially be exploited to adapt the policy. This difficult do on two counts. First, the meta-trann meta-training transitions do not come from neww, Second, even for transitions from the same task, it is non-trivial to update the policy because of extrapolation error (Fujimoto et al., 201 2018a): the value function has high error on states has not seen before. Our use of the propensity score to reweigh transitions is a simpler version of the conditional generative model used by Fujimoto al. (2018a) in this context.
MQL fits a logistic classifier on a mini-batch of transitions from the meta-training replay buffer and the transitions collected from the new task in step 1. The context variable 2; is the feature for this classifier. The logistic classifier estimates the importance ratio B(r; neew :Dmeaa and can be used to reweigh data from the meta-training replay buffer for taking updates as
arg max
B(:; new D Dmmtaa gnew (8)
|10 mmetall3
(19)
Tr
2
5

---

Published as a conference paper at ICLR 2020
We have again included a quadratie penalty ||0 meuull? that keeps the parameters close to metas- Estimating the importance ratio involves solving convex optimization problem on few samples (typically, 200 from the new task and 200-400 from the meta-training tasks). This classifier allows MQL exploit the large amount of past data. In practice, we perform as many x00x more weight updates using (19) than (18).
Remark 3 (Picking the coefficient A). Following Fakoor al. (2019), pick
A ESS
for both the steps (18-19). This relaxes the quadratic penalty if the new task is similar to the meta- training tasks (ESS is large) and vers- While À could be tuned empirical results show that adapting using ESS a simple and effective heuristic.
Remark (Details estimating the importance ratio). is crucial to ensure that the logistic classifier for estimating 3 generalizes well if are to reweigh transitions in the meta-training replay buffer that are different than the ones the logistic was fitted upon. We do so in two ways: (i) the regularization co-efficient in (11) is chosen to be relatively large, that way we prefer false negatives than risk false positives; (ii) transitions with very high / are valuable for updating (19) but cause large variance in stochastic gradient descent-based updates, we clip 3 before taking the update in (19). The clipping constant a
and is given in Sec. 4.
MQL requires having access to the meta- meta-training replay buffer during adaptation. This is not a debilitating requirement and there are a number of clustering techniques that can pick important transitions from the replay-buffer if robotic agent is limited by available hard-disk space. The meta-training replay buffer is at most 3 GB for the experiments in Sec. 4.
4
EXPERIMENTS
This section presents the experimental results of MQL. We first discuss the setup and provide details the benchmark in Sec. 4.1. This is followed by empirical results and ablation experiments in Sec, 4.2.
4. SETUP
Tasks and algorithms: We use the Mu- JoCo (Todorov et al., 2012) simulator with OpenAl Gym (Brockman al., 2016) continuous-control meta-RL benchmark tasks. These tasks have different rewards, randomized system parameters (Walker-DD-Paramn) have been used in previous papers such as Finn et al. (2017); Rothfuss et al. (2018); Rakelly et al. (2019). We compare against standard base- line algorithms, namely MAML (TRPO (Schul- man et al., 2015) variant) (Finn et al., 2017), RL2 (Duan et al., 2016), ProMP (Rothfuss et al., 2018) and PEARL (Rakelly al., 2019). We ob- tained the training curves and dlyerr-paameee for all the three algorithms from the published code by Rakelly et al. (2019).
750
and
Her 2oring
Uesurarii Threstape
We will compare the above algorithms against: (i) vanilla TD3 (Fujimoto et al., 2018a) without any adaptation on new tasks, (ii) TD3-context: TD3 with GRU-based context Sec. 3.1.1 with- out any adaptation, and (iii) MQL: TD3 with context and adaptation on new task using the procedure in Sec. 3.2. All the three variants use
Figure 2: Average undiscounted return of TD3 and TD3-context compared with PEARL. for validation tasks from four meta-RL environments. The agent fails to learn if the policy is conditioned only on the state. contrast, everything else remaining same. TD3 provided access to context, the rewards are much higher. In spite of not adaptating on the validation tasks, TD3- context is comparable to PEARL.

---

Published as a conference paper at ICLR 2020
the multi-task objective for meta- meta-training (15). We use Adam (Kingma & Ba, 2014) for opti- mizing all the loss functions in this paper.
Current meta-RL benchmarks lack a systematic evaluation procedure. 2 For each envi- ronment, Rakelly et al. (2019) constructed a fixed set of meta-training tasks (Dmeta and a validation set of tasks Dusw that are disjoint from the meta-training set. To enable direct comparison with pub- lished empirical results, we closely followed the evaluation code of Rakelly et al. (2019) to create these tasks. We also use the exact same evaluation protocol as that of these authors, e.g.. 200 time- steps of data from the new task, or the number of evaluation episodes. We report the undiscounted return the validation tasks with statistics computed across 5 random seeds.
Evaluation:
4.2 RESULTS
Half Chestahs Ford Back
Ant Goa--Gol--D
Find Back
1400
2000
Assragg Vakdation Neraar
1900
1030
1000
Aveegge
600
300
-1000
0.4
0.6
2.6
0.0
1.0
Mett training Tiim-steps
Meto: Time
Time-stape
Hatt-Checta-VVel
Humonold
Walker
-100
Averape 744 dar.am Nuturn
830
200
800
203
100
05
0.2
1-5
Matte-raiiingg TTrroodagga
Mataetraiiing T'mmooalggi
Moba-bariing Timeakos
Figure 3: Comparison of the average undiscounted return of MQL (orange) against existing meta-RL al- gorithms continuous-coutrol environments. We compare against four existing algorithmss namely MAML (green), RL2 (red), PROMP (purple) and PEARL (blue). In all environments except Walker-2D Malker-2D-Params and Ant-Goal-2D,, MQL is better or comparable to existing algorithms in terms of both sample complexity and final returns.
Our first result, in Fig. 2, is to show that vanilla off off-policy learning with context, without any adap- tation is competitive with state of the art meta-RL algorithms. We used standard implementation of TD3 and train on the meta-training tasks using the multi-task objective (15). Hyper-parameters for these tasks are provided in Appendix D. This result is surprising and had gone unnoticed in the current literature. Policies that have access to the context can easily generalize to the validation tasks and achieve performance that is comparable to more sophisticated meta-RI. algorithms.
We next evaluate MQL against existing meta-RL benchmarks on all environments. The results are shown in Fig. 3. We see that for all environments except Walker- 2D-aar-s and Ant-Goal- Goal-2D, MQL obtains comparable or better returns on the validation tasks. In most cases, in particular for the challenging Humanood--ireee
environment, MQL converges faster than existing algorithms. MAML and ProMP require about 100M time-steps to converge returns that are significantly worse
FFor instance, training and validation tasks not explicitly disjoint in Finn et al. (2017); Rothfuss et al. (2018) and these algorithms may benefit during adaptation from having seen the same task before. The OpenAl Gym environments used in Finn et al. (2017); Rothfuss et al. (2018); Rakelly et al. (2019) provide different rewards for the same task. The evaluation protocol in existing papers, e.g., length of episode for a new task, amount of data available for adaptation from the new task, is not consistent. This makes reproducing experiments and comparing numerical results extremely difficult.
7

---

Published as a conference paper at ICLR 2020
than the returns of off-policy algorithms like MQL and PEARL. Compare the training curve for TD3-context for the Ant-Goal-2D environment in Fig. 2 with that of the same environment in Fig. 3: the former shows a prominent dip in performance as meta-training progresses; this dip is absent in Fig. and can be attributed to the adaptation phase of MQL.
4.3 ABLATION EXPERIMENTS
We conduct series of ablation studies to analyze the different components of the MQL algorithm. We use two environments for this purpose, namely Half-Chheethh-Fwd
and Ant-Fwd-Baa Bacck. Fig. 4a shows that the adaptation in MQL in (18) and (19) improves performance. Also observe that MQL has a smaller standard deviation in the returns as compared to TD3-context which does not perform any adaptation; this can be seen as the adaptation phase making up for the lost performance of the meta-trained policy on a difficult task. Next, we evaluate the importance of the additional data from the replay buffer in MQL. Fig. 4b compares the performance of MQL with and without updates in (19). We see that the old data, even if comes from different tasks, useful improve the performance on top of (18). Fig. 4c shows the effectiveness of setting À = - ESS as compared to a fixed value of À = 0.5. We see that modulating the quadratic penalty with ESS helps, the effect is minor for Sec. 4.3. The ideal value of À depends
given task and using 1 ESS can help to adjust to different tasks without the need to do hyper-parameter search per task. Finally, Fig. 5 shows the evolution of À and (()) during meta-training. The coefficient
about 0.55 and (3x) is 0.8 for a large fraction of the time. The latter indicates that propensity score estimation is successful in sampling transitions from the meta-training replay buffer that are similar to the validation tasks. The value of À remains relatively unchanged during training. This value indicates the fraction of transitions in the old data that are similar to those from the new tasks; since there are two distinct tasks in Ant-Fwd-Back the value À 0.55 is appropriate.
Hif-Chrwent-Furs-RRae
Anerapa Valioiton Betmm
1530
Avee aa Variaadann Paeuun
1500
5500
1330
1000
5000
300
800
000
-m)
A
A
E50
2.00
0.25
0.53
0.75
1.00
125
150
0.00
0.25
5.50
2.75
LE3
1.25
1.50
0.00
0.25
5.50
2.75
153
1.25
1.50
(a.1)
(b.1)
(c.1)
Fwe Back
Pwd Back
Patt Pwd Back
Avergg Valk caltion Ratun
Maadyy
Aeoeago Yalloatoon Ratuu
0.32
0:32
+00
800
430
100
400
Am
NMOL 17 10))
60
0.5
5.5
1.5
2.0
255
0.0
0.5
1.0
2.0
2.5
0.0
0.5
1.0
1.5
2.
255
ning
(a.2)
(b.2)
(c.2)
(a) MQL TD3-context
(b) MQL MQL (3 0)
(c) Choice of A = 1 ESS
Figure 4: Ablation studies to examine various components of MQL.
4.4 RELATED WORK
Learning to learn: The idea of building inductive bias for learning new task by training large number of related tasks was established in series of works (Utgoff, 1986; Schmidhuber, 1987; Baxter, 1995; Thrun, 1996; Thrun& & Pratt, 2012). These papers propose building a base learner that fits on each task and meta-learner that learns properties of the base leamers to output new base
8

---

Published as a conference paper at ICLR 2020
learner for new task. The recent literature instantiates this idea in two forms: (i) the meta-learner directly predicts the base-learner (Wang et al., 2016; Snell et al., 2017) and (ii) the meta-learner learns the updates of the base-learner (Bengio et al., 1992; Hochreiter al., 2001; Finn et al., 2017).
Meta-training versus multi-task training: Meta- training aims to train a policy that can be adapted efficiently on new task. Conceptually, the improved efficiency of a meta-learner comes from two things:
Ant-F Sot-Fwo Back
0.1
Bi2l
1
1-1-888
foo Ra Wiy pam
(i) building a better inductive bias to initialize the learning (Schmidhuber al., 1997: Baxter, 1995; 2000; Mitchell, 1980), (ii) learning better learning pro- cedure (Bengio et al.. 1997; Lee et al., 2019). The two notions of meta-learning above are complementary to each other and in fact, most recent literature using deep neural networks, e.g., MAML (Finn et al., 2017) and Prototypical Networks (Snell et al., 2017) confirms to the first notion of building a better inductive bias.
0.6
0.5
0.4
0.3
1.0 1.5
3.0
Mata-traiiing Time-staaa
tel
Figure 5: Evolution of A and 3(z) during meta-training.
The multi-task training objective in MQL is the simplest possible instantiation of this idea: maximizes the aver-
age reward on all tasks and learns a better prior without explicitly training for improving adaptation. This aspect of MQL. coincides with recent trend in meta-learning for image classification where has been observed that modifications to episodic meta-training (Snell et al., 2017: Gidaris & Ko- modakis, 2018: Chen al., 2018), or even foregoing meta-training completely (Dhillon al., 2019) performs better. We speculate two reasons for this phenomenon: (i) meta-training methods are com- plex to implement and tune, and (ii) powerful function classes such as deep neural networks may have leftover capacity to adapt to new task even if they are not explicitly trained for adaptation.
Context-based approaches: Both forms of meta-lcarning above have been employed relatively successfully for image classification (Snell et al., 2017; Ravi & Larochelle, 2016; Finn et al., 2017). It has however been difficult to replicate that empirical performance in RL: sensitivity to hyper- parameters (Henderson et al., 2018) precludes directly predicting the base-learner while long-range temporal dependencies make difficult to learn the updates of the base learner (Nichol et al., 2018). Recent methods for RL. instead leverage context and learn a policy that depends on just on the current state Ie but on the previous history. This may be done in a recurrent fashion (Heess et al., 2015; Hausknecht & Stone, 2015) or by learning a latent representation of the task (Rakelly et al., 2019). Context is a powerful construct: as Fig. shows, even a simple vanilla RL algorithm (TD3) when combined with context performs comparably to tfte-of-theearr meta-RL algorithms. However, context is meta-training technique, it does not suggest way to adapt policy to new task. For instance, Rakelly et al. (2019) do not update parameters of the policy on a new task. They rely on the latent representation the context variable generalizing to new tasks. This is difficult if the new task is different from the training tasks; we discuss this further in Sec. 3.
saiey--rrddient-baeed algorithms versus off-policy methods: policy-gradient-bee have high sample complexity (Ilyas et al., 2018). This is particularly limiting for meta-RL (Finn et al.. 2017; Rothfuss et al., 2018; Houthooft et al., 2018) where one (i) trains on large number of tasks and, (ii) aims to adapt to a new task with few data. Off-policy methods offer substantial gains in sample complexity. This motivates our use of off- off-policy updates for both traaiitainnng and adaptation. Off-policy updates allow using past data from other policies. MQL exploits this substan- tially, it takes up to 100x more updates using old data than new data during adaptation. Off- Off-poliiy algorithms are typically very sensitive to
methods
(Fujimoto et al., 201 2018a) but we show that MQL is robust such sensitivity because it adapts automatically the distribution shift using the Effective Sample Size (ESS).
Propensity score estimation has been extensively studied in both statistics (Robert & Casella, 2013; Muionero-Candell et al., 2009) and RL (Dudík et al., 2011; Jiang & Li, 2015; Kang et al., 2007: Bang & Robins, 2005). It is typically used to reweigh data from the proposal distribution to compute estimators on the target distribution. MQL. uses propensity scores in novel way: we fit propensity score estimator on subset of the meta-training replay buffer and use this model to sample transitions

---

Published as a conference paper at ICLR 2020
from the replay buffer that are similar to the new task. The off off-policy updates in MQL essential to exploiting this data. The coefficient of the proximal term in the adaptation objective (18-19) using the effective sample size (ESS) is inspired from the recent work of Fakoor al. (2019).
5 DISCUSSION
The algorithm proposed in this paper, namely MQL, builds upon on three simple ideas. First, Q- learning with context sufficient be competitive on current meta-RL. RL benchmarks. Second, maxi- mizing the average reward of training tasks is effective meta- veta-learning technique. The meta- veta-training phase of MQL is significantly simpler than that of existing algorithmss and yet it achieves comparable performance to the state of the art. This suggests that we need to re-think meta-learning in the context of rich function approximators such as deep networks. Third, if one is to adapt new tasks with few data, it is essential to exploit every available avenue. MQL recycles data from the meta-training replay buffer using propensity estimation techniques. This data is essentially free and is completely neglected by other algorithms. This idea can potentially be used problems outside RL. such as few-shot and zero-shot image classification.
Finally, this paper sheds light on the nature of benchmark environments in meta-RL. The fact that even vanilla Q-learning with a context variable-withol meta-training and without any adaptation- is competitive with state of the art algorithms indicates that (i) training and validation tasks in the current meta-RI. benchmarks are quite similar to each other and (ii) current benchmarks may be insufficient to evaluate meta-RL. algorithms. Both of these are a call to action and point the need to invest resources towards creating better benchmark problems for meta-Re RL that drive the innovation of new algorithms.
REFERENCES
Deepak Agarwal, Lihong Li, Alexander Smola. Linear-time estimators for propensity scores. Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pp. 93-100, 201 1.
Heejung Bang and James Robins. Doubly robust estimation in missing data and causal inference models. Biometries, 61(4):962-973, 2005.
Jonathan Baxter. Learning internal representations. Flinders University of S. Aust., 1995.
Jonathan Baxter. A model of inductive bias learning. Journal artificial intelligence research, 12:149-198, 2000.
Samy Bengio. Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of synaptic learning rule. In Preprints Conf. Optimality Artificial and Biological Neural Networks, pp. 6-8. Univ. of Texas, 1992.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Geesei. On the optimization of synaptic learning rule, 1997.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. OpenAl Gym. UrXiv:1606.015.0 2016.
Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look few-shot classification. 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv: 1406::/406.1111 2014.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. arXiv: 1909. wtt.:/9999....... 2019.
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning, arXiv:1611..0277) 2016.
Miroslav Dudik, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. arXiv: 1rXiv:1103.4600 460,4601, 2011.
Víctor Elvira, Luca Martino, and Christian P Robert. Rethinking the effective sample size. arXiv: arXXiv1809.000 1809.04129 2018.
10

---

Published as a conference paper at ICLR 2020
Rasool Fakoor, Pratik Chaudhari, and Alexander Smola. P3o: Policy-on policy-off policy optimization. 19Xiv:9905.017560 2019.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning-Vonn Volume 70, pp. 1126- 1135. JMLR. org, 2017.
Scott Fujimoto. David Meger, and Doina Precup. Off-po deep reinforcement learning without exploration. 1/X2000//. 2018a.
Scott Fujimoto, Herke van Hoof, and Dave Meger. Addressing function approximation error in actor-critic methods. Xrv::: 802.09477, 2018b.
Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEFR Conference Computer Vision and Pattern Recognition, pp. 4367-4377, 2018.
Matthew Hausknecht and Peter Stone. Deep recurrent q-lcarning for partially observable mdps. In 2015 AAAI Fall Symposium Series, 2015.
Nicolas Heess, Jonathan J Hunt, Timothy P Lillicrap, and David Silver. Memory-based control with recurrent neural networks. 1rXiv:1512.044:
2015.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep rein- forcement learning that matters. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pp. 87-94. Springer, 2001.
Rein Houthooft, Yuhua Chen, Phillip Isola, Bradly Stadie, Filip Wolski. OpenAI Jonathan Ho. and Pieter Abbeel. Evolved policy gradients. Advances Neural Information Processing Systems, pp. 5400-5409, 2018.
Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Alek- sander Madry. Are deep policy gradient algorithms truly policy gradient algorithms? arXiv:1811.02553. 2018.
Nan Jiang Lihong Li. 1rXiv:/51.00322. 2015.
Doubly robust off-policy value evaluation for reinforcement learning.
Joseph DY Kang. for estimating population mean from incomplete data. Statistical science, 22(4):523-33) 2007.
Joseph I. Schafer, al. Demystifying double robustness: comparison alternative strategies
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.699 2014.
Augustine Kong. A note importance sampling using standardized weights. University of Chicago. Dept. of Statistics, Tech. Rep, 348, 1992.
Kwonjoon Lee, Subhransu Maji. Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. arXiv: 0e5//994000755 2019.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. 150i 029700299, 2015.
Tom Mitchell. The need for biases in learning generalizations. Department of Computer Science, Laboratory for Computer Science Research... 1980.
Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. 19X.99880.00299, 2018.
Joaquin Cnionero-Candele Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning. The MIT Press, 2009.
Kate Rakelly, Aurick Zhou, Deirdre Quillen, Chelsea Finn, and Sergey Levine. Efficient off-policy meta- reinforcement learning via probabilistic context variables. 1903.09900..2555 2019.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Sashank Reddi, Barnabás Póezos, and Alexander J. Smola. Doubly robust covariate shift correction. In AAAI, 2015.
Sidney Resnick. A probability path. Springer Science & Business Media, 2013.
Christian Robert and George Casella. Monte Carlo statistical methods. Springer Science & Business Media, 2013.
Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv: arXiv:/810.060 1810.06784, 2018.
11

---

Published as a conference paper at ICLR 2020
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how learn: The meta-meta-... hook.) Diploma thesis, Instina f. Informatik, Tech. Univ. Munich, 1987.
Jürgen Schmidhuber, Jieyu Zhao, and Marco Wiering. Shifting inductive bias with success-story algorithm, adaptive levin search, and incremental self-imprrvement. Machine Learning, 28(1))105-133 Jul 1997. ISSN
1573-0565.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning, volume 37, pp. 1889-1897, 2015.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International Conference Machine Learning, 2014.
Adrian Smith. Sequential Monte Carlo methods in practice. Springer Science & Business Media, 2013.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances Neural Information Processing Systems, pp. 4077-4087, 2017.
Sebastian Thrun. Is learning n-th thing any easier than learning the first? In Advances in neural information processing systems, pp. 640-640 646, 1996.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science Business Media, 2012.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujeco: A physics engine for model-based control. In 2012
JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
Paul E Utgoff. Shift of bias for inductive concept learning. Machine learning: An artificial intelligence approach, 2:107-148, 1986.
Jane Wang. Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer. Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforeement learn. CoRR, abs/1611 010///611.076666 105763, 2016. URL http: /t////axxxc /arxiv. org/ abs 1611.161111 05763.
12

---

Published as a conference paper at ICLR 2020
A PSEUDO-CODE
The pseudo-code for MQL during training and adaption are given in Algorithm 1 and Algorithm 2. After MQL is trained for given environment described in Algorithm 1, returns the meta-trained policy 0 and replay buffer containing train tasks.
Next, Algorithm 2 runs the adaptation procedure which adapts the meta-trainee policy to test task D with few data. To do so, MQL optimizes the adaptation objective into two steps. After gathering data from a test task D. MQL first updates the policy using the new data (line 4). MQL then fits logistic classifier on mini-batch of transitions from the meta-training replay buffer and the transitions collected from the test task and then estimates ESS (lines 5-6). Finally, the adaptation step runs for iterations (lines 7 10) in which MQL exploit past data in which it uses propensity decide whether or not a given sample is related to the current test task.
Algorithm 1: MQI. Mcta-training
Input: Set of training tasks Pmeta
Initialize the replay buffer
2 Initialize parameters 0 of an off-policy method, e.g., TD3
3 while not done do
If Rollout and update policy
Sample task D Diees
Gather data from task using policy wo while feeding transitions through context GRU. Add trajectory to the replay buffer.
7
6 +
Sample mini-batch from buffer
8
Update parameters using mini-batch Eqn. (15)
Ometa f-
10 return 0meaa replay buffer
Algorithm 2: MQL- Adaptation
Input: Test task D, meta- replay buffer, meta-traines policy Ometa
Initialize temporary buffer buf
0 <- Ometa
3 buf <- Gather data from using Wemeta
4 Update Eqn. (18) using buf
s (DD using and meta-training replay buffer using Eqn. (12)
Estimate ESS using 3(D) using Eqn. (13)
for i < do
6 f- sample mini-batch from meta-training replay buffer
Calculate 3 for 6
Update 8 using Eqn. (19)
Evaluate new rollout from task D
12 return
B OUT-OF-DISTRIBUTION TASKS
MQL is designed for explicitly using data from the new task along with off-policy data from old, possibly very different tasks. This on account two things: (i) the loss function of MQL does not use the old data if is very different from the new task, close to zero for all samples, and (ii) the first term in (18) makes multiple updates using data from the new task. To explore this aspect, we create an out-of-distribution task using the "Half-Cheetah- Vel environment wherein we use disjoint sets of velocities for meta-training and testing. The setup as follows: Salf-Cheetahh-e Nel-OOD-Medium: target velocity for training task sampled uniformly randomly from [0, 2.5) while that for test task is sampled uniformly randomly from [2.5, 3.0).
13

---

Published as a conference paper at ICLR 2020
This is what we call "medium" hardness task because although the distributions of train and test velocities is disjoint, they are close to each other.
target velocity for a training task is sampled uniformly randomly from [O, 1.5] while that for test task is sampled uniformly randomly from [2.5, 3.0) This is a "hard" task because the distributions of train and test velocities are far away from each other.
Saal--Cheeta--el--OOO
Fig. 6a shows that MQL significantly outperforms PEARL when the train and test target velocities come from disjoint sets. We used the published code of PEARL (Rakelly et al., 2019) for this experiment. This shows that the adaptation in MQL is crucial to generalizing new situations which are not part of the meta-traine meta-training process. Fig. 6b shows the evolution of the proximal penalty coefficient À and the propensity score (22) during meta-training for the medium-hard task. We see that À = 0.8 while ((z) = 0.2 throughout training. This indicates that MQL automatically adjusts its test-time adaptation
only few samples in (19) if the test task provides transitions quite different than those in the replay buffer.
We next discuss results the harder task Half-Cheetah--ee Cheetah-Vel-OOO------ Vel-OOD-Hard There is very large gap between training and test target velocities in this case. Fig. 7a shows the comparison with the same test protocol as the other experiments this paper. In particular, we collect 200 time-steps from the new task and use it for adaptation in both MQL and TD3-context. Since this task is particularly hard, we also ran an experiment where 1200 time-steps (6 episodes) are given to the two algorithms for adaptation. The results are shown in Fig. 7b. In both cases, we see that MQL is better than TD3-context by a large margin (the standard deviation on these plots is high because the environment is hard). Note that since re-initialize the hidden state of the context network at the beginning of each episode. TD3-context cannot take advantage of the extra time-steps. MQL on the other hand updates the policy explicitly and can take ad advantage this extra data.
For sake of being thorough, we collected 800 time- from the new task from the same episode, the results are shown in Fig. 8a. We again notice that MQL results in slightly higher rewards than TD3-context in spite of the fact that both the algorithms suffer large degradation performance as compared to Figs. 7a and 7b.
Figs. 7c, 7d and 8b show that the proximal penalty coefficient À = and the propensity score B(2) z 0 for a large fraction of training. This proves that MQL is able to automatically discard samples unrelated to the new test during the adaptation phase.
Half Cheelsh Vel OOD Medium
Half Cheelah Vel OOD Medium
Average Valitetion Reeuur
-100
0.8
-200
0.6
-300
0.4
-400
.000
0.2
PEARL
MCL (oum)
800
0.00
1.00
1.25
1.50
Meta
1uB
166
(a)
(b)
Figure 6: Comparison of the average return of MQL (orange) against existing PEARL algorithms (blue).
Fig. 6a shows that MQL significantly outperforms PEARL when the train and test target velocities come from disjoint sets. Fig. 6b shows the evolution of the proximal penalty coefficient À and the propensity score B(z). We see that ((x) is always small which demonstrates that MQL automatically adjusts the adaptation in (19) if the test task is different from the training tasks.
14

---

Published as a conference paper at ICLR 2020
Half- Seaf-Chetthh-eilee Vel---D-----O-rrr
Helf-Chheeah- eel-OOD-Hardd
Average Velddsion Relum
-330
Valicaaion Ratun
-350
-450
-450
M
-000
-000
-530
-350
112s coneeet (num)
TEIS contect (nars)
-000
N00, (ouns)
-500
MGL (ours)
5
Mala-tränning Timeesttapa
Mets-traininn Time-staps
(a)
(b)
Hall-Cheeean-ke--00D-aard
Taal-Chostah--ee-o00--aa
0.8
0.5
8121
(12)
A1
0.4
0.4
02
0.0
Melax leainnnniinngg Time elegre
166
Mela trainin Time eleepe
1cG
(c)
(d)
Figure 7: (a.b) Comparison of the average return of MQL (orange) against TD3-context (blue). Fig. 7a shows the comparison with the same test protocol as the other experiments in this paper. In particular, we collect 200 time-steps from the new task and use it for adaptation in both MQL and TD3-context. Since this task is particularly hard, we also ran experiment where 1200 time-steps (6 episodes) where results are shown in Fig. 7b. In both cases, we that MQI. better than TD3-context environment hard). (c.d) Evolution of and 3(2) during meta-training.
large margin (the standard deviation
on these plots is high because shows the evolution of the proximal penalty coefficient À and the propensity score ((2). We see in Fig. 7c and Fig. 7d that (=z) is always small which demonstrates that MQL automatically adjusts the adaptation if the test task is different from the training tasks.
Half- Cwwethh--ee----ece Vaattaah-Ve-OOO-HHawa
Hall- Cheethh-etthh Swt--Chetthh--e--OO0 Vei-OOD---arrr
1.0
Average Validtion Delumn
0.8
0.6
.1 ESS
2000
-2200
02
TD3 contect (oU1))
2400
MCL.
0.0
2
Meta training Time steps
Meta training Time stepe
1c6
(a)
(b)
Figure 8: (a) Comparison of the average return of MQL (orange) against TD3-context (blue). For these experiments, collected 800 simessteps from the new task from the same episode, the results shown Fig. 8a. We again notice that MQL results in slightly higher rewards than TD3-context in spite of the fact that both the algorithms suffer large degradation performance as compared to Figs. and 7b. (b) Evolution and ((2) during meta-training shows the evolution of the proximal penalty cocfficient À and the propensity score (()).
15

---

Published as a conference paper at ICLR 2020
Istt Cheetah
Cheetah
1sst Cheete
Aoengg Vall Surtan Plouun
-100
-900
-190
-160
-190
-200
-20
-300
A-32
720 2e00
0.0
3.0
1.0
1.5
20
50
0.5
1.0
20
0.
2.5
1.3
1.5
20
(c)
Walker Params
Walker
Viskeer Params
800
Avtrgg Valkattn Rosamn
Valiation
IKOL 12
triniig steps
Meta-rokiing
training Time-stops
(d)
(e)
(I)
Figure 9: Ablation studies examine various components of MQL. Fig. shows that the adaptation in MQL in (18) and (19) improves performance. One reason for that is because test and training tasks in Walker- DD-Params are very similar it shown in Fig. 10b. Next, we evaluate the importance of the additional data from the replay buffer in MQL. Fig. 9b compares the performance of MQL. with and without updates in (19). We that the old data, even if comes from different tasks, useful to improve the performance top (18). Fig. 9c and Fig. 9f show the effectiveness of setting À
ESS compared to fixed value of X 0.5. We see that modulating the quadratic penalty with ESS helps, the effect is minor for Sec. 4.3. The ideal value of A depends on given task and using ESS can help to adjust to different tasks without the need to do hyper-parameter search per task.
Half- Chaf-Cheetah--el
Bllkker-DDPPramms
0.8
0.8
- PIS
F55
3.6
0.4
34
Meta training Time
168
Meta training Time
1e8
(a)
(b)
Figure Evolution À and ((z) during meta-training shows the evolution of proximal penalty cneffi-
cient À and the propensity score ((z). We see in Fig. 10a that (z)) stays around 0.4 which demonstrates MQL automatically adjusts the adaptation if the test task is different from the training tasks. Fig. 10b shows that test and training tasks very similar ((z) is around 0.6.
16

---

Published as a conference paper at ICLR 2020
C MORE ABLATION STUDIES
We conduct a series additional ablation studies to analyze the different components the MQL algorithm. We
environments for this purpose, namely Half-Cheetah-Val Vel and Walker-2D- Params. Fig. 9 and Fig. 10 show the result of these experiments. These experiments show that adaptation phase is more useful for Half-Cheetah-VV than Walker-2D-Parann as test and training tasks are very similar in Walker-2D-PParan which helps TD3-context achieves strong performance that leaves window for improvement with adaptation.
D HYPER-PARAMETERS AND MORE DETAILS OF THE EMPIRICAL RESULTS
Table Hyper-parameters for MQL and TD3 for continuous-control meta-RL benchmark tasks. We use a network with two full-connected layers for all environments. The batch-size in Adam is fixed to 256 for all environments. The abbreviation HC stands for Half-Cheetah, These hyper-paramelers were tuned by grid-search.
Humanoid
HC-Vel
Ant-FB
Ant-Goal
Walker
HC-FB
3 clipping
1
1.1
1.2
2
0.8
TD3 exploration noise
0.3
0.3
0.2
0.3
TD3 policy noise
0.2
0.3
0.3
0.4
0.3
0.2
TD3 policy update frequency
2
4
Parameter updates per iteration (meta-training)
500
1000
200
1000
200
200
Adaptation parameter updates per
10
5
10
5
10
10
episode (eq. 18)
Adaptation parameter updates per
200
400
100
400
400
300
episode (cq. 19)
GRU sequence length
20
10
Context dimension
20
20
15
30
30
30
Adam learning rate
0.0003
0.001
0.0003
0.0004
0.0008
0.0003
17


Reviews:
[{"review_id": "SJxhwmZqFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors investigate meta-learning in reinforcement learning with respect to sample efficiency and the necessity of meta-learning an adaptation scheme. Based on their findings, they propose a new algorithm 'MQL' (Meta-Q-Learning) that is off-policy and has a fixed adaptation scheme but is still competitive on meta-RL benchmarks (a distribution of environments that differ slightly in their reward functions).\n\nThey motivate the paper by data-inefficiency of current meta-learning approaches and empirical results suggesting that meta-learning the adaptation scheme is less important than feature reuse. \n\nOn the other hand, their introduction section would benefit from additional references to the kind of meta-learning they describe. In particular, their so-called \"definition of meta-learning\" is mostly about domain randomization (e.g. Tobin et al. 2017 https://arxiv.org/abs/1703.06907) and not about the broader 'learning to learn' RL methodology (in particular Schmidhuber 1994 \"On learning how to learn learning strategies\").\n\n**The authors make the following contributions:**\n\n1. They show that Q-Learning trained on multiple tasks with a context variable as an input (an RNN state summarizing previous transitions) is competitive to related work when evaluated on a test task even though no adaptation is performed\n\n2. Based on these observations, they introduce a new method for off-policy RL that does not directly optimize for adaptation but instead uses a fixed adaptation scheme\n\n3. The new method leverages data during meta-testing that was collected during meta-training using importance weights for increased sample efficiency\n\n**Overall, we believe the contributions are significant and sufficiently empirically justified.**\n\nThere are strong similarities, however, to parallel work on analyzing whether MAML relies on feature reuse or rapid learning (Raghu et al. 2019 https://arxiv.org/abs/1909.09157).\n\nThis work and the present submission conclude that feature reuse is much more significant than meta-learning an adaptation scheme when evaluated on current meta-RL benchmarks. This is a significant result and supports the new method developed in this paper.\n\nDuring meta-training, their proposed method maximizes only the average return across tasks, not the ability to adapt from the resulting parameters.\n\nTheir method introduces a fixed (non-learned) adaptation scheme that performs favorably compared to certain methods from the existing meta-learning literature and demonstrates that even dropping this adaptation still does well.\n\nThere are strong similarities to Nichol et al. 2018 (https://arxiv.org/abs/1803.02999). We encourage the authors to relate this work to Raghu et al. 2019 and Nichol et al. 2018.\n\n**Despite these interesting results, we strongly disagree with the meta-learning narrative of their new method.**\n\nBecause the adaptation scheme is no longer optimized directly, instead a fixed adaptation scheme is assumed, hence the approach in this paper is no longer a meta-learning algorithm.\n\nInstead, this method has strong similarities with transfer-learning and domain adaptation (first training on one distribution of tasks, then fine-tuning on another task).\n\nThe authors should discuss the links to these fields of research, and clarify what's really novel, already in the abstract.\n\nFor example, on page 2 the authors claim that optimizing the multi-task objective (the mean error across tasks) is the simplest form of meta-learning. This objective, however, is NOT meta-learning.\n\n**Decision.**\n\nThe submission contains strong empirical results emphasizing the significance of feature reuse and the insignificance of learned adaptation on the tested meta-RL benchmarks.\n\nThe reuse of experience from meta-training during meta-testing by employing importance weights is also an interesting contribution.\n\nIn contrast, we are not satisfied with the presentation of their new approach as a meta-learning approach. This method should be introduced along the lines of: 'Transfer Learning / Feature Reuse in RL is competitive to meta-learning across similar tasks'.\n\nIn its current form, we tend to reject the paper because it further obscures what the term meta-learning refers to. The authors are confusing it with more limited transfer learning. \n\nAdditionally, it was not clear to us whether the quadratic penalty they add to their adaptation scheme is only empirically valid or whether there is a theoretical reason.\n\nFor now, we'd lean towards rejecting this submission, but we might change our minds, provided the comments above were addressed in a satisfactory way - let us wait for the rebuttal.\n\nEdit after rebuttal: score increased!", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxOyn32KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\nThe authors propose meta Q-learning, an algorithm for off-policy meta RL. The idea is to meta-train a context-dependent policy to maximize the expected return averaged over all training tasks, and then adapt this policy to any new task by leveraging both novel and past experience using importance sampling corrections. The proposed approach is evaluated on standard Mujoco benchmarks and compared to other relevant meta-rl algorithms.\n\nComments\n--------------\n\nMeta-rl is a relevant direction for mitigating the sample-complexity of rl agents and allowing them to scale to larger domains. This work proposes interesting ideas and overall it constitutes an nice contribution. In particular, I found interesting (and at the same time worrying) that a simple q-learning algorithm with hidden contexts compares favorably to state-of-art meta-rl approaches in standard benchmarks. The paper is well-organized and easy to read. Some comments/questions follow.\n\n1. (15) is probably miss-written (theta is trained to maximize the TD error)\n\n2. In the adaptation phase, are (18) and (19) performed one after the other? Could they be done at the same time by setting to 1 the importance weights of the new trajectories and sampling from the whole experience (new and old)?\n\n3. Note that the ESS estimator (13) diverges to infinity when all weights are close to zero. Is it clipped to [0,1] in the experiments? See e.g. [1] or [2] for more robust estimators that are bounded.\n\n4. Since many recent works try to improve the generalization capabilities of meta-rl algorithms, I was wondering how the proposed approach generalizes to out-of-distribution tasks (i.e., tasks that are unlikely to occur at meta-training). Though it is never mentioned in the paper, I believe the proposed method has the potential to be robust to negative transfer since the importance weights (which would be very small for very different tasks) should automatically discard old data and focus on new data alone. This is in contrast to many existing methods where the meta-trained model might negatively bias the learning of very different tasks. I think an experiment of this kind would be valuable to improve the paper.\n\n[1] Elvira, V., Martino, L., & Robert, C. P. (2018). Rethinking the effective sample size. arXiv preprint arXiv:1809.04129.\n[2] Tirinzoni, A., Salvini, M., & Restelli, M. (2019, May). Transfer of Samples in Policy Search via Multiple Importance Sampling. In International Conference on Machine Learning (pp. 6264-6274).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJgWhC1CtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Meta Q-Learning (MQL), an algorithm for efficient off-policy meta-learning. The method relies on a simple multi-task objective which provides initial parameter values for the adaptation phase. Adaptation is performed by gradient descent, minimizing TD-error on the new validation task (regularizing towards initial parameter values). To make adaptation data efficient, the method makes heavy use of off-policy data generated during meta-training, by minimizing its importance weighted TD-error. Importance weights are estimated via a likelihood ratio estimator, and are also used to derive the effective sample size of the meta-training batch, which is used to adaptively weight the regularization term. Intuitively, this has the effect of turning off regularization when meta-training trajectories are “close” to validation trajectories. One important but somewhat orthogonal contribution of the paper is to highlight the importance of context in meta-learning and fast adaptation. Concretely, the authors show that a simple actor-critic algorithm (TD3), whose policy and value are conditioned on a context variable derived from a recurrent network performs surprisingly well in comparison to SoTA meta-learning algorithms like PEARL. MQL is evaluated on benchmark meta-RL environments from continuous control tasks and is shown to perform competitively with PEARL.\n\nI have mixed opinions on this paper. On the positive side, and subject to further clarifications (see below), the paper seems to confirm that multi-task learning is almost sufficient to solve current meta-RL benchmarks in continuous control, without adaptation, as long as policy and critic are conditioned on a recurrent task context. This either highlights the strength of multi-task learning, or the inadequacies of current meta-RL benchmarks: either of which will be of interest to the community. On the other hand, the proposed MQL algorithm is only shown to significantly outperform this new baseline TD3-context agent on 1 of 6 tasks (Ant-Goal-2D), and furthermore the ablative analysis seems to suggest that the importance weighting and adaptive weighting of trust region are not very effective, and do not significantly change the performance of the method. The take-away seems to be that while context is crucial to generalization on these validation tasks, adaptation is not but can indeed be improved in a data-efficient manner with fine-tuning. MQL is only then a second thread to this story.\n\n\nClarifying Questions:\n\n* The text and figures could be much clearer with respect to what is being measured/presented. Can you confirm that you report average validation returns as a function of meta-training steps, where validation performance is estimated after training on (at most) 2x200 transitions from the validation task (|D_new|=400)? This would match the protocol from PEARL. Fig 4(a) would then imply that one can get close to SoTA results on Ant-Fwd-Back + Half-Cheetah-Fwd-Back with no adaptation whatsoever (using TD3-context).\n* Could the authors provide more details about how context is generated, in particular whether the GRU is reset on episode boundaries or not? If the recurrent context does span episode boundaries, are rewards being maximizing over a horizon greater than the episode length (similar to RL2)?\n* How do you reconcile your result that a deterministic encoder is sufficient, compared to Figure 7 of PEARL which shows stochasticity is paramount for performing structured exploration during adaptation?\n* Ablative analysis seems to suggest that off-policy learning plays a very minimal role during adaptation (beta=0). Can you confirm this interpretation? Would this not suggest that regularization via fine-tuning (regularized to multi-task prior) and context are sufficient to solve these meta-RL tasks? This would be a sufficient contribution by itself but unfortunately does little to validate the proposed method.\n* Could you repeat the ablative analysis on all of the 6 tasks? Currently, this is performed on the two tasks for which TD3-context does best which leaves little room for improvement.\n* Section 4.4. Propensity Score Estimation. “Use this model to sample transitions from the replay buffer that are similar to the new task”. Is this done via rejection sampling? This should be described more prominently in Section 3, along with a detailed description of the proposed MQL algorithm.\n\nDetailed Comments:\n* Paper desperately needs an algorithmic box, which clear and concise description of algorithm (how losses are interleaved, etc). Importantly, do the authors pretrain \\theta_meta using the multi-task loss before doing adapation?\n* Please add more informative labels to axes for all your figures: timesteps for [validation,training] ? Same for return. Please also augment captions to make figures as stand-alone as possible.\n* MQL encompasses technique for off-policy training using a discriminator to estimate a likelihood ratio. It would be nice to evaluate this in standard off-policy learning setting, instead of it being limited to meta-learning.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper’s contribution is twofold: 1) it proposes a new meta-RL method that leverages off-policy meta-learning by importance weighting, and 2) it demonstrates that current popular meta-RL benchmarks don’t necessarily require meta-learning, as a simple non-meta-learning algorithm (TD3) conditioned on a context variable of the trajectory is competitive with SoTA meta-learning approaches. 

The reviewers all agreed that the approach is interesting and the contributions are significant. I’d like to thank the reviewers for engaging in a spirited discussion about this paper, both with each other and with the authors. There was also a disagreement about the semantics of whether the approach can be classified as “meta-learning”, but in my opinion this argument is orthogonal to the practical contributions. After the revisions and rebuttal, reviewers agreed that the paper was improved and increased their ratings as a result, with all recommending accept.

There’s a good chance this work will make an impactful contribution to the field of meta-reinforcement learning and therefore I recommend it for an oral presentation.

Decision: Accept (Talk)