On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
Authors: Alireza Fallah
Kristian Georgiev
Aryan Mokhtari
Asuman E. Ozdaglar
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta-learning theory
reinforcement learning theory
optimization

Abstract: We consider Model-Agnostic Meta-Learning (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Markov Decision Processes (MDPs) that can be updated by one step of \textit{stochastic} policy gradient for the realized MDP. In particular, using stochastic gradients in MAML update steps is crucial for RL problems since computation of exact gradients requires access to a large number of possible trajectories. For this formulation, we propose a variant of the MAML method, named Stochastic Gradient Meta-Reinforcement Learning (SG-MRL), and study its convergence properties. We derive the iteration and sample complexity of SG-MRL to find an $\epsilon$-first-order stationary point, which, to the best of our knowledge, provides the first convergence guarantee for model-agnostic meta-reinforcement learning algorithms. We further show how our results extend to the case where more than one step of stochastic policy gradient method is used at test time. Finally, we empirically compare SG-MRL and MAML in several deep RL environments.

On the Convergence Theory of Debiased Model-Agnostic Meta-Reinforcement Learning
Alireza Fallah
Kristian Georgiev
EECS Department
EECS Department
Massachusetts Institute of Technology afallah@mit. edu
Massachusetts Institute of Technology krisgrgamit edu
Aryan Mokhtari
Asuman Ozdaglar
ECE Department
EECS Department
The University of Texas Austin mokhtar @aust tin. utexas edu
Massachusetts Institute of Technology as asuman@mit. edu
Abstract
We consider Model-Agnostic Meta-Learming (MAML) methods for Reinforcement Learning (RL) problems, where the goal is to find a policy using data from several tasks represented by Markov Decision Processes (MDPs) that can be updated by one step of stochastic policy gradient for the realized MDP. In particular, using stochastic gradients in MAML update steps crucial for RL problems since computation of exact gradients requires access to a large number of possible trajectories. For this formulation, we propose a variant of the MAML method, named Stochastic Gradient Meta-Reinforcement Learning (SG-MRL), and study its convergence properties. We derive the iteration and sample complexity of SG- MRL to find e- f-frrst-order stationary point, which, to the best of our knowledge, provides the first convergence guarantee for model-agnostic meta reinforcemnn learning algorithms. We further show how results extend to the case where more than one step of stochastic policy gradient method is used at test time. Finally, we empirically compare SG-MRL and MAML in several deep RL environments.
Introduction
Meta-learning has recently attracted much attention learning to learn approach that enables quick adaptation to new tasks using past experience and data. This is particularly promising approach for Reinforcement Learning (RL) where in several applications, such robotics, group of agents encounter new tasks and need to learn new behaviors or policies through a few interactions with the environment building previous experience [1-9]. Among various forms of Meta-learning gradient-basse based Model-Agnostic Meta-Learning (MAML) formulation [1] is a particularly effective approach which, its name suggests, be applied any learning problem that is trained with gradient-based updates. In MAML, we exploit observed tasks at training time to find an initial model that is trained in way that rapidly adapts of gradient-baat brasient-based update with respect to the loss of the new task.
unseen task test time, after running a few steps
The MAML formulation can be extended to RL problems if we represent each task Markov Decision Process (MDP). In this setting, we assume that we are given set of MDPs corresponding to the tasks that we observe during the training phase and assume that the new task at test time is drawn from an underlying probability distribution. The goal in Model-Agnostic Meta-Reiiffrcemeett Learning (MAMRL) is to exploit this data come up with an initial policy that adapts to new task (drawn from the same distribution) at test time by taking a few stochastic policy gradient steps [1].
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Several algorithms have been proposed in the context of MAMRL [1. 9-12] which demonstrate the advantage of this framework in practice. None of these methods, however, are supported by theoretical guarantees for their convergence rate or overall sample complexity. Moreover, these methods aim to solve specific form of MAMRI that does not fully take into account the stochasticity aspect of RL problems. To be more specific, the original MAMRL formulation proposed in [1] assumes performing one step of policy gradient to update the initial model at test time. However, as mentioned in the experimental evaluation section in [1], it is more common in practice to use stochastic policy gradient, computed over a batch of trajectories, to update the initial model at test time. This is mainly due to the fact that computing the exact gradient of the expected reward is not computationally tractable due to the massive number of possible state-action trajectories. As a result, the algorithm developed in [1] is designed for finding a proper initial policy that performs well after one step of policy gradient, while in practice it is implemented with stochastic policy gradient steps. Due to this difference between the formulation and what is used in practice, the ascent step used in MAML. takes a gradient estimate which suffers from a non-diminishing bias. As the variance of gradient estimation also non-diminishing, the resulting algorithm would not achieve exact first-order optimality. To be precise, in stochastic nonconvex optimization, if we use an unbiased gradient estimator, along with a small stepsize large batch size control the variance, the iterates converge to a stationary point. However, if we use a biased estimator with non-vanishing bias and variance, exact convergence to a stationary point not achievable, even if the variance is small.
Contributions. The goal of this paper is to solve the modified formulation of aggnos-agnosicc meta- reinforcement learning problem in which perform stochastic policy gradient update at test time instead of (deterministic) policy gradient. To do so, we propose a novel stochastic gradient- based method for Meta- RentaRRRiiforeementt Learning (SG-MRL), which is designed for stochastic policy gradient steps at test time. We show that SG-MRL implements an unbiased estimate of its objective function gradient which allows achieving first-order optimality in non-concave settings. Moreover, we characterize the relation between batch sizes and other problem parameters and the best accuracy that SG-MRL can achieve in terms of gradient norm. We show that, for any e >> 0, SG-MRL can find an e- e-firtt-order stationary point if the learning rate is sufficiently small or the batch of tasks is large enough. To the best of our knowledge, this is the first result on the convergence of MAMRL methods. Moreover, we show that our analysis can be extended to the case where more than one step of stochastic policy gradient is taken during test time. For simplicity, we state all the results in the body of the paper for the single-step case and include the derivations of the general multiple steps case in the appendices. We also empirically validate the proposed SG-MRL algorithm in larger- scale environments standard in modern reinforcement learning applications, including a 2DD-navigation problem, and a more challenging locomotion problem simulated with the MuJoCo library.
Related work. Although this paper provides the first theoretical study of MAML for RL, several recent papers have studied the complexity analysis of MAML in other contexts. In particular, the iMAML algorithm which performs an approximation of one step of proximal point method (instead of few steps of gradient descent) in the inner loop was proposed in [13]. The authors focus on the deterministic case, and show that, assuming the inner loop loss function is sufficiently smooth, i.c., the regularized inner loop function is strongly convex, iMAML converges to first-order stationary point. Another recent work [14] establishes convergence guarantees of the MAML method to first-order stationarity for non-convex settings. Also, [15] extends the theoretical framework in [14] to the multiple-step case. However, the results in [14, 15] cannot be applied to the reinforcement learning setting. This is mainly due to the fact that the probubility distribution over possible trajectories of states and actions varies with the policy parameter, leading to a different algorithm that has an additional term which makes the analysis, such as deriving an upper bound on the smoothness parameter, more challenging. We will discuss this point in subsequent sections.
The online meta-nearning setting has also been studied in number of recent works [16-18]. In particular, [17] studies this problem for convex objective functions by casting it in the online convex optimization framework. Also, [16] extends the model-agnostic setup to the online learning case by considering a competitor which adapts to new tasks, and propose the follow the meta leader method which obtains a sublinear regret for strongly convex loss functions.
It is also worth noting that another notion of bias that has been studied in the MAMRL literature [10, 19] differs from what we consider in our paper. More specifically, as will show later, the derivative of the MAML objective function requires access to the second-order information, i.c., Hessian. In [1], the authors suggest a first-order approximation which ignores this second-order term.
2

---

Let [M,}: be the set of Markov Decision Processes (MDPs) representing different tasks'. We assume these MDPs are drawn from distribution p (which we can only draw samples from), and also the time horizon is fixed and is equal to {0, 1,.,, H) for all tasks. For the i-th MDP denoted by M, which corresponds to task i, we denote the set of states and actions by S; and A,, respectively. We also assume the initial distribution over states in S; is given by (.(.) and the transition kernel is denoted by Ps, i.e., the probability of going from state 8 E S; to 8 E S; given taking action a E A; is P((''Is, 18, a). Finally, we assume at state and by taking action the agent receives reward ri a). To summarize, an MDP M, is defined by the tuple (S,, As; [bg, chosen according to random policy which is mixed strategy Pa: rt). For MDP M,, the actions are the set of actions and depends on This leads to biased estimate of the derivative of the MAML objective function, and number of recent works [10, 19] focus on providing unbiased estimates for the second-order term. contrast, here we focus on biased gradient estimates where the bias stems from the fact that in most real settings we do not have access all possible trajectories and we only have access to a mini-batch of possible trajectories. In this case, even if one has access to the second-order term required in the update of MAML, the bias issue we discuss here will remain.
2 Problem formulation
the current state, i.c., the system is in state s E S, the agent chooses action a E A, with probability (als). ). To search over the space of all policies, 0 E RR and denote the policy corresponding to parameter 0 by (-(--:)))
assume these policies are parametrized with
realization of states and actions in this setting is called trajectory, i.e., trajectory of MDP M, can be written as = (8o, 4g ..., 8 ah) where E A, and 8y, E S; for any 0 S h S H. Note that, given the above assumptions, the probability of this particular trajectory is given by
H
H-1
q:(T; 0) = (4(00) II h=O
(aaan 181; 0)
II
Pe(su4ilssh an)-
(1)
A=
Also, the total reward received over this trajectory is RT(T)) == Edn (s az), where 0 S < 1 the discount factor. As a result, for MDP M,, the expected reward obtained by choosing policy (-1:;;) B) is given by
(,( := E.-a. (:00) [T,
(2)
It is worth noting that the gradient (.) admits the following characterization [20-22]
J.((f) B. (9: ()]
(3)
where gi (T; is defined as
H
9: (T: =
v log n. ( (n: ())( (T),
(4)
(st, if we define (T) as (+) = >'r az). In practice, evaluating the exact value of (3)
is not computationally tractable. Instead, one could first acquire batch Di-0 of trajectories drawn independently from distribution (=(,)), and then, estimate (.) by
J. Di.))
Σ (T:
(5)
D
TEDİ.
Also, we denote the probability of choosing (with replacement) an independent batch of trajectories Di. by q. (Di."; ()) (see Appendix A.I for a remark on this).
In this setting, the goal of Model-Agnostic Meta- Eeta-Reinforemment Learning problem introduced in [1] find a good initial policy that performs well in expectation when it updated using one or few
steps of stochastic policy gradient with respect to new task. In particular, for the case of performing one step of stochastic policy gradient, the problem can be written as²
max (i( == Ei-p EEii.. J1 (0 ((.. Piese) PERd
(6)
TTo simplify the analysis, we assume the number of tasks finite
2 From now on, suppress the dependence of batches to simplify the notation.
3

---

Note that by solving this problem we find an initial policy (Meta-policy) that in expectation performs well if we evaluate the output of our procedure after running one step of stochastic policy gradient on this initial policy for new task.
This formulation can be extended to the setting with more than step of stochastic policy gradient as well. To state the problem formulation in this case, let us first define V which is an operator that takes model / and batch 2 input and performs one step of stochastic gradient policy at point and with respect to function Jj and batch D i.e., V2 D*) == 0 + a a (0, Di). Now, we extend problem (6) to the case where we are looking for an initial point which performs well on expectation after it is updated with Ç steps of stochastic policy gradient with respect to a new MDP drawn from distribution p. This problem can be written as
max V0( == Binp E (P...... [ (V. (0, Diest,) ..), Diest Diest, DRRd
(7)
where the operator V; is applied ( times inside the expectation. In this paper, we establish convergence properties of policy gradient methods for both single step and multiple steps of stochastic gradient cases, but for simplicity in the main text we focus on the single step case.
2.1 Second-order information of the expected reward
Due to the inner gradient in (0),, i.e., the objective function of the MAML problem (6), the gradient of the function Vi (0) requires access to the second-order information of the expected reward function (().. To facilitate further analysis, in this subsection we formally present a characterization of expected reward Hessian and its unbiased estimate over batch of trajectories. In particular, the expected reward Hessian (()) is given by (see [22] for more details)
V = E.,gg. (T: 0)],
0) =- Vou: 0) Vo log +)
(8)
where (T; is given by (T; == Σm= log n. (an s8; (1::n)8000))))))))) () (r).
Recall that the reward function is defined RT) == Σt 2 2"ri(st, ut). It is worth noting that based on the expression in (4) we can write gı 0) (T)
Similar to policy gradient, policy Hessian be estimated over a batch of trajectories D indepen- dently drawn with respect to q:(; 0). Specifically, for given dataset Di, we can define (0 Di)
Σ uT(T;) TED
DD
as an unbiased estimator of the Hessian V²),(0). We will use the expressions for the Hessian (0), (8) and the Hessian approximation (0, D*) (9) introduce our proposed method for solving the Meta-RL problem in (6) and its generalized version in (7).
3 Model-agnostic meta reinforcement learning
In this section, we first propose method to solve the stochastic gradient-based MAML Reinforcement Learning problem introduced in (6). Then, we discuss how extend the proposed method the setting that we solve a multi-step MAML problem as introduced in (7). We close the section by discussing the differences between our proposed method and the Meta-RL method proposed in [1] and clarify why these two methods are solving two different problems.
3.1 MAML for stochastic meta-RL
Our goal in this section is to propose an efficient method for solving the stochastic Meta-RL. problem in (6). To do so, we propose a stochastic gradient MAML method for ReinfoReinfrremmmntt Learning (SG-MRL) that aims solving problem (6) by following the update of stochastic gradient descent for the objective function Vi (0). To achieve this goal one need to find an unbiased estimator of the gradient (0) which in some MAMI. settings not trivial (for more details see Section 4. in [14]), but we show that for problem (6) an unbiased estimate of (V) can be efficiently computed.

---

Let us start by pointing out that the gradient of the function (0) defined in (6) is given by
(0) V E4 EDi
J (0 aVJ,(0, Diext) E,ED, (T+aV%) Diest))
VJ (e, Dlest)) + J.( Di Diest))
E
Volog Vologm((T:0)
(10)
TED
with the convention that for T = (50, d0, HH, aH) we define (T(T;)
as
(T: II T. (anlsn:)) h=
(11)
Recall that the expected reward function J(( and its gradient VJ((0) are defined in (2) and (3), respectively, and V (0, Diege) and (0, Diest) are the stochastic estimates of the gradient and Hessian corresponding to (0) that are formally defined in (5) and (9), respectively.
Note that the first term in the definition of (0) in (10), i.e., (T (I+ ( V J.O, D)e is the term that gives the gradient of an MAML problem (see, e.g., [16]), while the second term, i.c., (0 + (4,00 Diest)) ΣTEDi Vo log (T; 0), is specific to the RL setting since the probability distribution p: itself depends on the parameter 0. For more details regarding the derivation VV. for any Ç 2 1, we refer the reader to Appendix C.
We solve the optimization problem in (6) by using gradient ascent step to update the parameter 6, ie.. following the update 0x+1 = 0g + V (0g) at iteration k. However, computing the gradient VV, may not be tractable in many cases due the large number of tasks and the size of the action and state spaces. In our proposed SG-MRL method we therefore replace the gradient V VV1 (0x) with its estimate computed as follows: At iteration k 1, we first choose subset B3 of the tasks (MDPs), where each task is drawn independently from the probability distribution p. The SG-MRL outlined in Algorithm is implemented two levels: (i) inner loop and (ii) outer loop. In the inner loop, for each task Ti with E Bg. we draw a batch of trajectories Di according to gi(:5 to compute the stochastic gradient J. Di as defined in Section 2. This estimate is then used to compute a model 6/+1 corresponding to task by single iteration of stochastic policy gradient,
0 = 0g + J Din).
(12)
For simplicity, we assume that the size of B& is equal to B for all k and the size of dataset Di, is fixed for all tasks and at each iteration, and we denote by D.n.
In the outer loop, we compute the next iterate OR+1 using the iterates (0i+IhieB, that are computed in the inner loop. In particular, follow the update Ox+1
02 (VVV (fx),, where
1 Σ [ aU+av%),(0n J,,0 (i,,) + (.. D() (Ox) = 1/3
(13)
ieB.
+ Jt On + +av Din), Σ TED:
Volog (T:
in which (,,,, Di,,) is policy Hessian estimate defined in (9) and for each task T,, the dataset D a new batch of trajectories that are drawn based on the probability distribution q:(:; 0 Again, for simplicity, we assume that the size of dataset D is fixed for all tasks and at each iteration denoted by D., SG-MRL is summarized in Algorithm 1.
can be verified that if all the gradients and Hessians in SG-MRL update were exact, then the outcome of the update of SG-MRL would be equivalent to the outcome of gradient ascent update for the function Vi, i.e., 0x+1 = 02 + VVV, (0&). Note that by computing the expected value of VVi first with respect to the random set D then with respect to Din. and finally with respect to B,, we obtain that ((VV((0,) = VV, (9w). Therefore, the stochastic gradient (V,(),) is an unbiased estimator of the gradient VV (Ox)..
The SG-MRL method can also be extended and used for solving the multi-step MAMI. problem defined in (7). To do so, at each iteration, we first perform Ç steps of policy stochastic gradient in the
5

---

Algorithm 1: Proposed SG-MRL method for Meta-RL
Input:Initial iterate 0g repeat
Draw batch of i.i.d. tasks B3 SI with size B |B&xl:
for all Ti with E By do
Sample a batch of trajectories Di, w.r.t. qi(:;un;
Set +1 = 0x + a (O,,, Di:):
end for
Sample a batch of trajectories D w.r.t. o(;
Set 0k+1 0g
Additional term SG-MRL
Σ
((.)) (I+aV7),(0E) Din)) JJ; (0k+1; D() + J, (b)+1:
Vo (o TEDi
Σ
IEB
h h +1
until not done
inner loop, and then take one step of stochastic gradient ascent with respect to an unbiased estimator of (O). More details on the implementation of SG-MRL for that case provided in Appendix C.
3.2 Comparing SG-MRL with other model-agnostic meta-RL methods
In this section, we discuss the difference between our SG-MRL method and recent Meta-RL methods.
In particular, focus the MAML method in [1] for solving RL problems. Before discussing the differences between these two methods, let us first recap the update of the MAML method in [1].
The main formulation proposed in [1] which was followed in other works such as [10] is slightly different from the one in this paper as they assume the agent has access to the exact gradient of the new task, and hence, they consider the following MAML problem
max AER
Vi (0) == E.-p [ (0 (()))
(14)
As mentioned, the main difference between (6) and (14) is that the former tries find good initial policy that leads to a good solution after running one step of stochastic gradient ascent, while the latter finds initial policy that produces good policy after running one step of gradient ascent.
Remark 1. Problems in (6) and (14) are both valid formulations for Meta-RI. In pructice, however, is often computationally intractable to evaluate the exact gradient of the expected reward and we often have only access to its stochastic gradient. Hence, it might be more practical to solve (6) instead of (14) it finds initial policy that performs well after running one step of stochastic gradient, unlike (14) that finds a policy that performs well after running one step of gradient update.
In a nutshell, the MAML method proposed in [1] tries to solve the problem in (14) by following the update of stochastic gradient ascent for the objective function Vi (0). To be more precise, note that the gradient of the loss function V, (0) defined in (14) can be expressed as
(15) Note that the expression for the gradient of (0) (15) is different from the expression for the gra- dient of V/1 (0) in (10). In particular, the extra term Je( + J Diess)) EreD; V log n, (T; 8) that appears in (15) is caused by the fact that we use stochastic gradients in the definition of the function (0), while exact gradients are used in the definition of (0).
(0) VeE.up [J (0 (V)) Es-p [[I + a (O)) VJ, (0 + (V))
Considering the expression for the gradient of (1) in (15), a natural approach to approximate (0) is to replace the gradients and Hessians corresponding to the expected reward J,(0) by their stochastic approximations. In other words, one use the approximation (Vi which is defined as the average over (I (, J + (.kk Di..), D() for all i Br, i.e.
(O,( = 1/ Σ (1+aV2J,
1
&J (0) D:)
(16)
B
iEB

---

where -- + a. Din). Here the procedure for computing the sample sets Di, and D. is the same the one in SG-MRL. Once VVi (f&) is computed the new variable 0k+1 can be computed by following the update of stochastic gradient ascent, i... 0x+1 = 2g + 3 V V (0x). The description of the Meta-RL method in [1] and its implementation at two levels (inner and outer) is similar to the one in Algorithm 1, except the highlighted additional term which is not included in MAML update.
Note that the gradient estimate VV, (0x) in (16) is a biased estimate of the exact gradient V defined in (15). This is due to the fact that (.,,0 + aVV., Pin): D:) is a biased estimate of VJ + a V (+)) because of the term (O,, (i.)) inside it. In other words, MAML method proposed by [1] uses biased estimate of the gradient in this case. Note that, in general optimiza- tion analyses, when we have access to biased gradient estimators, even with diminishing or small stepsize, we might only converge to a neighborhood of the optimal solution, where the radius of our convergence depends on the bias. To resolve this issue, one needs to control the bias in the gradient directions and lower the bias time progresses using some debiasing techniques. For instance, the work in [23] studies this problem in detail for debiasing MAML in the supervised learning setting.
On the other hand, our proposed SG-MRL method does not suffer from this issue since computing an unbiased estimator of the gradient for the objective function considered in (6) is relatively simple. In fact, in the following section, we show that SG-MRL is provably convergent and characterize its complexity to find an approximate first-order stationary point of (6) and its generalized version defined in (7).
4 Theoretical results
In this section, we study the convergence properties of the proposed SG-MRL method and characterize its overall complexity for finding policy that satisfies the first-order optimality condition for the objective function (e( defined in (7). To do so, we first formally define the first-order optimality condition that we aim to achieve.
Definition A random vector 8e Wdd is called an c-approximate first-order stationary point for problem (7) if it satisfies ([]VVV c.
We next state the main assumptions that Assumption 1. The reward functions r; are nonnegative and uniformly bounded, i.e., there exists a constant R such that for any task i, state E S,, and action a AA,, we have 0 S r:(als) S R.. Assumption 2. There exist constants G and L such that for any and for any state s E S. action E A., and parameter 0 E Rd. we have Vo log (a|s;; 0)] I G and 1V log [a (s]ss 0) I L.
derive results.
Both assumptions are customary in the policy gradient literature and have been used in other papers to obtain convergence guarantees for policy gradient methods [24, 22, 25].
Assumption 3. There exists a constant p such that for any and for any state s S action a A,, and parameters 0₁, 02 E Ied, we have V log Te [a]s; 02) V log (als; 02)1 S p|l0] 02]1.
This assumption is also customary in the analysis of MAML-type algorithms [14, 16]. In particular, in Appendix we provide more insight into the conditions in Assumptions and by focusing on the special case of softmax policy parametrization
4.1 Convergence of SG-MRL
Next, we study the convergence of our proposed SG-MRL for solving the Model-Agnostic Meta- Reinforcement Learning problem in (7), To do so, we show two important intermediate results. First, we show that the function Ve (0) is smooth. Second, we show the unbiased estimator of the gradient (0) denoted by VV:(e)) has a bounded norm. Building on these two results, we will derive the convergence of SG-MRL. To prove these two intermediate results, we first state the following lemma on the Lipschitz property of the expected reward function J/ and its first and second derivatives for any MDP M.. This lemma not only plays key role in our analysis, but also can be of independent interest in general for analyzing meta-reinnorceement learning algorithms.
Lemma 1. Recall the definitions of (e(T; 8) in (4) and (T; f) in (8) for trajectory T E (S,x and policy parameter 0 Rd. If Asssmmttions 1-3 hold, then for any MDP M, we have:
x A.) #+1
7

---

i) For any and 8, we have lIgi(T;)) 0)
<
GG
GR As consequence, (1-))²
[()Il (. 1N D')ll S nG for any 9 and any batch of trajectories D' Further, this implies that J&. smooth with parameter nG.
ii For any T and 8, we have [|us(T;0)]l I πH
((H+1)G24 As a consequence, (1-7)²
J. 1. [((,, D'Il S лH for any 0 and any batch of trajectories D', Further, this im- plies that J.()) is smooth with parameter TH-
iii For any batch of trajectories D', JJ D') is smooth with parameter Up = (2 12[++])G++pM
(1-5))
By exploiting the results in Lemma 1, we can prove the promised results the Lipschitz property of (0) well boundedness of its unbiased estimator (0). In the following proposition, due to space limitation and for the the ease of notation we only state the result for the case that Ç 1; however, the general version of these results along with their proofs are available in Appendix F. Proposition Consider the objective function V1 defined in (6) for the case that a E (0, 1/nal where πn is given in Lemma 1. Suppose that the conditions in Assumptions 1-3 are satisfied. Then, (0) is smooth with parameter
Lv == amp + 4ли S..,, +] + G., (H + 1))
(17)
where GGG and 7o are defined in Lemma 1.
ii For any choices of BR. {Di)s and (Din the norm of stochastic gradient VV (Ox) defined in (13) at iteration l is bounded above byl (V&())) s (Gv==GR[[1-))== Din (H + 1)].
The smoothness parameter for the RL problem has been previously characterized (as example (22]), but, to the best of our knowledge, this is the first result on the smoothness parameter of the meta-RL function. Proving Proposition is the main challenge in our analysis, since it establishes that our formulation satisfies the relevant assumptions needed for our main result in the next theorem.
Now, we present our main result on the convergence of SG-MRL to first-order stationary point for the Meta-reinforcement learning problem in defined (7). We state our main result for the special case of Ç = 1, but the general statement of the theorem along with its proof can be found in Appendix G. Theorem 1. Consider V1 defined in (6) for the case that E (0,1/mm] where aH is defined in Lemma 1. Suppose Assumptions 1-3 are satisfied, and recall the definitions of Ly and Gy from Proposition 1. Consider running SG-MRL (Algorithm 1) with B E (0,1/Lv) Then, for any solution 02 such that ([[ < LvB + +e,, after running for BD.
1>< E 0, SG-MRL at most ((1) R min 22
BDD Lv
iterations.
Next we characterize the complexity of SG-MRL for finding an e-first-order stationary point solution. Corollary 1. Suppose the hypotheses of Theorem hold. Then, for any > 0, SG-MRL achieves e-firstooddrr stationarity by setting: (i) BD, 2 / and 3 = L/ requiring O () iterations and computing O(E 2) stochastic gradients per iteration; (ii) 3 O(e and BD, O()) which requires O 4) iterations and (11) stochastic gradient evaluations per iteration.
The conditions in Corollary I identify two settings under which SG-MRL finds an e-FOSP after a finite number of iterations, abd both settings overall require O( -1) stochastic gradient evaluations.
Remark 2. While we mainly focused on the case ( = we provide the general statement of the results for any in the Appendix. Note that the downside of increasing Ç is that the smoothness parameter grows exponentially with respect to Ç (see Theorem 3), which means that we need to take smaller learning rate that leads slower convergence rate. However, on the positive side, by increasing Ç we train a model that better adapts to a new task.
Numerical experiments
In this section, we empirically validate the proposed SG-MRL algorithm in larger-scale environments standard in modern reinforcement learning applications. The code available o
The code is available attps://gite httpgghu.../////issseeeeeeeeeeee www.oo/o/iiiangeeeeeeeeeeeeeeeee http://rrittaangooori///////
8

---

Table 1: Mean meta-test reward (negative square distance to goal location) of SG-MRL, MAML, and E-MAML after adaptation step.
Algorithm Meta-Test Reward
SG-MRL -16.901 = 0.699
MAML -17.767 + 0.106
E-MAML
-17.803 + 0.115
Table 2: The mean meta-test reward for SG-MRL and MAMI. on additional environments when trained and adapted with 1, 2, and 3 inner updates over 4 random seeds.
environment
SG-MRI.
MAMI reward
Half-Cheetah Random Direction, step
580.143 38.22
465.624 54.07
Half-Cheetah Random Direction, step
580.203 + 33.63
441.247 58.34
Half-Cheetah Random Direction, step
504.747 45.07
477.086 = 66.71
Half-Cheetah Random Velocity, 1 step
-91.73 0.34
-92.92 0.70
Half-Cheetah Random Velocity, 2 step
-52.64 6.86
-56.71 6.73
Half-Cheetah Random Velocity, 3 step
-33.39 =: 0.67
-32.48 + 0.50
Swimmer Random Velocity, I step
118.77 9.99
104.53 + 24.18
Swimmer Random Velocity, 2 step
134.57 1.67
108.47 + 23.36
Swimmer Random Velocity, 3 step
110.91 + 12.56
90.60 + 14.99
We conduct two experiments: 2D- D-Diivigatiin problem, and a more challenging locomotion problem simulated with the MuJoCo library [26]. For both experiments, we use neural network policy with a standard feed-forward neural network and optimize it with vanilla policy gradient [27]. Further implementation details are outlined in Appendix H.
All experiments were conducted in MIT's Supercloud [28]. Similar to FO-MAML proposed in [1], we use first order implementation of SG-MRL. It is also worth noting that SG-MRL is straightforward to implement as a modification to MAML and requires no additional hyperparameter tuning. Also, SG- MRL does not reduce the scalability of MAML. In particular, across experiments, the clock time of SG-MRL against MAML. and SG-MRL is consistently at most 1.05 times slower over the course of training. Next, we demonstrate the practicality of SG-MRL in modern deep reinforcement learning problems.
benchmarked
2D nD-navigation. We consider the problem of point-mass agent navigating from the origin to random goal lo- cation within unit-size square centered at the origin (l-0.5,0.5 0.5] x [-0.5, 0.5)). We consider the negative squared distance to the goal location servations consist of the position of the agent within the unit-size square. The action space comprises of all veloe- ities with components clipped in the interval 0-0.1, 0.1]. An example of trajectory is illustrated in Figure In Table 1, we compare the performance of SG-MRL against MAML [1] and E-MAML. [29]. We make comparison with E-MAML since it has a similar spirit to our proposed SG-MRL method, but unlike the proposed algorithm, E- MAML is derived from heuristic arguments.
start
MAML
goal
reward. Ob-
0.2
0.1
-0.3
Figure 1: Trajectories generated by poli- cies trained with SG-MRL and MAML for the 2D-navigation problem.
Locomotion: MuJoCo environments. In addition to the 2D-navigation example, we provide a benchmark on a more challenging set of tasks MuJoCo's locomotion environments. We benchmark our algorithm against MAML on three different tasks and report the results in Table 2. The tasks involve learning to move in goal direction (forward/backward), reach target velocity. We describe each task in more detail in Appendix H.
6 Conclusion and future work
We studied MAMI. for RL problems, considering performing a few steps of stochastic policy gradient at test time. Given this formulation, introduced SG-MRL, and discussed how it differs from the

---

original MAML algorithm in [1]. Further, we characterized the convergence of SG-MRL method in terms of gradient norm and under set of assumptions on the policy and reward functions. Our results show that, for any c, SG-MRL achieves e-first-order stationarity, given that either the learning rate is small enough or the multiplication of task and outer loop batch sizes is sufficiently large.
A shortcoming of our analysis is the requirement on the boundedness of gradient norm (Assumption 2). natural extension of our work would be extending the theoretical results the setting that gradient norm is possibly unbounded. Moreover, our results are limited to achieving first-order optimality, while one can exploit techniques for escaping from saddle points obtain second-order stationarity.
7 Acknowledgment
Alireza Fallah acknowledges support from the Apple Scholars in AI/ML PhD fellowship and the Math Works Engineering Fellowship. This research is sponsored by the United States Air Force Research Laboratory and the United States Air Force Artificial Intelligence Accelerator and was accomplished under Cooperative Agreement Number contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the United States Air Force or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. This research of Aryan Mokhtari is supported in part by NSF Grant 2007668, ARO Grant W91 10226, the Machine Learning Laboratory at UT Austin, and the NSF Al Institute for Foundations of Machine Learning.
1--8750-192-1100 9-00- The views and conclusions
References
[1] C. Finn, P. Abbeel, and S. Levine, "Model-agnostic meta-learning for fast adaptation of deep networks," in Proceedings of the 34th International Conference on Machine Learning, (Sydney, Australia), 06-11 Aug 2017.
[2] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel, "R12: Fast reinforce- ment learning via slow reinforcement learning," arXiv preprint drXiv:1611.0277,, 2016.
[3] J. X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z.. Leibo, R. Munos, C. Blundell, D. Ku- maran, and M. Botvinick, "Learning to reinforcement learn," arXiv preprint arXiv: 161 I.05663, 2016.
[4] N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel, "A simple neural attentive meta-leamer," 2017.
arXiv preprint arXiv:1707.0000
[5] J. Rothfuss, D. Lee, I. Clavera, T. Asfour, and P. Abbeel, "Promp: Proximal meta-policy search," arXiv preprint 11110:481,.06784, 2018
[6] J. X. Wang. Z. Kurth-Nelson, D. Kumaran, D. Tirumala, H. Soyer, J. Z. Leibo, D. Hassabis, and M. Botvinick, "Prefrontal cortex as a mete-reinforcement learning system," Nature neuroscience, vol. 21, no. 6. pp. 860-868, 2018.
[7] A. Nagabandi, I. Clavera, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn, "Learning to adapt in dynamic, real-world environments through meta-reinforremmnt learning," arXiv preprint 1803 2018
[8] K. Rakelly, A. Zhou, D. Quillen, C. Finn, and S. Levine, "Efficient off-policy meta- reinforcement learning via probabilistic context variables." arXiv preprint arXiv: 1eXXv:1903.0825: 2019.
[9] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine, "Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning," " arXiv preprint arXiv: 1910. 10897, 2019.
[10] H. Liu, R. Socher, and C. Xiong, "Taming maml: Efficient unbiased meta-reinforcemmet learning," in International Conference on Machine Learning, pp. 4061-4071, 2019.
10

---

[11] R. Mendonca, A. Gupta, R. Kralev, P. Abbeel, S. Levine, and C. Finn, "Guided meta-policy search," in Advances in Neural Information Processing Systems, pp. 9653-9664, 2019.
[12] A. Gupta, R. Mendonca, Y. Liu, P. Abbeel, and S. Levine, "Meta-reinforcement learning of structured exploration strategies," in Advances in Neural Information Processing Systems, pp. 5302-5311, 1, 2018.
[13] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine, "Meta-learingg with implicit gradients." in Advances in Neural Information Processing Systems, pp. 113-124, 2019.
[14] A. Fallah, A. Mokhtari, and A. Ozdaglar, "On the convergence theory of gradient- gradient-based model- agnostic meta-learning algorithms," in International Conference on Artificial Intelligence and Statistics, pp. 1082-1092, PMLR, 2020.
[15] K. Ji, J. Yang, and Y. Liang, "Multi-step model-agnostic meta-learning: Convergence and improved algorithms," arXiv preprint arXiv:2002.07830 2020.
[16] C. Finn, A. Rajeswaran, S. Kakade, and S. Levine, "Online meta-learming," in Proceedings of the 36th International Conference on Machine Learning, vol. 97 of Proceedings of Machine Learning Research, (Long Beach, California, USA). pp. 1920-1930, PMLR, 09-15 Jun 2019.
[17] M. Khodak, M.-F. Balcan, and A. Talwalkar, "Provable guarantees for gradient-based meta- learning," in Proceedings of the 36th International Conference on Machine Learning, vol. 97 of Proceedings of Machine Learning Research, (Long Beach, California, USA), PMLR, 09-15 Jun 2019.
[18] M. Khodak, M.-F. F. Balcan, and A. Talwalkar, "Adaptive gradient- maddentt-basdd meta- meta-learning methods," in Advances in Neural Information Processing Systems, pp. 5915-5926 2019.
[19] Foerster, G. Farquhar, M. Al-Shedivat, T. Rocktäschel, E. P. Xing, and S. Whiteson, "Dice: The infinitely differentiable monte-carlo estimator," arXiv preprint arXiv: 1802.05098, 2018.
[20] R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
[21] Peters and Schaal, "Reinforcement learning of motor skills with policy gradients," " Neural networks, vol. 21, no. 4, pp. 682 697, 2008
[22] Z. Shen, A. Ribeiro, H. Hassani, H. Qian, and C. Mi, "Hessian aided policy gradient," in International Conference on Machine Learning, pp. 5729-5738, 2019.
[23] Y. Hu, S. Zhang, X. Chen, and N. He, "Biased stochastic first-order methods for conditional stochastic optimization and applications in meta learning," " Advances in Neural Information Processing Systems, vol. 33, 2020.
[24] M. Papini, D. Binaghi, Canonaco, M. Pirotta, and M. Restelli, "Stochastic variance-reduced policy gradient," in Proceedings of the 35th International Conference on Machine Learning, vol. 80 of Proceedings of Machine Learning Research, (StockholmsmAAssee Stockholm Sweden), pp. 4026-4035, PMLR, 10-15 Jul 2018
[25] A. Agarwal, M. Kakade, J. D. Lee, and G. Mahajan, "Optimality and approximation with policy gradient methods in markov decision processes," arXiv preprint arXiv: 00261, 2019.
(26) E. Todorov, T. Erez, and Y. Tassa, "Mujoco: A physics engine for model-based control," in 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
[27] R. J. Williams, "Simple statistical suadient-following algorithms for connectionist reinforccmmmn learning," Machine learning. vol. 8, no. 3-4, pp. 229-2222222222 229-256, 1992.
[28] A. Reuther, Kepner, C. Byun, S. Samsi, W. Arcand, D. Bestor, B. Bergeron, Gadepally, M. Houle, M. Hubbell, et al., "Interactive supercomputing on 40,000 cores for machine learning and data analysis." " in 20/8 IEFE High Performance extreme Computing Conference (HPEC), pp. 1 1-6, IEEE, 2018.

---

[29] B. Stadie, G. Yang, R. Houthooft, P. Chen, Y. Duan, Y. Wu, P. Abbeel, and I. Sutskever, "The importance sampling inmeta-reinforcemmnn learning"" in Advances in Neural Information Processing Systems (S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, eds.), vol. 31, pp. 9280-9290, Curran Associates, Inc., 2018
[30] Y. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course, vol. 87. Springer, 2004.
12


Reviews:
[{"review_id": "qCBaRXVecCJ", "reviewer": "Reviewer_yqwx", "summary": "This paper studies the Model-Agnostic Meta-Reinforcement-Learning (MAMRL) from a theoretical perspective. In particular, it proposes a modified MAMRL objective along with its unbiased gradient estimator, establishing a new Meta-RL algorithm called SR-MRL. It theoreticallly proves the convergence of SR-MRL and experimentally shows its comparable performance to the original MAML algorithm.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, alghouth the techniques used in this paper do not seem to be very novel, I consider its results as a significant progress of Meta-RL because it is the first provably convergent Meta-RL algorithm, which meanwhile also achieves comparable performance to the MAML algorithm. The whole paper is also very well-written with clear logic flow.\n\nOne concern I have is about the assumptions this paper use. It seems that the convergence does not rely on any assumptions about underlying MDPs at all, which looks a little bit wired to me. Did I miss anything?\n\nQuestions:\n- Is the exponential dependency of $V_{\\zeta}$'s smoothness parameter in $\\zeta$ fundamental? Is there any intuitive explanation on why its dependency is exponential?\n- For the Remark 2, is there any benefits of increasing $\\zeta$ from a theoretical perspective?\n- Does the convergence result rely on any assumptions about the transition kernel of $\\mathcal{M}_i$ or it can be completely arbitrary?\n\nMinor suggestions on writing:\n- At line 301 on page 8, *Theorem 1* -> *Proposition 1*\n\n---\n\nScore increased after rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations of their work in Section 6 by comments on their assumptions. The potential negative social impact is not addressed, which will generally be all potential malicious usage of any reinforcement learning algorithms.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "IJsTZ-zu8KD", "reviewer": "Reviewer_43di", "summary": "This paper proposed a new debiased model-agnostic meta-reinforcement learning algorithm. The main improvement is that the paper considers a more accurate setting where one does not have accurate estimate on the gradient for each task, which is assumed by previous works. One has to estimate the gradient from samples in each task, and the samples themselves depend on the global parameter $\\theta$, there will be an extra term for the global gradient.  The authors hence write the exact gradient and show the convergence. Numerical results show good performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes a solid improvement over the previous methods by considering a more accurate setting, i.e. the gradient of each task is estimated from batch data. By considering the more accurate setting, they are also able to achieve a better numerical performance.\n\nThe paper is well-written and easy to follow.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Despite the improvement shown in the numerical experiments, it would be sound if the authors can theoretically show the advantage of the proposed method versus the methods in the literature. To be more specific, how much does assuming the knowledge of the second-order information of the expected reward hurt the actual performance, when the information is not available. Will it cause bias or increased variance?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "6_swLm-qcuw", "reviewer": "Reviewer_r6Av", "summary": "This paper aims to tackle the non-diminishing bias in MAMRL (MAML for RL) due to the stochastic policy gradient update for subtasks. Such bias hinders the first-order optimality when solving MAML with gradient descent. The authors compute the exact gradient of the MAMRL objective and identify the additional term that arises in RL. The authors then estimate such a gradient with an unbiased approximation based on samples and propose SG-MRL that utilizes such approximation to solve MAMRL. The authors show that SG-MRL converges to first-order stationary points under the standard assumptions and further conduct experiments on SG-MRL against baselines in the MuJoCo environment.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is a fairly solid work on MAML, with both theoretical insights and empirical improvements over baselines. The algorithm is simple and well-motivated, and the theory is reasonable and well-explained. Experiments demonstrate the improvement of the proposed SG-MRL against baselines, which also partially corroborates the theory. \n\nIt would be better if the authors could provide a more comprehensive comparison of SG-MRL against MAML on more tasks. Still, this is a minor issue. The main point of this work is on the theoretical side, for which the experiments are sufficient to me.\n\nSome inquiries:\n\n1. Is the bias correction term only adaptable to the update proposed in (6)? As the authors pointed out, the MAML gradient in (16) is biased and would also require a similar term to the bias correction term of SG-MRL. Does the difference in the objective (between (6) and (14)) affect the formulation of the bias correction term?\n\n2. In lines 235-244, the authors mentioned that having a biased gradient requires careful selection of stepsize for MAML, while SG-MRL does not suffer from such stepsize selection. Could the authors provide more details regarding this point?\n\n3. The theory involves the inner learning rate $\\alpha$ as a smoothness parameter of the objective and requires $\\alpha$ to be small for convergence. Can the theory say something about the effect of $\\alpha$ on convergence and the overall performance of SG-MRL? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See inquiries in the Main review. I do not have concerns regarding the societal impact of this work.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "2ncUeSgTxfq", "reviewer": "Reviewer_WxkP", "summary": "This paper deals with the problem of few-shot meta learning, i.e. where one explicitly optimizes for the performance of doing one (or more) steps of SGD on a new task. This was intially introduced through the MAML method. However, the authors argue that MAML only discusses gradient descent while in practise, approximations lead to doing stochastic gradient desecent instead, thus making the original formulation of the problem biased. To that end, this paper proposes an SGD based meta learning objective, which ensures simpler convergence analysis as well as better performance in two benchmark problems from the RL setting. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The motivation of the work is solid in my opinion. The main question the authors ask is indeed important and the authors do a good job at highlighting it. \n\nQuality: The overall quality of the work is quite high. The explanations of the SG-MRL theory are well detailed and intuitive.\n\nClarity: The paper reads quite clearly overall. The flow of content is pretty nice as well.\n\nSignificance: I believe the paper contributes in both fronts, theoretically and empirically. In the theory part, it sheds important insight on the issues with prior work and how those can be fixed, in the case when we do not have access to exact gradients. In the practical part, the results are quite strong and show that a simple fix can provide better gains. It would be even better in my opinion if there were some ablations on how much the gradient is biased by (across training) and how/if it matches with the reward curves. \n\nQuestions/Clarifications:\n\n- Going through the MAML paper, it seems like the authors proposed an objective that can be optimized by auto-diff end-to-end. Therefore, they did not use the inner/outer loops explicitly to explain the application of gradients in their work. Is it accurate to say that SG-MRL can also be used with auto-diff directly and that the inner/outer loop computations happen the same way as described in the paper for both SG-MRL and MAML. Or is it that because MAML induces a biased gradient, it can work with auto-diff but SG-MRL cannot, since it requires computing independent averages at both the inner and outer loops?\n\n- The empirical results look really convincing, in that computing an unbiased estimate of the stochastic gradient can have considerable difference in performance. However, the overall story of the paper leans heavily towards the theoretical contributions. Is there any specific reason why? Is the implementation more complicated than MAML in any sense? ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Suggestions: My sole primary suggestion would be to test this method in the supervised learning setting as well, just as in the MAML paper, so as to really drive the point home that correcting for the bias can have considerable empirical gains. This will again highlight that the paper does not only contribute theoretically. \n\nI do not think there any explicit negative societal impacts of this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I do not think there any explicit ethical concerns raised by this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hrs"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviews find that this paper is a solid contribution making a reasonable contribution to the empirical as well as the theoretical state-of-the-art of MAML, so that I recommend to accept it. Please take into account the points raised in the reviews when preparing the final version.
Decision: Accept (Poster)