Unifying Gradient Estimators for Meta-Reinforcement Learning  via Off-Policy Evaluation
Authors: Yunhao Tang
Tadashi Kozuno
Mark Rowland
Remi Munos
Michal Valko
Venue: NeurIPS 2021 Poster (2021)
Keywords: meta learning
reinforcement learning
off-policy learning
off-policy evaluation

Abstract: Model-agnostic meta-reinforcement learning requires estimating the Hessian matrix of value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. In this work, we provide a unifying framework for estimating higher-order derivatives of value functions, based on off-policy evaluation. Our framework interprets a number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door to a new family of estimates, which can be easily implemented with auto-differentiation libraries, and lead to performance gains in practice.

Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
Yunhao Tang*
Tadashi Kozuno University of Alberta
Columbia University yt25410columbia. edu
tadashi koozno@ggailll.......o....... com
Mark Rowland
Rémi Munos
Michal Valko
DeepMind London markrrovlaandddeeminnddc com
DeepMind Paris
DeepMind Paris
munos@deepmind. com
valkoo@deepmind.commmm com
Abstract
Model-agnostic meta-reinforcementt learning requires estimating the Hessian matrix value functions. This is challenging from an implementation perspective, as repeatedly differentiating policy gradient estimates may lead to biased Hessian estimates. this work, we provide a unifying framework for estimating higher- order derivatives of value functions, based on off-policy evaluation. Our framework interprets number of prior approaches as special cases and elucidates the bias and variance trade-off of Hessian estimates. This framework also opens the door new family of estimates, which can easily implemented with auto-differrntiation libraries, and lead performance gains in practice. We open source the code reproduce our results
Introduction
Recent years have witnessed the success of reinforcement learning (RL) challenging domains such as game playing [1]. board games [2], and robotics control [3]. However, despite such breakthroughs, state-of-the-art RI algorithms are still plagued by sample inefficiency, as training agents requires orders of magnitude more samples than humans would experience to reach similar levels of perfor- mance [1, 2]. One hypothesis on the source of such inefficiencies is that standard RL algorithms are not good at leveraging prior knowledge, which implies that whenever presented with new task, the algorithms must learn from scratch. On the contrary, humans are much better at transferring prior skills to new scenarios, an innate ability arguably obtained through evolution over thousand of years.
Meta-reiinooreement learning (meta-RI.) formalizes the learning and transfer of prior knowledge in RL [4]. The high-level idea is to have an RI. agent that interacts with distribution of environments meta-training time. The objective is that at meta-testing time, when the agent interacts with previously unseen environments, it can learn much faster than meta-training time. Here, faster learning is measured by the number of new samples needed to achieve good level of performance. If an agent can achieve good performance at meta-testing time, it embodies the ability to transfer knowledge from prior experiences at meta-training time. meta-RL. RL algorithms can be constructed in many ways, such those based recurrent memory [5, 6], gradient based adaptations [4], learning loss functions [7-9], probabilistic inference of context variables [10-12], online adaptation of hyper-paramettrss parameters within single lifetime [13, 14] and on. Some of these formulations have mathematical connections; see e.g. [15] for an in-depth discussion.
We focus gradient-based based adaptations [4], where the agent carries out policy gradient updates [16] at both meta-training and meta-testing time. Conceptually, since meta-RL seeks to optimize the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Table I: Interpretation of prior work on higher-order derivative estimations as special instances of differentiating off-policy evaluation estimates. Given any off-policy evaluation estimate V from the second row, we recover higher-order derivative estimates in prior work in the first row, by differentiating through the estimate V Pine.
Off-policy evaluation estimates
STEP-WISE IS [26]
DOUBLY- ROBUST [27]
TAYPO-1 [28, 29]
TAYPO-2 [29]
Loaded DiCE [22, 23]
Second-order (this work)
Prior work
DiCE [21]
LVC [24]
way in which the agent adapts itself in face of new environments. it needs to differentiate through the policy gradient update itself. This effectively reduces the meta-RI. problem into estimations of Hessian matrices of value functions.
Challenges for computing Hessian matrix of value functions. To calculate Hessian matrices (or unbiased estimates thereof) for supervised learning objectives, suffices to differentiate through gradient estimates, which can be easily implemented with auto-differentiation packages [17-19]. However, is not the case for value functions in RI.. Intuitively, this is because value functions are defined via expectations with respect to distributions that themselves depend on the policy parameters of interest, whereas in supervised learning the expectations are defined with respect to a fixed data distribution. As result, implementations that do not take this into account may lead to estimates [4] whose bias is not properly characterized and might have a negative impact on downstream applications [20].
Motivated by this observation, a number of prior works suggest implementation alternatives that lead to unbiased Hessian estimates [21], with potential variance reduction [22, 23, 20]: or biased estimates with small variance [24]. However, different algorithms in this space are motivated and derived in seemingly unrelated ways: for example, [21-23] derive code-level implementations within the general context of stochastic computation graphs [25]. On the other hand, [24] derives the estimates by explicitly analyzing certain terms with potentially high variance, which naturally produces bias in the final estimate. Due to apparently distinct ways of deriving estimates, it is not immediately clear how all such methods are related, and whether there could be other alternatives.
Central contribution. We present a unified framework for estimating higher-order derivatives of value functions, based on the concept of off-policy evaluation. The main insights are summarized in Table 1, where most aforementioned prior work can be interpreted as special cases of our framework. Our framework has few advantages: (1) it conceptually unifies a few seemingly unrelated prior methods; (2) it elucidates the bias and variance trade-off of the estimates; (3) naturally produces new methods based on Taylor expansions of value functions [29].
After a brief background introduction on meta-RI. Section 2, we will discuss the above aspects in detail in Section 3. From an implementation perspective, will show in Section 1 that both the general framework and the new method can be conveniently implemented in auto-differentiation libraries [30, 19], making it amenable in a practical setup. Finally in Section 5, we validate important claims based on this framework with experimental insights.
2 Background
We begin with the notation and background on RL and meta-RL
2.1 Task-based reinforcement learning
Consider a Markov decision process (MDP) with state space X and action space A. At time t 2 0, the agent takes action E A in state I; X, receives reward and transitions to a next state Z++1 p(-|z,, at). Without loss of generality, we assume a single starting state zg-- Here, we assume the reward r = r(I,, C1. g) to be deterministic function of state-action pair (wx, at) and the task variable g E g. The task variable g ~ Pg is resampled for every episode. For example, Ia E X' is the sensory space of running robot, a E A is the control, y is the episodic target direction in which to run and r(r,, (t,g) g) is the speed in direction g. A policy X + P(A) specifies a distribution over actions each state. For convenience, we define

---

the value function (z 9) = E, شニ= 2',( at. 2 =x] and Q-function (z) = En 1 7'r(",, (t 30 = I, 4o = a). Without loss of generality, we assume the policy to be smoothly parameterized as T with parameter 0 E DRD We further assume that the MDPs terminate within a finite horizon of H < 0o under any policy.
2.2 Gradient-based meta-reinforcemmnn learning
The motivation of meta-RL. is identify policy wo such that given task g. after updating the parameter from 0 to 0' with a parameter update computed under rewards r(x, a, g). the resulting policy Te' performs well. Formally, define (((0.) E _2 parameter update to 0, for example, one policy gradient ascent step. Model-agnostic meta-learning (MAML) [4] formulates meta-RL as optimizing the value function the updated policy Hyrpp [V
To (x, 9)] where 0' = 0 U(0,9) g) is the updated policy. The optimization is with respect to the initial policy parameter A. The aim is to find A such that it entails fast learning (or adaptation) to the environment, through the inner loop update operator U(0, g). that leads to nigh-performing updated policy 0'. Consider the following problem.
max F(O)= = Enrpg [VTV (zo: g)l, 0' = 0 + U)) 0
(1)
We can decompose the meta-gradient into two terms based on the chain rule,
V VoF = E Eg-po [oF g)] = Egupo [V V V (Io. g)l
(2)
where Vo6' = I
DD is matrix, and Ve V (zo. g) is the vanilla policy gradient evaluated at the updated parameter 0' [31]. A straightforward way to optimizing Eqn (1) is to carry out the outer loop update -
0 + a aVF((0) with learning rate a >
Policy gradient update. Following MAML [4], we focus policy gradient update where (0.9)) = VoV (zo. for a fixed step size 17 > 0. The matrix Vol = V/ (zo. g) is the Hessian of the value function with respect to policy parameters; henceforth, we define Ho ( == ( g) E DDD
Estimating meta-gradients. Practical algorithms construct stochastic estimates of the meta- gradients using finite number of samples. In Eqn (2), we decompose the meta-gradients into the multiplication of a Hessian matrix with a policy gradient vector. Given a fixed task variable g. a common practice of prior work is to construct the gradient estimate the product of the Hessian estimate and the policy gradient estimate F(0, g) = (T + Ho V V (æo, (Po,g)); see, e.g., [4, 32, 33, 24, 21-23, 20]. See Algorithm 1 for the full pseudocode for estimating meta-gradients by sampling multiple tasks. Since there is large literature on constructing accurate estimates policy gradient (e.g.. actor-critic algorithms use baselines for variance reduction [34]), the main challenge consists in estimating the Hessian matrix accurately.
Bias of common plug-in in estimators. Common practices in meta-RL algorithms rely on the premise that if both Hessian and policy gradient estimates are unbiased, then the meta-gradient estimate is unbiased too.
[oo(zo o)} = He (z. 9). E V V VT = V Th' (xo. g) + (oF(0,9) = (0)
Unfortunately, this is not true. This is because the two estimates are in general correlated when the sample size is finite, leading to the bias of the overall estimate. We provide further discussions in Appendix A. For the rest of the paper, we follow practices of prior work and focus on the properties of Hessian estimates, leaving more proper treatment of this bias to future work.
3 Deriving Hessian estimates with off-policy evaluation
Since the meta-gradient estimates are computed by averaging over task variables g ~ Pg, in the following, we focus on Hessian estimates at a single state and task variable Ho( g) with a fixed In this section, we also drop the dependency of the value function on g. such that, e.g.. V (xo) = VT (zv, g) and Q(I, a, g) Qzx.a).
3.1 Off-polics evaluation: maintaining higher-order dependencies on parameters
We assume access to data (Iz, ot, r generated under a behavior policy Off-policy evaluation [26] consists in building estimators ** V** (x,g) using the behavior data such that (x) V (=) (x)
3

---

Algorithm Pseudocode for computing meta-gradients for the MAML objective
for i=1,2.... do
Sample task variable Ji ~ Pg-
Sample B trajectories under policy To: Compute B-trajectory policy gradient estimate
VoVT (z). update parameter 0' 0 VoV (z).g.).
Compute B-trajectory Hessian estimate Ho (To: 9:) and an unbiased policy gradient estimate at i.c. V. (2o.
Compute the i-th meta-gradient estimate VF gi) (T (7 Vo (zo,
end for
Output averaged meta- gratdggradiet estimate 1, - VoF(0.9:).
for range of target policies TT. Note that the estimate (I) is random variable depending (I4, at, I' == also function of 9. The approximation 1/*V (x) R V** (x) implies that 1TT (x) is indicative of how the value function V (z) depends on 8, and hence maintains the higher-order dependencies on 6. Throughout, we assume [T(a > 0. ( (x) > for all (æ,a) E X A.
Example: step-wise importance sampling (IS) estimate. As concrete example, consider the unbiased step-wise IS estimate V750 (mr) = Ei 7' (T.sun?) ra where p To (a.)ar))/// u((, la..). Since the value function V (x) is in general highly non-linear function (see discussions in [35]) we see that Vi (zo) retains such dependencies via the sum of product of IS ratios.
3.2 Warming up: deriving unbiased estimates with variance reduction
We start with general result based on the intuition above: given an estimate VTH0 (x) to Vaa (x),, we can directly use the morder derivative V ( (xx) E RDx as estimate to V T (z). We introduce two assumptions:
(A.1) n. (x)) is th -order differentiable w.r.t. 0 almost surely. (A.2) .M for some constant M for the order of interest. These assumptions are fairly mild; see further details in Appendix C. The following result applies to general unbiased off-policy evaluation estimates.
Vigi (xx) oo
Proposition 3.1. Assume (A.1) and (A.2) are satisfied. Further assume we have an estimator VTTe which is unbiased (E,, VT (x) = VT (z)) for all 0' E (() where (()) is some open set that contains 0. Under some additional mild conditions, the -oder derivative of the estimate Vi (zo) are unbiased estimates to the -order derivative of the value function E,, V Vm (x)) = V= (x) for m 2 1.
Doubly-robust estimates. As special case, describe the doubly-robust (DR) off off-policy eval- uation estimator [36, 27, 37]. Assume we have access to state-action dependent critic Q(x, a, g), and we use the notation Qxx (x)) = Ea ((x,a))(aaa)) The DR estimate is defined recursively as follows,
The DR (x.)
= = Q(x,, (x)))) + p (re + 7 DiT (5141) (((e,))
(3)
The DR estimate unbiased for all rTy and subsumes the step-wise IS estimate special case when Q(I, a) = O. If the critic is properly chosen, e.g., Q(z, a) ~ QT* a), it can lead to significant variance reduction compared to Vi (xx). By directly differentiating the estimate V DR (æ)) derive estimators for higher-order derivatives of the value function; the result for the gradient in Proposition 3.2 was shown in [38].
(x)
Proposition 3.2. Define n, To(a) (22) and let 5, = + Vi (It-1) Q(2s, 0z) ) be the sampled temporal difference error at time t. Note that Vo log T. R and v log ne DDD The estimates

---

of higher-order derivatives can be deduced recursively, and in particular for m = 1,2,
V DR
(Ix) = V Q ne (z:)) + ग., + Tp. (I(+1):
(4)
DR (I+) = p. V log म. - Va log n. V log क| + Vo Vo (Iz) Va log
TP, Vo log T Vo DR ( (Z no (x)))) + Ypt V DR (I(+1).
(5)
Bias and variance of Hessian estimates. Proposition 3.1 implies that Vo m. (mr) and V (250) are both unbiased. To analyze the variance, we start with m = 1: note that when on-policy / Te DR (ro) recovers form of gradient estimates similar actor-critic policy gradient with [16], Vo V action-depended baselines [39-41]; when Q(r, a) is only state dependent, Vo DR (x) recovers the common policy gradient estimate with detendeennddnt baselines [34]. As such, the estimates computed with potential variance reduction due to the critic. Previously, [38] started with the DR estimate and derived we expect a similar effect variance reduction as shown in experiments.
general result for the on-policy first-order case. For the Hessian estimate,
Recovering prior work on estimates to higher-order derivatives. When applied to meta-RL, DiCE [21] and its follow-up variants [22, 42] can seen as special cases V DR (ar) with different choices of the critic Q when evaluated on-policy μ тT. See Table I for the correspondence between off-policy evaluation. will discuss detailed pseudocode in Section 4.1. See also Appendix F for more details.
prior work and their equivalent formulations under the framework
3.3 Trading-off bias and variance with Taylor expansions
Starting from unbiased off-policy evaluation estimates 77 (x), we can directly construct unbiased estimates to higher-order derivatives by differentiating the original estimate to obtain V (x). However, unbiased estimates can have large variance. Though is possible to reduce variance through the critic, as we will show experimentally. this is not enough to counter the high variance due to products of ratios. This leads us to consider trading off bias with variance [43].
Since we postulate that the products of IS ratios lead to high variance, we might seek to control the number of IS ratios in the estimate. We briefly introduce Taylor expansion policy optimization (TayPO) [29], a natural framework to control for the number iS ratios the value estimate.
Taylor expansions of value functions. Consider the value function W (zo) as a function of mi. Using the idea of Taylor expansions, we can express * (xo) as a sum of polynomials of We start by defining the -rder increment as U (zo) Vi(o)) which does not contain any IS ratio (zeroth order); and for K 2 1,
20
20
UK (2o) = E,
Σ 12=01=-
Σ
g th (pL, 1)) (xtt anx)
-1))
(6)
1x=tx-1t1
(70))
Intuitively, the orddr increment only contains product of K IS ratios. Equation (6) also yields natural sample-based estimate U (xo), which will discuss later. The -oh-rrder Taylor expansion defined as the partial sum of increments V (xx) == Σ UZ* (zo). Since V (æo) consists of products of up to K ratios, is effectively the thth-orde Taylor expansion of the value function. The properties summarized follows.
Proposition 3.3. (Adapted from Theorem of (29].) Define || μ|l max, E. (a( [(alx)). Let be a constant and
1-7 Then the following holds for all K 0,
nn (No: g))
VK" (ND, g)
+ +([ITT 1lh K+1
K onder expansion
residual
If [| μ|]1 then 45 (xo) lim v (zo) E, E k U (zo)).
5

---

Sample-based estimates to Taylor expansions of value functions. As shown in Equation (6), U (zo) is an unbiased estimate to UK (æo). We can naturally define the sample-based estimate to the -order Taylor expansion, called the TayPO-K estimate,
Vi (ro) = Σ Ug (zo).
The expression of TT (2o) contains O(TK) terms if the trajectory is of length T. Please refer to Appendix E for further details computing the estimates in linear time OT)) with sub-sampling. Note that V (xo) is a sample-based estimate whose bias against the value function controlled by the residual term which decays exponentially when To and μ are close. Similar to how we derived the differentiate through the TayPO-K value estimate to produce estimates higher-order derivatives V V750 (mr)
unbiased estimate V V (I),
Bias and variance Hessian estimates. TayPO-K trades-off bias and variance with choices of K. To understand the variance, note that TayPO-K K limits the number multiplicative IS ratios to be K. Though it difficult to compute the variance, we argue that the variance generally increases with K the number of IS ratios increase [26, 27, 44J. We characterize the bias of TayPO-K follows.
Proposition 3.4. Assume (A.1) and (A.2) hold. Also assume || - -wll v = (1 - 7/// For any tensor x, define ||= = max; [z]]]] The -ordrr TayPO objective produces the following bias in estimating high-order derivatives,
E
2 K
(an)
<
(9)
k=K+1
Hence the upper bound for the bias decreases as K increases. Importantly, when on-policy μ = Te. the -order TayPO objective preserves up to -order derivatives for any K 2 0,
V V (20)
V (zo), Vm S K.
(10)
Though IS ratios of evaluate to 1 when on-policy, they maintain the parameter dependencies in differentiations. As such, higher-order expansions contains products of IS ratios of higher orders, and maintains the high-order dependencies on parameters more accurately. There is clear trade-off between bias and variance mediated by K. When K increases, the higher-order derivatives are maintained more accurately in expectation, leading to less bias. However, the variance increases too.
Recovering prior work special cases. Recently, [24] proposed a low variance curvature (LVC) Hessian estimate. This estimate is equivalent to the TayPO-K estimate with K = 1. As also noted by [24], their objective function bears similarities first-order policy search algorithms [28, 45, 46], which have in fact been interpreted as first-order special cases of V.Vm (x)) with K 1 [29]. Importantly, based on Proposition 3.4, the LVC estimate only maintains the first-order dependency perfectly but introduces bias when approximating the Hessian, even when on-policy.
Limitations. Though the above framework interprets large number of prior methods as special cases, it has some limitations. For example, the derivation of Hessian estimates based on the DR estimate (Proposition 3.2) involves estimates of the value function ViTe (3t). In practice, when near on-policy wo μu, one might replace the DR estimate T by other value function estimate, such as plain cumulative sum of returns or TD(A), VTT((A) [22, 23] in Eqn (4). As such, the practical implementation might not strictly adhere to the conceptual framework. In addition, TMAML [20] is not incorporated as part of this framework: we show in Appendix B that the control variate introduced by TMAML in fact biases the overall estimate.
From Hessian estimates to meta-gradient estimates
practical desiteratum for meta-gradient estimates is that can be implemented scalable way using auto-differentiation frameworks [30, 19]. Below, discuss how this can be achieved.

---

4.1 Auto-differentiating off-policy evaluation estimates for Hessian estimates
In practice, we seek Hessian estimates that could be implemented with the help of an established framework, such as auto-differentiattio libraries [30, 19]. Now we discuss how to conveniently implement ideas discussed in the previous section.
Algorithm 2 Example: off-policy evaluation subroutine for computing the DR estimate
Require: Inputs: Trajectory (Ie, at, rz) target policy To. behavior policy μa. (optional) critic Q. Initialize V Q2 TT(x))).)) T fort=
1, do
Compute IS ratio pf na (2)) p()
Recursion: F < Q To (a;), g) + tpp + (((())))(0))))0))))) Q(2,, tpf.
end for
Output V as an estimate to V*a (xo.g).
Auto-differentiating the estimates. We can abstract the off-policy evaluation as a function eval(D, 0) that takes in some data 1 and parameter 0, and outputs an estimate for V (xo.g)) In particular, D) includes the trajectories and / is imput via the policy Te. As an example, Algorithm shows that for the doubly-robust estimator, the dependency of the estimator on 0 is built through the re- cursive computation by eval(D,0) 0). In fact, if we implement Algorithm 2 with an auto-differentiation framework (for example, Tensorflow or PyTorch [17-19]), the higher-order dependency of V on 0 is maintained through the computation graph. We can compute the Hessian estimate by directly differentiating through the function output V V = V eval (D, 0) Rs V V (xr.. g). In Appendix we show how the estimates could be conveniently implemented, as in many deep RL agents (see, e.g., [44, 5, 47, 48D. We also show concrete ways to compute estimates with TayPO-K K based on [29].
4.2 Practical implementations of meta-gradient estimates
In Equation (2), we write the meta-gradien estimate as product between an Hessian estimate and a policy gradient. In practice, the meta-gradiem estimate is computed via Hessian-vector products avoid explicitly computing the Hessian estimate of size D As such, the veta-gradient estimate could be computed by auto nutoodiiferrnntating through a scalar objective. See Appendix F for details.
Bias and variance meta-gradient estimates. Intuitively, the bias and variance of the Hessian estimates translate into bias and variance of the downstream meta-gradient estimates. Prior work has showed that low variance of meta-gradient estimates lead to faster convergence [49, 50]. However, it is not clear how the bias (such bias introduced by the Hessian estimates, the bias due to correlated estimates) theoretically impacts the convergence. We will study empirically the effect of bias and variance in experiments, and leave further theoretical study to future work.
5 Experiments
We now carry out several empirical studies to complement the framework developed above. In Section 5.1, we use a tabular example to investigate the bias and variance trade-offs of various estimates, to assess the validity of our theoretical insights. We choose tabular example because it is straightforward to compute exact higher-order derivatives of value functions and make comparison. In Section 5.2. and Section 5.2.2, we apply the new second-order estimate in high-dimensional meta-RL RL experiments, to assess the potential performance gains in more practical setup. Though can compute TayPO-K order estimates for general K, we focus on K < 2 in experiments. Below, we also address TayPO-1 and TayPO-2 estimates as the first and second-orded estimates respectively.
5.1
Investigating the bias and variance trade-off of different estimates
We study the bias and variance trade-off of various estimates using tabular exmaple. We consider random MDPs with | 10, |A| = 5. The transition matrix of the MDPs are sampled from Dirichlet distribution. See Appendix for further details. The policy To is parameterized as To(a[z) = cxp(6(x, a))/ E, (xp(8(z. b)). The behavior policy is uniform and B is initialized so that (x, a) = log ग (a|x) where π (1 - =)) + ema for some deterministic policy a. and parameter & (0, 1). The hyper-parameter measures the off aff-policcnnes. In this example, there is no task variable. As performance metries, we measure the component-wise correlation between the true
derivatives V ya (zo) (computed via an oracle) where zg is a fixed starting state, and its estimates
7

---

Corolation
Corrolation
Returre
Momse DEOO
crpeeddrrio LSC)
FINN: SLAME
off policyntos
of
pollccnnoo
Sampleize(kg00)
Borntioo
(a) Gradient off policy
(b) Hessian off-policy
(c) Hessian sample size (d) 2-D control: training
Figure Fig (a)-(b): performance measure as function of off- off-poeicyness measured by
I 1ll1 (a) shows results for gradient estimation and (b) shows Hessians. Plots show the accuracy measure between the estimates and the ground truth. Overall, second-order estimate achieves better bias and variance trade-off. Fig (c): performance measure function of off-policyness measured by sample size N for Hessians. Fig (d): training curves for the 2-D control environment. The second-order estimate is generally more robust. All curves are averaged over 10 runs. The left three plots use share the same legends.
V Vi (Iro), commonly used in prior work [21-23]. The estimates are averaged across N samples. We study the effect of different choices of the off-policy estimate 77 as function off off-policyness E and sample size N. We report results with mean + standard error over 10 seeds.
Effect of off-policyness. In Figure ((a) and (b), we let function of When increases (more off-policy), the accuracy measures of most estimates decrease. This is because off off-policyness generally increases both bias and variance (large ratios ,²)) At this level of sample size, the performance of the second-order estimate degrades more slowly than other estimates, making it the dominating estimate across values of 8. We also include truncated DR estimate using min(pl P)
1000 and show the performance as
baseline inspired by V-trace [44, 47]. The truncation is motivated by controlling the variance of the overall estimate. However, the estimate is heavily biased and does not perform well unless x p. See Appendix F for more.
Consider the case when 0 and the setup is on-policy. When estimating the policy gradient, almost all estimates converge to the optimal level of accuracy, except for the step-wise IS estimate, where the variance still renders the performance sub-optimal. However, when estimating the Hessian matrix, the first-order estimate converges to a lower accuracy than both second-orde estimate and DR estimate. This validates our theoretical analysis, both the second-order estimate and DR estimate are unbiased when on-policy.
Effect of sample size. Figure ((c) shows the accuracy measures function of sample size N when fixing E = 0.5. Note that since sample sizes directly influence the variance, the results show the variance properties of different estimates. When N is small, the first-order estimate dominates due smaller variance; however, when N increases, the first-order estimate is surpassed by the second-order estimate, due to higher bias. For more results and ablation high-dimensional environments, Appendix G.
5.2 Hiig-ddimensional meta-RI, problems
Next, we study the practical gains entailed by the second-order estimate high-dimensional meta-RI. problems. We first introduce few important algorithmic baselines, and how the second-order estimate is incorporated into meta-RI. algorithm.
Baseline algorithms. All baseline algorithms plain stochastic gradient ascent the inner loop optimizer: 8' + v/ Ve (o (Zo. g) where ~ Pg is a sampled goal and 77 (2o. g) is sample-based estimate of policy gradients averaged over trajectories. Different algorithms differ in how the inner loop loss implemented, such that auto-differentiation produces different Hessian estimates: these include TRPO-DiCE [21-23], TRPO-MAMI. [4], TRPO-FMAMI. [4], TRPO-EMAMI. [32, 33]. Please refer to Appendix for further details. Note despite the name, TRPO-DiCO baseline uses DR estimate to estimate Hessians. We implement the second-order

---

estimate using the proximal meta policy search (PROMP) [24] as the base algorithm. By default, PROMP uses the first-order estimate. Our new algorithm named PROMP- -ayP--2
5.2.1 Continuous control in 2D environments
Environment. We consider simple 2-D navigation task introduced in [24]. The state I, is the coordinate of a ball placed inside a room, the action a, is the direction which to push the ball. The goal g I+ is one-hot encoding of which corner of the room contains positive rewards. With 3 adaptation steps, the agent should ideally navigate to the desired corner indicated by g.
Training performance. In Figure I(d), show the performance curves of various baseline algorithms. Though MAML and EMAML learns quickly during the initial phase of training, they ultimately become unstable. TRPO-DiCE generally underperforms other methods potentially due to bias. On the other hand, FMAML and PROMP both reduce variance at the cost bias, but they both achieve slightly lower level of performance compared to PROMP-TayPO- Overall, PROMP-TAYPO-2 achieves much more stable training curves compared to others, potentially owing to the better bias and variance trade-off in the Hessian estimates.
5.2.2 Large scale locomotion experiments
Environments. We consider the set of meta-RL tasks based on simulated locomotion in MuJoCo [51]. Across these tasks, the states 24 consist of robotic sensory inputs, the actions de are torque controls applied to the robots. The task y is defined per environment: for example, in random goal environment, y E IR2 a random 2-d goal location that the robot should aim to reach.
Experiment setup. We adapt the open source code base by [24] and adopt exactly the same experimental setup as [24]. At each iteration, the agent samples n = 40 task variables. For each task, the agent carries out K = 1 adaptation computed based on B 20 trajectories sampled from the environment, each of length 7 = 100. See Appendix for further details on the architecture and other typer-parameters We report averaged results with mean + std over 10 seeds.
Tertume
PRanae Tuhs-1 FIEN
Recclionn
Eoccaioons
Exccalions
Excalooue
(a) Ant goal
(b) HalfCheetah
(c) Ant
(d) Walker2D
Figure 2: Comparison of baselines over a range of simulated locomotion tasks. For task (b)-(d), the goal space consists of 2-d random direction in which the robot should run obtain positive rewards. For task (a), the goal space is a 2-d location on the plane. Each curve shows the mean + std across 5 seeds. Overall, the second-order estimate achieves marginal gains over the first-order estimate.
Results. The training performance of different algorithmic baselines are shown in Figure 2. Com- paring TRPO-DiCE, TRPO-MAML and PROMP: reported in [24], where PROMP outperforms TRPO-MAML, while TRPO-DiCE generally per- forms the worst potentially due to high variance in the gradient estimates. As side observation, TRPO-FMAMI, generally underperforms TRPO-MAMI which implies the necessity of carrying out approximations to the Hessian matrix beyond the identity matrix. PROMP- -ayP22 slightly outperforms PROMP in a few occasions, where the new algorithm achieves slightly faster learning speed and sometimes higher final performance. However, overall, see that the empirical gains are marginal. This implies that under the default setup of these meta-RL experiments, the variance might be a major factor in gradient estimates, and the first-order estimate near optimal compared to other estimates. See Appendix G for additional experiments.
that the results are compatible with those
6 Conclusion
We have unified number of important prior work on meta reta-gradient estimations for model-agnostic mL. Our analysis entails the derivations of prior methods based on the unifying framework of

---

differentiating through off- off-po evaluation estimates. This framework provides principled way to reason about the bias and variance the higher-order derivative estimates of value functions, and opens the door to new family of estimates based on novel off- off-policy evaluation estimates. As an important example, we have theoretically and empirically studied the properties of the family of TayPO-based estimates. It is worth noting that this framework further suggests any future advances ooolpo evaluation could be conveniently imported into potential improvements in meta-gradient estimates. As future work, we hope to see the applications of such principled estimates in broader meta-RL applications.
The authors thank David Abel for reviewing an early draft of this work and providing very useful feedback. Yunhao and Tadashi are thankful for the Scientific Computation and Data Analysis section at the Okinawa Institute of Science and Technology (OIST), which maintains cluster we used for many of our experiments.
Acknowledgement.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
[2] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
[3] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-1373, 2016.
[4] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1135 PMLR, 2017.
[5] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando De Freitas. Dueling network architectures for deep reinforcement learning. arXiv preprint 15X 2015.
[6] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. R12: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 161 1611022777 2016.
[7] Rein Houthooft, Richard Y Chen, Phillip Isola, Bradly Stadie, Filip Wolski, Jonathan Ho, and Pieter Abbeel. Evolved policy gradients. arXiv preprint 1rXiv:1802.04882 2018.
[8] Junhyuk Oh, Matteo Hessel, Wojciech M Czarnecki, Zhongwen Xu, Hado van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. arXiv preprint arXiv:2007.0877 2020.
[9] Zhongwen Xu, Hado van Hasselt, Matteo Hessel, Junhyuk Oh, Satinder Singh, and David Silver. Meta-gradient reinforcement learning with an objective discovered online. arXiv preprint arXiv:2007.08433, 2020.
[10] Kate Rakelly, Aurick Zhou, Chelsea Fino, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcemment learning via probabilistic context variables. In International conference on machine learning, pages 5331-5340. PMLR, 2019.
[11] Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: very good method for bayes-adaptive deep via meta- learning, arXiv preprint arXiv:1910.088 2019.
[12] Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning, arXiv preprint 10XX:1190000125,, 2019.
[13] Zhongwen Xu, Hado van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in neural information processing systems, pages 2396-2407, 2018.
10

---

[14] Tom Zahavy, Zhongwen Xu, Vivek Veeriah, Matteo Hessel, Junhyuk Oh, Hado van Has- selt. David Silver, and Satinder Singh. A self-tuning actor-critic algorithm. arXiv preprint arXiv:2002. /2928, 2020.
[15] Pedro A Ortega, Jane X Wang, Mark Rowland, Tim Genewein, Zeb Kurth-Nelson, Razvan Pascanu, Nicolas Heess, Joel Veness, Alex Pritzel, Pablo Sprechmann, et al. Meta-learning of sequential strategies. arXiv preprint arw.::::05.00000000 030000 2019.
[16] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057-1063, 2000.
[17] Kavosh Asadi and Michael L. Littman. An alternative softmax operator for reinforcement learning. In International Conference on Machine Learning, pages 243-252, 2017.
[18] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Boauderman-Milngg and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.
[19] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[20] Hao Liu, Richard Socher, and Caiming Xiong. Taming maml: Efficient unbiased meta- reinforcement learning. In International Conference on Machine Learning, pages 4061-4071. PMLR, 2019.
[21] Jakob Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktäschel, Eric Xing, and Shimon Whiteson. Dice: The infinitely differentiable monte carlo estimator. In International Conference on Machine Learning, pages 1529-1538. PMLR, 2018.
[22] Gregory Farquhar, Shimon Whiteson, and Jakob Foerster. Loaded dice: Trading off bias and variance in any-order score function gradient estimators for reinforcement learning. 2019.
[23] Jingkai Mao, Jakob Foerster, Tim Rocktäschel, Maruan Al-Shedivat, Gregory Farquhar, and Shimon Whiteson. A baseline for any order gradient estimation in stochastic computation graphs. In International Conference on Machine Learning, pages 4343 4351. PMLR, 2019.
[24] Jonas Rothfuss, Dennis Lee, Ignasi Clavera, Tamim Asfour, and Pieter Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv:1810.06784 2018.
[25] John Schulman, Nicolas Heess, Theophane Weber, and Picter Abbeel. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems, pages 3528 03086,36,2200 2015.
[26] Doina Precup, Richard S Sutton, and Sanjoy Dasgupta. Off-policy temporal-difference learning with function approximation. In ICML, pages 417-424, 2001.
[27] Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning, pages 662-661. PMLR, 2016.
[28] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In ICML, volume 2, pages 267-274, 2002.
[29] Yunhao Tang, Michal Valko, and Rémi Munos. Taylor expansion policy optimization. arXiv preprint arXiv:2003.062559 2020.
[30] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large-scale machine lcarning. In 12th { [UENNIX] symposium on operating systems design and implementation ( ((ODDI) 16), pages 265-283 283,,83,222 2016.
[31] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.

---

[32] Maruan Al-Shedivat, Trapit Bansal, Yuri Burda, Ilya Sutskever, Igor Mordatch, and Pieter Abbeel. Continuous adaptation via meta-learing in nonstationary and competitive environments. arXiv preprini 171.03337100064/1 2017.
[33] Bradly C Stadie, Ge Yang, Rein Houthooft, Xi Chen, Yan Duan, Yuhuai Wu, Pieter Abbeel, and Ilya Sutskever. Some considerations on learning to explore via metfrreinforrement learning. arXiv preprint 1rXiv:1803.01111 111,, 2018.
[34] Vijay R Konda and John N Tsitsiklis. Actor-critic algorithms. In Advances in neural information processing systems, pages 1008-1014. Citeseer, 2000.
[35] Alckh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approxi- mation with policy gradient methods in markov decision processes. In Conference on Learning Theory, pages 66. PMLR, 2020.
[36] Miroslav Dudík, Dumitru Erhan, John Langford, Lihong Li, et al. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485- 2014.
[37] Philip Thomas and Emma Brunskill. Data-efficient off- pol-cy policy evaluation for reinforce- ment learning. In International Conference Machine Learning, pages 2139-2148. PMLR, 2016.
[38] Jiawei Huang and Nan Jiang. From importance sampling to doubly robust policy gradient. In International Conference on Machine Learning. pages 4434-4443. PMLR, 2020.
[39] Hao Liu, Yihao Feng, Yi Mao, Dengyong Zhou, Jian Peng, and Qiang Liu. Action Action-depedent control variates for policy optimization via stein's identity, arXiv preprint 1rXiv:1710.11199 2017.
[40] Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M Bayen, Sham Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. arXiv preprint
2018
[41] George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. In International conference on machine learning, pages 5015-5024, PMLR, 2018.
[42] Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides a competitive approach to reinforcement learning. arXiv preprint arXiv:/803.07005. 2018
[43] Mark Rowland, Will Dabney, and Rémi Munos. Adaptive trade-offs in off-policy learning. arXiv preprint 191.0:111, 2019.
[44] Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pages 1054-1062, 2016.
[45] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1897, 2015.
[46] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0634f 2017.
[47] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint htt.::/8000050000 2018.
[48] Steven Kapturowski. Georg Ostrovski. John Quan. Remi Munos, and Will Dabney, Recurrent experience replay in distributed reinforcement learning. 2018.
[49] Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostie meta-learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1082-1092. PMLR, 2020.
12

---

[50] Kaiyi Ji, Junjie Yang, and Yingbin Liang, Multi-step agnesste meta-learning: Conver- gence and improved algorithms. arXiv preprint arXiv:2002.07836, 2020.
[51] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In Intelligent Robots and Systems (IROS), 2012 IEEE/RSJ International Conference on, pages 5026-5033. IEEE, 2012.
[52] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
13


Reviews:
[{"review_id": "t_gny7btH6W", "reviewer": "Reviewer_WYwM", "summary": "This paper uses Huang and Jiang (2020) to talk about gradient-based meta learning in reinforcement learning using the language of off-policy evaluation. The logical connection behind those concepts stands from the fact that policy gradient methods in reinforcement learning are in fact instances of the likelihood ratio derivative estimator, which itself stems from important sampling. Proposition 3.2 and 3.3 are obtained from Huang and Jiang. Proposition 3.1 tries to extend their results to talk about higher order derivatives. The author's finally propose to use ideas from Taylor policy optimization from Tang et al. (2020) as a variance reduction method within the doubly robust policy gradient framework of Huang and Jiang.\n\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In my opinion the paper doesn't go deep enough into the individual ideas being put forward. The doubly robust estimator of Huang and Jiang is at the core of this paper.  In the specific context of meta learning, what does it allow us to say or do that we couldn't achieve before? Why is doubly robust necessary? The doubly robust being a variance reduction technique why this VRT in particular and not another one, in the meta learning context? Regarding the Taylor approximation, why is it related to the doubly robust estimator? The paper does not do a convincing job at showing that these two components strongly depend on each other: conceptually, or experimentally. Overall, the paper lacks cohesion and depth.\n\nOn line 83 the notation suggests that the expectation is not taken over x and only over the task g. While technically correct, you don't really have to write the objective as an expectation over the value function. Your objective should simply be the expected discounted or on discounted return, written plainly as a sum of rewards.\n\nIn equation 2, I would suggest using a different notation when you're writing nabla of theta prime. Theta prime should be a function, but the way that it's written suggests that it's a variable instead.\n\nAnother suggestion for notation would be to write \"v of theta\" (instead of a superscript) in equation 2. This way it would be clearer how you have to apply the chain rule later on.\n\nReading more carefully on line 92 and equation 1 now I realize that you assume a fixed initial distribution starting from x0. This is confusing because on line 83 you simply write v of x and it looks like x should be a random variable: your initial state. Perhaps you should state explicitly that you assume (without loss of generality) a given initial state.\n\nOn line 92, the expression the \"Hessian of the value function\" is misleading because really the Hessian that you're taking is that of the expected return objective, not that of the value function.\n\nI think that the unnumbered equation below line 106 is redundant: it simply says that the two quantities are unbiased estimators. Words suffice here. Instead, I would suggest that you explain the nature of the issue. The correlation problem that you're describing comes from taking the expectation over a product of two random variables. The fact that it's not simply the product of the individual expectations and that here is also a covariance term.\n\nI'm confused by proposition 3.1 especially around line 137 when you introduce a new notation for the estimator nabla hat v. What you have in this proposition should be something about a derivative estimator: score function/REINFORCE, reparametrization. Now you don't seem to be defining how you obtain this nabla theta m  v hat, and in that sense proposition 3.1 does not quite stand on its own.\n\nOn line 171 you allude to the fact that the variance of policy learning is essentially due to having a large number of importance sampling terms. While this is a reasonable intuition to have, note that we can show formally that PDIS-type estimators need not have larger variance than let's say stationary or marginal IS.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation of the framework per se are not quite explored. The authors talk about what can be said within that framework (dice) but not quite what it cannot say. Experimentally the paper looks at the performance of the combination of doubly robust plus Taylor approximation as an algorithm of its own. It does not quite try to understand what the doubly robust formulation allows us to address specifically in meta learning. It would be interesting to gain more insights into the value of the proposed method by taking a more ablative approach. Since the authors pitch a framework as the main contribution, performance results matter much less than experimental insights at the conceptual level.  I think that the paper could be restructured around the algorithm and to de-emphasize the framework part. To go all in on the algorithm (which might be how things happened chronologically anyways).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "dpjHFg4-IV1", "reviewer": "Reviewer_DoN5", "summary": "The paper uses the recently developed framework of Taylor expansions for value functions (ref [29] in the paper), expands the results, and shows how they could be applied to Off-Policy Evaluation, and by a non-trivial extension to meta-learning. The main algorthmic contribution of the paper is the application of the ideas to meta-learning, but I feel the main contribution is a conceptual one, further developing the ideas from [29] and drawing theoretical connections between OPE and meta-learning.\n\nThe paper includes an experimental section demonstrating marginal improvement over baselines in several domains.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overview: Despite some flaws, I think this is a good paper and leaning towards acceptance. I think the two main limitations of the paper are:\n(1) It draws very heavily on the theory and novelty of [29] and therefore its own novelty is a bit limited.\n(2) The empirical improvements obtained for meta-learning (the problem the paper sets out to advance) are limited.\nAdditionally, there are some changes that should be made to make the paper clearer, but I think those can be easily fixed for the camera ready if the paper is accepted. Despite the drawbacks, the reason I support acceptance is that I feel the additional contributions on top of the already interesting results of [29] are non-trivial, and this is a potentially interesting research direction which could provide insight into many aspects of RL. For example, viewing meta-learning through the lens of Taylor expansions for value functions allows for a very clear connection and unification of derivatives estimations of value functions and OPE.\n\nA general suggestion for the authors which I feel might improve the overall story of the paper, is to lean more heavily on the theoretical results and their ability to shed light on interesting insights such as the connection between OPE and meta-learning, and OPE to high order derivatives. At least my personal experience reading the paper was expecting significant novelty and improvement of results on meta-learning, which set me up for a disappointment. It was only on a second read that I appreciated the theoretical insights in the first half of the paper more. Additionally, I would set up the connection with [29] more - the authors do a good enough job of properly citing [29], but not enough to make it clear that the reader should be familiar with [29] to fully appreciate this paper. After emphasizing the theory and insights, I think the meta-learning algorithm and result can be presented as a cool implication (this is obviously a very subjective suggestion).\n\nSome more specific comments:\n- In lines 120-122 regarding how the approximation maintains the higher-order dependencies on \\theta. I feel like I'm missing something and the sentence is failing to convey an idea it is trying to. Obviously when you make an approximation in some parameter you maintain higher order terms in the small parameter, and choose how many terms to keep. So this statement seems trivial and I'd like to know if there's something deeper which the authors are trying to convey and I'm missing.\n- In section 3.2 and Proposition 3.1 - Is K a typo and should be m? (if yes, it is a consistent typo, which is why I'm confused). If it isn't a typo, I couldn't see where K is defined, and also can't follow the proof of Proportion 3.1, so this is a point I would really like clarification on.\n- In section 5.1, the authors claim to investigate the bias/variance tradeoff of their method, but only show results for the overall MSE of estimating the derivative. I think the authors should show explicit plots of the decomposed variance and bias of each estimator shown.\n- very minor - the y-labels in Figure 2 a-b are cut.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do a good job of discussing the limitations of their method compared to alternative methods.\n\nThe paper is mostly theoretical and I don't see a potential negative soceital impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 to 6"}, "extra_text": {}}, {"review_id": "IXK7zd28WLl", "reviewer": "Reviewer_eXfG", "summary": "The paper is concerned with meta reinforcement learning. The author(s) introduced a framework for learning high-order derivatives of value functions using off-policy evaluation. Specifically, the author(s) proposed to adopt existing off-policy evaluation methods to derive the value estimator and then take the derivative with respect to the target parameters to construct the estimate. This technique is further adopted to implement meta reinforcement learning. ", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have a few major comments regarding the proposed procedure.\n\n1. The author(s) proposed to adopt importance sampling and doubly-robust estimators to evaluate the value function. These estimators suffer from the curse of horizon, leading to the sub-optimality of the learned optimal policy. Given that the author(s) considered an MDP setting, there are more efficient estimators for value evaluation (see, e.g., https://arxiv.org/abs/1810.12429; https://arxiv.org/abs/1909.05850).\n\n2. Instead of constructing estimators for the value, it might be more efficient to directly evaluate the gradient of the value and its Hessian matrix. In the single agent setting, there are efficient off-policy policy gradient estimators developed in the literature (see, e.g., https://arxiv.org/abs/2002.04014).\n\n3. I think the author(s) focused on offline meta reinforcement learning problems. It remains unclear to me how the proposed method would work when the positivity assumption (e.g., the policy ratio shall be bounded away from infinity) in the offline dataset is violated. In the single agent setting, the pessimism principle is commonly adopted to mitigate this issue. Would it be useful to combine the pessimism principle with the proposed method?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations of their work on Page 6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3h"}, "extra_text": {}}, {"review_id": "GyJOvC8yIKO", "reviewer": "Reviewer_RG8H", "summary": "This paper considers meta-RL with gradient-based adaptations, which relies heavily on the estimation of the Hessians of value functions. Though there are previous approaches that focus on unbiased/biased estimations of Hessian. This paper gives a unified view for estimating higher-order derivatives of value functions, through the lens of off-policy evaluation. This subsumes several previous works, also sheds some light on the bias-variance tradeoff of the estimate. Empirically, the new proposed second-order estimate is also incorporated in meta-RL algorithms to showcase the empirical efficiency. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(Originality): The main contributions I see from this paper are: (1). a unified framework for estimating higher-order derivatives of value functions, through OPE methods. (2). the use of Taylor expansion in the value function, which controls the number of IS ratios in the value estimate, hence sheds some light on the bias-variance trade-off. (3). also a new method arising from this framework.\n\n(Quality): The motivation is clear, and the method is clearly stated. Some comments here include:\n[1]. For the bias-variance trade-off with the Taylor expansion, how to select the optimal $K$? Also, how does this bias-variance trade-off translate into the downstream meta-gradient estimations? Any comments on this?\n[2]. In experiment section 5.1, the bias-variance trade-off examination seems a little bit confusing to me. It seems the argument here is the second-order method achieves a better trade-off through high correlation with the ground truth. Is there any way to empirically examine the bias and variance in this tabular setting?\n[3]. For the continuous control problem, it seems the improvement from the new proposed approach is pretty marginal since the bias might be a dominating issue here. Are there any environments/MDPs/settings that we can expect a larger improvement of it? It would be great to include some discussions/new experiments on this.\n\n(Clarity): The paper is well-written and easy to follow.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are discussed in Section 3, while it seems it does not have any societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The expert reviewers for the most part appreciated the paper and were mostly positive. One issue that arose in discussion is that the intent and contribution of the paper is not sufficiently clearly stated, leading to some potential misunderstandings; the authors should very carefully use the feedback of all reviewers, both positive and negative, to more clearly frame the idea of bridging of meta-learning gradients and OPE. An addition, given the significant limitations of sequential IS (including as part of DR), the authors should offer a clear discussion of more efficient methods for OPE and gradient estimation such as pointed out by the reviewers and what might the implications of these potentially be in the authors' setting; this is an important discussion even if fully studying these implications in depth may be beyond the scope of the paper. The authors should similarly address the other points raised in the reviews and implement fixes suggested in their response.
Decision: Accept (Poster)